Kimi-K2
Kimi-K2
Kimi-K2는 Moonshot AI가 만든 최첨단(State-of-the-Art) MoE 언어 모델로, 활성화된 파라미터 32B와 총 파라미터 1T를 가져요. 이 페이지에서는 SGLang에서 Kimi-K2를 설치·배포·호출하는 방법과 추론 파서, 툴 호출, 벤치마크 결과를 상세히 설명해요.
출처: 문서
본문
1. Model Introduction (모델 소개)
Kimi-K2는 Moonshot AI의 최첨단 MoE 언어 모델로, 활성화 32B, 총 1T 파라미터를 가져요.
모델 변형:
- Kimi-K2-Instruct: 범용 채팅과 에이전트 작업에 최적화된 post-trained 모델. vLLM, SGLang, KTransformers, TensorRT-LLM과 호환돼요.
- Kimi-K2-Thinking: 단계별 추론과 툴 호출을 지원하는 고급 thinking 모델. 네이티브 INT4 양자화에 256k 컨텍스트 윈도우. 복잡한 추론과 다단계 툴 사용에 이상적이에요.
- ROCm 지원: SGLang을 통해 AMD MI300X GPU와 호환돼요(검증됨).
2. SGLang Installation (SGLang 설치)
공식 SGLang 설치 가이드를 참고해요.
3. Model Deployment (모델 배포)
이 섹션은 빠른 배포에서 성능 최적화까지 단계별로 안내하며, 다양한 수준의 사용자를 위한 내용을 담고 있어요.
3.1 Basic Configuration
인터랙티브 명령 생성기: 아래 설정 선택기를 사용해 하드웨어 플랫폼, 모델 변형, 배포 전략, 기능에 맞는 배포 명령을 자동으로 생성해요.
3.2 Configuration Tips
- 메모리: 각각 ≥140GB인 8개 GPU(H200/B200)가 필요해요.
--context-length 128000을 사용해 메모리를 아껴요. - Expert Parallelism (EP): MoE 처리량을 높이려면
--ep을 사용해요. EP 문서 참고. - Data Parallel (DP): 프로덕션 처리량을 위해
--dp 4 --enable-dp-attention으로 활성화해요. - KV Cache:
--kv-cache-dtype fp8_e4m3로 메모리를 50% 줄여요 (CUDA 11.8+). - 추론 파서: Kimi-K2-Thinking에서 thinking과 content를 분리하려면
--reasoning-parser kimi_k2를 추가해요. - 툴 호출 파서: 구조화된 툴 호출을 위해
--tool-call-parser kimi_k2를 추가해요. - AMD GPU: AMD GPU에서 실행하기 전에
SGLANG_ROCM_FUSED_DECODE_MLA=0을 설정해요.
4. Model Invocation (모델 호출)
4.1 Basic Usage
기본 API 사용법을 참고해요.
4.2 Advanced Usage
4.2.1 Reasoning Parser
Kimi-K2-Thinking용 추론 파서를 활성화해요.
python -m sglang.launch_server \
--model moonshotai/Kimi-K2-Thinking \
--reasoning-parser kimi_k2 \
--tp 8 \
--host 0.0.0.0 \
--port 8000
예제:
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="EMPTY"
)
# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
model="moonshotai/Kimi-K2-Thinking",
messages=[
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
],
temperature=0.6,
max_tokens=2048,
stream=True
)
# Process the stream
has_thinking = False
has_answer = False
thinking_started = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Print answer content
if delta.content:
# Close thinking section and add content header
if has_thinking and not has_answer:
print("\n=============== Content =================", flush=True)
has_answer = True
print(delta.content, end="", flush=True)
print()
출력 예시:
=============== Thinking =================
The user asks: "What is 15% of 240?" This is a straightforward percentage calculation problem. I need to solve it step by step.
Step 1: Understand what "percent" means.
- "Percent" means "per hundred". So 15% means 15 per 100, or 15/100, or 0.15.
Step 2: Convert the percentage to a decimal.
- 15% = 15 / 100 = 0.15
Step 3: Multiply the decimal by the number.
- 0.15 * 240
Step 4: Perform the multiplication.
- 0.15 * 240 = (15/100) * 240
- = 15 * 240 / 100
- = 3600 / 100
- = 36
Alternatively, I can calculate it directly:
- 0.15 * 240
- 15 * 240 = 3600
- 3600 / 100 = 36
Or, break it down:
- 10% of 240 = 24
- 5% of 240 = half of 10% = 12
- 15% of 240 = 10% + 5% = 24 + 12 = 36
I should present the solution clearly with steps. The most standard method is converting to decimal and multiplying.
Let me structure the answer:
1. Convert the percentage to a decimal.
2. Multiply the decimal by the number.
3. Show the calculation.
4. State the final answer.
This is simple and easy to follow.
=============== Content =================
Here is the step-by-step solution:
**Step 1: Convert the percentage to a decimal**
15% means 15 per 100, which is 15 ÷ 100 = **0.15**
**Step 2: Multiply the decimal by the number**
0.15 × 240
**Step 3: Calculate the result**
0.15 × 240 = **36**
**Answer:** 15% of 240 is **36**.
참고: 추론 파서는 모델의 단계별 사고 과정을 캡처해서, 모델이 결론에 어떻게 도달하는지 볼 수 있게 해줘요.
4.2.2 Tool Calling
Kimi-K2-Instruct와 Kimi-K2-Thinking은 툴 호출 기능을 지원해요. 배포 시 툴 호출 파서를 활성화해요.
배포 명령:
python -m sglang.launch_server \
--model moonshotai/Kimi-K2-Instruct \
--tool-call-parser kimi_k2 \
--tp 8 \
--trust-remote-code \
--host 0.0.0.0 \
--port 8000
Python 예제 (Thinking Process 포함):
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="EMPTY"
)
# Define available tools
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city name"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "Temperature unit"
}
},
"required": ["location"]
}
}
}
]
# Make request with streaming to see thinking process
response = client.chat.completions.create(
model="moonshotai/Kimi-K2-Thinking",
messages=[
{"role": "user", "content": "What's the weather in Beijing?"}
],
tools=tools,
temperature=0.7,
stream=True
)
# Process streaming response
thinking_started = False
has_thinking = False
tool_calls_accumulator = {}
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Accumulate tool calls
if hasattr(delta, 'tool_calls') and delta.tool_calls:
# Close thinking section if needed
if has_thinking and thinking_started:
print("\n=============== Content =================\n", flush=True)
thinking_started = False
for tool_call in delta.tool_calls:
index = tool_call.index
if index not in tool_calls_accumulator:
tool_calls_accumulator[index] = {
'name': None,
'arguments': ''
}
if tool_call.function:
if tool_call.function.name:
tool_calls_accumulator[index]['name'] = tool_call.function.name
if tool_call.function.arguments:
tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
# Print content
if delta.content:
print(delta.content, end="", flush=True)
# Print accumulated tool calls
for index, tool_call in sorted(tool_calls_accumulator.items()):
print(f"🔧 Tool Call: {tool_call['name']}")
print(f" Arguments: {tool_call['arguments']}")
print()
출력 예시:
=============== Thinking =================
The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information. Beijing is a major city in China, so I should be able to get weather data for it. The location parameter is required, but the unit parameter is optional. Since the user didn't specify a temperature unit, I can just provide the location and let the function use its default. I'll check the weather in Beijing for you.
=============== Content =================
🔧 Tool Call: get_weather
Arguments: {"location":"Beijing"}
참고:
- 추론 파서는 모델이 툴을 어떻게 사용하기로 결정하는지 보여줘요
- 툴 호출은 함수 이름과 인자로 명확히 표시돼요
- 함수를 실행하고 결과를 다시 보내 대화를 계속할 수 있어요
툴 호출 결과 처리:
# After getting the tool call, execute the function
def get_weather(location, unit="celsius"):
# Your actual weather API call here
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
# Send tool result back to the model
messages = [
{"role": "user", "content": "What's the weather in Beijing?"},
{
"role": "assistant",
"content": None,
"tool_calls": [{
"id": "call_123",
"type": "function",
"function": {
"name": "get_weather",
"arguments": '{"location": "Beijing", "unit": "celsius"}'
}
}]
},
{
"role": "tool",
"tool_call_id": "call_123",
"content": get_weather("Beijing", "celsius")
}
]
final_response = client.chat.completions.create(
model="moonshotai/Kimi-K2-Thinking",
messages=messages,
temperature=0.7
)
print(final_response.choices[0].message.content)
# Output: "The weather in Beijing is currently 22°C and sunny."
5. Benchmark (벤치마크)
5.1 Speed Benchmark (속도 벤치마크)
테스트 환경:
- Hardware: NVIDIA B200 GPU (8x)
- Model: Kimi-K2-Instruct
- sglang version: 0.5.6.post1
ShareGPT_Vicuna_unfiltered 데이터셋에서 SGLang 내장 벤치마킹 도구로 성능 평가를 수행했어요. 이 데이터셋은 실제 대화 데이터를 포함해 실제 사용 시나리오의 성능을 더 잘 반영해요.
5.1.1 Latency-Sensitive Benchmark (지연 민감 벤치마크)
- 모델 배포 명령:
python3 -m sglang.launch_server \
--model-path moonshotai/Kimi-K2-Instruct \
--tp 8 \
--dp 4 \
--enable-dp-attention \
--trust-remote-code \
--host 0.0.0.0 \
--port 8000
- 벤치마크 명령:
python3 -m sglang.bench_serving \
--backend sglang \
--host 127.0.0.1 \
--port 8000 \
--model moonshotai/Kimi-K2-Instruct\
--num-prompts 10 \
--max-concurrency 1
- 테스트 결과:
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 44.93
Total input tokens: 1951
Total input text tokens: 1951
Total input vision tokens: 0
Total generated tokens: 2755
Total generated tokens (retokenized): 2748
Request throughput (req/s): 0.22
Input token throughput (tok/s): 43.42
Output token throughput (tok/s): 61.32
Peak output token throughput (tok/s): 64.00
Peak concurrent requests: 3
Total token throughput (tok/s): 104.74
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 4489.56
Median E2E Latency (ms): 4994.53
---------------Time to First Token----------------
Mean TTFT (ms): 141.22
Median TTFT (ms): 158.28
P99 TTFT (ms): 166.90
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 18.40
Median TPOT (ms): 15.63
P99 TPOT (ms): 39.88
---------------Inter-Token Latency----------------
Mean ITL (ms): 15.78
Median ITL (ms): 15.76
P95 ITL (ms): 16.36
P99 ITL (ms): 16.59
Max ITL (ms): 19.94
==================================================
5.1.2 Throughput-Sensitive Benchmark (처리량 민감 벤치마크)
- 모델 배포 명령:
python3 -m sglang.launch_server \
--model-path moonshotai/Kimi-K2-Instruct \
--tp 8 \
--dp 4 \
--ep 4 \
--enable-dp-attention \
--trust-remote-code \
--host 0.0.0.0 \
--port 8000
- 벤치마크 명령:
python3 -m sglang.bench_serving \
--backend sglang \
--host 127.0.0.1 \
--port 8000 \
--model moonshotai/Kimi-K2-Instruct\
--num-prompts 1000 \
--max-concurrency 100
- 테스트 결과:
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 1000
Benchmark duration (s): 174.11
Total input tokens: 296642
Total input text tokens: 296642
Total input vision tokens: 0
Total generated tokens: 193831
Total generated tokens (retokenized): 168687
Request throughput (req/s): 5.74
Input token throughput (tok/s): 1703.73
Output token throughput (tok/s): 1113.25
Peak output token throughput (tok/s): 2383.00
Peak concurrent requests: 112
Total token throughput (tok/s): 2816.97
Concurrency: 89.60
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 15601.09
Median E2E Latency (ms): 10780.52
---------------Time to First Token----------------
Mean TTFT (ms): 457.42
Median TTFT (ms): 221.62
P99 TTFT (ms): 2475.32
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 97.23
Median TPOT (ms): 85.61
P99 TPOT (ms): 435.95
---------------Inter-Token Latency----------------
Mean ITL (ms): 78.61
Median ITL (ms): 43.66
P95 ITL (ms): 169.53
P99 ITL (ms): 260.91
Max ITL (ms): 1703.21
==================================================
5.2 Accuracy Benchmark (정확도 벤치마크)
5.2.1 GSM8K Benchmark
- 서버 명령
python3 -m sglang.launch_server \
--model-path moonshotai/Kimi-K2-Instruct \
--tp 8 \
--dp 4 \
--trust-remote-code \
--host 0.0.0.0 \
--port 8000
- 벤치마크 명령
python3 -m sglang.test.few_shot_gsm8k --num-questions 200 --port 8000
- 결과:
Accuracy: 0.960
Invalid: 0.000
Latency: 15.956 s
Output throughput: 1231.699 token/s
더 알아보기 (Learn more)
- Kimi-K2 공식 문서 — 모델 상세 및 사용법
- Kimi-K2 기술 보고서 — 아키텍처와 평가
- Expert Parallelism 문서 — EP 설정
- SGLang 기본 사용법 — 기본 API 호출