MiniMax-M2.5
MiniMax-M2.5
MiniMax-M2.5는 MiniMax가 개발한 강력한 언어 모델로, 실사용 생산성을 목표로 코딩·reasoning·agentic 작업·tool use 전반에서 최첨단 성능을 제공해요. 최신 MiniMax 모델 시리즈로서 여러 도메인에서 종합적인 개선을 이루었어요.
출처: 문서
본문
1. Model Introduction
MiniMax-M2.5는 MiniMax가 개발한 강력한 언어 모델로, 실사용 생산성을 위해 만들어졌으며 코딩·reasoning·agentic 작업·tool use 전반에서 최첨단 성능을 제공해요.
MiniMax 모델 시리즈의 최신 버전으로서 MiniMax-M2.5는 여러 도메인에서 종합적인 개선을 이뤄요. 세부 내용은 다음과 같아요:
- 뛰어난 코딩 성능: Droid에서 79.7, OpenCode에서 76.1을 달성해 Opus 4.6(각각 78.9와 75.9)을 능가해요. SWE-bench Verified, SWE-bench Multilingual, SWE-bench-pro, Multi-SWE-bench에서도 강력한 결과를 보여줘요.
- 고급 reasoning: AIME25와 다른 reasoning 벤치마크에서 강력한 성능을 보여주며, 추론 중 견고한 tool use를 제공해요.
- 더 강력한 에이전트: 웹 브라우징(BrowseComp, Wide Search), 정보 검색(RISE), 복잡한 tool use 시나리오(Terminal Bench 2, MEWC, Finance Modeling)를 포함한 agentic 작업에서 탁월해요.
- 실사용 생산성: 실용적인 코딩·데이터 분석·다단계 reasoning 작업에서 강력한 성능을 가진 프로덕션급 워크로드용으로 설계됐어요.
자세한 내용은 공식 MiniMax-M2.5 발표를 참조하세요.
2. SGLang Installation
SGLang은 여러 설치 방법을 제공해요. 하드웨어 플랫폼과 요구사항에 따라 가장 적합한 설치 방법을 선택할 수 있어요.
설치 지침은 공식 SGLang 설치 가이드를 참조하세요.
AMD MI300X/MI325X/MI355X GPU용:
# Docker (AMD MI300X/MI325X)
docker pull lmsysorg/sglang:v0.5.9-rocm720-mi30x
# Docker (AMD MI355X)
docker pull lmsysorg/sglang:v0.5.9-rocm720-mi35x
3. Model Deployment
이 섹션은 다양한 하드웨어 플랫폼과 사용 사례에 최적화된 배포 구성을 제공해요.
3.1 Basic Configuration
상단의 Command Generator를 사용해 하드웨어 플랫폼·배포 전략·기능 구성에 맞는 배포 명령을 자동 생성하세요. 예시 기본 명령(H200, 4-GPU):
python -m sglang.launch_server \
--model-path MiniMaxAI/MiniMax-M2.5 \
--tp 4 \
--trust-remote-code \
--mem-fraction-static 0.85
H200/B200 (allreduce fusion):
SGLANG_USE_FUSED_PARALLEL_QKNORM=1 ... --enable-flashinfer-allreduce-fusion. Blackwell (B200):--moe-runner-backend flashinfer_trtllm_routed --fp8-gemm-backend flashinfer_trtllm --dtype bfloat16. AMD:--kv-cache-dtype fp8_e4m3 --attention-backend triton --ep <TP>(2-GPU는 TP=2, EP=2).
3.2 Configuration Tips
핵심 파라미터:
| Parameter | Description | Recommended Value |
|---|---|---|
--tool-call-parser |
함수 호출 지원용 tool call parser | minimax-m2 |
--reasoning-parser |
thinking 모드용 reasoning parser | minimax-append-think |
--trust-remote-code |
MiniMax 모델 로딩에 필요 | 항상 활성화 |
--mem-fraction-static |
KV cache용 정적 메모리 분율 | 0.85 |
--tp |
텐서 병렬화 크기 | 2 (2-GPU) 또는 4 (4-GPU) 또는 8 (8-GPU) |
--ep |
전문가 병렬화 크기 | 8 (NVIDIA 8-GPU) 또는 EP=TP (AMD) |
--kv-cache-dtype |
KV cache 데이터 타입 (AMD 전용) | fp8_e4m3 |
--attention-backend |
어텐션 백엔드 (AMD 전용) | triton |
하드웨어 요구사항: NVIDIA
- 4-GPU 배포: 4× 고메모리 GPU(예: H200, B200, A100, H100)와 TP=4 필요
- 8-GPU 배포: 8× GPU(예: H200, B200, A100, H100)와 TP=8·EP=8 필요
하드웨어 요구사항: AMD
- 2-GPU 배포: 2× 고메모리 GPU(예: MI300X, MI325X, MI355X)와 TP=2·EP=2 필요
- 4-GPU 배포: 4× GPU(예: MI300X, MI325X, MI355X)와 TP=4·EP=4 필요
- 8-GPU 배포: 8× GPU(예: MI300X, MI325X, MI355X)와 TP=8·EP=8 필요
4. Model Invocation
4.1 Basic Usage
기본 API 사용과 요청 예시는 다음을 참조하세요:
배포 테스트:
시작 후 다음 명령으로 SGLang OpenAI 호환 API를 테스트할 수 있어요:
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "MiniMaxAI/MiniMax-M2.5",
"messages": [
{"role": "system", "content": [{"type": "text", "text": "You are a helpful assistant."}]},
{"role": "user", "content": [{"type": "text", "text": "Who won the world series in 2020?"}]}
]
}'
간단 완료 예시:
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
response = client.chat.completions.create(
model="MiniMaxAI/MiniMax-M2.5",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Who won the world series in 2020?"}
],
max_tokens=1024
)
print(response.choices[0].message.content)
예시 출력:
thinkingThe user asks: "Who won the world series in 2020?" That is a straightforward factual question. The answer: the Los Angeles Dodgers. They won the 2020 World Series, beating the Tampa Bay Rays. The user is presumably expecting that answer.
We must follow the policies. The question is safe: no disallowed content. It's just a factual question. Provide answer.
We must ensure compliance: Use no disallowed content. Should we provide context? Just answer straightforwardly.
The user simply asks "Who won the world series in 2020?" We'll answer: The Los Angeles Dodgers.
No additional relevant info needed, but could elaborate briefly: They beat the Tampa Bay Rays in six games, the series was played in a bubble at Globe Life Field in Arlington, Texas due to COVID-19.
No need for any extra. That's it.
response
The Los Angeles Dodgers won the 2020 World Series, defeating the Tampa Bay Rays in six games.
4.2 Advanced Usage
4.2.1 Reasoning Parser
MiniMax-M2.5는 Thinking 모드를 지원해요. 배포 중 reasoning parser를 켜서 thinking과 content 섹션을 분리하세요:
python -m sglang.launch_server \
--model-path MiniMaxAI/MiniMax-M2.5 \
--tp 4 \
--reasoning-parser minimax-append-think \
--trust-remote-code \
--mem-fraction-static 0.85
Thinking 프로세스 스트리밍
minimax-append-think에서는 thinking 콘텐츠가 content 필드 안의 thinking... response 태그로 감싸져요. 클라이언트 쪽에서 이 태그를 파싱해 thinking과 content 섹션을 분리할 수 있어요:
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="EMPTY"
)
# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
model="MiniMaxAI/MiniMax-M2.5",
messages=[
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
],
temperature=0.7,
max_tokens=2048,
stream=True
)
# Process the stream, separating thinking... response from content
in_think = False
think_printed_header = False
content_printed_header = False
buffer = ""
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
if delta.content:
buffer += delta.content
while buffer:
if in_think:
# Look for closing response tag
end_idx = buffer.find(" response")
if end_idx != -1:
print(buffer[:end_idx], end="", flush=True)
buffer = buffer[end_idx + len(" response"):]
in_think = False
else:
# Still in thinking, print what we have
print(buffer, end="", flush=True)
buffer = ""
else:
# Look for opening thinking tag
start_idx = buffer.find(" thinking")
if start_idx != -1:
# Print any content before thinking
before = buffer[:start_idx]
if before:
if not content_printed_header:
print("=============== Content =================", flush=True)
content_printed_header = True
print(before, end="", flush=True)
buffer = buffer[start_idx + len(" thinking"):]
in_think = True
if not think_printed_header:
print("=============== Thinking =================", flush=True)
think_printed_header = True
else:
# No thinking tag, print as content
if not content_printed_header and think_printed_header:
print("\n=============== Content =================", flush=True)
content_printed_header = True
print(buffer, end="", flush=True)
buffer = ""
print()
출력 예시:
=============== Thinking =================
The user asks: "Solve this problem step by step: What is 15% of 240?" This is straightforward: 15% = 0.15; 0.15*240 = 36. So answer: 36. Provide step-by-step: convert percent to decimal, multiply.
We need to obey policies. There's no policy violation. Just answer. Provide step by step. Should respond with solution.
We can also mention alternative method: 15% = 15/100 = 3/20. Multiply 240 * 3/20 = (240/20)*3 = 12*3 = 36.
Thus answer 36.
We can add step-by-step. That's it.
=============== Content =================
**Step‑by‑step solution**
1. **Convert the percent to a decimal**
\[
15\% = \frac{15}{100}=0.15
\]
2. **Multiply the decimal by the number**
\[
0.15 \times 240 = 36
\]
(You can also think of it as \(15\% = \frac{3}{20}\) and then \(240 \times \frac{3}{20}=12 \times 3 = 36\).)
\[
\boxed{36}
\]
참고: minimax-append-think reasoning parser는 thinking 프로세스를 content 필드 안의 thinking... response 태그에 내장해요. 위 코드는 이 태그를 실시간으로 파싱해 thinking과 content를 분리해 보여줘요.
4.2.2 Tool Calling
MiniMax-M2.5는 tool calling 능력을 지원해요. tool call parser를 켜세요:
python -m sglang.launch_server \
--model-path MiniMaxAI/MiniMax-M2.5 \
--tp 4 \
--tool-call-parser minimax-m2 \
--reasoning-parser minimax-append-think \
--trust-remote-code \
--mem-fraction-static 0.85
Python 예시:
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="EMPTY"
)
# Define available tools
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city name"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "Temperature unit"
}
},
"required": ["location"]
}
}
}
]
# Non-streaming request
response = client.chat.completions.create(
model="MiniMaxAI/MiniMax-M2.5",
messages=[
{"role": "user", "content": "What's the weather in Beijing?"}
],
tools=tools,
temperature=0.7
)
message = response.choices[0].message
# Check for tool calls
if message.tool_calls:
for tool_call in message.tool_calls:
print(f"Tool Call: {tool_call.function.name}")
print(f" Arguments: {tool_call.function.arguments}")
else:
print(message.content)
출력 예시:
Tool Call: get_weather
Arguments: {"location": "Beijing"}
참고:
- tool call은 함수 이름과 인자와 함께
message.tool_calls로 반환돼요 - 그런 다음 함수를 실행하고 결과를 다시 보내 대화를 계속할 수 있어요
Tool Call 결과 처리:
# After getting the tool call, execute the function
def get_weather(location, unit="celsius"):
# Your actual weather API call here
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
# Send tool result back to the model
messages = [
{"role": "user", "content": "What's the weather in Beijing?"},
{
"role": "assistant",
"content": None,
"tool_calls": [{
"id": "call_123",
"type": "function",
"function": {
"name": "get_weather",
"arguments": '{"location": "Beijing", "unit": "celsius"}'
}
}]
},
{
"role": "tool",
"tool_call_id": "call_123",
"content": get_weather("Beijing", "celsius")
}
]
final_response = client.chat.completions.create(
model="MiniMaxAI/MiniMax-M2.5",
messages=messages,
temperature=0.7
)
print(final_response.choices[0].message.content)
# Output: "The weather in Beijing is currently 22°C and sunny."
5. Benchmark
이 섹션은 비교 가능한 벤치마크 결과를 위해 업계 표준 구성을 사용해요.
5.1 Speed Benchmark
테스트 환경:
- 하드웨어: NVIDIA B200 GPU (8x)
- 모델: MiniMax-M2.5
- 텐서 병렬화: 8
- 전문가 병렬화: 8
- sglang 버전: 0.5.8
5.1.1 Standard Scenario Benchmark
- 모델 배포 명령:
sglang serve \
--model-path MiniMaxAI/MiniMax-M2.5 \
--tp 8 \
--ep 8 \
--reasoning-parser minimax-append-think \
--trust-remote-code \
--mem-fraction-static 0.85 \
--tool-call-parser minimax-m2
5.1.1.1 Low Concurrency
- 벤치마크 명령:
python3 -m sglang.bench_serving \
--backend sglang \
--model MiniMaxAI/MiniMax-M2.5 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1
- 테스트 결과:
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 42.99
Total input tokens: 6091
Total input text tokens: 6091
Total generated tokens: 4220
Total generated tokens (retokenized): 3804
Request throughput (req/s): 0.23
Input token throughput (tok/s): 141.70
Output token throughput (tok/s): 98.17
Peak output token throughput (tok/s): 102.00
Peak concurrent requests: 2
Total token throughput (tok/s): 239.87
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 4295.92
Median E2E Latency (ms): 3419.28
P90 E2E Latency (ms): 7832.04
P99 E2E Latency (ms): 9601.40
---------------Time to First Token----------------
Mean TTFT (ms): 130.57
Median TTFT (ms): 116.10
P99 TTFT (ms): 190.90
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 9.89
Median TPOT (ms): 9.89
P99 TPOT (ms): 9.91
---------------Inter-Token Latency----------------
Mean ITL (ms): 9.89
Median ITL (ms): 9.89
P95 ITL (ms): 10.15
P99 ITL (ms): 10.32
Max ITL (ms): 14.46
==================================================
5.1.1.2 Medium Concurrency
- 벤치마크 명령:
python3 -m sglang.bench_serving \
--backend sglang \
--model MiniMaxAI/MiniMax-M2.5 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 80 \
--max-concurrency 16
- 테스트 결과:
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 48.43
Total input tokens: 39588
Total input text tokens: 39588
Total generated tokens: 40805
Total generated tokens (retokenized): 37142
Request throughput (req/s): 1.65
Input token throughput (tok/s): 817.37
Output token throughput (tok/s): 842.49
Peak output token throughput (tok/s): 1184.00
Peak concurrent requests: 21
Total token throughput (tok/s): 1659.86
Concurrency: 13.67
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 8274.32
Median E2E Latency (ms): 8692.90
P90 E2E Latency (ms): 13690.70
P99 E2E Latency (ms): 16104.18
---------------Time to First Token----------------
Mean TTFT (ms): 305.44
Median TTFT (ms): 106.75
P99 TTFT (ms): 1053.26
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 16.20
Median TPOT (ms): 16.06
P99 TPOT (ms): 26.75
---------------Inter-Token Latency----------------
Mean ITL (ms): 15.65
Median ITL (ms): 13.63
P95 ITL (ms): 14.90
P99 ITL (ms): 87.99
Max ITL (ms): 483.53
==================================================
5.1.1.3 High Concurrency
- 벤치마크 명령:
python3 -m sglang.bench_serving \
--backend sglang \
--model MiniMaxAI/MiniMax-M2.5 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 500 \
--max-concurrency 100
- 테스트 결과:
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 500
Benchmark duration (s): 92.31
Total input tokens: 249331
Total input text tokens: 249331
Total generated tokens: 252662
Total generated tokens (retokenized): 218975
Request throughput (req/s): 5.42
Input token throughput (tok/s): 2700.94
Output token throughput (tok/s): 2737.02
Peak output token throughput (tok/s): 4479.00
Peak concurrent requests: 109
Total token throughput (tok/s): 5437.97
Concurrency: 91.19
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 16835.82
Median E2E Latency (ms): 16042.08
P90 E2E Latency (ms): 31027.63
P99 E2E Latency (ms): 34787.91
---------------Time to First Token----------------
Mean TTFT (ms): 391.06
Median TTFT (ms): 133.12
P99 TTFT (ms): 1712.92
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 33.04
Median TPOT (ms): 34.29
P99 TPOT (ms): 41.98
---------------Inter-Token Latency----------------
Mean ITL (ms): 32.61
Median ITL (ms): 21.67
P95 ITL (ms): 87.76
P99 ITL (ms): 118.81
Max ITL (ms): 1145.62
==================================================
5.1.2 Summarization Scenario Benchmark
- 모델 배포 명령:
sglang serve \
--model-path MiniMaxAI/MiniMax-M2.5 \
--tp 8 \
--ep 8 \
--reasoning-parser minimax-append-think \
--trust-remote-code \
--mem-fraction-static 0.85 \
--tool-call-parser minimax-m2
5.1.2.1 Low Concurrency
- 벤치마크 명령:
python3 -m sglang.bench_serving \
--backend sglang \
--model MiniMaxAI/MiniMax-M2.5 \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1
- 테스트 결과:
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 43.49
Total input tokens: 41941
Total input text tokens: 41941
Total generated tokens: 4220
Total generated tokens (retokenized): 4220
Request throughput (req/s): 0.23
Input token throughput (tok/s): 964.42
Output token throughput (tok/s): 97.04
Peak output token throughput (tok/s): 102.00
Peak concurrent requests: 2
Total token throughput (tok/s): 1061.46
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 4346.83
Median E2E Latency (ms): 3508.84
P90 E2E Latency (ms): 7972.23
P99 E2E Latency (ms): 9659.71
---------------Time to First Token----------------
Mean TTFT (ms): 131.50
Median TTFT (ms): 126.76
P99 TTFT (ms): 182.52
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 10.00
Median TPOT (ms): 10.01
P99 TPOT (ms): 10.12
---------------Inter-Token Latency----------------
Mean ITL (ms): 10.01
Median ITL (ms): 10.02
P95 ITL (ms): 10.29
P99 ITL (ms): 10.44
Max ITL (ms): 14.11
==================================================
5.1.2.2 Medium Concurrency
- 벤치마크 명령:
python3 -m sglang.bench_serving \
--backend sglang \
--model MiniMaxAI/MiniMax-M2.5 \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 80 \
--max-concurrency 16
- 테스트 결과:
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 50.12
Total input tokens: 300020
Total input text tokens: 300020
Total generated tokens: 41669
Total generated tokens (retokenized): 41662
Request throughput (req/s): 1.60
Input token throughput (tok/s): 5986.00
Output token throughput (tok/s): 831.38
Peak output token throughput (tok/s): 1152.00
Peak concurrent requests: 20
Total token throughput (tok/s): 6817.38
Concurrency: 13.93
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 8727.66
Median E2E Latency (ms): 9170.52
P90 E2E Latency (ms): 14220.00
P99 E2E Latency (ms): 16896.54
---------------Time to First Token----------------
Mean TTFT (ms): 282.56
Median TTFT (ms): 149.37
P99 TTFT (ms): 1278.62
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 16.60
Median TPOT (ms): 16.61
P99 TPOT (ms): 25.17
---------------Inter-Token Latency----------------
Mean ITL (ms): 16.24
Median ITL (ms): 13.89
P95 ITL (ms): 15.96
P99 ITL (ms): 105.79
Max ITL (ms): 1065.02
==================================================
5.1.2.3 High Concurrency
- 벤치마크 명령:
python3 -m sglang.bench_serving \
--backend sglang \
--model MiniMaxAI/MiniMax-M2.5 \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 320 \
--max-concurrency 64
- 테스트 결과:
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 64
Successful requests: 320
Benchmark duration (s): 93.92
Total input tokens: 1273893
Total input text tokens: 1273893
Total generated tokens: 170000
Total generated tokens (retokenized): 169999
Request throughput (req/s): 3.41
Input token throughput (tok/s): 13563.30
Output token throughput (tok/s): 1810.01
Peak output token throughput (tok/s): 2881.00
Peak concurrent requests: 71
Total token throughput (tok/s): 15373.31
Concurrency: 58.87
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 17277.69
Median E2E Latency (ms): 16827.33
P90 E2E Latency (ms): 29045.40
P99 E2E Latency (ms): 33496.77
---------------Time to First Token----------------
Mean TTFT (ms): 692.26
Median TTFT (ms): 188.46
P99 TTFT (ms): 4932.70
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 32.19
Median TPOT (ms): 32.69
P99 TPOT (ms): 50.46
---------------Inter-Token Latency----------------
Mean ITL (ms): 31.28
Median ITL (ms): 21.59
P95 ITL (ms): 101.35
P99 ITL (ms): 136.74
Max ITL (ms): 4649.23
==================================================
5.1.3 H100 Benchmark
테스트 환경:
-
하드웨어: NVIDIA H100 80GB HBM3 GPU (8x)
-
모델: MiniMax-M2.5
-
텐서 병렬화: 8
-
전문가 병렬화: 8
-
sglang 버전: 0.5.9
-
모델 배포 명령:
sglang serve \
--model-path MiniMaxAI/MiniMax-M2.5 \
--tp 8 \
--ep 8 \
--reasoning-parser minimax-append-think \
--trust-remote-code \
--mem-fraction-static 0.85 \
--tool-call-parser minimax-m2
5.1.3.1 Low Concurrency
- 벤치마크 명령:
python3 -m sglang.bench_serving \
--backend sglang \
--model MiniMaxAI/MiniMax-M2.5 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1
- 테스트 결과:
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 35.44
Total input tokens: 6101
Total input text tokens: 6101
Total generated tokens: 4220
Total generated tokens (retokenized): 4220
Request throughput (req/s): 0.28
Input token throughput (tok/s): 172.16
Output token throughput (tok/s): 119.08
Peak output token throughput (tok/s): 127.00
Peak concurrent requests: 2
Total token throughput (tok/s): 291.24
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 3542.38
Median E2E Latency (ms): 2791.92
P90 E2E Latency (ms): 6317.77
P99 E2E Latency (ms): 7780.15
---------------Time to First Token----------------
Mean TTFT (ms): 145.20
Median TTFT (ms): 80.38
P99 TTFT (ms): 633.08
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 8.05
Median TPOT (ms): 8.08
P99 TPOT (ms): 8.09
---------------Inter-Token Latency----------------
Mean ITL (ms): 8.07
Median ITL (ms): 8.08
P95 ITL (ms): 8.12
P99 ITL (ms): 8.16
Max ITL (ms): 10.10
==================================================
5.1.3.2 Medium Concurrency
- 벤치마크 명령:
python3 -m sglang.bench_serving \
--backend sglang \
--model MiniMaxAI/MiniMax-M2.5 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 80 \
--max-concurrency 16
- 테스트 결과:
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 43.68
Total input tokens: 39668
Total input text tokens: 39668
Total generated tokens: 40805
Total generated tokens (retokenized): 40805
Request throughput (req/s): 1.83
Input token throughput (tok/s): 908.19
Output token throughput (tok/s): 934.22
Peak output token throughput (tok/s): 1184.00
Peak concurrent requests: 20
Total token throughput (tok/s): 1842.42
Concurrency: 13.83
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 7551.91
Median E2E Latency (ms): 8094.28
P90 E2E Latency (ms): 12606.99
P99 E2E Latency (ms): 14977.84
---------------Time to First Token----------------
Mean TTFT (ms): 116.86
Median TTFT (ms): 82.33
P99 TTFT (ms): 240.59
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 14.81
Median TPOT (ms): 14.98
P99 TPOT (ms): 17.98
---------------Inter-Token Latency----------------
Mean ITL (ms): 14.61
Median ITL (ms): 13.50
P95 ITL (ms): 14.15
P99 ITL (ms): 66.52
Max ITL (ms): 107.39
==================================================
5.1.3.3 High Concurrency
- 벤치마크 명령:
python3 -m sglang.bench_serving \
--backend sglang \
--model MiniMaxAI/MiniMax-M2.5 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 500 \
--max-concurrency 100
- 테스트 결과:
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 500
Benchmark duration (s): 80.63
Total input tokens: 249831
Total input text tokens: 249831
Total generated tokens: 252662
Total generated tokens (retokenized): 252331
Request throughput (req/s): 6.20
Input token throughput (tok/s): 3098.45
Output token throughput (tok/s): 3133.56
Peak output token throughput (tok/s): 4800.00
Peak concurrent requests: 113
Total token throughput (tok/s): 6232.01
Concurrency: 90.56
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 14604.59
Median E2E Latency (ms): 14044.04
P90 E2E Latency (ms): 26456.53
P99 E2E Latency (ms): 30136.68
---------------Time to First Token----------------
Mean TTFT (ms): 149.32
Median TTFT (ms): 95.16
P99 TTFT (ms): 374.62
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 28.92
Median TPOT (ms): 30.09
P99 TPOT (ms): 34.31
---------------Inter-Token Latency----------------
Mean ITL (ms): 28.66
Median ITL (ms): 21.52
P95 ITL (ms): 66.90
P99 ITL (ms): 96.76
Max ITL (ms): 376.34
==================================================
5.2 Accuracy Benchmark
5.2.1 GSM8K Benchmark
- 벤치마크 명령:
python benchmark/gsm8k/bench_sglang.py --port 30000
- 테스트 결과:
Accuracy: 0.950
Invalid: 0.000
Latency: 18.033 s
Output throughput: 1130.161 token/s
5.2.2 MMLU Benchmark
- 벤치마크 명령:
cd benchmark/mmlu
bash download_data.sh
python3 bench_sglang.py --port 30000
- 테스트 결과:
subject: abstract_algebra, #q:100, acc: 0.620
subject: anatomy, #q:135, acc: 0.830
subject: astronomy, #q:152, acc: 0.928
subject: business_ethics, #q:100, acc: 0.810
subject: clinical_knowledge, #q:265, acc: 0.891
subject: college_biology, #q:144, acc: 0.951
subject: college_chemistry, #q:100, acc: 0.670
subject: college_computer_science, #q:100, acc: 0.820
subject: college_mathematics, #q:100, acc: 0.660
subject: college_medicine, #q:173, acc: 0.832
subject: college_physics, #q:102, acc: 0.814
subject: computer_security, #q:100, acc: 0.880
subject: conceptual_physics, #q:235, acc: 0.915
subject: econometrics, #q:114, acc: 0.719
subject: electrical_engineering, #q:145, acc: 0.834
subject: elementary_mathematics, #q:378, acc: 0.902
subject: formal_logic, #q:126, acc: 0.698
subject: global_facts, #q:100, acc: 0.710
subject: high_school_biology, #q:310, acc: 0.926
subject: high_school_chemistry, #q:203, acc: 0.793
subject: high_school_computer_science, #q:100, acc: 0.910
subject: high_school_european_history, #q:165, acc: 0.879
subject: high_school_geography, #q:198, acc: 0.955
subject: high_school_government_and_politics, #q:193, acc: 0.964
subject: high_school_macroeconomics, #q:390, acc: 0.908
subject: high_school_mathematics, #q:270, acc: 0.600
subject: high_school_microeconomics, #q:238, acc: 0.954
subject: high_school_physics, #q:151, acc: 0.781
subject: high_school_psychology, #q:545, acc: 0.956
subject: high_school_statistics, #q:216, acc: 0.847
subject: high_school_us_history, #q:204, acc: 0.922
subject: high_school_world_history, #q:237, acc: 0.916
subject: human_aging, #q:223, acc: 0.839
subject: human_sexuality, #q:131, acc: 0.893
subject: international_law, #q:121, acc: 0.934
subject: jurisprudence, #q:108, acc: 0.861
subject: logical_fallacies, #q:163, acc: 0.890
subject: machine_learning, #q:112, acc: 0.750
subject: management, #q:103, acc: 0.883
subject: marketing, #q:234, acc: 0.944
subject: medical_genetics, #q:100, acc: 0.920
subject: miscellaneous, #q:783, acc: 0.936
subject: moral_disputes, #q:346, acc: 0.829
subject: moral_scenarios, #q:895, acc: 0.632
subject: nutrition, #q:306, acc: 0.863
subject: philosophy, #q:311, acc: 0.833
subject: prehistory, #q:324, acc: 0.907
subject: professional_accounting, #q:282, acc: 0.720
subject: professional_law, #q:1534, acc: 0.640
subject: professional_medicine, #q:272, acc: 0.923
subject: professional_psychology, #q:612, acc: 0.871
subject: public_relations, #q:110, acc: 0.773
subject: security_studies, #q:245, acc: 0.845
subject: sociology, #q:201, acc: 0.930
subject: us_foreign_policy, #q:100, acc: 0.940
subject: virology, #q:166, acc: 0.614
subject: world_religions, #q:171, acc: 0.895
Total latency: 81.468
Average accuracy: 0.825