DeepSeek V3.1
DeepSeek V3.1
이 문서는 DeepSeek이 개발한 고급 Mixture-of-Experts(MoE) 대규모 언어 모델인 DeepSeek V3.1을 SGLang으로 배포하고 호출하는 방법을 설명해요. DeepSeek V3 대비 주요 기능·사용성 업그레이드 버전으로, 빠른 비사고(non-thinking) 응답과 명시적 다단계 추론을 모두 지원하는 하이브리드 추론 패러다임을 도입했어요. 원문 페이지에는 하드웨어 플랫폼과 배포 전략을 골라 명령을 자동 생성해 주는 대화형 선택기가 포함되어 있어요.
출처: 문서
본문
1. 모델 소개
DeepSeek V3.1은 DeepSeek이 개발한 고급 Mixture-of-Experts(MoE) 대규모 언어 모델로, DeepSeek V3 대비 주요 기능·사용성 업그레이드를 나타내요. DeepSeek V3 계열의 정제된 반복 버전으로서, DeepSeek V3.1은 빠른 비사고 응답과 명시적 다단계 추론을 모두 지원하는 하이브리드 추론 패러다임을 도입했고, 크게 개선된 도구 호출과 에이전트 동작을 제공해요. 이 모델은 지속적인 훈련, 정렬 최적화, 추론 시점 개선 덕분에 추론, 수학, 코딩, 장문 맥락 이해, 실제 에이전트 워크플로우 전반에서 강력한 성능을 보여줘요. DeepSeek V3.1은 견고한 범용 기반 모델로 설계되어 대화형 AI, 구조화된 도구 호출, 검색 보강 생성, 복잡한 다단계 작업에 적합하며, 희소 MoE 아키텍처를 통해 높은 효율을 유지해요.
**DeepSeek-V3.1-Terminus**는 일반 대화와 장문 맥락 처리를 위해 설계된 실험 버전이에요. 의도적인 추론을 위한 "Think" 모드와 더 빠른 응답을 위한 "Non-Think" 모드 사이를 전환할 수 있는 하이브리드 사고 기능을 갖추고 있어요. 일반 대화, 장문 맥락 처리, 실험적 사용 사례에 권장돼요.
2. SGLang 설치
SGLang은 여러 설치 방법을 제공해요. 하드웨어 플랫폼과 요구 사항에 가장 적합한 설치 방법을 선택할 수 있어요.
설치 지침은 공식 SGLang 설치 가이드를 참고하세요.
SGLang CPU 설치는 CPU 버전 설치 가이드를 참고하세요.
3. 모델 배포
이 섹션은 빠른 배포에서 성능 최적화까지 이어지는 점진적인 가이드를 제공하며, 다양한 수준의 사용자에게 적합해요.
3.1 기본 설정
대화형 명령 생성기 (Interactive Command Generator): 아래 설정 선택기를 사용해 하드웨어 플랫폼, 모델 변형, 배포 전략, 사고 능력에 맞는 배포 명령을 자동으로 생성할 수 있어요. (대화형 위젯은 원문 페이지에서 동작하므로, 기본 설정 기준 명령은 다음과 같아요.)
python -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-V3.1 \
--tp 8
3.2 설정 팁
DeepSeek-V3.1은 DeepSeek-V3와 동일한 모델 아키텍처를 공유하므로, 동일한 하드웨어·최적화 권장 사항이 적용돼요.
가중치 유형별 권장 GPU 설정:
| 가중치 유형 | 지원 하드웨어 |
|---|---|
| FP8 (권장) | 8× H200, 8× B200, 8× MI300X, 2×8× H100/H800/H20 |
| BF16 (FP8에서 upcast) | 2×8× H200, 2×8× MI300X, 4×8× H100/H800, 4×8× A100/A800 |
| INT8 | 16× A100/A800, 32× L40S, Xeon 6980P CPU, 4× Ascend A3 Series |
| W4A8 / AWQ / MXFP4 / NVFP4 | 8× H20/H100, 4× H200; 8× H100/A100; 8/4× MI355X/MI350X; 8/4× B200 |
공식 DeepSeek-V3.1 체크포인트는 이미 FP8 형식이에요 — 서빙할 때
--quantization fp8을 추가하지 마세요.
DeepGEMM 사전 컴파일 (NVIDIA Hopper / Blackwell): 첫 서버 실행 전에 GEMM 커널을 사전 컴파일해 JIT 오버헤드(약 10분)를 피하세요:
python3 -m sglang.compile_deep_gemm --model deepseek-ai/DeepSeek-V3.1 --tp 8 --trust-remote-code
DeepGEMM은 Hopper/Blackwell에서 기본적으로 활성화되며 SGLANG_ENABLE_JIT_DEEPGEMM=0으로 비활성화할 수 있어요.
데이터 병렬화 어텐션 (--enable-dp-attention): 큰 배치 크기의 높은 처리량 시나리오에 권장돼요. TP 랭크 간 KV-cache 중복을 줄여줘요. 단일 8-GPU 노드에서는 --enable-dp-attention --tp 8 --dp 8을 사용하세요. 저지연·소배치 워크로드에는 권장되지 않아요.
NCCL 타임아웃: 모델 로딩이 느리고 NCCL 타임아웃이 발생하면 늘리세요: --dist-timeout 3600.
Xeon CPU 서비스 설정: SGLang CPU 서버 문서의 서빙 엔진 실행 부분에 있는 Notes 항목을 참고해 인자, 특히 TP(텐서 병렬) 및 NUMA 바인딩 설정을 이해하세요.
4. 모델 호출
4.1 기본 사용법
기본 API 사용법과 요청 예시는 다음을 참고하세요:
4.2 고급 사용법
4.2.1 추론 파서 (Reasoning Parser)
DeepSeek-V3.1은 추론 모드를 지원해요. 배포 중 reasoning parser를 활성화해 thinking 섹션과 content 섹션을 분리하세요:
python -m sglang.launch_server \
--model deepseek-ai/DeepSeek-V3.1-Terminus \
--reasoning-parser deepseek-v3 \
--tp 8 \
--host 0.0.0.0 \
--port 8000
사고 과정이 포함된 스트리밍:
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="EMPTY"
)
# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-V3.1-Terminus",
messages=[
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
],
temperature=0.7,
max_tokens=2048,
extra_body = {"chat_template_kwargs": {"thinking": True}},
stream=True
)
# Process the stream
has_thinking = False
has_answer = False
thinking_started = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Print answer content
if delta.content:
# Close thinking section and add content header
if has_thinking and not has_answer:
print("\n=============== Content =================", flush=True)
has_answer = True
print(delta.content, end="", flush=True)
print()
출력 예시:
=============== Thinking =================
First, the problem is asking for 15% of 240. Percent means per hundred, so 15% is the same as 15 out of 100, or 15/100.
To find a percentage of a number, I can multiply the number by the percentage expressed as a decimal. So, I need to convert 15% to a decimal. To do that, I divide 15 by 100, which gives me 0.15.
Now, I multiply 0.15 by 240. So, the calculation is 0.15 × 240.
I can compute this step by step. First, I know that 15% of 100 is 15, but since 240 is larger, I need to adjust. Alternatively, I can think of 10% of 240, which is easy because 10% is just 240 divided by 10, which is 24. Then, 5% is half of 10%, so half of 24 is 12. Therefore, 15% is 10% plus 5%, so 24 plus 12, which equals 36.
I should also do the multiplication to confirm. 0.15 × 240. I can break it down: 0.15 × 200 = 30, and 0.15 × 40 = 6, so 30 + 6 = 36. Same answer.
So, 15% of 240 is 36.
The problem says "step by step," so I should present it clearly.
=============== Content =================
To find 15% of 240, follow these steps:
1. Understand that "percent" means "per hundred," so 15% is equivalent to \( \frac{15}{100} \).
2. Convert 15% to a decimal by dividing by 100: \( 15\% = \frac{15}{100} = 0.15 \).
3. Multiply the decimal by 240: \( 0.15 \times 240 \).
4. Perform the multiplication:
- \( 0.15 \times 200 = 30 \)
- \( 0.15 \times 40 = 6 \)
- Add the results: \( 30 + 6 = 36 \).
Alternatively, you can find 15% by breaking it into parts:
- 10% of 240 is \( \frac{10}{100} \times 240 = 0.10 \times 240 = 24 \).
- 5% of 240 is half of 10%, so \( \frac{24}{2} = 12 \).
- Add 10% and 5%: \( 24 + 12 = 36 \).
Thus, 15% of 240 is 36.
참고: reasoning parser는 모델의 단계별 사고 과정을 캡처해 모델이 결론에 도달하는 방식을 볼 수 있게 해줘요.
4.2.2 도구 호출 (Tool Calling)
DeepSeek-V3.1과 DeepSeek-V3.1-Terminus는 도구 호출 기능을 지원해요. tool call parser를 활성화하세요:
배포 명령:
python -m sglang.launch_server \
--model deepseek-ai/DeepSeek-V3.1-Terminus \
--tool-call-parser deepseekv31 \
--reasoning-parser deepseek-v3 \
--chat-template ./examples/chat_template/tool_chat_template_deepseekv31.jinja \
--tp 8 \
--host 0.0.0.0 \
--port 8000
DeepSeek-V3.1에도 --tool-call-parser deepseekv31을 사용하세요.
Python 예시 (사고 과정 포함):
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="EMPTY"
)
# Define available tools
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city name"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "Temperature unit"
}
},
"required": ["location"]
}
}
}
]
# Make request with streaming to see thinking process
response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-V3.1-Terminus",
messages=[
{"role": "user", "content": "What's the weather in Beijing?"}
],
tools=tools,
extra_body = {"chat_template_kwargs": {"thinking": True}},
temperature=0.7,
stream=True
)
# Process streaming response
thinking_started = False
has_thinking = False
tool_calls_accumulator = {}
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Accumulate tool calls
if hasattr(delta, 'tool_calls') and delta.tool_calls:
# Close thinking section if needed
if has_thinking and thinking_started:
print("\n=============== Content =================\n", flush=True)
thinking_started = False
for tool_call in delta.tool_calls:
index = tool_call.index
if index not in tool_calls_accumulator:
tool_calls_accumulator[index] = {
'name': None,
'arguments': ''
}
if tool_call.function:
if tool_call.function.name:
tool_calls_accumulator[index]['name'] = tool_call.function.name
if tool_call.function.arguments:
tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
# Print content
if delta.content:
print(delta.content, end="", flush=True)
# Print accumulated tool calls
for index, tool_call in sorted(tool_calls_accumulator.items()):
print(f"🔧 Tool Call: {tool_call['name']}")
print(f" Arguments: {tool_call['arguments']}")
print()
출력 예시:
=============== Thinking =================
Hmm, the user is asking for the weather in Beijing. This is a straightforward request that matches exactly what the weather tool can provide.
I need to call the get_weather function with Beijing as the location parameter. The user didn't specify a temperature unit, so I'll default to Celsius since that's commonly used in most parts of the world.
The tool call format needs to be precise - just the city name and unit selection. Once I get the weather data back, I'll present it clearly to the user.I'll check the weather in Beijing for you.
=============== Content =================
🔧 Tool Call: get_weather
Arguments: {"location": "Beijing", "unit": "celsius"}
참고:
- reasoning parser는 모델이 도구를 어떻게 사용하기로 결정하는지 보여줘요
- 도구 호출은 함수 이름과 인자로 명확하게 표시돼요
- 그런 다음 함수를 실행하고 결과를 다시 보내 대화를 이어갈 수 있어요
도구 호출 결과 처리:
아래 코드 블록을 이전 Python 스크립트에 붙이세요.
# After getting the tool call, execute the function
def get_weather(location, unit="celsius"):
# Your actual weather API call here
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
# Send tool result back to the model
messages = [
{"role": "user", "content": "What's the weather in Beijing?"},
{
"role": "assistant",
"content": None,
"tool_calls": [{
"id": "call_123",
"type": "function",
"function": {
"name": "get_weather",
"arguments": '{"location": "Beijing", "unit": "celsius"}'
}
}]
},
{
"role": "tool",
"tool_call_id": "call_123",
"content": get_weather("Beijing", "celsius")
}
]
final_response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-V3.1-Terminus",
messages=messages,
temperature=0.7
)
print(final_response.choices[0].message.content)
# Output: "Currently, it is **22°C and sunny** in Beijing."
4.2.3 다중 토큰 예측 (Multi-Token Prediction, EAGLE 추측 디코딩)
DeepSeek-V3.1은 DeepSeek-V3와 동일한 아키텍처를 공유하며 동일한 EAGLE 기반 MTP 추측 디코딩 경로를 지원해요. 전체 설정, 튜닝 가이드 및 bench_speculative.py 참고 자료는 DeepSeek-V3 §4.2.3을 참고하세요. --speculative-num-steps, --speculative-eagle-topk, --max-running-requests 권장 사항은 V3.1에도 동일하게 적용돼요.
5. 벤치마크
5.1 속도 벤치마크
테스트 환경:
- 하드웨어: AMD MI300X GPU (8x)
- 모델: DeepSeek-V3.1-Terminus
- 텐서 병렬화(Tensor Parallelism): 8
- sglang 버전: 0.5.7
벤치마크 방법론:
결과가 프레임워크와 하드웨어 플랫폼 전반에서 비교 가능하도록 업계 표준 벤치마크 설정을 사용해요.
5.1.1 표준 테스트 시나리오
세 가지 핵심 시나리오가 실제 사용 패턴을 반영해요:
| 시나리오 | 입력 길이 | 출력 길이 | 사용 사례 |
|---|---|---|---|
| Chat | 1K | 1K | 가장 흔한 대화형 AI 워크로드 |
| Reasoning | 1K | 8K | 장문 생성, 복잡한 추론 작업 |
| Summarization | 8K | 1K | 문서 요약, RAG 검색 |
5.1.2 동시성 수준
처리량과 지연 시간의 트레이드오프를 포착하기 위해 각 시나리오를 서로 다른 동시성 수준에서 테스트해요:
- 낮은 동시성 (Low Concurrency):
--max-concurrency 1(지연 시간 최적화) - 중간 동시성 (Medium Concurrency):
--max-concurrency 16(균형) - 높은 동시성 (High Concurrency):
--max-concurrency 100(처리량 최적화)
5.1.3 프롬프트 수
각 동시성 수준에 대해 num_prompts를 구성해 현실적인 사용자 부하를 시뮬레이션하세요:
- 빠른 테스트 (Quick Test):
num_prompts = concurrency × 1(최소 테스트) - 권장 (Recommended):
num_prompts = concurrency × 5(표준 벤치마크) - 안정적 측정 (Stable Measurements):
num_prompts = concurrency × 10(프로덕션 등급)
5.1.4 벤치마크 명령
시나리오 1: Chat (1K/1K) - 가장 중요
- 모델 배포
python -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-V3.1 \
--tp 8
- 낮은 동시성 (지연 시간 최적화)
python -m sglang.bench_serving \
--backend sglang \
--model deepseek-ai/DeepSeek-V3.1 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1 \
--request-rate inf
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 106.24
Total input tokens: 6101
Total input text tokens: 6101
Total input vision tokens: 0
Total generated tokens: 4220
Total generated tokens (retokenized): 4201
Request throughput (req/s): 0.09
Input token throughput (tok/s): 57.43
Output token throughput (tok/s): 39.72
Peak output token throughput (tok/s): 43.00
Peak concurrent requests: 2
Total token throughput (tok/s): 97.15
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 10620.29
Median E2E Latency (ms): 8868.09
---------------Time to First Token----------------
Mean TTFT (ms): 557.85
Median TTFT (ms): 213.58
P99 TTFT (ms): 1625.28
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 23.84
Median TPOT (ms): 23.90
P99 TPOT (ms): 24.03
---------------Inter-Token Latency----------------
Mean ITL (ms): 23.90
Median ITL (ms): 23.92
P95 ITL (ms): 24.15
P99 ITL (ms): 24.25
Max ITL (ms): 25.44
==================================================
- 중간 동시성 (균형)
python -m sglang.bench_serving \
--backend sglang \
--model deepseek-ai/DeepSeek-V3.1 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 80 \
--max-concurrency 16 \
--request-rate inf
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 107.71
Total input tokens: 39668
Total input text tokens: 39668
Total input vision tokens: 0
Total generated tokens: 40805
Total generated tokens (retokenized): 40625
Request throughput (req/s): 0.74
Input token throughput (tok/s): 368.28
Output token throughput (tok/s): 378.84
Peak output token throughput (tok/s): 508.00
Peak concurrent requests: 19
Total token throughput (tok/s): 747.12
Concurrency: 13.72
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 18473.65
Median E2E Latency (ms): 19558.42
---------------Time to First Token----------------
Mean TTFT (ms): 607.91
Median TTFT (ms): 191.32
P99 TTFT (ms): 2135.13
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 35.50
Median TPOT (ms): 35.99
P99 TPOT (ms): 43.62
---------------Inter-Token Latency----------------
Mean ITL (ms): 35.10
Median ITL (ms): 32.18
P95 ITL (ms): 33.03
P99 ITL (ms): 159.99
Max ITL (ms): 453.99
==================================================
- 높은 동시성 (처리량 최적화)
python -m sglang.bench_serving \
--backend sglang \
--model deepseek-ai/DeepSeek-V3.1 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 500 \
--max-concurrency 100 \
--request-rate inf
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 500
Benchmark duration (s): 207.65
Total input tokens: 249831
Total input text tokens: 249831
Total input vision tokens: 0
Total generated tokens: 252662
Total generated tokens (retokenized): 251238
Request throughput (req/s): 2.41
Input token throughput (tok/s): 1203.15
Output token throughput (tok/s): 1216.79
Peak output token throughput (tok/s): 2100.00
Peak concurrent requests: 106
Total token throughput (tok/s): 2419.94
Concurrency: 91.02
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 37800.20
Median E2E Latency (ms): 35921.56
---------------Time to First Token----------------
Mean TTFT (ms): 835.15
Median TTFT (ms): 236.88
P99 TTFT (ms): 2868.52
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 73.33
Median TPOT (ms): 76.35
P99 TPOT (ms): 97.63
---------------Inter-Token Latency----------------
Mean ITL (ms): 73.30
Median ITL (ms): 50.82
P95 ITL (ms): 180.67
P99 ITL (ms): 186.83
Max ITL (ms): 1661.39
==================================================
시나리오 2: Reasoning (1K/8K)
- 낮은 동시성
python -m sglang.bench_serving \
--backend sglang \
--model deepseek-ai/DeepSeek-V3.1 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 8000 \
--num-prompts 10 \
--max-concurrency 1 \
--request-rate inf
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 1097.29
Total input tokens: 6101
Total input text tokens: 6101
Total input vision tokens: 0
Total generated tokens: 44462
Total generated tokens (retokenized): 44313
Request throughput (req/s): 0.01
Input token throughput (tok/s): 5.56
Output token throughput (tok/s): 40.52
Peak output token throughput (tok/s): 43.00
Peak concurrent requests: 2
Total token throughput (tok/s): 46.08
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 109725.52
Median E2E Latency (ms): 117748.67
---------------Time to First Token----------------
Mean TTFT (ms): 156.67
Median TTFT (ms): 156.19
P99 TTFT (ms): 159.87
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 24.41
Median TPOT (ms): 24.51
P99 TPOT (ms): 24.96
---------------Inter-Token Latency----------------
Mean ITL (ms): 24.65
Median ITL (ms): 24.58
P95 ITL (ms): 25.68
P99 ITL (ms): 25.93
Max ITL (ms): 29.80
==================================================
- 중간 동시성
python -m sglang.bench_serving \
--backend sglang \
--model deepseek-ai/DeepSeek-V3.1 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 8000 \
--num-prompts 80 \
--max-concurrency 16 \
--request-rate inf
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 775.02
Total input tokens: 39668
Total input text tokens: 39668
Total input vision tokens: 0
Total generated tokens: 318306
Total generated tokens (retokenized): 317426
Request throughput (req/s): 0.10
Input token throughput (tok/s): 51.18
Output token throughput (tok/s): 410.70
Peak output token throughput (tok/s): 512.00
Peak concurrent requests: 18
Total token throughput (tok/s): 461.89
Concurrency: 13.86
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 134236.65
Median E2E Latency (ms): 135181.28
---------------Time to First Token----------------
Mean TTFT (ms): 214.35
Median TTFT (ms): 194.12
P99 TTFT (ms): 300.27
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 33.72
Median TPOT (ms): 34.00
P99 TPOT (ms): 34.75
---------------Inter-Token Latency----------------
Mean ITL (ms): 33.69
Median ITL (ms): 33.71
P95 ITL (ms): 34.50
P99 ITL (ms): 34.92
Max ITL (ms): 164.76
==================================================
- 높은 동시성
python -m sglang.bench_serving \
--backend sglang \
--model deepseek-ai/DeepSeek-V3.1 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 8000 \
--num-prompts 320 \
--max-concurrency 64 \
--request-rate inf
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 64
Successful requests: 320
Benchmark duration (s): 1231.97
Total input tokens: 158939
Total input text tokens: 158939
Total input vision tokens: 0
Total generated tokens: 1301025
Total generated tokens (retokenized): 1296845
Request throughput (req/s): 0.26
Input token throughput (tok/s): 129.01
Output token throughput (tok/s): 1056.05
Peak output token throughput (tok/s): 1472.00
Peak concurrent requests: 67
Total token throughput (tok/s): 1185.07
Concurrency: 56.17
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 216256.25
Median E2E Latency (ms): 224192.84
---------------Time to First Token----------------
Mean TTFT (ms): 317.68
Median TTFT (ms): 235.28
P99 TTFT (ms): 649.39
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 53.30
Median TPOT (ms): 55.10
P99 TPOT (ms): 56.58
---------------Inter-Token Latency----------------
Mean ITL (ms): 53.13
Median ITL (ms): 52.95
P95 ITL (ms): 56.23
P99 ITL (ms): 181.04
Max ITL (ms): 208.61
==================================================
시나리오 3: Summarization (8K/1K)
- 낮은 동시성
python -m sglang.bench_serving \
--backend sglang \
--model deepseek-ai/DeepSeek-V3.1 \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1 \
--request-rate inf
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 114.47
Total input tokens: 41941
Total input text tokens: 41941
Total input vision tokens: 0
Total generated tokens: 4220
Total generated tokens (retokenized): 4194
Request throughput (req/s): 0.09
Input token throughput (tok/s): 366.39
Output token throughput (tok/s): 36.87
Peak output token throughput (tok/s): 42.00
Peak concurrent requests: 2
Total token throughput (tok/s): 403.26
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 11442.86
Median E2E Latency (ms): 9508.87
---------------Time to First Token----------------
Mean TTFT (ms): 883.78
Median TTFT (ms): 481.38
P99 TTFT (ms): 2217.45
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 24.93
Median TPOT (ms): 25.05
P99 TPOT (ms): 26.11
---------------Inter-Token Latency----------------
Mean ITL (ms): 25.08
Median ITL (ms): 25.08
P95 ITL (ms): 26.18
P99 ITL (ms): 26.28
Max ITL (ms): 27.41
==================================================
- 중간 동시성
python -m sglang.bench_serving \
--backend sglang \
--model deepseek-ai/DeepSeek-V3.1 \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 80 \
--max-concurrency 16 \
--request-rate inf
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 162.33
Total input tokens: 300020
Total input text tokens: 300020
Total input vision tokens: 0
Total generated tokens: 41669
Total generated tokens (retokenized): 41443
Request throughput (req/s): 0.49
Input token throughput (tok/s): 1848.27
Output token throughput (tok/s): 256.70
Peak output token throughput (tok/s): 467.00
Peak concurrent requests: 19
Total token throughput (tok/s): 2104.97
Concurrency: 14.52
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 29456.89
Median E2E Latency (ms): 27628.16
---------------Time to First Token----------------
Mean TTFT (ms): 1784.30
Median TTFT (ms): 1347.21
P99 TTFT (ms): 5384.54
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 53.65
Median TPOT (ms): 52.09
P99 TPOT (ms): 74.39
---------------Inter-Token Latency----------------
Mean ITL (ms): 53.23
Median ITL (ms): 34.52
P95 ITL (ms): 35.81
P99 ITL (ms): 513.25
Max ITL (ms): 2865.73
==================================================
- 높은 동시성
python -m sglang.bench_serving \
--backend sglang \
--model deepseek-ai/DeepSeek-V3.1 \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 320 \
--max-concurrency 64 \
--request-rate inf
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 64
Successful requests: 320
Benchmark duration (s): 282.55
Total input tokens: 1273893
Total input text tokens: 1273893
Total input vision tokens: 0
Total generated tokens: 170000
Total generated tokens (retokenized): 169081
Request throughput (req/s): 1.13
Input token throughput (tok/s): 4508.6
Output token throughput (tok/s): 601.67
Peak output token throughput (tok/s): 1216
Peak concurrent requests: 68
Total token throughput (tok/s): 5110.27
Concurrency: 59.81
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 52810.32
Median E2E Latency (ms): 50981.81
---------------Time to First Token----------------
Mean TTFT (ms): 786.69
Median TTFT (ms): 499.38
P99 TTFT (ms): 2925.98
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 97.93
Median TPOT (ms): 103.45
P99 TPOT (ms): 157.84
---------------Inter-Token Latency----------------
Mean ITL (ms): 98.11
Median ITL (ms): 55.7
P95 ITL (ms): 240.71
P99 ITL (ms): 1114.36
==================================================
5.1.5 결과 이해하기
핵심 지표:
- 요청 처리량 (Request Throughput, req/s): 초당 처리되는 요청 수
- 출력 토큰 처리량 (Output Token Throughput, tok/s): 초당 생성되는 총 토큰 수
- 평균 TTFT (ms): Time to First Token - 응답성을 측정
- 평균 TPOT (ms): Time Per Output Token - 생성 속도를 측정
- 평균 ITL (ms): Inter-Token Latency - 스트리밍 일관성을 측정
이 설정들이 중요한 이유:
- 1K/1K (Chat): 가장 흔한 대화형 AI 워크로드를 나타내요. 대부분의 배포에서 최우선 시나리오예요.
- 1K/8K (Reasoning): 복잡한 추론, 코드 생성, 상세한 설명에 중요한 장문 생성 능력을 테스트해요.
- 8K/1K (Summarization): RAG 시스템, 문서 Q&A, 요약 작업에 필수적인 큰 컨텍스트 입력 성능을 평가해요.
- 가변 동시성 (Variable Concurrency): 파레토 프론티어를 포착해요 — 서로 다른 부하 수준에서 처리량과 지연 시간 간의 최적 트레이드오프를 보여줘요. 낮은 동시성은 최상의 지연 시간을, 높은 동시성은 최대 처리량을 보여줘요.
결과 해석:
- 결과를 하드웨어에 대한 기준 수치와 비교하세요
- 같은 지연 시간에서 더 높은 처리량 = 더 나은 성능
- 더 낮은 TTFT = 더 반응성 좋은 사용자 경험
- 더 낮은 TPOT = 더 빠른 생성 속도
5.2 정확도 벤치마크
표준 벤치마크에서 모델 정확도를 문서화하세요:
5.2.1 GSM8K 벤치마크
- 벤치마크 명령
python3 benchmark/gsm8k/bench_sglang.py \
--num-shots 8 \
--num-questions 1316 \
--parallel 1316
테스트 결과:
Accuracy: 0.959
Invalid: 0.000
Latency: 29.185 s
Output throughput: 4854.672 token/s