Qwen3-Coder
Qwen3-Coder
Qwen3-Coder는 Qwen 팀의 최신 코드 특화 대규모 언어 모델 시리즈예요. Qwen3를 기반으로 구축되어 코드 생성, 이해, 추론 작업에서 뛰어난 성능을 제공해요.
출처: 문서
본문
1. Model Introduction
Qwen3-Coder는 Qwen 팀의 최신 코드 특화 대규모 언어 모델 시리즈예요. Qwen3의 기반 위에 구축되어 코드 생성, 이해, 추론 작업에서 뛰어난 성능을 제공해요.
주요 특징:
- 최고 수준의 코딩 성능: HumanEval, MBPP, LiveCodeBench 및 기타 주요 코딩 벤치마크에서 최상위 결과를 달성해요.
- 도구 호출 지원: 함수 호출과 도구 사용의 네이티브 지원으로 외부 API·서비스와 원활하게 통합돼요.
- 확장된 컨텍스트 길이: 큰 코드베이스와 긴 문서 처리를 위해 최대 256K 토큰 지원.
- 다국어 코드 지원: Python, JavaScript, TypeScript, Java, C++, Go, Rust 등 다양한 프로그래밍 언어에 능숙.
- MoE 아키텍처: 최적의 성능-비용 비율을 위한 효율적인 Mixture-of-Experts 설계.
- ROCm 지원: SGLang을 통해 AMD MI300X, MI325X, MI355X GPU와 호환(검증됨).
- NVIDIA GPU 지원: SGLang을 통해 NVIDIA GB200 및 B200 GPU와 호환(검증됨).
자세한 내용은 공식 Qwen3-Coder GitHub 저장소를 참조하세요.
2. SGLang Installation
SGLang은 여러 설치 방법을 제공해요. 하드웨어 플랫폼과 요구사항에 따라 가장 적합한 설치 방법을 선택할 수 있어요.
설치 지침은 공식 SGLang 설치 가이드를 참조하세요.
SGLang CPU 설치에 대해서는 CPU 버전 설치 가이드를 참조하세요.
3. Model Deployment
이 섹션은 AMD MI300X, MI325X, MI355X, NVIDIA B200, GB200, Intel Xeon CPU 하드웨어 플랫폼에서 검증된 배포 구성을 제공해요.
3.1 Configuration
대화형 명령 생성기: 아래 구성 선택기를 사용해 하드웨어 플랫폼, 모델 크기, 양자화 방법에 맞는 배포 명령을 자동 생성하세요.
3.2 Configuration Tips
AMD (MI300X/MI325X/MI355X):
- 메모리 관리: MI300X/MI325X/MI355X에서
--context-length 8192로 성공적인 배포를 검증했어요. 더 큰 컨텍스트 길이도 지원될 수 있지만 추가 메모리가 필요해요. - 전문가 병렬 처리(Expert Parallelism): FP8 양자화의 480B-A35B 모델은 차원 정렬 요구를 충족하려면
--ep 2가 필요해요. - 페이지 크기: MoE 모델의 메모리 사용 최적화를 위해
--page-size 32를 권장해요. - 환경 변수: aiter 관련 문제가 발생하면
SGLANG_USE_AITER=0을 설정해 보세요.
NVIDIA (B200/GB200):
- GB200 병렬 처리: GB200에서
--tp 4 --ep 4를 사용하세요. B200은 위에서 생성된 기본 NVIDIA 설정을 사용해요. - NVFP4 양자화:
--quantization modelopt_fp4이 필요하며 다른 모델 경로(nvidia/Qwen3-Coder-...)를 사용해요. - DP Attention: NVFP4 구성은 처리량 향상을 위해
--enable-dp-attention을 지원해요.
Intel Xeon CPU:
- SGLang CPU 서버 문서의 serving engine 실행 섹션에서
Notes부분을 참조해 TP(텐서 병렬) 및 NUMA 바인딩 설정을 올바르게 구성하는 방법을 이해하세요.
일반:
- 도구 사용: 도구 호출 기능을 활성화하려면 실행 명령에
--tool-call-parser qwen3_coder를 추가하세요.
4. Model Invocation
4.1 Basic Usage
기본 API 사용법과 요청 예시는 다음을 참조하세요:
4.2 Advanced Usage
4.2.1 Code Generation Example
from openai import OpenAI
client = OpenAI(
api_key="EMPTY",
base_url="http://localhost:30000/v1",
timeout=3600
)
messages = [
{
"role": "user",
"content": "Write a Python function that implements binary search on a sorted list. Include docstring and type hints."
}
]
response = client.chat.completions.create(
model="Qwen/Qwen3-Coder-480B-A35B-Instruct",
messages=messages,
max_tokens=2048,
temperature=0.7
)
print(response.choices[0].message.content)
출력 예시:
```python
from typing import List, Optional, TypeVar
T = TypeVar('T')
def binary_search(arr: List[T], target: T) -> Optional[int]:
"""
Perform binary search on a sorted list to find the index of a target element.
This function implements the binary search algorithm, which efficiently finds
a target value in a sorted array by repeatedly dividing the search interval
in half.
Args:
arr (List[T]): A sorted list of elements to search through.
target (T): The element to search for in the list.
Returns:
Optional[int]: The index of the target element if found, None otherwise.
Time Complexity:
O(log n) where n is the number of elements in the array.
Space Complexity:
O(1) - iterative implementation uses constant extra space.
Examples:
>>> binary_search([1, 2, 3, 4, 5], 3)
2
>>> binary_search([1, 2, 3, 4, 5], 6)
None
>>> binary_search(['a', 'b', 'c', 'd'], 'b')
1
>>> binary_search([], 1)
None
"""
if not arr:
return None
left: int = 0
right: int = len(arr) - 1
while left <= right:
mid: int = (left + right) // 2
if arr[mid] == target:
return mid
elif arr[mid] < target:
left = mid + 1
else:
right = mid - 1
return None
# Alternative recursive implementation
def binary_search_recursive(arr: List[T], target: T, left: int = 0, right: Optional[int] = None) -> Optional[int]:
"""
Perform binary search recursively on a sorted list to find the index of a target element.
Args:
arr (List[T]): A sorted list of elements to search through.
target (T): The element to search for in the list.
left (int): Left boundary of the search range (inclusive).
right (Optional[int]): Right boundary of the search range (inclusive).
Returns:
Optional[int]: The index of the target element if found, None otherwise.
Time Complexity:
O(log n) where n is the number of elements in the array.
Space Complexity:
O(log n) due to recursive call stack.
Examples:
>>> binary_search_recursive([1, 2, 3, 4, 5], 3)
2
>>> binary_search_recursive([1, 2, 3, 4, 5], 6)
None
"""
if not arr:
return None
if right is None:
right = len(arr) - 1
if left > right:
return None
mid: int = (left + right) // 2
if arr[mid] == target:
return mid
elif arr[mid] < target:
return binary_search_recursive(arr, target, mid + 1, right)
else:
return binary_search_recursive(arr, target, left, mid - 1)
This implementation provides:
- Main function (
binary_search): An iterative implementation that's more memory-efficient - Alternative function (
binary_search_recursive): A recursive implementation for educational purposes - Type hints: Using generics (
TypeVar) to work with any comparable type - Comprehensive docstring: Including description, parameters, return value, complexity analysis, and examples
- Edge case handling: Empty lists, elements not found, etc.
- Clear variable names: Self-documenting code
- Examples: Doctest-style examples in the docstring
The function works with any sorted list of comparable elements (integers, strings, etc.) and returns the index of the target element if found, or None if not found.
#### 4.2.2 Tool Calling Example
Qwen3-Coder는 도구 호출 기능을 지원해요. 배포 중 도구 호출 파서를 활성화하세요. 다음 예시는 30B-A3B 모델을 사용해요:
```shell
SGLANG_USE_AITER=0 python -m sglang.launch_server \
--model Qwen/Qwen3-Coder-30B-A3B-Instruct \
--tp 1 \
--context-length 8192 \
--page-size 32 \
--tool-call-parser qwen3_coder
Python 예시:
from openai import OpenAI
client = OpenAI(
api_key="EMPTY",
base_url="http://localhost:30000/v1",
timeout=3600
)
# Define available tools
tools = [
{
"type": "function",
"function": {
"name": "execute_code",
"description": "Execute Python code and return the result",
"parameters": {
"type": "object",
"properties": {
"code": {
"type": "string",
"description": "The Python code to execute"
}
},
"required": ["code"]
}
}
}
]
response = client.chat.completions.create(
model="Qwen/Qwen3-Coder-30B-A3B-Instruct",
messages=[
{"role": "user", "content": "Calculate the factorial of 10 using Python"}
],
tools=tools,
temperature=0.7
)
# Check if the model wants to call a tool
if response.choices[0].message.tool_calls:
tool_call = response.choices[0].message.tool_calls[0]
print(f"Tool: {tool_call.function.name}")
print(f"Arguments: {tool_call.function.arguments}")
else:
# Model may return tool call in content format
print(response.choices[0].message.content)
출력 예시:
Tool: execute_code
Arguments: {"code": "def factorial(n):\n if n == 0 or n == 1:\n return 1\n else:\n return n * factorial(n-1)\n\nresult = factorial(10)\nresult"}
5. Benchmark
5.1 Speed Benchmark
테스트 환경:
- 하드웨어: AMD MI300X GPU (8x)
- 모델: Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8
- 텐서 병렬 처리(Tensor Parallelism): 8
- 전문가 병렬 처리(Expert Parallelism): 2
- sglang 버전: 0.5.7
SGLang의 내장 벤치마킹 도구를 임의 데이터셋과 함께 사용해 성능 평가를 수행해요.
5.1.1 AMD Standard Scenario Benchmark
- 모델 배포 명령:
SGLANG_USE_AITER=0 python -m sglang.launch_server \
--model Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \
--tp 8 \
--ep 2 \
--context-length 8192 \
--page-size 32 \
--trust-remote-code
5.1.1.1 Low Concurrency
- 벤치마크 명령:
python3 -m sglang.bench_serving \
--backend sglang \
--model Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1
- 테스트 결과:
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 73.79
Total input tokens: 6101
Total input text tokens: 6101
Total generated tokens: 4220
Total generated tokens (retokenized): 4104
Request throughput (req/s): 0.14
Input token throughput (tok/s): 82.68
Output token throughput (tok/s): 57.19
Peak output token throughput (tok/s): 59.00
Peak concurrent requests: 2
Total token throughput (tok/s): 139.86
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 7376.26
Median E2E Latency (ms): 5851.51
P90 E2E Latency (ms): 13351.89
P99 E2E Latency (ms): 16908.32
---------------Time to First Token----------------
Mean TTFT (ms): 191.93
Median TTFT (ms): 126.06
P99 TTFT (ms): 662.15
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 17.06
Median TPOT (ms): 17.07
P99 TPOT (ms): 17.08
---------------Inter-Token Latency----------------
Mean ITL (ms): 17.06
Median ITL (ms): 17.06
P95 ITL (ms): 17.14
P99 ITL (ms): 17.19
Max ITL (ms): 18.53
==================================================
5.1.1.2 Medium Concurrency
- 벤치마크 명령:
python3 -m sglang.bench_serving \
--backend sglang \
--model Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 80 \
--max-concurrency 16
- 테스트 결과:
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 87.04
Total input tokens: 39668
Total input text tokens: 39668
Total generated tokens: 40805
Total generated tokens (retokenized): 40364
Request throughput (req/s): 0.92
Input token throughput (tok/s): 455.77
Output token throughput (tok/s): 468.83
Peak output token throughput (tok/s): 608.00
Peak concurrent requests: 20
Total token throughput (tok/s): 924.59
Concurrency: 13.76
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 14966.88
Median E2E Latency (ms): 15871.93
P90 E2E Latency (ms): 24983.41
P99 E2E Latency (ms): 29504.85
---------------Time to First Token----------------
Mean TTFT (ms): 388.94
Median TTFT (ms): 157.49
P99 TTFT (ms): 1318.63
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 29.41
Median TPOT (ms): 29.22
P99 TPOT (ms): 43.48
---------------Inter-Token Latency----------------
Mean ITL (ms): 28.64
Median ITL (ms): 26.42
P95 ITL (ms): 27.51
P99 ITL (ms): 131.63
Max ITL (ms): 995.11
==================================================
5.1.1.3 High Concurrency
- 벤치마크 명령:
python3 -m sglang.bench_serving \
--backend sglang \
--model Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 320 \
--max-concurrency 64
- 테스트 결과:
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 64
Successful requests: 320
Benchmark duration (s): 177.82
Total input tokens: 158939
Total input text tokens: 158939
Total generated tokens: 170134
Total generated tokens (retokenized): 168387
Request throughput (req/s): 1.80
Input token throughput (tok/s): 893.84
Output token throughput (tok/s): 956.80
Peak output token throughput (tok/s): 1728.00
Peak concurrent requests: 70
Total token throughput (tok/s): 1850.64
Concurrency: 58.88
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 32716.53
Median E2E Latency (ms): 30896.37
P90 E2E Latency (ms): 65605.24
P99 E2E Latency (ms): 80970.63
---------------Time to First Token----------------
Mean TTFT (ms): 372.97
Median TTFT (ms): 181.67
P99 TTFT (ms): 529.01
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 62.98
Median TPOT (ms): 50.44
P99 TPOT (ms): 204.24
---------------Inter-Token Latency----------------
Mean ITL (ms): 60.95
Median ITL (ms): 37.87
P95 ITL (ms): 143.98
P99 ITL (ms): 148.02
Max ITL (ms): 36863.32
==================================================
5.1.2 NVIDIA (B200/GB200) Standard Scenario Benchmark
아래 실행은 AMD 섹션과 동일한 임의 데이터셋 벤치마크 클라이언트 명령을 사용해요. B200에서는 다음 명령으로 서버를 시작하세요:
sglang serve --model Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 --tp 8 --ep 8 --context-length 8192 --page-size 32 --trust-remote-code
5.1.2.1 FP8 Model
- 낮은 동시성:
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 42.68
Total input tokens: 6101
Total input text tokens: 6101
Total generated tokens: 4220
Total generated tokens (retokenized): 4204
Request throughput (req/s): 0.23
Input token throughput (tok/s): 142.95
Output token throughput (tok/s): 98.88
Peak output token throughput (tok/s): 102.00
Peak concurrent requests: 2
Total token throughput (tok/s): 241.83
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 4266.06
Median E2E Latency (ms): 3420.24
P90 E2E Latency (ms): 7717.19
P99 E2E Latency (ms): 9504.50
---------------Time to First Token----------------
Mean TTFT (ms): 112.03
Median TTFT (ms): 112.70
P99 TTFT (ms): 115.35
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 9.87
Median TPOT (ms): 9.86
P99 TPOT (ms): 9.92
---------------Inter-Token Latency----------------
Mean ITL (ms): 9.87
Median ITL (ms): 9.87
P95 ITL (ms): 10.06
P99 ITL (ms): 10.18
Max ITL (ms): 14.80
==================================================
- 중간 동시성:
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 60.80
Total input tokens: 39668
Total input text tokens: 39668
Total generated tokens: 40805
Total generated tokens (retokenized): 40543
Request throughput (req/s): 1.32
Input token throughput (tok/s): 652.43
Output token throughput (tok/s): 671.13
Peak output token throughput (tok/s): 864.00
Peak concurrent requests: 20
Total token throughput (tok/s): 1323.57
Concurrency: 13.93
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 10587.26
Median E2E Latency (ms): 11486.18
P90 E2E Latency (ms): 17374.75
P99 E2E Latency (ms): 21107.18
---------------Time to First Token----------------
Mean TTFT (ms): 155.27
Median TTFT (ms): 121.57
P99 TTFT (ms): 294.31
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 20.77
Median TPOT (ms): 21.13
P99 TPOT (ms): 23.62
---------------Inter-Token Latency----------------
Mean ITL (ms): 20.49
Median ITL (ms): 18.73
P95 ITL (ms): 19.65
P99 ITL (ms): 98.85
Max ITL (ms): 536.87
==================================================
- 높은 동시성:
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 64
Successful requests: 320
Benchmark duration (s): 100.07
Total input tokens: 158939
Total input text tokens: 158939
Total generated tokens: 170134
Total generated tokens (retokenized): 169119
Request throughput (req/s): 3.20
Input token throughput (tok/s): 1588.32
Output token throughput (tok/s): 1700.19
Peak output token throughput (tok/s): 2303.00
Peak concurrent requests: 71
Total token throughput (tok/s): 3288.51
Concurrency: 57.93
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 18114.01
Median E2E Latency (ms): 18279.15
P90 E2E Latency (ms): 30557.22
P99 E2E Latency (ms): 35889.84
---------------Time to First Token----------------
Mean TTFT (ms): 346.40
Median TTFT (ms): 129.75
P99 TTFT (ms): 1370.20
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 33.76
Median TPOT (ms): 34.62
P99 TPOT (ms): 39.97
---------------Inter-Token Latency----------------
Mean ITL (ms): 33.48
Median ITL (ms): 25.70
P95 ITL (ms): 99.36
P99 ITL (ms): 132.30
Max ITL (ms): 1132.39
==================================================
5.1.2.2 NVFP4 Model
- 낮은 동시성:
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 34.49
Total input tokens: 6101
Total input text tokens: 6101
Total generated tokens: 4220
Total generated tokens (retokenized): 4218
Request throughput (req/s): 0.29
Input token throughput (tok/s): 176.87
Output token throughput (tok/s): 122.34
Peak output token throughput (tok/s): 127.00
Peak concurrent requests: 2
Total token throughput (tok/s): 299.21
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 3448.01
Median E2E Latency (ms): 2768.11
P90 E2E Latency (ms): 6225.73
P99 E2E Latency (ms): 7668.26
---------------Time to First Token----------------
Mean TTFT (ms): 104.55
Median TTFT (ms): 105.38
P99 TTFT (ms): 105.63
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 7.94
Median TPOT (ms): 7.95
P99 TPOT (ms): 7.97
---------------Inter-Token Latency----------------
Mean ITL (ms): 7.94
Median ITL (ms): 7.94
P95 ITL (ms): 8.05
P99 ITL (ms): 8.11
Max ITL (ms): 24.64
==================================================
- 중간 동시성:
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 43.30
Total input tokens: 39668
Total input text tokens: 39668
Total generated tokens: 40805
Total generated tokens (retokenized): 39975
Request throughput (req/s): 1.85
Input token throughput (tok/s): 916.16
Output token throughput (tok/s): 942.42
Peak output token throughput (tok/s): 1264.00
Peak concurrent requests: 21
Total token throughput (tok/s): 1858.57
Concurrency: 13.90
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 7521.95
Median E2E Latency (ms): 8246.89
P90 E2E Latency (ms): 12370.93
P99 E2E Latency (ms): 15023.96
---------------Time to First Token----------------
Mean TTFT (ms): 137.27
Median TTFT (ms): 109.59
P99 TTFT (ms): 208.78
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 14.69
Median TPOT (ms): 14.87
P99 TPOT (ms): 17.63
---------------Inter-Token Latency----------------
Mean ITL (ms): 14.51
Median ITL (ms): 12.75
P95 ITL (ms): 13.33
P99 ITL (ms): 92.85
Max ITL (ms): 113.70
==================================================
- 높은 동시성:
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 64
Successful requests: 320
Benchmark duration (s): 73.93
Total input tokens: 158939
Total input text tokens: 158939
Total generated tokens: 170134
Total generated tokens (retokenized): 168841
Request throughput (req/s): 4.33
Input token throughput (tok/s): 2149.98
Output token throughput (tok/s): 2301.42
Peak output token throughput (tok/s): 3497.00
Peak concurrent requests: 71
Total token throughput (tok/s): 4451.40
Concurrency: 58.28
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 13463.58
Median E2E Latency (ms): 13498.74
P90 E2E Latency (ms): 22957.10
P99 E2E Latency (ms): 26656.95
---------------Time to First Token----------------
Mean TTFT (ms): 239.00
Median TTFT (ms): 113.42
P99 TTFT (ms): 713.87
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 25.13
Median TPOT (ms): 26.02
P99 TPOT (ms): 30.90
---------------Inter-Token Latency----------------
Mean ITL (ms): 24.92
Median ITL (ms): 16.68
P95 ITL (ms): 93.33
P99 ITL (ms): 119.26
Max ITL (ms): 548.82
==================================================
5.2 Accuracy Benchmark
5.2.1 GSM8K Benchmark
- 벤치마크 명령:
python3 -m sglang.test.few_shot_gsm8k --num-questions 200
AMD (MI300X/MI325X/MI355X)
-
결과:
- Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8
Accuracy: 0.965 Invalid: 0.000 Latency: 23.084 s Output throughput: 1148.425 token/s
- Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8
NVIDIA (B200/GB200)
배포 명령은 Section 3.1을 참조하세요.
-
Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8
Accuracy: 0.965 Invalid: 0.000 Latency: 14.870 s Output throughput: 1777.726 token/s -
nvidia/Qwen3-Coder-480B-A35B-Instruct-NVFP (NVFP4)
Accuracy: 0.960 Invalid: 0.000 Latency: 13.948 s Output throughput: 1988.548 token/s