Qwen3-Next

3.2 Configuration Tips

• --max-mamba-cache-size: --max-mamba-cache-size를 조정해 mamba 캐시 공간과 최대 실행 요청 수를 늘릴 수 있어요. 대가로 KV 캐시 공간은 줄어들어요. 워크로드에 따라 조정하세요.

• --mamba-ssm-dtype: bfloat16 또는 float32. bfloat16은 mamba 캐시 크기를 아끼고 float32는 더 정확한 결과를 얻어요. 기본값은 float32예요.

• --mamba-full-memory-ratio: --mamba-full-memory-ratio를 조정해 전체 kv 캐시 메모리 대비 mamba 상태 메모리의 비율을 설정해요. 기본값은 0.9예요.

• Mamba Radix Cache: Qwen3-Next의 하이브리드 Gated Delta Networks 아키텍처는 --mamba-radix-cache-strategy를 통해 두 가지 mamba 스케줄링 전략을 지원해요:

  • V1 (no_buffer): 기본값. overlap 스케줄러 없음, 더 낮은 메모리 사용.
  • V2 (extra_buffer): --mamba-radix-cache-strategy extra_buffer --page-size 64로 overlap 스케줄링과 branching point 캐싱을 활성화. FLA 커널 백엔드 필요. 더 높은 mamba 상태 메모리를 처리량 향상과 맞바꿔요. 비-KV-cache-바운드 시나리오에서는 엄격히 우월하고, KV-cache-바운드 경우에는 overlap 스케줄링 이점과 줄어든 최대 동시성을 저울질해야 해요. --page-size는 FLA_CHUNK_SIZE % page_size == 0 또는 page_size % FLA_CHUNK_SIZE == 0을 만족해야 해요(FLA_CHUNK_SIZE는 현재 64).

• Xeon CPU 서비스 구성: SGLang CPU 서버 문서의 serving engine 실행 섹션에서 Notes 부분을 참조해 TP(텐서 병렬) 및 NUMA 바인딩 설정을 올바르게 구성하는 방법을 이해하세요.

4. Model Invocation

4.1 Basic Usage

기본 API 사용법과 요청 예시는 다음을 참조하세요:

• SGLang Basic Usage Guide

4.2 Advanced Usage

4.2.1 Reasoning Parser

1. 생각 과정(Thinking Process)이 있는 스트리밍:

   Qwen3-Next-80B-A3B-Thinking은 thinking 모드만 지원해요. 배포 중 reasoning 파서를 활성화해 thinking 섹션과 content 섹션을 분리하세요.

python -m sglang.launch_server \
  --model Qwen/Qwen3-Next-80B-A3B-Thinking \
  --reasoning-parser qwen3 \
  --tp 8 \
  --host 0.0.0.0 \
  --port 8000

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY"
)

# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
    model="Qwen/Qwen3-Next-80B-A3B-Thinking",
    messages=[
        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
    ],
    temperature=0.7,
    max_tokens=2048,
    stream=True
)

# Process the stream
has_thinking = False
has_answer = False
thinking_started = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Print answer content
        if delta.content:
            # Close thinking section and add content header
            if has_thinking and not has_answer:
                print("\n=============== Content =================", flush=True)
                has_answer = True
            print(delta.content, end="", flush=True)

print()

출력 예시:

=============== Thinking =================
Okay, let's see. I need to find 15% of 240. Hmm, percentages. Right, "percent" means per hundred, so 15% is 15 per 100, or 15/100. To find a percentage of a number, I think you multiply the number by the percentage as a decimal. So first, maybe convert 15% to a decimal. To convert a percentage to a decimal, you divide by 100. So 15 divided by 100 is 0.15. Then, multiply that by 240. Let me check that. So 0.15 times 240. Let's calculate that. Maybe break it down. 10% of 240 is 24, because 10% is just moving the decimal one place left, so 240 becomes 24. Then 5% would be half of 10%, so half of 24 is 12. So 10% + 5% = 15%, so 24 + 12 = 36. Oh, that's another way to do it. Let me verify with the multiplication. 0.15 * 240. Let's do 240 * 0.1 = 24, 240 * 0.05 = 12, so 24 + 12 = 36. Yep, that works. Alternatively, 240 * 15 = 3600, then divide by 100, which is 36. Because 15% of 240 is (15/100)*240 = (15*240)/100. 15*240: 10*240=2400, 5*240=1200, so 2400+1200=3600. Then 3600/100=36. So that's 36. So the answer should be 36. Let me make sure. 15% of 240. If I take 240 and multiply by 0.15, 240*0.15. Let's compute 240*0.1=24, 240*0.05=12, so 24+12=36. Yep, that's right. So 15% of 240 is 36.

=============== Content =================

To find **15% of 240**, follow these steps:

---

### **Step 1: Understand what "percent" means**
- "Percent" means **per hundred**, so **15% = 15/100 = 0.15** in decimal form.

---

### **Step 2: Multiply the number by the decimal**
- To find 15% of 240, multiply:
  $$
  240 \times 0.15
  $$

---

### **Step 3: Break it down for clarity (optional but helpful)**
- **10% of 240** = $ 240 \times 0.1 = 24 $
- **5% of 240** = $ 240 \times 0.05 = 12 $
- Add them together:
  $$
  24 + 12 = 36
  $$

---

### **Step 4: Confirm with direct multiplication**
- $ 240 \times 0.15 = 36 $

---

### ✅ Final Answer:
$$
\boxed{36}
$$

참고: reasoning 파서는 모델의 단계별 사고 과정을 포착해 모델이 결론에 어떻게 도달하는지 볼 수 있게 해 줘요.

2. Thinking 끄기:

   Qwen3-Next-80B-A3B-Instruct는 instruct(비-thinking) 모드만 지원해요.

python -m sglang.launch_server \
  --model Qwen/Qwen3-Next-80B-A3B-Instruct \
  --tp 8 \
  --host 0.0.0.0 \
  --port 8000

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY"
)

# Turn off thinking process
response = client.chat.completions.create(
    model="Qwen/Qwen3-Next-80B-A3B-Instruct",
    messages=[
        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
    ],
    temperature=0.7,
    max_tokens=2048,
    stream=True,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}}
)

# Process the stream
has_thinking = False
has_answer = False
thinking_started = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Print answer content
        if delta.content:
            # Close thinking section and add content header
            if has_thinking and not has_answer:
                print("\n=============== Content =================", flush=True)
                has_answer = True
            print(delta.content, end="", flush=True)

print()

출력 예시:

To find **15% of 240**, follow these steps:

---

### **Step 1: Understand what percentage means**
"Percent" means "per hundred," so **15%** is the same as **15 per 100**, or the fraction:

$$
\frac{15}{100}
$$

---

### **Step 2: Multiply the fraction by the number**
To find 15% of 240, multiply:

$$
\frac{15}{100} \times 240
$$

---

### **Step 3: Simplify the multiplication**
You can simplify this in a couple of ways.

#### **Option A: Multiply first, then divide**
$$
15 \times 240 = 3600
$$
Then divide by 100:
$$
\frac{3600}{100} = 36
$$

#### **Option B: Simplify the fraction first**
$$
\frac{15}{100} = \frac{3}{20} \quad \text{(divided numerator and denominator by 5)}
$$
Now multiply:
$$
\frac{3}{20} \times 240 = \frac{3 \times 240}{20} = \frac{720}{20} = 36
$$

---

### **Step 4: Final Answer**
$$
\boxed{36}
$$

So, **15% of 240 is 36**.

4.2.2 Tool Calling

Qwen/Qwen3-Next-80B-A3B-Instruct와 Qwen/Qwen3-Next-80B-A3B-Thinking 모두 도구 호출 기능을 지원해요. 도구 호출 파서를 활성화하세요:

Python 예시 (Thinking Process 없이):

sglang 서버를 시작하세요:

python -m sglang.launch_server \
  --model Qwen/Qwen3-Next-80B-A3B-Instruct \
  --tool-call-parser qwen \
  --tp 8 \
  --host 0.0.0.0 \
  --port 8000

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY"
)

# Define available tools
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city name"
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "Temperature unit"
                    }
                },
                "required": ["location"]
            }
        }
    }
]

# Make request with streaming to see thinking process
response = client.chat.completions.create(
    model="Qwen/Qwen3-Next-80B-A3B-Instruct",
    messages=[
        {"role": "user", "content": "What's the weather in Beijing?"}
    ],
    tools=tools,
    temperature=0.7,
    stream=True
)

# Process streaming response
thinking_started = False
has_thinking = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Print tool calls
        if hasattr(delta, 'tool_calls') and delta.tool_calls:
            # Close thinking section if needed
            if has_thinking and thinking_started:
                print("\n=============== Content =================", flush=True)
                thinking_started = False

            for tool_call in delta.tool_calls:
                if tool_call.function:
                    print(f"🔧 Tool Call: {tool_call.function.name}")
                    print(f"   Arguments: {tool_call.function.arguments}")

        # Print content
        if delta.content:
            print(delta.content, end="", flush=True)

print()

출력 예시:

<tool_call>
{"name": "get_weather", "arguments": {"location": "Beijing"}}
</tool_call>

Python 예시 (Thinking Process 포함):

sglang 서버를 시작하세요:

python -m sglang.launch_server \
  --model Qwen/Qwen3-Next-80B-A3B-Thinking \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen \
  --tp 8 \
  --host 0.0.0.0 \
  --port 8000

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY"
)

# Define available tools
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city name"
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "Temperature unit"
                    }
                },
                "required": ["location"]
            }
        }
    }
]

# Make request with streaming to see thinking process
response = client.chat.completions.create(
    model="Qwen/Qwen3-Next-80B-A3B-Thinking",
    messages=[
        {"role": "user", "content": "What's the weather in Beijing?"}
    ],
    tools=tools,
    temperature=0.7,
    stream=True
)

# Process streaming response
thinking_started = False
has_thinking = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Print tool calls
        if hasattr(delta, 'tool_calls') and delta.tool_calls:
            # Close thinking section if needed
            if has_thinking and thinking_started:
                print("\n=============== Content =================", flush=True)
                thinking_started = False

            for tool_call in delta.tool_calls:
                if tool_call.function:
                    print(f"🔧 Tool Call: {tool_call.function.name}")
                    print(f"   Arguments: {tool_call.function.arguments}")

        # Print content
        if delta.content:
            print(delta.content, end="", flush=True)

print()

출력 예시:

=============== Thinking =================
Okay, the user is asking for the weather in Beijing. Let me check the available tools. There's a get_weather function that requires location and optionally unit. The location is needed, so I need to provide Beijing as the location. The unit is optional, but the user didn't specify Celsius or Fahrenheit. Since the default might be Celsius, but maybe I should check if the parameters require unit. Wait, the required field is only location, so unit is optional. So I can just call get_weather with location "Beijing" and not include the unit. Let me confirm the parameters. The parameters for get_weather have location as required, and unit is an enum with celsius or fahrenheit, but not required. So the correct call is to send location as Beijing, and omit unit. So the tool call should be {"name": "get_weather", "arguments": {"location": "Beijing"}}.

<tool_call>
{"name": "get_weather", "arguments": {"location": "Beijing"}}
</tool_call>

참고:

• reasoning 파서는 모델이 도구를 어떻게 사용하기로 결정하는지 보여줘요
• 도구 호출은 함수 이름과 인자로 명확히 표시돼요
• 그런 다음 함수를 실행하고 결과를 다시 보내 대화를 계속할 수 있어요

도구 호출 결과 처리:

# After getting the tool call, execute the function
def get_weather(location, unit="celsius"):
    # Your actual weather API call here
    return f"The weather in {location} is 22°{unit[0].upper()} and sunny."

# Send tool result back to the model
messages = [
    {"role": "user", "content": "What's the weather in Beijing?"},
    {
        "role": "assistant",
        "content": None,
        "tool_calls": [{
            "id": "call_123",
            "type": "function",
            "function": {
                "name": "get_weather",
                "arguments": '{"location": "Beijing", "unit": "celsius"}'
            }
        }]
    },
    {
        "role": "tool",
        "tool_call_id": "call_123",
        "content": get_weather("Beijing", "celsius")
    }
]

final_response = client.chat.completions.create(
    model="Qwen/Qwen3-Next-80B-A3B-Thinking",
    messages=messages,
    temperature=0.7
)

print(final_response.choices[0].message.content)
# Output: "The weather in Beijing is currently 22°C and sunny."

4.2.3 Processing Ultra-Long Texts

Qwen3-Next는 네이티브로 최대 262,144 토큰의 컨텍스트 길이를 지원해요. 총 길이(입력과 출력 모두 포함)가 이 한도를 크게 초과하는 대화에서는 RoPE 스케일링 기법을 사용해 긴 텍스트를 효과적으로 처리하는 것을 권장해요. YaRN 방법을 사용해 최대 100만 토큰의 컨텍스트 길이에서 모델 성능을 검증했어요.

Qwen3-Next-80B-A3B-Instruct

SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 python -m sglang.launch_server --model Qwen/Qwen3-Next-80B-A3B-Instruct    --tp 8   --host 0.0.0.0   --port 8000 --json-model-override-args '{"rope_scaling":{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":262144}}' --context-length 1010000

Qwen3-Next-80B-A3B-Thinking

SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 python -m sglang.launch_server --model Qwen/Qwen3-Next-80B-A3B-Thinking   --reasoning-parser qwen3     --tp 8   --host 0.0.0.0   --port 8000 --json-model-override-args '{"rope_scaling":{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":262144}}' --context-length 1010000

4.2.4 Multi-Token Prediction (NEXTN Speculative Decoding)

Qwen3-Next는 내장 Multi-Token Prediction (MTP) 레이어를 탑재하고 NEXTN 알고리즘을 통해 EAGLE 스타일 추측 디코딩을 지원해요. MTP 가중치는 메인 체크포인트에 포함되어 있어 별도의 draft 모델이 필요 없어요.

python3 -m sglang.launch_server \
  --model Qwen/Qwen3-Next-80B-A3B-Instruct \
  --speculative-algorithm NEXTN \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --tp 4

bench_speculative.py로 워크로드에 맞게 --speculative-num-steps, --speculative-eagle-topk, --speculative-num-draft-tokens를 조정하세요. 구현 세부 사항은 PR #10233을 참조하세요.

5. Benchmark

5.1 Speed Benchmark

테스트 환경:

• 하드웨어: NVIDIA B200 GPU (8x)
• 텐서 병렬 처리(Tensor Parallelism): 8
• 모델: Qwen/Qwen3-Next-80B-A3B-Instruct
• sglang 버전: 0.5.6

SGLang의 내장 벤치마킹 도구를 ShareGPT_Vicuna_unfiltered 데이터셋에서 성능 평가를 수행해요. 이 데이터셋은 실제 대화 데이터를 포함해 실제 사용 시나리오의 성능을 더 잘 반영해요.

5.1.1 Latency-Sensitive Benchmark

• 서버 명령:

python -m sglang.launch_server \
  --model Qwen/Qwen3-Next-80B-A3B-Instruct \
  --tp 8

• 테스트 명령:

python3 -m sglang.bench_serving \
  --backend sglang \
  --num-prompt 100 \
  --max-concurrency 1

• 테스트 결과:

============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     100
Benchmark duration (s):                  146.52
Total input tokens:                      33839
Total input text tokens:                 33839
Total input vision tokens:               0
Total generated tokens:                  21640
Total generated tokens (retokenized):    21619
Request throughput (req/s):              0.68
Input token throughput (tok/s):          230.95
Output token throughput (tok/s):         147.70
Peak output token throughput (tok/s):    164.00
Peak concurrent requests:                6
Total token throughput (tok/s):          378.65
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   1464.81
Median E2E Latency (ms):                 1077.48
---------------Time to First Token----------------
Mean TTFT (ms):                          127.88
Median TTFT (ms):                        132.88
P99 TTFT (ms):                           212.85
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          6.19
Median TPOT (ms):                        6.17
P99 TPOT (ms):                           6.64
---------------Inter-Token Latency----------------
Mean ITL (ms):                           6.21
Median ITL (ms):                         6.16
P95 ITL (ms):                            6.51
P99 ITL (ms):                            6.71
Max ITL (ms):                            10.07
==================================================

5.1.2 Throughput-Sensitive Benchmark

• 서버 명령:

python -m sglang.launch_server \
  --model Qwen/Qwen3-Next-80B-A3B-Instruct \
  --tp 8

• 테스트 명령:

python3 -m sglang.bench_serving \
  --backend sglang \
  --num-prompt 1000 \
  --max-concurrency 100

테스트 결과:

============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     1000
Benchmark duration (s):                  100.32
Total input tokens:                      302118
Total input text tokens:                 302118
Total input vision tokens:               0
Total generated tokens:                  195775
Total generated tokens (retokenized):    195016
Request throughput (req/s):              9.97
Input token throughput (tok/s):          3011.69
Output token throughput (tok/s):         1951.60
Peak output token throughput (tok/s):    5909.00
Peak concurrent requests:                120
Total token throughput (tok/s):          4963.29
Concurrency:                             93.05
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   9333.98
Median E2E Latency (ms):                 6054.12
---------------Time to First Token----------------
Mean TTFT (ms):                          161.77
Median TTFT (ms):                        137.94
P99 TTFT (ms):                           503.29
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          50.87
Median TPOT (ms):                        50.28
P99 TPOT (ms):                           122.87
---------------Inter-Token Latency----------------
Mean ITL (ms):                           47.11
Median ITL (ms):                         13.84
P95 ITL (ms):                            195.33
P99 ITL (ms):                            289.56
Max ITL (ms):                            486.38
==================================================

5.2 Accuracy Benchmark

5.2.1 GSM8K Benchmark

• 벤치마크 명령:

python3 -m sglang.test.few_shot_gsm8k --num-questions 200 --port 8000

• 결과:

  • Qwen3-Next-80B-A3B-Instruct

    Accuracy: 0.960
    Invalid: 0.000
    Latency: 12.673 s
    Output throughput: 2538.255 token/s
    

  • Qwen3-Next-80B-A3B-Thinking

    Accuracy: 0.935
    Invalid: 0.000
    Latency: 9.912 s
    Output throughput: 3288.737 token/s
    

5.2.2 MMLU Benchmark

• 벤치마크 명령:

cd sglang
bash benchmark/mmlu/download_data.sh
python3 benchmark/mmlu/bench_sglang.py --nsub 10

• 결과:

  • Qwen3-Next-80B-A3B-Instruct

    subject: abstract_algebra, #q:100, acc: 0.800
    subject: anatomy, #q:135, acc: 0.807
    subject: astronomy, #q:152, acc: 0.947
    subject: business_ethics, #q:100, acc: 0.810
    subject: clinical_knowledge, #q:265, acc: 0.894
    subject: college_biology, #q:144, acc: 0.972
    subject: college_chemistry, #q:100, acc: 0.680
    subject: college_computer_science, #q:100, acc: 0.860
    subject: college_mathematics, #q:100, acc: 0.780
    subject: college_medicine, #q:173, acc: 0.861
    Total latency: 10.098
    Average accuracy: 0.856
    

  • Qwen3-Next-80B-A3B-Thinking

    subject: abstract_algebra, #q:100, acc: 0.780
    subject: anatomy, #q:135, acc: 0.815
    subject: astronomy, #q:152, acc: 0.941
    subject: business_ethics, #q:100, acc: 0.870
    subject: clinical_knowledge, #q:265, acc: 0.894
    subject: college_biology, #q:144, acc: 0.965
    subject: college_chemistry, #q:100, acc: 0.670
    subject: college_computer_science, #q:100, acc: 0.840
    subject: college_mathematics, #q:100, acc: 0.770
    subject: college_medicine, #q:173, acc: 0.861
    Total latency: 10.236
    Average accuracy: 0.855
    

더 알아보기

• Qwen3-Next 컬렉션 (Hugging Face)
• 공식 Qwen3-Next 블로그