Qwen3.6-35B-A3B
Qwen3.6-35B-A3B
이 페이지는 Ascend NPU에서 Qwen3.6-35B-A3B의 최적 구성과 벤치마크 결과에 집중해요. 환경 설정, 모델 가중치 다운로드, 기능 구성, 배포 지침 등은 Qwen3.6-35B-A3B 모델 튜토리얼을 참고하세요.
A3 시리즈에서는 각 카드에 2개의 die가 있어서 --tp-size가 카드 수의 2배예요. 자세한 내용은 Ascend NPU 참조를 확인해요.
출처: 문서
본문
저지연 (Low Latency)
| 모델 (Model) | 하드웨어 (Hardware) | 카드 (Cards) | 배포 모드 (Deploy Mode) | 데이터셋 (Dataset) | TPOT | 양자화 (Quantization) | 구성 (Configuration) |
|---|---|---|---|---|---|---|---|
| Qwen3.6-35B-A3B | Ascend A3 Series Products | 1 | PD Mixed | 254k+1k | 16.1ms | BF16 | 최적 구성 |
고처리량 (High Throughput)
| 모델 (Model) | 하드웨어 (Hardware) | 카드 (Cards) | 배포 모드 (Deploy Mode) | 데이터셋 (Dataset) | TPOT | 양자화 (Quantization) | 구성 (Configuration) |
|---|---|---|---|---|---|---|---|
| Qwen3.6-35B-A3B | Ascend A3 Series Products | 1 | PD Mixed | 1024x1024 (30)+1024 | 50ms | BF16 | 최적 구성 |
| Qwen3.6-35B-A3B | Ascend A3 Series Products | 1 | PD Mixed | 1080p\_30+256 | 50ms | BF16 | 최적 구성 |
| Qwen3.6-35B-A3B | Ascend A3 Series Products | 1 | PD Mixed | 128k+1k | 50ms | BF16 | 최적 구성 |
| Qwen3.6-35B-A3B | Ascend A3 Series Products | 1 | PD Mixed | 128k+1k (90% prefix cache hit rate) | 50ms | BF16 | 최적 구성 |
| Qwen3.6-35B-A3B | Ascend A3 Series Products | 1 | PD Mixed | 3.5k+1.5k | 50ms | BF16 | 최적 구성 |
| Qwen3.6-35B-A3B | Ascend A3 Series Products | 1 | PD Mixed | 64k+1k | 50ms | BF16 | 최적 구성 |
| Qwen3.6-35B-A3B | Ascend A3 Series Products | 1 | PD Mixed | 64k+1k (90% prefix cache hit rate) | 50ms | BF16 | 최적 구성 |
| Qwen3.6-35B-A3B | Ascend A3 Series Products | 2 | PD Mixed | 984k+1k | 40.91ms | BF16 | 최적 구성 |
최적 구성 (Optimal Configuration)
Qwen3.6-35B-A3B 1P IN1024X1024 30 OUT1024 50ms
모델 (Model): Qwen3.6-35B-A3B
하드웨어 (Hardware): Ascend A3 Series Products
카드 (Cards): 1
배포 모드 (Deploy Mode): PD Mixed
양자화 (Quantization): BF16
데이터셋 (Dataset): 1024x1024 (30)+1024
형식 (Format): 해상도 (입력 토큰) + 출력 토큰
TPOT: 50ms
모델 배포 (Model Deployment)
# ============================================================
# Before running, update the following variables:
# MODEL_PATH: path to the model weights directory
# HCCL_SOCKET_IFNAME: network interface name for HCCL
# GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================
MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000
unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING
source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh
export ASCEND_USE_FIA=1
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_SET_CPU_AFFINITY=1
export SGLANG_VIT_ENABLE_CUDA_GRAPH=1
export STREAMS_PER_DEVICE=32
python3 -m sglang.launch_server \
--model-path $MODEL_PATH \
--host 127.0.0.1 --port 6688 \
--tp-size 2 \
--nnodes 1 \
--attention-backend ascend \
--device npu \
--chunked-prefill-size -1 \
--max-prefill-tokens 9999999 \
--max-total-tokens 365000 \
--prefill-max-requests 30 \
--disable-radix-cache \
--trust-remote-code \
--max-running-requests 120 \
--max-mamba-cache-size 120 \
--mem-fraction-static 0.85 \
--cuda-graph-bs-decode 4 16 32 48 64 110 165 \
--enable-multimodal \
--mm-attention-backend ascend_attn \
--dtype bfloat16 \
--mamba-ssm-dtype bfloat16 \
--speculative-algorithm NEXTN \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
--reasoning-parser qwen3 \
--tool-call-parser qwen3_coder
벤치마크 (Benchmark)
IMAGE 데이터셋을 1024x1024 해상도로 테스트했어요.
python -m sglang.bench_serving \
--dataset-name image \
--backend sglang-oai-chat \
--host 127.0.0.1 \
--port 6688 \
--max-concurrency 120 \
--num-prompts 120 \
--warmup-requests 120 \
--random-input-len 30 \
--random-output-len 1024 \
--random-range-ratio 1 \
--image-resolution 1024x1024 \
--image-count 1 \
--seed 1 \
--request-rate inf
Qwen3.6-35B-A3B 1P IN1080P 30 OUT256 50ms
모델 (Model): Qwen3.6-35B-A3B
하드웨어 (Hardware): Ascend A3 Series Products
카드 (Cards): 1
배포 모드 (Deploy Mode): PD Mixed
양자화 (Quantization): BF16
데이터셋 (Dataset): 1080p\_30+256
TPOT: 50ms
모델 배포 (Model Deployment)
# ============================================================
# Before running, update the following variables:
# MODEL_PATH: path to the model weights directory
# HCCL_SOCKET_IFNAME: network interface name for HCCL
# GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================
MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000
unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING
source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh
export ASCEND_USE_FIA=1
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_SET_CPU_AFFINITY=1
export SGLANG_VIT_ENABLE_CUDA_GRAPH=1
export STREAMS_PER_DEVICE=32
python3 -m sglang.launch_server \
--model-path $MODEL_PATH \
--host 127.0.0.1 --port 6688 \
--tp-size 2 \
--nnodes 1 \
--attention-backend ascend \
--device npu \
--chunked-prefill-size -1 \
--max-prefill-tokens 150000 \
--max-total-tokens 200000 \
--disable-radix-cache \
--trust-remote-code \
--max-running-requests 42 \
--max-mamba-cache-size 42 \
--mem-fraction-static 0.75 \
--cuda-graph-bs-decode 4 8 16 24 48 64 80 \
--enable-multimodal \
--mm-attention-backend ascend_attn \
--dtype bfloat16 \
--mamba-ssm-dtype bfloat16 \
--speculative-algorithm NEXTN \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
--reasoning-parser qwen3 \
--tool-call-parser qwen3_coder
벤치마크 (Benchmark)
IMAGE 데이터셋을 1920x1080 해상도로 테스트했어요.
python -m sglang.bench_serving \
--dataset-name image \
--backend sglang-oai-chat \
--host 127.0.0.1 \
--port 6688 \
--max-concurrency 42 \
--num-prompts 42 \
--warmup-requests 42 \
--random-input-len 30 \
--random-output-len 256 \
--random-range-ratio 1 \
--image-resolution 1920x1080 \
--image-count 1 \
--seed 1 \
--request-rate inf
Qwen3.6-35B-A3B 1P IN128K OUT1K 50ms
모델 (Model): Qwen3.6-35B-A3B
하드웨어 (Hardware): Ascend A3 Series Products
카드 (Cards): 1
배포 모드 (Deploy Mode): PD Mixed
양자화 (Quantization): BF16
데이터셋 (Dataset): 128k+1k
TPOT: 50ms
모델 배포 (Model Deployment)
# ============================================================
# Before running, update the following variables:
# MODEL_PATH: path to the model weights directory
# HCCL_SOCKET_IFNAME: network interface name for HCCL
# GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================
MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000
unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING
source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh
export ASCEND_USE_FIA=1
export DEEPEP_HCCL_BUFFSIZE=1600
export GDN_ATTN_BACKEND_TRITON=1
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=20
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32
python3 -m sglang.launch_server \
--model-path $MODEL_PATH \
--host 127.0.0.1 --port 6688 \
--tp-size 2 \
--nnodes 1 \
--attention-backend ascend \
--device npu \
--chunked-prefill-size -1 \
--max-total-tokens 420000 \
--max-prefill-tokens 128000 \
--disable-radix-cache \
--trust-remote-code \
--max-running-requests 3 \
--max-mamba-cache-size 3 \
--mem-fraction-static 0.9 \
--cuda-graph-bs-decode 1 2 3 \
--enable-multimodal \
--mm-attention-backend ascend_attn \
--dtype bfloat16 \
--mamba-ssm-dtype bfloat16 \
--speculative-algorithm NEXTN \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
--reasoning-parser qwen3 \
--tool-call-parser qwen3_coder
벤치마크 (Benchmark)
RANDOM 데이터셋으로 테스트했어요.
python -m sglang.bench_serving \
--dataset-name random \
--backend sglang \
--host 127.0.0.1 \
--port 6688 \
--max-concurrency 3 \
--warmup-requests 3 \
--num-prompts 3 \
--random-input-len 128000 \
--random-output-len 1000 \
--random-range-ratio 1 \
--seed 1
Qwen3.6-35B-A3B 1P IN128K OUT1K PREFIX90 50ms
모델 (Model): Qwen3.6-35B-A3B
하드웨어 (Hardware): Ascend A3 Series Products
카드 (Cards): 1
배포 모드 (Deploy Mode): PD Mixed
양자화 (Quantization): BF16
데이터셋 (Dataset): 128k+1k (90% prefix cache hit rate)
TPOT: 50ms
모델 배포 (Model Deployment)
# ============================================================
# Before running, update the following variables:
# MODEL_PATH: path to the model weights directory
# HCCL_SOCKET_IFNAME: network interface name for HCCL
# GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================
MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000
unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING
source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh
export ASCEND_USE_FIA=1
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=30
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32
python3 -m sglang.launch_server \
--model-path $MODEL_PATH \
--host 127.0.0.1 --port 6688 \
--tp-size 2 \
--nnodes 1 \
--attention-backend ascend \
--device npu \
--chunked-prefill-size 16384 \
--max-prefill-tokens 65536 \
--trust-remote-code \
--enable-prefill-delayer \
--mamba-radix-cache-strategy extra_buffer \
--max-running-requests 103 \
--max-mamba-cache-size 85 \
--mem-fraction-static 0.85 \
--cuda-graph-bs-decode 2 4 8 16 32 48 64 80 96 103 \
--enable-multimodal \
--mm-attention-backend ascend_attn \
--dtype bfloat16 \
--mamba-ssm-dtype bfloat16 \
--speculative-algorithm NEXTN \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
--reasoning-parser qwen3 \
--tool-call-parser qwen3_coder
벤치마크 (Benchmark)
90% 캐시 히트(repeat_rate = 0.9)를 가진 generated-shared-prefix 데이터셋으로 테스트했어요:
--gsp-system-prompt-len 115200 = round(128000 * 0.9)은 공유 프리픽스(shared prefix) 부분이에요.
--gsp-question-len 12800 = round(128000 * (1 - 0.9))은 요청별 고유한 접미사(suffix) 부분이에요.
--gsp-num-groups 1은 캐시 재사용을 최대화하도록 모든 요청을 하나의 프리픽스 그룹으로 유지해요.
python -m sglang.bench_serving \
--dataset-name generated-shared-prefix \
--backend sglang \
--host 127.0.0.1 \
--port 6688 \
--gsp-num-groups 1 \
--gsp-prompts-per-group 412 \
--gsp-system-prompt-len 115200 \
--gsp-question-len 12800 \
--gsp-output-len 1000 \
--max-concurrency 103 \
--num-prompts 412 \
--request-rate inf
Qwen3.6-35B-A3B 1P IN254K OUT1K
모델 (Model): Qwen3.6-35B-A3B
하드웨어 (Hardware): Ascend A3 Series Products
카드 (Cards): 1
배포 모드 (Deploy Mode): PD Mixed
양자화 (Quantization): BF16
데이터셋 (Dataset): 254k+1k
TPOT: 16.1ms
모델 배포 (Model Deployment)
# ============================================================
# Before running, update the following variables:
# MODEL_PATH: path to the model weights directory
# HCCL_SOCKET_IFNAME: network interface name for HCCL
# GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================
MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000
unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING
source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh
export ASCEND_USE_FIA=1
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32
python3 -m sglang.launch_server \
--model-path $MODEL_PATH \
--host 127.0.0.1 --port 6688 \
--tp-size 2 \
--nnodes 1 \
--attention-backend ascend \
--device npu \
--chunked-prefill-size 131072 \
--max-prefill-tokens 254000 \
--disable-radix-cache \
--trust-remote-code \
--max-running-requests 1 \
--max-mamba-cache-size 6 \
--mem-fraction-static 0.65 \
--cuda-graph-bs-decode 1 \
--enable-multimodal \
--mm-attention-backend ascend_attn \
--dtype bfloat16 \
--mamba-ssm-dtype bfloat16 \
--speculative-algorithm NEXTN \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
--reasoning-parser qwen3 \
--tool-call-parser qwen3_coder
벤치마크 (Benchmark)
RANDOM 데이터셋으로 테스트했어요.
python -m sglang.bench_serving \
--dataset-name random \
--backend sglang \
--host 127.0.0.1 \
--port 6688 \
--max-concurrency 1 \
--warmup-requests 1 \
--num-prompts 1 \
--random-input-len 254000 \
--random-output-len 1000 \
--random-range-ratio 1 \
--seed 1
Qwen3.6-35B-A3B 1P IN3K5 OUT1K5 50ms
모델 (Model): Qwen3.6-35B-A3B
하드웨어 (Hardware): Ascend A3 Series Products
카드 (Cards): 1
배포 모드 (Deploy Mode): PD Mixed
양자화 (Quantization): BF16
데이터셋 (Dataset): 3.5k+1.5k
TPOT: 50ms
모델 배포 (Model Deployment)
# ============================================================
# Before running, update the following variables:
# MODEL_PATH: path to the model weights directory
# HCCL_SOCKET_IFNAME: network interface name for HCCL
# GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================
MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000
unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING
source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh
export ASCEND_USE_FIA=1
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_BUFFSIZE=100
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32
python3 -m sglang.launch_server \
--model-path $MODEL_PATH \
--host 127.0.0.1 --port 6688 \
--tp-size 2 \
--nnodes 1 \
--attention-backend ascend \
--device npu \
--chunked-prefill-size -1 \
--max-total-tokens 659840 \
--max-prefill-tokens 43400 \
--disable-radix-cache \
--trust-remote-code \
--prefill-max-requests 12 \
--max-running-requests 122 \
--max-mamba-cache-size 122 \
--mem-fraction-static 0.9 \
--cuda-graph-bs-decode 4 16 32 64 96 116 120 122 \
--enable-multimodal \
--mm-attention-backend ascend_attn \
--dtype bfloat16 \
--mamba-ssm-dtype bfloat16 \
--speculative-algorithm NEXTN \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
--reasoning-parser qwen3 \
--tool-call-parser qwen3_coder
벤치마크 (Benchmark)
RANDOM 데이터셋으로 테스트했어요.
python -m sglang.bench_serving \
--dataset-name random \
--backend sglang \
--host 127.0.0.1 \
--port 6688 \
--max-concurrency 122 \
--warmup-requests 122 \
--num-prompts 122 \
--random-input-len 3500 \
--random-output-len 1500 \
--random-range-ratio 1 \
--seed 1
Qwen3.6-35B-A3B 1P IN64K OUT1K 50ms
모델 (Model): Qwen3.6-35B-A3B
하드웨어 (Hardware): Ascend A3 Series Products
카드 (Cards): 1
배포 모드 (Deploy Mode): PD Mixed
양자화 (Quantization): BF16
데이터셋 (Dataset): 64k+1k
TPOT: 50ms
모델 배포 (Model Deployment)
# ============================================================
# Before running, update the following variables:
# MODEL_PATH: path to the model weights directory
# HCCL_SOCKET_IFNAME: network interface name for HCCL
# GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================
MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000
unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING
source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh
export ASCEND_USE_FIA=1
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=1
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32
python3 -m sglang.launch_server \
--model-path $MODEL_PATH \
--host 127.0.0.1 --port 6688 \
--tp-size 2 \
--nnodes 1 \
--attention-backend ascend \
--device npu \
--chunked-prefill-size -1 \
--max-total-tokens 600000 \
--max-prefill-tokens 65536 \
--disable-radix-cache \
--trust-remote-code \
--enable-prefill-delayer \
--max-running-requests 10 \
--max-mamba-cache-size 20 \
--mem-fraction-static 0.65 \
--cuda-graph-bs-decode 2 4 8 12 14 16 \
--enable-multimodal \
--mm-attention-backend ascend_attn \
--dtype bfloat16 \
--mamba-ssm-dtype bfloat16 \
--speculative-algorithm NEXTN \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
--reasoning-parser qwen3 \
--tool-call-parser qwen3_coder
벤치마크 (Benchmark)
RANDOM 데이터셋으로 테스트했어요.
python -m sglang.bench_serving \
--dataset-name random \
--backend sglang \
--host 127.0.0.1 \
--port 6688 \
--max-concurrency 10 \
--warmup-requests 10 \
--num-prompts 40 \
--random-input-len 64000 \
--random-output-len 1000 \
--random-range-ratio 1 \
--seed 1
Qwen3.6-35B-A3B 1P IN64K OUT1K PREFIX90 50ms
모델 (Model): Qwen3.6-35B-A3B
하드웨어 (Hardware): Ascend A3 Series Products
카드 (Cards): 1
배포 모드 (Deploy Mode): PD Mixed
양자화 (Quantization): BF16
데이터셋 (Dataset): 64k+1k (90% prefix cache hit rate)
TPOT: 50ms
모델 배포 (Model Deployment)
# ============================================================
# Before running, update the following variables:
# MODEL_PATH: path to the model weights directory
# HCCL_SOCKET_IFNAME: network interface name for HCCL
# GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================
MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000
unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING
source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh
export ASCEND_USE_FIA=1
export DEEPEP_HCCL_BUFFSIZE=300
export GDN_ATTN_BACKEND_TRITON=1
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32
python3 -m sglang.launch_server \
--model-path $MODEL_PATH \
--host 127.0.0.1 --port 6688 \
--tp-size 2 \
--nnodes 1 \
--attention-backend ascend \
--device npu \
--chunked-prefill-size -1 \
--max-total-tokens 470784 \
--max-prefill-tokens 65536 \
--trust-remote-code \
--mamba-radix-cache-strategy extra_buffer \
--max-running-requests 40 \
--max-mamba-cache-size 200 \
--mem-fraction-static 0.9 \
--cuda-graph-bs-decode 2 8 16 24 32 36 40 \
--enable-multimodal \
--mm-attention-backend ascend_attn \
--dtype bfloat16 \
--mamba-ssm-dtype bfloat16 \
--speculative-algorithm NEXTN \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
--reasoning-parser qwen3 \
--tool-call-parser qwen3_coder
벤치마크 (Benchmark)
90% 캐시 히트(repeat_rate = 0.9)를 가진 generated-shared-prefix 데이터셋으로 테스트했어요:
--gsp-system-prompt-len 58982 = round(65536 * 0.9)은 공유 프리픽스(shared prefix) 부분이에요.
--gsp-question-len 6554 = round(65536 * (1 - 0.9))은 요청별 고유한 접미사(suffix) 부분이에요.
--gsp-num-groups 1은 캐시 재사용을 최대화하도록 모든 요청을 하나의 프리픽스 그룹으로 유지해요.
python -m sglang.bench_serving \
--dataset-name generated-shared-prefix \
--backend sglang \
--host 127.0.0.1 \
--port 6688 \
--gsp-num-groups 1 \
--gsp-prompts-per-group 40 \
--gsp-system-prompt-len 58982 \
--gsp-question-len 6554 \
--gsp-output-len 1024 \
--max-concurrency 40 \
--num-prompts 40 \
--request-rate inf
Qwen3.6-35B-A3B 2P IN984K OUT1K
모델 (Model): Qwen3.6-35B-A3B
하드웨어 (Hardware): Ascend A3 Series Products
카드 (Cards): 2
배포 모드 (Deploy Mode): PD Mixed
양자화 (Quantization): BF16
데이터셋 (Dataset): 984k+1k
TPOT: 40.91ms
모델 배포 (Model Deployment)
# ============================================================
# Before running, update the following variables:
# MODEL_PATH: path to the model weights directory
# HCCL_SOCKET_IFNAME: network interface name for HCCL
# GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================
MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000
unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING
source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh
export ASCEND_USE_FIA=1
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32
python3 -m sglang.launch_server \
--model-path $MODEL_PATH \
--host 127.0.0.1 --port 6688 \
--tp-size 4 \
--nnodes 1 \
--attention-backend ascend \
--device npu \
--chunked-prefill-size 131072 \
--max-prefill-tokens 984000 \
--disable-radix-cache \
--trust-remote-code \
--max-running-requests 1 \
--max-mamba-cache-size 6 \
--mem-fraction-static 0.68 \
--cuda-graph-bs-decode 1 \
--enable-multimodal \
--mm-attention-backend ascend_attn \
--dtype bfloat16 \
--mamba-ssm-dtype bfloat16 \
--speculative-algorithm NEXTN \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
--context-length 1010000 \
--reasoning-parser qwen3 \
--tool-call-parser qwen3_coder
벤치마크 (Benchmark)
RANDOM 데이터셋으로 테스트했어요.
python -m sglang.bench_serving \
--dataset-name random \
--backend sglang \
--host 127.0.0.1 \
--port 6688 \
--max-concurrency 1 \
--warmup-requests 1 \
--num-prompts 1 \
--random-input-len 984000 \
--random-output-len 1000 \
--random-range-ratio 1 \
--seed 1