Qwen3.6-27B

이 페이지는 Ascend NPU에서 Qwen3.6-27B의 최적 구성과 벤치마크 결과에 집중해요. 환경 설정, 모델 가중치 다운로드, 기능 구성, 배포 지침 등은 Qwen3.6-27B 모델 튜토리얼을 참고하세요.

A3 시리즈에서는 각 카드에 2개의 die가 있어서 --tp-size가 카드 수의 2배예요. 자세한 내용은 Ascend NPU 참조를 확인하세요.

출처: 문서

본문

고처리량 (High Throughput)

모델	하드웨어	카드	배포 모드	데이터셋	TPOT	양자화	구성
Qwen3.6-27B	Ascend A3 Series Products	1	PD Mixed	1024x1024 (30)+1024	50ms	BF16	최적 구성
Qwen3.6-27B	Ascend A3 Series Products	1	PD Mixed	1080p_30+256	50ms	BF16	최적 구성
Qwen3.6-27B	Ascend A3 Series Products	1	PD Mixed	64k+1k (90% prefix cache hit rate)	50ms	BF16	최적 구성
Qwen3.6-27B	Ascend A3 Series Products	1	PD Mixed	3.5k+1.5k	50ms	W8A8 INT8	최적 구성
Qwen3.6-27B	Ascend A3 Series Products	1	PD Mixed	64k+1k	50ms	W8A8 INT8	최적 구성
Qwen3.6-27B	Ascend A3 Series Products	2	PD Mixed	128k+1k	50ms	W8A8 INT8	최적 구성
Qwen3.6-27B	Ascend A3 Series Products	2	PD Mixed	16k+1k	50ms	W8A8 INT8	최적 구성
Qwen3.6-27B	Ascend A3 Series Products	2	PD Mixed	64k+1k	50ms	W8A8 INT8	최적 구성

최적 구성 (Optimal Configuration)

Qwen3.6-27B 1P IN1024X1024 30 OUT1024 50ms

모델: Qwen3.6-27B

하드웨어: Ascend A3 Series Products

카드: 1

배포 모드: PD Mixed

양자화: BF16

데이터셋: 1024x1024 (30)+1024

형식: 해상도 (입력 토큰) + 출력 토큰

TPOT: 50ms

모델 배포 (Model Deployment)

# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=300
export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1
export SGLANG_SET_CPU_AFFINITY=1
export SGLANG_VIT_ENABLE_CUDA_GRAPH=1
export STREAMS_PER_DEVICE=32

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --tp-size 2 \
    --nnodes 1 \
    --attention-backend ascend \
    --device npu \
    --chunked-prefill-size -1 \
    --max-prefill-tokens 52000 \
    --disable-radix-cache \
    --trust-remote-code \
    --max-running-requests 60 \
    --max-mamba-cache-size 60 \
    --mem-fraction-static 0.74 \
    --cuda-graph-bs-decode 2 4 8 14 16 24 26 32 36 37 40 42 44 45 46 50 52 60 \
    --enable-multimodal \
    --mm-attention-backend ascend_attn \
    --dtype bfloat16 \
    --mamba-ssm-dtype bfloat16 \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --mm-enable-dp-encoder \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder

벤치마크 (Benchmark)

1024x1024 해상도의 IMAGE 데이터셋을 기반으로 테스트했어요.

python -m sglang.bench_serving \
    --dataset-name image \
    --backend sglang-oai-chat \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 60 \
    --num-prompts 240 \
    --random-input-len 30 \
    --random-output-len 1024 \
    --random-range-ratio 1 \
    --image-resolution 1024x1024 \
    --image-count 1 \
    --seed 1

Qwen3.6-27B 1P IN1080P 30 OUT256 50ms

모델: Qwen3.6-27B

하드웨어: Ascend A3 Series Products

카드: 1

배포 모드: PD Mixed

양자화: BF16

데이터셋: 1080p_30+256

TPOT: 50ms

모델 배포 (Model Deployment)

# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=<network-interface>
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_SET_CPU_AFFINITY=1
export SGLANG_VIT_ENABLE_CUDA_GRAPH=1
export STREAMS_PER_DEVICE=32

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --tp-size 2 \
    --nnodes 1 \
    --attention-backend ascend \
    --device npu \
    --chunked-prefill-size -1 \
    --max-prefill-tokens 82688 \
    --disable-radix-cache \
    --trust-remote-code \
    --max-running-requests 38 \
    --max-mamba-cache-size 38 \
    --mem-fraction-static 0.7 \
    --cuda-graph-bs-decode 1 2 4 8 10 12 16 20 24 28 30 32 35 38 \
    --enable-prefill-delayer \
    --prefill-delayer-queue-min-ratio 0.45 \
    --prefill-delayer-max-delay-ms 5500 \
    --enable-multimodal \
    --mm-attention-backend ascend_attn \
    --dtype bfloat16 \
    --mamba-ssm-dtype bfloat16 \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder

벤치마크 (Benchmark)

1920x1080 해상도의 IMAGE 데이터셋을 기반으로 테스트했어요.

python -m sglang.bench_serving \
    --dataset-name image \
    --backend sglang-oai-chat \
    --host 127.0.0.1 \
    --port 6688 \
    --warmup-requests 38 \
    --max-concurrency 42 \
    --num-prompts 152 \
    --random-input-len 30 \
    --random-output-len 256 \
    --random-range-ratio 1 \
    --image-resolution 1920x1080 \
    --image-count 1 \
    --seed 1

Qwen3.6-27B 1P IN64K OUT1K PREFIX90 50ms

모델: Qwen3.6-27B

하드웨어: Ascend A3 Series Products

카드: 1

배포 모드: PD Mixed

양자화: BF16

데이터셋: 64k+1k (90% prefix cache hit rate)

TPOT: 50ms

모델 배포 (Model Deployment)

# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export GDN_ATTN_BACKEND_TRITON=1
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --tp-size 2 \
    --nnodes 1 \
    --attention-backend ascend \
    --device npu \
    --chunked-prefill-size 32768 \
    --max-prefill-tokens 32768 \
    --mamba-radix-cache-strategy extra_buffer \
    --trust-remote-code \
    --max-running-requests 20 \
    --max-mamba-cache-size 160 \
    --mem-fraction-static 0.82 \
    --cuda-graph-bs-decode 1 2 5 10 15 17 19 20 \
    --dtype bfloat16 \
    --mamba-ssm-dtype bfloat16 \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder

벤치마크 (Benchmark)

90% 캐시 히트(repeat_rate = 0.9)를 갖는 generated-shared-prefix 데이터셋을 기반으로 테스트했어요: --gsp-system-prompt-len 57600 = round(64000 * 0.9)가 공유 프리픽스 부분이에요. --gsp-question-len 6400 = round(64000 * (1 - 0.9))가 요청별 고유 접미사예요. --gsp-num-groups 1은 최대 캐시 재사용을 위해 모든 요청을 하나의 프리픽스 그룹으로 유지해요.

python -m sglang.bench_serving \
    --dataset-name generated-shared-prefix \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --gsp-num-groups 1 \
    --gsp-prompts-per-group 20 \
    --gsp-system-prompt-len 57600 \
    --gsp-question-len 6400 \
    --gsp-output-len 1000 \
    --max-concurrency 20 \
    --num-prompts 20 \
    --request-rate inf

Qwen3.6-27B W8A8 1P IN3K5 OUT1K5 50ms

모델: Qwen3.6-27B

하드웨어: Ascend A3 Series Products

카드: 1

배포 모드: PD Mixed

양자화: W8A8 INT8

데이터셋: 3.5k+1.5k

TPOT: 50ms

모델 배포 (Model Deployment)

# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0
export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=130
export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --tp-size 2 \
    --nnodes 1 \
    --attention-backend ascend \
    --device npu \
    --chunked-prefill-size -1 \
    --max-prefill-tokens 60000 \
    --disable-radix-cache \
    --trust-remote-code \
    --max-running-requests 64 \
    --max-mamba-cache-size 74 \
    --mem-fraction-static 0.7 \
    --cuda-graph-bs-decode 2 8 16 32 40 45 50 54 \
    --enable-multimodal \
    --quantization modelslim \
    --mm-attention-backend ascend_attn \
    --dtype bfloat16 \
    --mamba-ssm-dtype bfloat16 \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder

벤치마크 (Benchmark)

RANDOM 데이터셋을 기반으로 테스트했어요.

python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 54 \
    --num-prompts 216 \
    --random-input-len 3500 \
    --random-output-len 1500 \
    --random-range-ratio 1 \
    --seed 1

Qwen3.6-27B W8A8 1P IN64K OUT1K 50ms

모델: Qwen3.6-27B

하드웨어: Ascend A3 Series Products

카드: 1

배포 모드: PD Mixed

양자화: W8A8 INT8

데이터셋: 64k+1k

TPOT: 50ms

모델 배포 (Model Deployment)

# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=<network-interface>
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --tp-size 2 \
    --nnodes 1 \
    --attention-backend ascend \
    --device npu \
    --chunked-prefill-size -1 \
    --max-prefill-tokens 48000 \
    --disable-radix-cache \
    --trust-remote-code \
    --max-running-requests 6 \
    --max-mamba-cache-size 16 \
    --mem-fraction-static 0.6 \
    --cuda-graph-bs-decode 1 2 4 5 6 \
    --quantization modelslim \
    --dtype bfloat16 \
    --mamba-ssm-dtype bfloat16 \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder

벤치마크 (Benchmark)

RANDOM 데이터셋을 기반으로 테스트했어요.

python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 6 \
    --num-prompts 12 \
    --random-input-len 64000 \
    --random-output-len 1000 \
    --random-range-ratio 1 \
    --seed 1

Qwen3.6-27B W8A8 2P IN128K OUT1K 50ms

모델: Qwen3.6-27B

하드웨어: Ascend A3 Series Products

카드: 2

배포 모드: PD Mixed

양자화: W8A8 INT8

데이터셋: 128k+1k

TPOT: 50ms

모델 배포 (Model Deployment)

# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=20
export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --tp-size 4 \
    --nnodes 1 \
    --attention-backend ascend \
    --device npu \
    --chunked-prefill-size -1 \
    --max-prefill-tokens 74000 \
    --disable-radix-cache \
    --trust-remote-code \
    --max-running-requests 6 \
    --max-mamba-cache-size 7 \
    --mem-fraction-static 0.63 \
    --cuda-graph-bs-decode 1 2 4 5 6 \
    --enable-multimodal \
    --quantization modelslim \
    --mm-attention-backend ascend_attn \
    --dtype bfloat16 \
    --mamba-ssm-dtype bfloat16 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder

벤치마크 (Benchmark)

RANDOM 데이터셋을 기반으로 테스트했어요.

python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 4 \
    --num-prompts 16 \
    --random-input-len 128000 \
    --random-output-len 1000 \
    --random-range-ratio 1 \
    --seed 1

Qwen3.6-27B W8A8 2P IN16K OUT1K 50ms

모델: Qwen3.6-27B

하드웨어: Ascend A3 Series Products

카드: 2

배포 모드: PD Mixed

양자화: W8A8 INT8

데이터셋: 16k+1k

TPOT: 50ms

모델 배포 (Model Deployment)

# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export GDN_ATTN_BACKEND_TRITON=1
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=<network-interface>
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=50
export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --tp-size 4 \
    --nnodes 1 \
    --attention-backend ascend \
    --device npu \
    --chunked-prefill-size -1 \
    --max-prefill-tokens 58000 \
    --disable-radix-cache \
    --trust-remote-code \
    --max-running-requests 37 \
    --max-mamba-cache-size 74 \
    --mem-fraction-static 0.7 \
    --cuda-graph-bs-decode 1 2 3 4 6 8 10 12 14 16 18 20 21 23 24 25 26 27 28 29 30 31 33 35 37 \
    --quantization modelslim \
    --dtype bfloat16 \
    --mamba-ssm-dtype bfloat16 \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 4 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 5 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder

벤치마크 (Benchmark)

RANDOM 데이터셋을 기반으로 테스트했어요.

python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 37 \
    --warmup-requests 4 \
    --num-prompts 37 \
    --random-input-len 16000 \
    --random-output-len 1000 \
    --random-range-ratio 1 \
    --seed 1

Qwen3.6-27B W8A8 2P IN64K OUT1K 50ms

모델: Qwen3.6-27B

하드웨어: Ascend A3 Series Products

카드: 2

배포 모드: PD Mixed

양자화: W8A8 INT8

데이터셋: 64k+1k

TPOT: 50ms

모델 배포 (Model Deployment)

# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=30
export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --tp-size 4 \
    --nnodes 1 \
    --attention-backend ascend \
    --device npu \
    --chunked-prefill-size -1 \
    --max-prefill-tokens 50000 \
    --disable-radix-cache \
    --trust-remote-code \
    --max-running-requests 28 \
    --max-mamba-cache-size 50 \
    --mem-fraction-static 0.7 \
    --cuda-graph-bs-decode 2 4 6 \
    --enable-multimodal \
    --quantization modelslim \
    --mm-attention-backend ascend_attn \
    --dtype bfloat16 \
    --mamba-ssm-dtype bfloat16 \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4

벤치마크 (Benchmark)

RANDOM 데이터셋을 기반으로 테스트했어요.

python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 6 \
    --num-prompts 24 \
    --random-input-len 64000 \
    --random-output-len 1000 \
    --random-range-ratio 1 \
    --seed 1

더 알아보기 (Learn more)

SGLang Qwen3.6-27B 최적 구성 공식 문서