GLM-5.1

GLM-5.1 W4A8 16P IN3K5 OUT1K5 50ms

모델: GLM-5.1

하드웨어: Ascend A3 Series Products

카드: 16

배포 모드: PD Mixed

양자화: W4A8 INT8

데이터셋: 3.5k+1.5k

TPOT: 50ms

모델 배포 (Model Deployment)

# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   NODE_IPS: IP addresses of each node in the cluster
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
NODE_IPS=('<your node1 ip>' '<your node2 ip>')

echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export DEEPEP_HCCL_BUFFSIZE=2500
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_SOCKET_IFNAME=<network-interface>
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
echo "${LOCAL_HOST1}"
echo "${LOCAL_HOST2}"

for i in "${!NODE_IPS[@]}";
do
    if [[ "$LOCAL_HOST1" == "${NODE_IPS[$i]}" || "$LOCAL_HOST2" == "${NODE_IPS[$i]}" ]];
    then
        echo "${NODE_IPS[$i]}"
        python3 -m sglang.launch_server \
        --model-path $MODEL_PATH \
        --host ${NODE_IPS[$i]} --port 6688 \
        --nnodes 2 \
        --dist-init-addr ${NODE_IPS[0]}:5000 \
        --node-rank $i \
        --attention-backend ascend \
        --device npu \
        --tp-size 32 \
        --dp-size 16 \
        --enable-dp-attention \
        --chunked-prefill-size 65536 \
        --max-prefill-tokens 280000 \
        --trust-remote-code \
        --mem-fraction-static 0.65 \
        --served-model-name glm-5 \
        --cuda-graph-max-bs-decode 16 \
        --max-running-requests 256 \
        --quantization modelslim \
        --speculative-draft-model-quantization unquant \
        --moe-a2a-backend deepep \
        --deepep-mode auto \
        --load-balance-method round_robin \
        --speculative-algorithm NEXTN \
        --speculative-num-steps 3 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 4 \
        --reasoning-parser glm45 \
        --tool-call-parser glm47
        break
    fi
done

벤치마크 (Benchmark)

RANDOM 데이터셋을 기반으로 테스트했어요.

python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 128 \
    --num-prompts 128 \
    --random-input-len 3500 \
    --random-output-len 1500 \
    --random-range-ratio 1 \
    --seed 1

GLM-5.1 W4A8 1P1D 32P IN128K OUT1K 56.4ms

모델: GLM-5.1

하드웨어: Ascend A3 Series Products

카드: 32

배포 모드: PD Disaggregation

양자화: W4A8 INT8

데이터셋: 128k+1k

TPOT: 56.4ms

TTFT: 13.1s

모델 배포 (Model Deployment)

# ============================================================
# Before running, update the following variables:
#   P_IP: prefill node IP address
#   D_IP: decode node IP address
#   ASCEND_MF_STORE_URL: prefill node IP with port
#   MODEL_PATH: path to the model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================


echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=1200
export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=1200
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

P_IP=('<your prefill ip1>' '<your prefill ip2>')
D_IP=('<your decode ip1>' '<your decode ip2>')

export ASCEND_MF_STORE_URL="tcp://<your prefill ip1>:24670"

MODEL_PATH=/path/to/model-weights

LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
echo "${LOCAL_HOST1}"
echo "${LOCAL_HOST2}"
# prefill
for i in "${!P_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
    then
        echo "${P_IP[$i]}"
        export DEEPEP_HCCL_BUFFSIZE=1200
        export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1
        export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=1024
        export DEEPEP_NORMAL_LONG_SEQ_ROUND=72
        export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0
        export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24699
        export SGLANG_ZBAL_LOCAL_MEM_SIZE=61184
        export TASK_QUEUE_ENABLE=2
        export ZBAL_ENABLE_GRAPH=1

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode prefill \
        --host ${P_IP[$i]} \
        --port 8000 \
        --dist-init-addr ${P_IP[0]}:5000 \
        --disaggregation-bootstrap-port 8998 \
        --node-rank $i \
        --tp-size 4 \
        --nnodes 2 \
        --mem-fraction-static 0.72 \
        --attention-backend ascend \
        --device npu \
        --quantization modelslim \
        --disaggregation-transfer-backend ascend \
        --max-running-requests 16 \
        --served-model-name glm-5 \
        --chunked-prefill-size 32768 \
        --max-prefill-tokens 180000 \
        --moe-a2a-backend deepep \
        --deepep-mode normal \
        --disable-shared-experts-fusion \
        --disable-cuda-graph \
        --dtype bfloat16 \
        --speculative-draft-model-quantization unquant \
        --enable-nsa-prefill-context-parallel \
        --nsa-prefill-cp-mode in-seq-split \
        --attn-cp-size 4 \
        --disable-radix-cache \
        --enable-dp-lm-head \
        --moe-dense-tp 1 \
        --pp-size 8 \
        --trust-remote-code
        break
    fi
done

# decode
for i in "${!D_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
    then
        echo "${D_IP[$i]}"
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_BUFFSIZE=200
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=16
        export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
        export SGLANG_NPU_USE_MULTI_STREAM=1
        export SGLANG_SPEC_ENABLE_OVERLAP_REFLOW=1
        export TASK_QUEUE_ENABLE=0

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode decode \
        --host ${D_IP[$i]} \
        --port 8001 \
        --dist-init-addr ${D_IP[0]}:5000 \
        --node-rank $i \
        --trust-remote-code \
        --tp-size 32 \
        --nnodes 2 \
        --dp-size 32 \
        --ep-size 32 \
        --enable-dp-attention \
        --mem-fraction-static 0.85 \
        --max-running-requests 32 \
        --attention-backend ascend \
        --device npu \
        --quantization modelslim \
        --served-model-name glm-5 \
        --moe-a2a-backend deepep \
        --deepep-mode low_latency \
        --cuda-graph-bs-decode 1 2 3 \
        --disaggregation-transfer-backend ascend \
        --watchdog-timeout 9000 \
        --context-length 180000 \
        --tokenizer-worker-num 16 \
        --disable-shared-experts-fusion \
        --dtype bfloat16 \
        --load-balance-method round_robin \
        --speculative-draft-model-quantization unquant \
        --speculative-algorithm NEXTN \
        --speculative-num-steps 3 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 4
        break
    fi
done

# ============================================================
# Before running, replace the following placeholders:
#   <your prefill ip>: prefill node IP address
#   <your decode ip1>: first decode node IP address (decode may have distributed nodes)
# ============================================================

python -m sglang_router.launch_router \
    --pd-disaggregation \
    --prefill http://<your prefill ip>:8000 8998 \
    --decode http://<your decode ip1>:8001 \
    --host 127.0.0.1 \
    --port 6688 \
    --policy round_robin

벤치마크 (Benchmark)

RANDOM 데이터셋을 기반으로 테스트했어요.

python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 32 \
    --num-prompts 32 \
    --random-input-len 131072 \
    --random-output-len 1024 \
    --random-range-ratio 1 \
    --seed 1

GLM-5.1 W4A8 1P1D 32P IN16K OUT1K 50ms

모델: GLM-5.1

하드웨어: Ascend A3 Series Products

카드: 32

배포 모드: PD Disaggregation

양자화: W4A8 INT8

데이터셋: 16k+1k

TPOT: 50ms

모델 배포 (Model Deployment)

# ============================================================
# Before running, update the following variables:
#   P_IP: prefill node IP address
#   D_IP: decode node IP address
#   ASCEND_MF_STORE_URL: prefill node IP with port
#   MODEL_PATH: path to the model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================


echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

P_IP=('<your prefill ip1>' '<your prefill ip2>')
D_IP=('<your decode ip1>' '<your decode ip2>')

export ASCEND_MF_STORE_URL="tcp://<your prefill ip1>:24670"

MODEL_PATH=/path/to/model-weights

LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
echo "${LOCAL_HOST1}"
echo "${LOCAL_HOST2}"
# prefill
for i in "${!P_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
    then
        echo "${P_IP[$i]}"
        export DEEPEP_HCCL_BUFFSIZE=1200
        export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1
        export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_SOCKET_IFNAME=<network-interface>
        export TASK_QUEUE_ENABLE=2

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode prefill \
        --host ${P_IP[$i]} \
        --port 8000 \
        --dist-init-addr ${P_IP[0]}:5000 \
        --disaggregation-bootstrap-port 8998 \
        --node-rank $i \
        --tp-size 32 \
        --nnodes 2 \
        --mem-fraction-static 0.75 \
        --attention-backend ascend \
        --device npu \
        --quantization modelslim \
        --disaggregation-transfer-backend ascend \
        --max-running-requests 64 \
        --served-model-name glm-5 \
        --chunked-prefill-size 524288 \
        --max-prefill-tokens 180000 \
        --moe-a2a-backend deepep \
        --deepep-mode normal \
        --disable-shared-experts-fusion \
        --disable-cuda-graph \
        --dtype bfloat16 \
        --dp-size 4 \
        --enable-dp-attention \
        --load-balance-method round_robin \
        --enable-prefill-cp \
        --cp-strategy zigzag \
        --attn-cp-size 8 \
        --enable-dp-lm-head \
        --moe-dense-tp 1 \
        --reasoning-parser glm45 \
        --tool-call-parser glm47 \
        --trust-remote-code
        break
    fi
done

# decode
for i in "${!D_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
    then
        echo "${D_IP[$i]}"
        export DEEPEP_HCCL_BUFFSIZE=650
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=64
        export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
        export SGLANG_SPEC_ENABLE_OVERLAP_REFLOW=1
        export TASK_QUEUE_ENABLE=0

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode decode \
        --host ${D_IP[$i]} \
        --port 8001 \
        --dist-init-addr ${D_IP[0]}:5000 \
        --node-rank $i \
        --tp-size 32 \
        --nnodes 2 \
        --dp-size 32 \
        --ep-size 32 \
        --enable-dp-attention \
        --mem-fraction-static 0.87 \
        --max-running-requests 96 \
        --attention-backend ascend \
        --device npu \
        --quantization modelslim \
        --served-model-name glm-5 \
        --moe-a2a-backend deepep \
        --deepep-mode low_latency \
        --cuda-graph-bs-decode 1 2 3 \
        --disaggregation-transfer-backend ascend \
        --watchdog-timeout 9000 \
        --context-length 180000 \
        --tokenizer-worker-num 4 \
        --disable-shared-experts-fusion \
        --dtype bfloat16 \
        --load-balance-method round_robin \
        --speculative-draft-model-quantization unquant \
        --speculative-algorithm NEXTN \
        --speculative-num-steps 3 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 4 \
        --reasoning-parser glm45 \
        --tool-call-parser glm47 \
        --trust-remote-code
        break
    fi
done

# ============================================================
# Before running, replace the following placeholders:
#   <your prefill ip>: prefill node IP address
#   <your decode ip1>: first decode node IP address (decode may have distributed nodes)
# ============================================================

python -m sglang_router.launch_router \
    --pd-disaggregation \
    --prefill http://<your prefill ip>:8000 8998 \
    --decode http://<your decode ip1>:8001 \
    --host 127.0.0.1 \
    --port 6688 \
    --policy round_robin

벤치마크 (Benchmark)

RANDOM 데이터셋을 기반으로 테스트했어요.

python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 128 \
    --num-prompts 512 \
    --random-input-len 16384 \
    --random-output-len 1024 \
    --random-range-ratio 1 \
    --seed 1

GLM-5.1 W4A8 1P1D 32P IN64K OUT1K 55.2ms

모델: GLM-5.1

하드웨어: Ascend A3 Series Products

카드: 32

배포 모드: PD Disaggregation

양자화: W4A8 INT8

데이터셋: 64k+1k

TPOT: 55.2ms

TTFT: 7.58s

모델 배포 (Model Deployment)

# ============================================================
# Before running, update the following variables:
#   P_IP: prefill node IP address
#   D_IP: decode node IP address
#   ASCEND_MF_STORE_URL: prefill node IP with port
#   MODEL_PATH: path to the model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================


echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=1200
export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=1200
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

P_IP=('<your prefill ip1>' '<your prefill ip2>')
D_IP=('<your decode ip1>' '<your decode ip2>')

export ASCEND_MF_STORE_URL="tcp://<your prefill ip1>:24670"

MODEL_PATH=/path/to/model-weights

LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
echo "${LOCAL_HOST1}"
echo "${LOCAL_HOST2}"
# prefill
for i in "${!P_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
    then
        echo "${P_IP[$i]}"
        export DEEPEP_HCCL_BUFFSIZE=1200
        export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1
        export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=1024
        export DEEPEP_NORMAL_LONG_SEQ_ROUND=72
        export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0
        export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24699
        export SGLANG_ZBAL_LOCAL_MEM_SIZE=61184
        export TASK_QUEUE_ENABLE=2
        export ZBAL_ENABLE_GRAPH=1
        export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode prefill \
        --host ${P_IP[$i]} \
        --port 8000 \
        --dist-init-addr ${P_IP[0]}:5000 \
        --disaggregation-bootstrap-port 8998 \
        --node-rank $i \
        --trust-remote-code \
        --tp-size 4 \
        --nnodes 2 \
        --mem-fraction-static 0.72 \
        --attention-backend ascend \
        --device npu \
        --quantization modelslim \
        --disaggregation-transfer-backend ascend \
        --max-running-requests 16 \
        --served-model-name glm-5 \
        --chunked-prefill-size 32768 \
        --max-prefill-tokens 180000 \
        --moe-a2a-backend deepep \
        --deepep-mode normal \
        --disable-shared-experts-fusion \
        --disable-cuda-graph \
        --dtype bfloat16 \
        --speculative-draft-model-quantization unquant \
        --enable-nsa-prefill-context-parallel \
        --nsa-prefill-cp-mode in-seq-split \
        --attn-cp-size 4 \
        --disable-radix-cache \
        --enable-dp-lm-head \
        --moe-dense-tp 1 \
        --pp-size 8
        break
    fi
done

# decode
for i in "${!D_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
    then
        echo "${D_IP[$i]}"
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_BUFFSIZE=200
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=16
        export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
        export SGLANG_NPU_USE_MULTI_STREAM=1
        export SGLANG_SPEC_ENABLE_OVERLAP_REFLOW=1
        export TASK_QUEUE_ENABLE=0

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode decode \
        --host ${D_IP[$i]} \
        --port 8001 \
        --dist-init-addr ${D_IP[0]}:5000 \
        --node-rank $i \
        --tp-size 32 \
        --nnodes 2 \
        --dp-size 32 \
        --enable-dp-attention \
        --ep-size 32 \
        --mem-fraction-static 0.85 \
        --max-running-requests 32 \
        --attention-backend ascend \
        --device npu \
        --quantization modelslim \
        --served-model-name glm-5 \
        --moe-a2a-backend deepep \
        --deepep-mode low_latency \
        --cuda-graph-bs-decode 1 2 3 \
        --disaggregation-transfer-backend ascend \
        --watchdog-timeout 9000 \
        --context-length 180000 \
        --tokenizer-worker-num 16 \
        --disable-shared-experts-fusion \
        --dtype bfloat16 \
        --load-balance-method round_robin \
        --speculative-draft-model-quantization unquant \
        --speculative-algorithm NEXTN \
        --speculative-num-steps 3 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 4 \
        --trust-remote-code
        break
    fi
done

# ============================================================
# Before running, replace the following placeholders:
#   <your prefill ip>: prefill node IP address
#   <your decode ip1>: first decode node IP address (decode may have distributed nodes)
# ============================================================

python -m sglang_router.launch_router \
    --pd-disaggregation \
    --prefill http://<your prefill ip>:8000 8998 \
    --decode http://<your decode ip1>:8001 \
    --host 127.0.0.1 \
    --port 6688 \
    --policy round_robin

벤치마크 (Benchmark)

RANDOM 데이터셋을 기반으로 테스트했어요.

python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 32 \
    --num-prompts 32 \
    --random-input-len 65536 \
    --random-output-len 1024 \
    --random-range-ratio 1 \
    --seed 1

GLM-5.1 W4A8 1P1D 32P IN64K OUT1K 50ms

모델: GLM-5.1

하드웨어: Ascend A3 Series Products

카드: 32

배포 모드: PD Disaggregation

양자화: W4A8 INT8

데이터셋: 64k+1k

TPOT: 50ms

모델 배포 (Model Deployment)

# ============================================================
# Before running, update the following variables:
#   P_IP: prefill node IP address
#   D_IP: decode node IP address
#   ASCEND_MF_STORE_URL: prefill node IP with port
#   MODEL_PATH: path to the model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================


echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=1200
export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=1200
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

P_IP=('<your prefill ip1>' '<your prefill ip2>')
D_IP=('<your decode ip1>' '<your decode ip2>')

export ASCEND_MF_STORE_URL="tcp://<your prefill ip1>:24670"

MODEL_PATH=/path/to/model-weights

LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
echo "${LOCAL_HOST1}"
echo "${LOCAL_HOST2}"
# prefill
for i in "${!P_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
    then
        echo "${P_IP[$i]}"
        export DEEPEP_HCCL_BUFFSIZE=1200
        export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1
        export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=1024
        export DEEPEP_NORMAL_LONG_SEQ_ROUND=72
        export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_SOCKET_IFNAME=<network-interface>
        export TASK_QUEUE_ENABLE=2

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode prefill \
        --host ${P_IP[$i]} \
        --port 8000 \
        --dist-init-addr ${P_IP[0]}:5000 \
        --disaggregation-bootstrap-port 8998 \
        --node-rank $i \
        --tp-size 4 \
        --nnodes 2 \
        --mem-fraction-static 0.72 \
        --attention-backend ascend \
        --device npu \
        --quantization modelslim \
        --disaggregation-transfer-backend ascend \
        --max-running-requests 16 \
        --served-model-name glm-5 \
        --chunked-prefill-size 16384 \
        --max-prefill-tokens 180000 \
        --moe-a2a-backend deepep \
        --deepep-mode normal \
        --disable-shared-experts-fusion \
        --disable-cuda-graph \
        --dtype bfloat16 \
        --speculative-draft-model-quantization unquant \
        --enable-nsa-prefill-context-parallel \
        --nsa-prefill-cp-mode in-seq-split \
        --attn-cp-size 4 \
        --enable-dp-lm-head \
        --moe-dense-tp 1 \
        --pp-size 8 \
        --reasoning-parser glm45 \
        --tool-call-parser glm47 \
        --trust-remote-code
        break
    fi
done

# decode
for i in "${!D_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
    then
        echo "${D_IP[$i]}"
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_BUFFSIZE=200
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=16
        export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
        export SGLANG_SPEC_ENABLE_OVERLAP_REFLOW=1
        export TASK_QUEUE_ENABLE=0

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode decode \
        --host ${D_IP[$i]} \
        --port 8001 \
        --dist-init-addr ${D_IP[0]}:5000 \
        --node-rank $i \
        --tp-size 32 \
        --nnodes 2 \
        --dp-size 32 \
        --enable-dp-attention \
        --ep-size 32 \
        --mem-fraction-static 0.85 \
        --max-running-requests 32 \
        --attention-backend ascend \
        --device npu \
        --quantization modelslim \
        --served-model-name glm-5 \
        --moe-a2a-backend deepep \
        --deepep-mode low_latency \
        --cuda-graph-bs-decode 1 2 3 \
        --disaggregation-transfer-backend ascend \
        --watchdog-timeout 9000 \
        --context-length 180000 \
        --tokenizer-worker-num 16 \
        --disable-shared-experts-fusion \
        --dtype bfloat16 \
        --load-balance-method round_robin \
        --speculative-draft-model-quantization unquant \
        --reasoning-parser glm45 \
        --tool-call-parser glm47 \
        --trust-remote-code
        break
    fi
done

# ============================================================
# Before running, replace the following placeholders:
#   <your prefill ip>: prefill node IP address
#   <your decode ip1>: first decode node IP address (decode may have distributed nodes)
# ============================================================

python -m sglang_router.launch_router \
    --pd-disaggregation \
    --prefill http://<your prefill ip>:8000 8998 \
    --decode http://<your decode ip1>:8001 \
    --host 127.0.0.1 \
    --port 6688 \
    --policy round_robin

벤치마크 (Benchmark)

RANDOM 데이터셋을 기반으로 테스트했어요.

python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 1 \
    --num-prompts 1 \
    --random-input-len 65536 \
    --random-output-len 1024 \
    --random-range-ratio 1 \
    --seed 1

GLM-5.1 W4A8 1P1D 32P IN65K OUT1K5 PREFIX90 25ms

모델: GLM-5.1

하드웨어: Ascend A3 Series Products

카드: 32

배포 모드: PD Disaggregation

양자화: W4A8 INT8

데이터셋: 65k+1.5k (90% prefix cache hit rate)

TPOT: 25ms

모델 배포 (Model Deployment)

# ============================================================
# Before running, update the following variables:
#   P_IP: prefill node IP address
#   D_IP: decode node IP address
#   ASCEND_MF_STORE_URL: prefill node IP with port
#   MODEL_PATH: path to the model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================


echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

P_IP=('<your prefill ip1>' '<your prefill ip2>')
D_IP=('<your decode ip1>' '<your decode ip2>')

export ASCEND_MF_STORE_URL="tcp://<your prefill ip1>:24670"

MODEL_PATH=/path/to/model-weights

LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
echo "${LOCAL_HOST1}"
echo "${LOCAL_HOST2}"
# prefill
for i in "${!P_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
    then
        echo "${P_IP[$i]}"
        export DEEPEP_HCCL_BUFFSIZE=1200
        export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1
        export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_SOCKET_IFNAME=<network-interface>
        export TASK_QUEUE_ENABLE=2

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode prefill \
        --host ${P_IP[$i]} \
        --port 8000 \
        --dist-init-addr ${P_IP[0]}:5000 \
        --disaggregation-bootstrap-port 8998 \
        --node-rank $i \
        --tp-size 32 \
        --nnodes 2 \
        --mem-fraction-static 0.75 \
        --attention-backend ascend \
        --device npu \
        --quantization modelslim \
        --disaggregation-transfer-backend ascend \
        --max-running-requests 64 \
        --served-model-name glm-5 \
        --chunked-prefill-size 53248 \
        --max-prefill-tokens 180000 \
        --moe-a2a-backend deepep \
        --deepep-mode normal \
        --disable-shared-experts-fusion \
        --disable-cuda-graph \
        --dtype bfloat16 \
        --dp-size 4 \
        --enable-dp-attention \
        --load-balance-method round_robin \
        --enable-prefill-cp \
        --cp-strategy zigzag \
        --attn-cp-size 8 \
        --enable-dp-lm-head \
        --moe-dense-tp 1 \
        --reasoning-parser glm45 \
        --tool-call-parser glm47 \
        --trust-remote-code
        break
    fi
done

# decode
for i in "${!D_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
    then
        echo "${D_IP[$i]}"
        export DEEPEP_HCCL_BUFFSIZE=650
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=64
        export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
        export SGLANG_SPEC_ENABLE_OVERLAP_REFLOW=1
        export TASK_QUEUE_ENABLE=0

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode decode \
        --host ${D_IP[$i]} \
        --port 8001 \
        --dist-init-addr ${D_IP[0]}:5000 \
        --node-rank $i \
        --tp-size 32 \
        --nnodes 2 \
        --dp-size 32 \
        --ep-size 32 \
        --enable-dp-attention \
        --mem-fraction-static 0.87 \
        --max-running-requests 96 \
        --attention-backend ascend \
        --device npu \
        --quantization modelslim \
        --served-model-name glm-5 \
        --moe-a2a-backend deepep \
        --deepep-mode low_latency \
        --cuda-graph-bs-decode 1 2 3 \
        --disaggregation-transfer-backend ascend \
        --watchdog-timeout 9000 \
        --context-length 180000 \
        --tokenizer-worker-num 4 \
        --disable-shared-experts-fusion \
        --dtype bfloat16 \
        --load-balance-method round_robin \
        --speculative-draft-model-quantization unquant \
        --speculative-algorithm NEXTN \
        --speculative-num-steps 3 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 4 \
        --reasoning-parser glm45 \
        --tool-call-parser glm47 \
        --trust-remote-code
        break
    fi
done

# ============================================================
# Before running, replace the following placeholders:
#   <your prefill ip>: prefill node IP address
#   <your decode ip1>: first decode node IP address (decode may have distributed nodes)
# ============================================================

python -m sglang_router.launch_router \
    --pd-disaggregation \
    --prefill http://<your prefill ip>:8000 8998 \
    --decode http://<your decode ip1>:8001 \
    --host 127.0.0.1 \
    --port 6688 \
    --policy round_robin

벤치마크 (Benchmark)

90% 캐시 히트(repeat_rate = 0.9)를 갖는 generated-shared-prefix 데이터셋을 기반으로 테스트했어요: --gsp-system-prompt-len 59904 = round(66560 * 0.9)가 공유 프리픽스 부분이에요. --gsp-question-len 6656 = round(66560 * (1 - 0.9))가 요청별 고유 접미사예요. --gsp-num-groups 1은 최대 캐시 재사용을 위해 모든 요청을 하나의 프리픽스 그룹으로 유지해요.

python -m sglang.bench_serving \
    --dataset-name generated-shared-prefix \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --gsp-num-groups 1 \
    --gsp-prompts-per-group 480 \
    --gsp-system-prompt-len 59904 \
    --gsp-question-len 6656 \
    --gsp-output-len 1536 \
    --max-concurrency 100 \
    --num-prompts 480 \
    --request-rate inf

GLM-5.1 W4A8 2P1D 48P IN128K OUT1K PREFIX90 50ms

모델: GLM-5.1

하드웨어: Ascend A3 Series Products

카드: 48

배포 모드: PD Disaggregation

양자화: W4A8 INT8

데이터셋: 128k+1k (90% prefix cache hit rate)

TPOT: 50ms

모델 배포 (Model Deployment)

# ============================================================
# Before running, update the following variables:
#   P_IP: prefill node IP address
#   D_IP: decode node IP address
#   ASCEND_MF_STORE_URL: prefill node IP with port
#   MODEL_PATH: path to the model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================


echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=1200
export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=1200
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

P_IP=('<your prefill ip1>' '<your prefill ip2>' '<your prefill ip3>' '<your prefill ip4>')
D_IP=('<your decode ip1>' '<your decode ip2>')

export ASCEND_MF_STORE_URL="tcp://<your prefill ip1>:24670"

MODEL_PATH=/path/to/model-weights

LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
echo "${LOCAL_HOST1}"
echo "${LOCAL_HOST2}"
# prefill
for i in "${!P_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
    then
        echo "${P_IP[$i]}"
        export DEEPEP_HCCL_BUFFSIZE=1200
        export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1
        export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=1024
        export DEEPEP_NORMAL_LONG_SEQ_ROUND=72
        export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_SOCKET_IFNAME=<network-interface>
        export TASK_QUEUE_ENABLE=2

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode prefill \
        --host ${P_IP[$i]} \
        --port 8000 \
        --dist-init-addr ${P_IP[$(( $i / 2 * 2 ))]}:5000 \
        --disaggregation-bootstrap-port $((8998 + $i / 2)) \
        --node-rank $(( $i % 2 )) \
        --tp-size 4 \
        --nnodes 2 \
        --mem-fraction-static 0.72 \
        --attention-backend ascend \
        --device npu \
        --quantization modelslim \
        --disaggregation-transfer-backend ascend \
        --max-running-requests 32 \
        --served-model-name glm-5 \
        --chunked-prefill-size 16384 \
        --max-prefill-tokens 180000 \
        --moe-a2a-backend deepep \
        --deepep-mode normal \
        --disable-shared-experts-fusion \
        --disable-cuda-graph \
        --dtype bfloat16 \
        --speculative-draft-model-quantization unquant \
        --enable-prefill-cp \
        --cp-strategy zigzag \
        --attn-cp-size 4 \
        --enable-dp-lm-head \
        --moe-dense-tp 1 \
        --pp-size 8 \
        --reasoning-parser glm45 \
        --tool-call-parser glm47 \
        --trust-remote-code
        break
    fi
done

# decode
for i in "${!D_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
    then
        echo "${D_IP[$i]}"
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_BUFFSIZE=200
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=24
        export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
        export SGLANG_SPEC_ENABLE_OVERLAP_REFLOW=1
        export TASK_QUEUE_ENABLE=0

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode decode \
        --host ${D_IP[$i]} \
        --port 8001 \
        --dist-init-addr ${D_IP[0]}:5000 \
        --node-rank $i \
        --tp-size 32 \
        --nnodes 2 \
        --dp-size 32 \
        --ep-size 32 \
        --enable-dp-attention \
        --mem-fraction-static 0.865 \
        --max-running-requests 96 \
        --attention-backend ascend \
        --device npu \
        --quantization modelslim \
        --served-model-name glm-5 \
        --moe-a2a-backend deepep \
        --deepep-mode low_latency \
        --cuda-graph-bs-decode 1 2 3 4 5 6 \
        --disaggregation-transfer-backend ascend \
        --watchdog-timeout 9000 \
        --context-length 180000 \
        --tokenizer-worker-num 32 \
        --disable-shared-experts-fusion \
        --dtype bfloat16 \
        --load-balance-method round_robin \
        --speculative-draft-model-quantization unquant \
        --speculative-algorithm NEXTN \
        --speculative-num-steps 3 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 4 \
        --reasoning-parser glm45 \
        --tool-call-parser glm47 \
        --trust-remote-code
        break
    fi
done

# ============================================================
# Before running, replace the following placeholders:
#   <your prefill ip1>, <your prefill ip2>: prefill node IP addresses
#   <your decode ip1>: first decode node IP address (decode may have distributed nodes)
# ============================================================

python -m sglang_router.launch_router \
    --pd-disaggregation \
    --prefill http://<your prefill ip1>:8000 8998 \
    --prefill http://<your prefill ip2>:8000 8999 \
    --decode http://<your decode ip1>:8001 \
    --host 127.0.0.1 \
    --port 6688 \
    --policy round_robin

벤치마크 (Benchmark)

90% 캐시 히트(repeat_rate = 0.9)를 갖는 generated-shared-prefix 데이터셋을 기반으로 테스트했어요: --gsp-system-prompt-len 117965 = round(131072 * 0.9)가 공유 프리픽스 부분이에요. --gsp-question-len 13107 = round(131072 * (1 - 0.9))가 요청별 고유 접미사예요. --gsp-num-groups 1은 최대 캐시 재사용을 위해 모든 요청을 하나의 프리픽스 그룹으로 유지해요.

python -m sglang.bench_serving \
    --dataset-name generated-shared-prefix \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --gsp-num-groups 1 \
    --gsp-prompts-per-group 576 \
    --gsp-system-prompt-len 117965 \
    --gsp-question-len 13107 \
    --gsp-output-len 1024 \
    --max-concurrency 144 \
    --num-prompts 576 \
    --request-rate inf

GLM-5.1 W4A8 4P1D 48P IN64K OUT1K PREFIX90 50ms

모델: GLM-5.1

하드웨어: Ascend A3 Series Products

카드: 48

배포 모드: PD Disaggregation

양자화: W4A8 INT8

데이터셋: 64k+1k (90% prefix cache hit rate)

TPOT: 50ms

모델 배포 (Model Deployment)

# ============================================================
# Before running, update the following variables:
#   P_IP: prefill node IP address
#   D_IP: decode node IP address
#   ASCEND_MF_STORE_URL: prefill node IP with port
#   MODEL_PATH: path to the model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================


echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=1200
export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=1200
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

P_IP=('<your prefill ip1>' '<your prefill ip2>' '<your prefill ip3>' '<your prefill ip4>')
D_IP=('<your decode ip1>' '<your decode ip2>')

export ASCEND_MF_STORE_URL="tcp://<your prefill ip1>:24670"

MODEL_PATH=/path/to/model-weights

LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
echo "${LOCAL_HOST1}"
echo "${LOCAL_HOST2}"
# prefill
for i in "${!P_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
    then
        echo "${P_IP[$i]}"
        export DEEPEP_HCCL_BUFFSIZE=1200
        export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1
        export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=1024
        export DEEPEP_NORMAL_LONG_SEQ_ROUND=72
        export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_SOCKET_IFNAME=<network-interface>
        export TASK_QUEUE_ENABLE=2

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode prefill \
        --host ${P_IP[$i]} \
        --port 8000 \
        --disaggregation-bootstrap-port $((8998 + $i)) \
        --node-rank 0 \
        --tp-size 4 \
        --nnodes 1 \
        --mem-fraction-static 0.72 \
        --attention-backend ascend \
        --device npu \
        --quantization modelslim \
        --disaggregation-transfer-backend ascend \
        --max-running-requests 16 \
        --served-model-name glm-5 \
        --chunked-prefill-size 16384 \
        --max-prefill-tokens 180000 \
        --moe-a2a-backend deepep \
        --deepep-mode normal \
        --disable-shared-experts-fusion \
        --disable-cuda-graph \
        --dtype bfloat16 \
        --speculative-draft-model-quantization unquant \
        --enable-prefill-cp \
        --cp-strategy zigzag \
        --attn-cp-size 4 \
        --enable-dp-lm-head \
        --moe-dense-tp 1 \
        --pp-size 4 \
        --reasoning-parser glm45 \
        --tool-call-parser glm47 \
        --trust-remote-code
        break
    fi
done

# decode
for i in "${!D_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
    then
        echo "${D_IP[$i]}"
        export DEEPEP_HCCL_BUFFSIZE=300
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=40
        export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
        export SGLANG_SPEC_ENABLE_OVERLAP_REFLOW=1
        export TASK_QUEUE_ENABLE=0

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode decode \
        --host ${D_IP[$i]} \
        --port 8001 \
        --dist-init-addr ${D_IP[0]}:5000 \
        --node-rank $i \
        --tp-size 32 \
        --nnodes 2 \
        --dp-size 32 \
        --ep-size 32 \
        --enable-dp-attention \
        --mem-fraction-static 0.85 \
        --max-running-requests 320 \
        --attention-backend ascend \
        --device npu \
        --quantization modelslim \
        --served-model-name glm-5 \
        --moe-a2a-backend deepep \
        --deepep-mode low_latency \
        --cuda-graph-bs-decode 1 2 3 4 5 6 7 8 9 10 \
        --disaggregation-transfer-backend ascend \
        --watchdog-timeout 9000 \
        --context-length 180000 \
        --tokenizer-worker-num 4 \
        --disable-shared-experts-fusion \
        --dtype bfloat16 \
        --load-balance-method round_robin \
        --speculative-draft-model-quantization unquant \
        --speculative-algorithm NEXTN \
        --speculative-num-steps 3 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 4 \
        --reasoning-parser glm45 \
        --tool-call-parser glm47 \
        --trust-remote-code
        break
    fi
done

# ============================================================
# Before running, replace the following placeholders:
#   <your prefill ip1>, <your prefill ip2>, <your prefill ip3>, <your prefill ip4>: prefill node IP addresses
#   <your decode ip1>: first decode node IP address (decode may have distributed nodes)
# ============================================================

python -m sglang_router.launch_router \
    --pd-disaggregation \
    --prefill http://<your prefill ip1>:8000 8998 \
    --prefill http://<your prefill ip2>:8000 8999 \
    --prefill http://<your prefill ip3>:8000 9000 \
    --prefill http://<your prefill ip4>:8000 9001 \
    --decode http://<your decode ip1>:8001 \
    --host 127.0.0.1 \
    --port 6688 \
    --policy round_robin

벤치마크 (Benchmark)

90% 캐시 히트(repeat_rate = 0.9)를 갖는 generated-shared-prefix 데이터셋을 기반으로 테스트했어요: --gsp-system-prompt-len 58982 = round(65536 * 0.9)가 공유 프리픽스 부분이에요. --gsp-question-len 6554 = round(65536 * (1 - 0.9))가 요청별 고유 접미사예요. --gsp-num-groups 1은 최대 캐시 재사용을 위해 모든 요청을 하나의 프리픽스 그룹으로 유지해요.

python -m sglang.bench_serving \
    --dataset-name generated-shared-prefix \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --gsp-num-groups 1 \
    --gsp-prompts-per-group 1280 \
    --gsp-system-prompt-len 58982 \
    --gsp-question-len 6554 \
    --gsp-output-len 1024 \
    --max-concurrency 320 \
    --num-prompts 1280 \
    --request-rate inf

더 알아보기 (Learn more)