DeepSeekV32-Exp RBG 기반 PD 배포
DeepSeekV32-Exp RBG 기반 PD 배포 (DeepSeekV32-Exp RBG Based PD Deploy)
이 페이지는 Kubernetes의 RBG(RoleBasedGroup)를 사용해 DeepSeek V3.2-Exp의 PD(Prefill/Decode) 분리 배포를 구성하는 방법을 설명해요. RBG는 prefill, decode, router 역할을 하나의 manifest로 정의해 관리하는 방식이에요.
출처: 문서
본문
0. 사전 조건 (Prerequisites)
- k8s >= 1.26
- k8s에 lws 설치
- k8s에 rbg 설치
RBG 설치는 https://github.com/sgl-project/rbg을 참고해 주세요.
1. 이미지 준비 (Image Preparation)
lmsysorg/sglang:latest
2. 올인원 매니페스트 파일 (All In One manifest file)
참고: NodeSelector 섹션, 모델 위치 섹션, taint toleration 섹션은 실제 배포 환경에 따라 조정할 수 있어요.
rbg-dsv32.yml
apiVersion: workloads.x-k8s.io/v1alpha1
kind: RoleBasedGroup
metadata:
name: deepseek-rbg-32exp
namespace: default
spec:
roles:
- name: prefill
replicas: 1
workload:
apiVersion: leaderworkerset.x-k8s.io/v1
kind: LeaderWorkerSet
restartPolicy: None
leaderWorkerSet:
size: 1
patchLeaderTemplate:
metadata:
labels:
role: leader
pd_role: prefill
spec:
containers:
- command:
- python3
- -m
- sglang.launch_server
- --model-path
- /work/models
- --port
- "30000"
- --trust-remote
- --host
- 0.0.0.0
- --disaggregation-ib-device
- mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7
- --disable-radix-cache
- --chunked-prefill-size
- "131072"
- --page-size
- "64"
- --ep-dispatch-algorithm
- dynamic
- --eplb-algorithm
- deepseek
- --enable-dp-lm-head
- --enable-dp-attention
- --dp-size
- "8"
- --moe-a2a-backend
- deepep
- --deepep-mode
- normal
- --disaggregation-mode
- prefill
- --mem-fraction-static
- "0.8"
- --max-prefill-tokens
- "32768"
- --context-length
- "32768"
- --tp
- "8"
- --dist-init-addr
- $(LWS_LEADER_ADDRESS):20102
- --nnodes
- $(LWS_GROUP_SIZE)
- --node-rank
- $(LWS_WORKER_INDEX)
- --trust-remote-code
- --ep-num-redundant-experts
- "32"
- --moe-dense-tp-size
- "1"
- --max-running-requests
- "1024"
env:
- name: LWS_WORKER_INDEX
valueFrom:
fieldRef:
fieldPath: metadata.labels['leaderworkerset.sigs.k8s.io/worker-index']
livenessProbe:
failureThreshold: 3000
httpGet:
path: /health
port: 30000
initialDelaySeconds: 300
periodSeconds: 60
successThreshold: 1
timeoutSeconds: 10
readinessProbe:
failureThreshold: 20
httpGet:
path: /health
port: 30000
periodSeconds: 30
successThreshold: 1
timeoutSeconds: 10
name: sglang
ports:
- containerPort: 30000
name: sglang-http
protocol: TCP
patchWorkerTemplate: {}
template:
metadata:
labels:
inference-framework: sglang
inference-stack.io/monitoring: "enabled"
spec:
containers:
- name: sglang
image: lmsysorg/sglang:latest
env:
- name: SGLANG_SKIP_SGL_KERNEL_VERSION_CHECK
value: "1"
- name: CUDA_LAUNCH_BLOCKING
value: "0"
- name: SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT
value: "1000000000"
- name: NVSHMEM_IB_TRAFFIC_CLASS
value: "16"
- name: NVSHMEM_DISABLE_P2P
value: "0"
- name: ENABLE_METRICS
value: "true"
- name: NVSHMEM_IB_GID_INDEX
value: "3"
- name: NVSHMEM_IB_SL
value: "5"
- name: SGLANG_SET_CPU_AFFINITY
value: "true"
- name: SGL_ENABLE_JIT_DEEPGEMM
value: "1"
- name: NCCL_IB_QPS_PER_CONNECTION
value: "8"
- name: NCCL_IB_SPLIT_DATA_ON_QPS
value: "1"
- name: NCCL_NET_PLUGIN
value: "none"
- name: NCCL_IB_TC
value: "136"
- name: NCCL_IB_SL
value: "5"
- name: NCCL_IB_TIMEOUT
value: "22"
- name: NCCL_IB_GID_INDEX
value: "3"
- name: NCCL_MIN_NCHANNELS
value: "4"
- name: NCCL_SOCKET_IFNAME
value: bond0
- name: GLOO_SOCKET_IFNAME
value: bond0
- name: NCCL_IB_HCA
value: ^=mlx5_0,mlx5_5,mlx5_6
- name: NVSHMEM_BOOTSTRAP_UID_SOCK_IFNAME
value: "bond0"
- name: MC_TE_METRIC
value: "false"
resources:
limits:
nvidia.com/gpu: "8"
securityContext:
capabilities:
add:
- IPC_LOCK
privileged: true
volumeMounts:
- mountPath: /root/.cache
name: sgl-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /work/models
name: model
- mountPath: /dev/infiniband
name: ib
- mountPath: /sgl-workspace/sglang
name: src
dnsPolicy: ClusterFirstWithHostNet
hostIPC: true
hostNetwork: true
nodeSelector:
pd: "yes"
tolerations:
- key: pd
operator: Exists
volumes:
- hostPath:
path: /var/run/sys-topology
name: topo
- hostPath:
path: /data1/sgl_cache4
type: DirectoryOrCreate
name: sgl-cache
- emptyDir:
medium: Memory
name: dshm
- hostPath:
path: /data/DeepSeek-V3.2-Exp
name: model
- hostPath:
path: /dev/infiniband
name: ib
- hostPath:
path: /data/src/sglang
type: DirectoryOrCreate
name: src
- name: decode
replicas: 1
workload:
apiVersion: leaderworkerset.x-k8s.io/v1
kind: LeaderWorkerSet
leaderWorkerSet:
size: 1
patchLeaderTemplate:
metadata:
labels:
role: leader
pd_role: decode
spec:
containers:
- command:
- python3
- -m
- sglang.launch_server
- --model-path
- /work/models
- --port
- "30000"
- --trust-remote
- --host
- 0.0.0.0
- --disaggregation-ib-device
- mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7
- --chunked-prefill-size
- "131072"
- --eplb-rebalance-layers-per-chunk
- "29"
- --page-size
- "64"
- --enable-dp-attention
- --enable-dp-lm-head
- --dp-size
- "8"
- --moe-a2a-backend
- deepep
- --deepep-mode
- low_latency
- --disaggregation-mode
- decode
- --mem-fraction-static
- "0.8"
- --context-length
- "32768"
- --max-running-requests
- "2048"
- --tp-size
- "8" # Size of Tensor Parallelism
- --cuda-graph-max-bs-decode
- "16"
- --dist-init-addr
- $(LWS_LEADER_ADDRESS):20102
- --nnodes
- $(LWS_GROUP_SIZE)
- --node-rank
- $(LWS_WORKER_INDEX)
- --trust-remote-code
- --ep-num-redundant-experts
- "32"
- --moe-dense-tp-size
- "1"
env:
- name: LWS_WORKER_INDEX
valueFrom:
fieldRef:
fieldPath: metadata.labels['leaderworkerset.sigs.k8s.io/worker-index']
livenessProbe:
failureThreshold: 30000
httpGet:
path: /health
port: 30000
initialDelaySeconds: 300
periodSeconds: 60
successThreshold: 1
timeoutSeconds: 10
name: sglang
readinessProbe:
failureThreshold: 20
httpGet:
path: /health
port: 30000
periodSeconds: 30
successThreshold: 1
timeoutSeconds: 10
patchWorkerTemplate:
spec:
containers:
- command:
- python3
- -m
- sglang.launch_server
- --model-path
- /work/models
- --crash-dump-folder
- /log
- --chunked-prefill-size
- "262144"
- --eplb-rebalance-layers-per-chunk
- "29"
- --page-size
- "64"
- --enable-dp-attention
- --enable-dp-lm-head
- --dp-size
- "32"
- --moe-a2a-backend
- "deepep"
- --deepep-mode
- low_latency
- --disaggregation-mode
- decode
- --mem-fraction-static
- "0.849"
- --context-length
- "32768"
- --disaggregation-ib-device
- mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7
- --max-running-requests
- "4096"
- --cuda-graph-max-bs-decode
- "16"
- --tp-size
- "8" # Size of Tensor Parallelism
- --dist-init-addr
- $(LWS_LEADER_ADDRESS):20102
- --nnodes
- $(LWS_GROUP_SIZE)
- --node-rank
- $(LWS_WORKER_INDEX)
- --trust-remote-code
- --ep-num-redundant-experts
- "32"
- --moe-dense-tp-size
- "1"
env:
- name: LWS_WORKER_INDEX
valueFrom:
fieldRef:
fieldPath: metadata.labels['leaderworkerset.sigs.k8s.io/worker-index']
name: sglang
template:
metadata:
labels:
inference-framework: sglang-unuse
inference-stack.io/monitoring: "enabled"
spec:
containers:
- image: lmsysorg/sglang:latest
name: sglang
resources:
limits:
nvidia.com/gpu: "8"
securityContext:
capabilities:
add:
- IPC_LOCK
privileged: true
volumeMounts:
- mountPath: /root/.cache
name: sgl-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /work/models
name: model
- mountPath: /dev/infiniband
name: ib
- mountPath: /sgl-workspace/sglang
name: src
env:
- name: SGLANG_SKIP_SGL_KERNEL_VERSION_CHECK
value: "1"
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: "100000000"
- name: NVSHMEM_DISABLE_P2P
value: "0"
- name: NVSHMEM_IB_TRAFFIC_CLASS
value: "16"
- name: NVSHMEM_IB_SL
value: "5"
- name: ENABLE_METRICS
value: "true"
- name: CUDA_LAUNCH_BLOCKING
value: "0"
- name: NVSHMEM_IB_GID_INDEX
value: "3"
- name: NCCL_IB_QPS_PER_CONNECTION
value: "8"
- name: NCCL_IB_SPLIT_DATA_ON_QPS
value: "1"
- name: NCCL_NET_PLUGIN
value: "none"
- name: NCCL_IB_TC
value: "136"
- name: NCCL_IB_SL
value: "5"
- name: NCCL_IB_TIMEOUT
value: "22"
- name: NCCL_IB_GID_INDEX
value: "3"
- name: NCCL_MIN_NCHANNELS
value: "4"
- name: NCCL_SOCKET_IFNAME
value: bond0
- name: GLOO_SOCKET_IFNAME
value: bond0
- name: NVSHMEM_BOOTSTRAP_UID_SOCK_IFNAME
value: "bond0"
- name: NCCL_IB_HCA
value: ^=mlx5_0,mlx5_5,mlx5_6
- name: MC_TE_METRIC
value: "false"
- name: SGL_ENABLE_JIT_DEEPGEMM
value: "1"
dnsPolicy: ClusterFirstWithHostNet
hostIPC: true
hostNetwork: true
nodeSelector:
pd: "yes"
tolerations:
- key: pd
operator: Exists
volumes:
- hostPath:
path: /var/run/sys-topology
name: topo
- hostPath:
path: /data1/sgl_cache4
type: DirectoryOrCreate
name: sgl-cache
- hostPath:
path: /data/src/sglang
type: DirectoryOrCreate
name: src
- emptyDir:
medium: Memory
name: dshm
- hostPath:
path: /data/DeepSeek-V3.2-Exp
name: model
- hostPath:
path: /dev/infiniband
name: ib
- name: router
replicas: 1
dependencies: [ "decode", "prefill" ]
template:
spec:
containers:
- name: scheduler
image: lmsysorg/sglang:latest
command:
- sh
- -c
- >
python3 -m sglang_router.launch_router
--host 0.0.0.0
--port 8080
--pd-disaggregation
--policy random
--service-discovery
--service-discovery-namespace ${NAMESPACE}
--service-discovery-port 30000
--prefill-selector pd_role=prefill
--decode-selector pd_role=decode
--max-payload-size 2147483648
--worker-startup-timeout-secs 1200
env:
- name: NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
***
apiVersion: v1
kind: Service
metadata:
labels:
app: deepseek-rbg-32exp
name: deepseek-rbg-32exp
namespace: default
spec:
ports:
- name: http
port: 8080
protocol: TCP
targetPort: 8080
nodePort: 30080
selector:
rolebasedgroup.workloads.x-k8s.io/name: deepseek-rbg-32exp
rolebasedgroup.workloads.x-k8s.io/role: router
type: NodePort
배포 후 파드 상태를 확인:
[root@ecs-001]# kubectl get po -n default
deepseek-rbg-32exp-decode-main-0 1/1 Running 0 74m
deepseek-rbg-32exp-decode-0-1 1/1 Running 0 74m
deepseek-rbg-32exp-router-9c5dbfc57 1/1 Running 0 22m
deepseek-rbg-32exp-prefill-0 1/1 Running 0 74m
이 시점에서 nodePort:30800을 선택해 접근해요:
curl -X POST "http://{nodePort}:30800/v1/chat/completions" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ***" \
-d '{
"rid":"ccccdd",
"model": "dsv32",
"messages": [
{"role": "system", "content": "0: You are a helpful AI assistant"},
{"role": "user", "content": "你是谁？."}
],
"max_tokens":221
}'
FAQ
-
현재 배포 시작 파라미터는 모든 RDMA 시나리오와 완전히 호환되지 않을 수 있어요. 네트워크 환경에 따라 다른 RDMA NCCL 관련 환경 구성이 필요할 수 있어요.
-
이미지의 sglang 코드가 PR #10912의 변경 사항을 포함했는지 확인해 주세요.