Llama 4
Llama 4
Llama 4는 Meta의 최신 오픈소스 LLM 세대로, 업계 최고 수준의 성능을 제공해요. SGLang은 v0.4.5부터 Llama 4 Scout (109B)와 Llama 4 Maverick (400B)를 지원해요. 지속적 최적화는 Roadmap에서 추적돼요.
출처: 문서
본문
1. Model Introduction
Llama 4는 Meta의 최신 오픈소스 LLM 세대로 업계 최고 수준의 성능을 보여줘요.
SGLang은 v0.4.5부터 Llama 4 Scout (109B)와 Llama 4 Maverick (400B)을 지원해요.
진행 중인 최적화는 Roadmap에서 추적돼요.
이 세대는 전반에 걸친 종합적 업그레이드를 제공해요:
매우 강력한 Llama 4 Maverick은 약 400B 총 파라미터 중 17B 활성 파라미터에 128개 전문가를 가져요. 효율적인 Llama 4 Scout 역시 약 109B 총 파라미터 중 17B 활성 파라미터에 16개 전문가만 사용해요. 두 모델 모두 네이티브 멀티모달을 위해 early fusion을 활용해 텍스트와 이미지 입력을 처리할 수 있어요. Maverick과 Scout 모두 200개 언어를 아우르는 데이터로 최대 40조 토큰으로 학습됐어요(아랍어, 스페인어, 독일어, 힌디어를 포함한 12개 언어에 대한 특정 파인튜닝 지원).
자세한 내용은 공식 llama4 저장소를 참조하세요: https://www.llama.com/models/llama-4/
2. SGLang Installation
SGLang은 여러 설치 방법을 제공해요. 하드웨어 플랫폼과 요구사항에 따라 가장 적합한 설치 방법을 선택할 수 있어요.
설치 지침은 공식 SGLang 설치 가이드를 참조하세요.
SGLang CPU 설치에 대해서는 CPU 버전 설치 가이드를 참조하세요.
3. Model Deployment
이 섹션은 빠른 배포에서 성능 최적화까지 단계별 가이드를 제공하며, 다양한 수준의 사용자에게 적합해요.
3.1 Basic Configuration
상단의 Command Generator를 사용해 하드웨어 플랫폼·모델 변형·배포 전략·thinking 능력에 맞는 배포 명령을 자동 생성하세요. 예시 기본 명령(MI300x, BF16):
python -m sglang.launch_server \
--model-path meta-llama/Llama-4-Maverick-17B-128E-Instruct \
--tp 8 \
--enable-multimodal \
--context-length 65536 \
--dtype bfloat16 \
--trust-remote-code \
--host 0.0.0.0 \
--port 8000
XEON (CPU):
--device cpu --disable-overlap-schedule --tp 6. Tool calling:--tool-call-parser pythonic. EAGLE3 스페큘레이티브:--speculative-algorithm EAGLE3 --speculative-draft-model-path lmsys/sglang-EAGLE3-Llama-4-{Maverick-17B-128E/Scout-17B-16E}-Instruct-v1 --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --mem-fraction-static 0.75 --cuda-graph-max-bs-decode 2.
3.2 Configuration Tips
- OOM 완화: GPU out-of-memory를 피하려면
--context-length를 줄이세요. 권장: Scout는 8×H100에서 최대 1M, 8×H200에서 최대 2.5M; Maverick은 8×H200에서 context-length를 설정할 필요 없음. 하이브리드 KV cache를 켜면 Scout는 8×H100에서 5M, 8×H200에서 10M에 도달할 수 있어요. - 어텐션 백엔드 자동 선택: SGLang이 최적 백엔드를 자동 선택해요.
--attention-backend로 수동 오버라이드:- Blackwell (B200/GB200):
trtllm_mha - Hopper (H100/H200):
fa3 - AMD GPU:
aiter - Intel XPU:
intel_xpu - 기타:
triton
- Blackwell (B200/GB200):
- Chat Template: 채팅 완료 작업에
--chat-template llama-4을 추가하세요. - Multi-Modal: 이미지 입력 지원에
--enable-multimodal을 추가하세요. - Hybrid KV Cache:
--swa-full-tokens-ratio로 SWA(로컬 어텐션) KV 토큰과 full-어텐션 KV 토큰의 비율을 제어하세요(기본: 0.8, 범위: 0–1). - EAGLE 스페큘레이티브 디코딩: EAGLE3를 통해 Llama 4 Scout와 Maverick에서 지원돼요. 위 대화형 명령 생성기로 켜세요.
- Xeon CPU 서비스 구성: SGLang CPU 서버 문서의 서빙 엔진 시작 섹션 Notes 부분을 참조해 TP(tensor parallel)와 NUMA 바인딩 설정을 잘 이해하세요.
4. Model Invocation
4.1 Basic Usage
기본 API 사용과 요청 예시는 다음을 참조하세요:
4.2 Advanced Usage
4.2.1 Launch the docker
docker pull lmsysorg/sglang:v0.5.9-rocm720-mi30x
docker run -d -it --ipc=host --network=host --privileged \
--cap-add=CAP_SYS_ADMIN \
--device=/dev/kfd --device=/dev/dri --device=/dev/mem \
--group-add video --cap-add=SYS_PTRACE \
--security-opt seccomp=unconfined \
-v /:/work \
-e SHELL=/bin/bash \
--name Llama4 \
lmsysorg/sglang:v0.5.9-rocm720-mi30x \
/bin/bash
4.2.2 Launch the server
Llama-4-Scout
8-GPU 배포 명령:
sglang serve \
--model-path meta-llama/Llama-4-Scout-17B-16E-Instruct \
--tp 8 \
--context-length 1000000 \
--trust-remote-code
Llama-4-Maverick
8-GPU 배포 명령:
sglang serve \
--model-path meta-llama/Llama-4-Maverick-17B-128E-Instruct \
--tp 8 \
--trust-remote-code
4.2.3 EAGLE Speculative Decoding
SGLang은 EAGLE 스페큘레이티브 디코딩으로 Llama 4 Maverick (400B)을 지원해요. EAGLE3 알고리즘과 SGLang EAGLE3 드래프트 모델로 켜세요:
python3 -m sglang.launch_server \
--model-path meta-llama/Llama-4-Maverick-17B-128E-Instruct \
--speculative-algorithm EAGLE3 \
--speculative-draft-model-path lmsys/sglang-EAGLE3-Llama-4-Maverick-17B-128E-Instruct-v1 \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
--trust-remote-code \
--tp 8
5. Benchmark
5.1 Speed Benchmark (Scout)
테스트 환경:
하드웨어: AMD MI300x GPU
모델: Llama-4-Scout
텐서 병렬화: 8
sglang 버전: 0.5.9
- 모델 배포
sglang serve \
--model-path meta-llama/Llama-4-Scout-17B-16E-Instruct \
--tp 8 \
--context-length 1000000 \
--trust-remote-code
5.1.1 Low Concurrency (Latency-Optimized)
- 벤치마크 명령:
python3 -m sglang.bench_serving \
--backend sglang \
--model meta-llama/Llama-4-Scout-17B-16E-Instruct \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1 \
--request-rate inf
- 테스트 결과:
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 74.62
Total input tokens: 6101
Total input text tokens: 6101
Total input vision tokens: 0
Total generated tokens: 4220
Total generated tokens (retokenized): 4211
Request throughput (req/s): 0.14
Input token throughput (tok/s): 82.88
Output token throughput (tok/s): 57.42
Peak output token throughput (tok/s): 146.00
Peak concurrent requests: 2
Total token throughput (tok/s): 140.20
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 7459.48
Median E2E Latency (ms): 4489.77
---------------Time to First Token----------------
Mean TTFT (ms): 4246.98
Median TTFT (ms): 68.57
P99 TTFT (ms): 48091.05
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 7.49
Median TPOT (ms): 7.40
P99 TPOT (ms): 7.40
---------------Inter-Token Latency----------------
Mean ITL (ms): 7.49
Median ITL (ms): 7.49
P95 ITL (ms): 7.47
P99 ITL (ms): 7.52
Max ITL (ms): 10.44
==================================================
5.1.2 Medium Concurrency (Balanced)
- 벤치마크 명령:
python3 -m sglang.bench_serving \
--backend sglang \
--model meta-llama/Llama-4-Scout-17B-16E-Instruct \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 80 \
--max-concurrency 16 \
--request-rate inf
- 테스트 결과:
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 45.41
Total input tokens: 49668
Total input text tokens: 49668
Total input vision tokens: 0
Total generated tokens: 40805
Total generated tokens (retokenized): 40516
Request throughput (req/s): 2.26
Input token throughput (tok/s): 1120.46
Output token throughput (tok/s): 1152.47
Peak output token throughput (tok/s): 1520.00
Peak concurrent requests: 21
Total token throughput (tok/s): 2272.84
Concurrency: 14.76
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 6089.22
Median E2E Latency (ms): 6568.80
---------------Time to First Token----------------
Mean TTFT (ms): 124.44
Median TTFT (ms): 87.42
P99 TTFT (ms): 268.72
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 11.88
Median TPOT (ms): 12.00
P99 TPOT (ms): 15.49
---------------Inter-Token Latency----------------
Mean ITL (ms): 11.72
Median ITL (ms): 10.54
P95 ITL (ms): 11.22
P99 ITL (ms): 67.88
Max ITL (ms): 74.05
==================================================
5.1.3 High Concurrency (Throughput-Optimized)
- 벤치마크 명령:
python3 -m sglang.bench_serving \
--backend sglang \
--model meta-llama/Llama-4-Scout-17B-16E-Instruct \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 500 \
--max-concurrency 100 \
--request-rate inf
- 테스트 결과:
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 500
Benchmark duration (s): 85.84
Total input tokens: 249841
Total input text tokens: 249841
Total input vision tokens: 0
Total generated tokens: 252662
Total generated tokens (retokenized): 250498
Request throughput (req/s): 5.84
Input token throughput (tok/s): 2910.84
Output token throughput (tok/s): 2944.82
Peak output token throughput (tok/s): 4100.00
Peak concurrent requests: 110
Total token throughput (tok/s): 5854.65
Concurrency: 92.24
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 15844.00
Median E2E Latency (ms): 15262.56
---------------Time to First Token----------------
Mean TTFT (ms): 204.46
Median TTFT (ms): 129.96
P99 TTFT (ms): 528.54
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 41.56
Median TPOT (ms): 42.90
P99 TPOT (ms): 47.48
---------------Inter-Token Latency----------------
Mean ITL (ms): 40.99
Median ITL (ms): 24.46
P95 ITL (ms): 84.46
P99 ITL (ms): 87.64
Max ITL (ms): 226.06
==================================================
5.2 Speed Benchmark (Maverick)
테스트 환경:
하드웨어: AMD MI300x GPU
모델: Llama-4-Maverick
텐서 병렬화: 8
sglang 버전: 0.5.9
- 모델 배포
sglang serve \
--model-path meta-llama/Llama-4-Maverick-17B-128E-Instruct \
--tp 8 \
--context-length 1000000 \
--trust-remote-code
5.2.1 Low Concurrency (Latency-Optimized)
- 벤치마크 명령:
python3 -m sglang.bench_serving \
--backend sglang \
--model meta-llama/Llama-4-Maverick-17B-128E-Instruct \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1 \
--request-rate inf
- 테스트 결과:
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 68.08
Total input tokens: 6101
Total input text tokens: 6101
Total input vision tokens: 0
Total generated tokens: 4220
Total generated tokens (retokenized): 4202
Request throughput (req/s): 0.15
Input token throughput (tok/s): 89.62
Output token throughput (tok/s): 61.99
Peak output token throughput (tok/s): 168.00
Peak concurrent requests: 2
Total token throughput (tok/s): 151.61
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 6805.62
Median E2E Latency (ms): 2733.91
---------------Time to First Token----------------
Mean TTFT (ms): 4296.56
Median TTFT (ms): 57.45
P99 TTFT (ms): 38633.95
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 5.95
Median TPOT (ms): 5.96
P99 TPOT (ms): 5.97
---------------Inter-Token Latency----------------
Mean ITL (ms): 5.96
Median ITL (ms): 5.96
P95 ITL (ms): 6.02
P99 ITL (ms): 6.08
Max ITL (ms): 7.02
==================================================
5.2.2 Medium Concurrency (Balanced)
- 벤치마크 명령:
python3 -m sglang.bench_serving \
--backend sglang \
--model meta-llama/Llama-4-Maverick-17B-128E-Instruct \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 80 \
--max-concurrency 16 \
--request-rate inf
- 테스트 결과:
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 30.72
Total input tokens: 39668
Total input text tokens: 39668
Total input vision tokens: 0
Total generated tokens: 40805
Total generated tokens (retokenized): 40923
Request throughput (req/s): 2.60
Input token throughput (tok/s): 1291.39
Output token throughput (tok/s): 1328.41
Peak output token throughput (tok/s): 1760.00
Peak concurrent requests: 22
Total token throughput (tok/s): 2619.80
Concurrency: 13.92
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 5345.15
Median E2E Latency (ms): 5679.73
---------------Time to First Token----------------
Mean TTFT (ms): 259.30
Median TTFT (ms): 72.60
P99 TTFT (ms): 1063.45
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 10.53
Median TPOT (ms): 10.22
P99 TPOT (ms): 20.27
---------------Inter-Token Latency----------------
Mean ITL (ms): 9.99
Median ITL (ms): 9.10
P95 ITL (ms): 9.87
P99 ITL (ms): 55.62
Max ITL (ms): 868.54
==================================================
5.2.3 High Concurrency (Throughput-Optimized)
- 벤치마크 명령:
python3 -m sglang.bench_serving \
--backend sglang \
--model meta-llama/Llama-4-Maverick-17B-128E-Instruct \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 500 \
--max-concurrency 100 \
--request-rate inf
- 테스트 결과:
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 500
Benchmark duration (s): 90.95
Total input tokens: 249831
Total input text tokens: 249831
Total input vision tokens: 0
Total generated tokens: 252662
Total generated tokens (retokenized): 251625
Request throughput (req/s): 5.50
Input token throughput (tok/s): 2746.77
Output token throughput (tok/s): 2777.90
Peak output token throughput (tok/s): 3700.00
Peak concurrent requests: 109
Total token throughput (tok/s): 5524.67
Concurrency: 93.04
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 16924.17
Median E2E Latency (ms): 16294.85
---------------Time to First Token----------------
Mean TTFT (ms): 188.19
Median TTFT (ms): 128.96
P99 TTFT (ms): 534.81
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 33.63
Median TPOT (ms): 35.37
P99 TPOT (ms): 38.26
---------------Inter-Token Latency----------------
Mean ITL (ms): 33.19
Median ITL (ms): 27.66
P95 ITL (ms): 76.91
P99 ITL (ms): 78.82
Max ITL (ms): 268.17
==================================================
5.3 Accuracy Benchmark
5.3.1 GSM8K Benchmark
- 벤치마크 명령:
python3 -m sglang.test.few_shot_gsm8k --num-questions 200
- Llama-4-Scout-17B-16E-Instruct
Accuracy: 0.945
Invalid: 0.000
Latency: 12.731 s
Output throughput: 1595.418 token/s
- Llama-4-Maverick-17B-128E-Instruct
Accuracy: 0.895
Invalid: 0.000
Latency: 9.739 s
Output throughput: 2405.505 token/s
5.3.2 MMLU Pro with lm-eval
MMLU Pro의 정확도는 8×H100에서 Meta 공식 벤치마크 수치와 일치해요(재현 세부사항: PR #5092):
| Model | Official | SGLang |
|---|---|---|
| Llama-4-Scout-17B-16E-Instruct | 74.3 | 75.2 |
| Llama-4-Maverick-17B-128E-Instruct | 80.5 | 80.7 |
Scout:
# Start the server
python -m sglang.launch_server \
--model-path meta-llama/Llama-4-Scout-17B-16E-Instruct \
--port 30000 \
--tp 8 \
--mem-fraction-static 0.8 \
--context-length 65536
# Run lm_eval
lm_eval --model local-chat-completions \
--model_args model=meta-llama/Llama-4-Scout-17B-16E-Instruct,base_url=http://localhost:30000/v1/chat/completions,num_concurrent=128,timeout=999999,max_gen_toks=2048 \
--tasks mmlu_pro \
--batch_size 128 \
--apply_chat_template \
--num_fewshot 0
Maverick:
# Start the server
python -m sglang.launch_server \
--model-path meta-llama/Llama-4-Maverick-17B-128E-Instruct \
--port 30000 \
--tp 8 \
--mem-fraction-static 0.8 \
--context-length 65536
# Run lm_eval
lm_eval --model local-chat-completions \
--model_args model=meta-llama/Llama-4-Maverick-17B-128E-Instruct,base_url=http://localhost:30000/v1/chat/completions,num_concurrent=128,timeout=999999,max_gen_toks=2048 \
--tasks mmlu_pro \
--batch_size 128 \
--apply_chat_template \
--num_fewshot 0