Llama-3.1
Llama-3.1
Llama 3.1은 Meta가 2024년 7월에 공개한 사전학습 및 지시 튜닝 생성 모델 모음이에요. 8B·70B·405B 세 가지 크기로 제공되며, 405B 변형이 당시 가장 강력한 완전 공개 모델이었어요. 코딩·tool use·reasoning에서 크게 개선됐고, 컨텍스트를 128K 토큰으로 확장했으며, 영어 외 7개 언어를 지원해요. SGLang으로 NVIDIA·AMD GPU, Intel Arc Pro B-Series GPU, Intel Xeon CPU에서 배포할 수 있어요.
출처: 문서
본문
1. Model Introduction
Llama 3.1은 Meta가 2024년 7월에 공개한 사전학습 및 지시 튜닝 생성 모델 모음이에요. 이 모델은 8B·70B·405B 크기로 제공되며, 405B 변형이 당시 가장 강력한 완전 공개 소스 모델이었어요.
이 모델들은 몇 가지 새로운 기능과 개선으로 모두에게 공개 지능을 가져다줘요:
- 더 강한 일반 지능: 이 모델들은 코딩, 최첨단 tool use, 전반적으로 더 강력한 reasoning 능력에서 큰 개선을 보여줘요.
- 확장된 컨텍스트 길이: Llama 3.1은 컨텍스트 길이를 128K 토큰으로 확장해 요약·코드 reasoning 같은 긴 컨텍스트 작업 성능을 향상시켜요.
- Tool Use: Llama 3.1은 검색 엔진·Python 인터프리터·수학 엔진과 상호작용하도록 학습됐고, 잠재적으로 본 적 없는 도구와 상호작용하는 zero-shot tool use 능력도 개선해요.
- 다국어: Llama 3.1은 영어 외 7개 언어를 지원해요: 프랑스어, 독일어, 힌디어, 이탈리아어, 포르투갈어, 스페인어, 태국어.
자세한 내용은 Llama 3.1 블로그와 Llama 3.1 model card를 참조하세요.
2. SGLang Installation
SGLang은 여러 설치 방법을 제공해요. 하드웨어 플랫폼과 요구사항에 따라 가장 적합한 설치 방법을 선택할 수 있어요.
설치 지침은 공식 SGLang 설치 가이드를 참조하세요.
3. Model Deployment
이 섹션은 NVIDIA GPU, AMD GPU, Intel Arc Pro B-Series GPU(코드네임: BMG (Battlemage)), Intel Xeon CPU를 포함한 다양한 하드웨어 플랫폼에 최적화된 배포 구성을 제공해요.
3.1 Basic Configuration
상단의 Command Generator를 사용해 Llama 3.1 모델 모음용 시작 명령을 생성하세요. 하드웨어·모델 크기(8B/70B/405B)·카테고리(Base/Instruct)·양자화(BF16/FP8)·tool-call parser·최적화 모드(Basic/Throughput/Latency)를 선택할 수 있어요. 예시 기본 명령(70B Instruct, BF16):
sglang serve \
--model-path meta-llama/Llama-3.1-70B-Instruct
Throughput 최적화(NVIDIA):
--enable-dp-attention --mem-fraction-static 0.85. Latency 최적화(NVIDIA):--speculative-algorithm EAGLE3 --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --speculative-draft-model-path ${EAGLE3_MODEL_PATH} --disable-shared-experts-fusion --max-running-requests 64 --mem-fraction-static 0.85 --kv-cache-dtype fp8_e4m3 --context-length 32768.
3.2 Configuration Tips
스페큘레이티브 디코딩 (NVIDIA GPU):
- 지연 시간에 민감한 시나리오에서 스페큘레이티브 디코딩 사용:
--speculative-algorithm EAGLE3: 스페큘레이티브 디코딩 알고리즘--speculative-num-steps 3: 스페큘레이티브 검증 라운드 수--speculative-eagle-topk 1: 드래프트 토큰 top-k 샘플링--speculative-num-draft-tokens 4: 스텝당 드래프트 토큰 수--speculative-draft-model-path: 드래프트 모델 가중치 경로. 로컬 폴더이거나yuhuili/EAGLE3-LLaMA3.1-Instruct-8B같은 Hugging Face repo ID.
AMD GPU 배포:
- 하드웨어 인지 TP: MI355X(256GB 메모리)는 MI300X/MI325X(192GB)보다 낮은 TP 값을 지원해요
- 검증된 TP 구성:
- MI300X/MI325X: 405B BF16 (TP=8), 405B FP8 (TP=4), 70B/8B (TP=1)
- MI355X: 405B BF16 (TP=4), 405B FP8 (TP=2), 70B/8B (TP=1)
- FP8 모델 변형:
- 405B: Meta 공식
meta-llama/Llama-3.1-405B-Instruct-FP8사용 - 70B/8B: AMD 최적화
amd/Llama-3.1-{size}-Instruct-FP8-KV사용
- 405B: Meta 공식
- Tool Calling: Instruct 모델에서
--tool-call-parser llama3으로 활성화
Xeon CPU 배포:
- SGLang CPU 서버 문서의 서빙 엔진 시작 섹션 Notes 부분을 참조해 TP(tensor parallel)와 NUMA 바인딩 설정을 잘 이해하세요.
4. Model Invocation
4.1 Basic Usage
SGLang은 OpenAI 호환 엔드포인트를 노출해요. 먼저 서버를 시작하세요:
sglang serve \
--model-path Meta-Llama/Llama-3.1-405B-Instruct \
--tp 8
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="EMPTY",
)
resp = client.chat.completions.create(
model="Meta-Llama/Llama-3.1-405B-Instruct",
messages=[
{"role": "system", "content": "You are a helpful coding assistant."},
{"role": "user", "content": "Write a Python function that retries a request with exponential backoff."},
],
temperature=0.2,
max_tokens=512,
)
print(resp.choices[0].message.content)
출력 예시:
**Exponential Backoff Retry Function in Python**
=====================================================
Below is a Python function that uses the `requests` library to retry a request with exponential backoff.
```python
import requests
import time
import random
def exponential_backoff_retry(url, method, retries=3, backoff_factor=1, max_delay=60):
"""
Retry a request with exponential backoff.
Args:
url (str): The URL to make the request to.
method (str): The HTTP method to use (e.g. 'GET', 'POST', etc.).
retries (int): The number of retries to attempt. Defaults to 3.
backoff_factor (int): The factor to multiply the delay by for each retry. Defaults to 1.
max_delay (int): The maximum delay to wait between retries in seconds. Defaults to 60.
Returns:
The response object from the successful request.
"""
delay = 1
for attempt in range(retries + 1):
try:
response = requests.request(method, url)
response.raise_for_status() # Raise an exception for HTTP errors
return response
except requests.RequestException as e:
if attempt < retries:
# Calculate the delay for this retry
delay = min(delay * backoff_factor, max_delay)
# Add a random jitter to the delay to prevent thundering herd problem
delay += random.uniform(0, delay * 0.1)
# Wait for the calculated delay before retrying
time.sleep(delay)
else:
# If all retries have failed, raise the exception
raise e
...
### 4.2 Advanced Usage
#### 4.2.1 Tool Calling
Llama3는 tool calling 능력을 지원해요. 먼저 tool call parser를 켠 채 서버를 시작하세요:
```shell
sglang serve \
--model-path Meta-Llama/Llama-3.1-405B-Instruct \
--tool-call-parser llama3 \
--tp 8
Python 예시
from openai import OpenAI
client = OpenAI(api_key="None", base_url=f"http://0.0.0.0:8000/v1")
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the weather in a given location",
"parameters": {
"type": "object",
"properties": {
"city": {
"type": "string",
"description": "The city to find the weather for, e.g. 'San Francisco'",
},
"unit": {
"type": "string",
"description": "The unit to fetch the temperature in",
"enum": ["celsius", "fahrenheit"],
},
},
"required": ["city", "unit"],
},
},
}
]
response = client.chat.completions.create(
model="meta-llama/Llama-3.1-405B-Instruct",
messages=[
{
"role": "user",
"content": "What's the weather like in Boston today?",
}
],
temperature=0.7,
stream=True,
tools=tools,
)
arguments = []
tool_calls_accumulator = {}
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
if hasattr(delta, 'tool_calls') and delta.tool_calls:
for tool_call in delta.tool_calls:
index = tool_call.index
if index not in tool_calls_accumulator:
tool_calls_accumulator[index] = {
'name': None,
'arguments': ''
}
if tool_call.function:
if tool_call.function.name:
tool_calls_accumulator[index]['name'] = tool_call.function.name
if tool_call.function.arguments:
tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
# Print content
if delta.content:
print(delta.content, end="", flush=True)
# Print accumulated tool calls
for index, tool_call in sorted(tool_calls_accumulator.items()):
print(f"🔧 Tool Call: {tool_call['name']}")
print(f" Arguments: {tool_call['arguments']}")
print()
참조: SGLang Tool Parser Documentation
출력 예시
🔧 Tool Call: get_weather
Arguments: {"city": "Boston", "unit": "fahrenheit"}
Tool Call 결과 처리 tool call을 얻은 후 함수를 실행할 수 있어요:
def get_weather(location, unit="celsius"):
# Your actual weather API call here
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
# Send tool result back to the model
messages = [
{"role": "user", "content": "What's the weather like in Boston today?"},
{
"role": "assistant",
"content": None,
"tool_calls": [{
"id": "call_123",
"type": "function",
"function": {
"name": "get_weather",
"arguments": '{"location": "Boston", "unit": "fahrenheit"}'
}
}]
},
{
"role": "tool",
"tool_call_id": "call_123",
"content": get_weather("Boston", "fahrenheit")
}
]
final_response = client.chat.completions.create(
model="Meta-Llama/Llama-3.1-405B-Instruct",
messages=messages,
temperature=0.7
)
print(final_response.choices[0].message.content)
# Output: "The current weather in Boston is **22°C** and **sunny**. A perfect day to spend outside"
5. Benchmark
5.1 Speed Benchmark
테스트 환경:
- 하드웨어: NVIDIA A100 GPU (8x)
- 모델: Meta-Llama/Llama-3.1-70B
- 텐서 병렬화: 8
- sglang 버전: 0.5.6
SGLang 내장 벤치마킹 도구로 ShareGPT_Vicuna_unfiltered 데이터셋의 성능 평가를 수행해요. 이 데이터셋은 실제 대화 데이터를 담고 있어 실제 사용 시나리오의 성능을 더 잘 반영해요.
5.1.1 Standard Scenario Benchmark
- 모델 배포 명령:
sglang serve \
--model-path Meta-Llama/Llama-3.1-70B \
--tp 8
5.1.1.1 Low Concurrency
- 벤치마크 명령:
sglang serve \
--backend sglang \
--model Meta-Llama/Llama-3.1-70B \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1
- 테스트 결과:
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 79.81
Total input tokens: 6101
Total input text tokens: 6101
Total input vision tokens: 0
Total generated tokens: 4220
Total generated tokens (retokenized): 4208
Request throughput (req/s): 0.13
Input token throughput (tok/s): 76.44
Output token throughput (tok/s): 52.88
Peak output token throughput (tok/s): 54.00
Peak concurrent requests: 2
Total token throughput (tok/s): 129.32
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 7977.81
Median E2E Latency (ms): 6373.48
---------------Time to First Token----------------
Mean TTFT (ms): 131.61
Median TTFT (ms): 131.77
P99 TTFT (ms): 163.88
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 18.63
Median TPOT (ms): 18.63
P99 TPOT (ms): 18.65
---------------Inter-Token Latency----------------
Mean ITL (ms): 18.64
Median ITL (ms): 18.64
P95 ITL (ms): 18.69
P99 ITL (ms): 18.74
Max ITL (ms): 21.95
==================================================
5.1.1.2 Medium Concurrency
sglang serve \
--backend sglang \
--model-path Meta-Llama/Llama-3.1-70B \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 80 \
--max-concurrency 16
- 테스트 결과:
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 79.47
Total input tokens: 39668
Total input text tokens: 39668
Total input vision tokens: 0
Total generated tokens: 40805
Total generated tokens (retokenized): 38450
Request throughput (req/s): 1.01
Input token throughput (tok/s): 499.17
Output token throughput (tok/s): 513.48
Peak output token throughput (tok/s): 674.00
Peak concurrent requests: 20
Total token throughput (tok/s): 1012.65
Concurrency: 13.47
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 13376.67
Median E2E Latency (ms): 14130.48
---------------Time to First Token----------------
Mean TTFT (ms): 264.84
Median TTFT (ms): 147.02
P99 TTFT (ms): 791.93
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 26.09
Median TPOT (ms): 26.08
P99 TPOT (ms): 34.65
---------------Inter-Token Latency----------------
Mean ITL (ms): 25.76
Median ITL (ms): 23.95
P95 ITL (ms): 24.72
P99 ITL (ms): 98.32
Max ITL (ms): 478.92
==================================================
5.1.1.3 High Concurrency
sglang serve \
--backend sglang \
--model-path Meta-Llama/Llama-3.1-70B \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 500 \
--max-concurrency 100
- 테스트 결과:
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 500
Benchmark duration (s): 131.64
Total input tokens: 249831
Total input text tokens: 249831
Total input vision tokens: 0
Total generated tokens: 252662
Total generated tokens (retokenized): 243641
Request throughput (req/s): 3.80
Input token throughput (tok/s): 1897.87
Output token throughput (tok/s): 1919.38
Peak output token throughput (tok/s): 3100.00
Peak concurrent requests: 107
Total token throughput (tok/s): 3817.25
Concurrency: 89.70
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 23616.71
Median E2E Latency (ms): 22770.44
---------------Time to First Token----------------
Mean TTFT (ms): 245.98
Median TTFT (ms): 184.22
P99 TTFT (ms): 1251.67
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 47.19
Median TPOT (ms): 48.67
P99 TPOT (ms): 56.37
---------------Inter-Token Latency----------------
Mean ITL (ms): 46.34
Median ITL (ms): 33.46
P95 ITL (ms): 108.61
P99 ITL (ms): 166.11
Max ITL (ms): 1107.09
==================================================
5.1.2 Summarization Scenario Benchmark
5.1.2.1 Low Concurrency
sglang serve \
--backend sglang \
--model-path Meta-Llama/Llama-3.1-70B\
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 83.25
Total input tokens: 41941
Total input text tokens: 41941
Total input vision tokens: 0
Total generated tokens: 4220
Total generated tokens (retokenized): 4220
Request throughput (req/s): 0.12
Input token throughput (tok/s): 503.77
Output token throughput (tok/s): 50.69
Peak output token throughput (tok/s): 54.00
Peak concurrent requests: 2
Total token throughput (tok/s): 554.46
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 8322.45
Median E2E Latency (ms): 6873.36
---------------Time to First Token----------------
Mean TTFT (ms): 395.25
Median TTFT (ms): 318.02
P99 TTFT (ms): 850.80
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 18.80
Median TPOT (ms): 18.81
P99 TPOT (ms): 19.03
---------------Inter-Token Latency----------------
Mean ITL (ms): 18.83
Median ITL (ms): 18.81
P95 ITL (ms): 19.06
P99 ITL (ms): 19.08
Max ITL (ms): 23.08
==================================================
5.1.2.2 Medium Concurrency
sglang serve \
--backend sglang \
--model-path Meta-Llama/Llama-3.1-70B \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 80 \
--max-concurrency 16
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 107.12
Total input tokens: 300020
Total input text tokens: 300020
Total input vision tokens: 0
Total generated tokens: 41669
Total generated tokens (retokenized): 41603
Request throughput (req/s): 0.75
Input token throughput (tok/s): 2800.81
Output token throughput (tok/s): 389.00
Peak output token throughput (tok/s): 624.00
Peak concurrent requests: 19
Total token throughput (tok/s): 3189.81
Concurrency: 14.18
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 18988.30
Median E2E Latency (ms): 20290.66
---------------Time to First Token----------------
Mean TTFT (ms): 603.42
Median TTFT (ms): 531.82
P99 TTFT (ms): 2607.95
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 36.94
Median TPOT (ms): 36.73
P99 TPOT (ms): 79.19
---------------Inter-Token Latency----------------
Mean ITL (ms): 35.36
Median ITL (ms): 25.72
P95 ITL (ms): 27.07
P99 ITL (ms): 439.74
Max ITL (ms): 2529.51
==================================================
5.1.2.3 High Concurrency
sglang serve \
--backend sglang \
--model-path Meta-Llama/Llama-3.1-70B \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 320 \
--max-concurrency 64
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 64
Successful requests: 320
Benchmark duration (s): 215.66
Total input tokens: 1273893
Total input text tokens: 1273893
Total input vision tokens: 0
Total generated tokens: 170000
Total generated tokens (retokenized): 169035
Request throughput (req/s): 1.48
Input token throughput (tok/s): 5906.92
Output token throughput (tok/s): 788.27
Peak output token throughput (tok/s): 1920.00
Peak concurrent requests: 69
Total token throughput (tok/s): 6695.19
Concurrency: 60.01
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 40443.85
Median E2E Latency (ms): 39813.12
---------------Time to First Token----------------
Mean TTFT (ms): 633.32
Median TTFT (ms): 616.38
P99 TTFT (ms): 1912.97
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 74.95
Median TPOT (ms): 82.85
P99 TPOT (ms): 118.46
---------------Inter-Token Latency----------------
Mean ITL (ms): 75.08
Median ITL (ms): 34.12
P95 ITL (ms): 261.18
P99 ITL (ms): 828.12
Max ITL (ms): 1970.03
==================================================
5.2 Accuracy Benchmark
5.2.1 GSM8K Benchmark
- 벤치마크 명령:
python3 -m sglang.test.few_shot_gsm8k --num-questions 200
- 결과:
Accuracy: 0.830
Invalid: 0.000
Latency: 11.794 s
Output throughput: 1406.961 token/s