Gemma 4
Gemma 4
Gemma 4는 Google의 차세대 오픈 모델 제품군으로, Gemma 3 아키텍처를 기반으로 성능을 개선하고 MoE 변형과 텍스트·비전·오디오 멀티모달 지원을 추가했어요. 슬라이딩 윈도우·풀 어텐션을 결합한 하이브리드 어텐션, 전용 비전·오디오 인코더를 통한 멀티모달, MoE 변형, 레이어별 임베딩(PLE), 내장 thinking 모드와 함수 호출 기능을 갖추고 있어요.
gemma-4-E2B-it(~2B), gemma-4-E4B-it(~4B), gemma-4-12B-it(12B), gemma-4-31B-it(31B), gemma-4-26B-A4B-it(MoE, 26B total / 4B active) 등 다양한 크기와 구조를 제공하며 NVIDIA·AMD·Intel Arc Pro B-Series(GMG) GPU를 지원해요.
이 문서는 SGLang으로 Gemma 4를 배포하고, 추론·함수 호출·비전·오디오 입력 같은 고급 기능과 벤치마크 실행 방법을 설명해요.
원문 페이지에는 하드웨어 플랫폼과 모델 변형을 골라 배포 명령을 자동 생성하는 대화형 선택기가 포함되어 있어요. 위키에서는 렌더링되지 않으니 아래 지침을 직접 참고하면 돼요.
출처: 문서
본문
1. 모델 소개
Gemma 4는 Google의 차세대 오픈 모델 제품군으로, Gemma 3 아키텍처를 기반으로 성능을 개선하고 MoE 변형과 텍스트·비전·오디오 멀티모달 지원을 추가했어요.
주요 특징:
- 하이브리드 어텐션: 슬라이딩 윈도우와 풀 어텐션 레이어를 결합해 효율적인 장문 컨텍스트 처리
- 멀티모달: 전용 비전·오디오 인코더를 통해 텍스트, 이미지, 오디오 입력 지원
- MoE 변형: 26B-A4B 모델은 효율적인 추론을 위해 Mixture-of-Experts 아키텍처 사용
- 레이어별 임베딩 (PLE): 향상된 표현을 위한 레이어별 토큰 임베딩
- 추론:
gemma4추론 파서를 통한 내장 thinking 모드 - 함수 호출:
gemma4함수 호출 파서와 스트리밍을 통한 함수 호출 지원 - 융합 연산: Triton 최적화 RMSNorm + residual + scalar 커널
제공 모델:
| 모델 | 구조 | 파라미터 |
|---|---|---|
| google/gemma-4-E2B-it | Dense | ~2B |
| google/gemma-4-E4B-it | Dense | ~4B |
| google/gemma-4-12B-it | Dense | 12B |
| google/gemma-4-31B-it | Dense | 31B |
| google/gemma-4-26B-A4B-it | MoE | 26B total / 4B active |
2. SGLang 설치
Gemma 4(인코더 없는 통합 12B 포함, sgl-project/sglang#27167)는 SGLang main에서 지원돼요. 일치하는 transformers 커밋과 함께 설치하세요:
# Install SGLang from main
pip install 'git+https://github.com/sgl-project/sglang.git#subdirectory=python'
# Install transformers with Gemma 4 support (encoder-free unified family included)
pip install 'git+https://github.com/huggingface/transformers.git@1423d22f7a3b62e8c70ad67b58ec25cd9b675897'
Docker
lmsysorg/sglang:latest(CUDA 13.0, 멀티 아키텍처 amd64 + arm64)는 Hopper(H200)와 Blackwell(B200 / GB200 / GB300) 모두에서 실행돼요:
docker run --gpus all --ipc=host --shm-size 32g \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-p 30000:30000 \
lmsysorg/sglang:latest \
sglang serve --model-path google/gemma-4-12B-it \
--reasoning-parser gemma4 --tool-call-parser gemma4 \
--host 0.0.0.0 --port 30000
다른 설치 방법은 공식 SGLang 설치 가이드를 참고하세요.
3. 모델 배포
3.1 기본 구성
Gemma 4 시리즈는 NVIDIA GPU, AMD GPU, Intel Arc Pro B-Series GPU(코드명 BMG (Battlemage))를 포함한 다양한 하드웨어 플랫폼에 최적화된 여러 크기·구조의 모델을 제공해요. 권장 시작 구성은 하드웨어와 모델 크기에 따라 달라져요.
대화형 명령 생성기: 아래 구성 선택기를 사용하면 하드웨어 플랫폼과 모델 변형에 맞는 배포 명령을 자동으로 생성할 수 있어요.
3.2 구성 팁
- SGLang은 Gemma 4 모델에 대해 자동으로 Triton 어텐션 백엔드를 선택해요(prefill 중 양방향 이미지 토큰 어텐션에 필요).
- Blackwell(B200/sm100)의 어텐션 백엔드: SGLang은 sm100에서 기본적으로
trtllm_mha백엔드를 사용하는데, 텍스트에는 가장 빠르지만 이미지 토큰에 인과적 어텐션을 적용해요. B200에서 멀티모달(이미지) 워크로드를 실행한다면--attention-backend triton을 전달해 양방향 이미지 토큰 어텐션과 완전한 비전 품질을 복원하세요. 텍스트 전용과 오디오 워크로드는 기본값의 영향을 받지 않아요. - B200의 Gemma 4 26B-A4B: Triton MoE 경로를 위한 워크스페이스 여유를 남기려면
--mem-fraction-static 0.75를 사용하세요. - 26B-A4B MoE 모델은 고처리량 워크로드에
--tp 2를 고려하세요. - 추측 디코딩 (MTP): 각 Gemma 4 변형은 NEXTN 다중 토큰 예측을 가능하게 하는 짝지어진
*-assistantdraft 모델과 함께 제공돼요. 위 선택기로 활성화하거나--speculative-algorithm NEXTN --speculative-draft-model-path google/gemma-4-<variant>-it-assistant --speculative-num-steps 5 --speculative-num-draft-tokens 6 --speculative-eagle-topk 1을 전달하세요. MTP는 대화형 사용 사례의 지연 시간을 크게 줄일 수 있어요. 26B-A4B MoE 모델은 MTP 활성화 시--tp 2가 필요해요. - QAT 체크포인트: 선택기에서 Checkpoint → QAT로 전환해
qat-q4_0-unquantized릴리스를 대상으로 하세요. 이들은 bf16 가중치를 유지하므로 메모리·TP 요구사항이 표준 체크포인트와 동일하며, 각각 MTP용 짝지어진*-qat-q4_0-unquantized-assistantdraft 모델이 있어요. - 하드웨어 요구사항:
| 모델 | 하드웨어 | TP |
|---|---|---|
| gemma-4-E2B-it | 1x H200 / 1x B200 / 1x B300 / 1x MI300X / 1x MI325X / 1x MI355X | 1 |
| gemma-4-E4B-it | 1x H200 / 1x B200 / 1x B300 / 1x MI300X / 1x MI325X / 1x MI355X | 1 |
| gemma-4-12B-it | 1x H200 / 1x B200 / 1x B300 | 1 |
| gemma-4-31B-it | 2x H200 / 1x B200 / 1x B300 / 1x MI300X / 1x MI325X / 1x MI355X | 2 (H200) / 1 (B200/B300/AMD) |
| gemma-4-26B-A4B-it | 1x H200 / 1x B200 / 1x B300 / 1x MI300X / 1x MI325X / 1x MI355X | 1 |
3.3 AMD GPU 배포 (MI300X / MI325X / MI355X)
SGLang은 AMD GPU에서 올바른 어텐션 백엔드를 자동 선택해요. 작은 E-모델(gemma-4-E2B-it, gemma-4-E4B-it)은 AMD GPU에서 AITER를 비활성화하고 나머지는 같은 명령줄을 사용하세요:
SGLANG_USE_AITER=0 sglang serve --model-path google/gemma-4-E4B-it \
--reasoning-parser gemma4 \
--tool-call-parser gemma4 \
--host 0.0.0.0 --port 30000
gemma-4-31B-it과 gemma-4-26B-A4B-it은 추가 명령줄 변경 없이 MI300X, MI325X, MI355X에서 위와 같은 명령으로 실행돼요.
상태: AMD 벤치마크는 섹션 5.1에 있어요.
4. 모델 호출
모든 기능을 활성화한 gemma-4-26B-A4B-it(MoE) 배포:
sglang serve --model-path google/gemma-4-26B-A4B-it \
--reasoning-parser gemma4 \
--tool-call-parser gemma4 \
--host 0.0.0.0 --port 30000
추측 디코딩 (MTP) 서버 명령
각 Gemma 4 변형은 NEXTN 다중 토큰 예측용 짝지어진 *-assistant draft 모델과 함께 제공돼요. 아래 명령으로 각 target 모델에 대해 MTP를 활성화하세요. 이것들은 대화형 선택기에서 Speculative Decoding (MTP) → Enabled를 켰을 때 생성되는 구성과 일치해요.
# Gemma 4 E2B + MTP
sglang serve \
--model-path google/gemma-4-E2B-it \
--speculative-algorithm NEXTN \
--speculative-draft-model-path google/gemma-4-E2B-it-assistant \
--speculative-num-steps 5 \
--speculative-num-draft-tokens 6 \
--speculative-eagle-topk 1 \
--mem-fraction-static 0.85
# Gemma 4 E4B + MTP
sglang serve \
--model-path google/gemma-4-E4B-it \
--speculative-algorithm NEXTN \
--speculative-draft-model-path google/gemma-4-E4B-it-assistant \
--speculative-num-steps 5 \
--speculative-num-draft-tokens 6 \
--speculative-eagle-topk 1 \
--mem-fraction-static 0.85
# Gemma 4 12B + MTP (~35% faster single-stream decode on H200)
sglang serve \
--model-path google/gemma-4-12B-it \
--speculative-algorithm NEXTN \
--speculative-draft-model-path google/gemma-4-12B-it-assistant \
--speculative-num-steps 5 \
--speculative-num-draft-tokens 6 \
--speculative-eagle-topk 1 \
--mem-fraction-static 0.85
# Gemma 4 31B + MTP
sglang serve \
--model-path google/gemma-4-31B-it \
--tp-size 2 \
--speculative-algorithm NEXTN \
--speculative-draft-model-path google/gemma-4-31B-it-assistant \
--speculative-num-steps 5 \
--speculative-num-draft-tokens 6 \
--speculative-eagle-topk 1 \
--mem-fraction-static 0.85
# Gemma 4 26B-A4B + MTP
sglang serve \
--model-path google/gemma-4-26B-A4B-it \
--tp-size 2 \
--speculative-algorithm NEXTN \
--speculative-draft-model-path google/gemma-4-26B-A4B-it-assistant \
--speculative-num-steps 5 \
--speculative-num-draft-tokens 6 \
--speculative-eagle-topk 1 \
--mem-fraction-static 0.85
4.1 기본 사용법
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
response = client.chat.completions.create(
model="google/gemma-4-26B-A4B-it",
messages=[
{"role": "user", "content": "What are the key differences between TCP and UDP?"}
],
max_tokens=1024
)
print(response.choices[0].message.content)
출력 예시:
The fundamental difference between **TCP (Transmission Control Protocol)** and **UDP (User Datagram
Protocol)** lies in how they prioritize data integrity versus speed.
### 1. Connection Type
* **TCP (Connection-Oriented):** Before any data is sent, TCP performs a "three-way handshake."
The sender and receiver exchange signals to establish a formal connection.
* **UDP (Connectionless):** UDP does not establish a connection. It simply starts blasting packets
to the destination IP address without checking if the receiver is ready.
### 2. Reliability and Error Checking
* **TCP (Reliable):** If a packet is lost or arrives corrupted, TCP detects the error and
retransmits the missing data.
* **UDP (Unreliable):** If a packet is lost or corrupted, it is simply discarded. There is no
mechanism to ask for a retransmission.
### 3. Ordering of Data
* **TCP (Ordered):** Segments are assigned sequence numbers and reassembled in the correct order.
* **UDP (Unordered):** Packets may arrive in a different order than sent.
### 4. Speed and Overhead
* **TCP (Slower):** Managing connections, tracking, and retransmissions adds significant overhead.
* **UDP (Faster):** No handshake, no tracking — extremely fast and ideal for real-time needs.
| Feature | TCP | UDP |
| :--- | :--- | :--- |
| **Connection** | Connection-oriented | Connectionless |
| **Reliability** | Guaranteed delivery | Best-effort |
| **Ordering** | Maintains strict order | No guaranteed order |
| **Speed** | Slower (High overhead) | Faster (Low overhead) |
4.2 비전 입력
Gemma 4 멀티모달 변형은 텍스트와 함께 이미지를 받아요:
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
response = client.chat.completions.create(
model="google/gemma-4-26B-A4B-it",
messages=[
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": "https://farm4.staticflickr.com/3175/2653711032_804ff86d81_z.jpg"
}
},
{
"type": "text",
"text": "Describe this image in detail."
}
]
}
],
max_tokens=1024
)
print(response.choices[0].message.content)
출력 예시:
A vertical, full shot shows a girl and a boy standing in front of a giant teddy bear. The boy, who
is on the left, is of South Asian descent, has short dark hair, and is smiling at the camera. He is
wearing a navy blue sweatshirt with a white collar, blue jeans, and white, black, and red sneakers.
The girl, on the right, is also of South Asian descent and has long, dark hair. She is smiling at
the camera and is wearing a pink t-shirt, a white long-sleeve shirt underneath, blue jeans, and pink
sneakers. The giant teddy bear is light brown and is standing behind the two children. The bear has
large, dark eyes and a black nose. In the background, on the left, there is a large wooden basket
filled with small teddy bears. To the left of the basket, an American flag is hanging on the wall.
On the right side of the image, there is a green leafy plant. The floor is a dark purple carpet. The
lighting is bright and even.
4.3 추론 (Thinking 모드)
Gemma 4는 하이브리드 추론을 지원해요. Thinking은 기본적으로 활성화되지 않아요 — extra_body를 통해 chat_template_kwargs: {"enable_thinking": true}를 전달해 활성화하세요. 추론 파서가 thinking과 content를 분리하고, thinking 과정은 스트리밍 응답의 reasoning_content로 반환돼요.
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
response = client.chat.completions.create(
model="google/gemma-4-26B-A4B-it",
messages=[
{"role": "user", "content": "Solve step by step: If a train travels at 60 km/h for 2.5 hours, how far does it go?"}
],
max_tokens=4096,
stream=True,
extra_body={"chat_template_kwargs": {"enable_thinking": True}}
)
thinking_started = False
has_thinking = False
has_answer = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Print answer content
if delta.content:
if has_thinking and not has_answer:
print("\n=============== Content =================", flush=True)
has_answer = True
print(delta.content, end="", flush=True)
print()
출력 예시:
=============== Thinking =================
* Input: Speed = 60 km/h, Time = 2.5 hours.
* Goal: Find the distance traveled.
* Distance = Speed × Time.
* Step 1: Identify given values. Speed = 60 km/h, Time = 2.5 hours
* Step 2: Formula. Distance = Speed × Time
* Step 3: Calculation. 60 × 2.5
Mental math: 60 × 2 = 120; 60 × 0.5 = 30; 120 + 30 = 150.
* Step 4: Final Result. 150 km.
=============== Content =================
To find the distance traveled, you can follow these steps:
### 1. Identify the given information:
* **Speed:** 60 km/h
* **Time:** 2.5 hours
### 2. Use the distance formula:
Distance = Speed × Time
### 3. Substitute the values:
Distance = 60 km/h × 2.5 hours
### 4. Perform the calculation:
* 60 × 2 = 120
* 60 × 0.5 = 30
* 120 + 30 = 150
**Final Answer: The train travels 150 km.**
4.4 함수 호출
Gemma 4는 gemma4 함수 호출 파서를 통한 함수 호출을 지원해요. 배포 시 --tool-call-parser gemma4로 활성화하세요.
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city name"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "Temperature unit"
}
},
"required": ["location"]
}
}
}
]
response = client.chat.completions.create(
model="google/gemma-4-26B-A4B-it",
messages=[
{"role": "user", "content": "What's the weather in Tokyo?"}
],
tools=tools,
stream=True
)
thinking_started = False
has_thinking = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
if hasattr(delta, 'tool_calls') and delta.tool_calls:
if has_thinking and thinking_started:
print("\n=============== Tool Calls ================", flush=True)
thinking_started = False
for tool_call in delta.tool_calls:
if tool_call.function:
print(f"Tool Call: {tool_call.function.name}")
print(f" Arguments: {tool_call.function.arguments}")
if delta.content:
print(delta.content, end="", flush=True)
print()
출력 예시:
=============== Tool Calls ================
Tool Call: get_weather
Arguments: {"location": "Tokyo"}
4.5 오디오 입력
오디오 지원 Gemma 4 변형(gemma-4-E2B-it, gemma-4-E4B-it, gemma-4-12B-it)은 텍스트와 함께 원시 오디오를 받아요. 파형을 base64 audio_url data URI(16 kHz 모노 WAV가 잘 동작)로 전달하거나, OpenAI의 input_audio 파트로 data에 base64 바이트, format는 wav로 전달하세요:
import base64
from openai import OpenAI
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
with open("sample.wav", "rb") as f:
audio_b64 = base64.b64encode(f.read()).decode()
response = client.chat.completions.create(
model="google/gemma-4-12B-it",
messages=[
{
"role": "user",
"content": [
{"type": "audio_url", "audio_url": {"url": f"data:audio/wav;base64,{audio_b64}"}},
{"type": "text", "text": "Transcribe the speech in this audio exactly."},
],
}
],
max_tokens=256,
temperature=0,
)
print(response.choices[0].message.content)
출력 예시:
Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.
최고의 ASR 품질을 위해 권장 전사 프롬프트 구조를 사용하세요:
Transcribe the following speech segment in {LANGUAGE} into {LANGUAGE} text.
Follow these specific instructions for formatting the answer:
* Only output the transcription, with no newlines.
* When transcribing numbers, write the digits, i.e. write 1.7 and not one point seven, and write 3 instead of three.
음성 번역(AST)의 경우 먼저 소스 언어 전사를 요청한 다음 번역을 요청하세요: "Transcribe the following speech segment in {SOURCE_LANGUAGE}, then translate it into {TARGET_LANGUAGE}. ..."
5. 벤치마크
5.1 속도 벤치마크
테스트 환경:
- 하드웨어: H200
- SGLang 버전: gemma4 branch
gemma-4-E2B-it (1x H200, TP=1)
서버 실행 명령:
sglang serve --model-path google/gemma-4-E2B-it
지연 시간 벤치마크 (텍스트)
python3 -m sglang.bench_serving --backend sglang \
--host 0.0.0.0 --port 30000 \
--dataset-name random --num-prompts 10 --max-concurrency 1
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 17.44
Total input tokens: 6101
Total generated tokens: 4220
Request throughput (req/s): 0.57
Output token throughput (tok/s): 242.03
Total token throughput (tok/s): 591.94
Mean TTFT (ms): 50.19
Median TTFT (ms): 54.22
Mean TPOT (ms): 3.99
Median ITL (ms): 4.05
==================================================
지연 시간 벤치마크 (이미지)
python3 -m sglang.bench_serving --backend sglang-oai-chat \
--host 0.0.0.0 --port 30000 \
--dataset-name image --image-count 2 --image-resolution 720p \
--random-input-len 128 --random-output-len 1024 \
--num-prompts 10 --max-concurrency 1
============ Serving Benchmark Result ============
Backend: sglang-oai-chat
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 18.05
Total input tokens: 6097
Total input vision tokens: 5340
Total generated tokens: 4220
Request throughput (req/s): 0.55
Output token throughput (tok/s): 233.84
Total token throughput (tok/s): 571.69
Mean TTFT (ms): 109.59
Median TTFT (ms): 112.62
Mean TPOT (ms): 4.01
Median ITL (ms): 4.04
==================================================
처리량 벤치마크 (텍스트)
python3 -m sglang.bench_serving --backend sglang \
--host 0.0.0.0 --port 30000 \
--dataset-name random --num-prompts 1000 --max-concurrency 100
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 1000
Benchmark duration (s): 51.73
Total input tokens: 512842
Total generated tokens: 510855
Request throughput (req/s): 19.33
Output token throughput (tok/s): 9876.36
Peak output token throughput (tok/s): 13863.00
Total token throughput (tok/s): 19791.14
Mean TTFT (ms): 86.57
Mean TPOT (ms): 9.56
Median ITL (ms): 5.99
==================================================
처리량 벤치마크 (이미지)
python3 -m sglang.bench_serving --backend sglang-oai-chat \
--host 0.0.0.0 --port 30000 \
--dataset-name image --image-count 2 --image-resolution 720p \
--random-input-len 128 --random-output-len 1024 \
--num-prompts 1000 --max-concurrency 100
============ Serving Benchmark Result ============
Backend: sglang-oai-chat
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 1000
Benchmark duration (s): 89.07
Total input tokens: 617799
Total input vision tokens: 534000
Total generated tokens: 510855
Request throughput (req/s): 11.23
Output token throughput (tok/s): 5735.75
Peak output token throughput (tok/s): 12823.00
Total token throughput (tok/s): 12672.23
Mean TTFT (ms): 636.46
Mean TPOT (ms): 16.34
Median ITL (ms): 5.68
==================================================
gemma-4-E4B-it (1x H200, TP=1)
서버 실행 명령:
sglang serve --model-path google/gemma-4-E4B-it
지연 시간 벤치마크 (텍스트)
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 24.49
Total input tokens: 6101
Total generated tokens: 4220
Request throughput (req/s): 0.41
Output token throughput (tok/s): 172.32
Total token throughput (tok/s): 421.45
Mean TTFT (ms): 52.76
Median TTFT (ms): 53.66
Mean TPOT (ms): 5.64
Median ITL (ms): 5.74
==================================================
지연 시간 벤치마크 (이미지)
============ Serving Benchmark Result ============
Backend: sglang-oai-chat
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 25.04
Total input tokens: 6124
Total input vision tokens: 5340
Total generated tokens: 4220
Request throughput (req/s): 0.40
Output token throughput (tok/s): 168.54
Total token throughput (tok/s): 413.13
Mean TTFT (ms): 110.15
Median TTFT (ms): 108.24
Mean TPOT (ms): 5.66
Median ITL (ms): 5.73
==================================================
처리량 벤치마크 (텍스트)
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 1000
Benchmark duration (s): 72.95
Total input tokens: 512842
Total generated tokens: 510855
Request throughput (req/s): 13.71
Output token throughput (tok/s): 7002.68
Peak output token throughput (tok/s): 9878.00
Total token throughput (tok/s): 14032.60
Mean TTFT (ms): 166.33
Mean TPOT (ms): 13.36
Median ITL (ms): 8.88
==================================================
처리량 벤치마크 (이미지)
============ Serving Benchmark Result ============
Backend: sglang-oai-chat
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 1000
Benchmark duration (s): 108.99
Total input tokens: 616952
Total input vision tokens: 534000
Total generated tokens: 510855
Request throughput (req/s): 9.18
Output token throughput (tok/s): 4687.38
Peak output token throughput (tok/s): 9277.00
Total token throughput (tok/s): 10348.25
Mean TTFT (ms): 626.17
Mean TPOT (ms): 20.00
Median ITL (ms): 8.64
==================================================
gemma-4-31B-it (2x H200, TP=2)
서버 실행 명령:
sglang serve --model-path google/gemma-4-31B-it --tp 2
지연 시간 벤치마크 (텍스트)
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 53.05
Total input tokens: 6101
Total generated tokens: 4220
Request throughput (req/s): 0.19
Output token throughput (tok/s): 79.55
Total token throughput (tok/s): 194.55
Mean TTFT (ms): 72.77
Median TTFT (ms): 75.05
Mean TPOT (ms): 12.32
Median ITL (ms): 12.53
==================================================
지연 시간 벤치마크 (이미지)
============ Serving Benchmark Result ============
Backend: sglang-oai-chat
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 53.78
Total input tokens: 6162
Total input vision tokens: 5340
Total generated tokens: 4220
Request throughput (req/s): 0.19
Output token throughput (tok/s): 78.46
Total token throughput (tok/s): 193.03
Mean TTFT (ms): 143.35
Median TTFT (ms): 146.85
Mean TPOT (ms): 12.37
Median ITL (ms): 12.48
==================================================
처리량 벤치마크 (텍스트)
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 1000
Benchmark duration (s): 182.00
Total input tokens: 512842
Total generated tokens: 510855
Request throughput (req/s): 5.49
Output token throughput (tok/s): 2806.82
Peak output token throughput (tok/s): 3798.00
Total token throughput (tok/s): 5624.56
Mean TTFT (ms): 324.67
Mean TPOT (ms): 33.95
Median ITL (ms): 25.44
==================================================
처리량 벤치마크 (이미지)
============ Serving Benchmark Result ============
Backend: sglang-oai-chat
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 1000
Benchmark duration (s): 236.46
Total input tokens: 621630
Total input vision tokens: 534000
Total generated tokens: 510855
Request throughput (req/s): 4.23
Output token throughput (tok/s): 2160.42
Peak output token throughput (tok/s): 3745.00
Total token throughput (tok/s): 4789.30
Mean TTFT (ms): 952.02
Mean TPOT (ms): 44.17
Median ITL (ms): 26.81
==================================================
gemma-4-26B-A4B-it (MoE, 1x H200, TP=1)
서버 실행 명령:
sglang serve --model-path google/gemma-4-26B-A4B-it
팁: 고처리량 워크로드에는
--tp 2를 고려하세요.
지연 시간 벤치마크 (텍스트)
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 25.00
Total input tokens: 6101
Total generated tokens: 4220
Request throughput (req/s): 0.40
Output token throughput (tok/s): 168.81
Total token throughput (tok/s): 412.85
Mean TTFT (ms): 103.74
Median TTFT (ms): 46.57
Mean TPOT (ms): 5.60
Median ITL (ms): 5.78
==================================================
지연 시간 벤치마크 (이미지)
============ Serving Benchmark Result ============
Backend: sglang-oai-chat
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 25.31
Total input tokens: 6164
Total input vision tokens: 5340
Total generated tokens: 4220
Request throughput (req/s): 0.40
Output token throughput (tok/s): 166.70
Total token throughput (tok/s): 410.20
Mean TTFT (ms): 129.22
Median TTFT (ms): 132.54
Mean TPOT (ms): 5.68
Median ITL (ms): 5.75
==================================================
처리량 벤치마크 (텍스트)
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 1000
Benchmark duration (s): 138.98
Total input tokens: 512842
Total generated tokens: 510855
Request throughput (req/s): 7.20
Output token throughput (tok/s): 3675.81
Peak output token throughput (tok/s): 4799.00
Total token throughput (tok/s): 7365.91
Mean TTFT (ms): 153.77
Mean TPOT (ms): 25.95
Median ITL (ms): 20.23
==================================================
처리량 벤치마크 (이미지)
============ Serving Benchmark Result ============
Backend: sglang-oai-chat
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 1000
Benchmark duration (s): 186.38
Total input tokens: 621146
Total input vision tokens: 534000
Total generated tokens: 510855
Request throughput (req/s): 5.37
Output token throughput (tok/s): 2740.86
Peak output token throughput (tok/s): 4962.00
Total token throughput (tok/s): 6073.47
Mean TTFT (ms): 854.71
Mean TPOT (ms): 34.64
Median ITL (ms): 19.08
==================================================
gemma-4-31B-it (1x MI300X, TP=1)
서버 실행 명령:
sglang serve --model-path google/gemma-4-31B-it
참고: 31B dense 모델은 TP=2가 필요한 H200(141 GB)과 달리 TP=1에서 단일 MI300X(192 GB VRAM)에 맞아요.
지연 시간 벤치마크 (텍스트)
python3 -m sglang.bench_serving --backend sglang \
--host 0.0.0.0 --port 30000 \
--dataset-name random --num-prompts 10 --max-concurrency 1
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 103.55
Total input tokens: 6101
Total generated tokens: 4220
Request throughput (req/s): 0.10
Output token throughput (tok/s): 40.75
Total token throughput (tok/s): 99.67
Mean TTFT (ms): 152.35
Median TTFT (ms): 169.66
Mean TPOT (ms): 24.13
Median ITL (ms): 24.23
==================================================
처리량 벤치마크 (텍스트)
python3 -m sglang.bench_serving --backend sglang \
--host 0.0.0.0 --port 30000 \
--dataset-name random --num-prompts 1000 --max-concurrency 100
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 1000
Benchmark duration (s): 441.59
Total input tokens: 512842
Total generated tokens: 510855
Request throughput (req/s): 2.26
Output token throughput (tok/s): 1156.85
Peak output token throughput (tok/s): 1759.00
Total token throughput (tok/s): 2318.19
Mean TTFT (ms): 819.22
Mean TPOT (ms): 82.51
Median ITL (ms): 63.45
==================================================
gemma-4-26B-A4B-it (MoE, 1x MI300X, TP=1)
서버 실행 명령:
sglang serve --model-path google/gemma-4-26B-A4B-it
지연 시간 벤치마크 (텍스트)
python3 -m sglang.bench_serving --backend sglang \
--host 0.0.0.0 --port 30000 \
--dataset-name random --num-prompts 10 --max-concurrency 1
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 43.73
Total input tokens: 6101
Total generated tokens: 4220
Request throughput (req/s): 0.23
Output token throughput (tok/s): 96.49
Total token throughput (tok/s): 236.00
Mean TTFT (ms): 185.58
Median TTFT (ms): 90.18
Mean TPOT (ms): 9.78
Median ITL (ms): 9.57
==================================================
처리량 벤치마크 (텍스트)
python3 -m sglang.bench_serving --backend sglang \
--host 0.0.0.0 --port 30000 \
--dataset-name random --num-prompts 1000 --max-concurrency 100
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 1000
Benchmark duration (s): 219.43
Total input tokens: 512842
Total generated tokens: 510855
Request throughput (req/s): 4.56
Output token throughput (tok/s): 2328.05
Peak output token throughput (tok/s): 3500.00
Total token throughput (tok/s): 4665.16
Mean TTFT (ms): 168.44
Mean TPOT (ms): 41.23
Median ITL (ms): 29.31
==================================================
gemma-4-12B-it (1x H200, TP=1)
서버 실행 명령:
sglang serve --model-path google/gemma-4-12B-it
지연 시간 벤치마크 (텍스트)
python3 -m sglang.bench_serving --backend sglang \
--host 0.0.0.0 --port 30000 \
--dataset-name random --num-prompts 10 --max-concurrency 1
============ Serving Benchmark Result ============
Backend: sglang
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 38.66
Total input tokens: 6101
Total generated tokens: 4220
Request throughput (req/s): 0.26
Output token throughput (tok/s): 109.15
Total token throughput (tok/s): 266.94
Mean TTFT (ms): 33.08
Median TTFT (ms): 33.71
Mean TPOT (ms): 9.02
Median ITL (ms): 9.19
==================================================
지연 시간 벤치마크 (이미지)
python3 -m sglang.bench_serving --backend sglang-oai-chat \
--host 0.0.0.0 --port 30000 \
--dataset-name image --image-count 2 --image-resolution 720p \
--random-input-len 128 --random-output-len 1024 \
--num-prompts 10 --max-concurrency 1
============ Serving Benchmark Result ============
Backend: sglang-oai-chat
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 39.36
Total input vision tokens: 5320
Total generated tokens: 4220
Request throughput (req/s): 0.25
Output token throughput (tok/s): 107.23
Total token throughput (tok/s): 263.62
Mean TTFT (ms): 94.98
Median TTFT (ms): 97.33
Mean TPOT (ms): 9.08
Median ITL (ms): 9.17
==================================================
처리량 벤치마크 (텍스트)
python3 -m sglang.bench_serving --backend sglang \
--host 0.0.0.0 --port 30000 \
--dataset-name random --num-prompts 1000 --max-concurrency 100
============ Serving Benchmark Result ============
Backend: sglang
Max request concurrency: 100
Successful requests: 1000
Benchmark duration (s): 130.44
Total input tokens: 512842
Total generated tokens: 510855
Request throughput (req/s): 7.67
Output token throughput (tok/s): 3916.46
Total token throughput (tok/s): 7848.15
Mean TTFT (ms): 207.49
Median TTFT (ms): 76.95
Mean TPOT (ms): 24.38
Median ITL (ms): 17.89
==================================================
처리량 벤치마크 (이미지)
============ Serving Benchmark Result ============
Backend: sglang-oai-chat
Max request concurrency: 100
Successful requests: 1000
Benchmark duration (s): 147.57
Total input tokens: 619609
Total input vision tokens: 532000
Total generated tokens: 510855
Request throughput (req/s): 6.78
Output token throughput (tok/s): 3461.79
Total token throughput (tok/s): 7660.54
Mean TTFT (ms): 438.40
Median TTFT (ms): 129.83
Mean TPOT (ms): 27.12
Median ITL (ms): 19.16
==================================================
gemma-4-12B-it (1x B200, TP=1)
서버 실행 명령:
# Text/audio: the sm100 default (trtllm_mha) is fastest.
# For image workloads add --attention-backend triton (bidirectional image attention).
sglang serve --model-path google/gemma-4-12B-it --attention-backend triton
지연 시간 벤치마크 (텍스트)
============ Serving Benchmark Result ============
Backend: sglang
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 30.46
Output token throughput (tok/s): 138.55
Total token throughput (tok/s): 338.85
Mean TTFT (ms): 28.14
Median TTFT (ms): 29.74
Mean TPOT (ms): 7.08
Median ITL (ms): 7.26
==================================================
지연 시간 벤치마크 (이미지)
============ Serving Benchmark Result ============
Backend: sglang-oai-chat
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 31.43
Total input vision tokens: 5320
Total generated tokens: 4220
Request throughput (req/s): 0.32
Output token throughput (tok/s): 134.26
Total token throughput (tok/s): 329.57
Mean TTFT (ms): 115.51
Median TTFT (ms): 74.27
Mean TPOT (ms): 7.14
Median ITL (ms): 7.24
==================================================
처리량 벤치마크 (텍스트)
============ Serving Benchmark Result ============
Backend: sglang
Max request concurrency: 100
Successful requests: 1000
Benchmark duration (s): 92.94
Request throughput (req/s): 10.76
Output token throughput (tok/s): 5496.55
Total token throughput (tok/s): 11014.49
Mean TTFT (ms): 120.89
Median TTFT (ms): 45.00
Mean TPOT (ms): 17.23
Median ITL (ms): 14.30
==================================================
처리량 벤치마크 (이미지)
============ Serving Benchmark Result ============
Backend: sglang-oai-chat
Max request concurrency: 100
Successful requests: 998
Benchmark duration (s): 107.82
Total input tokens: 617971
Total input vision tokens: 530936
Total generated tokens: 508951
Request throughput (req/s): 9.26
Output token throughput (tok/s): 4720.29
Total token throughput (tok/s): 10451.68
Mean TTFT (ms): 425.89
Median TTFT (ms): 109.57
Mean TPOT (ms): 19.45
Median ITL (ms): 15.11
==================================================
성능 튜닝: B200에서
--scheduler-recv-interval을 16으로 올리면 스케줄러의 단계별 Python 오버헤드를 줄여, 동시성 100에서 정확도 변화 없이 텍스트 처리량이 5497에서 5673 tok/s 출력(≈ +3%)으로 올랐어요. 고동시성 서빙에 안전하고 저위험한 노브예요.
5.2 정확도 벤치마크
테스트 환경:
- 하드웨어: H200
- SGLang 버전: gemma4 branch
MMLU
| 모델 | Humanities | Social Sciences | STEM | Other | Overall |
|---|---|---|---|---|---|
| gemma-4-E2B-it | 0.621 | 0.739 | 0.830 | 0.736 | 0.720 |
| gemma-4-E4B-it | 0.703 | 0.862 | 0.902 | 0.825 | 0.810 |
| gemma-4-12B-it | 0.784 | 0.888 | 0.946 | 0.861 | 0.859 |
| gemma-4-31B-it | 0.878 | 0.921 | 0.884 | 0.911 | 0.896 |
| gemma-4-26B-A4B-it | 0.853 | 0.906 | 0.938 | 0.886 | 0.891 |
GSM8K
| 모델 | Accuracy | Invalid | Latency (s) | Output Throughput (tok/s) |
|---|---|---|---|---|
| gemma-4-E2B-it | 0.170 | 0.000 | 3.990 | 8041.739 |
| gemma-4-E4B-it | 0.745 | 0.000 | 4.174 | 4672.030 |
| gemma-4-12B-it | 0.431 | 0.052 | 55.105 | 6580.229 |
| gemma-4-31B-it | 0.805 | 0.005 | 16.148 | 1559.914 |
| gemma-4-26B-A4B-it | 0.450 | 0.010 | 13.001 | 4089.457 |
참고: 이 GSM8K 수치는 원시 few-shot 완성 하네스(
sglang.test.few_shot_gsm8k)를 사용해요.gemma-4-12B-it은 추론 지향적이라 원시 few-shot 프롬프팅으로는 덜 채집되고, 채팅 템플릿과 함께 동일한 1319개 GSM8K 테스트 질문에서 0.950을 기록해요(sglang.test.run_eval --eval-name gsm8k).
gemma-4-12B-it with sgl-eval
gemma-4-12B-it은 추론 지향적이라 간결한 마지막 줄 대신 장황하게(단계별로) 답해요. 엄격한 마지막 줄 Answer: $LETTER 추출(sglang.test.run_eval처럼)은 올바른 답을 과소 계산해요. sgl-project의 평가 CLI인 sgl-eval은 견고한 답 추출을 사용해 서빙 모델에 대한 신뢰할 만한 점수를 제공해요:
| 벤치마크 | 예시 | 정확도 |
|---|---|---|
| MMLU | 2000 | 0.878 |
| GSM8K | 1319 | 0.960 |
실행 중인 서버에 대해 재현(--base-url은 엔드포인트를 가리킴):
pip install sgl-eval
# Sanity-check the endpoint
sgl-eval ping --base-url http://localhost:30000/v1
# Run the benchmarks (greedy, single-shot)
sgl-eval run gsm8k --base-url http://localhost:30000/v1
sgl-eval run mmlu --base-url http://localhost:30000/v1 --num-examples 2000
MMMU
| 모델 | Overall |
|---|---|
| gemma-4-E2B-it | 0.307 |
| gemma-4-E4B-it | 0.396 |
| gemma-4-12B-it | 0.683 |
| gemma-4-31B-it | 0.589 |
| gemma-4-26B-A4B-it | 0.549 |
MMMU 상세 점수 (도메인별):
gemma-4-E2B-it
{"Overall-Art and Design": {"num": 120, "acc": 0.45}, "Art": {"num": 30, "acc": 0.5}, "Art_Theory": {"num": 30, "acc": 0.467}, "Design": {"num": 30, "acc": 0.5}, "Music": {"num": 30, "acc": 0.333}, "Overall-Business": {"num": 150, "acc": 0.26}, "Accounting": {"num": 30, "acc": 0.367}, "Economics": {"num": 30, "acc": 0.233}, "Finance": {"num": 30, "acc": 0.2}, "Manage": {"num": 30, "acc": 0.233}, "Marketing": {"num": 30, "acc": 0.267}, "Overall-Science": {"num": 150, "acc": 0.273}, "Biology": {"num": 30, "acc": 0.233}, "Chemistry": {"num": 30, "acc": 0.267}, "Geography": {"num": 30, "acc": 0.367}, "Math": {"num": 30, "acc": 0.233}, "Physics": {"num": 30, "acc": 0.267}, "Overall-Health and Medicine": {"num": 150, "acc": 0.273}, "Basic_Medical_Science": {"num": 30, "acc": 0.5}, "Clinical_Medicine": {"num": 30, "acc": 0.233}, "Diagnostics_and_Laboratory_Medicine": {"num": 30, "acc": 0.233}, "Pharmacy": {"num": 30, "acc": 0.3}, "Public_Health": {"num": 30, "acc": 0.1}, "Overall-Humanities and Social Science": {"num": 120, "acc": 0.4}, "History": {"num": 30, "acc": 0.4}, "Literature": {"num": 30, "acc": 0.567}, "Sociology": {"num": 30, "acc": 0.333}, "Psychology": {"num": 30, "acc": 0.3}, "Overall-Tech and Engineering": {"num": 210, "acc": 0.252}, "Agriculture": {"num": 30, "acc": 0.333}, "Architecture_and_Engineering": {"num": 30, "acc": 0.267}, "Computer_Science": {"num": 30, "acc": 0.233}, "Electronics": {"num": 30, "acc": 0.1}, "Energy_and_Power": {"num": 30, "acc": 0.3}, "Materials": {"num": 30, "acc": 0.2}, "Mechanical_Engineering": {"num": 30, "acc": 0.333}, "Overall": {"num": 900, "acc": 0.307}}
gemma-4-E4B-it
{"Overall-Art and Design": {"num": 120, "acc": 0.458}, "Art": {"num": 30, "acc": 0.433}, "Art_Theory": {"num": 30, "acc": 0.567}, "Design": {"num": 30, "acc": 0.667}, "Music": {"num": 30, "acc": 0.167}, "Overall-Business": {"num": 150, "acc": 0.287}, "Accounting": {"num": 30, "acc": 0.233}, "Economics": {"num": 30, "acc": 0.467}, "Finance": {"num": 30, "acc": 0.133}, "Manage": {"num": 30, "acc": 0.3}, "Marketing": {"num": 30, "acc": 0.3}, "Overall-Science": {"num": 150, "acc": 0.28}, "Biology": {"num": 30, "acc": 0.333}, "Chemistry": {"num": 30, "acc": 0.133}, "Geography": {"num": 30, "acc": 0.4}, "Math": {"num": 30, "acc": 0.2}, "Physics": {"num": 30, "acc": 0.333}, "Overall-Health and Medicine": {"num": 150, "acc": 0.427}, "Basic_Medical_Science": {"num": 30, "acc": 0.4}, "Clinical_Medicine": {"num": 30, "acc": 0.533}, "Diagnostics_and_Laboratory_Medicine": {"num": 30, "acc": 0.4}, "Pharmacy": {"num": 30, "acc": 0.4}, "Public_Health": {"num": 30, "acc": 0.4}, "Overall-Humanities and Social Science": {"num": 120, "acc": 0.7}, "History": {"num": 30, "acc": 0.633}, "Literature": {"num": 30, "acc": 0.867}, "Sociology": {"num": 30, "acc": 0.733}, "Psychology": {"num": 30, "acc": 0.567}, "Overall-Tech and Engineering": {"num": 210, "acc": 0.324}, "Agriculture": {"num": 30, "acc": 0.533}, "Architecture_and_Engineering": {"num": 30, "acc": 0.3}, "Computer_Science": {"num": 30, "acc": 0.367}, "Electronics": {"num": 30, "acc": 0.133}, "Energy_and_Power": {"num": 30, "acc": 0.4}, "Materials": {"num": 30, "acc": 0.2}, "Mechanical_Engineering": {"num": 30, "acc": 0.333}, "Overall": {"num": 900, "acc": 0.396}}
gemma-4-12B-it
{"Overall-Art and Design": {"num": 120, "acc": 0.667}, "Art": {"num": 30, "acc": 0.7}, "Art_Theory": {"num": 30, "acc": 0.867}, "Design": {"num": 30, "acc": 0.767}, "Music": {"num": 30, "acc": 0.333}, "Overall-Business": {"num": 150, "acc": 0.747}, "Accounting": {"num": 30, "acc": 0.767}, "Economics": {"num": 30, "acc": 0.767}, "Finance": {"num": 30, "acc": 0.633}, "Manage": {"num": 30, "acc": 0.7}, "Marketing": {"num": 30, "acc": 0.867}, "Overall-Science": {"num": 150, "acc": 0.647}, "Biology": {"num": 30, "acc": 0.6}, "Chemistry": {"num": 30, "acc": 0.633}, "Geography": {"num": 30, "acc": 0.567}, "Math": {"num": 30, "acc": 0.6}, "Physics": {"num": 30, "acc": 0.833}, "Overall-Health and Medicine": {"num": 150, "acc": 0.68}, "Basic_Medical_Science": {"num": 30, "acc": 0.667}, "Clinical_Medicine": {"num": 30, "acc": 0.633}, "Diagnostics_and_Laboratory_Medicine": {"num": 30, "acc": 0.267}, "Pharmacy": {"num": 30, "acc": 0.833}, "Public_Health": {"num": 30, "acc": 1.0}, "Overall-Humanities and Social Science": {"num": 120, "acc": 0.817}, "History": {"num": 30, "acc": 0.8}, "Literature": {"num": 30, "acc": 0.9}, "Sociology": {"num": 30, "acc": 0.8}, "Psychology": {"num": 30, "acc": 0.767}, "Overall-Tech and Engineering": {"num": 210, "acc": 0.6}, "Agriculture": {"num": 30, "acc": 0.467}, "Architecture_and_Engineering": {"num": 30, "acc": 0.667}, "Computer_Science": {"num": 30, "acc": 0.733}, "Electronics": {"num": 30, "acc": 0.567}, "Energy_and_Power": {"num": 30, "acc": 0.667}, "Materials": {"num": 30, "acc": 0.567}, "Mechanical_Engineering": {"num": 30, "acc": 0.533}, "Overall": {"num": 900, "acc": 0.683}}
gemma-4-31B-it
{"Overall-Art and Design": {"num": 120, "acc": 0.667}, "Art": {"num": 30, "acc": 0.667}, "Art_Theory": {"num": 30, "acc": 0.867}, "Design": {"num": 30, "acc": 0.8}, "Music": {"num": 30, "acc": 0.333}, "Overall-Business": {"num": 150, "acc": 0.573}, "Accounting": {"num": 30, "acc": 0.633}, "Economics": {"num": 30, "acc": 0.733}, "Finance": {"num": 30, "acc": 0.433}, "Manage": {"num": 30, "acc": 0.533}, "Marketing": {"num": 30, "acc": 0.533}, "Overall-Science": {"num": 150, "acc": 0.527}, "Biology": {"num": 30, "acc": 0.667}, "Chemistry": {"num": 30, "acc": 0.567}, "Geography": {"num": 30, "acc": 0.5}, "Math": {"num": 30, "acc": 0.267}, "Physics": {"num": 30, "acc": 0.633}, "Overall-Health and Medicine": {"num": 150, "acc": 0.673}, "Basic_Medical_Science": {"num": 30, "acc": 0.733}, "Clinical_Medicine": {"num": 30, "acc": 0.533}, "Diagnostics_and_Laboratory_Medicine": {"num": 30, "acc": 0.467}, "Pharmacy": {"num": 30, "acc": 0.8}, "Public_Health": {"num": 30, "acc": 0.833}, "Overall-Humanities and Social Science": {"num": 120, "acc": 0.825}, "History": {"num": 30, "acc": 0.833}, "Literature": {"num": 30, "acc": 0.867}, "Sociology": {"num": 30, "acc": 0.767}, "Psychology": {"num": 30, "acc": 0.833}, "Overall-Tech and Engineering": {"num": 210, "acc": 0.405}, "Agriculture": {"num": 30, "acc": 0.667}, "Architecture_and_Engineering": {"num": 30, "acc": 0.2}, "Computer_Science": {"num": 30, "acc": 0.567}, "Electronics": {"num": 30, "acc": 0.333}, "Energy_and_Power": {"num": 30, "acc": 0.533}, "Materials": {"num": 30, "acc": 0.3}, "Mechanical_Engineering": {"num": 30, "acc": 0.233}, "Overall": {"num": 900, "acc": 0.589}}
gemma-4-26B-A4B-it
{"Overall-Art and Design": {"num": 120, "acc": 0.717}, "Art": {"num": 30, "acc": 0.733}, "Art_Theory": {"num": 30, "acc": 0.833}, "Design": {"num": 30, "acc": 0.867}, "Music": {"num": 30, "acc": 0.433}, "Overall-Business": {"num": 150, "acc": 0.493}, "Accounting": {"num": 30, "acc": 0.533}, "Economics": {"num": 30, "acc": 0.533}, "Finance": {"num": 30, "acc": 0.333}, "Manage": {"num": 30, "acc": 0.5}, "Marketing": {"num": 30, "acc": 0.567}, "Overall-Science": {"num": 150, "acc": 0.473}, "Biology": {"num": 30, "acc": 0.633}, "Chemistry": {"num": 30, "acc": 0.367}, "Geography": {"num": 30, "acc": 0.533}, "Math": {"num": 30, "acc": 0.267}, "Physics": {"num": 30, "acc": 0.567}, "Overall-Health and Medicine": {"num": 150, "acc": 0.62}, "Basic_Medical_Science": {"num": 30, "acc": 0.767}, "Clinical_Medicine": {"num": 30, "acc": 0.533}, "Diagnostics_and_Laboratory_Medicine": {"num": 30, "acc": 0.433}, "Pharmacy": {"num": 30, "acc": 0.7}, "Public_Health": {"num": 30, "acc": 0.667}, "Overall-Humanities and Social Science": {"num": 120, "acc": 0.758}, "History": {"num": 30, "acc": 0.8}, "Literature": {"num": 30, "acc": 0.833}, "Sociology": {"num": 30, "acc": 0.733}, "Psychology": {"num": 30, "acc": 0.667}, "Overall-Tech and Engineering": {"num": 210, "acc": 0.376}, "Agriculture": {"num": 30, "acc": 0.633}, "Architecture_and_Engineering": {"num": 30, "acc": 0.367}, "Computer_Science": {"num": 30, "acc": 0.533}, "Electronics": {"num": 30, "acc": 0.167}, "Energy_and_Power": {"num": 30, "acc": 0.367}, "Materials": {"num": 30, "acc": 0.367}, "Mechanical_Engineering": {"num": 30, "acc": 0.2}, "Overall": {"num": 900, "acc": 0.549}}
ASR
| 모델 | WER | Avg Latency (s) | Throughput (req/s) |
|---|---|---|---|
| gemma-4-E2B-it | 23.86% | 0.212 | 2.99 |
| gemma-4-E4B-it | 29.55% | 0.366 | 2.46 |
| gemma-4-12B-it | Supported (see §4.5) | — | — |
| gemma-4-31B-it | Not Supported | — | — |
| gemma-4-26B-A4B-it | Not Supported | — | — |
FLEUR (EN_US)
| 모델 | WER | Avg Latency (s) | Throughput (req/s) |
|---|---|---|---|
| gemma-4-E2B-it | 7.37% | 0.8963s | 16.25 |
| gemma-4-E4B-it | 6.08% | 0.8707s | 16.20 |
| gemma-4-12B-it | Supported (see §4.5) | — | — |
| gemma-4-31B-it | Not Supported | — | — |
| gemma-4-26B-A4B-it | Not Supported | — | — |
5.3 Logits 정확성 검증
gemma-4-E2B-it
$ python -m sglang.benchmark.one_batch --correct --model google/gemma-4-E2B-it ....
prefill logits (final): tensor([[-25.3063, -2.5718, -10.3674, ..., -25.3779, -25.5181, -25.2337]],
device='cuda:0')
....
$ python scripts/playground/reference_hf.py --model-path google/gemma-4-E2B-it
....
prefill logits (final) tensor([-25.3281, -2.1367, -10.2266, ..., -25.4375, -25.5000, -25.2500],
device='cuda:0', dtype=torch.float16)
....
gemma-4-E4B-it
$ python -m sglang.benchmark.one_batch --correct --model google/gemma-4-E4B-it ....
prefill logits (final): tensor([[-17.6478, 7.9901, -5.6505, ..., -17.5658, -17.6478, -17.7293]],
device='cuda:0')
....
$ python scripts/playground/reference_hf.py --model-path google/gemma-4-E4B-it
....
prefill logits (final) tensor([-17.5625, 8.0469, -5.5742, ..., -17.4688, -17.5625, -17.6719],
device='cuda:0', dtype=torch.float16)
....
gemma-4-31B-it
$ python -m sglang.benchmark.one_batch --correct --model google/gemma-4-31B-it ....
prefill logits (final): tensor([[-2.0748, 1.1245, -7.4356, ..., -2.1059, -2.1525, -2.2303]],
device='cuda:0')
....
$ python scripts/playground/reference_hf.py --model-path google/gemma-4-31B-it
....
prefill logits (final) tensor([-2.1133, 1.2656, -7.4766, ..., -2.1523, -2.2012, -2.2695],
device='cuda:0', dtype=torch.float16)
....