NVIDIA Nemotron3-Ultra

3.2 Configuration Tips

• Attention backend:

  H100/H200: 기본적으로 flash attention 3 backend를 사용해요. B200/GB200/B300/GB300: --attention-backend trtllm_mha를 설정해 주세요. Blackwell에서는 flashinfer 기본값이 overlap scheduler를 깨뜨리므로 trtllm_mha가 필요해요.

• Mamba 스케줄러 전략:

  항상 --mamba-radix-cache-strategy extra_buffer로 시작해 주세요. 이 하이브리드 Transformer-Mamba 모델은 Mamba 상태를 올바르게 스케줄링하기 위해 extra_buffer 전략이 필요해요.

• Mamba backend:

  Mamba 레이어는 기본적으로 Triton SSM 커널을 사용해요. 더 나은 성능을 위해 --mamba-backend flashinfer를 설정해서 대신 FlashInfer Mamba 커널을 사용할 수 있어요.

• Mamba SSM 정밀도:

  SSM 상태 dtype은 기본적으로 모델 설정 값을 사용해요. --mamba-ssm-dtype float16을 설정하면 Mamba 상태를 FP16으로 저장해서, 정확도 손실이 크지 않으면서 mamba 캐시 메모리를 줄여 줘요.

• Mamba SSM 확률적 라운딩:

  Mamba 상태를 FP16으로 저장할 때 --enable-mamba-cache-stochastic-rounding을 추가하면 SSM 캐시 쓰기를 확률적으로 반올림해서 누적 편향을 줄여 줘요. 이 옵션은 --mamba-ssm-dtype float16과 CUDA가 필요하며, 기본 --mamba-backend triton에서는 추가로 SM100이 필요해요. --mamba-cache-philox-rounds로 Philox 라운드 수를 제어할 수 있어요(0은 backend 기본값 사용).

• TP 지원:

  TP 크기를 설정하려면 --tp <4|8|16>을 사용해요. 권장 조합:

  • BF16: H100/H200에서는 --tp 16, B200/B300에서는 --tp 8
  • NVFP4: B200/B300에서는 --tp 4 또는 --tp 8, GB200/GB300에서는 --tp 4

• H100에서 멀티노드 BF16:

  16×H100 BF16 설정은 두 노드에 걸쳐 있어요. 각 노드에서 --dist-init-addr <head-node-ip>:5000 --nnodes 2 --node-rank <0|1>을 사용하고 --tp 16을 유지해 주세요.

• DP attention:

  기본적으로 attention 레이어는 텐서 병렬(모든 TP rank에 걸쳐 샤드)로 실행돼요. DP attention을 활성화하면(위 토글, 또는 --dp <N> --enable-dp-attention) attention이 N개의 데이터 병렬 그룹으로 실행돼요: 각 DP rank가 자체 KV 캐시로 요청의 자체 슬라이스를 처리해요. --dp는 --tp를 나눌 수 있어야 해요.

• Expert parallel (EP):

  이 MoE는 ep_size == 1(off) 또는 ep_size == tp_size만 지원해요. 전문가 병렬 처리를 활성화하려면 --tp와 같은 값으로 --ep <tp>를 설정해 주세요.

• Multi-token prediction (MTP):

  낮은 지연 시간을 위해 MTP 추측 디코딩(위 토글)을 활성화해 주세요.

• FP8 KV cache:

  fp8 kv cache를 활성화하려면 --kv-cache-dtype fp8_e4m3를 설정해 주세요. 이 옵션은 NVFP4 체크포인트에서 기본적으로 활성화돼요.

• Reasoning parser:

  --reasoning-parser nemotron_3을 설정하면 구조화된 추론 트레이스(응답의 reasoning_content 필드)를 활성화해요.

• Tool calling:

  --tool-call-parser qwen3_coder를 설정하면 도구 호출 지원을 활성화해요.

4. Model Invocation

python3 -m sglang.launch_server \
  --model-path nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 \
  --trust-remote-code \
  --tp 8 \
  --mamba-radix-cache-strategy extra_buffer \
  --attention-backend trtllm_mha \
  --tool-call-parser qwen3_coder \
  --reasoning-parser nemotron_3

4.1 Basic Usage (OpenAI-Compatible API)

SGLang은 OpenAI 호환 엔드포인트를 제공해요. OpenAI Python 클라이언트 사용 예시:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY",
)

resp = client.chat.completions.create(
    model="nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16",
    messages=[
        {"role": "system", "content": "You are a helpful AI assistant."},
        {"role": "user", "content": "Give me 3 bullet points about SGLang."},
    ],
    temperature=0.6,
    max_tokens=1024,
)
print("Reasoning:", resp.choices[0].message.reasoning_content, "\nContent:", resp.choices[0].message.content)
print("\n")

출력:

Reasoning: The user wants 3 bullet points about SGLang. Let me recall what I know about SGLang — it's a high-performance serving framework for large language models with a focus on structured generation and efficient KV cache reuse...(more tokens)

Content: - **Radix Attention** — SGLang reuses KV cache across requests sharing a common prefix, dramatically reducing memory and compute for multi-turn agent loops and few-shot workloads.
- **OpenAI-compatible API and structured generation** — Drop-in replacement for the OpenAI client, with first-class support for constrained decoding (JSON schema, regex) and OpenAI-style tool calling.
- **High-throughput serving on NVIDIA GPUs** — Continuous batching, chunked prefill, FP8/NVFP4 quantization, and optimized CUDA kernels deliver state-of-the-art throughput across H100, H200, B200, and GB200.

스트리밍 채팅 완성:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY",
)

stream = client.chat.completions.create(
    model="nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16",
    messages=[
        {"role": "system", "content": "You are a helpful AI assistant."},
        {"role": "user", "content": "What are the first 5 prime numbers?"}
    ],
    temperature=0.7,
    max_tokens=1024,
    stream=True,
)
for chunk in stream:
    delta = chunk.choices[0].delta
    if delta and delta.content:
        print(delta.content, end="", flush=True)

출력:

The first 5 prime numbers are:
**2, 3, 5, 7, 11**.

### Explanation:
- A **prime number** is a natural number greater than 1 whose only positive divisors are 1 and itself.
- **2** is the smallest prime and the only even prime.
- **3, 5, 7, 11** are each divisible only by 1 and themselves.
- **1** is not prime by definition (it has only one positive divisor).
- **4, 6, 8, 9, 10** are composite.

4.2 Reasoning

이 모델은 Reasoning ON(기본값)과 OFF 두 가지 모드를 지원해요. 아래 예시처럼 enable_thinking을 False로 설정해서 전환할 수 있어요.

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY",
)

# Reasoning on (default)
print("Reasoning on")
resp = client.chat.completions.create(
    model="nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Plan a 3-step approach to debug a flaky integration test. Keep the thinking process short."}
    ],
    temperature=1,
    max_tokens=1024,
)
print(f"Reasoning: \n{resp.choices[0].message.reasoning_content[:200]}... \nContent: \n{resp.choices[0].message.content[:200]}...")
print("\n")
# Reasoning off
print("Reasoning off")
resp = client.chat.completions.create(
    model="nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Give me 3 facts about SGLang."}
    ],
    temperature=0,
    max_tokens=256,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}}
)
print(f"Content: \n{resp.choices[0].message.content[:200]}...")

출력:

Reasoning on
Reasoning:
The user wants a short reasoning chain plus a 3-step debug plan for a flaky integration test. I'll think briefly about common causes (timing/race, shared state, external service variance) and pick a t...
Content:
1. **Reproduce deterministically** — run the test in a loop (e.g. 50–100x) with logging at the suspected race points to confirm the failure rate and surface ordering.
2. **Isolate state** — re-run with...

Reasoning off
Content:
Here are 3 facts about SGLang:

1. **High-performance LLM serving system** developed at UC Berkeley with contributions from a broad open-source community, focused on throughput and latency at scale.
...

4.3 Tool Calling

OpenAI Tools 스키마를 사용해 함수를 호출하고 반환된 tool_calls를 확인해요. 서버는 --tool-call-parser qwen3_coder로 시작되어야 해요.

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY",
)

# Tool calling via OpenAI tools schema
TOOLS = [
    {
        "type": "function",
        "function": {
            "name": "search_codebase",
            "description": "Search the project codebase for a symbol or pattern.",
            "parameters": {
                "type": "object",
                "properties": {
                    "query": {
                        "type": "string",
                        "description": "The symbol, function name, or regex to search for"
                    },
                    "path": {
                        "type": "string",
                        "description": "Optional sub-path to restrict the search to"
                    }
                },
                "required": ["query"]
            }
        }
    }
]

completion = client.chat.completions.create(
    model="nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16",
    messages=[
        {"role": "system", "content": "You are a coding agent. Use tools to inspect the repo before answering."},
        {"role": "user", "content": "Where is the `RadixCache` class defined?"}
    ],
    tools=TOOLS,
    temperature=0.6,
    top_p=0.95,
    max_tokens=512,
    stream=False
)

print(completion.choices[0].message.reasoning_content)
print(completion.choices[0].message.tool_calls)

출력:

The user is asking where the RadixCache class is defined. I should search the codebase for the symbol "RadixCache" to find the file and line. I'll call search_codebase with that query.

[ChatCompletionMessageFunctionToolCall(id='call_8a7f2c4e1b9d4a3e8c2f1d6b', function=Function(arguments='{"query": "class RadixCache"}', name='search_codebase'), type='function', index=0)]

4.4 Controlling Reasoning Budget

reasoning_budget 파라미터는 모델의 추론 트레이스 길이를 제한해요. 추론 출력이 지정된 토큰 예산에 도달하면, 모델은 다음 newline 문자에서 추론을 자연스럽게 종료하려고 시도해요.

예산 임계값에 도달한 후 500 토큰 이내에 newline을 만나지 못하면, 추론 트레이스는 reasoning_budget + 500 토큰에서 강제로 종료돼요.

from typing import Any, Dict, List
import openai
from transformers import AutoTokenizer

class ThinkingBudgetClient:
    def __init__(self, base_url: str, api_key: str, tokenizer_name_or_path: str):
        self.base_url = base_url
        self.api_key = api_key
        self.tokenizer = AutoTokenizer.from_pretrained(tokenizer_name_or_path)
        self.client = openai.OpenAI(base_url=self.base_url, api_key=self.api_key)

    def chat_completion(
        self,
        model: str,
        messages: List[Dict[str, Any]],
        reasoning_budget: int = 512,
        max_tokens: int = 1024,
        **kwargs,
    ) -> Dict[str, Any]:
        assert (
            max_tokens > reasoning_budget
        ), f"reasoning_budget must be smaller than max_tokens. Given {max_tokens=} and {reasoning_budget=}"

        # 1. first call chat completion to get reasoning content
        response = self.client.chat.completions.create(
            model=model,
            messages=messages,
            max_tokens=reasoning_budget,
            **kwargs
        )

        reasoning_content = response.choices[0].message.reasoning_content or ""

        if " response" not in reasoning_content:
            # reasoning content is too long, closed with a period (.)
            reasoning_content = f"{reasoning_content}.\n response\n\n"

        reasoning_tokens_used = len(
            self.tokenizer.encode(reasoning_content, add_special_tokens=False)
        )
        remaining_tokens = max_tokens - reasoning_tokens_used

        assert (
            remaining_tokens > 0
        ), f"remaining tokens must be positive. Given {remaining_tokens=}. Increase max_tokens or lower reasoning_budget."

        # 2. append reasoning content to messages and call completion
        messages.append({"role": "assistant", "content": reasoning_content})
        prompt = self.tokenizer.apply_chat_template(
            messages,
            tokenize=False,
            continue_final_message=True,
        )

        response = self.client.completions.create(
            model=model,
            prompt=prompt,
            max_tokens=remaining_tokens,
            **kwargs
        )

        response_data = {
            "reasoning_content": reasoning_content.strip().strip(" response").strip(),
            "content": response.choices[0].text,
            "finish_reason": response.choices[0].finish_reason,
        }
        return response_data

reasoning_budget=256을 사용한 예시:

SERVED_MODEL_NAME = "nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16"

# Client
client = ThinkingBudgetClient(
    base_url="http://127.0.0.1:30000/v1",
    api_key="null",
    tokenizer_name_or_path=SERVED_MODEL_NAME
)

resp = client.chat_completion(
    model=SERVED_MODEL_NAME,
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Outline a research plan to evaluate the throughput of two MoE serving strategies."}
    ],
    temperature=1,
    max_tokens=1024,
    reasoning_budget=256
)
print("Reasoning:", resp["reasoning_content"], "\nContent:", resp["content"])

출력:

Reasoning: The user wants a research plan to compare throughput of two MoE serving strategies. I should outline goals, baselines, datasets, metrics (tokens/s, TTFT, ITL, MFU), variables to sweep (TP, batch size, sequence length, concurrency), and statistical handling. Keep it concise since reasoning_budget is 256...
Content:

**Research plan**

1. **Define goal & metrics** — peak token throughput (input+output), TTFT, P99 ITL, MFU; measured at fixed accuracy.
2. **Choose baselines** — Strategy A (TP-only) vs Strategy B (TP + expert-parallel). Hold model checkpoint, precision, and KV-cache dtype constant.
3. **Sweep** — `{batch ∈ 1,4,16,64, concurrency ∈ 16,64,256, seq_len ∈ 1k,8k,32k}` per strategy.
4. **Workload** — `sglang.bench_serving --dataset-name random` with matched input/output budgets.
5. **Analysis** — per-config throughput table + roofline overlay; bootstrap CIs over 3 reruns to bound noise.

5. Benchmark

5.1 Speed Benchmark

테스트 환경:

• 하드웨어: GB200 (4x)

• 모델: nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4

• 텐서 병렬 처리(Tensor Parallelism): 4

• SGLang 버전: main branch

• 모델 배포 명령:

python3 -m sglang.launch_server \
  --model-path nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4 \
  --trust-remote-code \
  --tp 4 \
  --mamba-radix-cache-strategy extra_buffer \
  --attention-backend trtllm_mha \
  --max-running-requests 1024

• 벤치마크 명령:

python3 -m sglang.bench_serving \
  --backend sglang \
  --model nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4 \
  --dataset-name random \
  --random-input-len 1024 \
  --random-output-len 1024 \
  --num-prompts 4096 \
  --max-concurrency 256

• 테스트 결과:

============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 256
Successful requests:                     4096
Benchmark duration (s):                  1184.58
Total input tokens:                      2081726
Total input text tokens:                 2081726
Total generated tokens:                  2087288
Total generated tokens (retokenized):    1990224
Request throughput (req/s):              3.46
Input token throughput (tok/s):          1757.35
Output token throughput (tok/s):         1762.05
Peak output token throughput (tok/s):    3150.00
Peak concurrent requests:                266
Total token throughput (tok/s):          3519.40
Concurrency:                             249.55
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   72169.95
Median E2E Latency (ms):                 71994.47
P90 E2E Latency (ms):                    99898.56
P99 E2E Latency (ms):                    107119.61
---------------Time to First Token----------------
Mean TTFT (ms):                          40057.33
Median TTFT (ms):                        41375.93
P99 TTFT (ms):                           46377.89
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          63.15
Median TPOT (ms):                        63.65
P99 TPOT (ms):                           78.16
---------------Inter-Token Latency----------------
Mean ITL (ms):                           63.14
Median ITL (ms):                         35.92
P95 ITL (ms):                            178.10
P99 ITL (ms):                            182.10
Max ITL (ms):                            2466.36
==================================================

5.2 Accuracy Benchmark

5.2.1 GSM8K Benchmark

환경

• 하드웨어: GB200 (4x)
• 모델: nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4
• 텐서 병렬 처리(Tensor Parallelism): 4
• SGLang 버전: main branch

모델 시작

python3 -m sglang.launch_server \
  --model-path nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4 \
  --trust-remote-code \
  --tp 4 \
  --mamba-radix-cache-strategy extra_buffer \
  --attention-backend trtllm_mha \
  --reasoning-parser nemotron_3

벤치마크 실행

python3 benchmark/gsm8k/bench_sglang.py

테스트 결과:

Accuracy: 0.970
Invalid: 0.000
Latency: 29.129 s
Output throughput: 745.333 token/s

5.2.2 MMLU Benchmark

벤치마크 실행

python3 benchmark/mmlu/bench_sglang.py

테스트 결과:

TBD

더 알아보기

• NVIDIA Nemotron3-Super
• NVIDIA Nemotron3-Nano