Hy3 preview

3.2 구성 팁

주요 파라미터:

하드웨어 요구사항: NVIDIA BF16 (Hy3-preview, ~552GB 가중치)

• H200 (141GB) / B200 (180GB): TP=8 (single-node에 BF16이 맞는 최소값).
• B300 (275GB) / GB300: TP=4.
• A100 / H100 (80GB): single-node 미지원 — 80GB급 GPU에서 BF16은 multi-node TP=16+ 필요.

Blackwell (B200 / B300 / GB300): Blackwell에서 자동 선택된 어텐션 백엔드가 HYV3를 잘못 라우팅할 수 있어요. Blackwell 하드웨어에서는 항상 --attention-backend trtllm_mha를 명시적으로 전달하세요 (위 구성 생성기가 이를 강제함).

Multi-Token Prediction (MTP): Hy3-preview 릴리스는 MTP draft 모듈을 번들해요. SGLang은 EAGLE 추측 디코딩 경로로 이를 실행하며, draft 모듈이 같은 --model-path에서 자동 로드돼요. 표준 MTP 플래그로 활성화하세요:

sglang serve \
  --model-path tencent/Hy3-preview \
  --tp 8 \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --reasoning-parser hunyuan \
  --tool-call-parser hunyuan \
  --trust-remote-code \
  --mem-fraction-static 0.85

위 생성기에서 "Speculative Decoding (MTP)" 옵션을 토글해 이 플래그들을 자동 추가하세요. 워크로드의 수락률(acceptance rate)에 따라 num-steps / num-draft-tokens을 튜닝하세요.

Xeon CPU 서비스 구성: SGLang CPU 서버 문서의 서빙 엔진 실행 절에 있는 Notes 부분을 참고해 인자(특히 TP(tensor parallel)와 NUMA 바인딩 설정)를 어떻게 구성하는지 이해하세요.

4. 모델 호출

4.1 기본 사용

기본 API 사용과 요청 예제는 다음을 참조하세요:

• SGLang Basic Usage Guide

배포 명령 (H200 × 8, BF16 기본):

sglang serve \
  --model-path tencent/Hy3-preview \
  --tp 8 \
  --reasoning-parser hunyuan \
  --tool-call-parser hunyuan \
  --trust-remote-code \
  --mem-fraction-static 0.9

배포 테스트:

시작 후 다음 명령으로 SGLang OpenAI 호환 API를 테스트할 수 있어요:

curl http://localhost:30000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "tencent/Hy3-preview",
        "messages": [
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": "Who won the world series in 2020?"}
        ]
    }'

간단한 완성 예제:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

response = client.chat.completions.create(
    model="tencent/Hy3-preview",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Who won the world series in 2020?"}
    ],
    max_tokens=1024
)

print("Reasoning:", response.choices[0].message.reasoning_content)
print("Content:  ", response.choices[0].message.content)

출력 예제:

Reasoning: None
Content:   The Los Angeles Dodgers won the 2020 World Series. They defeated the Tampa Bay Rays in six games (4-2). This was the Dodgers' first World Series championship since 1988. The series was notable for being played in a neutral-site bubble at Globe Life Field in Arlington, Texas, due to the COVID-19 pandemic.

reasoning_effort를 설정하지 않으면 서버는 기본적으로 instant 모드로 동작해요 (thinking 없음, reasoning_content=None). thinking을 선택하려면 요청에서 reasoning_effort="high" / "medium" / "low"를 전달하세요 — 아래 Hybrid Thinking 절 참고.

4.2 고급 사용

4.2.1 Reasoning Parser (Hybrid Thinking)

Hy3-preview는 하이브리드 thinking 모델이에요. OpenAI 표준 reasoning_effort로 thinking 예산을 제어하세요:

• high / medium / low — reasoning_content에 점점 더 많은 chain-of-thought 포함
• none — 생각을 완전히 건너뜀 (즉시 응답, content만)

배포 시 reasoning parser를 활성화해 thinking 섹션( thinking... response)이 reasoning_content로 분리되게 하세요:

sglang serve \
  --model-path tencent/Hy3-preview \
  --tp 8 \
  --reasoning-parser hunyuan \
  --trust-remote-code \
  --mem-fraction-static 0.9

Thinking Mode — High Effort:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="tencent/Hy3-preview",
    messages=[{"role": "user", "content": "Solve step by step: What is 15% of 240?"}],
    reasoning_effort="high",
    max_tokens=2048,
)

msg = response.choices[0].message
print("=============== Thinking =================")
print(msg.reasoning_content)
print("=============== Content =================")
print(msg.content)

출력 예제:

=============== Thinking =================
We need to solve: "What is 15% of 240?" Step by step. So we need to compute 15% of 240. The process: 15% means 15 per hundred, i.e., 15/100 = 0.15. Multiply 0.15 by 240. Or we can do: 10% of 240 = 24, 5% is half of 10% = 12, so sum = 36. Or do multiplication: 15/100 * 240 = (15*240)/100 = (3600)/100 = 36. So answer is 36.

We need to produce step-by-step explanation. The instruction: "Solve step by step: What is 15% of 240?" So we should provide a clear solution with steps. The final answer: 36. Also maybe include units? No units.

We'll output the solution in a clear manner.
=============== Content =================
To find 15% of 240, follow these steps:

1. **Understand that percent means "per hundred."**
   So, 15% = 15/100 or 0.15.

2. **Multiply the number (240) by the percentage in decimal form.**
   0.15 × 240.

   Alternatively, you can use fractions:
   (15/100) × 240.

3. **Perform the multiplication.**
   0.15 × 240 = 36.
   Or:
   (15 × 240) / 100 = 3600 / 100 = 36.

4. **Check using an alternative method:**
   - 10% of 240 = 24.
   - 5% of 240 = half of 10% = 12.
   - 15% = 10% + 5% = 24 + 12 = 36.

Thus, **15% of 240 is 36**.

Instant Mode — No Thinking:

response = client.chat.completions.create(
    model="tencent/Hy3-preview",
    messages=[{"role": "user", "content": "Give me a one-line summary of relativity."}],
    reasoning_effort="none",
    max_tokens=256,
)

print("Content:", response.choices[0].message.content)

출력 예제:

Content: Relativity is Einstein's theory that space, time, mass, and gravity are interconnected and relative, not fixed, fundamentally changing our understanding of the universe.

4.2.2 Tool Calling

Hy3-preview는 스트리밍 OpenAI 호환 툴 호출을 지원해요. 두 파서를 함께 활성화하세요 — reasoning parser가 툴 호출 파서 실행 전에 thinking 토큰을 제거해요:

sglang serve \
  --model-path tencent/Hy3-preview \
  --tp 8 \
  --reasoning-parser hunyuan \
  --tool-call-parser hunyuan \
  --trust-remote-code \
  --mem-fraction-static 0.9

비스트리밍 예제:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a city.",
            "parameters": {
                "type": "object",
                "properties": {
                    "city": {"type": "string"},
                    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
                },
                "required": ["city"],
            },
        },
    }
]

response = client.chat.completions.create(
    model="tencent/Hy3-preview",
    messages=[{"role": "user", "content": "What's the weather in Beijing? Use fahrenheit."}],
    tools=tools,
)

msg = response.choices[0].message
print("Reasoning:", msg.reasoning_content)
print("Content:  ", msg.content)
for tc in msg.tool_calls or []:
    print(f"Tool Call: {tc.function.name}")
    print(f"  Arguments: {tc.function.arguments}")

출력 예제:

Reasoning: None
Content:   I'll get the current weather for Beijing in Fahrenheit for you.
Tool Call: get_weather
  Arguments: {"city": "Beijing", "unit": "fahrenheit"}

스트리밍 예제 (증분 인자 델타):

Hy3-preview의 hunyuan 툴 호출 파서는 먼저 툴 이름을 방출한 다음 인자 JSON을 증분 조각으로 방출해요 — OpenAI 스트리밍 계약과 일치해요:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")

stream = client.chat.completions.create(
    model="tencent/Hy3-preview",
    messages=[{"role": "user", "content": "What's the weather in Beijing? Use fahrenheit."}],
    tools=tools,
    stream=True,
)

tool_buffer = {}
for chunk in stream:
    delta = chunk.choices[0].delta
    if delta.content:
        print(delta.content, end="", flush=True)
    for tc in delta.tool_calls or []:
        buf = tool_buffer.setdefault(tc.index, {"name": "", "args": ""})
        if tc.function and tc.function.name:
            buf["name"] += tc.function.name
        if tc.function and tc.function.arguments:
            buf["args"] += tc.function.arguments

for idx, buf in tool_buffer.items():
    print(f"\nTool[{idx}] {buf['name']}({buf['args']})")

출력 예제:

I'll check the current weather in Beijing for you using Fahrenheit.
Tool[0] get_weather({"city": "Beijing", "unit": "fahrenheit"})

5. 벤치마크

5.1 정확도 벤치마크

테스트 환경:

• 하드웨어: 8× NVIDIA H200 (141GB)
• Docker 이미지: lmsysorg/sglang:hy3-preview
• 모델: tencent/Hy3-preview (BF16)
• Tensor Parallelism: 8
• SGLang 버전: 최신 main

5.1.1 GSM8K

• 벤치마크 방법: 200개 질문에 대한 5-shot CoT, SGLang 네이티브 백엔드로 평가
• 벤치마크 명령:

python3 benchmark/gsm8k/bench_sglang.py --num-questions 200 --parallel 64

• 테스트 결과:

TODO — replace with real GSM8K accuracy after benchmark run on Hy3-preview (BF16).

5.1.2 MMLU

• 벤치마크 방법: 5-shot, 전체 57개 과목
• 벤치마크 명령:

python3 benchmark/mmlu/bench_sglang.py --nsub 60 --parallel 64

• 테스트 결과:

TODO — replace with real MMLU accuracy after benchmark run on Hy3-preview (BF16).

5.1.3 툴 호출 정확도 (MiniMax-Provider-Verifier)

• 벤치마크 도구: MiniMax-Provider-Verifier
• 메트릭: 함수 호출 스키마 유효성, 인자 매칭, 엔드투엔드 응답 정확도
• 테스트 결과:

TODO — replace with real tool-call accuracy after benchmark run on Hy3-preview (BF16).

5.2 속도 벤치마크

5.2.1 저동시성

• 벤치마크 명령:

python3 -m sglang.bench_serving \
  --backend sglang \
  --model tencent/Hy3-preview \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 10 \
  --max-concurrency 1

• 테스트 결과:

TODO — replace with real low-concurrency output on Hy3-preview (BF16).

5.2.2 고동시성

• 벤치마크 명령:

python3 -m sglang.bench_serving \
  --backend sglang \
  --model tencent/Hy3-preview \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 500 \
  --max-concurrency 100

• 테스트 결과:

TODO — replace with real high-concurrency output on Hy3-preview (BF16).

더 알아보기 (Learn more)

• tencent/Hy3-preview · tencent/Hy3-preview-Base
• SGLang 설치 가이드