Mistral Small 4

3.2 Configuration Tips

• Tensor Parallelism: Mistral Small 4 FP8(약 119 GB)은 Hopper(H100/H200)에서 tp=2, Blackwell(B200/B300)에서 tp=1이 필요해요. NVFP4(약 60 GB, Blackwell 전용)는 tp=1로 실행돼요.
• 추론 강도: reasoning_effort("none", "high")로 요청마다 추론 깊이를 설정할 수 있어요. 재시작 없이 호출마다 전환 가능해요.
• 컨텍스트 길이와 메모리: 모델은 256K 컨텍스트 윈도우를 가져요. 메모리가 부족하면 --context-length(예: 32768)를 낮춰 시작하고 안정되면 늘려요.
• 툴 호출: --tool-call-parser mistral을 활성화해 네이티브 함수 호출을 지원해요.
• 추론 파서: --reasoning-parser mistral을 활성화하면 reasoning_content를 메인 응답 콘텐츠와 분리해요.
• 추측 디코딩 (EAGLE): EAGLE 가중치로 --speculative-algorithm EAGLE --speculative-draft-model-path mistralai/Mistral-Small-4-119B-2603-eagle을 활성화해 지연 시간을 낮춰요.

4. Model Invocation (모델 호출)

4.1 Thinking Mode

Mistral Small 4는 하이브리드 추론 모델이에요. 기본적으로 추론 응답을 생성하지 않아요. --reasoning_effort high로 추론을 켜요.

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY",
)

response = client.chat.completions.create(
    model="mistralai/Mistral-Small-4-119B-2603",
    messages=[
        {"role": "user", "content": "Solve step by step: what is 17 × 23 + 144 / 12?"},
    ],
    extra_body={"reasoning_effort": "high"},
)

print("Reasoning:", response.choices[0].message.reasoning_content)
print("Answer:", response.choices[0].message.content)

출력:

Reasoning: First, I'll break down the problem into two parts: the multiplication and
the division. According to the order of operations (PEMDAS/BODMAS), multiplication and
division are performed from left to right before addition.

17 × 23 = 17 × (20 + 3) = (17 × 20) + (17 × 3) = 340 + 51 = 391
144 / 12 = 12

Finally, add the results: 391 + 12 = 403

Answer: The solution to the problem is as follows:

1. First, perform the multiplication: 17 × 23.
   - 17 × 20 = 340
   - 17 × 3 = 51
   - 340 + 51 = 391

2. Then, perform the division: 144 / 12 = 12.

3. Finally, add the results:
   - 391 + 12 = 403

**Answer:** \boxed{403}

4.2 Instruct Mode (Reasoning Off)

추론 트레이스를 건너뛰고 빠른 직접 응답을 받으려면 reasoning_effort를 "none"으로 설정해요.

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY",
)

response = client.chat.completions.create(
    model="mistralai/Mistral-Small-4-119B-2603",
    messages=[
        {"role": "user", "content": "Write a Python function to reverse a string."},
    ],
    extra_body={"reasoning_effort": "none"},
)

print(response.choices[0].message.content)

출력:

# Python Function to Reverse a String

Here are several ways to write a Python function to reverse a string:

## Method 1: Using String Slicing (Most Pythonic)
```python
def reverse_string(s):
    """Reverse a string using slicing."""
    return s[::-1]
```

## Method 2: Using a Loop
```python Example
def reverse_string(s):
    """Reverse a string using a loop."""
    reversed_str = ""
    for char in s:
        reversed_str = char + reversed_str
    return reversed_str
```

## Method 3: Using reversed() function
```python Example
def reverse_string(s):
    """Reverse a string using reversed() function."""
    return ''.join(reversed(s))
```

The first method using string slicing (`s[::-1]`) is generally the most efficient and
recommended approach in Python.

Example usage:
```python Example
original = "Hello, World!"
reversed_str = reverse_string(original)
print(reversed_str)  # Output: "!dlroW ,olleH"
```

4.3 Streaming with Reasoning

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY",
)

stream = client.chat.completions.create(
    model="mistralai/Mistral-Small-4-119B-2603",
    messages=[
        {"role": "user", "content": "Explain the difference between async and threading in Python."},
    ],
    extra_body={"reasoning_effort": "high"},
    stream=True,
)

print("=== Reasoning ===")
for chunk in stream:
    delta = chunk.choices[0].delta
    if hasattr(delta, "reasoning_content") and delta.reasoning_content:
        print(delta.reasoning_content, end="", flush=True)
    elif delta.content:
        print("\n=== Response ===")
        print(delta.content, end="", flush=True)
print()

출력:

=== Reasoning ===
Okay, the user is asking about the difference between async and threading in Python.
I need to break this down clearly, covering the key aspects of both, like their
purposes, performance characteristics, and use cases...
=== Response ===
In Python, **`async`/`asyncio`** and **`threading`** are two different concurrency
models, each suited for specific use cases. Here's a breakdown of their key differences:

### 1. Model of Concurrency
- **Threading**: Based on preemptive multitasking using OS threads.
- **Async** (`asyncio`): Based on cooperative multitasking. Tasks voluntarily yield...

4.4 Tool Calling

Mistral Small 4는 네이티브 함수 호출을 지원해요. --tool-call-parser mistral로 활성화해요.

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY",
)

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a city",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {"type": "string", "description": "City name"},
                    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
                },
                "required": ["location"],
            },
        },
    }
]

response = client.chat.completions.create(
    model="mistralai/Mistral-Small-4-119B-2603",
    messages=[{"role": "user", "content": "What's the weather in Paris?"}],
    tools=tools,
    tool_choice="auto",
)

tool_calls = response.choices[0].message.tool_calls
for tc in tool_calls:
    print(f"Tool: {tc.function.name}")
    print(f"Args: {tc.function.arguments}")

출력:

Tool: get_weather
Args: {"location": "Paris"}

4.5 Vision (Image Input)

Mistral Small 4는 텍스트와 함께 이미지 입력을 받아요.

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY",
)

response = client.chat.completions.create(
    model="mistralai/Mistral-Small-4-119B-2603",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe what you see in this image."},
                {
                    "type": "image_url",
                    "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/assets/logo.png"},
                },
            ],
        }
    ],
)

print(response.choices[0].message.content)

출력:

The image is a copyright symbol, represented by a stylized version of the lowercase
letter "c" inside a circle. The "c" is depicted in a white or light-colored font, and
the circle is orange. The design is simple yet striking, using oval and elliptical
shapes to create a distinct symbol which signifies copyright protection.

5. Benchmarks (벤치마크)

5.1 Accuracy Benchmarks

GSM8K

python3 benchmark/gsm8k/bench_sglang.py --port 30000

결과:

TODO

MMLU

python3 benchmark/mmlu/bench_sglang.py --port 30000

결과:

TODO

5.2 Speed Benchmarks

Latency (Low Concurrency)

python3 -m sglang.bench_serving \
  --backend sglang \
  --num-prompts 10 \
  --max-concurrency 1 \
  --random-input-len 1024 \
  --random-output-len 512 \
  --port 30000

결과:

TODO

Throughput (High Concurrency)

python3 -m sglang.bench_serving \
  --backend sglang \
  --num-prompts 1000 \
  --max-concurrency 100 \
  --random-input-len 1024 \
  --random-output-len 512 \
  --port 30000

결과:

TODO

더 알아보기 (Learn more)

• 공식 SGLang 설치 가이드 — SGLang 설치 방법
• Mistral Small 4 (Hugging Face) — FP8 체크포인트
• Mistral Small 4 EAGLE (Hugging Face) — EAGLE 추측 디코딩 가중치
• Leanstral 2603 (Hugging Face) — 동일 아키텍처 모델