GPT-OSS

3.2 Configuration Tips

• 네이티브 웹 검색: SGLang 서버 환경에 EXA_API_KEY를 설정하면 내장 웹 검색(Exa)을 활성화해요. --tool-server는 필요 없고, 요청에 x-exa-integration: sglang이 태그됩니다.
• 웹 검색 기본값: numResults=10, 검색 type="auto", contents.highlights=true. SGLANG_EXA_NUM_RESULTS, SGLANG_EXA_SEARCH_TYPE, SGLANG_EXA_INCLUDE_HIGHLIGHTS로 재정의할 수 있어요.
• Python 도구: --tool-server demo를 추가하면 Python 인터프리터를 활성화해요. 기본적으로 Docker 샌드박스에서 실행되며, PYTHON_EXECUTION_BACKEND=UV로 설정하면 호스트에서 실행돼요(모델 생성 코드가 로컬에서 실행되므로 주의해서 사용하세요).
• MCP 도구 서버: 프로덕션에서는 --tool-server ip-1:port-1,ip-2:port-2로 SGLang을 외부 MCP SSE 서버에 연결하세요.
• Responses API: GPT-OSS는 표준 Chat Completions API 외에도 OpenAI의 Responses API(client.responses.create)를 지원해요(섹션 4.2.4 참조).
• 데모 Python 도구 실행 시 Python 3.12를 사용하세요.
• Xeon CPU 서비스 구성: SGLang CPU 서버 문서의 serving engine 실행 섹션에서 Notes 부분을 참조해 TP(텐서 병렬) 및 NUMA 바인딩 설정을 올바르게 구성하는 방법을 이해하세요.

4.Model Invocation

4.1 Basic Usage

기본 API 사용법과 요청 예시는 다음을 참고하세요:

• SGLang Basic Usage Guide

4.2 Advanced Usage

4.2.1 Reasoning Parser

GPT-OSS는 reasoning 모드를 지원해요. 배포 시 reasoning 파서를 활성화해 thinking 섹션과 content 섹션을 분리하세요:

python -m sglang.launch_server \
  --model openai/gpt-oss-120b \
  --reasoning-parser gpt-oss \
  --tp 8

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY"
)

# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
    model="openai/gpt-oss-120b",
    messages=[
        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
    ],
    temperature=0.7,
    max_tokens=2048,
    stream=True
)

# Process the stream
has_thinking = False
has_answer = False
thinking_started = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Print answer content
        if delta.content:
            # Close thinking section and add content header
            if has_thinking and not has_answer:
                print("\n=============== Content =================", flush=True)
                has_answer = True
            print(delta.content, end="", flush=True)

print()

출력 예시:

=============== Thinking =================
The user asks: "Solve this problem step by step: What is 15% of 240?" So we need to provide step-by-step solution. Compute 15% of 240: 0.15 * 240 = 36. Provide steps: convert percent to decimal, multiply, maybe use fraction. Provide answer.
=============== Content =================
**Step‑by‑step solution**

1. **Understand what “percent” means**
   “15 %” means 15 out of every 100 parts, i.e. the fraction \(\displaystyle \frac{15}{100}\).

2. **Convert the percent to a decimal (or fraction)**
   \[
   \frac{15}{100}=0.15
   \]

3. **Set up the multiplication**
   To find 15 % of 240 we multiply 240 by the decimal 0.15:
   \[
   240 \times 0.15
   \]

4. **Do the multiplication**
   One convenient way is to break it into two easier parts:
   \[
   240 \times 0.15 = 240 \times \left(\frac{15}{100}\right)
                = \frac{240 \times 15}{100}
   \]

   - First compute \(240 \times 15\):
     \[
     240 \times 15 = 240 \times (10 + 5) = 2400 + 1200 = 3600
     \]

   - Then divide by 100:
     \[
     \frac{3600}{100} = 36
     \]

5. **Write the result**
   \[
   15\% \text{ of } 240 = 36
   \]

---

**Answer:** \(36\)

4.2.2 Tool Calling

GPT-OSS는 도구 호출 기능을 지원해요. 도구 호출 파서를 활성화하세요:

Python 예시 (Thinking Process 없이):

sglang 서버를 시작하세요:

python -m sglang.launch_server \
  --model openai/gpt-oss-120b \
  --tool-call-parser gpt-oss \
  --tp 8

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY"
)

# Define available tools
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city name"
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "Temperature unit"
                    }
                },
                "required": ["location"]
            }
        }
    }
]

# Make request with streaming to see thinking process
response = client.chat.completions.create(
    model="openai/gpt-oss-120b",
    messages=[
        {"role": "user", "content": "What's the weather in Beijing?"}
    ],
    tools=tools,
    temperature=0.7,
    stream=True
)

# Process streaming response
thinking_started = False
has_thinking = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Print tool calls
        if hasattr(delta, 'tool_calls') and delta.tool_calls:
            # Close thinking section if needed
            if has_thinking and thinking_started:
                print("\n=============== Content =================", flush=True)
                thinking_started = False

            for tool_call in delta.tool_calls:
                if tool_call.function:
                    print(f"🔧 Tool Call: {tool_call.function.name}")
                    print(f"   Arguments: {tool_call.function.arguments}")

        # Print content
        if delta.content:
            print(delta.content, end="", flush=True)

print()

출력 예시:

🔧 Tool Call: get_weather
   Arguments: {"location": "Beijing", "unit": "celsius"}

Python 예시 (Thinking Process 포함):

sglang 서버를 시작하세요:

python -m sglang.launch_server \
  --model openai/gpt-oss-120b \
  --reasoning-parser gpt-oss \
  --tool-call-parser gpt-oss \
  --tp 8

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY"
)

# Define available tools
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city name"
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "Temperature unit"
                    }
                },
                "required": ["location"]
            }
        }
    }
]

# Make request with streaming to see thinking process
response = client.chat.completions.create(
    model="openai/gpt-oss-120b",
    messages=[
        {"role": "user", "content": "What's the weather in Beijing?"}
    ],
    tools=tools,
    temperature=0.7,
    stream=True
)

# Process streaming response
thinking_started = False
has_thinking = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Print tool calls
        if hasattr(delta, 'tool_calls') and delta.tool_calls:
            # Close thinking section if needed
            if has_thinking and thinking_started:
                print("\n=============== Content =================", flush=True)
                thinking_started = False

            for tool_call in delta.tool_calls:
                if tool_call.function:
                    print(f"🔧 Tool Call: {tool_call.function.name}")
                    print(f"   Arguments: {tool_call.function.arguments}")

        # Print content
        if delta.content:
            print(delta.content, end="", flush=True)

print()

출력 예시:

=============== Thinking =================
User asks: "What's the weather in Beijing?" We need to get current weather. Use function get_weather with location "Beijing". No unit specified; default? Probably use default (maybe Celsius). We can specify unit as "celsius". We'll call function.
=============== Content =================
🔧 Tool Call: get_weather
   Arguments: {"location": "Beijing", "unit": "celsius"}

참고:

• reasoning 파서는 모델이 도구를 어떻게 사용하기로 결정하는지 보여줘요
• 도구 호출은 함수 이름과 인자로 명확히 표시돼요
• 그런 다음 함수를 실행하고 결과를 다시 보내 대화를 계속할 수 있어요

도구 호출 결과 처리:

# After getting the tool call, execute the function
def get_weather(location, unit="celsius"):
    # Your actual weather API call here
    return f"The weather in {location} is 22°{unit[0].upper()} and sunny."

# Send tool result back to the model
messages = [
    {"role": "user", "content": "What's the weather in Beijing?"},
    {
        "role": "assistant",
        "content": None,
        "tool_calls": [{
            "id": "call_123",
            "type": "function",
            "function": {
                "name": "get_weather",
                "arguments": '{"location": "Beijing", "unit": "celsius"}'
            }
        }]
    },
    {
        "role": "tool",
        "tool_call_id": "call_123",
        "content": get_weather("Beijing", "celsius")
    }
]

final_response = client.chat.completions.create(
    model="openai/gpt-oss-120b",
    messages=messages,
    temperature=0.7
)

print(final_response.choices[0].message.content)
# Output: "The current weather in Beijing is 22 °C and sunny. Let me know if you'd like a forecast for the next few days or any other details!"

4.2.3 EAGLE3 Speculative Decoding

SGLang은 EAGLE3 알고리즘을 사용한 GPT-OSS 모델의 추측 디코딩을 지원해요. 특히 작은 배치 크기에서 디코딩 속도를 크게 높일 수 있어요.

python3 -m sglang.launch_server \
  --model-path openai/gpt-oss-120b \
  --speculative-algorithm EAGLE3 \
  --speculative-draft-model-path lmsys/EAGLE3-gpt-oss-120b-bf16 \
  --tp 2

4.2.4 Responses API and Built-in Tools

GPT-OSS는 내장 도구 사용(웹 검색 및 Python 인터프리터)과 함께 OpenAI Responses API를 지원해요. EXA_API_KEY를 설정하면 네이티브 웹 검색을 활성화하고, Python 도구도 원한다면 --tool-server demo를 추가하세요:

export EXA_API_KEY=YOUR_EXA_KEY
# Optional: server-side Exa tuning (defaults shown)
export SGLANG_EXA_NUM_RESULTS=10
export SGLANG_EXA_SEARCH_TYPE=auto
export SGLANG_EXA_INCLUDE_HIGHLIGHTS=true
# Optional: run Python tool on host instead of Docker (model code executes locally)
export PYTHON_EXECUTION_BACKEND=UV

python3 -m sglang.launch_server \
  --model-path openai/gpt-oss-120b \
  --tp 2

프로덕션에서는 demo 대신 외부 MCP SSE 서버를 사용하세요:

mcp run -t sse browser_server.py:mcp
mcp run -t sse python_server.py:mcp

python -m sglang.launch_server \
  --model-path openai/gpt-oss-120b \
  --tool-server ip-1:port-1,ip-2:port-2 \
  --tp 2

Responses API 사용 예시:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="sk-123456")

search_tools = [{"type": "web_search"}]
python_tools = [{"type": "code_interpreter"}]

# Configurable reasoning effort: "high", "medium", or "low"
response = client.responses.create(
    model="openai/gpt-oss-120b",
    instructions="You are a helpful assistant.",
    reasoning_effort="high",
    input="In one sentence, explain the transformer architecture.",
)
print(response.output_text)

# Web search (requires EXA_API_KEY on the SGLang server)
response = client.responses.create(
    model="openai/gpt-oss-120b",
    instructions="You are a helpful assistant, you can search the web when needed.",
    input="Search the web for the latest news about Nvidia stock price",
    tools=search_tools,
)
print(response.output_text)

# Python tool (requires launching SGLang with --tool-server demo)
response = client.responses.create(
    model="openai/gpt-oss-120b",
    instructions="You are a helpful assistant, you could use python tool to execute code.",
    input="Use python tool to calculate the sum of 29138749187 and 29138749187",
    tools=python_tools,
)
print(response.output_text)
# Output: The sum is 58,277,498,374.

5.Benchmark

5.1 Speed Benchmark

• 하드웨어: NVIDIA B200 GPU (8x)
• 텐서 병렬 처리(Tensor Parallelism): 8
• 모델: openai/gpt-oss-120b
• sglang 버전: 0.5.6

SGLang의 내장 벤치마킹 도구를 사용해 ShareGPT_Vicuna_unfiltered 데이터셋에서 성능 평가를 수행해요. 이 데이터셋은 실제 대화 데이터를 포함해 실제 사용 시나리오에서의 성능을 더 잘 반영할 수 있어요.

5.1.1 Latency-Sensitive Benchmark

• 서버 명령:

python -m sglang.launch_server \
  --model openai/gpt-oss-120b \
  --tp 8

• 테스트 명령:

python3 -m sglang.bench_serving \
  --backend sglang \
  --num-prompt 100 \
  --max-concurrency 1

• 테스트 결과:

============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     100
Benchmark duration (s):                  52.35
Total input tokens:                      33178
Total input text tokens:                 33178
Total input vision tokens:               0
Total generated tokens:                  21251
Total generated tokens (retokenized):    20868
Request throughput (req/s):              1.91
Input token throughput (tok/s):          633.76
Output token throughput (tok/s):         405.93
Peak output token throughput (tok/s):    433.00
Peak concurrent requests:                8
Total token throughput (tok/s):          1039.69
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   523.30
Median E2E Latency (ms):                 389.91
---------------Time to First Token----------------
Mean TTFT (ms):                          33.71
Median TTFT (ms):                        31.79
P99 TTFT (ms):                           108.98
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          2.31
Median TPOT (ms):                        2.31
P99 TPOT (ms):                           2.39
---------------Inter-Token Latency----------------
Mean ITL (ms):                           2.31
Median ITL (ms):                         2.31
P95 ITL (ms):                            2.35
P99 ITL (ms):                            2.38
Max ITL (ms):                            3.54
==================================================

5.1.2 Throughput-Sensitive Benchmark

• 서버 명령:

python -m sglang.launch_server \
  --model openai/gpt-oss-120b \
  --tp 8

• 테스트 명령:

python3 -m sglang.bench_serving \
  --backend sglang \
  --num-prompt 1000 \
  --max-concurrency 100

테스트 결과:

============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     1000
Benchmark duration (s):                  24.76
Total input tokens:                      297156
Total input text tokens:                 297156
Total input vision tokens:               0
Total generated tokens:                  192432
Total generated tokens (retokenized):    187145
Request throughput (req/s):              40.39
Input token throughput (tok/s):          12003.57
Output token throughput (tok/s):         7773.26
Peak output token throughput (tok/s):    13780.00
Peak concurrent requests:                156
Total token throughput (tok/s):          19776.83
Concurrency:                             89.23
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   2208.97
Median E2E Latency (ms):                 1591.11
---------------Time to First Token----------------
Mean TTFT (ms):                          102.94
Median TTFT (ms):                        31.53
P99 TTFT (ms):                           674.32
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          14.31
Median TPOT (ms):                        11.00
P99 TPOT (ms):                           91.28
---------------Inter-Token Latency----------------
Mean ITL (ms):                           11.00
Median ITL (ms):                         5.75
P95 ITL (ms):                            25.35
P99 ITL (ms):                            43.18
Max ITL (ms):                            621.42
==================================================

5.2 Accuracy Benchmark

5.2.1 GSM8K Benchmark

• 벤치마크 명령:

python3 -m sglang.test.few_shot_gsm8k --num-questions 200 --port 8000

• 결과:

  • GPT-OSS-120b

    Accuracy: 0.880
    Invalid: 0.005
    Latency: 5.262 s
    Output throughput: 12143.675 token/s
    

  • GPT-OSS-20b

    Accuracy: 0.535
    Invalid: 0.165
    Latency: 4.157 s
    Output throughput: 19589.165 token/s
    

더 알아보기

• GPT-OSS-20b (Hugging Face)
• GPT-OSS (OpenAI)