Qwen3-VL

3.2 Configuration Tips

• 멀티모달 attention 백엔드: 보통 --mm-attention-backend는 H100/H200/A100에서 성능 향상을 위해 fa3가 기본값이지만, B200에서는 호환성을 위해 triton_attn이 기본값이에요.
• TTFT 최적화: SGLANG_USE_CUDA_IPC_TRANSPORT=1을 설정해 CUDA IPC로 멀티모달 피처를 전송하면 TTFT가 크게 개선돼요. 이는 추가 메모리를 소비하며 --mem-fraction-static 및/또는 --max-running-requests 조정이 필요할 수 있어요. (추가 메모리는 현재 실행 중인 요청의 이미지 크기 × 이미지 수에 비례해요.)
• 메모리 관리: --context-length를 낮게 설정해 메모리를 절약하세요. 기본 262K 대비 128000이 대부분 시나리오에 충분해요.
• Expert Parallelism: SGLang은 --ep로 Expert Parallelism(EP)을 지원해 MoE 모델의 전문가를 별도 GPU에 배포해 처리량을 높일 수 있어요. 양자화 모델의 경우 --ep를 (moe_intermediate_size / moe_tp_size) % weight_block_size_n == 0(여기서 moe_tp_size는 tp_size를 ep_size로 나눈 값) 조건을 만족하는 값으로 설정해야 해요. EP는 추가 통신 오버헤드 때문에 저동시성 시나리오에서 성능이 더 나빠질 수 있다는 점에 유의하세요. 자세한 내용은 Expert Parallelism 배포를 참조하세요.
• 커널 튜닝: 특정 하드웨어에서 MoE Triton 커널 튜닝은 fused_moe_triton을 참조하세요.

하드웨어별 참고사항:

• H100 (FP8): 최상의 메모리 효율을 위해 Qwen/Qwen3-VL-235B-A22B-Instruct-FP8 체크포인트를 사용하세요.
• A100 / H100 (BF16): 처리량과 GPU 메모리 사용을 관리하기 위해 표준 멀티모달 파라미터를 사용하세요.
• H200 / B200: 기본으로 동작하며 전체 컨텍스트 길이와 동시 이미지+비디오 처리를 지원해요.

추가 멀티모달 서버 파라미터:

• --mm-feature-transport=cuda_ipc: 멀티모달 피처 텐서를 CUDA IPC로 공유되는 제한된 GPU 풀에 유지해 device-to-host 메모리 복사를 피하고 고주파 추론 성능을 개선해요.

전체 멀티모달 최적화 예시:

SGLANG_USE_CUDA_IPC_TRANSPORT=1 \
SGLANG_VLM_CACHE_SIZE_MB=0 \
python -m sglang.launch_server \
  --model-path Qwen/Qwen3-VL-235B-A22B-Instruct \
  --host 0.0.0.0 \
  --port 30000 \
  --trust-remote-code \
  --tp-size 8 \
  --enable-cache-report \
  --log-level info \
  --max-running-requests 64 \
  --mem-fraction-static 0.65 \
  --chunked-prefill-size 8192 \
  --attention-backend fa3 \
  --mm-attention-backend fa3 \
  --enable-metrics

• Xeon CPU 서비스 구성: SGLang CPU 서버 문서의 serving engine 실행 섹션에서 Notes 부분을 참조해 인자, 특히 TP(텐서 병렬)와 NUMA 바인딩 설정을 올바르게 구성하는 방법을 이해하세요.

4. Model Invocation

4.1 Basic Usage

기본 API 사용법과 요청 예시는 다음을 참조하세요:

• SGLang Basic Usage Guide
• SGLang OpenAI Vision API Guide

4.2 Advanced Usage

4.2.1 Multi-Modal Inputs

Qwen3-VL은 이미지와 비디오 입력을 모두 지원해요. 이미지 입력의 기본 예시는 다음과 같아요:

import time
from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url="http://localhost:30000/v1",
    timeout=3600
)

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
                }
            },
            {
                "type": "text",
                "text": "Read all the text in the image."
            }
        ]
    }
]

start = time.time()
response = client.chat.completions.create(
    model="Qwen/Qwen3-VL-235B-A22B-Instruct",
    messages=messages,
    max_tokens=2048
)
print(f"Response costs: {time.time() - start:.2f}s")
print(f"Generated text: {response.choices[0].message.content}")

출력 예시:

Response costs: 3.37s
Generated text: Auntie Anne's

CINNAMON SUGAR
1 x 17,000                    17,000

SUB TOTAL                    17,000

GRAND TOTAL                  17,000

CASH IDR                     20,000

CHANGE DUE                  3,000

멀티 이미지 입력 예시:

Qwen3-VL은 비교나 분석을 위해 단일 요청에서 여러 이미지를 처리할 수 있어요:

import time
from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url="http://localhost:30000/v1",
    timeout=3600
)

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://www.civitatis.com/f/china/hong-kong/guia/taxi.jpg"
                }
            },
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://cdn.cheapoguides.com/wp-content/uploads/sites/7/2025/05/GettyImages-509614603-1280x600.jpg"
                }
            },
            {
                "type": "text",
                "text": "Compare these two images and describe the differences in 100 words or less. Focus on the key visual elements, colors, textures, and any notable contrasts between the two scenes. Be specific about what you see in each image."
            }
        ]
    }
]

start = time.time()
response = client.chat.completions.create(
    model="Qwen/Qwen3-VL-235B-A22B-Instruct",
    messages=messages,
    max_tokens=2048
)
print(f"Response costs: {time.time() - start:.2f}s")
print(f"Generated text: {response.choices[0].message.content}")

출력 예시:

Response costs: 10.18s
Generated text: The two images present starkly different portrayals of Hong Kong's iconic red taxis, contrasting a dynamic street-level moment with a static, large-scale gathering.
...

비디오 입력 예시:

Qwen3-VL은 비디오 URL을 처리해 비디오 이해를 지원해요:

import time
from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url="http://localhost:30000/v1",
    timeout=3600
)

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "video_url",
                "video_url": {
                    "url": "https://videos.pexels.com/video-files/4114797/4114797-uhd_3840_2160_25fps.mp4"
                }
            },
            {
                "type": "text",
                "text": "Describe what happens in this video."
            }
        ]
    }
]

start = time.time()
response = client.chat.completions.create(
    model="Qwen/Qwen3-VL-235B-A22B-Instruct",
    messages=messages,
    max_tokens=2048
)
print(f"Response costs: {time.time() - start:.2f}s")
print(f"Generated text: {response.choices[0].message.content}")

참고:

• 비디오 처리를 위해 충분한 컨텍스트 길이(최대 262K 토큰)를 구성해야 해요.
• 비디오 처리는 더 많은 메모리가 필요할 수 있으므로 --mem-fraction-static을 그에 맞게 조정하세요.
• file:// 프로토콜로 로컬 파일 경로를 제공할 수도 있어요.

출력 예시:

Response costs: 3.89s
Generated text: A person wearing blue gloves is using a microscope. They are adjusting the focus knob with one hand while holding a pipette with the other, suggesting they are preparing or examining a sample on the slide beneath the objective lens. The microscope's 40x objective lens is positioned over the slide, indicating a high-magnification observation. The person carefully manipulates the slide and the microscope controls, likely to achieve a clear view of the specimen.

4.2.2 Reasoning Parser

Qwen3-VL-Thinking은 reasoning 모드를 지원해요. 배포 중 reasoning 파서를 활성화해 thinking 섹션과 content 섹션을 분리하세요:

python -m sglang.launch_server \
  --model Qwen/Qwen3-VL-235B-A22B-Thinking \
  --reasoning-parser qwen3 \
  --tp 8 \
  --host 0.0.0.0 \
  --port 30000

생각 과정(Thinking Process)이 있는 스트리밍:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
    model="Qwen/Qwen3-VL-235B-A22B-Thinking",
    messages=[
        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
    ],
    temperature=0.7,
    max_tokens=2048,
    stream=True
)

# Process the stream
has_thinking = False
has_answer = False
thinking_started = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Print answer content
        if delta.content:
            # Close thinking section and add content header
            if has_thinking and not has_answer:
                print("\n=============== Content =================", flush=True)
                has_answer = True
            print(delta.content, end="", flush=True)

print()

출력 예시:

=============== Thinking ================
To solve this problem, I need to calculate 15% of 240.
Step 1: Convert 15% to decimal: 15% = 0.15
Step 2: Multiply 240 by 0.15
Step 3: 240 × 0.15 = 36
=============== Content ================

The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36.

참고: reasoning 파서는 모델의 단계별 사고 과정을 포착해 모델이 어떻게 결론에 도달하는지 볼 수 있게 해 줘요.

4.2.3 Tool Calling

Qwen3-VL은 도구 호출 기능을 지원해요. 도구 호출 파서를 활성화하세요:

python -m sglang.launch_server \
  --model Qwen/Qwen3-VL-235B-A22B-Thinking \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen \
  --tp 8 \
  --host 0.0.0.0 \
  --port 30000

Python 예시 (생각 과정 포함):

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

# Define available tools
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city name"
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "Temperature unit"
                    }
                },
                "required": ["location"]
            }
        }
    }
]

# Make request with streaming to see thinking process
response = client.chat.completions.create(
    model="Qwen/Qwen3-VL-235B-A22B-Thinking",
    messages=[
        {"role": "user", "content": "What's the weather in Beijing?"}
    ],
    tools=tools,
    temperature=0.7,
    stream=True
)

# Process streaming response
thinking_started = False
has_thinking = False
tool_calls_accumulator = {}

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Accumulate tool calls
        if hasattr(delta, 'tool_calls') and delta.tool_calls:
            # Close thinking section if needed
            if has_thinking and thinking_started:
                print("\n=============== Content ================\n", flush=True)
                thinking_started = False

            for tool_call in delta.tool_calls:
                index = tool_call.index
                if index not in tool_calls_accumulator:
                    tool_calls_accumulator[index] = {
                        'name': None,
                        'arguments': ''
                    }

                if tool_call.function:
                    if tool_call.function.name:
                        tool_calls_accumulator[index]['name'] = tool_call.function.name
                    if tool_call.function.arguments:
                        tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments

        # Print content
        if delta.content:
            print(delta.content, end="", flush=True)

# Print accumulated tool calls
for index, tool_call in sorted(tool_calls_accumulator.items()):
    print(f"🔧 Tool Call: {tool_call['name']}")
    print(f"   Arguments: {tool_call['arguments']}")

print()

출력 예시:

=============== Thinking ================
The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information.
I should call the function with location="Beijing".
=============== Content ================

🔧 Tool Call: get_weather
   Arguments: {"location": "Beijing", "unit": "celsius"}

참고:

• reasoning 파서는 모델이 도구를 어떻게 사용하기로 결정하는지 보여줘요.
• 도구 호출은 함수 이름과 인자로 명확히 표시돼요.
• 그런 다음 함수를 실행하고 결과를 다시 보내 대화를 계속할 수 있어요.

도구 호출 결과 처리:

# After getting the tool call, execute the function
def get_weather(location, unit="celsius"):
    # Your actual weather API call here
    return f"The weather in {location} is 22°{unit[0].upper()} and sunny."

# Send tool result back to the model
messages = [
    {"role": "user", "content": "What's the weather in Beijing?"},
    {
        "role": "assistant",
        "content": None,
        "tool_calls": [{
            "id": "call_123",
            "type": "function",
            "function": {
                "name": "get_weather",
                "arguments": '{"location": "Beijing", "unit": "celsius"}'
            }
        }]
    },
    {
        "role": "tool",
        "tool_call_id": "call_123",
        "content": get_weather("Beijing", "celsius")
    }
]

final_response = client.chat.completions.create(
    model="Qwen/Qwen3-VL-235B-A22B-Thinking",
    messages=messages,
    temperature=0.7
)

print(final_response.choices[0].message.content)
# Output: "The weather in Beijing is currently 22°C and sunny."

5. Benchmark

5.1 Speed Benchmark

테스트 환경:

• 하드웨어: NVIDIA B200 GPU (8x)
• 모델: Qwen3-VL-235B-A22B-Instruct
• 텐서 병렬 처리(Tensor Parallelism): 8
• sglang 버전: 0.5.6

SGLang의 내장 벤치마킹 도구로 무작위 이미지를 사용해 성능 평가를 수행해요. 실제 사용을 시뮬레이션하려면 각 요청에 서로 다른 입력·출력 길이를 지정할 수 있어요. 예를 들어 각 요청은 128 입력 토큰, 720p 이미지 2장, 1024 출력 토큰을 가질 수 있어요.

5.1.1 Latency-Sensitive Benchmark

• 모델 배포 명령:

python -m sglang.launch_server \
  --model Qwen/Qwen3-VL-235B-A22B-Instruct \
  --tp 8 \
  --host 0.0.0.0 \
  --port 30000

• 벤치마크 명령:

python3 -m sglang.bench_serving \
  --backend sglang-oai-chat \
  --host 127.0.0.1 \
  --port 30000 \
  --model Qwen/Qwen3-VL-235B-A22B-Instruct \
  --dataset-name image \
  --image-count 2 \
  --image-resolution 720p \
  --random-input-len 128 \
  --random-output-len 1024 \
  --num-prompts 10 \
  --max-concurrency 1

• 테스트 결과:

============ Serving Benchmark Result ============
Backend:                                 sglang-oai-chat
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  45.97
Total input tokens:                      18348
Total input text tokens:                 708
Total input vision tokens:               17640
Total generated tokens:                  4220
Total generated tokens (retokenized):    3423
Request throughput (req/s):              0.22
Input token throughput (tok/s):          399.17
Output token throughput (tok/s):         91.81
Peak output token throughput (tok/s):    96.00
Peak concurrent requests:                2
Total token throughput (tok/s):          490.98
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   4594.52
Median E2E Latency (ms):                 3725.04
---------------Time to First Token----------------
Mean TTFT (ms):                          193.35
Median TTFT (ms):                        196.32
P99 TTFT (ms):                           222.75
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          10.44
Median TPOT (ms):                        10.44
P99 TPOT (ms):                           10.47
---------------Inter-Token Latency----------------
Mean ITL (ms):                           11.78
Median ITL (ms):                         10.48
P95 ITL (ms):                            21.01
P99 ITL (ms):                            31.40
Max ITL (ms):                            31.92
==================================================

최적화 결과 (CUDA IPC Transport 사용):

추가 TTFT 최적화를 위해 SGLANG_USE_CUDA_IPC_TRANSPORT=1을 설정해 멀티모달 피처를 CUDA IPC로 전송하면 TTFT가 크게 줄어들어요.

• 모델 배포 명령:

SGLANG_USE_CUDA_IPC_TRANSPORT=1 python -m sglang.launch_server \
  --model Qwen/Qwen3-VL-235B-A22B-Instruct \
  --tp 8 \
  --host 0.0.0.0 \
  --port 30000

• 벤치마크 명령:

python3 -m sglang.bench_serving \
  --backend sglang-oai-chat \
  --host 127.0.0.1 \
  --port 30000 \
  --model Qwen/Qwen3-VL-235B-A22B-Instruct \
  --dataset-name image \
  --image-count 2 \
  --image-resolution 720p \
  --random-input-len 128 \
  --random-output-len 1024 \
  --num-prompts 100 \
  --max-concurrency 1

• 테스트 결과:

  SGLANG_USE_CUDA_IPC_TRANSPORT=1을 사용하면 TTFT가 크게 개선돼요:

============ Serving Benchmark Result ============
Backend:                                 sglang-oai-chat
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     100
Benchmark duration (s):                  566.84
Total input tokens:                      183667
Total input text tokens:                 7267
Total input vision tokens:               176400
Total generated tokens:                  52444
Total generated tokens (retokenized):    28702
Request throughput (req/s):              0.18
Input token throughput (tok/s):          324.02
Output token throughput (tok/s):         92.52
Peak output token throughput (tok/s):    96.00
Peak concurrent requests:                3
Total token throughput (tok/s):          416.54
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   5667.50
Median E2E Latency (ms):                 5830.00
---------------Time to First Token----------------
Mean TTFT (ms):                          191.16
Median TTFT (ms):                        182.58
P99 TTFT (ms):                           244.58
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          10.46
Median TPOT (ms):                        10.46
P99 TPOT (ms):                           10.48
---------------Inter-Token Latency----------------
Mean ITL (ms):                           13.91
Median ITL (ms):                         10.56
P95 ITL (ms):                            21.35
P99 ITL (ms):                            31.55
Max ITL (ms):                            42.36
==================================================

5.1.2 Throughput-Sensitive Benchmark

• 모델 배포 명령:

python -m sglang.launch_server \
  --model Qwen/Qwen3-VL-235B-A22B-Instruct \
  --tp 8 \
  --host 0.0.0.0 \
  --port 30000

• 벤치마크 명령:

python3 -m sglang.bench_serving \
  --backend sglang-oai-chat \
  --host 127.0.0.1 \
  --port 30000 \
  --model Qwen/Qwen3-VL-235B-A22B-Instruct \
  --dataset-name image \
  --image-count 2 \
  --image-resolution 720p \
  --random-input-len 128 \
  --random-output-len 1024 \
  --num-prompts 1000 \
  --max-concurrency 100

• 테스트 결과:

============ Serving Benchmark Result ============
Backend:                                 sglang-oai-chat
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     1000
Benchmark duration (s):                  584.65
Total input tokens:                      1839015
Total input text tokens:                 75015
Total input vision tokens:               1764000
Total generated tokens:                  510855
Total generated tokens (retokenized):    284284
Request throughput (req/s):              1.71
Input token throughput (tok/s):          3145.50
Output token throughput (tok/s):         873.78
Peak output token throughput (tok/s):    2855.00
Peak concurrent requests:                107
Total token throughput (tok/s):          4019.29
Concurrency:                             98.35
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   57502.05
Median E2E Latency (ms):                 54301.08
---------------Time to First Token----------------
Mean TTFT (ms):                          5802.23
Median TTFT (ms):                        1444.75
P99 TTFT (ms):                           46675.92
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          100.22
Median TPOT (ms):                        105.43
P99 TPOT (ms):                           144.37
---------------Inter-Token Latency----------------
Mean ITL (ms):                           134.20
Median ITL (ms):                         25.57
P95 ITL (ms):                            558.14
P99 ITL (ms):                            1449.01
Max ITL (ms):                            33453.23
==================================================

5.2 Accuracy Benchmark

5.2.1 MMMU Benchmark

lmms_eval로 MMMU 데이터셋을 사용해 모델의 정확도를 평가할 수 있어요:

• 벤치마크 명령:

uv pip install lmms_eval

python3 -m lmms_eval \
  --model openai_compatible \
  --model_args "model=Qwen/Qwen3-VL-235B-A22B-Instruct,api_key=EMPTY,base_url=http://127.0.0.1:30000/v1/" \
  --tasks mmmu_val \
  --batch_size 128 \
  --log_samples \
  --log_samples_suffix "openai_compatible" \
  --output_path ./logs \
  --gen_kwargs "max_new_tokens=4096"

• 테스트 결과:

더 알아보기

• Qwen3-VL GitHub 저장소