Kimi-K2.7-Code

3.2 Configuration Tips

• 메모리: 각각 ≥140GB인 GPU가 필요해요. 네이티브 INT4 체크포인트는 H200 (8×, TP=8), B300 (8×, TP=8), GB300 (4×, TP=4), MI300X/MI325X (4×, TP=4), MI350X/MI355X (4×, TP=4)를 지원해요. 메모리를 아끼려면 --context-length 128000을 사용해요.
• 컨텍스트 길이: 모델은 256K 컨텍스트 길이를 지원해요. 더 큰 배치를 위해 메모리를 예약해야 할 때는 더 짧은 --context-length을 사용해요.
• Transformers 버전: 모델 카드는 transformers>=4.57.1,<5.0.0을 요구해요.
• AMD GPU TP 제약: AMD GPU에서 TP는 8이 아니라 ≤ 4여야 해요. Kimi-K2.7-Code는 64개의 attention heads를 가져요. AITER MLA 커널은 heads_per_gpu % 16 == 0을 요구해요. TP=4면 GPU당 16개 head(유효), TP=8이면 GPU당 8개 head(무효)가 돼요.
• AMD Docker 이미지: MI350X/MI355X에는 lmsysorg/sglang:v0.5.9-rocm700-mi35x, MI300X/MI325X에는 lmsysorg/sglang:v0.5.9-rocm700-mi30x를 사용해요.
• DP Attention: 프로덕션 처리량을 위해 --dp <N> --enable-dp-attention으로 활성화해요. 보통 --dp를 --tp와 같게 설정하지만 필수는 아니에요.
• 추론 파서: --reasoning-parser kimi_k2를 추가해 모델 출력에서 thinking과 content를 분리해요.
• 툴 호출 파서: 구조화된 툴 호출을 위해 --tool-call-parser kimi_k2를 추가해요.
• AMD FP8 KV Cache: AMD 플랫폼에서 생성기가 기본적으로 --kv-cache-dtype fp8_e4m3을 추가하고 --mem-fraction-static 0.8을 설정해 INT4 가중치와 KV cache를 맞춰요. FP8 KV cache는 약간의 정확도를 메모리로 바꾸는 거예요. 워크로드에서 정확도 저하가 보이면 이 플래그를 빼요.
• MXFP4 체크포인트: MXFP4 체크포인트는 rocm/sgl-dev:v0.5.17-rocm720-mi35x-20260812 이미지(ROCm 7.2)로 MI350X/MI355X(TP=4)에서 검증되었어요. Quantization 컨트롤에서 MXFP4를 선택해 amd/Kimi-K2.7-Code-MXFP4를 타깃으로 해요. MXFP4 양자화는 체크포인트에서 자동 감지되므로 --quantization 플래그가 필요 없어요. 생성기는 AITER/ROCm 환경 블록과 --attention-backend aiter, --mem-fraction-static 0.90, FP8 KV cache, --disable-radix-cache, --enable-aiter-allreduce-fusion을 내보내요. 추론·툴 호출 파서는 자동 감지되므로 명시적으로 전달하지 않아요.

4. Model Invocation (모델 호출)

4.1 Basic Usage

기본 API 사용법을 참고해요.

4.2 Advanced Usage

4.2.1 Multimodal (Vision + Text) Input

Kimi-K2.7-Code는 이미지가 있는 네이티브 멀티모달 입력을 지원해요.

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

response = client.chat.completions.create(
    model="moonshotai/Kimi-K2.7-Code",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
                    }
                },
                {
                    "type": "text",
                    "text": "What is in this image? Describe it in detail."
                }
            ]
        }
    ]
)

print(response.choices[0].message.content)

출력 예시:

This image shows a **paper receipt from Auntie Anne's**, the pretzel chain restaurant. Here's a detailed breakdown:

## Header
- At the top left is the Auntie Anne's logo (a pretzel with a halo)
- The store name "**Auntie Anne's**" is printed prominently at the top
- Some text below the store name appears blurred/redacted (likely store location, address, or transaction details)

## Purchase Details
- **Item**: CINNAMON SUGAR
- **Quantity & Price**: 1 × 17,000
- **Item Total**: 17,000

## Financial Summary
- **SUB TOTAL**: 17,000
- **GRAND TOTAL**: 17,000
- **CASH IDR**: 20,000 (customer paid 20,000 Indonesian Rupiah)
- **CHANGE DUE**: 3,000

## Physical Description
- The receipt is printed on white thermal paper
- Some information in the middle section and toward the bottom is intentionally blurred/obscured
- The paper appears slightly curved/wrinkled and is placed on a dark brown surface (likely a table or counter)

The transaction is in **Indonesian Rupiah (IDR)**, indicating this purchase was made at an Auntie Anne's location in Indonesia. The customer bought one Cinnamon Sugar pretzel for 17,000 IDR and received 3,000 IDR in change after paying with 20,000 IDR cash.

4.2.2 Reasoning Output

Kimi-K2.7-Code는 thinking 모드와 preserve-thinking 동작을 강제해요.

Thinking Mode (기본값) — 추론 콘텐츠가 자동으로 분리돼요:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

response = client.chat.completions.create(
    model="moonshotai/Kimi-K2.7-Code",
    messages=[
        {"role": "user", "content": "Which one is bigger, 9.11 or 9.9? Think carefully."}
    ]
)

print("====== Reasoning Content (Thinking Mode) ======")
print(response.choices[0].message.reasoning_content)
print("====== Response (Thinking Mode) ======")
print(response.choices[0].message.content)

출력 예시:

====== Reasoning Content (Thinking Mode) ======
The user is asking which number is bigger: 9.11 or 9.9. This seems straightforward, but there's a viral internet debate about this due to decimal confusion.

Let me think carefully:
- 9.11 means 9 + 11/100 = 9.11
- 9.9 means 9 + 9/10 = 9.90

So 9.9 = 9.90, and 9.90 > 9.11 because 0.90 > 0.11.

The confusion often comes from people thinking of software versioning (where 9.11 comes after 9.9) or comparing the numbers after the decimal as whole numbers (11 vs 9, thinking 11 > 9).

So mathematically, 9.9 is clearly bigger. 9.9 - 9.11 = 0.79.

I should explain this clearly and address the common misconception.
====== Response (Thinking Mode) ======
Mathematically, **9.9 is bigger**.

Here's why:

**9.9 = 9.90**

When comparing decimals, you need to look at the same place values:
- 9.11 = 9 ones, 1 tenth, and 1 hundredth
- 9.9 = 9 ones, 9 tenths, and 0 hundredths (9.90)

Since **0.90 > 0.11**, it follows that **9.9 > 9.11**.

The difference is:
9.9 - 9.11 = 0.79

**Why people get confused:** Many mistakenly treat the decimals like whole numbers (thinking "11 is bigger than 9") or confuse this with software version numbering (where version 9.11 comes after version 9.9). But in standard mathematics, 9.9 is definitively larger.

4.2.3 Preserve Thinking

Kimi-K2.7-Code는 다중 턴 상호작용에서도 추론 콘텐츠를 유지해요. 이 동작은 기본적으로 활성화되며 비활성화할 수 없어요.

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

messages = [
    {
        "role": "user",
        "content": "Tell me three random numbers."
    },
    {
        "role": "assistant",
        "reasoning_content": "I'll start by listing five numbers: 473, 921, 235, 215, 222, and I'll tell you the first three.",
        "content": "473, 921, 235"
    },
    {
        "role": "user",
        "content": "What are the other two numbers you have in mind?"
    }
]

response = client.chat.completions.create(
    model="moonshotai/Kimi-K2.7-Code",
    messages=messages,
    stream=False,
    max_tokens=4096,
)

print(response.choices[0].message.content)

일부 OpenAI 호환 배포는 어시스턴트 메시지에서 reasoning_content 대신 reasoning을 사용해요. 서빙 스택이 노출하는 필드를 사용해요.

4.2.4 Tool Calling

Kimi-K2.7-Code는 에이전트 작업을 위한 툴 호출 기능을 지원해요.

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

# Define available tools
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city name"
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "Temperature unit"
                    }
                },
                "required": ["location"]
            }
        }
    }
]

response = client.chat.completions.create(
    model="moonshotai/Kimi-K2.7-Code",
    messages=[
        {"role": "user", "content": "What's the weather in Beijing?"}
    ],
    tools=tools,
    stream=True
)

# Process streaming response
tool_calls_accumulator = {}

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        if hasattr(delta, 'tool_calls') and delta.tool_calls:
            for tool_call in delta.tool_calls:
                index = tool_call.index
                if index not in tool_calls_accumulator:
                    tool_calls_accumulator[index] = {'name': None, 'arguments': ''}
                if tool_call.function:
                    if tool_call.function.name:
                        tool_calls_accumulator[index]['name'] = tool_call.function.name
                    if tool_call.function.arguments:
                        tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments

        if delta.content:
            print(delta.content, end="", flush=True)

for index, tool_call in sorted(tool_calls_accumulator.items()):
    print(f"Tool Call: {tool_call['name']}")
    print(f"  Arguments: {tool_call['arguments']}")

출력 예시:

Tool Call: get_weather
  Arguments: {"location": "Beijing"}

툴 호출 결과 처리:

# Send tool result back to the model
messages = [
    {"role": "user", "content": "What's the weather in Beijing?"},
    {
        "role": "assistant",
        "content": None,
        "tool_calls": [{
            "id": "call_123",
            "type": "function",
            "function": {
                "name": "get_weather",
                "arguments": '{"location": "Beijing", "unit": "celsius"}'
            }
        }]
    },
    {
        "role": "tool",
        "tool_call_id": "call_123",
        "content": "The weather in Beijing is 22°C and sunny."
    }
]

final_response = client.chat.completions.create(
    model="moonshotai/Kimi-K2.7-Code",
    messages=messages
)

print(final_response.choices[0].message.content)

출력 예시:

The weather in Beijing is currently **22°C and sunny**. ☀️

It's a nice, warm day there—great for being outdoors!

4.2.5 Multimodal + Tool Calling (Agentic Vision)

비전 이해와 툴 호출을 결합해 고급 에이전트 작업을 수행해요.

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

tools = [
    {
        "type": "function",
        "function": {
            "name": "search_product",
            "description": "Search for a product by name or description",
            "parameters": {
                "type": "object",
                "properties": {
                    "query": {
                        "type": "string",
                        "description": "The product name or description to search for"
                    }
                },
                "required": ["query"]
            }
        }
    }
]

response = client.chat.completions.create(
    model="moonshotai/Kimi-K2.7-Code",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
                    }
                },
                {
                    "type": "text",
                    "text": "Can you identify this product and search for similar items?"
                }
            ]
        }
    ],
    tools=tools
)

msg = response.choices[0].message

# Print reasoning process
if msg.reasoning_content:
    print("=== Reasoning ===")
    print(msg.reasoning_content)

# Print response content
if msg.content:
    print("=== Content ===")
    print(msg.content)

# Print tool calls
if msg.tool_calls:
    print("=== Tool Calls ===")
    for tc in msg.tool_calls:
        print(f"  Function: {tc.function.name}")
        print(f"  Arguments: {tc.function.arguments}")

출력 예시:

=== Reasoning ===
The user wants me to identify the product from the receipt and search for similar items. Looking at the receipt, it's from Auntie Anne's and the item purchased is "CINNAMON SUGAR" for 17,000 IDR. This is likely a Cinnamon Sugar Pretzel from Auntie Anne's, which is a popular pretzel chain.

I should search for this product using the search_product function. The query should be something like "Auntie Anne's Cinnamon Sugar Pretzel" or just "Cinnamon Sugar Pretzel" to find similar items.
=== Content ===
Based on the receipt, the product is a **Cinnamon Sugar Pretzel** from **Auntie Anne's** (a popular pretzel bakery chain). The receipt shows it was purchased for 17,000 Indonesian Rupiah (IDR).

Let me search for this product and similar items for you.
=== Tool Calls ===
  Function: search_product
  Arguments: {"query":"Auntie Anne's Cinnamon Sugar Pretzel"}

4.2.6 Deployment Command Example

다음 명령으로 Kimi-K2.7-Code를 배포해요 (H200/B300, reasoning 및 tool parsing 활성화):

sglang serve \
  --model-path moonshotai/Kimi-K2.7-Code \
  --tp 8 \
  --reasoning-parser kimi_k2 \
  --tool-call-parser kimi_k2 \
  --trust-remote-code \
  --host 0.0.0.0 \
  --port 30000

GB300에서는 --tp 4를 사용해요.

AMD MI350X/MI355X에서 MXFP4 체크포인트를 배포해요 (추론·툴 호출 파서는 체크포인트에서 자동 감지):

SGLANG_USE_AITER=1 \
HIP_FORCE_DEV_KERNARG=1 \
SGLANG_EXPERT_PARALLEL_SIZE=1 \
SGLANG_USE_DYNAMIC_MXFP4_LINEAR=0 \
TORCH_BLAS_PREFER_HIPBLASLT=1 \
TENSILE_STREAMK_DYNAMIC_GRID=6 \
AITER_QUICK_REDUCE_QUANTIZATION=INT4 \
AITER_USE_FLYDSL_MOE_SORTING=1 \
AITER_AR_1STAGE_MAX_KB=512 \
AITER_MXFP4_INTERMEDIATE=1 \
ROCM_QUICK_REDUCE_QUANTIZATION=INT4 \
sglang serve \
  --model-path amd/Kimi-K2.7-Code-MXFP4 \
  --tp 4 \
  --trust-remote-code \
  --attention-backend aiter \
  --mem-fraction-static 0.90 \
  --kv-cache-dtype fp8_e4m3 \
  --disable-radix-cache \
  --enable-aiter-allreduce-fusion \
  --host 0.0.0.0 \
  --port 30000

5. Benchmark (벤치마크)

5.1 Model Card Benchmarks

다음 결과는 공식 Kimi-K2.7-Code 모델 카드에서 가져온 것이에요. 별도로 명시되지 않는 한 Kimi Code CLI를 통해 thinking 모드를 활성화하고 temperature=1.0, top_p=0.95, 262,144-token 컨텍스트 길이로 평가되었어요.

5.2 MXFP4 Benchmark

테스트 환경:

• Hardware: 4× AMD MI355X
• Model: amd/Kimi-K2.7-Code-MXFP4 (MXFP4)
• Tensor Parallelism: 4
• Docker Image: rocm/sgl-dev:v0.5.17-rocm720-mi35x-20260812 (ROCm 7.2)

5.2.1 Accuracy Benchmark

여러 클라이언트 병렬 수준(각각 2,000문제)에서 배포된 서버로 측정한 MXFP4 체크포인트의 GSM8K 정확도:

python3 benchmark/gsm8k/bench_sglang.py \
  --num-questions 2000 \
  --parallel <PARALLEL> \
  --port 30000

5.2.2 Latency Benchmark

SGLang 내장 bench_serving 도구를 random 데이터셋(input 8192, output 1024, range ratio 0.8)과 함께 사용해요. 각 동시성 C에 대해 --num-prompts는 C × 5로 설정돼요.

python3 -m sglang.bench_serving \
  --backend sglang \
  --model amd/Kimi-K2.7-Code-MXFP4 \
  --dataset-name random \
  --random-input-len 8192 \
  --random-output-len 1024 \
  --random-range-ratio 0.8 \
  --max-concurrency <C> \
  --num-prompts <C x 5> \
  --host 0.0.0.0 \
  --port 30000

더 알아보기 (Learn more)

• Kimi-K2.7-Code (Hugging Face) — 공식 모델 카드
• Kimi-K2.7-Code-MXFP4 (Hugging Face) — AMD MXFP4 체크포인트
• SGLang 설치 가이드 — SGLang 설치 방법