Bedrock Imported Models

Bedrock Imported Models

Bedrock Imported Models(Deepseek, Deepseek R1, Qwen, OpenAI 호환 모델)을 LiteLLM으로 호출해요.

출처: 문서

본문

Deepseek R1

chat 템플릿이 달라 별도의 라우트예요.

속성	설명
공급자 라우트	bedrock/deepseek_r1/{model_arn}
공급자 문서	Bedrock Imported Models, Deepseek Bedrock Imported Model

SDK:

from litellm import completion
import os

response = completion(
    model="bedrock/deepseek_r1/arn:aws:bedrock:us-east-1:086734376398:imported-model/r4c4kewx2s0n",  # bedrock/deepseek_r1/{your-model-arn}
    messages=[{"role": "user", "content": "Tell me a joke"}],
)

Proxy:

model_list:
  - model_name: DeepSeek-R1-Distill-Llama-70B
    litellm_params:
      model: bedrock/deepseek_r1/arn:aws:bedrock:us-east-1:086734376398:imported-model/r4c4kewx2s0n

litellm --config /path/to/config.yaml
# RUNNING at http://0.0.0.0:4000

curl --location 'http://0.0.0.0:4000/chat/completions' \
  --header "Authorization: Bearer ***" \
  --header 'Content-Type: application/json' \
  --data '{
    "model": "DeepSeek-R1-Distill-Llama-70B",  # 👈 the 'model_name' in config
    "messages": [
      {
        "role": "user",
        "content": "what llm are you"
      }
    ],
  }'

Deepseek (R1 아님)

llama Invoke Request / Response spec을 따르는 Bedrock Imported Models를 호출하는 데 쓰는 라우트예요.

속성	설명
공급자 라우트	bedrock/llama/{model_arn}
공급자 문서	Bedrock Imported Models, Deepseek Bedrock Imported Model

SDK:

from litellm import completion
import os

response = completion(
    model="bedrock/llama/arn:aws:bedrock:us-east-1:086734376398:imported-model/r4c4kewx2s0n",  # bedrock/llama/{your-model-arn}
    messages=[{"role": "user", "content": "Tell me a joke"}],
)

Proxy:

model_list:
  - model_name: DeepSeek-R1-Distill-Llama-70B
    litellm_params:
      model: bedrock/llama/arn:aws:bedrock:us-east-1:086734376398:imported-model/r4c4kewx2s0n

litellm --config /path/to/config.yaml
# RUNNING at http://0.0.0.0:4000

curl --location 'http://0.0.0.0:4000/chat/completions' \
  --header "Authorization: Bearer ***" \
  --header 'Content-Type: application/json' \
  --data '{
    "model": "DeepSeek-R1-Distill-Llama-70B",  # 👈 the 'model_name' in config
    "messages": [
      {
        "role": "user",
        "content": "what llm are you"
      }
    ],
  }'

Qwen3 Imported Models

속성	설명
공급자 라우트	bedrock/qwen3/{model_arn}
공급자 문서	Bedrock Imported Models, Qwen3 Models

SDK:

from litellm import completion
import os

response = completion(
    model="bedrock/qwen3/arn:aws:bedrock:us-east-1:086734376398:imported-model/your-qwen3-model",  # bedrock/qwen3/{your-model-arn}
    messages=[{"role": "user", "content": "Tell me a joke"}],
    max_tokens=100,
    temperature=0.7,
)

Proxy:

model_list:
  - model_name: Qwen3-32B
    litellm_params:
      model: bedrock/qwen3/arn:aws:bedrock:us-east-1:086734376398:imported-model/your-qwen3-model

Qwen2 Imported Models

속성	설명
공급자 라우트	bedrock/qwen2/{model_arn}
공급자 문서	Bedrock Imported Models

Qwen2와 Qwen3 아키텍처는 대체로 비슷해요. 주요 차이는 응답 형식: Qwen2는 text 필드를, Qwen3는 generation 필드를 사용해요.

SDK:

from litellm import completion
import os

response = completion(
    model="bedrock/qwen2/arn:aws:bedrock:us-east-1:086734376398:imported-model/your-qwen2-model",  # bedrock/qwen2/{your-model-arn}
    messages=[{"role": "user", "content": "Tell me a joke"}],
    max_tokens=100,
    temperature=0.7,
)

Proxy:

model_list:
  - model_name: Qwen2-72B
    litellm_params:
      model: bedrock/qwen2/arn:aws:bedrock:us-east-1:086734376398:imported-model/your-qwen2-model

OpenAI 호환 Imported Models (Qwen 2.5 VL 등)

OpenAI Chat Completions API spec을 따르는 Bedrock imported models에 쓰는 라우트예요. Qwen 2.5 VL처럼 OpenAI 형식 메시지를 받고 비전(이미지), tool calling 등 OpenAI 기능을 지원하는 모델이 포함돼요.

속성	설명
공급자 라우트	bedrock/openai/{model_arn}
공급자 문서	Bedrock Imported Models
지원 기능	비전(이미지), tool calling, 스트리밍, system messages

기본 사용법 (LiteLLM SDK)

from litellm import completion

response = completion(
    model="bedrock/openai/arn:aws:bedrock:us-east-1:046319184608:imported-model/0m2lasirsp6z",  # bedrock/openai/{your-model-arn}
    messages=[{"role": "user", "content": "Tell me a joke"}],
    max_tokens=300,
    temperature=0.5,
)

비전 (이미지) 사용

import base64
from litellm import completion

# Load and encode image
with open("image.jpg", "rb") as f:
    image_base64 = base64.b64encode(f.read()).decode("utf-8")

response = completion(
    model="bedrock/openai/arn:aws:bedrock:us-east-1:046319184608:imported-model/0m2lasirsp6z",
    messages=[
        {
            "role": "system",
            "content": "You are a helpful assistant that can analyze images."
        },
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What's in this image?"},
                {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image_base64}"}}
            ]
        }
    ],
    max_tokens=300,
    temperature=0.5,
)

여러 이미지 비교

import base64
from litellm import completion

# Load images
with open("image1.jpg", "rb") as f:
    image1_base64 = base64.b64encode(f.read()).decode("utf-8")
with open("image2.jpg", "rb") as f:
    image2_base64 = base64.b64encode(f.read()).decode("utf-8")

response = completion(
    model="bedrock/openai/arn:aws:bedrock:us-east-1:046319184608:imported-model/0m2lasirsp6z",
    messages=[
        {
            "role": "system",
            "content": "You are a helpful assistant that can analyze images."
        },
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Spot the difference between these two images?"},
                {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image1_base64}"}},
                {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image2_base64}"}}
            ]
        }
    ],
    max_tokens=300,
    temperature=0.5,
)

LiteLLM Proxy 사용

1. config에 추가:

model_list:
  - model_name: qwen-25vl-72b
    litellm_params:
      model: bedrock/openai/arn:aws:bedrock:us-east-1:046319184608:imported-model/0m2lasirsp6z

2. Proxy 시작:

litellm --config /path/to/config.yaml
# RUNNING at http://0.0.0.0:4000

3. 테스트:

기본 텍스트 요청:

curl --location 'http://0.0.0.0:4000/chat/completions' \
  --header "Authorization: Bearer ***" \
  --header 'Content-Type: application/json' \
  --data '{
    "model": "qwen-25vl-72b",
    "messages": [
      {
        "role": "user",
        "content": "what llm are you"
      }
    ],
    "max_tokens": 300
  }'

비전(이미지) 포함:

curl --location 'http://0.0.0.0:4000/chat/completions' \
  --header "Authorization: Bearer ***" \
  --header 'Content-Type: application/json' \
  --data '{
    "model": "qwen-25vl-72b",
    "messages": [
      {
        "role": "system",
        "content": "You are a helpful assistant that can analyze images."
      },
      {
        "role": "user",
        "content": [
          {"type": "text", "text": "What is in this image?"},
          {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,/9j/4AAQSkZ..."}}
        ]
      }
    ],
    "max_tokens": 300,
    "temperature": 0.5
  }'

Moonshot Kimi K2 Thinking

Moonshot AI의 Kimi K2 Thinking 모델이 이제 Amazon Bedrock에서 사용 가능해요. 이 모델은 자동 reasoning content 추출이 포함된 고급 추론 기능을 제공해요.

속성	설명
공급자 라우트	bedrock/moonshot.kimi-k2-thinking, bedrock/invoke/moonshot.kimi-k2-thinking
공급자 문서	AWS Bedrock Moonshot Announcement
지원 파라미터	temperature, max_tokens, top_p, stream, tools, tool_choice
특수 기능	Reasoning content 추출, Tool calling

지원 기능

• Reasoning Content 추출: <reasoning> 태그를 자동 추출해 reasoning_content로 반환 (OpenAI o1 모델과 유사)
• Tool Calling: tool 응답이 포함된 function/tool calling 완전 지원
• 스트리밍: 스트리밍 및 비스트리밍 응답 모두 지원
• 시스템 메시지: 시스템 메시지 지원

기본 사용법

SDK:

from litellm import completion
import os

os.environ["AWS_ACCESS_KEY_ID"] = "your-aws-access-key"
os.environ["AWS_SECRET_ACCESS_KEY"] = "your-aws-secret-key"
os.environ["AWS_REGION_NAME"] = "us-west-2"  # or your preferred region

# Basic completion
response = completion(
    model="bedrock/moonshot.kimi-k2-thinking",  # or bedrock/invoke/moonshot.kimi-k2-thinking
    messages=[
        {"role": "user", "content": "What is 2+2? Think step by step."}
    ],
    temperature=0.7,
    max_tokens=200,
)
print(response.choices[0].message.content)

# Access reasoning content if present
if response.choices[0].message.reasoning_content:
    print("Reasoning:", response.choices[0].message.reasoning_content)

Proxy:

model_list:
  - model_name: kimi-k2
    litellm_params:
      model: bedrock/moonshot.kimi-k2-thinking
      aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
      aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
      aws_region_name: us-west-2

litellm --config /path/to/config.yaml
# RUNNING at http://0.0.0.0:4000

curl --location 'http://0.0.0.0:4000/chat/completions' \
  --header "Authorization: Bearer ***" \
  --header 'Content-Type: application/json' \
  --data '{
    "model": "kimi-k2",
    "messages": [
      {
        "role": "user",
        "content": "What is 2+2? Think step by step."
      }
    ],
    "temperature": 0.7,
    "max_tokens": 200
  }'

Tool Calling 예시

from litellm import completion
import os

os.environ["AWS_ACCESS_KEY_ID"] = "your-aws-access-key"
os.environ["AWS_SECRET_ACCESS_KEY"] = "your-aws-secret-key"
os.environ["AWS_REGION_NAME"] = "us-west-2"

# Tool calling example
response = completion(
    model="bedrock/moonshot.kimi-k2-thinking",
    messages=[
        {"role": "user", "content": "What's the weather in Tokyo?"}
    ],
    tools=[
        {
            "type": "function",
            "function": {
                "name": "get_weather",
                "description": "Get the current weather in a location",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "location": {
                            "type": "string",
                            "description": "The city name"
                        }
                    },
                    "required": ["location"]
                }
            }
        }
    ],
)

if response.choices[0].message.tool_calls:
    tool_call = response.choices[0].message.tool_calls[0]
    print(f"Tool called: {tool_call.function.name}")
    print(f"Arguments: {tool_call.function.arguments}")

스트리밍 예시

from litellm import completion
import os

os.environ["AWS_ACCESS_KEY_ID"] = "your-aws-access-key"
os.environ["AWS_SECRET_ACCESS_KEY"] = "your-aws-secret-key"
os.environ["AWS_REGION_NAME"] = "us-west-2"

response = completion(
    model="bedrock/moonshot.kimi-k2-thinking",
    messages=[
        {"role": "user", "content": "Explain quantum computing in simple terms."}
    ],
    stream=True,
    temperature=0.7,
)

for chunk in response:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")
        # Check for reasoning content in streaming
        if hasattr(chunk.choices[0].delta, 'reasoning_content') and chunk.choices[0].delta.reasoning_content:
            print(f"\n[Reasoning: {chunk.choices[0].delta.reasoning_content}]")

더 알아보기 (Learn more)

• LiteLLM 개요
• LiteLLM Python SDK 사용법
• LiteLLM Proxy Server 설정
• 지원되는 공급업체 전체 목록