Bedrock

Bedrock (boto3) SDK

Bedrock의 패스스루 엔드포인트를 소개할게요. Bedrock의 프로바이더 고유 엔드포인트를 네이티브 형식 그대로(변환 없이) 호출할 수 있는 기능이에요. /invoke, /converse 같은 Bedrock 네이티브 엔드포인트를 그대로 사용할 수 있어요.

출처: 문서

본문

연결 대상 호스트는 다음과 같아요.

https://bedrock-runtime.{aws_region_name}.amazonaws.com

프록시를 통한 주소는 이렇게 구성돼요.

LITELLM_PROXY_BASE_URL/bedrock

개요 (Overview)

1. config.yaml 사용하기 (모델 엔드포인트 권장)

모델은 config.yaml에 등록하고 /converse, /converse-stream, /invoke, /invoke-with-response-stream 같은 엔드포인트를 호출하는 방식이에요.

model_list:
  - model_name: my-bedrock-model
    litellm_params:
      model: bedrock/us.anthropic.claude-sonnet-5
      aws_region_name: us-west-2
      custom_llm_provider: bedrock

curl -X POST 'http://0.0.0.0:4000/bedrock/model/my-bedrock-model/converse' \
-H "Authorization: Bearer ***" \
-H 'Content-Type: application/json' \
-d '{"messages": [{"role": "user", "content": [{"text": "Hello"}]}]}'

2. 비모델 엔드포인트를 위한 직접 패스스루

가드레일(guardrail), 지식 베이스 등 모델이 아닌 엔드포인트는 AWS 자격 증명 환경 변수를 설정한 뒤 직접 호출할 수 있어요.

export AWS_ACCESS_KEY_ID=""
export AWS_SECRET_ACCESS_KEY=""
export AWS_REGION_NAME="us-west-2"

curl "http://0.0.0.0:4000/bedrock/guardrail/my-guardrail-id/version/1/apply" \
-H "Authorization: Bearer ***" \
-H 'Content-Type: application/json' \
-d '{"contents": [{"text": {"text": "Hello"}}], "source": "INPUT"}'

빠른 시작 (Quick Start)

/converse 엔드포인트를 호출하는 예시예요. 먼저 config.yaml에 모델을 등록해요.

model_list:
  - model_name: my-bedrock-model
    litellm_params:
      model: bedrock/us.anthropic.claude-sonnet-5
      aws_region_name: us-west-2
      custom_llm_provider: bedrock

AWS 자격 증명을 설정해요.

export AWS_ACCESS_KEY_ID=""  # Access key
export AWS_SECRET_ACCESS_KEY="" # Secret access key

그다음 LiteLLM 프록시를 실행해요.

litellm --config config.yaml
# RUNNING on http://0.0.0.0:4000

이제 /converse 엔드포인트를 호출해요.

curl -X POST 'http://0.0.0.0:4000/bedrock/model/my-bedrock-model/converse' \
-H "Authorization: Bearer ***" \
-H 'Content-Type: application/json' \
-d '{
    "messages": [
        {
            "role": "user",
            "content": [{"text": "Hello, how are you?"}]
        }
    ],
    "inferenceConfig": {
        "maxTokens": 100
    }
}'

config.yaml로 설정하기 (Setup with config.yaml)

1. config.yaml에 모델 정의하기

model_list:
  - model_name: my-claude-model
    litellm_params:
      model: bedrock/us.anthropic.claude-sonnet-5
      aws_region_name: us-west-2
      custom_llm_provider: bedrock
  - model_name: my-cohere-model
    litellm_params:
      model: bedrock/cohere.command-r-v1:0
      aws_region_name: us-east-1
      custom_llm_provider: bedrock

2. 설정과 함께 프록시 시작하기

litellm --config config.yaml
# RUNNING on http://0.0.0.0:4000

3. Bedrock converse 엔드포인트 호출하기

URL의 model_name에는 config.yaml의 model_name을 사용해요.

curl -X POST 'http://0.0.0.0:4000/bedrock/model/my-claude-model/converse' \
-H "Authorization: Bearer ***" \
-H 'Content-Type: application/json' \
-d '{
    "messages": [
        {
            "role": "user",
            "content": [{"text": "Hello, how are you?"}]
        }
    ],
    "inferenceConfig": {
        "temperature": 0.5,
        "maxTokens": 100
    }
}'

4. Bedrock converse-stream 엔드포인트 호출하기

curl -X POST 'http://0.0.0.0:4000/bedrock/model/my-claude-model/converse-stream' \
-H "Authorization: Bearer ***" \
-H 'Content-Type: application/json' \
-d '{
    "messages": [
        {
            "role": "user",
            "content": [{"text": "Tell me a short story"}]
        }
    ],
    "inferenceConfig": {
        "temperature": 0.7,
        "maxTokens": 200
    }
}'

config.yaml에서 지원되는 Bedrock 엔드포인트

/model/{model_name}/converse

http://0.0.0.0:4000/bedrock/model/my-claude-model/converse

/model/{model_name}/converse-stream

http://0.0.0.0:4000/bedrock/model/my-claude-model/converse-stream

/model/{model_name}/invoke

http://0.0.0.0:4000/bedrock/model/my-claude-model/invoke

/model/{model_name}/invoke-with-response-stream

http://0.0.0.0:4000/bedrock/model/my-claude-model/invoke-with-response-stream

model_name은 config.yaml의 model_name 필드를 사용해요.

여러 배포 간 로드 밸런싱 (Load Balancing across Multiple Deployments)

1. config.yaml에 여러 배포 정의하기

model_list:
  # First deployment - us-west-2
  - model_name: my-claude-model
    litellm_params:
      model: bedrock/us.anthropic.claude-sonnet-5
      aws_region_name: us-west-2
      custom_llm_provider: bedrock
  # Second deployment - us-east-1 (load balanced)
  - model_name: my-claude-model
    litellm_params:
      model: bedrock/us.anthropic.claude-sonnet-5
      aws_region_name: us-east-1
      custom_llm_provider: bedrock

2. 설정과 함께 프록시 시작하기

litellm --config config.yaml
# RUNNING on http://0.0.0.0:4000

3. 엔드포인트 호출 — 요청이 자동으로 로드 밸런싱됨

curl -X POST 'http://0.0.0.0:4000/bedrock/model/my-claude-model/invoke' \
-H "Authorization: Bearer ***" \
-H 'Content-Type: application/json' \
-d '{
    "max_tokens": 100,
    "messages": [
        {
            "role": "user",
            "content": "Hello, how are you?"
        }
    ],
    "anthropic_version": "bedrock-2023-05-31"
}'

이렇게 하면 us-west-2와 us-east-1 배포 간에 /invoke, /invoke-with-response-stream, /converse, /converse-stream 모두 자동으로 로드 밸런싱돼요.

boto3 SDK로 로드 밸런싱 사용하기

import boto3
import json
import os
# Set dummy AWS credentials (required by boto3, but not used by LiteLLM proxy)
os.environ['AWS_ACCESS_KEY_ID'] = 'dummy'
os.environ['AWS_SECRET_ACCESS_KEY'] = 'dummy'
os.environ['AWS_BEARER_TOKEN_BEDROCK'] = "sk-<your-litellm-api-key>"  # your litellm proxy api key
# Point boto3 to the LiteLLM proxy
bedrock_runtime = boto3.client(
    service_name='bedrock-runtime',
    region_name='us-west-2',
    endpoint_url='http://0.0.0.0:4000/bedrock')
# Call the load-balanced model
response = bedrock_runtime.invoke_model(
    modelId='my-claude-model',  # Your model_name from config.yaml
    contentType='application/json',
    accept='application/json',
    body=json.dumps({
        "max_tokens": 100,
        "messages": [
            {
                "role": "user",
                "content": "Hello, how are you?"
            }
        ],
        "anthropic_version": "bedrock-2023-05-31"
    }))
# Parse response
response_body = json.loads(response['body'].read())
print(response_body['content'][0]['text'])

예시 (Examples)

핵심 아이디어는 단순해요. Bedrock API의 호스트를 http://0.0.0.0:4000/bedrock로 바꾸면 돼요. 원래 https://bedrock-runtime.{aws_region_name}.amazonaws.com으로 인증(AWS4-HMAC-SHA256) 하던 것을 LiteLLM 키(Bearer anything 또는 가상 키 Bearer LITELLM_VIRTUAL_KEY)로 바꾸면 됩니다.

예시 1: converse API

LiteLLM 프록시 호출

curl -X POST 'http://0.0.0.0:4000/bedrock/model/cohere.command-r-v1:0/converse' \
-H 'Authorization: Bearer ***' \
-H 'Content-Type: application/json' \
-d '{
    "messages": [
        {"role": "user",
        "content": [{"text": "Hello"}]
    }
    ]
}'

Bedrock 직접 API 호출 (변환 전)

curl -X POST 'https://bedrock-runtime.us-west-2.amazonaws.com/model/cohere.command-r-v1:0/converse' \
-H 'Authorization: AWS4-H...56..' \
-H 'Content-Type: application/json' \
-d '{
    "messages": [
        {"role": "user",
        "content": [{"text": "Hello"}]
    }
    ]
}'

예시 2: 가드레일 적용 (Apply Guardrail)

AWS 자격 증명을 설정하고 프록시를 실행해요.

export AWS_ACCESS_KEY_ID="your-access-key"
export AWS_SECRET_ACCESS_KEY="your-secret-key"
export AWS_REGION_NAME="us-west-2"

litellm
# RUNNING on http://0.0.0.0:4000

LiteLLM 프록시 호출

curl "http://0.0.0.0:4000/bedrock/guardrail/guardrailIdentifier/version/guardrailVersion/apply" \
    -H 'Authorization: Bearer ***' \
    -H 'Content-Type: application/json' \
    -X POST \
    -d '{
      "contents": [{"text": {"text": "Hello world"}}],
      "source": "INPUT" 
       }'

Bedrock 직접 API 호출 (변환 전)

curl "https://bedrock-runtime.us-west-2.amazonaws.com/guardrail/guardrailIdentifier/version/guardrailVersion/apply" \
    -H 'Authorization: AWS4-H...56..' \
    -H 'Content-Type: application/json' \
    -X POST \
    -d '{
      "contents": [{"text": {"text": "Hello world"}}],
      "source": "INPUT" 
       }'

예시 3: 지식 베이스 쿼리 (Query Knowledge Base)

LiteLLM 프록시 호출

curl -X POST "http://0.0.0.0:4000/bedrock/knowledgebases/{knowledgeBaseId}/retrieve" \
-H 'Authorization: Bearer ***' \
-H 'Content-Type: application/json' \
-d '{
    "nextToken": "string",
    "retrievalConfiguration": {
        "vectorSearchConfiguration": {
          "filter": { ... },
          "numberOfResults": number,
          "overrideSearchType": "string"
        }
    },
    "retrievalQuery": {
        "text": "string"
    }
}'

Bedrock 직접 API 호출 (변환 전)

curl -X POST "https://bedrock-agent-runtime.us-west-2.amazonaws.com/knowledgebases/{knowledgeBaseId}/retrieve" \
-H 'Authorization: AWS4-H...56..' \
-H 'Content-Type: application/json' \
-d '{
    "nextToken": "string",
    "retrievalConfiguration": {
        "vectorSearchConfiguration": {
          "filter": { ... },
          "numberOfResults": number,
          "overrideSearchType": "string"
        }
    },
    "retrievalQuery": {
        "text": "string"
    }
}'

고급: 가상 키(Virtual Keys)와 함께 사용하기

가상 키는 LiteLLM 프록시에 데이터베이스가 설정된 경우에 사용할 수 있어요. 가상 키 설정 문서를 참고해 주세요.

환경 변수를 설정해요.

export DATABASE_URL=""
export LITELLM_MASTER_KEY=""
export AWS_ACCESS_KEY_ID=""  # Access key
export AWS_SECRET_ACCESS_KEY="" # Secret access key
export AWS_REGION_NAME="" # us-east-1, us-east-2, us-west-1, us-west-2

프록시를 실행해요.

litellm
# RUNNING on http://0.0.0.0:4000

가상 키를 생성해요.

curl -X POST 'http://0.0.0.0:4000/key/generate' \
-H "Authorization: Bearer ***" \
-H 'Content-Type: application/json' \
-d '{}'

응답에서 키를 받아요.

{
    ...
    "key": "sk-<virtual-key>"
}

이제 converse 엔드포인트를 가상 키로 호출해요.

curl -X POST 'http://0.0.0.0:4000/bedrock/model/cohere.command-r-v1:0/converse' \
-H "Authorization: Bearer ***" \
-H 'Content-Type: application/json' \
-d '{
    "messages": [
        {"role": "user",
        "content": [{"text": "Hello"}]
    }
    ]
}'

고급: Bedrock Agents

AWS 자격 증명을 설정하고 프록시를 실행한 뒤,

export AWS_ACCESS_KEY_ID="your-access-key"
export AWS_SECRET_ACCESS_KEY="your-secret-key"
export AWS_REGION_NAME="us-west-2"

litellm
# RUNNING on http://0.0.0.0:4000

boto3로 Bedrock Agent를 호출할 수 있어요. AWS 키는 더미 값을 쓰고, LiteLLM 키는 AWS_BEARER_TOKEN_BEDROCK 환경 변수로 전달해요.

import os 
import boto3
# Set dummy AWS credentials (required by boto3, but not used by LiteLLM proxy)
os.environ["AWS_ACCESS_KEY_ID"] = "dummy"
os.environ["AWS_SECRET_ACCESS_KEY"] = "dummy"
os.environ["AWS_BEARER_TOKEN_BEDROCK"] = "sk-<your-litellm-api-key>"  # your litellm proxy api key
# Create the client
runtime_client = boto3.client(
    service_name="bedrock-agent-runtime", 
    region_name="us-west-2", 
    endpoint_url="http://0.0.0.0:4000/bedrock")
response = runtime_client.invoke_agent(
    agentId="L1RT58GYRW",
    agentAliasId="MFPSBCXYTW",
    sessionId="12345",
    inputText="Who do you know?")
completion = ""
for event in response.get("completion"):
    chunk = event["chunk"]
    completion += chunk["bytes"].decode()
print(completion)

LangChain AWS SDK와 함께 사용하기

빠른 시작 (Quick Start)

langchain-aws 패키지를 설치해요.

uv add langchain-aws

config.yaml에 모델을 등록해요.

model_list:
  - model_name: claude-sonnet
    litellm_params:
      model: bedrock/us.anthropic.claude-sonnet-5
      aws_region_name: us-east-1
      custom_llm_provider: bedrock

export AWS_ACCESS_KEY_ID="your-access-key"
export AWS_SECRET_ACCESS_KEY="your-secret-key"
litellm --config config.yaml
# RUNNING on http://0.0.0.0:4000

ChatBedrockConverse를 프록시로 연결하는 예시예요. aws_access_key_id에는 "Bearer sk-<your-litellm-api-key>"를 넣어요.

from langchain_aws import ChatBedrockConverse
from langchain_core.messages import HumanMessage
# Your LiteLLM API key
API_KEY = "Bearer sk-<your-litellm-api-key>"
# Initialize ChatBedrockConverse pointing to LiteLLM proxy
llm = ChatBedrockConverse(
    model_id="us.anthropic.claude-sonnet-5",
    endpoint_url="http://localhost:4000/bedrock",
    region_name="us-east-1",
    aws_access_key_id=API_KEY,
    aws_secret_access_key="bedrock"  # Any non-empty value works
)
# Invoke the model
messages = [HumanMessage(content="Hello, how are you?")]
response = llm.invoke(messages)
print(response.content)

고급 예시: 인용(citations)이 있는 PDF 문서 처리

import os
import json
from langchain_aws import ChatBedrockConverse
from langchain_core.messages import HumanMessage
# Your LiteLLM API key
API_KEY = "Bearer sk-<your-litellm-api-key>"
def get_llm() -> ChatBedrockConverse:
    """Initialize LLM pointing to LiteLLM proxy"""
    llm = ChatBedrockConverse(
        model_id="us.anthropic.claude-sonnet-5",
        base_model_id="anthropic.claude-sonnet-5",
        endpoint_url="http://localhost:4000/bedrock",
        region_name="us-east-1",
        aws_access_key_id=API_KEY,
        aws_secret_access_key="bedrock"
    )
    return llm
if __name__ == "__main__":
    # Initialize the LLM
    llm = get_llm()
    
    # Read PDF file as bytes (Converse API requires raw bytes)
    with open("your-document.pdf", "rb") as file:
        file_bytes = file.read()
        # Prepare messages with document attachment
        messages = [
            HumanMessage(content=[
                {"text": "What is the policy number in this document?"},
                {
                    "document": {
                        "format": "pdf",
                        "name": "PolicyDocument",
                        "source": {"bytes": file_bytes},
                        "citations": {"enabled": True}
                    }
                }
            ])
    ]
    
    # Invoke the LLM
    response = llm.invoke(messages)
    
    # Print response with citations
    print(json.dumps(response.content, indent=4))

지원되는 LangChain 기능

스트리밍은 stream() 메서드로 지원돼요.

문제 해결 (Troubleshooting)

UnknownOperationException 오류가 나면, 베이스 URL에서 /v2 경로가 제외되었는지 확인해요. 최신 버전의 Litellm은 /converse, /invoke 같은 경로를 사용하기 때문에 http://localhost:4000/bedrock(즉 /v2 없이)을 사용해야 해요. 또한 aws_access_key_id="Bearer sk-<your-litellm-api-key>"처럼 Bearer 접두사를 반드시 포함해야 해요.

더 알아보기 (Learn more)

• Bedrock Converse API 참조
• 가상 키 설정