Vercel AI Gateway

Vercel AI Gateway

Vercel AI Gateway를 LiteLLM에서 사용하는 방법을 알아봐요. 단일 엔드포인트로 여러 AI 프로바이더에 접근하며, 내장 캐싱·레이트 리밋·분석을 제공해요.

출처: 문서

본문

개요

Vercel AI Gateway를 통해 사용 가능한 모든 모델을 지원하며, completion 요청을 보낼 때 vercel_ai_gateway/를 접두사로 붙이면 돼요.

필수 변수

os.environ["VERCEL_AI_GATEWAY_API_KEY"] = ""  # your Vercel AI Gateway API key
# OR
os.environ["VERCEL_OIDC_TOKEN"] = ""  # your Vercel OIDC token for authentication

선택 변수

os.environ["VERCEL_SITE_URL"] = ""  # your site url
# OR
os.environ["VERCEL_APP_NAME"] = ""  # your app name

참고: 키 획득 방법은 Vercel AI Gateway 문서를 참고해요.

LiteLLM Python SDK 사용법

비스트리밍

import os
import litellm
from litellm import completion

os.environ["VERCEL_AI_GATEWAY_API_KEY"] = "your-api-key"

messages = [{"content": "Hello, how are you?", "role": "user"}]

# Vercel AI Gateway call
response = completion(
    model="vercel_ai_gateway/openai/gpt-5.6-terra",
    messages=messages
)

print(response)

스트리밍

import os
import litellm
from litellm import completion

os.environ["VERCEL_AI_GATEWAY_API_KEY"] = "your-api-key"

messages = [{"content": "Hello, how are you?", "role": "user"}]

# Vercel AI Gateway call with streaming
response = completion(
    model="vercel_ai_gateway/openai/gpt-5.6-terra",
    messages=messages,
    stream=True
)

for chunk in response:
    print(chunk)

임베딩

import os
from litellm import embedding

os.environ["VERCEL_AI_GATEWAY_API_KEY"] = "your-api-key"

# Vercel AI Gateway embedding call
response = embedding(
    model="vercel_ai_gateway/openai/text-embedding-3-small",
    input="Hello world"
)

print(response.data[0]["embedding"][:5])  # Print first 5 dimensions

dimensions 파라미터도 지정할 수 있어요:

response = embedding(
    model="vercel_ai_gateway/openai/text-embedding-3-small",
    input=["Hello world", "Goodbye world"],
    dimensions=768
)

LiteLLM Proxy 사용법

LiteLLM Proxy 설정 파일에 다음을 추가해요.

config.yaml:

model_list:
  - model_name: gpt-4o-gateway
    litellm_params:
      model: vercel_ai_gateway/openai/gpt-5.6-terra
      api_key: os.environ/VERCEL_AI_GATEWAY_API_KEY

  - model_name: claude-4-sonnet-gateway
    litellm_params:
      model: vercel_ai_gateway/anthropic/claude-4-sonnet
      api_key: os.environ/VERCEL_AI_GATEWAY_API_KEY

  - model_name: text-embedding-3-small-gateway
    litellm_params:
      model: vercel_ai_gateway/openai/text-embedding-3-small
      api_key: os.environ/VERCEL_AI_GATEWAY_API_KEY

Proxy 서버 시작:

litellm --config config.yaml

# RUNNING on http://0.0.0.0:4000

Proxy를 통한 Vercel AI Gateway - 비스트리밍 (OpenAI SDK):

from openai import OpenAI

# Initialize client with your proxy URL
client = OpenAI(
    base_url="http://localhost:4000",  # Your proxy URL
    api_key="your-proxy-api-key"       # Your proxy API key
)

# Non-streaming response
response = client.chat.completions.create(
    model="gpt-4o-gateway",
    messages=[{"role": "user", "content": "Hello, how are you?"}]
)

print(response.choices[0].message.content)

스트리밍:

from openai import OpenAI

# Initialize client with your proxy URL
client = OpenAI(
    base_url="http://localhost:4000",  # Your proxy URL
    api_key="your-proxy-api-key"       # Your proxy API key
)

# Streaming response
response = client.chat.completions.create(
    model="gpt-4o-gateway",
    messages=[{"role": "user", "content": "Hello, how are you?"}],
    stream=True
)

for chunk in response:
    if chunk.choices[0].delta.content is not None:
        print(chunk.choices[0].delta.content, end="")

LiteLLM SDK:

import litellm

# Configure LiteLLM to use your proxy
response = litellm.completion(
    model="litellm_proxy/gpt-4o-gateway",
    messages=[{"role": "user", "content": "Hello, how are you?"}],
    api_base="http://localhost:4000",
    api_key="your-proxy-api-key"
)

print(response.choices[0].message.content)

LiteLLM SDK 스트리밍:

import litellm

# Configure LiteLLM to use your proxy with streaming
response = litellm.completion(
    model="litellm_proxy/gpt-4o-gateway",
    messages=[{"role": "user", "content": "Hello, how are you?"}],
    api_base="http://localhost:4000",
    api_key="your-proxy-api-key",
    stream=True
)

for chunk in response:
    if hasattr(chunk.choices[0], 'delta') and chunk.choices[0].delta.content is not None:
        print(chunk.choices[0].delta.content, end="")

cURL:

curl http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your-p...-key" \
  -d '{
    "model": "gpt-4o-gateway",
    "messages": [{"role": "user", "content": "Hello, how are you?"}]
  }'

cURL 스트리밍:

curl http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your-p...-key" \
  -d '{
    "model": "gpt-4o-gateway",
    "messages": [{"role": "user", "content": "Hello, how are you?"}],
    "stream": true
  }'

LiteLLM Proxy 사용에 대한 자세한 내용은 LiteLLM Proxy 문서를 참고해요.

더 알아보기 (Learn more)

• Vercel AI Gateway 문서
• LiteLLM Proxy 문서