Vertex AI SDK

Vertex AI SDK

Vertex AI의 패스스루 엔드포인트를 소개할게요. Vertex AI 고유 엔드포인트를 네이티브 형식 그대로(변환 없이) 호출할 수 있는 기능이에요. /generateContent 같은 엔드포인트를 그대로 사용할 수 있어요.

출처: 문서

본문

지원 엔드포인트 (Supported Endpoints)

• /vertex_ai → https://{vertex_location}-aiplatform.googleapis.com/
• /vertex_ai/discovery → https://discoveryengine.googleapis.com (Discovery Engine API, 자세한 내용은 Vertex AI Search Datastores 참고)
• /vertex_ai/live → google.cloud.aiplatform.v1.LlmBidiService/BidiGenerateContent (Vertex AI Live WebSocket 참고)

사용 방법 (How to use)

연결 대상 호스트는 이렇게 구성돼요.

https://REGION-aiplatform.googleapis.com

프록시를 통한 주소는 이렇게 구성돼요.

LITELLM_PROXY_BASE_URL/vertex_ai

사용 예시 (Example Usage)

config.yaml에 모델을 등록할 때 use_in_pass_through: true를 지정해야 해요. 이게 핵심 변경점이에요 👈

model_list:
  - model_name: gemini-3.1-pro-preview
    litellm_params:
      model: vertex_ai/gemini-3.1-pro-preview
      vertex_project: adroit-crow-413218
      vertex_location: us-central1
      vertex_credentials: /path/to/credentials.json
      use_in_pass_through: true # 👈 KEY CHANGE

또는 전역 기본 설정(default_vertex_config)으로 자격 증명을 구성할 수 있어요.

default_vertex_config:
  vertex_project: adroit-crow-413218
  vertex_location: us-central1
  vertex_credentials: /path/to/credentials.json

환경 변수로도 설정할 수 있어요.

export DEFAULT_VERTEXAI_PROJECT="adroit-crow-413218"
export DEFAULT_VERTEXAI_LOCATION="us-central1"
export DEFAULT_GOOGLE_APPLICATION_CREDENTIALS="/path/to/credentials.json"

모델과 프로젝트 ID를 변수로 둔 뒤,

MODEL_ID="gemini-3.8-flash"
PROJECT_ID="YOUR_PROJECT_ID"

streamGenerateContent 같은 Vertex 네이티브 엔드포인트를 호출할 수 있어요. (아래 예시는 원본 Vertex 요청 방식입니다.)

curl \
  -X POST \
  -H "Authorization: Bearer *** auth application-default print-access-token)" \
  -H "Content-Type: application/json" \
  "${LITELLM_PROXY_BASE_URL}/vertex_ai/v1/projects/${PROJECT_ID}/locations/us-central1/publishers/google/models/${MODEL_ID}:streamGenerateContent" -d \
  $'{
    "contents": {
      "role": "user",
      "parts": [
        {
        "fileData": {
          "mimeType": "image/png",
          "fileUri": "gs://generativeai-downloads/images/scones.jpg"
          }
        },
        {
          "text": "Describe this picture."
        }
      ]
    }
  }'

사용 예시 1

x-litellm-api-key 헤더로 LiteLLM 키를 전달하며 generateContent를 호출해요.

curl http://localhost:4000/vertex_ai/v1/projects/${PROJECT_ID}/locations/us-central1/publishers/google/models/${MODEL_ID}:generateContent \
  -H "Content-Type: application/json" \
  -H "x-litellm-api-key: *** $LITELLM_API_KEY" \
  -d '{
    "contents":[{
      "role": "user", 
      "parts":[{"text": "How are you doing today?"}]
    }]
  }'

Vertex AI JS SDK(@google-cloud/vertexai)를 프록시로 연결하는 예시예요. apiEndpoint에 /vertex_ai 경로를 지정하고, customHeaders에 x-litellm-api-key를 넣어요.

const { VertexAI } = require('@google-cloud/vertexai');
const vertexAI = new VertexAI({
    project: 'your-project-id', // enter your vertex project id
    location: 'us-central1', // enter your vertex region
    apiEndpoint: "localhost:4000/vertex_ai" // <proxy-server-url>/vertex_ai # note, do not include 'https://' in the url
});
const model = vertexAI.getGenerativeModel({
    model: 'gemini-3.1-pro-preview'
}, {
    customHeaders: {
        "x-litellm-api-key": "sk-<your-litellm-api-key>" // Your litellm Virtual Key
    }
});
async function generateContent() {
    try {
        const prompt = {
            contents: [{
                role: 'user',
                parts: [{ text: 'How are you doing today?' }]
            }]
        };
        const response = await model.generateContent(prompt);
        console.log('Response:', response);
    } catch (error) {
        console.error('Error:', error);
    }
}
generateContent();

Vertex AI Live API (WebSocket)

• default_vertex_config에 자격 증명을 설정하면 WebSocket 엔드포인트 wss://<PROXY_URL>/vertex_ai/live에서 vertex_project, vertex_location, model을 사용해 Gemini 모델과 실시간 양방향 통신할 수 있어요. 자세한 내용은 Vertex AI Live WebSocket 패스스루 문서를 참고해 주세요.

import asyncio
import json
from websockets.asyncio.client import connect
async def main() -> None:
    headers = {
        "x-litellm-api-key": "Bearer sk-you...-key",
        "Content-Type": "application/json",
    }
    async with connect(
        "ws://localhost:4000/vertex_ai/live",
        additional_headers=headers,
    ) as ws:
        await ws.send(
            json.dumps(
                {
                    "setup": {
                        "model": "projects/your-project/locations/us-central1/publishers/google/models/gemini-2.0-flash-live-preview-04-09",
                        "generation_config": {"response_modalities": ["TEXT"]},
                    }
                }
            )
        )
        async for message in ws:
            print("server:", message)
if __name__ == "__main__":
    asyncio.run(main())

빠른 시작 (Quick Start)

/generateContent 엔드포인트를 호출하는 예시예요. 먼저 환경 변수를 설정해요.

export DEFAULT_VERTEXAI_PROJECT="" # "adroit-crow-413218"
export DEFAULT_VERTEXAI_LOCATION="" # "us-central1"
export DEFAULT_GOOGLE_APPLICATION_CREDENTIALS="" # "/Users/Downloads/adroit-crow-413218-a956eef1a2a8.json"

그다음 LiteLLM 프록시를 실행해요.

litellm
# RUNNING on http://0.0.0.0:4000

이제 generateContent 엔드포인트를 호출해요.

curl http://localhost:4000/vertex-ai/v1/projects/${PROJECT_ID}/locations/us-central1/publishers/google/models/gemini-3.1-pro-preview:generateContent \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ***" \
  -d '{
    "contents":[{
      "role": "user",
      "parts":[{"text": "How are you doing today?"}]
    }]
  }'

지원되는 API 엔드포인트 (Supported API Endpoints)

다음과 같은 Vertex 네이티브 엔드포인트를 지원해요: generateContent, predict(임베딩/Imagen), countTokens, tuningJobs 등.

Vertex AI로 인증하기 (Authentication to Vertex AI)

자격 증명은 기본적으로 DEFAULT_VERTEXAI_PROJECT, DEFAULT_VERTEXAI_LOCATION, DEFAULT_GOOGLE_APPLICATION_CREDENTIALS 환경 변수 또는 default_vertex_config/모델별 vertex_* 파라미터로 설정해요.

사용 예시 (Usage Examples)

Gemini API — Generate Content

curl http://localhost:4000/vertex_ai/v1/projects/${PROJECT_ID}/locations/us-central1/publishers/google/models/gemini-3.8-flash:generateContent \
  -H "Content-Type: application/json" \
  -H "x-litellm-api-key: *** $LITELLM_API_KEY" \
  -d '{"contents":[{"role": "user", "parts":[{"text": "hi"}]}]}'

Embeddings API

curl http://localhost:4000/vertex_ai/v1/projects/${PROJECT_ID}/locations/us-central1/publishers/google/models/textembedding-gecko@001:predict \
  -H "Content-Type: application/json" \
  -H "x-litellm-api-key: *** $LITELLM_API_KEY" \
  -d '{"instances":[{"content": "gm"}]}'

Imagen API

curl http://localhost:4000/vertex_ai/v1/projects/${PROJECT_ID}/locations/us-central1/publishers/google/models/imagen-3.0-generate-001:predict \
  -H "Content-Type: application/json" \
  -H "x-litellm-api-key: *** $LITELLM_API_KEY" \
  -d '{"instances":[{"prompt": "make an otter"}], "parameters": {"sampleCount": 1}}'

토큰 계산 (Count Tokens API)

curl http://localhost:4000/vertex_ai/v1/projects/${PROJECT_ID}/locations/us-central1/publishers/google/models/gemini-3.8-flash:countTokens \
  -H "Content-Type: application/json" \
  -H "x-litellm-api-key: *** $LITELLM_API_KEY" \
  -d '{"contents":[{"role": "user", "parts":[{"text": "hi"}]}]}'

튜닝 (Tuning API)

curl http://localhost:4000/vertex_ai/v1/projects/${PROJECT_ID}/locations/us-central1/publishers/google/models/gemini-3.8-flash:tuningJobs \
      -H "Content-Type: application/json" \
      -H "x-litellm-api-key: *** $LITELLM_API_KEY" \
      -d '{
  "baseModel": "gemini-3.8-flash",
  "supervisedTuningSpec" : {
      "training_dataset_uri": "gs://cloud-samples-data/ai-platform/generative_ai/sft_train_data.jsonl"
  }
}'

고급 (Advanced)

가상 키(Virtual Keys)와 함께 사용하기

가상 키는 LiteLLM 프록시에 데이터베이스가 설정된 경우에 사용할 수 있어요. 가상 키 설정 문서를 참고해 주세요.

환경 변수를 설정해요.

export DATABASE_URL=""
export LITELLM_MASTER_KEY=""
# vertex ai credentials
export DEFAULT_VERTEXAI_PROJECT="" # "adroit-crow-413218"
export DEFAULT_VERTEXAI_LOCATION="" # "us-central1"
export DEFAULT_GOOGLE_APPLICATION_CREDENTIALS="" # "/Users/Downloads/adroit-crow-413218-a956eef1a2a8.json"

프록시를 실행해요.

litellm
# RUNNING on http://0.0.0.0:4000

가상 키를 생성해요.

curl -X POST 'http://0.0.0.0:4000/key/generate' \
-H "x-litellm-api-key: *** $LITELLM_API_KEY" \
-H 'Content-Type: application/json' \
-d '{}'

응답에서 키를 받아요.

{
    ...
    "key": "sk-<virtual-key>"
}

이제 가상 키로 호출해요.

curl http://localhost:4000/vertex_ai/v1/projects/${PROJECT_ID}/locations/us-central1/publishers/google/models/gemini-3.1-pro-preview:generateContent \
  -H "Content-Type: application/json" \
  -H "x-litellm-api-key: *** $LITELLM_API_KEY" \
  -d '{
    "contents":[{
      "role": "user", 
       "parts":[{"text": "How are you doing today?"}]
    }]
  }'

요청 헤더로 태그(tags) 보내기

요청에 tags 헤더를 넣어 비용/사용량에 태그를 붙일 수 있어요.

tags: ["vertex-js-sdk", "pass-through-endpoint"]

curl 예시

curl http://localhost:4000/vertex_ai/v1/projects/${PROJECT_ID}/locations/us-central1/publishers/google/models/gemini-3.1-pro-preview:generateContent \
  -H "Content-Type: application/json" \
  -H "x-litellm-api-key: *** $LITELLM_API_KEY" \
  -H "tags: vertex-js-sdk,pass-through-endpoint" \
  -d '{
    "contents":[{
      "role": "user", 
       "parts":[{"text": "How are you doing today?"}]
    }]
  }'

JS SDK 예시

const { VertexAI } = require('@google-cloud/vertexai');
const vertexAI = new VertexAI({
    project: 'your-project-id', // enter your vertex project id
    location: 'us-central1', // enter your vertex region
    apiEndpoint: "localhost:4000/vertex_ai" // <proxy-server-url>/vertex_ai # note, do not include 'https://' in the url
});
const model = vertexAI.getGenerativeModel({
    model: 'gemini-3.1-pro-preview'
}, {
    customHeaders: {
        "x-litellm-api-key": "sk-<your-litellm-api-key>", // Your litellm Virtual Key
        "tags": "vertex-js-sdk,pass-through-endpoint"
    }
});
async function generateContent() {
    try {
        const prompt = {
            contents: [{
                role: 'user',
                parts: [{ text: 'How are you doing today?' }]
            }]
        };
        const response = await model.generateContent(prompt);
        console.log('Response:', response);
    } catch (error) {
        console.error('Error:', error);
    }
}
generateContent();

Vertex AI에서 Anthropic 베타 기능 사용하기

Vertex의 rawPredict 또는 streamRawPredict 엔드포인트(예: Anthropic Claude 모델)에서 베타 헤더를 보내려면 anthropic-beta 헤더를 사용하면 돼요.

curl http://localhost:4000/vertex_ai/v1/projects/${PROJECT_ID}/locations/us-east5/publishers/anthropic/models/claude-sonnet-5:rawPredict \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ***" \
  -H "anthropic-beta: context-1m-2025-08-07" \
  -d '{
    "anthropic_version": "vertex-2023-10-16",
    "messages": [{"role": "user", "content": "Hello"}],
    "max_tokens": 500
  }'

x-pass- 접두사로 커스텀 헤더 전달하기

일반적으로 패스스루 시 인증/제어에 쓰이는 헤더는 제거되는데, x-pass- 접두사를 붙이면 해당 헤더를 그대로 전달할 수 있어요.

• x-pass-anthropic-beta: value → anthropic-beta: value로 전달
• x-pass-custom-header: value → custom-header: value로 전달

curl http://localhost:4000/vertex_ai/v1/projects/${PROJECT_ID}/locations/us-east5/publishers/anthropic/models/claude-sonnet-5:rawPredict \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ***" \
  -H "x-pass-anthropic-beta: context-1m-2025-08-07" \
  -H "x-pass-custom-feature: enabled" \
  -d '{
    "anthropic_version": "vertex-2023-10-16",
    "messages": [{"role": "user", "content": "Hello"}],
    "max_tokens": 500
  }'

전달하고 싶은 모든 헤더에 x-pass- 접두사를 붙이면 돼요.

더 알아보기 (Learn more)

• Vertex AI 추론 문서
• Vertex AI Search Datastores 패스스루
• Vertex AI Live WebSocket 패스스루
• 가상 키 설정