Azure OpenAI

Azure OpenAI

Azure OpenAI Service는 OpenAI의 강력한 언어 모델(o1, o1-mini, GPT-5, GPT-4o, GPT-4o mini, GPT-4 Turbo with Vision, GPT-4, GPT-3.5-Turbo, Embeddings 모델 시리즈)에 REST API 접근을 제공해요. Azure Foundry의 Claude 모델도 지원해요.

출처: 문서

본문

개요 (Overview)

속성	설명
설명	Azure OpenAI Service는 OpenAI의 강력한 언어 모델(o1, o1-mini, GPT-5, GPT-4o, GPT-4o mini, GPT-4 Turbo with Vision, GPT-4, GPT-3.5-Turbo, Embeddings 시리즈)에 REST API 접근을 제공. Azure Foundry의 Claude 모델도 지원
LiteLLM 라우트	azure/, azure/o_series/, azure/gpt5_series/. Azure Foundry의 Claude 모델은 azure_ai/claude-* 라우트 사용
지원 작업	/chat/completions, /responses, /completions, /embeddings, /audio/speech, /audio/transcriptions, /fine_tuning, /batches, /files, /images
공급자 문서	Azure OpenAI, Azure Foundry Claude

Azure Foundry의 Claude 모델은 azure/가 아니라 azure_ai/ 공급자를 통해 라우팅돼요. azure_ai/claude-* 모델 이름(예: azure_ai/claude-sonnet-5)을 Azure 인증과 함께 사용하세요. 자세한 내용은 Azure Anthropic 문서를 참고해요.

설정 (Setup)

API 키, api_base, api_version 등은 litellm.completion에 직접 전달하거나 litellm.api_key 파라미터로 설정할 수 있어요.

import os

os.environ["AZURE_API_KEY"] = ""  # "my-azure-api-key"
os.environ["AZURE_API_BASE"] = ""  # "https://example-endpoint.openai.azure.com"
os.environ["AZURE_API_VERSION"] = ""  # "2023-05-15"

# optional
os.environ["AZURE_AD_TOKEN"] = ""
os.environ["AZURE_API_TYPE"] = ""

LiteLLM Python SDK 사용법

.env 변수 사용

from litellm import completion

## set ENV variables
os.environ["AZURE_API_KEY"] = ""
os.environ["AZURE_API_BASE"] = ""
os.environ["AZURE_API_VERSION"] = ""

# azure call
response = completion(
    model = "azure/<your_deployment_name>",
    messages = [{ "content": "Hello, how are you?","role": "user"}]
)

api_key, api_base, api_version 직접 전달

import litellm

# azure call
response = litellm.completion(
    model = "azure/<your deployment name>",  # model = azure/<your deployment name>
    api_base = "",  # azure api base
    api_version = "",  # azure api version
    api_key = "",  # azure api key
    messages = [{"role": "user", "content": "good morning"}],
)

azure_ad_token 사용

import litellm

# azure call
response = litellm.completion(
    model = "azure/<your deployment name>",  # model = azure/<your deployment name>
    api_base = "",  # azure api base
    api_version = "",  # azure api version
    azure_ad_token="",  # azure_ad_token
    messages = [{"role": "user", "content": "good morning"}],
)

LiteLLM Proxy Server 사용법

Azure OpenAI 모델을 LiteLLM Proxy Server로 호출하는 방법이에요.

1. 환경에 키 저장

export AZURE_API_KEY=""

2. Proxy 시작

model_list:
  - model_name: gpt-5.6-luna
    litellm_params:
      model: azure/chatgpt-v-2
      api_base: https://openai-gpt-4-test-v-1.openai.azure.com/
      api_version: "2023-05-15"
      api_key: os.environ/AZURE_API_KEY  # The `os.environ/` prefix tells litellm to read this from the env.

3. 테스트

cURL:

curl --location 'http://0.0.0.0:4000/chat/completions' \
  --header 'Content-Type: application/json' \
  --data ' {
    "model": "gpt-5.6-luna",
    "messages": [
      {
        "role": "user",
        "content": "what llm are you"
      }
    ]
  }'

OpenAI v1.0.0+:

import openai

client = openai.OpenAI(
    api_key="anything",
    base_url="http://0.0.0.0:4000"
)

response = client.chat.completions.create(
    model="gpt-5.6-luna",
    messages = [
        {
            "role": "user",
            "content": "this is a test request, write a short poem"
        }
    ]
)
print(response)

Langchain:

from langchain.chat_models import ChatOpenAI
from langchain.prompts.chat import (
    ChatPromptTemplate,
    HumanMessagePromptTemplate,
    SystemMessagePromptTemplate,
)
from langchain.schema import HumanMessage, SystemMessage

chat = ChatOpenAI(
    openai_api_base="http://0.0.0.0:4000",  # set openai_api_base to the LiteLLM Proxy
    model = "gpt-5.6-luna",
    temperature=0.1
)

messages = [
    SystemMessage(
        content="You are a helpful assistant that im using to make a test request to."
    ),
    HumanMessage(
        content="test from litellm. tell me why it's amazing in 1 sentence"
    ),
]
response = chat(messages)
print(response)

API 버전 설정하기

proxy config.yaml에서 Azure OpenAI의 api_version을 다음과 같이 설정할 수 있어요.

Option 1: 모델별 설정

model_list:
  - model_name: gpt-5.6-terra
    litellm_params:
      model: azure/my-gpt4-deployment
      api_base: https://your-resource.openai.azure.com/
      api_version: "2024-08-01-preview"  # Set version per model
      api_key: os.environ/AZURE_API_KEY

Azure OpenAI Chat Completion 모델

모든 Azure 모델을 지원해요. litellm 요청 시 model=azure/<your deployment name> 접두사로 설정하기만 하면 돼요.

모델 이름	함수 호출
o1-mini	response = completion(model="azure/<your deployment name>", messages=messages)
o1-preview	response = completion(model="azure/<your deployment name>", messages=messages)
gpt-5	response = completion(model="azure/<your deployment name>", messages=messages)
gpt-4o-mini	completion('azure/<your deployment name>', messages)
gpt-4o	completion('azure/<your deployment name>', messages)
gpt-4	completion('azure/<your deployment name>', messages)
gpt-4-0314	completion('azure/<your deployment name>', messages)
gpt-4-0613	completion('azure/<your deployment name>', messages)
gpt-4-32k	completion('azure/<your deployment name>', messages)
gpt-4-32k-0314	completion('azure/<your deployment name>', messages)
gpt-4-32k-0613	completion('azure/<your deployment name>', messages)
gpt-4-1106-preview	completion('azure/<your deployment name>', messages)
gpt-4-0125-preview	completion('azure/<your deployment name>', messages)
gpt-3.5-turbo	completion('azure/<your deployment name>', messages)
gpt-3.5-turbo-0301	completion('azure/<your deployment name>', messages)
gpt-3.5-turbo-0613	completion('azure/<your deployment name>', messages)
gpt-3.5-turbo-16k	completion('azure/<your deployment name>', messages)
gpt-3.5-turbo-16k-0613	completion('azure/<your deployment name>', messages)

Azure OpenAI Vision 모델

모델 이름	함수 호출
gpt-4-vision	completion(model="azure/<your deployment name>", messages=messages)
gpt-4o	completion('azure/<your deployment name>', messages)

기본 사용법:

import os
from litellm import completion

os.environ["AZURE_API_KEY"] = "your-api-key"

# azure call
response = completion(
    model = "azure/<your deployment name>",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "What's in this image?"
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://awsmp-logos.s3.amazonaws.com/seller-xw5kijmvmzasy/c233c9ade2ccb5491072ae232c814942.png"
                    }
                }
            ]
        }
    ],
)

Azure Vision enhancements 사용:

참고: Azure는 base_url을 /extensions로 설정해야 해요.

base_url="https://gpt-4-vision-resource.openai.azure.com/openai/deployments/gpt-4-vision/extensions"
# base_url="{azure_endpoint}/openai/deployments/{azure_deployment}/extensions"

import os
from litellm import completion

os.environ["AZURE_API_KEY"] = "your-api-key"

# azure call
response = completion(
    model="azure/gpt-4-vision",
    timeout=5,
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Whats in this image?"},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://avatars.githubusercontent.com/u/29436595?v=4"
                    },
                },
            ],
        }
    ],
    base_url="https://gpt-4-vision-resource.openai.azure.com/openai/deployments/gpt-4-vision/extensions",
    api_key=os.getenv("AZURE_VISION_API_KEY"),
    enhancements={"ocr": {"enabled": True}, "grounding": {"enabled": True}},
    dataSources=[
        {
            "type": "AzureComputerVision",
            "parameters": {
                "endpoint": "https://gpt-4-vision-enhancement.cognitiveservices.azure.com/",
                "key": os.environ["AZURE_VISION_ENHANCE_KEY"],
            },
        }
    ],
)

O-Series 모델

Azure OpenAI O-Series 모델은 LiteLLM에서 지원돼요. LiteLLM은 모델 이름에 o1이나 o3가 있는 배포 이름을 O-Series 변환 로직으로 라우팅해요. 명시적으로 설정하려면 model을 azure/o_series/<your-deployment-name>으로 설정하세요.

자동 라우팅 (SDK):

import litellm

litellm.completion(
    model="azure/my-o3-deployment",
    messages=[{"role": "user", "content": "Hello, world!"}]
)  # 👈 Note: 'o3' in the deployment name

자동 라우팅 (Proxy):

model_list:
  - model_name: o3-mini
    litellm_params:
      model: azure/o3-model
      api_base: os.environ/AZURE_API_BASE
      api_key: os.environ/AZURE_API_KEY

명시적 라우팅 (SDK):

import litellm

litellm.completion(
    model="azure/o_series/my-random-deployment-name",
    messages=[{"role": "user", "content": "Hello, world!"}]
)  # 👈 Note: 'o_series/' in the deployment name

명시적 라우팅 (Proxy):

model_list:
  - model_name: o3-mini
    litellm_params:
      model: azure/o_series/my-random-deployment-name
      api_base: os.environ/AZURE_API_BASE
      api_key: os.environ/AZURE_API_KEY

GPT-5 모델

LiteLLM은 Azure GPT-5 모델을 두 가지 방식으로 지원해요:

• 명시적 라우팅: model = azure/gpt5_series/<deployment-name>
• 추론 라우팅(Azure 배포 이름에 gpt-5 포함): model = azure/gpt-5.6-luna

명시적 라우팅 (SDK):

import litellm

response = litellm.completion(
    model="azure/gpt5_series/my-gpt-5-deployment",
    messages=[{"role": "user", "content": "Hello, world!"}]
)

명시적 라우팅 (Proxy):

model_list:
  - model_name: gpt-5.6-terra
    litellm_params:
      model: azure/gpt5_series/my-gpt-5-deployment
      api_base: os.environ/AZURE_API_BASE
      api_key: os.environ/AZURE_API_KEY

추론 라우팅 (SDK):

import litellm

# Deployment name contains 'gpt-5' - automatically inferred
response = litellm.completion(
    model="azure/my-gpt-5-deployment",
    messages=[{"role": "user", "content": "Hello, world!"}]
)

추론 라우팅 (Proxy):

model_list:
  - model_name: gpt-5.6-luna
    litellm_params:
      model: azure/my-gpt-5-deployment  # deployment name contains 'gpt-5'
      api_base: os.environ/AZURE_API_BASE
      api_key: os.environ/AZURE_API_KEY

Azure Audio 모델

SDK:

from litellm import completion
import os

os.environ["AZURE_API_KEY"] = ""
os.environ["AZURE_API_BASE"] = ""
os.environ["AZURE_API_VERSION"] = ""

response = completion(
    model="azure/azure-openai-4o-audio",
    messages=[
        {
            "role": "user",
            "content": "I want to try out speech to speech"
        }
    ],
    modalities=["text","audio"],
    audio={"voice": "alloy", "format": "wav"}
)
print(response)

Proxy:

model_list:
  - model_name: azure-openai-4o-audio
    litellm_params:
      model: azure/azure-openai-4o-audio
      api_base: os.environ/AZURE_API_BASE
      api_key: os.environ/AZURE_API_KEY
      api_version: os.environ/AZURE_API_VERSION

litellm --config /path/to/config.yaml

curl http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer ***" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "azure-openai-4o-audio",
    "messages": [{"role": "user", "content": "I want to try out speech to speech"}],
    "modalities": ["text","audio"],
    "audio": {"voice": "alloy", "format": "wav"}
  }'

Azure Instruct 모델

model="azure_text/<your-deployment>" 사용.

모델 이름	함수 호출
gpt-3.5-turbo-instruct	response = completion(model="azure_text/<your deployment name>", messages=messages)
gpt-3.5-turbo-instruct-0914	response = completion(model="azure_text/<your deployment name>", messages=messages)

import litellm

## set ENV variables
os.environ["AZURE_API_KEY"] = ""
os.environ["AZURE_API_BASE"] = ""
os.environ["AZURE_API_VERSION"] = ""

response = litellm.completion(
    model="azure_text/<your-deployment-name",
    messages=[{"role": "user", "content": "What is the weather like in Boston?"}]
)
print(response)

인증 (Authentication)

Entra ID - azure_ad_token 사용

Azure Active Directory 토큰(Microsoft Entra ID)으로 litellm.completion() 호출하는 방법이에요. 이 과정은 다른 모든 Azure 엔드포인트(chat, embeddings, image, audio 등)에도 동일하게 적용돼요.

Step 1 - Azure CLI 설치: https://learn.microsoft.com/cli/azure/install-azure-cli

Step 2 - az로 로그인:

az login --output table

Step 3 - Azure AD 토큰 생성:

az account get-access-token --resource https://cognitiveservices.azure.com

이 단계에서 accessToken이 생성된 것을 확인할 수 있어요:

{
  "accessToken": "eyJ0eX...WSJ9",
  "expiresOn": "2023-11-14 15:50:46.000000",
  "expires_on": 1700005846,
  "subscription": "db38de1f-4bb3..",
  "tenant": "bdfd79b3-8401-47..",
  "tokenType": "Bearer"
}

Step 4 - Azure AD 토큰으로 litellm.completion 호출:

response = litellm.completion(
    model = "azure/<your deployment name>",  # model = azure/<your deployment name>
    api_base = "",  # azure api base
    api_version = "",  # azure api version
    azure_ad_token="",  # your accessToken from step 3
    messages = [{"role": "user", "content": "good morning"}],
)

Proxy config.yaml:

model_list:
  - model_name: gpt-5.6-luna
    litellm_params:
      model: azure/chatgpt-v-2
      api_base: https://openai-gpt-4-test-v-1.openai.azure.com/
      api_version: "2023-05-15"
      azure_ad_token: os.environ/AZURE_AD_TOKEN

Entra ID - tenant_id, client_id, client_secret 사용

model_list:
  - model_name: gpt-5.6-luna
    litellm_params:
      model: azure/chatgpt-v-2
      api_base: https://openai-gpt-4-test-v-1.openai.azure.com/
      api_version: "2023-05-15"
      tenant_id: os.environ/AZURE_TENANT_ID
      client_id: os.environ/AZURE_CLIENT_ID
      client_secret: os.environ/AZURE_CLIENT_SECRET
      azure_scope: os.environ/AZURE_SCOPE  # defaults to "https://cognitiveservices.azure.com/.default"

Entra ID - client_id, username, password 사용

model_list:
  - model_name: gpt-5.6-luna
    litellm_params:
      model: azure/chatgpt-v-2
      api_base: https://openai-gpt-4-test-v-1.openai.azure.com/
      api_version: "2023-05-15"
      client_id: os.environ/AZURE_CLIENT_ID
      azure_username: os.environ/AZURE_USERNAME
      azure_password: os.environ/AZURE_PASSWORD
      azure_scope: os.environ/AZURE_SCOPE  # defaults to "https://cognitiveservices.azure.com/.default"

Azure AD Token Refresh - DefaultAzureCredential

요청에 Azure DefaultAzureCredential 인증을 쓰고 싶을 때 사용해요. DefaultAzureCredential은 여러 소스에서 사용 가능한 Azure 자격 증명을 자동으로 발견해요.

Option 1: 명시적 DefaultAzureCredential (권장)

from litellm import completion
from azure.identity import DefaultAzureCredential, get_bearer_token_provider

# DefaultAzureCredential automatically discovers credentials from:
# - Environment variables (AZURE_CLIENT_ID, AZURE_CLIENT_SECRET, AZURE_TENANT_ID)
# - Managed Identity (AKS, Azure VMs, etc.)
# - Azure CLI credentials
# - And other Azure identity sources
token_provider = get_bearer_token_provider(
    DefaultAzureCredential(),
    "https://cognitiveservices.azure.com/.default"
)

response = completion(
    model = "azure/<your deployment name>",  # model = azure/<your deployment name>
    api_base = "",  # azure api base
    api_version = "",  # azure api version
    azure_ad_token_provider=token_provider,
    messages = [{"role": "user", "content": "good morning"}],
)

Option 2: LiteLLM 자동 폴백

import litellm

# Enable automatic fallback to DefaultAzureCredential
litellm.enable_azure_ad_token_refresh = True

response = litellm.completion(
    model = "azure/<your deployment name>",
    api_base = "",
    api_version = "",
    messages = [{"role": "user", "content": "good morning"}],
)

Proxy 설정:

export AZURE_TENANT_ID=""
export AZURE_CLIENT_ID=""
export AZURE_CLIENT_SECRET=""

model_list:
  - model_name: gpt-5.6-luna
    litellm_params:
      model: azure/your-deployment-name
      api_base: https://openai-gpt-4-test-v-1.openai.azure.com/

litellm_settings:
  enable_azure_ad_token_refresh: true  # 👈 KEY CHANGE

동작 방식:

• LiteLLM은 먼저 Service Principal 인증을 시도 (환경 변수가 있으면)
• 실패하면 DefaultAzureCredential로 자동 폴백
• DefaultAzureCredential은 Managed Identity, Azure CLI 자격 증명 또는 다른 Azure 신원 소스를 사용
• 이로써 AKS 같은 관리형 환경에서 하드코딩 자격 증명이 불필요

Azure Batches API

속성	설명
설명	Azure OpenAI Batches API
LiteLLM의 custom_llm_provider	azure/
지원 작업	/v1/batches, /v1/files
비용/로깅 지원	✅ LiteLLM이 Batch API 요청 로깅·비용 추적

1. 파일 업로드

OpenAI Python SDK:

from openai import OpenAI

# Initialize the client
client = OpenAI(
    base_url="http://localhost:4000",
    api_key="your-api-key",
)

batch_input_file = client.files.create(
    file=open("mydata.jsonl", "rb"),
    purpose="batch",
    extra_headers={"custom-llm-provider": "azure"}
)
file_id = batch_input_file.id

cURL:

curl http://localhost:4000/v1/files \
  -H "Authorization: Bearer ***" \
  -F purpose="batch" \
  -F file="@mydata.jsonl"

예시 파일 형식:

{"custom_id": "task-0", "method": "POST", "url": "/chat/completions", "body": {"model": "REPLACE-WITH-MODEL-DEPLOYMENT-NAME", "messages": [{"role": "system", "content": "You are an AI assistant that helps people find information."}, {"role": "user", "content": "When was Microsoft founded?"}]}}
{"custom_id": "task-1", "method": "POST", "url": "/chat/completions", "body": {"model": "REPLACE-WITH-MODEL-DEPLOYMENT-NAME", "messages": [{"role": "system", "content": "You are an AI assistant that helps people find information."}, {"role": "user", "content": "When was the first XBOX released?"}]}}
{"custom_id": "task-2", "method": "POST", "url": "/chat/completions", "body": {"model": "REPLACE-WITH-MODEL-DEPLOYMENT-NAME", "messages": [{"role": "system", "content": "You are an AI assistant that helps people find information."}, {"role": "user", "content": "What is Altair Basic?"}]}}

2. 배치 요청 생성

OpenAI Python SDK:

batch = client.batches.create(  # re use client from above
    input_file_id=file_id,
    endpoint="/v1/chat/completions",
    completion_window="24h",
    metadata={"description": "My batch job"},
    extra_headers={"custom-llm-provider": "azure"}
)

cURL:

curl http://localhost:4000/v1/batches \
  -H "Authorization: Bearer ***" \
  -H "Content-Type: application/json" \
  -d '{
    "input_file_id": "file-abc123",
    "endpoint": "/v1/chat/completions",
    "completion_window": "24h"
  }'

3. 배치 조회 / 취소 / 목록

retrieved_batch = client.batches.retrieve(
    batch.id,
    extra_headers={"custom-llm-provider": "azure"}
)

cancelled_batch = client.batches.cancel(
    batch.id,
    extra_headers={"custom-llm-provider": "azure"}
)

client.batches.list(extra_headers={"custom-llm-provider": "azure"})

LiteLLM SDK (비동기)

import litellm
import os

os.environ["AZURE_API_KEY"] = ""
os.environ["AZURE_API_BASE"] = ""

file_name = "azure_batch_completions.jsonl"
_current_dir = os.path.dirname(os.path.abspath(__file__))
file_path = os.path.join(_current_dir, file_name)

file_obj = await litellm.acreate_file(
    file=open(file_path, "rb"),
    purpose="batch",
    custom_llm_provider="azure",
)
print("Response from creating file=", file_obj)

create_batch_response = await litellm.acreate_batch(
    completion_window="24h",
    endpoint="/v1/chat/completions",
    input_file_id=batch_input_file_id,
    custom_llm_provider="azure",
    metadata={"key1": "value1", "key2": "value2"},
)
print("response from litellm.create_batch=", create_batch_response)

retrieved_batch = await litellm.aretrieve_batch(
    batch_id=create_batch_response.id,
    custom_llm_provider="azure"
)
print("retrieved batch=", retrieved_batch)

# Get file content
file_content = await litellm.afile_content(
    file_id=batch_input_file_id,
    custom_llm_provider="azure"
)
print("file content = ", file_content)

list_batches_response = litellm.list_batches(
    custom_llm_provider="azure",
    limit=2
)
print("list_batches_response=", list_batches_response)

[BETA] 여러 Azure 배포 로드밸런싱

config.yaml에서 enable_loadbalancing_on_batch_endpoints: true 설정. 이 기능은 {PROXY_BASE_URL}/v1/files와 {PROXY_BASE_URL}/v1/batches에서 동작해요. 응답은 OpenAI 형식이에요.

model_list:
  - model_name: "batch-gpt-4o-mini"
    litellm_params:
      model: "azure/gpt-5.6-luna"
      api_key: os.environ/AZURE_API_KEY
      api_base: os.environ/AZURE_API_BASE
      model_info:
        mode: batch

litellm_settings:
  enable_loadbalancing_on_batch_endpoints: true  # 👈 KEY CHANGE

.jsonl에 model: batch-gpt-4o-mini를 설정하세요 (모델은 Azure 배포 이름이어야 해요).

고급 (Advanced)

Azure API 로드밸런싱

여러 Azure/OpenAI 배포를 로드밸런싱할 때 사용해요. Router가 rate-limit 아래이고 사용 토큰이 가장 적은 배포를 골라 실패한 요청을 방지해요. 프로덕션에서 Router는 Redis Cache에 연결해 여러 배포의 사용량을 추적해요.

from litellm import Router

model_list = [
    {  # list of model deployments
        "model_name": "gpt-5.6-luna",  # openai model name
        "litellm_params": {  # params for litellm completion/embedding call
            "model": "azure/chatgpt-v-2",
            "api_key": os.getenv("AZURE_API_KEY"),
            "api_version": os.getenv("AZURE_API_VERSION"),
            "api_base": os.getenv("AZURE_API_BASE"),
        },
        "tpm": 240000,
        "rpm": 1800
    },
    {
        "model_name": "gpt-5.6-luna",  # openai model name
        "litellm_params": {
            "model": "azure/chatgpt-functioncalling",
            "api_key": os.getenv("AZURE_API_KEY"),
            "api_version": os.getenv("AZURE_API_VERSION"),
            "api_base": os.getenv("AZURE_API_BASE"),
        },
        "tpm": 240000,
        "rpm": 1800
    },
    {
        "model_name": "gpt-5.6-luna",  # openai model name
        "litellm_params": {
            "model": "gpt-5.6-luna",
            "api_key": os.getenv("OPENAI_API_KEY"),
        },
        "tpm": 1000000,
        "rpm": 9000
    }
]

router = Router(model_list=model_list)

# openai.chat.completions.create replacement
response = router.completion(model="gpt-5.6-luna",
                             messages=[{"role": "user", "content": "Hey, how's it going?"}])
print(response)

Redis Queue 사용:

router = Router(model_list=model_list,
                redis_host=os.getenv("REDIS_HOST"),
                redis_password=os.getenv("REDIS_PASSWORD"),
                redis_port=os.getenv("REDIS_PORT"))
print(response)

Tool Calling / Function Calling

# set Azure env variables
import os
import litellm
import json

os.environ['AZURE_API_KEY'] = ""  # litellm reads AZURE_API_KEY from .env and sends the request
os.environ['AZURE_API_BASE'] = "https://openai-gpt-4-test-v-1.openai.azure.com/"
os.environ['AZURE_API_VERSION'] = "2023-07-01-preview"

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_current_weather",
            "description": "Get the current weather in a given location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city and state, e.g. San Francisco, CA",
                    },
                    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
                },
                "required": ["location"],
            },
        },
    }
]

response = litellm.completion(
    model="azure/chatgpt-functioncalling",  # model = azure/<your-azure-deployment-name>
    messages=[{"role": "user", "content": "What's the weather like in San Francisco, Tokyo, and Paris?"}],
    tools=tools,
    tool_choice="auto",  # auto is default, but we'll be explicit
)
print("\nLLM Response1:\n", response)

response_message = response.choices[0].message
tool_calls = response.choices[0].message.tool_calls
print("\nTool Choice:\n", tool_calls)

Proxy:

model_list:
  - model_name: azure-gpt-3.5
    litellm_params:
      model: azure/chatgpt-functioncalling
      api_base: os.environ/AZURE_API_BASE
      api_key: os.environ/AZURE_API_KEY
      api_version: "2023-07-01-preview"

litellm --config config.yaml

Azure OpenAI 모델 지출 추적 (PROXY)

이미지 생성 호출에 대한 비용 추적을 위해 base model 설정:

model_list:
  - model_name: dall-e-3
    litellm_params:
      model: azure/dall-e-3-test
      api_version: 2023-06-01-preview
      api_base: https://openai-gpt-4-test-v-1.openai.azure.com/
      api_key: os.environ/AZURE_API_KEY
      base_model: dall-e-3  # 👈 set dall-e-3 as base model
      model_info:
        mode: image_generation

또한 일반 모델에도 base_model을 설정할 수 있어요:

model_list:
  - model_name: azure-gpt-3.5
    litellm_params:
      model: azure/chatgpt-v-2
      api_base: os.environ/AZURE_API_BASE
      api_key: os.environ/AZURE_API_KEY
      api_version: "2023-07-01-preview"
      model_info:
        base_model: azure/gpt-5.6-terra

더 알아보기 (Learn more)

• LiteLLM 개요
• LiteLLM Python SDK 사용법
• LiteLLM Proxy Server 설정
• 지원되는 공급업체 전체 목록