DeepInfra
DeepInfra
DeepInfra의 모든 모델을 지원해요. litellm 요청 시 model=deepinfra/<any-model-on-deepinfra> 접두사로 설정하기만 하면 돼요.
출처: 문서
본문
API 키
# env variable
os.environ['DEEPINFRA_API_KEY']
샘플 사용법 (Sample Usage)
from litellm import completion
import os
os.environ['DEEPINFRA_API_KEY'] = ""
response = completion(
model="deepinfra/meta-llama/Llama-2-70b-chat-hf",
messages=[{"role": "user", "content": "write code for saying hi from LiteLLM"}]
)
샘플 사용법 - 스트리밍 (Streaming)
from litellm import completion
import os
os.environ['DEEPINFRA_API_KEY'] = ""
response = completion(
model="deepinfra/meta-llama/Llama-2-70b-chat-hf",
messages=[{"role": "user", "content": "write code for saying hi from LiteLLM"}],
stream=True
)
for chunk in response:
print(chunk)
Chat 모델
| 모델 이름 | 함수 호출 |
|---|---|
| meta-llama/Meta-Llama-3-8B-Instruct | completion(model="deepinfra/meta-llama/Meta-Llama-3-8B-Instruct", messages) |
| meta-llama/Meta-Llama-3-70B-Instruct | completion(model="deepinfra/meta-llama/Meta-Llama-3-70B-Instruct", messages) |
| meta-llama/Llama-2-70b-chat-hf | completion(model="deepinfra/meta-llama/Llama-2-70b-chat-hf", messages) |
| meta-llama/Llama-2-7b-chat-hf | completion(model="deepinfra/meta-llama/Llama-2-7b-chat-hf", messages) |
| meta-llama/Llama-2-13b-chat-hf | completion(model="deepinfra/meta-llama/Llama-2-13b-chat-hf", messages) |
| codellama/CodeLlama-34b-Instruct-hf | completion(model="deepinfra/codellama/CodeLlama-34b-Instruct-hf", messages) |
| mistralai/Mistral-7B-Instruct-v0.1 | completion(model="deepinfra/mistralai/Mistral-7B-Instruct-v0.1", messages) |
| jondurbin/airoboros-l2-70b-gpt4-1.4.1 | completion(model="deepinfra/jondurbin/airoboros-l2-70b-gpt4-1.4.1", messages) |
Rerank 엔드포인트
LiteLLM은 DeepInfra rerank 모델용 Cohere API 호환 /rerank 엔드포인트를 제공해요.
지원 Rerank 모델
| 모델 이름 | 설명 |
|---|---|
| deepinfra/Qwen/Qwen3-Reranker-0.6B | 경량 rerank 모델 (0.6B 파라미터) |
| deepinfra/Qwen/Qwen3-Reranker-4B | 중형 rerank 모델 (4B 파라미터) |
| deepinfra/Qwen/Qwen3-Reranker-8B | 대형 rerank 모델 (8B 파라미터) |
LiteLLM Python SDK 사용법
from litellm import rerank
import os
os.environ["DEEPINFRA_API_KEY"] = "your-api-key"
response = rerank(
model="deepinfra/Qwen/Qwen3-Reranker-0.6B",
query="What is the capital of France?",
documents=[
"Paris is the capital of France.",
"London is the capital of the United Kingdom.",
"Berlin is the capital of Germany.",
"Madrid is the capital of Spain.",
"Rome is the capital of Italy."
]
)
print(response)
config.yaml에 추가:
model_list:
- model_name: Qwen/Qwen3-Reranker-0.6B
litellm_params:
model: deepinfra/Qwen/Qwen3-Reranker-0.6B
api_key: os.environ/DEEPINFRA_API_KEY
Proxy 시작 후 테스트:
litellm --config /path/to/config.yaml
# RUNNING on http://0.0.0.0:4000/
curl -L -X POST 'http://0.0.0.0:4000/rerank' \
-H "Authorization: Bearer ***" \
-H 'Content-Type: application/json' \
-d '{
"model": "Qwen/Qwen3-Reranker-0.6B",
"query": "What is the capital of France?",
"documents": [
"Paris is the capital of France.",
"London is the capital of the United Kingdom.",
"Berlin is the capital of Germany.",
"Madrid is the capital of Spain.",
"Rome is the capital of Italy."
]
}'
지원 Cohere Rerank API 파라미터
| 파라미터 | 타입 | 설명 |
|---|---|---|
| query | str | 문서를 rerank할 쿼리 |
| documents | list[str] | rerank할 문서 |
공급자별 파라미터
다른 deepinfra 전용 파라미터를 keyword argument로 rerank 함수에 전달할 수 있어요.
response = rerank(
model="deepinfra/Qwen/Qwen3-Reranker-0.6B",
query="What is the capital of France?",
documents=[
"Paris is the capital of France.",
"London is the capital of the United Kingdom.",
"Berlin is the capital of Germany.",
"Madrid is the capital of Spain.",
"Rome is the capital of Italy."
],
my_custom_param="my_custom_value", # any other deepinfra specific parameters
)
응답 형식:
{
"id": "request-id",
"results": [
{
"index": 0,
"relevance_score": 0.9975274205207825
},
{
"index": 1,
"relevance_score": 0.011687257327139378
}
],
"meta": {
"billed_units": {
"total_tokens": 427
},
"tokens": {
"input_tokens": 427,
"output_tokens": 0
}
}
}