Infinity

Infinity

Infinity는 텍스트 임베딩, reranking 모델, clip을 서빙하기 위한 고처리량, 저지연 REST API예요.

출처: 문서

본문

개요 (Overview)

LiteLLM Python SDK 사용법

from litellm import rerank, embedding
import os

os.environ["INFINITY_API_BASE"] = "http://localhost:8080"

response = rerank(
    model="infinity/rerank",
    query="What is the capital of France?",
    documents=["Paris", "London", "Berlin", "Madrid"],
)

LiteLLM Proxy 사용법

LiteLLM은 Rerank 호출용 Cohere API 호환 /rerank 엔드포인트를 제공해요.

config.yaml에 추가:

model_list:
  - model_name: custom-infinity-rerank
    litellm_params:
      model: infinity/rerank
      api_base: https://localhost:8080
      api_key: os.environ/INFINITY_API_KEY

litellm --config /path/to/config.yaml
# RUNNING on http://0.0.0.0:4000

테스트 요청:

curl http://0.0.0.0:4000/rerank \
  -H "Authorization: Bearer ***" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "custom-infinity-rerank",
    "query": "What is the capital of the United States?",
    "documents": [
      "Carson City is the capital city of the American state of Nevada.",
      "The Commonwealth of the Northern Mariana Islands is a group of islands in the Pacific Ocean. Its capital is Saipan.",
      "Washington, D.C. is the capital of the United States.",
      "Capital punishment has existed in the United States since before it was a country."
    ],
    "top_n": 3
  }'

지원 Cohere Rerank API 파라미터

Return Documents 사용

response = rerank(
    model="infinity/rerank",
    query="What is the capital of France?",
    documents=["Paris", "London", "Berlin", "Madrid"],
    return_documents=True,
)

curl http://0.0.0.0:4000/rerank \
  -H "Authorization: Bearer ***" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "custom-infinity-rerank",
    "query": "What is the capital of France?",
    "documents": [
      "Paris",
      "London",
      "Berlin",
      "Madrid"
    ],
    "return_documents": True,
  }'

공급자별 파라미터 전달

매핑되지 않은 파라미터는 그대로 공급자로 전달돼요.

SDK:

from litellm import rerank
import os

os.environ["INFINITY_API_BASE"] = "http://localhost:8080"

response = rerank(
    model="infinity/rerank",
    query="What is the capital of France?",
    documents=["Paris", "London", "Berlin", "Madrid"],
    raw_scores=True,  # 👈 PROVIDER-SPECIFIC PARAM
)

Proxy:

model_list:
  - model_name: custom-infinity-rerank
    litellm_params:
      model: infinity/rerank
      api_base: https://localhost:8080
      raw_scores: True  # 👈 EITHER SET PROVIDER-SPECIFIC PARAMS HERE OR IN REQUEST BODY

litellm --config /path/to/config.yaml
# RUNNING on http://0.0.0.0:4000

curl http://0.0.0.0:4000/rerank \
  -H "Authorization: Bearer ***" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "custom-infinity-rerank",
    "query": "What is the capital of the United States?",
    "documents": [
      "Carson City is the capital city of the American state of Nevada.",
      "The Commonwealth of the Northern Mariana Islands is a group of islands in the Pacific Ocean. Its capital is Saipan.",
      "Washington, D.C. is the capital of the United States.",
      "Capital punishment has existed in the United States since before it was a country."
    ],
    "raw_scores": True  # 👈 PROVIDER-SPECIFIC PARAM
  }'

Embeddings

LiteLLM은 embedding 호출용 OpenAI API 호환 /embeddings 엔드포인트를 제공해요.

config.yaml에 추가:

model_list:
  - model_name: custom-infinity-embedding
    litellm_params:
      model: infinity/provider/custom-embedding-v1
      api_base: http://localhost:8080
      api_key: os.environ/INFINITY_API_KEY

기본 예시:

from litellm import embedding
import os

os.environ["INFINITY_API_BASE"] = "http://localhost:8080"

response = embedding(
    model="infinity/bge-small",
    input=["good morning from litellm"]
)
print(response.data[0]['embedding'])

curl http://0.0.0.0:4000/embeddings \
  -H "Authorization: Bearer ***" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "custom-infinity-embedding",
    "input": ["hello"]
  }'

지원 Embedding API 파라미터

OpenAI SDK 사용:

from openai import OpenAI

client = OpenAI(
    api_key="<LITELLM_MASTER_KEY>",
    base_url="<LITELLM_URL>"
)

response = client.embeddings.create(
    model="bge-small",
    input=["The food was delicious and the waiter..."],
    encoding_format="float"
)
print(response.data[0].embedding)

curl:

curl http://0.0.0.0:4000/embeddings \
  -H "Authorization: Bearer ***" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "bge-small",
    "input": ["The food was delicious and the waiter..."],
    "encoding_format": "float"
  }'

더 알아보기 (Learn more)

• LiteLLM 개요
• LiteLLM Python SDK 사용법
• LiteLLM Proxy Server 설정
• 지원되는 공급업체 전체 목록