Xinference

Xinference (Xorbits Inference)

https://inference.readthedocs.io/en/latest/index.html

개요 (Overview)

LiteLLM은 Xinference Embedding + Image Generation 호출을 지원해요.

API Base, Key

# env variable
os.environ['XINFERENCE_API_BASE'] = "http://127.0.0.1:9997/v1"
os.environ['XINFERENCE_API_KEY'] = "anything" #[optional] no api key required

샘플 사용법 - Embedding

from litellm import embedding
import os

os.environ['XINFERENCE_API_BASE'] = "http://127.0.0.1:9997/v1"
response = embedding(
    model="xinference/bge-base-en",
    input=["good morning from litellm"],
)
print(response)

api_base 파라미터 사용법 (Sample Usage api_base param)

from litellm import embedding
import os

response = embedding(
    model="xinference/bge-base-en",
    api_base="http://127.0.0.1:9997/v1",
    input=["good morning from litellm"],
)
print(response)

이미지 생성 (Image Generation)

LiteLLM Python SDK 사용법

from litellm import image_generation
import os

# xinference image generation call
response = image_generation(
    model="xinference/stabilityai/stable-diffusion-3.5-large",
    prompt="A beautiful sunset over a calm ocean",
    api_base="http://127.0.0.1:9997/v1",
)
print(response)

LiteLLM Proxy Server 사용법

1. config.yaml 설정

model_list:
  - model_name: xinference-sd
    litellm_params:
      model: xinference/stabilityai/stable-diffusion-3.5-large
      api_base: http://127.0.0.1:9997/v1
      api_key: anything
    model_info:
      mode: image_generation

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

2. 프록시 시작

litellm --config config.yaml

# RUNNING on http://0.0.0.0:4000

3. 테스트

curl --location 'http://0.0.0.0:4000/v1/images/generations' \
--header 'Content-Type: application/json' \
--header "Authorization: Bearer ***" \
--data '{
    "model": "xinference-sd",
    "prompt": "A beautiful sunset over a calm ocean",
    "n": 1,
    "size": "1024x1024",
    "response_format": "url"
}'

고급 사용법 - 추가 파라미터 포함 (Advanced Usage - With Additional Parameters)

from litellm import image_generation
import os

os.environ['XINFERENCE_API_BASE'] = "http://127.0.0.1:9997/v1"

response = image_generation(
    model="xinference/stabilityai/stable-diffusion-3.5-large",
    prompt="A beautiful sunset over a calm ocean",
    n=1,                           # number of images
    size="1024x1024",             # image size
    response_format="b64_json",   # return format
)
print(response)

지원 이미지 생성 모델 (Supported Image Generation Models)

Xinference는 다양한 stable diffusion 모델을 지원해요. 몇 가지 예시:

지원되는 이미지 생성 모델의 전체 목록은 다음을 참고해 주세요: https://inference.readthedocs.io/en/latest/models/builtin/image/index.html

지원 모델 (Supported Models)

https://inference.readthedocs.io/en/latest/models/builtin/embedding/index.html 에 나열된 모든 모델이 지원돼요.

출처: 문서

본문

더 알아보기 (Learn more)

• Xinference 문서
• LiteLLM 이미지 생성
• LiteLLM 임베딩
• LiteLLM 공식 문서