Llamafile

Llamafile

LiteLLM은 Llamafile의 모든 모델을 지원해요.

출처: 문서

본문

개요 (Overview)

빠른 시작 (Quick Start)

litellm.completion 사용 (OpenAI 호환 엔드포인트 호출)

llamafile은 chat completions용 OpenAI 호환 엔드포인트를 제공해요. LiteLLM으로 호출하려면 completion 호출에 다음을 추가하세요:

• model="llamafile/<your-llamafile-model-name>"
• api_base = "your-hosted-llamafile"

import litellm

response = litellm.completion(
    model="llamafile/mistralai/mistral-7b-instruct-v0.2",  # pass the llamafile model name for completeness
    messages=messages,
    api_base="http://localhost:8080/v1",
    temperature=0.2,
    max_tokens=80
)
print(response)

LiteLLM Proxy Server 사용법 (OpenAI 호환 엔드포인트 호출)

config.yaml 수정:

model_list:
  - model_name: my-model
    litellm_params:
      model: llamafile/mistralai/mistral-7b-instruct-v0.2  # add llamafile/ prefix to route as OpenAI provider
      api_base: http://localhost:8080/v1  # add api base for OpenAI compatible provider

Proxy 시작:

$ litellm --config /path/to/config.yaml

LiteLLM Proxy Server로 요청:

OpenAI Python v1.0.0+:

import openai

client = openai.OpenAI(
    api_key="sk-<your-litellm-api-key>",  # pass litellm proxy key, if you're using virtual keys
    base_url="http://0.0.0.0:4000"  # litellm-proxy-base url
)

response = client.chat.completions.create(
    model="my-model",
    messages = [
        {
            "role": "user",
            "content": "what llm are you"
        }
    ],
)
print(response)

curl:

curl --location 'http://0.0.0.0:4000/chat/completions' \
  --header "Authorization: Bearer ***" \
  --header 'Content-Type: application/json' \
  --data '{
    "model": "my-model",
    "messages": [
      {
        "role": "user",
        "content": "what llm are you"
      }
    ],
  }'

Embedding

from litellm import embedding
import os

os.environ["LLAMAFILE_API_BASE"] = "http://localhost:8080/v1"
embedding = embedding(model="llamafile/sentence-transformers/all-MiniLM-L6-v2", input=["Hello world"])
print(embedding)

Proxy 설정:

model_list:
  - model_name: my-model
    litellm_params:
      model: llamafile/sentence-transformers/all-MiniLM-L6-v2  # add llamafile/ prefix to route as OpenAI provider
      api_base: http://localhost:8080/v1  # add api base for OpenAI compatible provider

$ litellm --config /path/to/config.yaml
# RUNNING on http://0.0.0.0:4000

curl -L -X POST 'http://0.0.0.0:4000/embeddings' \
  -H "Authorization: Bearer ***" \
  -H 'Content-Type: application/json' \
  -d '{"input": ["hello world"], "model": "my-model"}'

더 알아보기 (Learn more)

• LiteLLM 개요
• LiteLLM Python SDK 사용법
• LiteLLM Proxy Server 설정
• 지원되는 공급업체 전체 목록