Docker Model Runner

Docker Model Runner

Docker Model Runner를 사용하면 Docker Desktop으로 대형 언어 모델을 로컬에서 실행할 수 있어요.

모든 Docker Model Runner 모델을 지원해요. completion 요청 시 docker_model_runner/ 접두사로 설정하기만 하면 돼요.

출처: 문서

본문

개요 (Overview)

빠른 시작 (Quick Start)

Docker Model Runner는 AI 모델을 로컬에서 실행하게 해주는 Docker Desktop 기능이에요. 다른 로컬 솔루션보다 더 나은 성능을 제공하면서 OpenAI 호환성을 유지해요.

설치 (Installation)

1. Docker Desktop 설치
2. Docker Desktop 설정에서 Docker Model Runner 활성화
3. Docker Desktop을 통해 원하는 모델 다운로드

환경 변수 (Environment Variables)

os.environ["DOCKER_MODEL_RUNNER_API_BASE"] = "http://localhost:22088/engines/llama.cpp"  # Optional - defaults to this
os.environ["DOCKER_MODEL_RUNNER_API_KEY"] = "dummy-key"  # Optional - Docker Model Runner may not require auth for local instances

참고:

• Docker Model Runner는 일반적으로 로컬에서 실행되며 인증이 필요하지 않을 수 있어요. LiteLLM은 키가 제공되지 않으면 기본적으로 dummy 키를 사용해요.
• API base에는 엔진 경로(예: /engines/llama.cpp)가 포함되어야 해요.

API Base 구조

Docker Model Runner는 고유한 URL 구조를 사용해요:

http://model-runner.docker.internal/engines/{engine}/v1/chat/completions

여기서 {engine}은 사용하려는 엔진(일반적으로 llama.cpp)이에요.

중요: 엔진을 모델 이름이 아닌 api_base URL에 지정하세요:

• ✅ 정확: api_base="http://localhost:22088/engines/llama.cpp", model="docker_model_runner/llama-3.1"
• ❌ 잘못됨: api_base="http://localhost:22088", model="docker_model_runner/llama.cpp/llama-3.1"

LiteLLM Python SDK 사용법

비스트리밍 (Non-streaming)

import os
import litellm
from litellm import completion

# Specify the engine in the api_base URL
os.environ["DOCKER_MODEL_RUNNER_API_BASE"] = "http://localhost:22088/engines/llama.cpp"

messages = [{"content": "Hello, how are you?", "role": "user"}]

# Docker Model Runner call
response = completion(
    model="docker_model_runner/llama-3.1",
    messages=messages
)
print(response)

스트리밍 (Streaming)

import os
import litellm
from litellm import completion

# Specify the engine in the api_base URL
os.environ["DOCKER_MODEL_RUNNER_API_BASE"] = "http://localhost:22088/engines/llama.cpp"

messages = [{"content": "Hello, how are you?", "role": "user"}]

# Docker Model Runner call with streaming
response = completion(
    model="docker_model_runner/llama-3.1",
    messages=messages,
    stream=True
)

for chunk in response:
    print(chunk)

사용자 지정 API Base와 엔진

import litellm
from litellm import completion

messages = [{"content": "Hello, how are you?", "role": "user"}]

# Specify the engine in the api_base URL
# Using a different host and engine
response = completion(
    model="docker_model_runner/llama-3.1",
    messages=messages,
    api_base="http://model-runner.docker.internal/engines/llama.cpp"
)
print(response)

다른 엔진 사용

import litellm
from litellm import completion

messages = [{"content": "Hello, how are you?", "role": "user"}]

# To use a different engine, specify it in the api_base
# For example, if Docker Model Runner supports other engines:
response = completion(
    model="docker_model_runner/mistral-7b",
    messages=messages,
    api_base="http://localhost:22088/engines/custom-engine"
)
print(response)

LiteLLM Proxy 사용법

config.yaml에 다음을 추가:

model_list:
  - model_name: llama-3.1
    litellm_params:
      model: docker_model_runner/llama-3.1
      api_base: http://localhost:22088/engines/llama.cpp
  - model_name: mistral-7b
    litellm_params:
      model: docker_model_runner/mistral-7b
      api_base: http://localhost:22088/engines/llama.cpp

LiteLLM Proxy 서버 시작:

litellm --config config.yaml
# RUNNING on http://0.0.0.0:4000

OpenAI SDK (비스트리밍):

from openai import OpenAI

# Initialize client with your proxy URL
client = OpenAI(
    base_url="http://localhost:4000",  # Your proxy URL
    api_key="your-proxy-api-key"  # Your proxy API key
)

# Non-streaming response
response = client.chat.completions.create(
    model="llama-3.1",
    messages=[{"role": "user", "content": "hello from litellm"}]
)
print(response.choices[0].message.content)

OpenAI SDK (스트리밍):

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:4000",  # Your proxy URL
    api_key="your-proxy-api-key"  # Your proxy API key
)

# Streaming response
response = client.chat.completions.create(
    model="llama-3.1",
    messages=[{"role": "user", "content": "hello from litellm"}],
    stream=True
)

for chunk in response:
    if chunk.choices[0].delta.content is not None:
        print(chunk.choices[0].delta.content, end="")

LiteLLM SDK (비스트리밍):

import litellm

# Configure LiteLLM to use your proxy
response = litellm.completion(
    model="litellm_proxy/llama-3.1",
    messages=[{"role": "user", "content": "hello from litellm"}],
    api_base="http://localhost:4000",
    api_key="your-proxy-api-key"
)
print(response.choices[0].message.content)

LiteLLM SDK (스트리밍):

import litellm

response = litellm.completion(
    model="litellm_proxy/llama-3.1",
    messages=[{"role": "user", "content": "hello from litellm"}],
    api_base="http://localhost:4000",
    api_key="your-proxy-api-key",
    stream=True
)

for chunk in response:
    if hasattr(chunk.choices[0], 'delta') and chunk.choices[0].delta.content is not None:
        print(chunk.choices[0].delta.content, end="")

cURL (비스트리밍):

curl http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your-p...-key" \
  -d '{
    "model": "llama-3.1",
    "messages": [{"role": "user", "content": "hello from litellm"}]
  }'

cURL (스트리밍):

curl http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your-p...-key" \
  -d '{
    "model": "llama-3.1",
    "messages": [{"role": "user", "content": "hello from litellm"}],
    "stream": true
  }'

LiteLLM Proxy에 대한 자세한 내용은 LiteLLM Proxy 설명서를 참고하세요.

더 알아보기 (Learn more)

• LiteLLM 개요
• LiteLLM Python SDK 사용법
• LiteLLM Proxy Server 설정
• 지원되는 공급업체 전체 목록