Ollama
Ollama
LiteLLM에서 Ollama의 모든 모델을 사용하는 방법을 알아봐요. JSON 모드, 도구 호출, FIM, 비전 모델까지 지원해요.
출처: 문서
본문
LiteLLM은 Ollama의 모든 모델을 지원해요.
info: 더 나은 응답을 위해
ollama_chat을 사용할 것을 권장해요.
사전 요구사항
ollama 서버가 실행 중인지 확인하세요.
사용 예시
from litellm import completion
response = completion(
model="ollama/llama2",
messages=[{ "content": "respond in 20 words. who are you?","role": "user"}],
api_base="http://localhost:11434"
)
print(response)
사용 예시 - 스트리밍
from litellm import completion
response = completion(
model="ollama/llama2",
messages=[{ "content": "respond in 20 words. who are you?","role": "user"}],
api_base="http://localhost:11434",
stream=True
)
print(response)
for chunk in response:
print(chunk['choices'][0]['delta'])
사용 예시 - 스트리밍 + Acompletion
ollama acompletion을 스트리밍으로 사용하려면 async_generator를 설치하세요.
uv add async_generator
async def async_ollama():
response = await litellm.acompletion(
model="ollama/llama2",
messages=[{ "content": "what's the weather" ,"role": "user"}],
api_base="http://localhost:11434",
stream=True
)
async for chunk in response:
print(chunk)
# call async_ollama
import asyncio
asyncio.run(async_ollama())
사용 예시 - JSON 모드
ollama JSON 모드를 사용하려면 litellm.completion()에 format="json"을 전달해요.
from litellm import completion
response = completion(
model="ollama/llama2",
messages=[
{
"role": "user",
"content": "respond in json, what's the weather"
}
],
max_tokens=10,
format = "json"
)
사용 예시 - 도구 호출
ollama 도구 호출을 사용하려면 litellm.completion()에 tools=[{..}]를 전달해요.
from litellm import completion
import litellm
## [OPTIONAL] REGISTER MODEL - not all ollama models support function calling, litellm defaults to json mode tool calls if native tool calling not supported.
# litellm.register_model(model_cost={
# "ollama_chat/llama3.1": {
# "supports_function_calling": true
# },
# })
tools = [
{
"type": "function",
"function": {
"name": "get_current_weather",
"description": "Get the current weather in a given location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city and state, e.g. San Francisco, CA",
},
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
},
"required": ["location"],
},
}
}
]
messages = [{"role": "user", "content": "What's the weather like in Boston today?"}]
response = completion(
model="ollama_chat/llama3.1",
messages=messages,
tools=tools
)
config.yaml:
model_list:
- model_name: "llama3.1"
litellm_params:
model: "ollama_chat/llama3.1"
keep_alive: "8m" # Optional: Overrides default keep_alive, use -1 for Forever
model_info:
supports_function_calling: true
Proxy 시작:
litellm --config /path/to/config.yaml
테스트:
curl -X POST 'http://0.0.0.0:4000/chat/completions' \
-H 'Content-Type: application/json' \
-H "Authorization: Bearer ***" \
-d '{
"model": "llama3.1",
"messages": [
{
"role": "user",
"content": "What'\''s the weather like in Boston today?"
}
],
"tools": [
{
"type": "function",
"function": {
"name": "get_current_weather",
"description": "Get the current weather in a given location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city and state, e.g. San Francisco, CA"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"]
}
},
"required": ["location"]
}
}
}
],
"tool_choice": "auto",
"stream": true
}'
/v1/completions에서 Ollama FIM 사용
LiteLLM은 /v1/completions 요청에서 Ollama의 /api/generate 엔드포인트 호출을 지원해요.
import litellm
litellm._turn_on_debug() # turn on debug to see the request
from litellm import completion
response = completion(
model="ollama/llama3.1",
prompt="Hello, world!",
api_base="http://localhost:11434"
)
print(response)
config.yaml:
model_list:
- model_name: "llama3.1"
litellm_params:
model: "ollama/llama3.1"
api_base: "http://localhost:11434"
Proxy 시작:
litellm --config /path/to/config.yaml --detailed_debug
# RUNNING ON http://0.0.0.0:4000
테스트:
from openai import OpenAI
client = OpenAI(
api_key="anything", # 👈 PROXY KEY (can be anything, if master_key not set)
base_url="http://0.0.0.0:4000" # 👈 PROXY BASE URL
)
response = client.completions.create(
model="ollama/llama3.1",
prompt="Hello, world!",
api_base="http://localhost:11434"
)
print(response)
ollama api/chat 사용
ollama 서버의 POST /api/chat로 요청을 보내려면 모델 접두사를 ollama_chat으로 설정해요.
from litellm import completion
response = completion(
model="ollama_chat/llama2",
messages=[{ "content": "respond in 20 words. who are you?","role": "user"}],
)
print(response)
Ollama 모델
Ollama 지원 모델: https://github.com/ollama/ollama
| 모델명 | 함수 호출 |
|---|---|
| Mistral | completion(model='ollama/mistral', messages, api_base="http://localhost:11434", stream=True) |
| Mistral-7B-Instruct-v0.1 | completion(model='ollama/mistral-7B-Instruct-v0.1', messages, api_base="http://localhost:11434", stream=False) |
| Mistral-7B-Instruct-v0.2 | completion(model='ollama/mistral-7B-Instruct-v0.2', messages, api_base="http://localhost:11434", stream=False) |
| Mixtral-8x7B-Instruct-v0.1 | completion(model='ollama/mistral-8x7B-Instruct-v0.1', messages, api_base="http://localhost:11434", stream=False) |
| Mixtral-8x22B-Instruct-v0.1 | completion(model='ollama/mixtral-8x22B-Instruct-v0.1', messages, api_base="http://localhost:11434", stream=False) |
| Llama2 7B | completion(model='ollama/llama2', messages, api_base="http://localhost:11434", stream=True) |
| Llama2 13B | completion(model='ollama/llama2:13b', messages, api_base="http://localhost:11434", stream=True) |
| Llama2 70B | completion(model='ollama/llama2:70b', messages, api_base="http://localhost:11434", stream=True) |
| Llama2 Uncensored | completion(model='ollama/llama2-uncensored', messages, api_base="http://localhost:11434", stream=True) |
| Code Llama | completion(model='ollama/codellama', messages, api_base="http://localhost:11434", stream=True) |
| Meta LLaMa3 8B | completion(model='ollama/llama3', messages, api_base="http://localhost:11434", stream=False) |
| Meta LLaMa3 70B | completion(model='ollama/llama3:70b', messages, api_base="http://localhost:11434", stream=False) |
| Orca Mini | completion(model='ollama/orca-mini', messages, api_base="http://localhost:11434", stream=True) |
| Vicuna | completion(model='ollama/vicuna', messages, api_base="http://localhost:11434", stream=True) |
| Nous-Hermes | completion(model='ollama/nous-hermes', messages, api_base="http://localhost:11434", stream=True) |
| Nous-Hermes 13B | completion(model='ollama/nous-hermes:13b', messages, api_base="http://localhost:11434", stream=True) |
| Wizard ... | (더 많은 모델은 Ollama 저장소 참고) |
JSON Schema 지원
from litellm import completion
response = completion(
model="ollama_chat/deepseek-r1",
messages=[{ "content": "respond in 20 words. who are you?","role": "user"}],
response_format={"type": "json_schema", "json_schema": {"schema": {"type": "object", "properties": {"name": {"type": "string"}}}}},
)
print(response)
config.yaml:
model_list:
- model_name: "deepseek-r1"
litellm_params:
model: "ollama_chat/deepseek-r1"
api_base: "http://localhost:11434"
Proxy 시작:
litellm --config /path/to/config.yaml
# RUNNING ON http://0.0.0.0:4000
테스트:
from pydantic import BaseModel
from openai import OpenAI
client = OpenAI(
api_key="anything", # 👈 PROXY KEY (can be anything, if master_key not set)
base_url="http://0.0.0.0:4000" # 👈 PROXY BASE URL
)
class Step(BaseModel):
explanation: str
output: str
class MathReasoning(BaseModel):
steps: list[Step]
final_answer: str
completion = client.beta.chat.completions.parse(
model="deepseek-r1",
messages=[
{"role": "system", "content": "You are a helpful math tutor. Guide the user through the solution step by step."},
{"role": "user", "content": "how can I solve 8x + 7 = -23"}
],
response_format=MathReasoning,
)
math_reasoning = completion.choices[0].message.parsed
Ollama 비전 모델
| 모델명 | 함수 호출 |
|---|---|
| llava | completion('ollama/llava', messages) |
Ollama 비전 모델 사용
ollama/llava를 OpenAI gpt-4-vision과 동일한 입출력 형식으로 호출해요.
LiteLLM은 url로 전달되는 다음 이미지 타입을 지원해요:
- Base64 인코딩 svg
요청 예시:
import litellm
response = litellm.completion(
model = "ollama/llava",
messages=[
{
"role": "user",
"content": [
{
"type": "text",
"text": "Whats in this image?"
},
{
"type": "image_url",
"image_url": {
"url": "iVBORw0KGgoAAAANSUhEUgAAAG0AAABmCAYAAADBPx+VAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAA3VSURBVHgB7Z27r0zdG8fX743i1bi1ikMoFMQloXRpKFFIqI7LH4BEQ+NWIkjQuSWCRIEoULk0gsK1kCBI0IhrQVT7tz/7zZo888yz1r7MnDl7z5xvsjkzs2fP3uu71nNfa7lkAsm7d++Sffv2JbNmzUqcc8m0adOSzZs3Z+/XES4ZckAWJEGWPiCxjsQNLWmQsWjRIpMseaxcuTKpG/7HP27I8P79e7dq1ars/yL4/v27S0ejqwv+cUOGEGGpKHR37tzJCEpHV9tnT58+dXXCJDdECBE2Ojrqjh071hpNECjx4cMHVycM1Uhbv359B2F79ux25s6daxN/+pyRkRFXKyRDAqxEp4yMlDDzXG1NPnnyJKkThoK0VFd1ELZu3TrzXKxKfW7dMBQ6bcuWLW2v0VlHjx41z717927ba22U9APcw7Nnz1oGEPeL3m3p2mTAYYnFmMOMXybPPXv2bNIPpFZr1NHn4HMw0KRBjg9NuRw95s8PEcz/6DZELQd/09C9QGq5RsmSRybqkwHGjh07OsJSsYYm3ijPpyHzoiacg35MLdDSIS/O1yM778jOTwYUkKNHWUzUWaOsylE00MyI0fcnOwIdjvtNdW/HZwNLGg+sR1kMepSNJXmIwxBZiG8tDTpEZzKg0GItNsosY8USkxDhD0Rinuiko2gfL/RbiD2LZAjU9zKQJj8RDR0vJBR1/Phx9+PHj9Z7REF4nTZkxzX4LCXHrV271qXkBAPGfP/atWvu/PnzHe4C97F48eIsRLZ9+3a3f/9+87dwP1JxaF7/3r17ba+5l4EcaVo0lj3SBq5kGTJSQmLWMjgYNei2GPT1MuMqGTDEFHzeQSP2wi/jGnkmPJ/nhccs44jvDAxpVcxnq0F6eT8h4ni/iIWpR5lPyA6ETkNXoSukvpJAD3AsXLiwpZs49+fPn5ke4j10TqYvegSfn0OnafC+Tv9ooA/JPkgQysqQNBzagXY55nO/oa1F7qvIPWkRL12WRpMWUvpVDYmxAPehxWSe8ZEXL20sadYIozfmNch4QJPAfeJgW3rNsnzphBKNJM2KKODo1rVOMRYik5ETy3ix4qWNI81qAAirizgMIc+yhTytx0JWZuNI03qsrgWlGtwjoS9XwgUhWGyhUaRZZQNNIEwCiXD16tXcAHUs79co0vSD8rrJCIW98pzvxpAWyyo3HYwqS0+H0BjStClcZJT5coMm6D2LOF8TolGJtK9fvyZpyiC5ePFi9nc/oJU4eiEP0jVoAnHa9wyJycITMP78+eMeP37sXrx44d6+fdt6f82aNdkx1pg9e3Zb5W+RSRE+n+VjksQWifvVaTKFhn5O8my63K8Qabdv33b379/PiAP//vuvW7BggZszZ072/+TJk91YgkafPn166zXB1rQHFvouAWHq9z3SEevSUerqCn2/dDCeta2jxYbr69evk4MHDyY7d+7MjhMnTiTPnz9Pfv/+nfQT2ggpO2dMF8cghuoM7Ygj5iWCqRlGFml0QC/ftGmTmzt3rmsaKDsgBSPh0/8yPeLLBihLkOKJc0jp8H8vUzcxIA1k6QJ/c78tWEyj5P3o4u9+jywNPdJi5rAH9x0KHcl4Hg570eQp3+vHXGyrmEeigzQsQsjavXt38ujRo44LQuDDhw+TW7duRS1HGgMxhNXHgflaNTOsHyKvHK5Ijo2jbFjJBQK9YwFd6RVMzfgRBmEfP37suBBm/p49e1qjEP2mwTViNRo0VJWH1deMXcNK08uUjVUu7s/zRaL+oLNxz1bpANco4npUgX4G2eFbpDFyQoQxojBCpEGSytmOH8qrH5Q9vuzD6ofQylkCUmh8DBAr+q8JCy"
}
}
]
}
],
)
print(response)
LiteLLM/Ollama Docker 이미지
Ollama의 경우 LiteLLM은 로컬 LLM(llama2, mistral, codellama)용 OpenAI API 호환 서버 Docker 이미지를 제공해요.
로컬 LLM용 OpenAI API 호환 서버 - llama2, mistral, codellama
빠른 시작:
docker pull litellm/ollama
docker run --name ollama litellm/ollama
서버 컨테이너 테스트
docker 컨테이너에서 test.py 파일을 python3 test.py로 실행하세요.
이 서버에 요청 보내기
import openai
api_base = f"http://0.0.0.0:4000" # base url for server
openai.api_base = api_base
openai.api_key = "temp-key"
print(openai.api_base)
print(f'LiteLLM: response from proxy with streaming')
response = openai.chat.completions.create(
model="ollama/llama2",
messages = [
{
"role": "user",
"content": "this is a test request, acknowledge that you got it"
}
],
stream=True
)
for chunk in response:
print(f'LiteLLM: streaming response from proxy {chunk}')
이 서버의 응답
{
"object": "chat.completion",
"choices": [
{
"finish_reason": "stop",
"index": 0,
"message": {
"content": " Hello! I acknowledge receipt of your test request. Please let me know if there's anything else I can assist you with.",
"role": "assistant",
"logprobs": null
}
}
],
"id": "chatcmpl-403d5a85-2631-4233-92cb-01e6dffc3c39",
"created": 1696992706.619709,
"model": "ollama/llama2",
"usage": {
"prompt_tokens": 18,
"completion_tokens": 25,
"total_tokens": 43
}
}
Docker 컨테이너 호출 (host.docker.internal)
이 지침을 따르세요.
더 알아보기 (Learn more)
- Ollama 공식 문서
- Ollama 지원 모델 목록