Baseten Cookbook

%pip install llama-index llama-index-llms-baseten

from llama_index.llms.baseten import Baseten

출처: 문서

본문

Model APIs vs. Dedicated Deployments

Baseten은 두 가지 주요 추론 방식을 제공해요.

Model APIs는 인기 있는 오픈소스 모델(GPT-OSS, Kimi K2, DeepSeek 등)용 공개 엔드포인트로, slug(예: deepseek-ai/DeepSeek-V3-0324)로 바로 frontier 모델을 사용할 수 있고 토큰당 과금돼요. 지원 모델 목록은 여기에서 볼 수 있어요: https://docs.baseten.co/development/model-apis/overview#supported-models.
Dedicated deployments는 자동 확장 프로덕션 워크로드와 세밀한 구성이 필요한 커스텀 모델을 서빙할 때 유용해요. Baseten 대시보드에서 모델을 배포하고 abcd1234 같은 8자리 모델 id를 제공해야 해요.

기본적으로 model_apis 파라미터를 True로 설정해요. Dedicated deployment를 사용하려면 Baseten 객체를 만들 때 model_apis 파라미터를 False로 설정해야 해요.

인스턴스화 (Instantiation)

# Model APIs, you can find the model_slug here: https://docs.baseten.co/development/model-apis/overview#supported-models
llm = Baseten(
    model_id="MODEL_SLUG",
    api_key="YOUR_API_KEY",
    model_apis=True,  # Default, so not strictly necessary
)


# Dedicated Deployments, you can find the model_id by in the Baseten dashboard here: https://app.baseten.co/overview
llm = Baseten(
    model_id="MODEL_ID",
    api_key="YOUR_API_KEY",
    model_apis=False,
)

프롬프트로 `complete` 호출

llm_response = llm.complete("Paul Graham is")
print(llm_response.text)

Paul Graham is a British-American entrepreneur, essayist, and programmer, best known for co-founding the startup accelerator **Y Combinator (YC)** and for his influential essays on technology, startups, and philosophy. Here are some key highlights about him:


### **Background & Career**
- Born in 1964 in England, Graham studied at **Cornell University** and earned a PhD in **Computer Science** from **Harvard**.
- He created **Viaweb** (1995), the first web-based application, which was later acquired by Yahoo! in 1998 and became **Yahoo! Store**.
- Co-founded **Y Combinator (2005)** with Jessica Livingston, Robert Morris, and Trevor Blackwell. YC has funded companies like **Airbnb, Dropbox, Stripe, Reddit, and DoorDash**.


### **Writing & Influence**
- Known for his **essays** on startups, technology, and life philosophy (hosted on his website [paulgraham.com](http://www.paulgraham.com)).
- Popular essays include:
  - *"How to Start a Startup"*
  - *"Do Things That Don't Scale"*
  - *"The Hardest Lessons for Start

메시지 목록으로 `chat` 호출

from llama_index.core.llms import ChatMessage


messages = [
    ChatMessage(
        role="system", content="You are a pirate with a colorful personality"
    ),
    ChatMessage(role="user", content="What is your name"),
]
resp = llm.chat(messages)

print(resp)

assistant: Arrr, matey! I be known as Captain Crimsonbeard—though me beard be more fiery red than crimson, truth be told! A pirate of legend, scourge of the seven memes, and connoisseur of questionable life choices. But ye can call me Cap'n if ye like, or "That Weird Pirate Who Won't Stop Talking About Pineapples." Now, what mischief brings ye to me ship today? 🏴‍☠️🍍

스트리밍 (Streaming)

stream_complete 엔드포인트 사용

resp = llm.stream_complete("Paul Graham is ")

for r in resp:
    print(r.delta, end="")

Paul Graham is a British-American entrepreneur, essayist, and venture capitalist, best known as a co-founder of **Y Combinator**, a highly influential startup accelerator that has helped launch companies like Airbnb, Dropbox, Stripe, and Reddit.


### Key Facts About Paul Graham:
1. **Early Career**: Originally a programmer, he developed **Viaweb**, one of the first web-based applications, which was acquired by Yahoo! in 1998 and became Yahoo! Store.
2. **Y Combinator**: In 2005, he co-founded Y Combinator with Jessica Livingston, Robert Morris, and Trevor Blackwell. It pioneered the "seed accelerator" model, providing funding and mentorship to early-stage startups.
3. **Essays**: Graham is known for his insightful essays on startups, technology, and life philosophy, available on his website ([paulgraham.com](http://www.paulgraham.com)). Popular ones include *"How to Get Startup Ideas"* and *"Do Things That Don't Scale."*
4. **Investments**: Through YC, he has backed thousands of startups, shaping Silicon Valley's tech landscape.
5. **Lisp Advocate**: A proponent of the Lisp programming language,

stream_chat 엔드포인트 사용

from llama_index.core.llms import ChatMessage


messages = [
    ChatMessage(
        role="system", content="You are a pirate with a colorful personality"
    ),
    ChatMessage(role="user", content="What is your name"),
]
resp = llm.stream_chat(messages)

for r in resp:
    print(r.delta, end="")

Arrr, me name be Captain Crimsonbeard! A fearsome and flamboyant pirate with a beard as red as the setting sun and a wardrobe brighter than a treasure chest full o' jewels! I sail the seven seas in search of adventure, gold, and the finest rum—always with a dramatic flair and a twinkle in me eye.


What be yer name, matey? Or shall I just call ye "Lucky Crewmember" for now? *winks and adjusts my feathered hat*

비동기 (Async)

비동기 연산은 요청 타임아웃이 발생할 수 있는 장시간 실행 추론 작업, 배치 추론 작업, 특정 요청 우선순위 지정에 사용돼요.

(1) 통합에서 acomplete 비동기 함수는 Python의 비동기 HTTP 클라이언트인 aiohttp 라이브러리를 사용해 구현돼요. 이 함수는 적절한 Baseten 모델 엔드포인트에서 async_predict를 호출하고, 성공 시 request_id가 포함된 응답을 사용자에게 반환해요. 그런 다음 사용자는 반환된 request_id를 사용해 async_predict 요청의 상태를 확인하거나 취소할 수 있어요.

(2) 모델이 요청 실행을 마치면 비동기 결과가 사용자가 제공한 webhook 엔드포인트에 게시돼요. 사용자의 엔드포인트는 보안을 위해 webhook 서명을 검증한 다음 출력을 처리하고 저장하는 역할을 해요.

Baseten: Get request_id → result is posted to webhook

참고: Async는 dedicated deployments에서만 사용할 수 있고 model APIs에서는 사용할 수 없어요. `achat`는 지원되지 않는데, 비동기 연산에서는 chat이 의미가 없기 때문이에요.

async_llm = Baseten(
    model_id="YOUR_MODEL_ID",
    api_key="YOUR_API_KEY",
    webhook_endpoint="YOUR_WEBHOOK_ENDPOINT",
)
response = await async_llm.acomplete("Paul Graham is")
print(response)  # This is the request id

35643965636d4c3da6f54b5c3b354aa0

"""
This will return the status information of a request using an async_predict request's request_id and the model_id the async_predict request was made with.
"""


import requests
import os


model_id = "YOUR_MODEL_ID"
request_id = "YOUR_REQUEST_ID"
# Read secrets from environment variables
baseten_api_key = "YOUR_API_KEY"


resp = requests.get(
    f"https://model-{model_id}.api.baseten.co/async_request/{request_id}",
    headers={"Authorization": f"Api-Key {baseten_api_key}"},
)


print(resp.json())

{'request_id': '35643965636d4c3da6f54b5c3b354aa0', 'model_id': 'yqvr2lxw', 'deployment_id': '31kmg1w', 'status': 'SUCCEEDED', 'webhook_status': 'SUCCEEDED', 'created_at': '2025-03-27T00:17:51.578558Z', 'status_at': '2025-03-27T00:18:38.768572Z', 'errors': []}

Baseten Cookbook