Letta 통합

Letta 통합 (Letta Integration)

Letta(구 MemGPT)는 지속 메모리를 가진 상태 유지형(stateful) LLM 에이전트를 구축하는 프레임워크예요. 이 가이드는 LiteLLM SDK와 LiteLLM Proxy를 각각 Letta와 통합해, 메모리 지원 에이전트를 만들면서 여러 LLM 프로바이더를 쓰는 방법을 알려드려요.

출처: 문서

본문

Letta란? (What is Letta?)

Letta를 사용하면 다음을 할 수 있는 LLM 에이전트를 만들 수 있어요:

• 대화 전반에 걸친 장기 메모리 유지
• 도구 상호작용을 위한 함수 호출 사용
• 대형 컨텍스트 윈도우를 효율적으로 처리
• 에이전트 상태와 메모리 영속화

사전 준비 (Prerequisites)

uv add letta litellm

퀵 스타트 (Quick Start)

LiteLLM Proxy로 시작하기

1. LiteLLM Proxy 시작 — 우선 프록시용 설정 파일을 만드세요:

# config.yaml
model_list:
  - model_name: gpt-5.6-terra
    litellm_params:
      model: openai/gpt-5.6-terra
      api_key: os.environ/OPENAI_API_KEY
  - model_name: claude-sonnet-5
    litellm_params:
      model: anthropic/claude-sonnet-5
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: gpt-5.6-luna
    litellm_params:
      model: azure/gpt-5.6-luna
      api_key: os.environ/AZURE_API_KEY
      api_base: os.environ/AZURE_API_BASE
      api_version: "2023-07-01-preview"

프록시 시작: litellm --config config.yaml --port 4000

2. Letta를 LiteLLM Proxy에 연결

import letta
from letta import create_client

# Configure Letta to use LiteLLM proxy
client = create_client()

# Configure the LLM endpoint
client.set_default_llm_config(
    model="gpt-5.6-terra",  # This should match a model from your LiteLLM config
    model_endpoint_type="openai",
    model_endpoint="http://localhost:4000",  # Your LiteLLM proxy URL
    context_window=8192
)

# Configure embedding endpoint (optional)
client.set_default_embedding_config(
    embedding_endpoint_type="openai",
    embedding_endpoint="http://localhost:4000",
    embedding_model="text-embedding-ada-002"
)

LiteLLM SDK로 시작하기

1. LiteLLM SDK 설정 — API 키를 설정하고 LiteLLM을 구성:

import os
import litellm

# Set your API keys
os.environ["OPENAI_API_KEY"] = "your-openai-key"
os.environ["ANTHROPIC_API_KEY"] = "your-anthropic-key"

# Optional: Configure default settings
litellm.set_verbose = True  # For debugging

2. Letta용 커스텀 LLM 래퍼 만들기

import letta
from letta import create_client
from letta.llm_api.llm_api_base import LLMConfig
import litellm
from typing import List, Dict, Any

class LiteLLMWrapper:
    def __init__(self, model: str):
        self.model = model

    def chat_completions_create(self, messages: List[Dict], **kwargs):
        # Use LiteLLM SDK for completion
        response = litellm.completion(
            model=self.model,
            messages=messages,
            **kwargs
        )
        return response

# Configure Letta with custom LiteLLM wrapper
client = create_client()

# Set up LLM configuration using direct SDK integration
llm_config = LLMConfig(
    model="gpt-5.6-terra",  # or "claude-sonnet-5", "azure/gpt-5.6-luna", etc.
    model_endpoint_type="openai",
    context_window=8192
)
client.set_default_llm_config(llm_config)

Letta 에이전트 만들고 사용하기

LiteLLM Proxy 사용:

import letta
from letta import create_client

# Create Letta client
client = create_client()

# Create a new agent
agent_state = client.create_agent(
    name="my-assistant",
    system="You are a helpful assistant with persistent memory.",
    llm_config=client.get_default_llm_config(),
    embedding_config=client.get_default_embedding_config()
)

# Send a message to the agent
response = client.user_message(
    agent_id=agent_state.id,
    message="Hi! My name is Alice and I love reading science fiction books."
)
print(f"Agent response: {response.messages[-1].text}")

# Send another message - the agent will remember previous context
response = client.user_message(
    agent_id=agent_state.id,
    message="What did I tell you about my interests?"
)
print(f"Agent response: {response.messages[-1].text}")

LiteLLM SDK 사용:

import letta
from letta import create_client
import litellm
import os

# Set up environment variables
os.environ["OPENAI_API_KEY"] = "your-openai-key"

# Create Letta client with LiteLLM integration
client = create_client()

# Create a new agent
agent_state = client.create_agent(
    name="my-assistant",
    system="You are a helpful assistant with persistent memory.",
    llm_config=client.get_default_llm_config(),
    embedding_config=client.get_default_embedding_config()
)

# Send a message to the agent
response = client.user_message(
    agent_id=agent_state.id,
    message="Hi! My name is Alice and I love reading science fiction books."
)
print(f"Agent response: {response.messages[-1].text}")

# Send another message - the agent will remember previous context
response = client.user_message(
    agent_id=agent_state.id,
    message="What did I tell you about my interests?"
)
print(f"Agent response: {response.messages[-1].text}")

고급 설정 (Advanced Configuration)

에이전트마다 다른 모델 사용하기

LiteLLM Proxy 사용:

from letta import LLMConfig, EmbeddingConfig

# Create different LLM configurations pointing to your proxy
gpt_config = LLMConfig(
    model="gpt-5.6-terra",
    model_endpoint_type="openai",
    model_endpoint="http://localhost:4000",
    context_window=8192
)

claude_config = LLMConfig(
    model="claude-sonnet-5",
    model_endpoint_type="openai",  # Using OpenAI-compatible endpoint
    model_endpoint="http://localhost:4000",
    context_window=200000
)

# Create agents with different configurations
research_agent = client.create_agent(
    name="research-agent",
    system="You are a research assistant specialized in analysis.",
    llm_config=claude_config  # Use Claude for research tasks
)

creative_agent = client.create_agent(
    name="creative-agent", 
    system="You are a creative writing assistant.",
    llm_config=gpt_config  # Use gpt-5.6-terra for creative tasks
)

LiteLLM SDK 사용:

import os
import litellm
from letta import LLMConfig, EmbeddingConfig

# Set up API keys for different providers
os.environ["OPENAI_API_KEY"] = "your-openai-key"
os.environ["ANTHROPIC_API_KEY"] = "your-anthropic-key"

# Create different LLM configurations for direct SDK usage
gpt_config = LLMConfig(
    model="openai/gpt-5.6-terra",  # Using LiteLLM model format
    model_endpoint_type="openai",
    context_window=8192
)

claude_config = LLMConfig(
    model="anthropic/claude-sonnet-5",  # Using LiteLLM model format
    model_endpoint_type="openai",
    context_window=200000
)

# Create agents with different configurations
research_agent = client.create_agent(
    name="research-agent",
    system="You are a research assistant specialized in analysis.",
    llm_config=claude_config  # Use Claude for research tasks
)

creative_agent = client.create_agent(
    name="creative-agent", 
    system="You are a creative writing assistant.",
    llm_config=gpt_config  # Use gpt-5.6-terra for creative tasks
)

도구로 함수 호출하기 (Function Calling with Tools)

LiteLLM Proxy 사용:

# Define custom tools for your agent
def search_web(query: str) -> str:
    """Search the web for information"""
    # Your web search implementation
    return f"Search results for: {query}"

def save_note(content: str) -> str:
    """Save a note to persistent storage"""
    # Your note saving implementation
    return f"Note saved: {content}"

# Create agent with tools (using proxy endpoint)
agent_state = client.create_agent(
    name="research-assistant",
    system="You are a research assistant that can search the web and save notes.",
    llm_config=client.get_default_llm_config(),
    embedding_config=client.get_default_embedding_config(),
    tools=[search_web, save_note]
)

# The agent can now use these tools
response = client.user_message(
    agent_id=agent_state.id,
    message="Search for recent developments in AI and save important findings."
)

LiteLLM SDK 사용:

import litellm
import os

# Set up API keys
os.environ["OPENAI_API_KEY"] = "your-openai-key"

# Define custom tools for your agent
def search_web(query: str) -> str:
    """Search the web for information"""
    # Your web search implementation
    return f"Search results for: {query}"

def save_note(content: str) -> str:
    """Save a note to persistent storage"""
    # Your note saving implementation
    return f"Note saved: {content}"

# Create agent with tools (using LiteLLM SDK directly)
agent_state = client.create_agent(
    name="research-assistant",
    system="You are a research assistant that can search the web and save notes.",
    llm_config=LLMConfig(
        model="openai/gpt-5.6-terra",  # Direct model specification
        model_endpoint_type="openai",
        context_window=8192
    ),
    embedding_config=client.get_default_embedding_config(),
    tools=[search_web, save_note]
)

# The agent can now use these tools
response = client.user_message(
    agent_id=agent_state.id,
    message="Search for recent developments in AI and save important findings."
)

인증 (Authentication)

LiteLLM Proxy에 인증이 필요하다면:

import os
from letta import LLMConfig

# Set up authenticated configuration
llm_config = LLMConfig(
    model="gpt-5.6-terra",
    model_endpoint_type="openai",
    model_endpoint="http://localhost:4000",
    model_wrapper="openai",
    context_window=8192
)

# If using API keys with your proxy
os.environ["OPENAI_API_KEY"] = "your-litellm-proxy-api-key"
client = create_client()
client.set_default_llm_config(llm_config)

인증이 활성화된 프록시의 경우:

# config.yaml with auth
general_settings:
  master_key: "your-master-key"
model_list:
  - model_name: gpt-5.6-terra
    litellm_params:
      model: openai/gpt-5.6-terra
      api_key: os.environ/OPENAI_API_KEY

# Configure Letta with authenticated proxy
llm_config = LLMConfig(
    model="gpt-5.6-terra",
    model_endpoint_type="openai",
    model_endpoint="http://localhost:4000",
    context_window=8192,
    api_key="your-master-key"  # Proxy master key
)

LiteLLM SDK를 쓰면 프로바이더 API 키를 직접 설정할 수 있어요:

import os
import litellm

# Set up API keys for different providers
os.environ["OPENAI_API_KEY"] = "your-openai-api-key"
os.environ["ANTHROPIC_API_KEY"] = "your-anthropic-api-key"
os.environ["AZURE_API_KEY"] = "your-azure-api-key"
os.environ["AZURE_API_BASE"] = "https://your-resource.openai.azure.com"
os.environ["AZURE_API_VERSION"] = "2023-07-01-preview"

# Optional: Configure default settings
litellm.api_key = os.environ.get("OPENAI_API_KEY")  # Default key
litellm.set_verbose = True  # For debugging

# Use in Letta configuration
from letta import LLMConfig
llm_config = LLMConfig(
    model="openai/gpt-5.6-terra",  # Will use OPENAI_API_KEY automatically
    model_endpoint_type="openai",
    context_window=8192
)

# Or for Azure
azure_config = LLMConfig(
    model="azure/gpt-5.6-luna",
    model_endpoint_type="openai",
    context_window=4096
)

로드 밸런싱과 폴백 (Load Balancing and Fallbacks)

LiteLLM 프록시의 로드 밸런싱·폴백 기능은 Letta와 함께 동작해요:

# config.yaml with fallbacks
model_list:
  - model_name: gpt-5.6-terra
    litellm_params:
      model: openai/gpt-5.6-terra
      api_key: os.environ/OPENAI_API_KEY
      tpm: 40000
      rpm: 500
  - model_name: gpt-5.6-terra  # Same model name for fallback
    litellm_params:
      model: azure/gpt-5.6-terra
      api_key: os.environ/AZURE_API_KEY
      api_base: os.environ/AZURE_API_BASE
      api_version: "2023-07-01-preview"
      tpm: 80000
      rpm: 800
router_settings:
  routing_strategy: "usage-based-routing"
  fallbacks: [ { "gpt-5.6-terra": [ "azure/gpt-5.6-terra" ] } ]

프록시가 Letta를 위해 라우팅·로드 밸런싱·폴백을 모두 투명하게 처리해요.

LiteLLM SDK로는 프로그래밍 방식으로 라우팅·폴백을 설정할 수 있어요:

import litellm
from litellm import Router

# Configure router with multiple models
router = Router(
    model_list=[
        {
            "model_name": "gpt-5.6-terra",
            "litellm_params": {
                "model": "openai/gpt-5.6-terra",
                "api_key": os.environ["OPENAI_API_KEY"]
            },
            "tpm": 40000,
            "rpm": 500
        },
        {
            "model_name": "gpt-5.6-terra",  # Same name for fallback
            "litellm_params": {
                "model": "azure/gpt-5.6-terra",
                "api_key": os.environ["AZURE_API_KEY"],
                "api_base": os.environ["AZURE_API_BASE"],
                "api_version": "2023-07-01-preview"
            },
            "tpm": 80000,
            "rpm": 800
        }
    ],
    fallbacks=[{ "gpt-5.6-terra": [ "azure/gpt-5.6-terra" ] }],
    routing_strategy="usage-based-routing"
)

# Create custom completion function for Letta
def custom_completion(messages, model="gpt-5.6-terra", **kwargs):
    return router.completion(model=model, messages=messages, **kwargs)

# Use with Letta by monkey-patching or custom wrapper
litellm.completion = custom_completion

모니터링과 관측성 (Monitoring and Observability)

프록시를 통해 Letta 에이전트의 LLM 사용량을 추적하도록 로깅을 활성화해요:

# config.yaml with logging
model_list:
  # ... your models
litellm_settings:
  success_callback: ["langfuse"]  # or other observability tools
environment_variables:
  LANGFUSE_PUBLIC_KEY: "your-key"
  LANGFUSE_SECRET_KEY: "your-secret"

프록시 대시보드에서 메트릭을 확인하세요:

# Start proxy with UI
litellm --config config.yaml --port 4000 --detailed_debug

SDK 통합에서 직접 옵저버빌리티를 설정할 수도 있어요:

import litellm
import os

# Configure observability callbacks
os.environ["LANGFUSE_PUBLIC_KEY"] = "your-key"
os.environ["LANGFUSE_SECRET_KEY"] = "your-secret"

# Set global callbacks
litellm.success_callback = ["langfuse"]
litellm.failure_callback = ["langfuse"]

# Optional: Set up custom logging
litellm.set_verbose = True

# Create custom completion wrapper with logging
def logged_completion(messages, model="gpt-5.6-terra", **kwargs):
    try:
        response = litellm.completion(model=model, messages=messages, **kwargs)
        # Custom logging logic here if needed
        return response
    except Exception as e:
        # Custom error handling
        print(f"LLM call failed: {e}")
        raise

# Use in Letta configuration
litellm.completion = logged_completion

예시: 멀티 에이전트 시스템 (Example: Multi-Agent System)

LiteLLM Proxy 사용:

import letta
from letta import create_client, LLMConfig

client = create_client()

# Create specialized agents using proxy endpoints
agents = {}

# Research agent using Claude for analysis
agents['researcher'] = client.create_agent(
    name="researcher",
    system="You are a research specialist. Analyze information thoroughly.",
    llm_config=LLMConfig(
        model="claude-sonnet-5",
        model_endpoint="http://localhost:4000",
        model_endpoint_type="openai"
    )
)

# Writer agent using gpt-5.6-terra for content creation
agents['writer'] = client.create_agent(
    name="writer",
    system="You are a content writer. Create engaging, well-structured content.",
    llm_config=LLMConfig(
        model="gpt-5.6-terra",
        model_endpoint="http://localhost:4000", 
        model_endpoint_type="openai"
    )
)

# Coordinator workflow
def research_and_write_workflow(topic: str):
    # Research phase
    research_response = client.user_message(
        agent_id=agents['researcher'].id,
        message=f"Research the topic: {topic}. Provide key insights and data."
    )
    
    research_results = research_response.messages[-1].text
    
    # Writing phase
    write_response = client.user_message(
        agent_id=agents['writer'].id,
        message=f"Based on this research: {research_results}\n\nWrite an article about {topic}."
    )
    
    return write_response.messages[-1].text

# Execute workflow
article = research_and_write_workflow("The future of AI in healthcare")
print(article)

LiteLLM SDK 사용:

import letta
from letta import create_client, LLMConfig
import litellm
import os

# Set up environment
os.environ["OPENAI_API_KEY"] = "your-openai-key"
os.environ["ANTHROPIC_API_KEY"] = "your-anthropic-key"

client = create_client()

# Create specialized agents using direct SDK models
agents = {}

# Research agent using Claude for analysis
agents['researcher'] = client.create_agent(
    name="researcher",
    system="You are a research specialist. Analyze information thoroughly.",
    llm_config=LLMConfig(
        model="anthropic/claude-sonnet-5",
        model_endpoint_type="openai"
    )
)

# Writer agent using gpt-5.6-terra for content creation
agents['writer'] = client.create_agent(
    name="writer",
    system="You are a content writer. Create engaging, well-structured content.",
    llm_config=LLMConfig(
        model="openai/gpt-5.6-terra",
        model_endpoint_type="openai"
    )
)

# Cost-conscious agent using gpt-5.6-luna
agents['reviewer'] = client.create_agent(
    name="reviewer",
    system="You are an editor. Review and improve content quality.",
    llm_config=LLMConfig(
        model="openai/gpt-5.6-luna",
        model_endpoint_type="openai"
    )
)

# Enhanced workflow with multiple agents
def enhanced_workflow(topic: str):
    # Research phase
    research_response = client.user_message(
        agent_id=agents['researcher'].id,
        message=f"Research the topic: {topic}. Provide key insights and data."
    )
    
    research_results = research_response.messages[-1].text
    
    # Writing phase
    write_response = client.user_message(
        agent_id=agents['writer'].id,
        message=f"Based on this research: {research_results}\n\nWrite an article about {topic}."
    )
    
    draft_article = write_response.messages[-1].text
    
    # Review phase
    review_response = client.user_message(
        agent_id=agents['reviewer'].id,
        message=f"Please review and improve this article:\n\n{draft_article}"
    )
    
    return review_response.messages[-1].text

# Execute enhanced workflow
article = enhanced_workflow("The future of AI in healthcare")
print(article)

모범 사례 (Best Practices)

• 모델 선택: 작업에 맞는 모델 사용 — 분석·추론에는 Claude, 창의적 작업엔 gpt-5.6-terra, 단순 상호작용엔 gpt-5.6-luna
• 프록시 설정: 적절한 rate limit과 타임아웃 설정, 안정성을 위한 폴백 사용, 운영 환경에선 인증 활성화
• 메모리 관리: Letta가 메모리를 자동 처리하지만, 큰 컨텍스트에선 사용량을 모니터링
• 비용 최적화: 프록시의 예산 기능으로 비용 제어, 사용자·팀별 rate limiting, 대시보드에서 토큰 사용량 모니터링
• 모니터링: 에이전트 성능·토큰 사용량 추적을 위해 옵저버빌리티 활성화
• 오류 처리: 재시도로 견고한 오류 처리 구현 — litellm.num_retries = 3, litellm.request_timeout = 60
• 비용 관리: 비핵심 작업엔 저렴한 모델 사용, 토큰 카운팅·예산 구현, 적절한 응답 캐싱
• 성능: 동시 요청에 비동기 사용, 연결 풀링 구현, 응답 시간 모니터링
• 보안: API 키를 안전하게 저장(환경 변수), 키 정기 교체, rate limiting 구현

문제 해결 (Troubleshooting)

연결 문제

# Test your LiteLLM proxy
curl -X POST http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-5.6-terra",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

설정 디버깅

# Enable verbose logging
import logging
logging.basicConfig(level=logging.DEBUG)

# Test Letta configuration
client = create_client()
print(client.get_default_llm_config())

일반적인 프록시 문제

• 포트 충돌: 포트 4000이 사용 중이지 않은지 확인
• 모델 없음: 모델 이름이 config.yaml과 일치하는지 확인
• 인증 오류: master key 설정 확인
• Rate limiting: 프록시 로그에서 rate limit 히트 모니터링

API 키 문제

import os
import litellm

# Check if API keys are set
print("OpenAI Key:", os.environ.get("OPENAI_API_KEY", "Not set"))
print("Anthropic Key:", os.environ.get("ANTHROPIC_API_KEY", "Not set"))

# Test direct LiteLLM call
try:
    response = litellm.completion(
        model="openai/gpt-5.6-luna",
        messages=[{"role": "user", "content": "Hello"}]
    )
    print("LiteLLM working:", response.choices[0].message.content)
except Exception as e:
    print("LiteLLM error:", e)

설정 디버깅

# Enable verbose logging
litellm.set_verbose = True

# Test model availability
models = ["openai/gpt-5.6-terra", "anthropic/claude-sonnet-5"]
for model in models:
    try:
        response = litellm.completion(
            model=model,
            messages=[{"role": "user", "content": "Test"}],
            max_tokens=10
        )
        print(f"✓ {model} working")
    except Exception as e:
        print(f"✗ {model} failed: {e}")

일반적인 SDK 문제

• Import 오류: uv add litellm letta가 실행됐는지 확인
• 모델 형식: provider/model 형식 사용 (예: openai/gpt-5.6-terra)
• API 키 형식: 프로바이더마다 키 형식이 다름
• Rate limits: 재시도에 지수 백오프(exponential backoff) 구현

리소스 (Resources)

• Letta Documentation
• LiteLLM Proxy Documentation
• LiteLLM SDK Documentation
• Function Calling 가이드
• Observability 설정
• Router 설정

더 알아보기 (Learn more)

• Integrations
• Function Calling
• Router
• Observability