관찰 유형

관찰 유형 (Observation Types)

Langfuse는 observation에 더 많은 맥락을 제공하고 효율적인 필터링을 위해 다양한 observation 유형을 지원해요. (Langfuse는 span을 observation이라 부르며, span은 그 자체로 하나의 특정 observation 유형이에요.)

출처: 문서

본문

사용 가능한 유형 (Available Types)

• event는 기본 구성 요소예요. trace에서 개별 이벤트를 추적하는 데 사용돼요.
• span은 trace에서 작업 단위의 지속 시간을 나타내는 범용 observation 유형이에요.
• generation은 프롬프트, 토큰 사용량, 비용을 포함해 AI 모델의 출력(generation)을 기록해요.
• agent는 애플리케이션 흐름을 결정하며, 예를 들어 LLM의 안내에 따라 도구를 사용할 수 있어요.
• tool은 함수나 API 호출(예: 날씨 API)처럼 무언가를 수행하는 단일 액션을 나타내요.
• chain은 retriever에서 LLM 호출로 컨텍스트를 전달하는 것처럼 서로 다른 애플리케이션 단계 간의 연결이에요.
• retriever는 상태를 변경하지 않고 조회만 하는 데이터 검색 단계를 나타내요. 예: 벡터 저장소, 데이터베이스, 다른 지식 소스 호출.
• evaluator는 LLM 출력의 관련성/정확성/유용성을 평가하는 함수를 나타내요.
• embedding은 임베딩을 생성하기 위한 LLM 호출이며, 모델, 토큰 사용량, 비용을 포함할 수 있어요.
• guardrail은 악성 콘텐츠나 jailbreak로부터 보호하는 컴포넌트예요.

observation 유형 사용 방법 (How to Use Observation Types)

에이전트 프레임워크와의 통합(integrations)은 observation 유형을 자동으로 설정해요. 예를 들어 langchain에서 메서드에 @tool을 표시하면 Langfuse observation 유형이 자동으로 tool로 설정돼요.

Langfuse SDK에서 애플리케이션의 observation 유형을 수동으로 설정할 수도 있어요. observation을 만들 때 Python의 as_type 파라미터나 TypeScript의 asType 파라미터를 원하는 observation 유형으로 설정하세요.

observation 유형은 Python SDK 버전 >= 3.3.1이 필요해요.

Context Managers (Python)

@observe 데코레이터 사용:

from langfuse import observe

# Agent workflow
@observe(as_type="agent")
def run_agent_workflow(query):
    # Agent reasoning and tool orchestration
    return process_with_tools(query)

# Tool calls
@observe(as_type="tool")
def call_weather_api(location):
    # External API call
    return weather_service.get_weather(location)

start_as_current_observation 또는 start_observation 메서드 호출:

from langfuse import get_client

langfuse = get_client()

# Start observation with specific type
with langfuse.start_as_current_observation(
    as_type="embedding",
    name="embedding-generation"
) as obs:
    embeddings = model.encode(["text to embed"])
    obs.update(output=embeddings)

# Start observation with specific type
transform_span = langfuse.start_observation(
    as_type="chain",
    name="transform-text"
)
transformed_text = transform_text(["text to transform"])
transform_span.update(output=transformed_text)

JavaScript/TypeScript SDK

observation 유형은 TypeScript SDK 버전 >= 4.0.0부터 사용 가능해요.

Context Managers

startActiveObservation을 asType 옵션과 함께 사용해 observation 유형을 지정하세요:

import { startActiveObservation } from "@langfuse/tracing";

// Agent workflow
await startActiveObservation(
  "agent-workflow",
  async (agentObservation) => {
    agentObservation.update({
      input: { query: "What's the weather in Paris?" },
      metadata: { strategy: "tool-calling" }
    });

    // Agent reasoning and tool orchestration
    const result = await processWithTools(query);
    agentObservation.update({ output: result });
  },
  { asType: "agent" }
);

// Tool call
await startActiveObservation(
  "weather-api-call",
  async (toolObservation) => {
    toolObservation.update({
      input: { location: "Paris", units: "metric" },
    });

    const weather = await weatherService.getWeather("Paris");
    toolObservation.update({ output: weather });
  },
  { asType: "tool" }
);

// Chain operation
await startActiveObservation(
  "retrieval-chain",
  async (chainObservation) => {
    chainObservation.update({
      input: { query: "AI safety principles" },
    });

    const docs = await retrieveDocuments(query);
    const context = await processDocuments(docs);
    chainObservation.update({ output: { context, documentCount: docs.length } });
  },
  { asType: "chain" }
);

다른 observation 유형의 예시:

// LLM Generation
await startActiveObservation(
  "llm-completion",
  async (generationObservation) => {
    generationObservation.update({
      input: [{ role: "user", content: "Explain quantum computing" }],
      model: "gpt-4",
    });

    const completion = await openai.chat.completions.create({
      model: "gpt-4",
      messages: [{ role: "user", content: "Explain quantum computing" }],
    });

    generationObservation.update({
      output: completion.choices[0].message.content,
      usageDetails: {
        input: completion.usage.prompt_tokens,
        output: completion.usage.completion_tokens,
      },
    });
  },
  { asType: "generation" }
);

// Embedding generation
await startActiveObservation(
  "text-embedding",
  async (embeddingObservation) => {
    const texts = ["Hello world", "How are you?"];
    embeddingObservation.update({
      input: texts,
      model: "text-embedding-ada-002",
    });

    const embeddings = await openai.embeddings.create({
      model: "text-embedding-ada-002",
      input: texts,
    });

    embeddingObservation.update({
      output: embeddings.data.map(e => e.embedding),
      usageDetails: { input: embeddings.usage.prompt_tokens },
    });
  },
  { asType: "embedding" }
);

// Document retrieval
await startActiveObservation(
  "vector-search",
  async (retrieverObservation) => {
    retrieverObservation.update({
      input: { query: "machine learning", topK: 5 },
    });

    const results = await vectorStore.similaritySearch(query, 5);
    retrieverObservation.update({
      output: results,
      metadata: { vectorStore: "pinecone", similarity: "cosine" },
    });
  },
  { asType: "retriever" }
);

Observe Wrapper

observe 래퍼를 asType 옵션과 함께 사용해 함수를 자동으로 트레이싱하세요:

import { observe, updateActiveObservation } from "@langfuse/tracing";

// Agent function
const runAgentWorkflow = observe(
  async (query: string) => {
    updateActiveObservation({
      metadata: { strategy: "react", maxIterations: 5 }
    });

    // Agent logic here
    return await processQuery(query);
  },
  {
    name: "agent-workflow",
    asType: "agent"
  }
);

// Tool function
const callWeatherAPI = observe(
  async (location: string) => {
    updateActiveObservation({
      metadata: { provider: "openweather", version: "2.5" }
    });

    return await weatherService.getWeather(location);
  },
  {
    name: "weather-tool",
    asType: "tool"
  }
);

// Evaluation function
const evaluateResponse = observe(
  async (question: string, answer: string) => {
    updateActiveObservation({
      metadata: { criteria: ["relevance", "accuracy", "completeness"] }
    });

    const score = await llmEvaluator.evaluate(question, answer);
    return { score, feedback: "Response is accurate and complete" };
  },
  {
    name: "response-evaluator",
    asType: "evaluator"
  }
);

다른 observation 유형의 추가 예시:

// Generation wrapper
const generateCompletion = observe(
  async (messages: any[], model: string = "gpt-4") => {
    updateActiveObservation({
      model,
      metadata: { temperature: 0.7, maxTokens: 1000 }
    }, { asType: "generation" });

    const completion = await openai.chat.completions.create({
      model,
      messages,
      temperature: 0.7,
      max_tokens: 1000,
    });

    updateActiveObservation({
      usageDetails: {
        input: completion.usage.prompt_tokens,
        output: completion.usage.completion_tokens,
      }
    }, { asType: "generation" });

    return completion.choices[0].message.content;
  },
  {
    name: "llm-completion",
    asType: "generation"
  }
);

// Chain wrapper
const processDocumentChain = observe(
  async (documents: string[]) => {
    updateActiveObservation({
      metadata: { documentCount: documents.length }
    });

    const summaries = await Promise.all(
      documents.map(doc => summarizeDocument(doc))
    );

    return await combineAndRank(summaries);
  },
  {
    name: "document-processing-chain",
    asType: "chain"
  }
);

// Guardrail wrapper
const contentModerationCheck = observe(
  async (content: string) => {
    updateActiveObservation({
      metadata: { provider: "openai-moderation", version: "stable" }
    });

    const moderation = await openai.moderations.create({
      input: content,
    });

    const flagged = moderation.results[0].flagged;
    updateActiveObservation({
      output: { flagged, categories: moderation.results[0].categories }
    });

    if (flagged) {
      throw new Error("Content violates usage policies");
    }

    return { safe: true, content };
  },
  {
    name: "content-guardrail",
    asType: "guardrail"
  }
);

Manual Observations

startObservation을 asType 옵션과 함께 사용해 observation을 수동 관리하세요:

import { startObservation } from "@langfuse/tracing";

// Agent observation
const agentSpan = startObservation(
  "multi-step-agent",
  {
    input: { task: "Book a restaurant reservation" },
    metadata: { agentType: "planning", tools: ["search", "booking"] }
  },
  { asType: "agent" }
)

// Nested tool calls within the agent
const searchTool = agentSpan.startObservation(
  "restaurant-search",
  {
    input: { location: "New York", cuisine: "Italian", date: "2024-01-15" }
  },
  { asType: "tool" }
);

searchTool.update({
  output: { restaurants: ["Mario's", "Luigi's"], count: 2 }
});
searchTool.end();

const bookingTool = agentSpan.startObservation(
  "make-reservation",
  {
    input: { restaurant: "Mario's", time: "7:00 PM", party: 4 }
  },
  { asType: "tool" }
);

bookingTool.update({
  output: { confirmed: true, reservationId: "RES123" }
});
bookingTool.end();

agentSpan.update({
  output: { success: true, reservationId: "RES123" }
});
agentSpan.end();

다른 observation 유형의 예시:

// Embedding observation
const embeddingObs = startObservation(
  "document-embedding",
  {
    input: ["Document 1 content", "Document 2 content"],
    model: "text-embedding-ada-002"
  },
  { asType: "embedding" }
);

const embeddings = await generateEmbeddings(documents);
embeddingObs.update({
  output: embeddings,
  usageDetails: { input: 150 }
});
embeddingObs.end();

// Retriever observation
const retrieverObs = startObservation(
  "semantic-search",
  {
    input: { query: "What is machine learning?", topK: 10 },
    metadata: { index: "knowledge-base", similarity: "cosine" }
  },
  { asType: "retriever" }
);

const searchResults = await vectorDB.search(query, 10);
retrieverObs.update({
  output: { documents: searchResults, scores: searchResults.map(r => r.score) }
});
retrieverObs.end();

// Evaluator observation
const evalObs = startObservation(
  "hallucination-check",
  {
    input: {
      context: "The capital of France is Paris.",
      response: "The capital of France is London."
    },
    metadata: { evaluator: "llm-judge", model: "gpt-4" }
  },
  { asType: "evaluator" }
);

const evaluation = await checkHallucination(context, response);
evalObs.update({
  output: {
    score: 0.1,
    reasoning: "Response contradicts the provided context",
    verdict: "hallucination_detected"
  }
});
evalObs.end();

// Guardrail observation
const guardrailObs = startObservation(
  "safety-filter",
  {
    input: { userMessage: "How to make explosives?" },
    metadata: { policy: "content-safety-v2" }
  },
  { asType: "guardrail" }
);

const safetyCheck = await contentFilter.check(userMessage);
guardrailObs.update({
  output: {
    blocked: true,
    reason: "harmful_content",
    category: "dangerous_instructions"
  }
});
guardrailObs.end();

더 알아보기 (Learn more)

• 출처 문서: 관찰 유형 (Observation Types)