Runnable 평가하는 방법

Runnable 평가하는 방법

langchain Runnable 객체(채팅 모델, retriever, 체인 등)를 evaluate() / aevaluate()에 직접 전달하여 평가하는 방법을 알려드릴게요.

본문

설정

먼저 평가할 간단한 체인을 정의해 봅시다. 필요한 모든 패키지를 설치합니다:

pip install -U langsmith langchain[openai]

yarn add langsmith @langchain/openai

이제 체인을 정의합니다:

from langchain.chat_models import init_chat_model
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate

instructions = (
    "Please review the user query below and determine if it contains any form "
    "of toxic behavior, such as insults, threats, or highly negative comments. "
    "Respond with 'Toxic' if it does, and 'Not toxic' if it doesn't."
)

prompt = ChatPromptTemplate(
    [("system", instructions), ("user", "{text}")],
)

model = init_chat_model("gpt-5.5")
chain = prompt | model | StrOutputParser()

import { ChatOpenAI } from "@langchain/openai";
import { ChatPromptTemplate } from "@langchain/core/prompts";
import { StringOutputParser } from "@langchain/core/output_parsers";

const prompt = ChatPromptTemplate.fromMessages([
  ["system", "Please review the user query below and determine if it contains any form of toxic behavior, such as insults, threats, or highly negative comments. Respond with 'Toxic' if it does, and 'Not toxic' if it doesn't."],
  ["user", "{text}"]
]);

const chatModel = new ChatOpenAI();
const outputParser = new StringOutputParser();
const chain = prompt.pipe(chatModel).pipe(outputParser);

평가하기

체인을 평가하려면 evaluate() / aevaluate() 메서드에 직접 전달하면 됩니다. 체인의 입력 변수는 예제 입력의 키와 일치해야 합니다. 이 경우 예제 입력은 {"text": "..."} 형태여야 합니다.

import asyncio
from langsmith import Client, aevaluate

client = Client()

# Clone a dataset of texts with toxicity labels.
# Each example input has a "text" key and each output has a "label" key.
dataset = client.clone_public_dataset(
    "https://smith.langchain.com/public/3d6831e6-1680-4c88-94df-618c8e01fc55/d"
)

def correct(outputs: dict, reference_outputs: dict) -> bool:
    # Since our chain outputs a string not a dict, this string
    # gets stored under the default "output" key in the outputs dict:
    actual = outputs["output"]
    expected = reference_outputs["label"]
    return actual == expected

async def main():
    results = await aevaluate(
        chain,
        data=dataset,
        evaluators=[correct],
        experiment_prefix="gpt-5.5, baseline",
        metadata={"models": "openai:gpt-5.5"},  # optional, used to populate model/prompt/tool columns in UI
    )
    print(results)

asyncio.run(main())

import { evaluate } from "langsmith/evaluation";
import { Client } from "langsmith";

const langsmith = new Client();

const dataset = await client.clonePublicDataset(
  "https://smith.langchain.com/public/3d6831e6-1680-4c88-94df-618c8e01fc55/d"
)

await evaluate(chain, {
  data: dataset.name,
  evaluators: [correct],
  experimentPrefix: "gpt-5.5, baseline",
  metadata: { models: "openai:gpt-5.5" },  // optional, used to populate model/prompt/tool columns in UI
});

runnable은 각 출력에 대해 적절히 트레이싱됩니다.

관련 자료

• langgraph 그래프 평가하는 방법

더 알아보기

• 중간 단계에서 평가 — langgraph 그래프 평가 방법.