LangSmith
LangSmith
LangSmith는 LLM(대규모 언어 모델)을 활용하는 애플리케이션의 개발·배포를 향상시키기 위해 설계된 고급 도구예요. LLM 워크플로우를 추적·분석·최적화하는 포괄적인 프레임워크를 제공해서, 개발자가 애플리케이션 내 복잡한 상호작용을 더 쉽게 관리할 수 있게 해 줘요.
출처: 문서
본문
이 튜토리얼은 LangSmith를 사용해 Ragas 평가의 trace를 로깅하는 방법을 설명해요. Ragas는 LangChain 기반으로 만들어졌으므로, LangSmith만 설정하면 trace 로깅을 자동으로 처리해요.
LangSmith 설정
LangSmith를 설정하려면 다음 환경 변수를 설정해야 해요(LangSmith 문서에서 자세한 내용을 참조하세요).
export LANGCHAIN_TRACING_V2=true
export LANGCHAIN_ENDPOINT=https://api.smith.langchain.com
export LANGCHAIN_API_KEY=<your-api-key>
export LANGCHAIN_PROJECT=<your-project> # Defaults to "default" if not set
데이터셋 가져오기
평가 데이터셋을 만들거나 인스턴스를 평가할 때는 용어가 SingleTurnSample 또는 MultiTurnSample 에 사용된 스키마와 일치하는지 확인하세요.
from ragas import EvaluationDataset
dataset = [
{
"user_input": "Which CEO is widely recognized for democratizing AI education through platforms like Coursera?",
"retrieved_contexts": [
"Andrew Ng, CEO of Landing AI, is known for his pioneering work in deep learning and for democratizing AI education through Coursera."
],
"response": "Andrew Ng is widely recognized for democratizing AI education through platforms like Coursera.",
"reference": "Andrew Ng, CEO of Landing AI, is known for democratizing AI education through Coursera.",
},
{
"user_input": "Who is Sam Altman?",
"retrieved_contexts": [
"Sam Altman, CEO of OpenAI, has advanced AI research and advocates for safe, beneficial AI technologies."
],
"response": "Sam Altman is the CEO of OpenAI and advocates for safe, beneficial AI technologies.",
"reference": "Sam Altman, CEO of OpenAI, has advanced AI research and advocates for safe AI.",
},
{
"user_input": "Who is Demis Hassabis and how did he gain prominence?",
"retrieved_contexts": [
"Demis Hassabis, CEO of DeepMind, is known for developing systems like AlphaGo that master complex games."
],
"response": "Demis Hassabis is the CEO of DeepMind, known for developing systems like AlphaGo.",
"reference": "Demis Hassabis, CEO of DeepMind, is known for developing AlphaGo.",
},
{
"user_input": "Who is the CEO of Google and Alphabet Inc., praised for leading innovation across Google's product ecosystem?",
"retrieved_contexts": [
"Sundar Pichai, CEO of Google and Alphabet Inc., leads innovation across Google's product ecosystem."
],
"response": "Sundar Pichai is the CEO of Google and Alphabet Inc., praised for leading innovation across Google's product ecosystem.",
"reference": "Sundar Pichai, CEO of Google and Alphabet Inc., leads innovation across Google's product ecosystem.",
},
{
"user_input": "How did Arvind Krishna transform IBM?",
"retrieved_contexts": [
"Arvind Krishna, CEO of IBM, transformed the company by focusing on cloud computing and AI solutions."
],
"response": "Arvind Krishna transformed IBM by focusing on cloud computing and AI solutions.",
"reference": "Arvind Krishna, CEO of IBM, transformed the company through cloud computing and AI.",
},
]
evaluation_dataset = EvaluationDataset.from_list(dataset)
ragas 메트릭 추적하기
데이터셋에 대해 Ragas 평가를 실행하면, trace가 지정한 프로젝트 이름 또는 "default" 아래의 LangSmith 대시보드에 나타나요.
from ragas import evaluate
from ragas.llms import LangchainLLMWrapper
from langchain_openai import ChatOpenAI
from ragas.metrics import LLMContextRecall, Faithfulness, FactualCorrectness
llm = ChatOpenAI(model="gpt-4o-mini")
evaluator_llm = LangchainLLMWrapper(llm)
result = evaluate(
dataset=evaluation_dataset,
metrics=[LLMContextRecall(), Faithfulness(), FactualCorrectness()],
llm=evaluator_llm,
)
result
Output
Evaluating: 0%| | 0/15 [00:00<?, ?it/s]
{'context_recall': 1.0000, 'faithfulness': 0.9333, 'factual_correctness': 0.8520}