LangChain 통합
LangChain 통합
이 튜토리얼에서는 LangChain으로 만든 RAG 기반 Q&A 애플리케이션을 Ragas로 평가하는 방법을 보여줘요. 또한 Ragas App이 애플리케이션 성능을 분석·향상하는 데 어떻게 도움이 되는지도 살펴볼 거예요.
출처: 문서
본문
간단한 Q&A 애플리케이션 구축
질문-답변 시스템을 만들기 위해 작은 데이터셋을 만들고 그것의 임베딩으로 벡터 데이터베이스에 인덱싱하는 것부터 시작해요.
import os
from dotenv import load_dotenv
from langchain_core.documents import Document
load_dotenv()
content_list = [
"Andrew Ng is the CEO of Landing AI and is known for his pioneering work in deep learning. He is also widely recognized for democratizing AI education through platforms like Coursera.",
"Sam Altman is the CEO of OpenAI and has played a key role in advancing AI research and development. He is a strong advocate for creating safe and beneficial AI technologies.",
"Demis Hassabis is the CEO of DeepMind and is celebrated for his innovative approach to artificial intelligence. He gained prominence for developing systems that can master complex games like AlphaGo.",
"Sundar Pichai is the CEO of Google and Alphabet Inc., and he is praised for leading innovation across Google's vast product ecosystem. His leadership has significantly enhanced user experiences on a global scale.",
"Arvind Krishna is the CEO of IBM and is recognized for transforming the company towards cloud computing and AI solutions. He focuses on providing cutting-edge technologies to address modern business challenges.",
]
langchain_documents = []
for content in content_list:
langchain_documents.append(
Document(
page_content=content,
)
)
from ragas.embeddings import OpenAIEmbeddings
from langchain_core.vectorstores import InMemoryVectorStore
import openai
openai_client = openai.OpenAI()
embeddings = OpenAIEmbeddings(client=openai_client, model="text-embedding-3-small")
vector_store = InMemoryVectorStore(embeddings)
_ = vector_store.add_documents(langchain_documents)
이제 retriever, LLM, 프롬프트를 Retrieval QA Chain에 통합한 RAG 기반 시스템을 구축할 거예요. retriever는 지식 베이스에서 관련 문서를 가져와요. LLM은 검색된 문서를 기반으로 응답을 생성하고, 프롬프트는 모델의 응답을 안내해 컨텍스트를 이해하고 관련성 있고 일관된 언어 기반 출력을 만들도록 도와줘요.
LangChain에서 벡터 스토어의 .as_retriever 메서드를 사용해 retriever를 만들 수 있어요. 자세한 내용은 LangChain의 벡터 스토어 retriever 문서를 참조하세요.
retriever = vector_store.as_retriever(search_kwargs={"k": 1})
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(model="gpt-4o-mini")
사용자 쿼리와 검색된 관련 데이터를 처리해 구조화된 프롬프트 안에서 모델에 전달하는 Chain을 정의할 거예요. 그런 다음 모델 출력을 파싱해 최종 응답을 문자열로 생성해요.
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
template = """Answer the question based only on the following context:
{context}
Question: {query}
"""
prompt = ChatPromptTemplate.from_template(template)
qa_chain = prompt | llm | StrOutputParser()
def format_docs(relevant_docs):
return "\n".join(doc.page_content for doc in relevant_docs)
query = "Who is the CEO of OpenAI?"
relevant_docs = retriever.invoke(query)
qa_chain.invoke({"context": format_docs(relevant_docs), "query": query})
'The CEO of OpenAI is Sam Altman.'
평가
sample_queries = [
"Which CEO is widely recognized for democratizing AI education through platforms like Coursera?",
"Who is Sam Altman?",
"Who is Demis Hassabis and how did he gained prominence?",
"Who is the CEO of Google and Alphabet Inc., praised for leading innovation across Google's product ecosystem?",
"How did Arvind Krishna transformed IBM?",
]
expected_responses = [
"Andrew Ng is the CEO of Landing AI and is widely recognized for democratizing AI education through platforms like Coursera.",
"Sam Altman is the CEO of OpenAI and has played a key role in advancing AI research and development. He strongly advocates for creating safe and beneficial AI technologies.",
"Demis Hassabis is the CEO of DeepMind and is celebrated for his innovative approach to artificial intelligence. He gained prominence for developing systems like AlphaGo that can master complex games.",
"Sundar Pichai is the CEO of Google and Alphabet Inc., praised for leading innovation across Google's vast product ecosystem. His leadership has significantly enhanced user experiences globally.",
"Arvind Krishna is the CEO of IBM and has transformed the company towards cloud computing and AI solutions. He focuses on delivering cutting-edge technologies to address modern business challenges.",
]
Q&A 시스템을 평가하려면 쿼리, expected_responses와 그 밖의 메트릭별 요구사항을 EvaluationDataset 으로 구조화해야 해요.
from ragas import EvaluationDataset
dataset = []
for query, reference in zip(sample_queries, expected_responses):
relevant_docs = retriever.invoke(query)
response = qa_chain.invoke({"context": format_docs(relevant_docs), "query": query})
dataset.append(
{
"user_input": query,
"retrieved_contexts": [rdoc.page_content for rdoc in relevant_docs],
"response": response,
"reference": reference,
}
)
evaluation_dataset = EvaluationDataset.from_list(dataset)
Q&A 애플리케이션을 평가하기 위해 다음 메트릭을 사용할 거예요.
LLMContextRecall: 참조 답변의 claim과 검색된 컨텍스트가 얼마나 잘 일치하는지 평가해서, 수동 참조 컨텍스트 어노테이션 없이 recall을 추정해요.Faithfulness: 생성된 답변의 모든 claim이 제공된 컨텍스트에서 직접 추론될 수 있는지 평가해요.Factual Correctness: claim 기반 평가와 자연어 추론을 사용해 생성된 응답을 reference와 비교해 사실적 정확성을 확인해요.
이 메트릭들에 대한 자세한 내용과 RAG 시스템 평가에 어떻게 적용되는지는 Ragas Metrics 문서를 방문하세요.
from ragas import evaluate
from ragas.llms import LangchainLLMWrapper
from ragas.metrics import LLMContextRecall, Faithfulness, FactualCorrectness
evaluator_llm = LangchainLLMWrapper(llm)
result = evaluate(
dataset=evaluation_dataset,
metrics=[LLMContextRecall(), Faithfulness(), FactualCorrectness()],
llm=evaluator_llm,
)
result
Output
{'context_recall': 1.0000, 'faithfulness': 0.9000, 'factual_correctness': 0.9260}