Haystack 통합
Haystack 통합
Haystack는 커스터마이즈 가능하고 프로덕션에 바로 쓸 수 있는 LLM 애플리케이션을 구축하기 위한 LLM 오케스트레이션 프레임워크예요. Haystack의 기본 개념은 문서 저장, 관련 데이터 검색, 응답 생성 같은 모든 개별 작업이 Document Stores, Retrievers, Generators 같은 모듈식 컴포넌트로 처리되고, 이들이 Pipelines로 매끄럽게 연결·오케스트레이션된다는 거예요.
출처: 문서
본문
개요
이 튜토리얼에서는 Haystack로 RAG 파이프라인을 만들고 Ragas로 평가할 거예요. 먼저 RAG 파이프라인의 다양한 컴포넌트를 설정하고, 평가를 위해 RagasEvaluator 컴포넌트를 초기화할 거예요. 컴포넌트가 설정되면 컴포넌트들을 연결해 완전한 파이프라인을 만들 거예요. 튜토리얼 후반에는 Ragas에서 커스텀 정의 메트릭으로 평가를 수행하는 방법을 살펴볼 거예요.
의존성 설치
%pip install ragas-haystack
데이터 준비
dataset = [
"OpenAI is one of the most recognized names in the large language model space, known for its GPT series of models. These models excel at generating human-like text and performing tasks like creative writing, answering questions, and summarizing content. GPT-4, their latest release, has set benchmarks in understanding context and delivering detailed responses.",
"Anthropic is well-known for its Claude series of language models, designed with a strong focus on safety and ethical AI behavior. Claude is particularly praised for its ability to follow complex instructions and generate text that aligns closely with user intent.",
"DeepMind, a division of Google, is recognized for its cutting-edge Gemini models, which are integrated into various Google products like Bard and Workspace tools. These models are renowned for their conversational abilities and their capacity to handle complex, multi-turn dialogues.",
"Meta AI is best known for its LLaMA (Large Language Model Meta AI) series, which has been made open-source for researchers and developers. LLaMA models are praised for their ability to support innovation and experimentation due to their accessibility and strong performance.",
"Meta AI with it's LLaMA models aims to democratize AI development by making high-quality models available for free, fostering collaboration across industries. Their open-source approach has been a game-changer for researchers without access to expensive resources.",
"Microsoft’s Azure AI platform is famous for integrating OpenAI’s GPT models, enabling businesses to use these advanced models in a scalable and secure cloud environment. Azure AI powers applications like Copilot in Office 365, helping users draft emails, generate summaries, and more.",
"Amazon’s Bedrock platform is recognized for providing access to various language models, including its own models and third-party ones like Anthropic’s Claude and AI21’s Jurassic. Bedrock is especially valued for its flexibility, allowing users to choose models based on their specific needs.",
"Cohere is well-known for its language models tailored for business use, excelling in tasks like search, summarization, and customer support. Their models are recognized for being efficient, cost-effective, and easy to integrate into workflows.",
"AI21 Labs is famous for its Jurassic series of language models, which are highly versatile and capable of handling tasks like content creation and code generation. The Jurassic models stand out for their natural language understanding and ability to generate detailed and coherent responses.",
"In the rapidly advancing field of artificial intelligence, several companies have made significant contributions with their large language models. Notable players include OpenAI, known for its GPT Series (including GPT-4); Anthropic, which offers the Claude Series; Google DeepMind with its Gemini Models; Meta AI, recognized for its LLaMA Series; Microsoft Azure AI, which integrates OpenAI’s GPT Models; Amazon AWS (Bedrock), providing access to various models including Claude (Anthropic) and Jurassic (AI21 Labs); Cohere, which offers its own models tailored for business use; and AI21 Labs, known for its Jurassic Series. These companies are shaping the landscape of AI by providing powerful models with diverse capabilities.",
]
RAG 파이프라인 컴포넌트 초기화
DocumentStore 초기화
from haystack import Document
from haystack.document_stores.in_memory import InMemoryDocumentStore
document_store = InMemoryDocumentStore()
docs = [Document(content=doc) for doc in dataset]
Document와 Text Embedder 초기화
from haystack.components.embedders import OpenAIDocumentEmbedder, OpenAITextEmbedder
document_embedder = OpenAIDocumentEmbedder(model="text-embedding-3-small")
text_embedder = OpenAITextEmbedder(model="text-embedding-3-small")
이제 document store와 document embedder가 있으니, 이를 사용해 벡터 데이터스토어를 채울 거예요.
docs_with_embeddings = document_embedder.run(docs)
document_store.write_documents(docs_with_embeddings["documents"])
Calculating embeddings: 1it [00:01, 1.74s/it]
10
Retriever 초기화
from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever
retriever = InMemoryEmbeddingRetriever(document_store, top_k=2)
템플릿 프롬프트 정의
from haystack.components.builders import ChatPromptBuilder
from haystack.dataclasses import ChatMessage
template = [
ChatMessage.from_user(
"""
Given the following information, answer the question.
Context:
{% for document in documents %}
{{ document.content }}
{% endfor %}
Question: {{question}}
Answer:
"""
)
]
prompt_builder = ChatPromptBuilder(template=template)
ChatGenerator 초기화
from haystack.components.generators.chat import OpenAIChatGenerator
chat_generator = OpenAIChatGenerator(model="gpt-4o-mini")
RagasEvaluator 설정
평가에 사용할 모든 Ragas 메트릭을 전달하되, 선택된 각 메트릭을 계산하는 데 필요한 모든 정보가 제공되는지 확인하세요.
예를 들어:
AnswerRelevancy: query 와 response 가 모두 필요해요.ContextPrecision: query , 검색된 documents , 그리고 reference 가 필요해요.Faithfulness: query , 검색된 documents , 그리고 response 가 필요해요.
정확한 평가를 위해 각 메트릭에 관련 데이터를 모두 포함해야 해요.
from haystack_integrations.components.evaluators.ragas import RagasEvaluator
from langchain_openai import ChatOpenAI
from ragas.llms import LangchainLLMWrapper
from ragas.metrics import AnswerRelevancy, ContextPrecision, Faithfulness
llm = ChatOpenAI(model="gpt-4o-mini")
evaluator_llm = LangchainLLMWrapper(llm)
ragas_evaluator = RagasEvaluator(
ragas_metrics=[AnswerRelevancy(), ContextPrecision(), Faithfulness()],
evaluator_llm=evaluator_llm,
)
파이프라인 구축 및 조립
파이프라인 생성
from haystack import Pipeline
rag_pipeline = Pipeline()
컴포넌트 추가
from haystack.components.builders import AnswerBuilder
rag_pipeline.add_component("text_embedder", text_embedder)
rag_pipeline.add_component("retriever", retriever)
rag_pipeline.add_component("prompt_builder", prompt_builder)
rag_pipeline.add_component("llm", chat_generator)
rag_pipeline.add_component("answer_builder", AnswerBuilder())
rag_pipeline.add_component("ragas_evaluator", ragas_evaluator)
컴포넌트 연결
rag_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
rag_pipeline.connect("retriever", "prompt_builder")
rag_pipeline.connect("prompt_builder.prompt", "llm.messages")
rag_pipeline.connect("llm.replies", "answer_builder.replies")
rag_pipeline.connect("retriever", "answer_builder.documents")
rag_pipeline.connect("llm.replies", "answer_builder.replies")
rag_pipeline.connect("retriever", "answer_builder.documents")
rag_pipeline.connect("retriever", "ragas_evaluator.documents")
rag_pipeline.connect("llm.replies", "ragas_evaluator.response")
<haystack.core.pipeline.pipeline.Pipeline object at 0x14b20fad0>
🚅 Components
- text_embedder: OpenAITextEmbedder
- retriever: InMemoryEmbeddingRetriever
- prompt_builder: ChatPromptBuilder
- llm: OpenAIChatGenerator
- answer_builder: AnswerBuilder
- ragas_evaluator: RagasEvaluator
🛤️ Connections
- text_embedder.embedding -> retriever.query_embedding (List[float])
- retriever.documents -> prompt_builder.documents (List[Document])
- retriever.documents -> answer_builder.documents (List[Document])
- retriever.documents -> ragas_evaluator.documents (List[Document])
- prompt_builder.prompt -> llm.messages (List[ChatMessage])
- llm.replies -> answer_builder.replies (List[ChatMessage])
- llm.replies -> ragas_evaluator.response (List[ChatMessage])
파이프라인 실행
question = "What makes Meta AI’s LLaMA models stand out?"
reference = "Meta AI’s LLaMA models stand out for being open-source, supporting innovation and experimentation due to their accessibility and strong performance."
result = rag_pipeline.run(
{
"text_embedder": {"text": question},
"prompt_builder": {"question": question},
"answer_builder": {"query": question},
"ragas_evaluator": {"query": question, "reference": reference},
# Each metric expects a specific set of parameters as input. Refer to the
# Ragas class' documentation for more details.
}
)
print(result["answer_builder"]["answers"][0].data, "\n")
print(result["ragas_evaluator"]["result"])
Evaluating: 100%|██████████| 3/3 [00:14<00:00, 4.72s/it]
Meta AI's LLaMA models stand out due to their open-source nature, which allows researchers and developers easy access to high-quality language models without the need for expensive resources. This accessibility fosters innovation and experimentation, enabling collaboration across various industries. Moreover, the strong performance of the LLaMA models further enhances their appeal, making them valuable tools for advancing AI development.
{'answer_relevancy': 0.9782, 'context_precision': 1.0000, 'faithfulness': 1.0000}
고급 사용법
기본 ragas 메트릭 대신 필요에 맞는 메트릭으로 바꾸거나 직접 커스텀 메트릭을 만들 수 있어요. 그런 다음 이를 RagasEvaluator 컴포넌트에 전달할 수 있어요. ragas 메트릭 커스터마이징에 대해 더 알아보려면 문서를 확인하세요.
아래 예시에서는 두 개의 커스텀 Ragas 메트릭을 정의할 거예요.
SportsRelevanceMetric: 질문과 응답이 스포츠와 관련이 있는지 평가하는 메트릭이에요.AnswerQualityMetric: LLM이 준 응답이 사용자의 질문을 얼마나 잘 답하는지 측정하는 메트릭이에요.
from ragas.metrics import AspectCritic, RubricsScore
SportsRelevanceMetric = AspectCritic(
name="sports_relevance_metric",
definition="Were the question and response related to sports?",
llm=evaluator_llm,
)
rubrics = {
"score1_description": "The response does not answer the user input.",
"score2_description": "The response partially answers the user input.",
"score3_description": "The response fully answer the user input",
}
evaluator = RagasEvaluator(
ragas_metrics=[
SportsRelevanceMetric,
RubricsScore(llm=evaluator_llm, rubrics=rubrics),
],
evaluator_llm=evaluator_llm,
)
output = evaluator.run(
query="Which is the most popular global sport?",
documents=[
"Football is undoubtedly the world's most popular sport with"
" major events like the FIFA World Cup and sports personalities"
" like Ronaldo and Messi, drawing a followership of more than 4"
" billion people."
],
response="Football is the most popular sport with around 4 billion"
" followers worldwide",
)
output["result"]
Evaluating: 100%|██████████| 2/2 [00:01<00:00, 1.62it/s]
{'sports_relevance_metric': 1.0000, 'domain_specific_rubrics': 3.0000}