Griptape 통합

Griptape 통합

Griptape의 RAG Engine에 익숙하고 RAG 시스템 성능 평가를 시작하고 싶다면 잘 찾아오셨어요. 이 튜토리얼에서는 Griptape RAG Engine이 생성한 응답을 Ragas로 평가하는 방법을 살펴볼 거예요.

출처: 문서

본문

Griptape 설정

환경 설정

먼저 필요한 패키지가 모두 설치되었는지 확인해요.

%pip install "griptape[all]" ragas -q

데이터셋 만들기

주요 LLM 프로바이더에 대한 텍스트 청크의 작은 데이터셋을 사용하고 간단한 RAG 파이프라인을 설정할 거예요.

chunks = [
    "OpenAI is one of the most recognized names in the large language model space, known for its GPT series of models. These models excel at generating human-like text and performing tasks like creative writing, answering questions, and summarizing content. GPT-4, their latest release, has set benchmarks in understanding context and delivering detailed responses.",
    "Anthropic is well-known for its Claude series of language models, designed with a strong focus on safety and ethical AI behavior. Claude is particularly praised for its ability to follow complex instructions and generate text that aligns closely with user intent.",
    "DeepMind, a division of Google, is recognized for its cutting-edge Gemini models, which are integrated into various Google products like Bard and Workspace tools. These models are renowned for their conversational abilities and their capacity to handle complex, multi-turn dialogues.",
    "Meta AI is best known for its LLaMA (Large Language Model Meta AI) series, which has been made open-source for researchers and developers. LLaMA models are praised for their ability to support innovation and experimentation due to their accessibility and strong performance.",
    "Meta AI with it's LLaMA models aims to democratize AI development by making high-quality models available for free, fostering collaboration across industries. Their open-source approach has been a game-changer for researchers without access to expensive resources.",
    "Microsoft’s Azure AI platform is famous for integrating OpenAI’s GPT models, enabling businesses to use these advanced models in a scalable and secure cloud environment. Azure AI powers applications like Copilot in Office 365, helping users draft emails, generate summaries, and more.",
    "Amazon’s Bedrock platform is recognized for providing access to various language models, including its own models and third-party ones like Anthropic’s Claude and AI21’s Jurassic. Bedrock is especially valued for its flexibility, allowing users to choose models based on their specific needs.",
    "Cohere is well-known for its language models tailored for business use, excelling in tasks like search, summarization, and customer support. Their models are recognized for being efficient, cost-effective, and easy to integrate into workflows.",
    "AI21 Labs is famous for its Jurassic series of language models, which are highly versatile and capable of handling tasks like content creation and code generation. The Jurassic models stand out for their natural language understanding and ability to generate detailed and coherent responses.",
    "In the rapidly advancing field of artificial intelligence, several companies have made significant contributions with their large language models. Notable players include OpenAI, known for its GPT Series (including GPT-4); Anthropic, which offers the Claude Series; Google DeepMind with its Gemini Models; Meta AI, recognized for its LLaMA Series; Microsoft Azure AI, which integrates OpenAI’s GPT Models; Amazon AWS (Bedrock), providing access to various models including Claude (Anthropic) and Jurassic (AI21 Labs); Cohere, which offers its own models tailored for business use; and AI21 Labs, known for its Jurassic Series. These companies are shaping the landscape of AI by providing powerful models with diverse capabilities.",
]

Vector Store에 데이터 수집

import getpass
import os

if "OPENAI_API_KEY" not in os.environ:
    os.environ["OPENAI_API_KEY"] = getpass.getpass("Enter your OpenAI API key: ")

from griptape.drivers.embedding.openai import OpenAiEmbeddingDriver
from griptape.drivers.vector.local import LocalVectorStoreDriver

# Set up a simple vector store with our data
vector_store = LocalVectorStoreDriver(embedding_driver=OpenAiEmbeddingDriver())
vector_store.upsert_collection({"major_llm_providers": chunks})

RAG Engine 설정

from griptape.engines.rag import RagContext, RagEngine
from griptape.engines.rag.modules import (
    PromptResponseRagModule,
    VectorStoreRetrievalRagModule,
)
from griptape.engines.rag.stages import (
    ResponseRagStage,
    RetrievalRagStage,
)

# Create a basic RAG pipeline
rag_engine = RagEngine(
    # Stage for retrieving relevant chunks
    retrieval_stage=RetrievalRagStage(
        retrieval_modules=[
            VectorStoreRetrievalRagModule(
                name="VectorStore_Retriever",
                vector_store_driver=vector_store,
                query_params={"namespace": "major_llm_providers"},
            ),
        ],
    ),
    # Stage for generating a response
    response_stage=ResponseRagStage(
        response_modules=[
            PromptResponseRagModule(),
        ]
    ),
)

RAG 파이프라인 테스트

RAG 파이프라인이 동작하는지 샘플 쿼리로 확인해 봅시다.

rag_context = RagContext(query="What makes Meta AI’s LLaMA models stand out?")
rag_context = rag_engine.process(rag_context)
rag_context.outputs[0].to_text()

"Meta AI's LLaMA models stand out for their open-source nature, which makes them accessible to researchers and developers. This accessibility supports innovation and experimentation, allowing for collaboration across industries. By making high-quality models available for free, Meta AI aims to democratize AI development, which has been a game-changer for researchers without access to expensive resources."

Ragas 평가

Ragas 평가 데이터셋 만들기

questions = [
    "Who are the major players in the large language model space?",
    "What is Microsoft’s Azure AI platform known for?",
    "What kind of models does Cohere provide?",
]

references = [
    "The major players include OpenAI (GPT Series), Anthropic (Claude Series), Google DeepMind (Gemini Models), Meta AI (LLaMA Series), Microsoft Azure AI (integrating GPT Models), Amazon AWS (Bedrock with Claude and Jurassic), Cohere (business-focused models), and AI21 Labs (Jurassic Series).",
    "Microsoft’s Azure AI platform is known for integrating OpenAI’s GPT models, enabling businesses to use these models in a scalable and secure cloud environment.",
    "Cohere provides language models tailored for business use, excelling in tasks like search, summarization, and customer support.",
]

griptape_rag_contexts = []

for que in questions:
    rag_context = RagContext(query=que)
    griptape_rag_contexts.append(rag_engine.process(rag_context))

from ragas.integrations.griptape import transform_to_ragas_dataset

ragas_eval_dataset = transform_to_ragas_dataset(
    grip_tape_rag_contexts=griptape_rag_contexts, references=references
)

ragas_eval_dataset.to_pandas()

Ragas 평가 실행

이제 Ragas 메트릭으로 RAG 시스템을 평가해 보겠습니다.

검색(Retrieval) 평가

검색 성능을 평가하기 위해 Ragas 내장 메트릭을 사용하거나 자신의 필요에 맞는 커스텀 메트릭을 만들 수 있어요. 사용 가능한 모든 메트릭과 커스터마이징 옵션의 포괄적인 목록은 문서를 참조하세요.

ContextPrecision, ContextRecall, ContextRelevance 를 사용해 검색 성능을 측정할 거예요.

  • ContextPrecision : 주어진 쿼리에 대해 RAG 시스템의 검색기가 관련 청크를 검색된 컨텍스트의 상위에 얼마나 잘 순위를 매기는지 측정하며, 모든 청크에 대한 mean precision@k로 계산돼요.
  • ContextRecall : 지식 베이스에서 관련 정보를 성공적으로 검색한 비율을 측정해요.
  • ContextRelevance : 이중 LLM 판단을 통한 관련성 평가로 검색된 컨텍스트가 사용자 쿼리를 얼마나 잘 다루는지 측정해요.
from ragas.metrics import ContextPrecision, ContextRecall, ContextRelevance
from ragas import evaluate
from langchain_openai import ChatOpenAI
from ragas.llms import LangchainLLMWrapper

llm = ChatOpenAI(model="gpt-4o-mini")
evaluator_llm = LangchainLLMWrapper(llm)

ragas_metrics = [
    ContextPrecision(llm=evaluator_llm),
    ContextRecall(llm=evaluator_llm),
    ContextRelevance(llm=evaluator_llm),
]

retrieval_results = evaluate(dataset=ragas_eval_dataset, metrics=ragas_metrics)
retrieval_results.to_pandas()

Evaluating: 100%|██████████| 9/9 [00:15<00:00,  1.77s/it]

생성(Generation) 평가

생성 성능을 측정하기 위해 FactualCorrectness, Faithfulness, ResponseGroundedness 를 사용할 거예요.

  • FactualCorrectness : 응답의 모든 문장이 reference 답변에 의해 뒷받침되는지 확인해요.
  • Faithfulness : 응답이 검색된 컨텍스트와 얼마나 사실적으로 일관된지 측정해요.
  • ResponseGroundedness : 응답이 제공된 컨텍스트에 근거(grounded)했는지 측정해 환각이나 지어낸 정보를 식별하는 데 도움을 줘요.
from ragas.metrics import FactualCorrectness, Faithfulness, ResponseGroundedness

ragas_metrics = [
    FactualCorrectness(llm=evaluator_llm),
    Faithfulness(llm=evaluator_llm),
    ResponseGroundedness(llm=evaluator_llm),
]

genration_results = evaluate(dataset=ragas_eval_dataset, metrics=ragas_metrics)
genration_results.to_pandas()

Evaluating: 100%|██████████| 9/9 [00:17<00:00,  1.90s/it]

결론

축하해요! Griptape RAG 시스템을 위한 Ragas 평가 파이프라인을 성공적으로 설정했어요. 이 평가는 시스템이 관련 정보를 얼마나 잘 검색하고 정확한 응답을 생성하는지에 대한 귀중한 인사이트를 제공해요.

RAG 평가는 반복적인 과정이라는 점을 기억하세요. 이 메트릭들을 사용해 시스템의 약점을 식별하고, 개선하고, 필요한 성능 수준에 도달할 때까지 재평가하세요.

Happy RAGging! 😄

더 알아보기 (Learn more)