R2R 연동하기

R2R 연동하기 (R2R Integration)

R2R은 멀티모달 콘텐츠 수집, 하이브리드 검색, 사용자·문서 관리 같은 프로덕션급 기능을 갖춘 AI RAG(retrieval-augmented generation) 올인원 솔루션이에요. 이 튜토리얼에서는 R2R의 /rag 엔드포인트를 이용해 작은 데이터셋에 대한 RAG를 수행하고, 생성된 응답을 Ragas로 평가해 볼게요.

출처: 문서

본문

개요

이 튜토리얼에서는 다음을 수행해요.

• R2R의 /rag 엔드포인트를 활용해 작은 데이터셋으로 RAG(Retrieval-Augmented Generation) 실행
• 생성된 응답 평가
• 평가 트레이스 분석

R2R 설정

의존성 설치하기

먼저 필요한 패키지를 설치해요.

%pip install r2r -q

로컬 환경 설정하기

R2R_API_KEY, OPENAI_API_KEY, RAGAS_APP_TOKEN(선택)을 설정해요.

from dotenv import load_dotenv

load_dotenv()

데이터 가져오기

dataset = [
    "OpenAI is one of the most recognized names in the large language model space, known for its GPT series of models. These models excel at generating human-like text and performing tasks like creative writing, answering questions, and summarizing content. GPT-4, their latest release, has set benchmarks in understanding context and delivering detailed responses.",
    "Anthropic is well-known for its Claude series of language models, designed with a strong focus on safety and ethical AI behavior. Claude is particularly praised for its ability to follow complex instructions and generate text that aligns closely with user intent.",
    "DeepMind, a division of Google, is recognized for its cutting-edge Gemini models, which are integrated into various Google products like Bard and Workspace tools. These models are renowned for their conversational abilities and their capacity to handle complex, multi-turn dialogues.",
    "Meta AI is best known for its LLaMA (Large Language Model Meta AI) series, which has been made open-source for researchers and developers. LLaMA models are praised for their ability to support innovation and experimentation due to their accessibility and strong performance.",
    "Meta AI with it's LLaMA models aims to democratize AI development by making high-quality models available for free, fostering collaboration across industries. Their open-source approach has been a game-changer for researchers without access to expensive resources.",
    "Microsoft’s Azure AI platform is famous for integrating OpenAI’s GPT models, enabling businesses to use these advanced models in a scalable and secure cloud environment. Azure AI powers applications like Copilot in Office 365, helping users draft emails, generate summaries, and more.",
    "Amazon’s Bedrock platform is recognized for providing access to various language models, including its own models and third-party ones like Anthropic’s Claude and AI21’s Jurassic. Bedrock is especially valued for its flexibility, allowing users to choose models based on their specific needs.",
    "Cohere is well-known for its language models tailored for business use, excelling in tasks like search, summarization, and customer support. Their models are recognized for being efficient, cost-effective, and easy to integrate into workflows.",
    "AI21 Labs is famous for its Jurassic series of language models, which are highly versatile and capable of handling tasks like content creation and code generation. The Jurassic models stand out for their natural language understanding and ability to generate detailed and coherent responses.",
    "In the rapidly advancing field of artificial intelligence, several companies have made significant contributions with their large language models. Notable players include OpenAI, known for its GPT Series (including GPT-4); Anthropic, which offers the Claude Series; Google DeepMind with its Gemini Models; Meta AI, recognized for its LLaMA Series; Microsoft Azure AI, which integrates OpenAI’s GPT Models; Amazon AWS (Bedrock), providing access to various models including Claude (Anthropic) and Jurassic (AI21 Labs); Cohere, which offers its own models tailored for business use; and AI21 Labs, known for its Jurassic Series. These companies are shaping the landscape of AI by providing powerful models with diverse capabilities.",
]

R2R 클라이언트 설정하기

from r2r import R2RClient

client = R2RClient()

데이터 수집하기

ingest_response = client.documents.create(
    chunks=dataset,
)

/rag 엔드포인트 사용하기

/rag 엔드포인트는 검색 결과와 언어 모델 출력을 통합해 RAG를 수행해요. rag_generation_config 파라미터로 생성 과정을, search_settings로 검색 과정을 설정할 수 있어요.

query = "What makes Meta AI’s LLaMA models stand out?"

search_settings = {
        "limit": 2,
        "graph_settings": {"enabled": False, "limit": 2},
    }

response = client.retrieval.rag(
    query=query,
    search_settings=search_settings
)

print(response.results.generated_answer)

출력

Meta AI’s LLaMA models stand out due to their open-source nature, which supports innovation and experimentation by making high-quality models accessible to researchers and developers [1]. This approach democratizes AI development, fostering collaboration across industries and enabling researchers without access to expensive resources to work with advanced AI models [2].

평가

R2R Client를 Ragas로 평가하기

R2R Client가 준비되면 Ragas의 r2r 연동을 사용해 평가할 수 있어요. 이 과정은 다음 핵심 컴포넌트로 구성돼요.

• 1. R2R 클라이언트와 설정 RAG 설정을 지정하는 R2RClient와 /rag 설정.
• 2. 평가 데이터셋 (Evaluation Dataset) Ragas 지표가 필요로 하는 모든 입력을 담은 Ragas EvaluationDataset이 필요해요.
• 3. Ragas 지표 (Ragas Metrics) Ragas는 faithfulness, answer relevance, context recall 등 RAG의 다양한 측면을 평가하는 여러 지표를 제공해요. 사용 가능한 전체 지표 목록은 Ragas 문서에서 확인할 수 있어요.

Ragas EvaluationDataset 구성하기

EvaluationDataset은 평가 샘플을 나타내는 Ragas의 데이터 타입이에요. 구조와 사용법에 대한 자세한 내용은 핵심 개념 섹션에서 확인할 수 있어요.

데이터용 EvaluationDataset을 얻기 위해 ragas의 transform_to_ragas_dataset 함수를 사용할게요.

questions = [
    "Who are the major players in the large language model space?",
    "What is Microsoft’s Azure AI platform known for?",
    "What kind of models does Cohere provide?",
]

references = [
    "The major players include OpenAI (GPT Series), Anthropic (Claude Series), Google DeepMind (Gemini Models), Meta AI (LLaMA Series), Microsoft Azure AI (integrating GPT Models), Amazon AWS (Bedrock with Claude and Jurassic), Cohere (business-focused models), and AI21 Labs (Jurassic Series).",
    "Microsoft’s Azure AI platform is known for integrating OpenAI’s GPT models, enabling businesses to use these models in a scalable and secure cloud environment.",
    "Cohere provides language models tailored for business use, excelling in tasks like search, summarization, and customer support.",
]

r2r_responses = []

search_settings = {
    "limit": 2,
    "graph_settings": {"enabled": False, "limit": 2},
}

for que in questions:
    response = client.retrieval.rag(query=que, search_settings=search_settings)
    r2r_responses.append(response)

from ragas.integrations.r2r import transform_to_ragas_dataset

ragas_eval_dataset = transform_to_ragas_dataset(
    user_inputs=questions, r2r_responses=r2r_responses, references=references
)

출력

EvaluationDataset(features=['user_input', 'retrieved_contexts', 'response', 'reference'], len=3)

지표 선택하기

RAG 엔드포인트를 평가하기 위해 다음 지표를 사용할게요.

• Response Relevancy: 응답이 사용자 입력(질의)과 얼마나 관련 있는지 측정
• Context Precision: 관련 문서(또는 정보 조각)가 얼마나 성공적으로 검색됐는지 측정
• Faithfulness: 응답이 검색된 컨텍스트와 사실적으로 일치하는지 측정

from ragas.metrics import AnswerRelevancy, ContextPrecision, Faithfulness
from ragas import evaluate
from langchain_openai import ChatOpenAI
from ragas.llms import LangchainLLMWrapper

llm = ChatOpenAI(model="gpt-4o-mini")
evaluator_llm = LangchainLLMWrapper(llm)

ragas_metrics = [AnswerRelevancy(llm=evaluator_llm), ContextPrecision(llm=evaluator_llm), Faithfulness(llm=evaluator_llm)]

results = evaluate(dataset=ragas_eval_dataset, metrics=ragas_metrics)

출력

Querying Client: 100%|██████████| 3/3 [00:00<?, ?it/s]

Evaluating: 100%|██████████| 9/9 [00:00<?, ?it/s]

평가 트레이싱

평가에서 나온 점수를 더 잘 이해하기 위해, 아래 코드로 판정의 트레이스와 이유를 얻을 수 있어요.

results.upload()

즐거운 코딩 되세요!