Ragas 메트릭으로 VertexAI가 제공하는 모델을 RAG 기반 Q&A 작업에서 비교

Ragas 메트릭으로 VertexAI가 제공하는 모델을 RAG 기반 Q&A 작업에서 비교 (Compare models provided by VertexAI on RAG-based Q&A task using Ragas metrics)

이 튜토리얼은 Ragas와 함께 Vertex AI 모델을 사용하는 3부작 시리즈의 일부예요. Getting Started: Ragas with Vertex AI를 먼저 보는 것을 권장하지만, 보지 않았어도 괜찮아요. 클릭해서 Align LLM Metrics 튜토리얼을 확인할 수 있어요.

출처: 문서

본문

이 튜토리얼에서는 Ragas를 사용해서 Question Answering(QA) 작업에 대해 다양한 LLM 모델을 채점하고 평가하는 방법을 배워요. 그런 다음 평가 결과를 시각화하고 비교해서 생성 모델을 선택해요.

시작하기

의존성 설치

%pip install --upgrade --user --quiet langchain-core langchain-google-vertexai langchain ragas rouge_score

런타임 재시작

이 Jupyter 런타임에서 새로 설치한 패키지를 사용하려면 런타임을 재시작해야 해요. 아래 셀을 실행해서 현재 커널을 재시작해요.

import IPython

app = IPython.Application.instance()
app.kernel.do_shutdown(True)

노트북 환경 인증 (Colab 전용)

Google Colab에서 이 노트북을 실행한다면 아래 셀을 실행해서 환경을 인증해요.

import sys

if "google.colab" in sys.modules:
    from google.colab import auth

    auth.authenticate_user()

Google Cloud 프로젝트 정보 설정 및 Vertex AI SDK 초기화

PROJECT_ID = "[your-project-id]"  # @param {type:"string"}
LOCATION = "us-central1"  # @param {type:"string"}

if not PROJECT_ID or PROJECT_ID == "[your-project-id]":
    raise ValueError("Please set your PROJECT_ID")

import vertexai

vertexai.init(project=PROJECT_ID, location=LOCATION)

헬퍼 함수

아래는 평가 보고서 표시와 평가 결과 시각화를 위한 몇 가지 헬퍼 함수예요.

import pandas as pd
import plotly.graph_objects as go
from IPython.display import HTML, Markdown, display

def display_eval_report(eval_result, metrics=None):
    """평가 결과 표시."""

    title, summary_metrics, report_df = eval_result
    metrics_df = pd.DataFrame.from_dict(summary_metrics, orient="index").T
    if metrics:
        metrics_df = metrics_df.filter(
            [
                metric
                for metric in metrics_df.columns
                if any(selected_metric in metric for selected_metric in metrics)
            ]
        )
        report_df = report_df.filter(
            [
                metric
                for metric in report_df.columns
                if any(selected_metric in metric for selected_metric in metrics)
            ]
        )

    # 강조를 위해 제목을 Markdown으로 표시
    display(Markdown(f"## {title}"))

    # 메트릭 DataFrame 표시
    display(Markdown("### Summary Metrics"))
    display(metrics_df)

    # 상세 보고서 DataFrame 표시
    display(Markdown("### Report Metrics"))
    display(report_df)

def plot_radar_plot(eval_results, max_score=5, metrics=None):
    fig = go.Figure()

    for eval_result in eval_results:
        title, summary_metrics, report_df = eval_result

        if metrics:
            summary_metrics = {
                k: summary_metrics[k]
                for k, v in summary_metrics.items()
                if any(selected_metric in k for selected_metric in metrics)
            }

        fig.add_trace(
            go.Scatterpolar(
                r=list(summary_metrics.values()),
                theta=list(summary_metrics.keys()),
                fill="toself",
                name=title,
            )
        )

    fig.update_layout(
        polar=dict(radialaxis=dict(visible=True, range=[0, max_score])), showlegend=True
    )

    fig.show()

def plot_bar_plot(eval_results, metrics=None):
    fig = go.Figure()
    data = []

    for eval_result in eval_results:
        title, summary_metrics, _ = eval_result
        if metrics:
            summary_metrics = {
                k: summary_metrics[k]
                for k, v in summary_metrics.items()
                if any(selected_metric in k for selected_metric in metrics)
            }

        data.append(
            go.Bar(
                x=list(summary_metrics.keys()),
                y=list(summary_metrics.values()),
                name=title,
            )
        )

    fig = go.Figure(data=data)

    # 막대 모드 변경
    fig.update_layout(barmode="group")
    fig.show()

Ragas 메트릭으로 평가 설정

evaluator_llm 정의

모델 기반 메트릭을 사용하려면 먼저 평가기 LLM과 임베딩을 정의해요.

from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper
from langchain_google_vertexai import VertexAI, VertexAIEmbeddings

evaluator_llm = LangchainLLMWrapper(VertexAI(model_name="gemini-pro"))
evaluator_embeddings = LangchainEmbeddingsWrapper(VertexAIEmbeddings(model_name="text-embedding-004"))

Ragas 메트릭

애플리케이션에 가장 관련 있는 메트릭을 선택하고 정의해요.

from ragas import evaluate
from ragas.metrics import ContextPrecision, Faithfulness, RubricsScore, RougeScore

rouge_score = RougeScore()

helpfulness_rubrics = {
    "score1_description": "Response is useless/irrelevant, contains inaccurate/deceptive/misleading information, and/or contains harmful/offensive content. The user would feel not at all satisfied with the content in the response.",
    "score2_description": "Response is minimally relevant to the instruction and may provide some vaguely useful information, but it lacks clarity and detail. It might contain minor inaccuracies. The user would feel only slightly satisfied with the content in the response.",
    "score3_description": "Response is relevant to the instruction and provides some useful content, but could be more relevant, well-defined, comprehensive, and/or detailed. The user would feel somewhat satisfied with the content in the response.",
    "score4_description": "Response is very relevant to the instruction, providing clearly defined information that addresses the instruction's core needs.  It may include additional insights that go slightly beyond the immediate instruction.  The user would feel quite satisfied with the content in the response.",
    "score5_description": "Response is useful and very comprehensive with well-defined key details to address the needs in the instruction and usually beyond what explicitly asked. The user would feel very satisfied with the content in the response.",
}

rubrics_score = RubricsScore(name="helpfulness", rubrics=helpfulness_rubrics)
context_precision = ContextPrecision(llm=evaluator_llm)
faithfulness = Faithfulness(llm=evaluator_llm)

데이터셋 준비

Ragas 메트릭으로 평가를 수행하려면 데이터를 Ragas의 핵심 데이터 타입인 EvaluationDataset으로 변환해야 해요. 구조에 대한 자세한 내용은 Ragas 문서를 참고해요.

# 사용자의 질문 또는 쿼리
user_inputs = [
    "Which part of the brain does short-term memory seem to rely on?",
    "What provided the Roman senate with exuberance?",
    "What area did the Hasan-jalalians command?",
]

# 답변 생성에 사용된 검색 데이터
retrieved_contexts = [
    ["Short-term memory is supported by transient patterns of neuronal communication, dependent on regions of the frontal lobe (especially dorsolateral prefrontal cortex) and the parietal lobe. Long-term memory, on the other hand, is maintained by more stable and permanent changes in neural connections widely spread throughout the brain. The hippocampus is essential (for learning new information) to the consolidation of information from short-term to long-term memory, although it does not seem to store information itself. Without the hippocampus, new memories are unable to be stored into long-term memory, as learned from patient Henry Molaison after removal of both his hippocampi, and there will be a very short attention span. Furthermore, it may be involved in changing neural connections for a period of three months or more after the initial learning."],
    ["In 62 BC, Pompey returned victorious from Asia. The Senate, elated by its successes against Catiline, refused to ratify the arrangements that Pompey had made. Pompey, in effect, became powerless. Thus, when Julius Caesar returned from a governorship in Spain in 61 BC, he found it easy to make an arrangement with Pompey. Caesar and Pompey, along with Crassus, established a private agreement, now known as the First Triumvirate. Under the agreement, Pompey's arrangements would be ratified. Caesar would be elected consul in 59 BC, and would then serve as governor of Gaul for five years. Crassus was promised a future consulship."],
    ["The Seljuk Empire soon started to collapse. In the early 12th century, Armenian princes of the Zakarid noble family drove out the Seljuk Turks and established a semi-independent Armenian principality in Northern and Eastern Armenia, known as Zakarid Armenia, which lasted under the patronage of the Georgian Kingdom. The noble family of Orbelians shared control with the Zakarids in various parts of the country, especially in Syunik and Vayots Dzor, while the Armenian family of Hasan-Jalalians controlled provinces of Artsakh and Utik as the Kingdom of Artsakh."],
]

# 기대 응답 또는 ground truth
references = [
    "frontal lobe and the parietal lobe",
    "Due to successes against Catiline.",
    "The Hasan-Jalalians commanded the area of Artsakh and Utik.",
]

from vertexai.generative_models import GenerativeModel

generation_config = {
    "max_output_tokens": 128,
    "temperature": 0.1,
}

model_a_name = "gemini-1.5-pro"
model_b_name = "gemini-1.0-pro"

gemini_model_15 = GenerativeModel(
    model_a_name,
    generation_config=generation_config,
)

gemini_model_1 = GenerativeModel(
    model_b_name,
    generation_config=generation_config,
)

responses_a = []
responses_b = []

# 프롬프트 생성을 위한 템플릿
template = """Answer the question based only on the following context:
{context}

Question: {query}

"""

# 각 사용자 입력과 해당 컨텍스트를 반복
for i in range(len(user_inputs)):
    # 검색된 컨텍스트 리스트를 단일 문자열로 결합
    context_str = "\n".join(retrieved_contexts[i])

    # Gemini 1.5 pro 모델에 대한 프롬프트 생성 및 응답
    gemini_15_prompt = template.format(context=context_str, query=user_inputs[i])

    gemini_15_response = gemini_model_15.generate_content(gemini_15_prompt)
    responses_a.append(gemini_15_response.text)

    # Gemini 1 pro 모델에 대한 프롬프트 생성 및 응답
    gemini_1_prompt = template.format(context=context_str, query=user_inputs[i])

    gemini_1_response = gemini_model_1.generate_content(gemini_1_prompt)
    responses_b.append(gemini_1_response.text)

이것들을 Ragas EvaluationDataset으로 변환해요.

from ragas.dataset_schema import SingleTurnSample, EvaluationDataset

n = len(user_inputs)

samples_a = []
samples_b = []

for i in range(n):
    sample_a = SingleTurnSample(
        user_input=user_inputs[i],
        retrieved_contexts=retrieved_contexts[i],
        response=responses_a[i],
        reference=references[i],
    )
    sample_b = SingleTurnSample(
        user_input=user_inputs[i],
        retrieved_contexts=retrieved_contexts[i],
        response=responses_b[i],
        reference=references[i],
    )

    samples_a.append(sample_a)
    samples_b.append(sample_b)

ragas_eval_dataset_a = EvaluationDataset(samples=samples_a)
ragas_eval_dataset_b = EvaluationDataset(samples=samples_b)

ragas_eval_dataset_a.to_pandas()

출력

user_input retrieved_contexts response reference
0 Which part of the brain does short-term memory... [Short-term memory is supported by transient p... Short-term memory relies on regions of the **f... frontal lobe and the parietal lobe
1 What provided the Roman senate with exuberance? [In 62 BC, Pompey returned victorious from Asi... The Roman Senate was elated by its successes a... Due to successes against Catiline.
2 What area did the Hasan-jalalians command? [The Seljuk Empire soon started to collapse. I... The Hasan-Jalalians controlled the provinces o... The Hasan-Jalalians commanded the area of Arts... 
ragas_eval_dataset_b.to_pandas()

출력

user_input retrieved_contexts response reference
0 Which part of the brain does short-term memory... [Short-term memory is supported by transient p... The frontal lobe, especially the dorsolateral ... frontal lobe and the parietal lobe
1 What provided the Roman senate with exuberance? [In 62 BC, Pompey returned victorious from Asi... The Roman Senate's exuberance stemmed from its... Due to successes against Catiline.
2 What area did the Hasan-jalalians command? [The Seljuk Empire soon started to collapse. I... The Hasan-Jalalians controlled the provinces o... The Hasan-Jalalians commanded the area of Arts...

평가 실행

데이터셋과 원하는 메트릭 목록을 evaluate 함수에 전달해서 데이터셋을 Ragas로 평가해요.

from ragas import evaluate

ragas_metrics = [
    context_precision,
    faithfulness,
    rouge_score,
    rubrics_score,
]

ragas_result_rag_a = evaluate(
    dataset=ragas_eval_dataset_a, metrics=ragas_metrics, llm=evaluator_llm
)

ragas_result_rag_b = evaluate(
    dataset=ragas_eval_dataset_b, metrics=ragas_metrics, llm=evaluator_llm
)

Evaluating: 100%|██████████| 12/12 [00:00<?, ?it/s]

Evaluating: 100%|██████████| 12/12 [00:00<?, ?it/s]

결과를 Google의 EvalResult 구조로 감싸요.

from vertexai.evaluation import EvalResult

result_rag_a = EvalResult(
    summary_metrics=ragas_result_rag_a._repr_dict,
    metrics_table=ragas_result_rag_a.to_pandas(),
)

result_rag_b = EvalResult(
    summary_metrics=ragas_result_rag_b._repr_dict,
    metrics_table=ragas_result_rag_b.to_pandas(),
)

평가 결과 비교

요약 결과 보기

모든 평가 메트릭의 포괄적인 요약을 단일 테이블로 보려면 display_eval_report() 헬퍼 함수를 호출해요.

display_eval_report(
    eval_result=(
        f"{model_a_name} Eval Result",
        result_rag_a.summary_metrics,
        result_rag_a.metrics_table,
    ),
)

출력

gemini-1.5-pro Eval Result

Summary Metrics

context_precision faithfulness rouge_score(mode=fmeasure) helpfulness
0 0.666667 1.0 0.56 4.333333

Report Metrics

user_input retrieved_contexts response reference context_precision faithfulness rouge_score(mode=fmeasure) helpfulness
0 Which part of the brain does short-term memory... [Short-term memory is supported by transient p... Short-term memory relies on regions of the **f... frontal lobe and the parietal lobe 1.0 1.0 0.48 5
1 What provided the Roman senate with exuberance? [In 62 BC, Pompey returned victorious from Asi... The Roman Senate was elated by its successes a... Due to successes against Catiline. 0.0 1.0 0.40 4
2 What area did the Hasan-jalalians command? [The Seljuk Empire soon started to collapse. I... The Hasan-Jalalians controlled the provinces o... The Hasan-Jalalians commanded the area of Arts... 1.0 1.0 0.80 4 
display_eval_report(
    (
        f"{model_b_name} Eval Result",
        result_rag_b.summary_metrics,
        result_rag_b.metrics_table,
    )
)

출력

gemini-1.0-pro Eval Result

Summary Metrics

context_precision faithfulness rouge_score(mode=fmeasure) helpfulness
0 1.0 0.916667 0.479034 4.0

Report Metrics

user_input retrieved_contexts response reference context_precision faithfulness rouge_score(mode=fmeasure) helpfulness
0 Which part of the brain does short-term memory... [Short-term memory is supported by transient p... The frontal lobe, especially the dorsolateral ... frontal lobe and the parietal lobe 1.0 1.00 0.666667 4
1 What provided the Roman senate with exuberance? [In 62 BC, Pompey returned victorious from Asi... The Roman Senate's exuberance stemmed from its... Due to successes against Catiline. 1.0 0.75 0.130435 4
2 What area did the Hasan-jalalians command? [The Seljuk Empire soon started to collapse. I... The Hasan-Jalalians controlled the provinces o... The Hasan-Jalalians commanded the area of Arts... 1.0 1.00 0.640000 4

평가 결과 시각화

eval_results = []

eval_results.append(
    (model_a_name, result_rag_a.summary_metrics, result_rag_a.metrics_table)
)
eval_results.append(
    (model_b_name, result_rag_b.summary_metrics, result_rag_b.metrics_table)
)

plot_radar_plot(eval_results, max_score=5)

Radar Plot

plot_bar_plot(eval_results)

Bar Plot

이 시리즈의 다른 튜토리얼도 확인해요:

  • Ragas with Vertex AI: Vertex AI 모델을 Ragas와 함께 사용해서 LLM 워크플로우를 평가하는 방법 학습.
  • Align LLM Metrics: LLM 평가기를 훈련하고 정렬해서 인간 판단과 더 잘 일치시키기.

더 알아보기 (Learn more)