루브릭 기반 평가(Rubric-Based Evaluation)

루브릭 기반 평가(Rubric-Based Evaluation)

루브릭 기반 평가 지표는 커스텀 채점 기준으로 LLM 응답을 평가하게 해줘요. Ragas는 두 가지 루브릭 지표를 제공해요. 데이터셋 전체에 같은 루브릭을 적용하는 DomainSpecificRubrics와, 샘플마다 고유한 루브릭을 쓸 수 있는 InstanceSpecificRubrics죠. 보통 각 점수(1~5)에 대한 설명으로 루브릭을 구성해요.

출처: 문서

본문

루브릭 기반 평가 지표는 커스텀 채점 기준으로 LLM 응답을 평가하게 해줘요. Ragas는 두 가지 유형의 루브릭 지표를 제공해요:

1. DomainSpecificRubrics: 데이터셋의 모든 샘플에 같은 루브릭을 사용해요(초기화 시 설정).
2. InstanceSpecificRubrics: 각 샘플이 자신만의 고유한 루브릭을 가질 수 있어요(평가 시 전달).

루브릭은 보통 1에서 5 사이의 각 점수에 대한 설명으로 구성돼요. 응답은 루브릭에 지정된 설명에 따라 LLM으로 평가·채점돼요.

도메인별 루브릭(Domain-Specific Rubrics)

모든 샘플에 동일한 평가 기준을 적용하려면 DomainSpecificRubrics를 사용해요. 채점 기준이 일정하게 유지되는 도메인 전체 평가에 유용해요.

예시(Example)

from openai import AsyncOpenAI
from ragas.llms.base import llm_factory
from ragas.metrics.collections import DomainSpecificRubrics

# Setup
client = AsyncOpenAI()
llm = llm_factory("gpt-4o-mini", client=client)

# Reference-free evaluation (default)
metric = DomainSpecificRubrics(llm=llm)
result = await metric.ascore(
    user_input="What's the longest river in the world?",
    response="The longest river in the world is the Nile, stretching approximately 6,650 kilometers through northeastern Africa.",
)
print(f"Score: {result.value}, Feedback: {result.reason}")

# Reference-based evaluation
metric_with_ref = DomainSpecificRubrics(llm=llm, with_reference=True)
result = await metric_with_ref.ascore(
    user_input="What's the longest river in the world?",
    response="The longest river in the world is the Nile.",
    reference="The Nile is a major north-flowing river in northeastern Africa.",
)

커스텀 루브릭(Custom Rubrics)

채점 기준을 커스터마이즈하려면 자신만의 루브릭을 정의할 수 있어요:

from ragas.metrics.collections import DomainSpecificRubrics

my_custom_rubrics = {
    "score1_description": "Answer and ground truth are completely different",
    "score2_description": "Answer and ground truth are somewhat different",
    "score3_description": "Answer and ground truth are somewhat similar",
    "score4_description": "Answer and ground truth are similar",
    "score5_description": "Answer and ground truth are exactly the same",
}

metric = DomainSpecificRubrics(llm=llm, rubrics=my_custom_rubrics, with_reference=True)

검색된 컨텍스트와 함께(With Retrieved Contexts)

이 지표는 검색된 컨텍스트와의 평가도 지원해요:

result = await metric.ascore(
    user_input="What's the longest river in the world?",
    response="Based on the context, the Nile is the longest river.",
    retrieved_contexts=[
        "Scientists debate whether the Amazon or the Nile is the longest river.",
        "The Nile River was central to Ancient Egyptians' wealth and power.",
    ],
)

편의 클래스(Convenience Classes)

더 명확한 의도를 위해 편의 클래스를 사용해요:

from ragas.metrics.collections import (
    RubricsScoreWithoutReference,
    RubricsScoreWithReference,
)

# Reference-free
metric_no_ref = RubricsScoreWithoutReference(llm=llm)

# Reference-based
metric_with_ref = RubricsScoreWithReference(llm=llm)

기본 루브릭(Default Rubrics)

Reference-Free 루브릭(기본값)

Reference-Based 루브릭

인스턴스별 루브릭(Instance-Specific Rubrics)

서로 다른 샘플에 서로 다른 평가 기준이 필요할 때 InstanceSpecificRubrics를 사용해요. 다음과 같은 경우에 유용해요:

• 서로 다른 질문이 서로 다른 평가 기준을 요구할 때
• 특정 작업 요구사항에 맞춰 채점을 커스터마이즈하고 싶을 때
• 데이터셋 전반에서 평가 기준이 다양할 때

예시(Example)

from openai import AsyncOpenAI
from ragas.llms.base import llm_factory
from ragas.metrics.collections import InstanceSpecificRubrics

# Setup
client = AsyncOpenAI()
llm = llm_factory("gpt-4o-mini", client=client)

metric = InstanceSpecificRubrics(llm=llm)

# Each sample can have its own rubrics
email_rubrics = {
    "score1_description": "The email is unprofessional or inappropriate",
    "score2_description": "The email lacks proper formatting or tone",
    "score3_description": "The email is acceptable but could be improved",
    "score4_description": "The email is professional with minor issues",
    "score5_description": "The email is highly professional and well-written",
}

result = await metric.ascore(
    user_input="Write a professional email declining a meeting invitation",
    response="Dear John, Thank you for the invitation...",
    rubrics=email_rubrics,
)
print(f"Score: {result.value}, Feedback: {result.reason}")

# Different rubrics for a different type of task
code_rubrics = {
    "score1_description": "The code doesn't work or has critical bugs",
    "score2_description": "The code has significant issues or is poorly structured",
    "score3_description": "The code works but lacks optimization or best practices",
    "score4_description": "The code is good with minor improvements possible",
    "score5_description": "The code is excellent, efficient, and follows best practices",
}

result = await metric.ascore(
    user_input="Write a function to sort a list",
    response="def sort_list(arr): return sorted(arr)",
    rubrics=code_rubrics,
)

Reference와 컨텍스트와 함께(With Reference and Contexts)

result = await metric.ascore(
    user_input="Explain the water cycle",
    response="The water cycle involves evaporation, condensation, and precipitation.",
    reference="The water cycle describes how water evaporates from surfaces, rises into the atmosphere, condenses into clouds, and falls as precipitation.",
    retrieved_contexts=["Water cycle information from encyclopedia..."],
    rubrics={
        "score1_description": "Explanation is completely wrong",
        "score2_description": "Explanation has major inaccuracies",
        "score3_description": "Explanation is partially correct",
        "score4_description": "Explanation is mostly correct",
        "score5_description": "Explanation is comprehensive and accurate",
    },
)

레거시 API(Legacy API)

Deprecated 아래 레거시 API는 사용이 중단됐어요. ragas.metrics.collections.DomainSpecificRubrics 또는 ragas.metrics.collections.InstanceSpecificRubrics를 사용해주세요.

from ragas import evaluate
from datasets import Dataset

from ragas.metrics import rubrics_score_without_reference, rubrics_score_with_reference

rows = {
    "question": [
        "What's the longest river in the world?",
    ],
    "ground_truth": [
        "The Nile is a major north-flowing river in northeastern Africa.",
    ],
    "answer": [
        "The longest river in the world is the Nile, stretching approximately 6,650 kilometers (4,130 miles) through northeastern Africa.",
    ],
    "contexts": [
        [
            "Scientists debate whether the Amazon or the Nile is the longest river in the world.",
            "The Nile River was central to the Ancient Egyptians' rise to wealth and power.",
        ],
    ]
}

dataset = Dataset.from_dict(rows)

result = evaluate(
    dataset,
    metrics=[
        rubrics_score_without_reference,
        rubrics_score_with_reference
    ],
)

레거시 API로 커스텀 루브릭 사용:

from ragas.metrics._domain_specific_rubrics import RubricsScore

my_custom_rubrics = {
    "score1_description": "answer and ground truth are completely different",
    "score2_description": "answer and ground truth are somewhat different",
    "score3_description": "answer and ground truth are somewhat similar",
    "score4_description": "answer and ground truth are similar",
    "score5_description": "answer and ground truth are exactly the same",
}

rubrics_score = RubricsScore(rubrics=my_custom_rubrics)

더 알아보기 (Learn more)

• 범용 지표(General Purpose Metrics)