노이즈 민감도(Noise Sensitivity)

노이즈 민감도(Noise Sensitivity)

노이즈 민감도는 시스템이 관련 있거나 관련 없는 검색 문서를 사용할 때 잘못된 응답을 내는 오류가 얼마나 자주 발생하는지 측정하는 지표예요. 점수는 0에서 1 사이이며 낮을수록 성능이 좋다는 뜻이에요. user_input, reference, response, retrieved_contexts로 계산돼요.

출처: 문서

본문

노이즈 민감도(Noise Sensitivity)

NoiseSensitivity는 시스템이 관련 있거나 관련 없는 검색 문서를 활용할 때 잘못된 응답을 제공하는 오류를 얼마나 자주 내는지 측정해요. 점수는 0에서 1 사이이며, 값이 낮을수록 성능이 좋다는 뜻이에요. 노이즈 민감도는 user_input, reference, response, retrieved_contexts를 사용해 계산돼요.

노이즈 민감도를 추정하기 위해 생성된 응답의 각 claim을 검사해, 정답(ground truth)을 기준으로 옳은지 그리고 관련(또는 관련 없는) 검색 컨텍스트에 귀속될 수 있는지 판단해요. 이상적으로는 답변의 모든 claim이 관련 검색 컨텍스트에 의해 뒷받침되어야 해요.

[ \text{noise sensitivity (relevant)} = {|\text{Total number of incorrect claims in response}| \over |\text{Total number of claims in the response}|} ]

예시(Example)

from openai import AsyncOpenAI
from ragas.llms import llm_factory
from ragas.metrics.collections import NoiseSensitivity

# Setup LLM
client = AsyncOpenAI()
llm = llm_factory("gpt-4o-mini", client=client)

# Create metric
scorer = NoiseSensitivity(llm=llm)

# Evaluate
result = await scorer.ascore(
    user_input="What is the Life Insurance Corporation of India (LIC) known for?",
    response="The Life Insurance Corporation of India (LIC) is the largest insurance company in India, known for its vast portfolio of investments. LIC contributes to the financial stability of the country.",
    reference="The Life Insurance Corporation of India (LIC) is the largest insurance company in India, established in 1956 through the nationalization of the insurance industry. It is known for managing a large portfolio of investments.",
    retrieved_contexts=[
        "The Life Insurance Corporation of India (LIC) was established in 1956 following the nationalization of the insurance industry in India.",
        "LIC is the largest insurance company in India, with a vast network of policyholders and huge investments.",
        "As the largest institutional investor in India, LIC manages substantial funds, contributing to the financial stability of the country.",
        "The Indian economy is one of the fastest-growing major economies in the world, thanks to sectors like finance, technology, manufacturing etc."
    ]
)
print(f"Noise Sensitivity Score: {result.value}")

출력(Output):

Noise Sensitivity Score: 0.3333333333333333

관련 없는 컨텍스트의 노이즈 민감도를 계산하려면 mode 파라미터를 irrelevant로 설정해요:

scorer = NoiseSensitivity(llm=llm, mode="irrelevant")
result = await scorer.ascore(
    user_input="What is the Life Insurance Corporation of India (LIC) known for?",
    response="The Life Insurance Corporation of India (LIC) is the largest insurance company in India, known for its vast portfolio of investments. LIC contributes to the financial stability of the country.",
    reference="The Life Insurance Corporation of India (LIC) is the largest insurance company in India, established in 1956 through the nationalization of the insurance industry. It is known for managing a large portfolio of investments.",
    retrieved_contexts=[
        "The Life Insurance Corporation of India (LIC) was established in 1956 following the nationalization of the insurance industry in India.",
        "LIC is the largest insurance company in India, with a vast network of policyholders and huge investments.",
        "As the largest institutional investor in India, LIC manages substantial funds, contributing to the financial stability of the country.",
        "The Indian economy is one of the fastest-growing major economies in the world, thanks to sectors like finance, technology, manufacturing etc."
    ]
)
print(f"Noise Sensitivity (Irrelevant) Score: {result.value}")

출력(Output):

Noise Sensitivity (Irrelevant) Score: 0.0

동기 사용법(Synchronous Usage) 동기 코드를 선호한다면 .ascore() 대신 .score() 메서드를 쓸 수 있어요: result = scorer.score( user_input="What is the Life Insurance Corporation of India (LIC) known for?", response="The Life Insurance Corporation of India (LIC) is the largest insurance company in India...", reference="The Life Insurance Corporation of India (LIC) is the largest insurance company...", retrieved_contexts=[...] )

계산 방법(How It's Calculated)

예시(Example) 질문: What is the Life Insurance Corporation of India (LIC) known for? Ground truth: The Life Insurance Corporation of India (LIC) is the largest insurance company in India, established in 1956 through the nationalization of the insurance industry. It is known for managing a large portfolio of investments. Relevant Retrieval: - The Life Insurance Corporation of India (LIC) was established in 1956 following the nationalization of the insurance industry in India. - LIC is the largest insurance company in India, with a vast network of policyholders and a significant role in the financial sector. - As the largest institutional investor in India, LIC manages a substantial life fund, contributing to the financial stability of the country. Irrelevant Retrieval: - The Indian economy is one of the fastest-growing major economies in the world, thanks to the sectors like finance, technology, manufacturing etc.

관련 컨텍스트에서 노이즈 민감도가 어떻게 계산됐는지 살펴볼게요:

  • Step 1: ground truth를 추론할 수 있는 관련 컨텍스트를 식별해요.

    Ground Truth: The Life Insurance Corporation of India (LIC) is the largest insurance company in India, established in 1956 through the nationalization of the insurance industry. It is known for managing a large portfolio of investments.

    Contexts:

    Context 1: The Life Insurance Corporation of India (LIC) was established in 1956 following the nationalization of the insurance industry in India. Context 2: LIC is the largest insurance company in India, with a vast network of policyholders and a significant role in the financial sector. Context 3: As the largest institutional investor in India, LIC manages a substantial funds`, contributing to the financial stability of the country.

  • Step 2: 생성된 답변의 claim이 관련 컨텍스트에서 추론될 수 있는지 확인해요.

    Answer: The Life Insurance Corporation of India (LIC) is the largest insurance company in India, known for its vast portfolio of investments. LIC contributes to the financial stability of the country.

    Contexts:

    Context 1: The Life Insurance Corporation of India (LIC) was established in 1956 following the nationalization of the insurance industry in India. Context 2: LIC is the largest insurance company in India, with a vast network of policyholders and a significant role in the financial sector. Context 3: As the largest institutional investor in India, LIC manages a substantial funds, contributing to the financial stability of the country.

  • Step 3: 답변에서 잘못된 claim을 식별해요(즉, ground truth로 뒷받침되지 않는 답변 문장).

    Ground Truth: The Life Insurance Corporation of India (LIC) is the largest insurance company in India, established in 1956 through the nationalization of the insurance industry. It is known for managing a large portfolio of investments.

    Answer: The Life Insurance Corporation of India (LIC) is the largest insurance company in India, known for its vast portfolio of investments. LIC contributes to the financial stability of the country.

    설명: ground truth는 LIC가 국가 금융 안정에 기여한다는 내용을 언급하지 않아요. 따라서 답변의 이 문장은 잘못된 문장이에요. 잘못된 문장: 1 전체 claim: 3

  • Step 4: 공식을 사용해 노이즈 민감도를 계산해요:

[ \text{noise sensitivity} = { \text{1} \over \text{3} } = 0.333 ]

그 결과 노이즈 민감도 점수는 0.333이 되며, 답변의 claim 3개 중 1개가 잘못됐다는 뜻이에요.

레거시 지표 API(Legacy Metrics API)

다음 예시는 레거시 지표 API 패턴을 사용해요. 새 프로젝트에는 위에서 보여준 컬렉션 기반 API를 권장해요.

폐지 일정(Deprecation Timeline) 이 API는 버전 0.4에서 폐지되고 버전 1.0에서 제거될 예정이에요. 위에 보여준 컬렉션 기반 API로 마이그레이션해주세요.

SingleTurnSample과 함께하는 예시(Example with SingleTurnSample)

from ragas.dataset_schema import SingleTurnSample
from ragas.metrics import NoiseSensitivity

sample = SingleTurnSample(
    user_input="What is the Life Insurance Corporation of India (LIC) known for?",
    response="The Life Insurance Corporation of India (LIC) is the largest insurance company in India, known for its vast portfolio of investments. LIC contributes to the financial stability of the country.",
    reference="The Life Insurance Corporation of India (LIC) is the largest insurance company in India, established in 1956 through the nationalization of the insurance industry. It is known for managing a large portfolio of investments.",
    retrieved_contexts=[
        "The Life Insurance Corporation of India (LIC) was established in 1956 following the nationalization of the insurance industry in India.",
        "LIC is the largest insurance company in India, with a vast network of policyholders and huge investments.",
        "As the largest institutional investor in India, LIC manages substantial funds, contributing to the financial stability of the country.",
        "The Indian economy is one of the fastest-growing major economies in the world, thanks to sectors like finance, technology, manufacturing etc."
    ]
)

scorer = NoiseSensitivity(llm=evaluator_llm)
await scorer.single_turn_ascore(sample)

출력(Output):

0.3333333333333333

관련 없는 컨텍스트의 노이즈 민감도를 계산하려면 mode 파라미터를 irrelevant로 설정해요:

scorer = NoiseSensitivity(mode="irrelevant")
await scorer.single_turn_ascore(sample)

Credit: 노이즈 민감도는 RAGChecker에서 도입되었어요.