Llama 4로 LlamaStack 웹 검색 Groundedness 평가하기

Llama 4로 LlamaStack 웹 검색 Groundedness 평가하기

이 튜토리얼에서는 LlamaStack의 웹 검색 에이전트가 생성한 응답의 groundedness를 측정할 거예요. LlamaStack은 meta가 유지 관리하는 오픈소스 프레임워크로, 대규모 언어 모델 기반 애플리케이션의 개발·배포를 간소화해요. 평가는 Ragas 메트릭으로 진행하고 Meta Llama 4 Maverick을 판정자(judge)로 사용할 거예요.

출처: 문서

본문

LlamaStack 서버 설정 및 실행

이 명령은 together inference 프로바이더와 함께 LlamaStack 서버에 필요한 모든 의존성을 설치해요.

conda 명령 사용:

!pip install ragas langchain-together uv 
!uv run --with llama-stack llama stack build --template together --image-type conda

venv 명령 사용:

!pip install ragas langchain-together uv 
!uv run --with llama-stack llama stack build --template together --image-type venv

import os
import subprocess


def run_llama_stack_server_background():
    log_file = open("llama_stack_server.log", "w")
    process = subprocess.Popen(
        "uv run --with llama-stack llama stack run together --image-type venv",
        shell=True,
        stdout=log_file,
        stderr=log_file,
        text=True,
    )

    print(f"Starting LlamaStack server with PID: {process.pid}")
    return process


def wait_for_server_to_start():
    import requests
    from requests.exceptions import ConnectionError
    import time

    url = "http://0.0.0.0:8321/v1/health"
    max_retries = 30
    retry_interval = 1

    print("Waiting for server to start", end="")
    for _ in range(max_retries):
        try:
            response = requests.get(url)
            if response.status_code == 200:
                print("\nServer is ready!")
                return True
        except ConnectionError:
            print(".", end="", flush=True)
            time.sleep(retry_interval)

    print("\nServer failed to start after", max_retries * retry_interval, "seconds")
    return False


# use this helper if needed to kill the server
def kill_llama_stack_server():
    # Kill any existing llama stack server processes
    os.system(
        "ps aux | grep -v grep | grep llama_stack.distribution.server.server | awk '{print $2}' | xargs kill -9"
    )

LlamaStack 서버 시작

server_process = run_llama_stack_server_background()
assert wait_for_server_to_start()

Starting LlamaStack server with PID: 95508
Waiting for server to start....
Server is ready!

검색 에이전트 구축

from llama_stack_client import LlamaStackClient, Agent, AgentEventLogger

client = LlamaStackClient(
    base_url="http://0.0.0.0:8321",
)

agent = Agent(
    client,
    model="meta-llama/Llama-3.1-8B-Instruct",
    instructions="You are a helpful assistant. Use web search tool to answer the questions.",
    tools=["builtin::websearch"],
)
user_prompts = [
    "In which major did Demis Hassabis complete his undergraduate degree? Search the web for the answer.",
    "Ilya Sutskever is one of the key figures in AI. From which institution did he earn his PhD in machine learning? Search the web for the answer.",
    "Sam Altman, widely known for his role at OpenAI, was born in which American city? Search the web for the answer.",
]

session_id = agent.create_session("test-session")


for prompt in user_prompts:
    response = agent.create_turn(
        messages=[
            {
                "role": "user",
                "content": prompt,
            }
        ],
        session_id=session_id,
    )
    for log in AgentEventLogger().log(response):
        log.print()

이제 에이전트의 실행 단계를 더 깊이 살펴보고 에이전트가 얼마나 잘 수행하는지 확인해 보겠습니다.

session_response = client.agents.session.retrieve(
    session_id=session_id,
    agent_id=agent.agent_id,
)

에이전트 응답 평가

LlamaStack 웹 검색 에이전트가 생성한 응답의 Groundedness를 측정하려고 해요. 이를 위해 EvaluationDataset 과 grounded 응답을 평가하는 메트릭이 필요해요. Ragas는 검색과 생성의 다양한 측면을 측정하는 데 쓸 수 있는 다양한 즉시 사용 가능한(off the shelf) 메트릭을 제공해요.

응답의 groundedness를 측정하기 위해 다음을 사용할 거예요.

• Faithfulness
• Response Groundedness

Ragas EvaluationDataset 구성

Ragas로 평가를 수행하기 위해 EvaluationDataset 을 만들 거예요.

import json

# This function extracts the search results for the trace of each query
def extract_retrieved_contexts(turn_object):
    results = []
    for step in turn_object.steps:
        if step.step_type == "tool_execution":
            tool_responses = step.tool_responses
            for response in tool_responses:
                content = response.content
                if content:
                    try:
                        parsed_result = json.loads(content)
                        results.append(parsed_result)
                    except json.JSONDecodeError:
                        print("Warning: Unable to parse tool response content as JSON.")
                        continue

    retrieved_context = []
    for result in results:
        top_content_list = [item["content"] for item in result["top_k"]]
        retrieved_context.extend(top_content_list)
    return retrieved_context

from ragas.dataset_schema import EvaluationDataset

samples = []

references = [
    "Demis Hassabis completed his undergraduate degree in Computer Science.",
    "Ilya Sutskever earned his PhD from the University of Toronto.",
    "Sam Altman was born in Chicago, Illinois.",
]

for i, turn in enumerate(session_response.turns):
    samples.append(
        {
            "user_input": turn.input_messages[0].content,
            "response": turn.output_message.content,
            "reference": references[i],
            "retrieved_contexts": extract_retrieved_contexts(turn),
        }
    )

ragas_eval_dataset = EvaluationDataset.from_list(samples)

ragas_eval_dataset.to_pandas()

Ragas 메트릭 설정

from ragas.metrics import AnswerAccuracy, Faithfulness, ResponseGroundedness
from langchain_together import ChatTogether
from ragas.llms import LangchainLLMWrapper

llm = ChatTogether(
    model="meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8",
)
evaluator_llm = LangchainLLMWrapper(llm)

ragas_metrics = [
    AnswerAccuracy(llm=evaluator_llm),
    Faithfulness(llm=evaluator_llm),
    ResponseGroundedness(llm=evaluator_llm),
]

평가

마지막으로 평가를 실행해 봅시다.

from ragas import evaluate

results = evaluate(dataset=ragas_eval_dataset, metrics=ragas_metrics)
results.to_pandas()

Evaluating: 100%|██████████| 9/9 [00:04<00:00,  2.03it/s]

kill_llama_stack_server()

더 알아보기 (Learn more)

• 통합 목록