LLM 평가하기 - MLflow Evals, Auto Eval
LLM 평가하기 - MLflow Evals, Auto Eval
LiteLLM을 MLflow와 AutoEvals와 함께 사용해 LLM을 평가하는 방법을 알려드릴게요.
출처: 문서
본문
LiteLLM과 MLflow 사용하기
MLflow는 LLM 평가를 돕는 mlflow.evaluate() API를 제공해요. https://mlflow.org/docs/latest/llms/llm-evaluate/index.html
전제 조건
uv add litellm
uv add mlflow
1단계: CLI에서 LiteLLM Proxy 시작
LiteLLM은 모든 지원 LLM을 위한 OpenAI 호환 서버를 만들 수 있게 해줘요. litellm proxy에 대한 자세한 정보는 여기를 참고하세요.
$ litellm --model huggingface/bigcode/starcoder#INFO: Proxy running on http://0.0.0.0:4000
다른 지원 LLM을 위한 프록시를 만드는 방법은 다음과 같아요.
- Bedrock
- Huggingface (TGI)
- Anthropic
- VLLM
- OpenAI Compatible Server
- TogetherAI
- Replicate
- Petals
- Palm
- Azure OpenAI
- AI21
- Cohere
$ export AWS_ACCESS_KEY_ID=""$ export AWS_REGION_NAME="" # e.g. us-west-2$ export AWS_SECRET_ACCESS_KEY=""
$ litellm --model bedrock/us.anthropic.claude-sonnet-5
$ export HUGGINGFACE_API_KEY=my-api-key #[OPTIONAL]
$ litellm --model huggingface/ --api_base https://k58ory32yinf1ly0.us-east-1.aws.endpoints.huggingface.cloud
$ export ANTHROPIC_API_KEY=my-api-key
$ litellm --model claude-sonnet-5
로컬에서 vllm을 실행 중이라고 가정해요.
$ litellm --model vllm/facebook/opt-125m
$ litellm --model openai/ --api_base
$ export TOGETHERAI_API_KEY=my-api-key
$ litellm --model together_ai/lmsys/vicuna-13b-v1.5-16k
$ export REPLICATE_API_KEY=my-api-key
$ litellm \ --model replicate/meta/llama-2-70b-chat:02e509c789964a7ea8736978a43525956ef40397be9033abf9fd2badfe68c9e3
$ litellm --model petals/meta-llama/Llama-2-70b-chat-hf
$ export PALM_API_KEY=my-palm-key
$ litellm --model palm/chat-bison
$ export AZURE_API_KEY=my-api-key$ export AZURE_API_BASE=my-api-base
$ litellm --model azure/my-deployment-name
$ export AI21_API_KEY=my-api-key
$ litellm --model j2-light
$ export COHERE_API_KEY=my-api-key
$ litellm --model command-nightly
2단계: MLflow 실행
평가를 실행하기 전에 openai.api_base를 1단계의 litellm 프록시로 설정할 거예요.
openai.api_base = "http://0.0.0.0:4000"
import openaiimport pandas as pdopenai.api_key = "anything" # this can be anything, we set the key on the proxyopenai.api_base = "http://0.0.0.0:4000" # set api base to the proxy from step 1import mlfloweval_data = pd.DataFrame( { "inputs": [ "What is the largest country", "What is the weather in sf?", ], "ground_truth": [ "India is a large country", "It's cold in SF today" ], })with mlflow.start_run() as run: system_prompt = "Answer the following question in two sentences" logged_model_info = mlflow.openai.log_model( model="gpt-5.6-luna", task=openai.ChatCompletion, artifact_path="model", messages=[ {"role": "system", "content": system_prompt}, {"role": "user", "content": "{question}"}, ], ) # Use predefined question-answering metrics to evaluate our model. results = mlflow.evaluate( logged_model_info.model_uri, eval_data, targets="ground_truth", model_type="question-answering", ) print(f"See aggregated evaluation results below: \n{results.metrics}") # Evaluation result for each data record is available in `results.tables`. eval_table = results.tables["eval_results_table"] print(f"See evaluation table below: \n{eval_table}")
MLflow 출력
{'toxicity/v1/mean': 0.00014476531214313582, 'toxicity/v1/variance': 2.5759661361262862e-12, 'toxicity/v1/p90': 0.00014604929747292773, 'toxicity/v1/ratio': 0.0, 'exact_match/v1': 0.0}Downloading artifacts: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00https://github.com/braintrustdata/autoevals
전제 조건
uv add litellm
uv add autoevals
빠른 시작
이 코드 샘플에서는 autoevals.llm의 Factuality() 평가기를 사용해 출력이 원본(기대) 값과 비교해 사실적인지 테스트해요.
AutoEvals는 기본적으로 gpt-3.5-turbo / gpt-4-turbo를 사용해 응답을 평가해요
지원되는 평가기(Translation, Summary, Security Evaluators 등)에 대한 자세한 내용은 autoevals 문서를 참고하세요.
# auto evals imports from autoevals.llm import *###################import litellm# litellm completion callquestion = "which country has the highest population"response = litellm.completion( model = "gpt-5.6-luna", messages = [ { "role": "user", "content": question } ],)print(response)# use the auto eval Factuality() evaluatorevaluator = Factuality()result = evaluator( output=response.choices[0]["message"]["content"], # response from litellm.completion() expected="India", # expected output input=question # question passed to litellm.completion)print(result)
AutoEvals의 평가 출력
Score( name='Factuality', score=0, metadata= {'rationale': "The expert answer is 'India'.\nThe submitted answer is 'As of 2021, China has the highest population in the world with an estimated 1.4 billion people.'\nThe submitted answer mentions China as the country with the highest population, while the expert answer mentions India.\nThere is a disagreement between the submitted answer and the expert answer.", 'choice': 'D' }, error=None)