LiteLLM으로 여러 LLM 제공자 신뢰성 테스트
LiteLLM으로 여러 LLM 제공자 신뢰성 테스트
여러 LLM 제공자에 걸쳐 품질(Quality), 부하(Load), 지속시간(Duration) 테스트를 어떻게 수행하는지 알아봐요.
출처: 문서
본문
-
품질 테스트
-
부하 테스트
-
지속시간 테스트
uv add litellm python-dotenv
import litellmfrom litellm import testing_batch_completionfrom litellm.utils import load_test_modelimport time
from dotenv import load_dotenvload_dotenv()
품질 테스트 엔드포인트
여러 LLM 제공자에 걸쳐 같은 프롬프트 테스트
이 예시에서는 Paul Graham에 관한 몇 가지 질문을 해볼게요.
models = ["gpt-5.6-luna", "gpt-5.6-terra", "claude-sonnet-5", "replicate/llama-2-70b-chat:58d078176e02c219e11eb4da5a02a7830a283b14cf8f94537af893ccff5ee781"]context = """Paul Graham (/ɡræm/; born 1964)[3] is an English computer scientist, essayist, entrepreneur, venture capitalist, and author. He is best known for his work on the programming language Lisp, his former startup Viaweb (later renamed Yahoo! Store), cofounding the influential startup accelerator and seed capital firm Y Combinator, his essays, and Hacker News. He is the author of several computer programming books, including: On Lisp,[4] ANSI Common Lisp,[5] and Hackers & Painters.[6] Technology journalist Steven Levy has described Graham as a "hacker philosopher".[7] Graham was born in England, where he and his family maintain permanent residence. However he is also a citizen of the United States, where he was educated, lived, and worked until 2016."""prompts = ["Who is Paul Graham?", "What is Paul Graham known for?" , "Is paul graham a writer?" , "Where does Paul Graham live?", "What has Paul Graham done?"]messages = [[{"role": "user", "content": context + "\n" + prompt}] for prompt in prompts] # pass in a list of messages we want to testresult = testing_batch_completion(models=models, messages=messages)
부하 테스트 엔드포인트
여러 제공자에 걸쳐 100개 이상의 동시 쿼리를 실행해 언제 실패하는지와 지연 시간 영향(latency)을 확인해 보세요. load_test_model은 단일 model을 받으므로 제공자마다 한 번씩 호출해야 해요.
models=["gpt-5.6-luna", "replicate/llama-2-70b-chat:58d078176e02c219e11eb4da5a02a7830a283b14cf8f94537af893ccff5ee781", "claude-sonnet-5"]context = """Paul Graham (/ɡræm/; born 1964)[3] is an English computer scientist, essayist, entrepreneur, venture capitalist, and author. He is best known for his work on the programming language Lisp, his former startup Viaweb (later renamed Yahoo! Store), cofounding the influential startup accelerator and seed capital firm Y Combinator, his essays, and Hacker News. He is the author of several computer programming books, including: On Lisp,[4] ANSI Common Lisp,[5] and Hackers & Painters.[6] Technology journalist Steven Levy has described Graham as a "hacker philosopher".[7] Graham was born in England, where he and his family maintain permanent residence. However he is also a citizen of the United States, where he was educated, lived, and worked until 2016."""prompt = "Where does Paul Graham live?"final_prompt = context + promptresult = {model: load_test_model(model=model, prompt=final_prompt, num_calls=5) for model in models}
데이터 시각화
import matplotlib.pyplot as plt## calculate avg response timeavg_response_time = {}for model, load_result in result.items(): avg_response_time[model] = load_result["total_response_time"] / load_result["calls_made"]models = list(avg_response_time.keys())response_times = list(avg_response_time.values())plt.bar(models, response_times)plt.xlabel('Model', fontsize=10)plt.ylabel('Average Response Time')plt.title('Average Response Times for each Model')plt.xticks(models, [model[:15]+'...' if len(model) > 15 else model for model in models], rotation=45)plt.show()
지속시간 테스트 엔드포인트
2분 동안 부하 테스트를 실행해 보세요. 15초마다 100개 이상의 쿼리로 엔드포인트를 때리는 거예요. load_test_model에는 interval이나 duration 옵션이 없으므로 직접 루프를 돌아야 해요.
models=["gpt-5.6-luna", "replicate/llama-2-70b-chat:58d078176e02c219e11eb4da5a02a7830a283b14cf8f94537af893ccff5ee781", "claude-sonnet-5"]context = """Paul Graham (/ɡræm/; born 1964)[3] is an English computer scientist, essayist, entrepreneur, venture capitalist, and author. He is best known for his work on the programming language Lisp, his former startup Viaweb (later renamed Yahoo! Store), cofounding the influential startup accelerator and seed capital firm Y Combinator, his essays, and Hacker News. He is the author of several computer programming books, including: On Lisp,[4] ANSI Common Lisp,[5] and Hackers & Painters.[6] Technology journalist Steven Levy has described Graham as a "hacker philosopher".[7] Graham was born in England, where he and his family maintain permanent residence. However he is also a citizen of the United States, where he was educated, lived, and worked until 2016."""prompt = "Where does Paul Graham live?"final_prompt = context + promptinterval = 15duration = 120result = []end_time = time.time() + durationwhile time.time() < end_time ...
import matplotlib.pyplot as plt## calculate avg response timemodel_dict = {model: {"response_time": []} for model in models}for iteration in result: for model, load_result in iteration.items(): model_dict[model]["response_time"].append(load_result["total_response_time"] / load_result["calls_made"])avg_response_time = {}for model, data in model_dict.items(): avg_response_time[model] = sum(data["response_time"]) / len(data["response_time"])models = list(avg_response_time.keys())response_times = list(avg_response_time.values())plt.bar(models, response_times)plt.xlabel('Model', fontsize=10)plt.ylabel('Average Response Time')plt.title('Average Response Times for each Model')plt.xticks(models, [model[:15]+'...' if len(model) > 15 else model for model in models], rotation=45)plt.show()