비영어 테스트셋 생성
비영어 테스트셋 생성
영어가 아닌 텍스트로도 합성 테스트셋을 만들 수 있어요. 이 글에서는 비영어 코퍼스(corpus) 환경에 맞게 합성 테스트 데이터 생성을 적용하는 방법을 배워요. 이 튜토리얼에서는 스페인어 위키피디아 문서에서 스페인어 질문을 생성할 거예요.
출처: 문서
본문
코퍼스 다운로드 및 로드
! git clone https://huggingface.co/datasets/vibrantlabsai/Sample_non_english_corpus
Cloning into 'Sample_non_english_corpus'...
remote: Enumerating objects: 12, done.
remote: Counting objects: 100% (8/8), done.
remote: Compressing objects: 100% (8/8), done.
remote: Total 12 (delta 0), reused 0 (delta 0), pack-reused 4 (from 1)
Unpacking objects: 100% (12/12), 11.43 KiB | 780.00 KiB/s, done.
from langchain_community.document_loaders import DirectoryLoader, TextLoader
path = "Sample_non_english_corpus/"
loader = DirectoryLoader(path, glob="**/*.txt")
docs = loader.load()
/opt/homebrew/Caskroom/miniforge/base/envs/ragas/lib/python3.9/site-packages/requests/__init__.py:102: RequestsDependencyWarning: urllib3 (1.26.20) or chardet (5.2.0)/charset_normalizer (None) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({})/charset_normalizer ({}) doesn't match a supported "
len(docs)
6
필요한 모델 초기화
from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import OpenAIEmbeddings
from langchain_openai import ChatOpenAI
import openai
generator_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o-mini"))
openai_client = openai.OpenAI()
generator_embeddings = OpenAIEmbeddings(client=openai_client)
/opt/homebrew/Caskroom/miniforge/base/envs/ragas/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
from .autonotebook import tqdm as notebook_tqdm
Persona와 변환 설정
이 노트북을 사용해서 persona를 자동으로 만들 수도 있어요. 단순함을 위해 미리 정의된 persona 하나와 두 개의 기본 변환, 단순한 query distribution을 사용할 거예요.
from ragas.testset.persona import Persona
personas = [
Persona(
name="curious student",
role_description="A student who is curious about the world and wants to learn more about different cultures and languages",
),
]
from ragas.testset.transforms.extractors.llm_based import NERExtractor
from ragas.testset.transforms.splitters import HeadlineSplitter
transforms = [HeadlineSplitter(), NERExtractor()]
테스트 생성기 초기화
from ragas.testset import TestsetGenerator
generator = TestsetGenerator(
llm=generator_llm, embedding_model=generator_embeddings, persona_list=personas
)
쿼리 로드 및 적용
여기서 필요한 쿼리 유형을 로드하고 타겟 언어로 적용해요.
from ragas.testset.synthesizers.single_hop.specific import (
SingleHopSpecificQuerySynthesizer,
)
distribution = [
(SingleHopSpecificQuerySynthesizer(llm=generator_llm), 1.0),
]
for query, _ in distribution:
prompts = await query.adapt_prompts("spanish", llm=generator_llm)
query.set_prompts(**prompts)
생성
dataset = generator.generate_with_langchain_docs(
docs[:],
testset_size=5,
transforms=transforms,
query_distribution=distribution,
)
Applying HeadlineSplitter: 0%| | 0/6 [00:00<?, ?it/s]unable to apply transformation: 'headlines' property not found in this node
unable to apply transformation: 'headlines' property not found in this node
unable to apply transformation: 'headlines' property not found in this node
unable to apply transformation: 'headlines' property not found in this node
unable to apply transformation: 'headlines' property not found in this node
unable to apply transformation: 'headlines' property not found in this node
Generating Scenarios: 100%|██████████| 1/1 [00:07<00:00, 7.75s/it]
Generating Samples: 100%|██████████| 5/5 [00:03<00:00, 1.65it/s]
eval_dataset = dataset.to_evaluation_dataset()
print("Query:", eval_dataset[0].user_input)
print("Reference:", eval_dataset[0].reference)
Query: Quelles sont les caractéristiques du Bronx en tant que borough de New York?
Reference: Le Bronx est l'un des cinq arrondissements de New York, qui est la plus grande ville des États-Unis. Bien que le contexte ne fournisse pas de détails spécifiques sur le Bronx, il mentionne que New York est une ville cosmopolite avec de nombreux quartiers ethniques, ce qui pourrait inclure des caractéristiques culturelles variées présentes dans le Bronx.
이것으로 끝이에요. 이제 요구사항에 맞게 테스트 생성 과정을 커스터마이즈할 수 있어요.