문서에서 커스텀 단일 홉(single-hop) 쿼리 만들기
문서에서 커스텀 단일 홉(single-hop) 쿼리 만들기
표준 쿼리 유형으로는 만들기 어려운, 자신의 문서에 특화된 질문을 생성하고 싶을 때가 있어요. 이 튜토리얼에서는 SingleHopQuerySynthesizer 를 상속받아 시나리오 생성 부분을 직접 바꿔서 커스텀 단일 홉 쿼리를 만드는 방법을 배워요.
출처: 문서
본문
샘플 문서 로드
GitLab handbook의 샘플 문서를 사용하고 있어요. 아래 명령으로 다운로드할 수 있어요.
! git clone https://huggingface.co/datasets/vibrantlabsai/Sample_Docs_Markdown
from langchain_community.document_loaders import DirectoryLoader
path = "Sample_Docs_Markdown/"
loader = DirectoryLoader(path, glob="**/*.md")
docs = loader.load()
KG 생성
문서로 기본 지식 그래프(knowledge graph)를 만들어요.
from ragas.testset.graph import KnowledgeGraph
from ragas.testset.graph import Node, NodeType
kg = KnowledgeGraph()
for doc in docs:
kg.nodes.append(
Node(
type=NodeType.DOCUMENT,
properties={
"page_content": doc.page_content,
"document_metadata": doc.metadata,
},
)
)
LLM과 Embedding 모델 설정
원하는 모델을 써도 되는데, 여기서는 open-ai 모델을 사용할게요.
from openai import OpenAI
from ragas.llms import llm_factory
from ragas.embeddings import OpenAIEmbeddings
openai_client = OpenAI()
llm = llm_factory("gpt-4o-mini", client=openai_client)
embedding = OpenAIEmbeddings(client=openai_client)
변환(transforms) 설정
여기서는 추출기(extractor) 2개와 관계 빌더 1개를 사용해요.
- Headline extractor: 문서에서 헤드라인을 추출해요
- Keyphrase extractor: 문서에서 핵심어(keyphrase)를 추출해요
- Headline splitter: 헤드라인을 기준으로 문서를 노드로 나눠요
from ragas.testset.transforms import apply_transforms
from ragas.testset.transforms import (
HeadlinesExtractor,
HeadlineSplitter,
KeyphrasesExtractor,
)
headline_extractor = HeadlinesExtractor(llm=llm)
headline_splitter = HeadlineSplitter(min_tokens=300, max_tokens=1000)
keyphrase_extractor = KeyphrasesExtractor(
llm=llm, property_name="keyphrases", max_num=10
)
transforms = [
headline_extractor,
headline_splitter,
keyphrase_extractor,
]
apply_transforms(kg, transforms=transforms)
Output
Applying KeyphrasesExtractor: 6%| | 2/36 [00:01<00:20, 1Property 'keyphrases' already exists in node '514fdc'. Skipping!
Applying KeyphrasesExtractor: 11%| | 4/36 [00:01<00:10, 2Property 'keyphrases' already exists in node '84a0f6'. Skipping!
Applying KeyphrasesExtractor: 64%|▋| 23/36 [00:03<00:01, Property 'keyphrases' already exists in node '93f19d'. Skipping!
Applying KeyphrasesExtractor: 72%|▋| 26/36 [00:04<00:00, 1Property 'keyphrases' already exists in node 'a126bf'. Skipping!
Applying KeyphrasesExtractor: 81%|▊| 29/36 [00:04<00:00, Property 'keyphrases' already exists in node 'c230df'. Skipping!
Applying KeyphrasesExtractor: 89%|▉| 32/36 [00:04<00:00, 1Property 'keyphrases' already exists in node '4f2765'. Skipping!
Property 'keyphrases' already exists in node '4a4777'. Skipping!
Persona 구성
자동 persona 생성기를 사용해 이 단계를 자동화할 수도 있어요.
from ragas.testset.persona import Persona
person1 = Persona(
name="gitlab employee",
role_description="A junior gitlab employee curious on workings on gitlab",
)
persona2 = Persona(
name="Hiring manager at gitlab",
role_description="A hiring manager at gitlab trying to underestand hiring policies in gitlab",
)
persona_list = [person1, persona2]
SingleHop 쿼리
SingleHopQuerySynthesizer 를 상속받아 쿼리 생성을 위한 시나리오를 만드는 함수를 수정해요.
단계:
- 쿼리 생성에 적합한 노드 집합을 찾아요. 여기서는 keyphrases가 추출된 모든 노드를 선택해요.
- 각 적합 집합에 대해
- keyphrase를 하나 이상의 persona와 매칭해요
- (Node, Persona, Query Style, Query Length)의 모든 가능한 조합을 만들어요
- 조합에서 필요한 수만큼 쿼리를 샘플링해요
from ragas.testset.synthesizers.single_hop import (
SingleHopQuerySynthesizer,
SingleHopScenario,
)
from dataclasses import dataclass
from ragas.testset.synthesizers.prompts import (
ThemesPersonasInput,
ThemesPersonasMatchingPrompt,
)
@dataclass
class MySingleHopScenario(SingleHopQuerySynthesizer):
theme_persona_matching_prompt = ThemesPersonasMatchingPrompt()
async def _generate_scenarios(self, n, knowledge_graph, persona_list, callbacks):
property_name = "keyphrases"
nodes = []
for node in knowledge_graph.nodes:
if node.type.name == "CHUNK" and node.get_property(property_name):
nodes.append(node)
number_of_samples_per_node = max(1, n // len(nodes))
scenarios = []
for node in nodes:
if len(scenarios) >= n:
break
themes = node.properties.get(property_name, [""])
prompt_input = ThemesPersonasInput(themes=themes, personas=persona_list)
persona_concepts = await self.theme_persona_matching_prompt.generate(
data=prompt_input, llm=self.llm, callbacks=callbacks
)
base_scenarios = self.prepare_combinations(
node,
themes,
personas=persona_list,
persona_concepts=persona_concepts.mapping,
)
scenarios.extend(
self.sample_combinations(base_scenarios, number_of_samples_per_node)
)
return scenarios
query = MySingleHopScenario(llm=llm)
scenarios = await query.generate_scenarios(
n=5, knowledge_graph=kg, persona_list=persona_list
)
scenarios[0]
SingleHopScenario(
nodes=1
term=what is an ally
persona=name='Hiring manager at gitlab' role_description='A hiring manager at gitlab trying to underestand hiring policies in gitlab'
style=Web search like queries
length=long)
result = await query.generate_sample(scenario=scenarios[-1])
쿼리 스타일을 커스터마이즈하는 프롬프트 수정
여기서는 기본 프롬프트를 Yes/No 질문만 생성하라는 지시문으로 교체해요. 이 단계는 선택적이에요.
instruction = """Generate a Yes/No query and answer based on the specified conditions (persona, term, style, length)
and the provided context. Ensure the answer is entirely faithful to the context, using only the information
directly from the provided context.
### Instructions:
1. **Generate a Yes/No Query**: Based on the context, persona, term, style, and length, create a question
that aligns with the persona's perspective, incorporates the term, and can be answered with 'Yes' or 'No'.
2. **Generate an Answer**: Using only the content from the provided context, provide a 'Yes' or 'No' answer
to the query. Do not add any information not included in or inferable from the context."""
prompt = query.get_prompts()["generate_query_reference_prompt"]
prompt.instruction = instruction
query.set_prompts(**{"generate_query_reference_prompt": prompt})
result = await query.generate_sample(scenario=scenarios[-1])
result.user_input
'Does the Diversity, Inclusion & Belonging (DIB) Team at GitLab have a structured approach to encourage collaborations among team members through various communication methods?'
result.reference
'Yes'