Weaviate와 Cohere
Weaviate와 Cohere (통합 가이드)
Cohere를 Weaviate 데이터베이스와 통합하는 방법을 알아볼 거예요.
Weaviate는 객체와 벡터를 모두 저장하는 오픈소스 벡터 검색 엔진으로, 벡터 검색과 구조화된 필터링을 결합할 수 있게 해 줘요. 여기서는 Weaviate Cluster를 만들어 Cohere Embed로 데이터를 인덱싱하고, Rerank와 Command로 처리할 거예요.
다음 단계가 포함돼요.
- Weaviate 클러스터 만들기 (자세한 내용은 이 게시물 참조)
- 클러스터가 생성되면 클러스터 URL과 API 키를 받게 됩니다.
- 제공된 URL과 API 키를 사용해 Weaviate 클러스터에 연결합니다.
- Weaviate Python 클라이언트를 사용해 데이터를 저장할 컬렉션을 만듭니다.
출처: 문서
설정하기 (Getting Set up)
먼저 임포트, URL, pip 설치를 처리해 보겠어요.
PYTHON
from google.colab import userdata
weaviate_url = userdata.get("WEAVIATE_ENDPOINT")
weaviate_key = userdata.get("WEAVIATE_API_KEY")
cohere_key = userdata.get("COHERE_API_KEY")
PYTHON
!pip install -U weaviate-client -q
PYTHON
# Import the weaviate modules to interact with the Weaviate vector database
import weaviate
from weaviate.classes.init import Auth
# Define headers for the API requests, including the Cohere API key
headers = {
"X-Cohere-Api-Key": cohere_key,
}
# Connect to the Weaviate cloud instance
client = weaviate.connect_to_weaviate_cloud(
cluster_url=weaviate_url, # `weaviate_url`: your Weaviate URL
auth_credentials=Auth.api_key(
weaviate_key
), # `weaviate_key`: your Weaviate API key
headers=headers,
)
Embed
이제 Weaviate 데이터베이스에 "Healthcare_Compliance"라는 새 컬렉션을 만들 거예요.
PYTHON
from weaviate.classes.config import Configure
# This is where the "Healthcare_Compliance" collection is created in Weaviate.
client.collections.create(
"Healthcare_Compliance",
vectorizer_config=[
# Configure a named vectorizer using Cohere's model
Configure.NamedVectors.text2vec_cohere(
name="title_vector", # Name of the vectorizer
source_properties=[
"title"
], # Property to vectorize (in this case, the "title" field)
model="embed-english-v3.0", # Cohere model to use for vectorization
)
],
)
다음과 같은 출력이 보이게 돼요.
<weaviate.collections.collection.sync.Collection at 0x7f48a5604590>
다음으로 헬스케어 규정 준수 문서 목록을 정의하고, Weaviate 클라이언트에서 "Healthcare_Compliance" 컬렉션을 가져온 뒤, 동적 배치(dynamic batch) 프로세스를 사용해 여러 문서를 효율적으로 컬렉션에 추가할 거예요.
PYTHON
# Define the list of healthcare compliance documents
hl_compliance_docs = [
{
"title": "HIPAA Compliance Guide",
"description": "Comprehensive overview of HIPAA regulations, including patient privacy rules, data security standards, and breach notification requirements.",
},
{
"title": "FDA Drug Approval Process",
"description": "Detailed explanation of the FDA's drug approval process, covering clinical trials, safety reviews, and post-market surveillance.",
},
{
"title": "Telemedicine Regulations",
"description": "Analysis of state and federal regulations governing telemedicine practices, including licensing, reimbursement, and patient consent.",
},
{
"title": "Healthcare Data Security",
"description": "Best practices for securing healthcare data, including encryption, access controls, and incident response planning.",
},
{
"title": "Medicare and Medicaid Billing",
"description": "Guide to billing and reimbursement processes for Medicare and Medicaid, including coding, claims submission, and audit compliance.",
},
{
"title": "Patient Rights and Consent",
"description": "Overview of patient rights under federal and state laws, including informed consent, access to medical records, and end-of-life decisions.",
},
{
"title": "Healthcare Fraud and Abuse",
"description": "Explanation of laws and regulations related to healthcare fraud, including the False Claims Act, Anti-Kickback Statute, and Stark Law.",
},
{
"title": "Occupational Safety in Healthcare",
"description": "Guidelines for ensuring workplace safety in healthcare settings, including infection control, hazard communication, and emergency preparedness.",
},
{
"title": "Health Insurance Portability",
"description": "Discussion of COBRA and other laws ensuring continuity of health insurance coverage during job transitions or life events.",
},
{
"title": "Medical Device Regulations",
"description": "Overview of FDA regulations for medical devices, including classification, premarket approval, and post-market surveillance.",
},
{
"title": "Electronic Health Records (EHR) Standards",
"description": "Explanation of standards and regulations for EHR systems, including interoperability, data exchange, and patient privacy.",
},
{
"title": "Pharmacy Regulations",
"description": "Overview of state and federal regulations governing pharmacy practices, including prescription drug monitoring, compounding, and controlled substances.",
},
{
"title": "Mental Health Parity Act",
"description": "Analysis of the Mental Health Parity and Addiction Equity Act, ensuring equal coverage for mental health and substance use disorder treatment.",
},
{
"title": "Healthcare Quality Reporting",
"description": "Guide to quality reporting requirements for healthcare providers, including measures, submission processes, and performance benchmarks.",
},
{
"title": "Advance Directives and End-of-Life Care",
"description": "Overview of laws and regulations governing advance directives, living wills, and end-of-life care decisions.",
},
]
# Retrieve the "Healthcare_Compliance" collection from the Weaviate client
collection = client.collections.get("Healthcare_Compliance")
# Use a dynamic batch process to add multiple documents to the collection efficiently
with collection.batch.dynamic() as batch:
for src_obj in hl_compliance_docs:
# Add each document to the batch, specifying the "title" and "description" properties
batch.add_object(
properties={
"title": src_obj["title"],
"description": src_obj["description"],
},
)
이제 검색한 객체들을 순회하며 결과를 출력할 거예요.
PYTHON
# Import the MetadataQuery class from weaviate.classes.query to handle metadata in queries
from weaviate.classes.query import MetadataQuery
# Retrieve the "Healthcare_Compliance" collection from the Weaviate client
collection = client.collections.get("Healthcare_Compliance")
# Perform a near_text search for documents related to "policies related to drug compounding"
response = collection.query.near_text(
query="policies related to drug compounding", # Search query
limit=2, # Limit the number of results to 2
return_metadata=MetadataQuery(
distance=True
), # Include distance metadata in the results
)
# Iterate over the retrieved objects and print their details
for obj in response.objects:
title = obj.properties.get("title")
description = obj.properties.get("description")
distance = (
obj.metadata.distance
) # Get the distance metadata (A lower value for a distance means that two vectors are closer to one another than a higher value)
print(f"Title: {title}")
print(f"Description: {description}")
print(f"Distance: {distance}")
print("-" * 50)
출력은 대략 다음과 같아요 (참고: Distance 값이 낮을수록 두 벡터가 값이 높은 경우보다 서로 더 가깝다는 뜻이에요).
Title: Pharmacy Regulations
Description: Overview of state and federal regulations governing pharmacy practices, including prescription drug monitoring, compounding, and controlled substances.
Distance: 0.5904817581176758
--------------------------------------------------
Title: FDA Drug Approval Process
Description: Detailed explanation of the FDA's drug approval process, covering clinical trials, safety reviews, and post-market surveillance.
Distance: 0.6262975931167603
--------------------------------------------------
Embed + Rerank
이제 더 관련성 높은 결과를 표면에 드러내기 위해 Cohere Rerank를 추가할 거예요. 이를 위해 좀 더 설정이 필요해요.
PYTHON
# Import the weaviate module to interact with the Weaviate vector database
import weaviate
from weaviate.classes.init import Auth
# Define headers for the API requests, including the Cohere API key
headers = {
"X-Cohere-Api-Key": cohere_key,
}
# Connect to the Weaviate cloud instance
client = weaviate.connect_to_weaviate_cloud(
cluster_url=weaviate_url, # `weaviate_url`: your Weaviate URL
auth_credentials=Auth.api_key(
weaviate_key
), # `weaviate_key`: your Weaviate API key
headers=headers, # Include the Cohere API key in the headers
)
여기서는 Weaviate 데이터베이스에 "Legal_Docs" 컬렉션을 만들 거예요.
PYTHON
from weaviate.classes.config import Configure, Property, DataType
# Create a new collection named "Legal_Docs" in the Weaviate database
client.collections.create(
name="Legal_Docs",
properties=[
# Define a property named "title" with data type TEXT
Property(name="title", data_type=DataType.TEXT),
],
# Configure the vectorizer to use Cohere's text2vec model
vectorizer_config=Configure.Vectorizer.text2vec_cohere(
model="embed-english-v3.0" # Specify the Cohere model to use for vectorization
),
# Configure the reranker to use Cohere's rerank model
reranker_config=Configure.Reranker.cohere(
model="rerank-english-v3.0" # Specify the Cohere model to use for reranking
),
)
PYTHON
legal_documents = [
{
"title": "Contract Law Basics",
"description": "An in-depth introduction to contract law, covering essential elements such as offer, acceptance, consideration, and mutual assent. Explores types of contracts, including express, implied, and unilateral contracts, as well as remedies for breach of contract, such as damages, specific performance, and rescission.",
},
{
"title": "Intellectual Property Rights",
"description": "Comprehensive overview of intellectual property laws, including patents, trademarks, copyrights, and trade secrets. Discusses the process of obtaining patents, trademark registration, and copyright protection, as well as strategies for enforcing intellectual property rights and defending against infringement claims.",
},
{
"title": "Employment Law Guide",
"description": "Detailed guide to employment laws, covering hiring practices, termination procedures, anti-discrimination laws, and workplace safety regulations. Includes information on employee rights, such as minimum wage, overtime pay, and family and medical leave, as well as employer obligations under federal and state laws.",
},
{
"title": "Criminal Law Procedures",
"description": "Step-by-step explanation of criminal law procedures, from arrest and booking to trial and sentencing. Covers the rights of the accused, including the right to counsel, the right to remain silent, and the right to a fair trial, as well as rules of evidence and burden of proof in criminal cases.",
},
{
"title": "Real Estate Transactions",
"description": "Comprehensive guide to real estate transactions, including purchase agreements, title searches, property inspections, and closing processes. Discusses common issues such as title defects, financing contingencies, and property disclosures, as well as the role of real estate agents and attorneys in the transaction process.",
},
{
"title": "Corporate Governance",
"description": "In-depth overview of corporate governance principles, including the roles and responsibilities of boards of directors, shareholder rights, and compliance with securities laws. Explores best practices for board composition, executive compensation, and risk management, as well as strategies for maintaining transparency and accountability in corporate decision-making.",
},
{
"title": "Family Law Overview",
"description": "Comprehensive introduction to family law, covering marriage, divorce, child custody, child support, and adoption processes. Discusses the legal requirements for marriage and divorce, factors considered in child custody determinations, and the rights and obligations of adoptive parents under state and federal laws.",
},
{
"title": "Tax Law for Businesses",
"description": "Detailed guide to tax laws affecting businesses, including corporate income tax, payroll taxes, sales and use taxes, and tax deductions. Explores tax planning strategies, such as deferring income and accelerating expenses, as well as compliance requirements and penalties for non-compliance with tax laws.",
},
{
"title": "Immigration Law Basics",
"description": "Comprehensive overview of immigration laws, including visa categories, citizenship requirements, and deportation processes. Discusses the rights and obligations of immigrants, including access to public benefits and protection from discrimination, as well as the role of immigration attorneys in navigating the immigration system.",
},
{
"title": "Environmental Regulations",
"description": "In-depth overview of environmental laws and regulations, including air and water quality standards, hazardous waste management, and endangered species protection. Explores the role of federal and state agencies in enforcing environmental laws, as well as strategies for businesses to achieve compliance and minimize environmental impact.",
},
{
"title": "Consumer Protection Laws",
"description": "Comprehensive guide to consumer protection laws, including truth in advertising, product safety, and debt collection practices. Discusses the rights of consumers under federal and state laws, such as the right to sue for damages and the right to cancel certain contracts, as well as the role of government agencies in enforcing consumer protection laws.",
},
{
"title": "Estate Planning Essentials",
"description": "Detailed overview of estate planning, including wills, trusts, powers of attorney, and advance healthcare directives. Explores strategies for minimizing estate taxes, protecting assets from creditors, and ensuring that assets are distributed according to the individual's wishes after death.",
},
{
"title": "Bankruptcy Law Overview",
"description": "Comprehensive introduction to bankruptcy law, including Chapter 7 and Chapter 13 bankruptcy proceedings. Discusses the eligibility requirements for filing bankruptcy, the process of liquidating assets and discharging debts, and the impact of bankruptcy on credit scores and future financial opportunities.",
},
{
"title": "International Trade Law",
"description": "In-depth overview of international trade laws, including tariffs, quotas, and trade agreements. Explores the role of international organizations such as the World Trade Organization (WTO) in regulating global trade, as well as strategies for businesses to navigate trade barriers and comply with international trade regulations.",
},
{
"title": "Healthcare Law and Regulations",
"description": "Comprehensive guide to healthcare laws and regulations, including patient privacy rights, healthcare provider licensing, and medical malpractice liability. Discusses the impact of laws such as the Affordable Care Act (ACA) and the Health Insurance Portability and Accountability Act (HIPAA) on healthcare providers and patients, as well as strategies for ensuring compliance with healthcare regulations.",
},
]
PYTHON
# Retrieve the "Legal_Docs" collection from the Weaviate client
collection = client.collections.get("Legal_Docs")
# Use a dynamic batch process to add multiple documents to the collection efficiently
with collection.batch.dynamic() as batch:
for src_obj in legal_documents:
# Add each document to the batch, specifying the "title" and "description" properties
batch.add_object(
properties={
"title": src_obj["title"],
"description": src_obj["description"],
},
)
이제 검색 쿼리를 정의해야 해요.
PYTHON
search_query = "eligibility requirements for filing bankruptcy"
이 코드 조각은 weaviate.classes.query에서 MetadataQuery 클래스를 임포트해 쿼리에서 메타데이터를 처리하고, 검색된 객체들을 순회하며 세부 정보를 출력해요.
PYTHON
# Import the MetadataQuery class from weaviate.classes.query to handle metadata in queries
from weaviate.classes.query import MetadataQuery
# Retrieve the "Legal_Docs" collection from the Weaviate client
collection = client.collections.get("Legal_Docs")
# Perform a near_text semantic search for documents
response = collection.query.near_text(
query=search_query, # Search query
limit=3, # Limit the number of results to 3
return_metadata=MetadataQuery(distance=True) # Include distance metadata in the results
)
print("Semantic Search")
print("*" * 50)
# Iterate over the retrieved objects and print their details
for obj in response.objects:
title = obj.properties.get("title")
description = obj.properties.get("description")
metadata_distance = obj.metadata.distance
print(f"Title: {title}")
print(f"Description: {description}")
print(f"Metadata Distance: {metadata_distance}")
print("-" * 50)
출력은 대략 다음과 같아요.
Semantic Search
**************************************************
Title: Bankruptcy Law Overview
Description: Comprehensive introduction to bankruptcy law, including Chapter 7 and Chapter 13 bankruptcy proceedings. Discusses the eligibility requirements for filing bankruptcy, the process of liquidating assets and discharging debts, and the impact of bankruptcy on credit scores and future financial opportunities.
Metadata Distance: 0.41729819774627686
--------------------------------------------------
Title: Tax Law for Businesses
Description: Detailed guide to tax laws affecting businesses, including corporate income tax, payroll taxes, sales and use taxes, and tax deductions. Explores tax planning strategies, such as deferring income and accelerating expenses, as well as compliance requirements and penalties for non-compliance with tax laws.
Metadata Distance: 0.6903179883956909
--------------------------------------------------
Title: Consumer Protection Laws
Description: Comprehensive guide to consumer protection laws, including truth in advertising, product safety, and debt collection practices. Discusses the rights of consumers under federal and state laws, such as the right to sue for damages and the right to cancel certain contracts, as well as the role of government agencies in enforcing consumer protection laws.
Metadata Distance: 0.7075160145759583
--------------------------------------------------
이 코드는 Rerank 인프라를 설정해요.
PYTHON
# Import the Rerank class from weaviate.classes.query to enable reranking in queries
from weaviate.classes.query import Rerank
# Perform a near_text search with reranking for documents related to "property contracts and zoning regulations"
rerank_response = collection.query.near_text(
query=search_query,
limit=3,
rerank=Rerank(
prop="description", # Property to rerank based on (description in this case)
query=search_query, # Query to use for reranking
),
)
# Display the reranked search results
print("Reranked Search Results:")
for obj in rerank_response.objects:
title = obj.properties.get("title")
description = obj.properties.get("description")
rerank_score = getattr(
obj.metadata, "rerank_score", None
) # Get the rerank score metadata
print(f"Title: {title}")
print(f"Description: {description}")
print(f"Rerank Score: {rerank_score}")
print("-" * 50)
출력은 다음과 같아요.
Reranked Search Results:
Title: Bankruptcy Law Overview
Description: Comprehensive introduction to bankruptcy law, including Chapter 7 and Chapter 13 bankruptcy proceedings. Discusses the eligibility requirements for filing bankruptcy, the process of liquidating assets and discharging debts, and the impact of bankruptcy on credit scores and future financial opportunities.
Rerank Score: 0.8951567
--------------------------------------------------
Title: Tax Law for Businesses
Description: Detailed guide to tax laws affecting businesses, including corporate income tax, payroll taxes, sales and use taxes, and tax deductions. Explores tax planning strategies, such as deferring income and accelerating expenses, as well as compliance requirements and penalties for non-compliance with tax laws.
Rerank Score: 7.071895e-06
--------------------------------------------------
Title: Consumer Protection Laws
Description: Comprehensive guide to consumer protection laws, including truth in advertising, product safety, and debt collection practices. Discusses the rights of consumers under federal and state laws, such as the right to sue for damages and the right to cancel certain contracts, as well as the role of government agencies in enforcing consumer protection laws.
Rerank Score: 6.4895394e-06
--------------------------------------------------
재정렬 점수에 따르면 Bankruptcy Law Overview가 가장 관련성 높은 결과라는 것이 분명해요. 반면 나머지 두 문서(Tax Law for Businesses와 Consumer Protection Laws)는 점수가 훨씬 낮아서 쿼리와의 관련성이 낮음을 나타내요. 따라서 가장 관련성 높은 결과에만 집중하고 나머지 두 개는 건너뛰면 돼요.
Embed + Rerank + Command
마지막으로 Command를 섞어 넣을 거예요. 이는 임포트를 처리하고 Weaviate 데이터베이스에 새 "Legal_Docs"를 만듭니다.
PYTHON
from weaviate.classes.config import Configure
from weaviate.classes.generate import GenerativeConfig
# Create a new collection named "Legal_Docs" in the Weaviate database
client.collections.create(
name="Legal_Docs_RAG",
properties=[
# Define a property named "title" with data type TEXT
Property(name="title", data_type=DataType.TEXT),
],
# Configure the vectorizer to use Cohere's text2vec model
vectorizer_config=Configure.Vectorizer.text2vec_cohere(
model="embed-english-v3.0" # Specify the Cohere model to use for vectorization
),
# Configure the reranker to use Cohere's rerank model
reranker_config=Configure.Reranker.cohere(
model="rerank-english-v3.0" # Specify the Cohere model to use for reranking
),
# Configure the generative model to use Cohere's command r plus model
generative_config=Configure.Generative.cohere(
model="command-r-plus"
),
)
다음과 같은 출력이 보이게 돼요.
<weaviate.collections.collection.sync.Collection at 0x7f48afc06410>
이 코드는 Weaviate에서 "Legal_Docs_RAG"를 가져와요.
PYTHON
# Retrieve the "Legal_Docs_RAG" collection from the Weaviate client
collection = client.collections.get("Legal_Docs_RAG")
# Use a dynamic batch process to add multiple documents to the collection efficiently
with collection.batch.dynamic() as batch:
for src_obj in legal_documents:
# Add each document to the batch, specifying the "title" and "description" properties
batch.add_object(
properties={
"title": src_obj["title"],
"description": src_obj["description"],
},
)
앞서와 마찬가지로 객체를 순회하며 결과를 출력할 거예요.
PYTHON
from weaviate.classes.config import Configure
from weaviate.classes.generate import GenerativeConfig
# To generate text for each object in the search results, use the single prompt method.
# The example below generates outputs for each of the n search results, where n is specified by the limit parameter.
collection = client.collections.get("Legal_Docs_RAG")
response = collection.generate.near_text(
query=search_query,
limit=1,
single_prompt="Translate this into French - {title}: {description}",
)
for obj in response.objects:
print("Retrieved results")
print("-----------------")
print(obj.properties["title"])
print(obj.properties["description"])
print("Generated output")
print("-----------------")
print(obj.generated)
다음과 같은 출력이 보이게 돼요.
Retrieved results
-----------------
Bankruptcy Law Overview
Comprehensive introduction to bankruptcy law, including Chapter 7 and Chapter 13 bankruptcy proceedings. Discusses the eligibility requirements for filing bankruptcy, the process of liquidating assets and discharging debts, and the impact of bankruptcy on credit scores and future financial opportunities.
Generated output
-----------------
Voici une traduction possible :
Aperçu du droit des faillites : Introduction complète au droit des faillites, y compris les procédures de faillite en vertu des chapitres 7 et 13. Discute des conditions d'admissibilité pour déposer une demande de faillite, du processus de liquidation des actifs et de libération des dettes, ainsi que de l'impact de la faillite sur les cotes de crédit et les opportunités financières futures.
결론 (Conclusion)
이 통합 가이드는 Cohere의 강력한 AI 기능과 Weaviate의 벡터 데이터베이스를 효과적으로 결합해 정교한 검색 및 검색(retrieval) 시스템을 만드는 방법을 보여줬어요. 우리는 세 가지 핵심 접근 방식을 다뤘어요.
- 기본 벡터 검색: Cohere의 Embed 모델을 Weaviate와 함께 사용해 시맨틱 검색을 수행하면, 키워드가 아닌 의미를 기반으로 관련 문서를 찾는 자연어 쿼리가 가능해요.
- Rerank로 강화된 검색: Cohere의 Rerank 모델을 추가해 관련성에 따라 결과를 재정렬함으로써 검색 결과를 개선하고, 가장 관련성 높은 문서가 먼저 나타나도록 보장해요.
- 전체 RAG 파이프라인: 임베딩, 재정렬, Cohere의 Command 모델을 결합한 완전한 검색 증강 생성(RAG) 시스템을 구현해 관련 정보를 찾는 것뿐만 아니라 맥락에 맞는 응답을 생성해요.
이 통합은 이러한 기술들이 함께 협력해 더 지능적이고 정확한 검색 시스템을 만드는 방법을 보여줘요. 헬스케어 규정 준수 데이터베이스, 법률 문서 시스템, 기타 어떤 지식 베이스를 만들든, 이 조합은 시맨틱 검색과 AI 기반 콘텐츠 생성을 위한 강력한 기반을 제공해요.
이 통합의 유연성 덕분에 검색과 검색(retrieval) 작업에서 높은 성능과 정확도를 유지하면서 다양한 사용 사례에 적응시킬 수 있어요.