Elasticsearch와 Cohere

Elasticsearch와 Cohere (통합 가이드)

Elasticsearch와 Cohere의 생성형 AI 기능으로 시맨틱 검색 파이프라인을 만드는 방법을 알아볼 거예요.

Elasticsearch에는 개발자가 생성형 AI로 차세대 검색 경험을 구축하는 데 필요한 모든 도구가 있고, Cohere를 inference API를 통해 네이티브로 통합할 수 있어요.

다음과 같은 목적으로 구축하고 싶다면 Elastic을 사용하세요.

  • 벡터 데이터베이스
  • 여러 개의 ML 모델 배포
  • 텍스트, 벡터, 하이브리드 검색 수행
  • 필터, facet, 집계(aggregation)로 검색
  • 문서 및 필드 수준 보안 적용
  • 온프레미스, 클라우드 또는 serverless(프리뷰)에서 실행

이 가이드는 위키백과 기사 데이터셋을 사용해 시맨틱 검색용 파이프라인을 구성해요. 다음 내용을 다룰 거예요.

  • Cohere 임베딩을 사용하는 Elastic inference 프로세서 만들기
  • 임베딩이 포함된 Elasticsearch 인덱스 만들기
  • Elasticsearch 인덱스에서 하이브리드 검색 수행 및 결과 재정렬(reranking)
  • 기본 RAG 수행

전체 코드 샘플을 보려면 이 노트북을 참조하세요. Elasticsearch 도큐사이트에서 통합 가이드도 확인할 수 있어요.

출처: 문서

사전 요구 사항 (Prerequisites)

이 튜토리얼에서는 다음이 있다고 가정해요.

참고: 이 튜토리얼은 Cohere를 Elastic Cloud serverless 프로젝트와 통합하지만, serverless에서 일반 language client로 전환하기만 하면 자체 관리(self-managed) Elasticsearch 배포 또는 Elastic Cloud 배포와도 통합할 수 있어요.

Elastic Serverless 배포 만들기

Elastic Cloud 배포가 없다면 가입해서 무료 트라이얼을 시작하고 Elastic Serverless에 접근을 요청하세요.

필요한 패키지 설치하기

필요한 Python 패키지를 설치하고 임포트하세요.

  • elasticsearch_serverless
  • cohere: 버전 5.2.5 이상인지 확인하세요

패키지를 설치하려면 다음 코드를 사용하세요.

PYTHON

!pip install elasticsearch_serverless==0.2.0.20231031
!pip install cohere==5.2.5

설치가 끝나면 Serverless 대시보드에서 엔드포인트 URL을 찾고 API 키를 만드세요.

필요한 패키지 임포트하기

다음으로 필요한 모듈을 임포트해요. 🔐 참고: getpass는 자격 증명을 터미널에 그대로 출력하거나 메모리에 저장하지 않으면서 안전하게 사용자에게 입력을 요청할 수 있게 해 줘요.

PYTHON

from elasticsearch_serverless import Elasticsearch, helpers
from getpass import getpass
import cohere
import json
import requests

Elasticsearch 클라이언트 만들기

이제 Python Elasticsearch 클라이언트를 인스턴스화할 수 있어요.

먼저 사용자에게 엔드포인트와 인코딩된 API 키를 입력받아요. 그런 다음 Elasticsearch 클래스의 인스턴스를 만드는 클라이언트 객체를 생성해요.

Elastic Serverless API 키를 만들 때 반드시 Control security privileges를 켜고, 클러스터 권한을 "cluster": ["all"]로 지정하도록 편집하세요.

PYTHON

ELASTICSEARCH_ENDPOINT = getpass("Elastic Endpoint: ")
ELASTIC_API_KEY = getpass(
    "Elastic encoded API key: "
)  # Use the encoded API key

client = Elasticsearch(
    ELASTICSEARCH_ENDPOINT, api_key=ELASTIC_API_KEY
)

# Confirm the client has connected
print(client.info())

Cohere와 Elasticsearch로 하이브리드 검색 인덱스 구축하기

Inference 엔드포인트 만들기

벡터 검색 인덱스를 구축할 때 가장 큰 고충 중 하나는 대규모 데이터 코퍼스에 대한 임베딩을 계산하는 일이에요. 다행히 Elastic은 대량 인덱싱(bulk indexing) 작업이 수행될 때 임베딩을 자동으로 계산하도록 인제스트 파이프라인에서 사용할 수 있는 inference 엔드포인트를 제공해요.

인제스트용 inference 파이프라인을 설정하려면 먼저 Cohere 임베딩을 사용하는 inference 엔드포인트를 만들어야 해요. 이를 위해 Cohere 계정의 API keys 섹션에서 찾을 수 있는 Cohere API 키가 필요해요.

embed-v4.0과 int8 또는 byte 압축을 사용해 저장 공간을 절약하는 inference 엔드포인트를 만들 거예요.

PYTHON

COHERE_API_KEY = getpass("Enter Cohere API key:  ")
# Delete the inference model if it already exists
client.options(ignore_status=[404]).inference.delete(
    inference_id="cohere_embeddings"
)

client.inference.put(
    task_type="text_embedding",
    inference_id="cohere_embeddings",
    body={
        "service": "cohere",
        "service_settings": {
            "api_key": COHERE_API_KEY,
            "model_id": "embed-v4.0",
            "embedding_type": "int8",
            "similarity": "cosine",
        },
        "task_settings": {},
    },
)

인덱스 만들기

대상 인덱스(모델이 입력 텍스트를 바탕으로 생성할 임베딩을 담을 인덱스)의 매핑을 만들어야 해요. 대상 인덱스는 Cohere 모델의 출력을 인덱싱하기 위해 semantic_text 필드 타입의 필드가 있어야 해요.

필요한 매핑으로 cohere-wiki-embeddings라는 이름의 인덱스를 만들어 보겠어요.

PYTHON

client.indices.delete(
    index="cohere-wiki-embeddings", ignore_unavailable=True
)
client.indices.create(
    index="cohere-wiki-embeddings",
    mappings={
        "properties": {
            "text_semantic": {
                "type": "semantic_text",
                "inference_id": "cohere_embeddings",
            },
            "text": {"type": "text", "copy_to": "text_semantic"},
            "wiki_id": {"type": "integer"},
            "url": {"type": "text"},
            "views": {"type": "float"},
            "langs": {"type": "integer"},
            "title": {"type": "text"},
            "paragraph_id": {"type": "integer"},
            "id": {"type": "integer"},
        }
    },
)

다음과 같은 응답을 볼 수 있을 거예요.

ObjectApiResponse({'acknowledged': True, 'shards_acknowledged': True, 'index': 'cohere-wiki-embeddings'})

그 API 호출에서 몇 가지 중요한 매개변수를 짚어볼게요.

  • semantic_text: inference 엔드포인트를 사용해 텍스트 콘텐츠에 대한 임베딩을 자동으로 생성하는 필드 타입이에요.
  • inference_id: 사용할 inference 엔드포인트의 ID를 지정해요. 이 예시에서 모델 ID는 cohere_embeddings로 설정됐어요.
  • copy_to: inference 결과를 담을 출력 필드를 지정해요.

문서 삽입하기

예시 위키 데이터셋을 삽입해 보겠어요. 이 단계를 완료하려면 프로덕션 Cohere 계정이 필요해요. 그렇지 않으면 API 요청 rate limit 때문에 문서 인제스트가 시간 초과될 수 있어요.

PYTHON

url = "https://raw.githubusercontent.com/cohere-ai/cohere-developer-experience/main/notebooks/data/embed_jobs_sample_data.jsonl"
response = requests.get(url)

# Load the response data into a JSON object
jsonl_data = response.content.decode("utf-8").splitlines()

# Prepare the documents to be indexed
documents = []
for line in jsonl_data:
    data_dict = json.loads(line)
    documents.append(
        {
            "_index": "cohere-wiki-embeddings",
            "_source": data_dict,
        }
    )

# Use the bulk endpoint to index
helpers.bulk(client, documents)

print("Done indexing documents into `cohere-wiki-embeddings` index!")

다음과 같은 출력이 보여야 해요.

Done indexing documents into `cohere-wiki-embeddings` index!

시맨틱 검색

데이터셋이 임베딩으로 풍부해졌으면, Elasticsearch가 제공하는 semantic 쿼리를 사용해 데이터를 쿼리할 수 있어요. Elasticsearch의 semantic_text는 시맨틱 검색을 크게 단순화해 줘요. Elasticsearch의 semantic text가 기술적 세부 사항 대신 모델과 결과에 집중할 수 있게 해 주는 방법을 자세히 알아보세요.

PYTHON

query = "When were the semi-finals of the 2022 FIFA world cup played?"

response = client.search(
    index="cohere-wiki-embeddings",
    size=100,
    query = {
        "semantic": {
                    "query": "When were the semi-finals of the 2022 FIFA world cup played?",
                     "field": "text_semantic"
        }
    }
)

raw_documents = response["hits"]["hits"]

# Display the first 10 results
for document in raw_documents[0:10]:
  print(f'Title: {document["_source"]["title"]}\nText: {document["_source"]["text"]}\n')

# Format the documents for ranking
documents = []
for hit in response["hits"]["hits"]:
    documents.append(hit["_source"]["text"])

결과는 대략 다음과 같아요.

Title: 2022 FIFA World Cup
Text: The 2022 FIFA World Cup was an international football tournament contested by the men's national teams of FIFA's member associations and 22nd edition of the FIFA World Cup. It took place in Qatar from 20 November to 18 December 2022, making it the first World Cup held in the Arab world and Muslim world, and the second held entirely in Asia after the 2002 tournament in South Korea and Japan. France were the defending champions, having defeated Croatia 4–2 in the 2018 final. At an estimated cost of over $220 billion, it is the most expensive World Cup ever held to date; this figure is disputed by Qatari officials, including organising CEO Nasser Al Khater, who said the true cost was $8 billion, and other figures related to overall infrastructure development since the World Cup was awarded to Qatar in 2010.

Title: 2022 FIFA World Cup
Text: The semi-finals were played on 13 and 14 December. Messi scored a penalty kick before Julián Álvarez scored twice to give Argentina a 3–0 victory over Croatia. Théo Hernandez scored after five minutes as France led Morocco for most of the game and later Randal Kolo Muani scored on 78 minutes to complete a 2–0 victory for France over Morocco as they reached a second consecutive final.

Title: 2022 FIFA World Cup
Text: The quarter-finals were played on 9 and 10 December. Croatia and Brazil ended 0–0 after 90 minutes and went to extra time. Neymar scored for Brazil in the 15th minute of extra time. Croatia, however, equalised through Bruno Petković in the second period of extra time. With the match tied, a penalty shootout decided the contest, with Croatia winning the shoot-out 4–2. In the second quarter-final match, Nahuel Molina and Messi scored for Argentina before Wout Weghorst equalised with two goals shortly before the end of the game. The match went to extra time and then penalties, where Argentina would go on to win 4–3. Morocco defeated Portugal 1–0, with Youssef En-Nesyri scoring at the end of the first half. Morocco became the first African and the first Arab nation to advance as far as the semi-finals of the competition. Despite Harry Kane scoring a penalty for England, it was not enough to beat France, who won 2–1 by virtue of goals from Aurélien Tchouaméni and Olivier Giroud, sending them to their second consecutive World Cup semi-final and becoming the first defending champions to reach this stage since Brazil in 1998.

Title: 2022 FIFA World Cup
Text: Unlike previous FIFA World Cups, which are typically played in June and July, because of Qatar's intense summer heat and often fairly high humidity, the 2022 World Cup was played in November and December. As a result, the World Cup was unusually staged in the middle of the seasons of domestic association football leagues, which started in late July or August, including all of the major European leagues, which had been obliged to incorporate extended breaks into their domestic schedules to accommodate the World Cup. Major European competitions had scheduled their respective competitions group matches to be played before the World Cup, to avoid playing group matches the following year.

Title: 2022 FIFA World Cup
Text: The match schedule was confirmed by FIFA in July 2020. The group stage was set to begin on 21 November, with four matches every day. Later, the schedule was tweaked by moving the Qatar vs Ecuador game to 20 November, after Qatar lobbied FIFA to allow their team to open the tournament. The final was played on 18 December 2022, National Day, at Lusail Stadium.

Title: 2022 FIFA World Cup
Text: Owing to the climate in Qatar, concerns were expressed over holding the World Cup in its traditional time frame of June and July. In October 2013, a task force was commissioned to consider alternative dates and report after the 2014 FIFA World Cup in Brazil. On 24 February 2015, the FIFA Task Force proposed that the tournament be played from late November to late December 2022, to avoid the summer heat between May and September and also avoid clashing with the 2022 Winter Olympics in February, the 2022 Winter Paralympics in March and Ramadan in April.

Title: 2022 FIFA World Cup
Text: Of the 32 nations qualified to play at the 2022 FIFA World Cup, 24 countries competed at the previous tournament in 2018. Qatar were the only team making their debut in the FIFA World Cup, becoming the first hosts to make their tournament debut since Italy in 1934. As a result, the 2022 tournament was the first World Cup in which none of the teams that earned a spot through qualification were making their debut. The Netherlands, Ecuador, Ghana, Cameroon, and the United States returned to the tournament after missing the 2018 tournament. Canada returned after 36 years, their only prior appearance being in 1986. Wales made their first appearance in 64 years – the longest ever gap for any team, their only previous participation having been in 1958.

Title: 2022 FIFA World Cup
Text: After UEFA were guaranteed to host the 2018 event, members of UEFA were no longer in contention to host in 2022. There were five bids remaining for the 2022 FIFA World Cup: Australia, Japan, Qatar, South Korea, and the United States.

Title: Cristiano Ronaldo
Text: Ronaldo was named in Portugal's squad for the 2022 FIFA World Cup in Qatar, making it his fifth World Cup. On 24 November, in Portugal's opening match against Ghana, Ronaldo scored a penalty kick and became the first male player to score in five different World Cups. In the last group game against South Korea, Ronaldo received criticism from his own coach for his reaction at being substituted. He was dropped from the starting line-up for Portugal's last 16 match against Switzerland, marking the first time since Euro 2008 that he had not started a game for Portugal in a major international tournament, and the first time Portugal had started a knockout game without Ronaldo in the starting line-up at an international tournament since Euro 2000. He came off the bench late on as Portugal won 6–1, their highest tally in a World Cup knockout game since the 1966 World Cup, with Ronaldo's replacement Gonçalo Ramos scoring a hat-trick. Portugal employed the same strategy in the quarter-finals against Morocco, with Ronaldo once again coming off the bench; in the process, he equalled Bader Al-Mutawa's international appearance record, becoming the joint–most capped male footballer of all time, with 196 caps. Portugal lost 1–0, however, with Morocco becoming the first CAF nation ever to reach the World Cup semi-finals.

Title: 2022 FIFA World Cup
Text: The final draw was held at the Doha Exhibition and Convention Center in Doha, Qatar, on 1 April 2022, 19:00 AST, prior to the completion of qualification. The two winners of the inter-confederation play-offs and the winner of the Path A of the UEFA play-offs were not known at the time of the draw. The draw was attended by 2,000 guests and was led by Carli Lloyd, Jermaine Jenas and sports broadcaster Samantha Johnson, assisted by the likes of Cafu (Brazil), Lothar Matthäus (Germany), Adel Ahmed Malalla (Qatar), Ali Daei (Iran), Bora Milutinović (Serbia/Mexico), Jay-Jay Okocha (Nigeria), Rabah Madjer (Algeria), and Tim Cahill (Australia).

하이브리드 검색

데이터셋이 임베딩으로 풍부해졌으면, 하이브리드 검색을 사용해 데이터를 쿼리할 수 있어요.

시맨틱 쿼리를 전달하고, 쿼리 텍스트와 임베딩을 만드는 데 사용한 모델을 제공하세요.

PYTHON

query = "When were the semi-finals of the 2022 FIFA world cup played?"

response = client.search(
    index="cohere-wiki-embeddings",
    size=100,
    query={
        "bool": {
            "must": {
                "multi_match": {
                "query": "When were the semi-finals of the 2022 FIFA world cup played?",
                "fields": ["text", "title"]
        }
            },
            "should": {
                "semantic": {
                    "query": "When were the semi-finals of the 2022 FIFA world cup played?",
                     "field": "text_semantic"
                }
            },
        }
    }

)

raw_documents = response["hits"]["hits"]

# Display the first 10 results
for document in raw_documents[0:10]:
  print(f'Title: {document["_source"]["title"]}\nText: {document["_source"]["text"]}\n')

# Format the documents for ranking
documents = []
for hit in response["hits"]["hits"]:
    documents.append(hit["_source"]["text"])

재정렬 (Ranking)

벡터 검색과 BM25 검색의 결과를 효과적으로 결합하기 위해, inference API를 통한 Cohere의 Rerank 3 모델을 사용해 결과에 더 정밀한 의미적 재정렬을 제공할 수 있어요.

먼저 Cohere API 키로 inference 엔드포인트를 만들어요. 엔드포인트 이름과 rerank 모델 중 하나의 model_id를 지정해야 해요. 이 예시에서는 Rerank 3을 사용할 거예요.

PYTHON

# Delete the inference model if it already exists
client.options(ignore_status=[404]).inference.delete(inference_id="cohere_rerank")

client.inference.put(
    task_type="rerank",
    inference_id="cohere_rerank",
    body={
        "service": "cohere",
        "service_settings":{
            "api_key": COHERE_API_KEY,
            "model_id": "rerank-english-v3.0"
           },
        "task_settings": {
            "top_n": 10,
        },
    }
)

이제 그 inference 엔드포인트를 사용해 결과를 재정렬할 수 있어요. 여기서는 검색에 사용한 쿼리와 하이브리드 검색으로 가져온 문서를 전달할 거예요.

inference 서비스는 관련성 내림차순으로 문서 목록을 응답해요. 각 문서는 인덱스(문서가 inference 엔드포인트에 전송된 순서를 반영)를 가지며, "return_documents" 작업 설정이 True라면 문서 텍스트도 함께 포함돼요.

이 경우에는 response를 False로 설정하고, 응답에 포함된 인덱스를 기반으로 입력 문서를 재구성할 거예요.

PYTHON

response = client.inference.inference(
    inference_id="cohere_rerank",
    body={
        "query": query,
        "input": documents,
        "task_settings": {
            "return_documents": False
            }
        }
)

# Reconstruct the input documents based on the index provided in the rereank response
ranked_documents = []
for document in response.body["rerank"]:
  ranked_documents.append({
      "title": raw_documents[int(document["index"])]["_source"]["title"],
      "text": raw_documents[int(document["index"])]["_source"]["text"]
  })

# Print the top 10 results
for document in ranked_documents[0:10]:
  print(f"Title: {document['title']}\nText: {document['text']}\n")

검색 증강 생성 (Retrieval augmented generation)

결과를 재정렬했으니, 이제 Cohere의 Chat API로 쉽게 RAG 시스템으로 만들 수 있어요. 검색된 문서와 쿼리를 전달하고, Cohere의 최신 생성 모델인 Command R+를 사용해 근거 있는(grounded) 응답을 확인해 보세요.

먼저 Cohere 클라이언트를 만들 거예요.

PYTHON

co = cohere.Client(COHERE_API_KEY)

다음으로 Cohere Chat API에서 인용(citations)이 포함된 근거 있는 생성 결과를 쉽게 얻을 수 있어요. 사용자 쿼리와 Elastic에서 검색한 문서를 API에 전달하고, 근거 있는 응답을 출력하기만 하면 돼요.

PYTHON

response = co.chat(
    message=query,
    documents=ranked_documents,
    model="command-a-03-2025",
)

source_documents = []
for citation in response.citations:
    for document_id in citation.document_ids:
        if document_id not in source_documents:
            source_documents.append(document_id)

print(f"Query: {query}")
print(f"Response: {response.text}")
print("Sources:")
for document in response.documents:
    if document["id"] in source_documents:
        print(f"{document['title']}: {document['text']}")

그게 전부예요! Cohere와 Elastic으로 하이브리드 검색과 RAG를 빠르고 쉽게 구현했습니다.

더 알아보기 (Learn more)