Nomic 임베딩

Nomic 임베딩

Nomic은 v1.5 🪆🪆🪆를 출시했는데, matryoshka 학습으로 64~768 사이의 다양한 임베딩 차원과 8192 컨텍스트를 지원해요. 이 노트북에서는 Nomic v1.5 임베딩을 여러 차원에서 사용하는 방법을 살펴볼게요.

출처: 문서

본문

설치 (Installation)

%pip install -U llama-index llama-index-embeddings-nomic

API 키 설정 (Setup API Keys)

nomic_api_key = "<NOMIC API KEY>"

import nest_asyncio


nest_asyncio.apply()


from llama_index.embeddings.nomic import NomicEmbedding

차원 128로 사용하기

embed_model = NomicEmbedding(
    api_key=nomic_api_key,
    dimensionality=128,
    model_name="nomic-embed-text-v1.5",
)


embedding = embed_model.get_text_embedding("Nomic Embeddings")

print(len(embedding))

128

embedding[:5]

[0.05569458, 0.057922363, -0.30126953, -0.09832764, 0.05947876]

차원 256으로 사용하기

embed_model = NomicEmbedding(
    api_key=nomic_api_key,
    dimensionality=256,
    model_name="nomic-embed-text-v1.5",
)


embedding = embed_model.get_text_embedding("Nomic Embeddings")

print(len(embedding))

256

embedding[:5]

[0.044708252, 0.04650879, -0.24182129, -0.07897949, 0.04776001]

차원 768로 사용하기

embed_model = NomicEmbedding(
    api_key=nomic_api_key,
    dimensionality=768,
    model_name="nomic-embed-text-v1.5",
)


embedding = embed_model.get_text_embedding("Nomic Embeddings")

print(len(embedding))

768

embedding[:5]

[0.027282715, 0.028381348, -0.14758301, -0.048187256, 0.029144287]

여전히 v1 Nomic 임베딩도 사용할 수 있어요

v1은 고정된 768 임베딩 차원을 가져요.

embed_model = NomicEmbedding(
    api_key=nomic_api_key, model_name="nomic-embed-text-v1"
)


embedding = embed_model.get_text_embedding("Nomic Embeddings")

print(len(embedding))

768

embedding[:5]

[0.0059013367, 0.03744507, 0.0035305023, -0.047180176, 0.0154418945]

Nomic v1.5 임베딩으로 End-to-End RAG 파이프라인 만들기

생성(Generation) 단계에는 OpenAI를 사용할게요.

임베딩 모델과 LLM 설정하기

from llama_index.core import settings
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.llms.openai import OpenAI


import os


os.environ["OPENAI_API_KEY"] = "<YOUR OPENAI API KEY>"


embed_model = NomicEmbedding(
    api_key=nomic_api_key,
    dimensionality=128,
    model_name="nomic-embed-text-v1.5",
)


llm = OpenAI(model="gpt-3.5-turbo")


settings.llm = llm
settings.embed_model = embed_model

데이터 다운로드

!mkdir -p 'data/paul_graham/'
!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham/paul_graham_essay.txt'

--2024-02-16 18:37:03--  https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 2606:50c0:8001::154, 2606:50c0:8003::154, 2606:50c0:8000::154, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|2606:50c0:8001::154|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 75042 (73K) [text/plain]
Saving to: 'data/paul_graham/paul_graham_essay.txt'


data/paul_graham/pa 100%[===================>]  73.28K  --.-KB/s    in 0.02s


2024-02-16 18:37:03 (3.87 MB/s) - 'data/paul_graham/paul_graham_essay.txt' saved [75042/75042]

데이터 로드

documents = SimpleDirectoryReader("./data/paul_graham").load_data()

인덱스 생성

index = VectorStoreIndex.from_documents(documents)

쿼리 엔진

query_engine = index.as_query_engine()

response = query_engine.query("what did author do growing up?")
print(response)

The author, growing up, worked on writing and programming. They wrote short stories and also tried writing programs on an IBM 1401 computer. Later, they got a microcomputer and started programming more extensively, writing simple games and a word processor.

더 알아보기 (Learn more)

• Nomic