Faiss 벡터 저장소
Faiss 벡터 저장소 (Faiss Vector Store)
이번엔 Facebook AI가 만든 효율적인 유사도 검색 라이브러리 Faiss를 LlamaIndex의 벡터 저장소로 활용하는 방법을 배워볼게요. 먼저 Faiss 인덱스를 만들고, 문서를 로드해 벡터 저장소 인덱스를 구축한 뒤, 디스크에 저장/로드하고 쿼리하는 전체 흐름을 다룬답니다. 콜랩 노트북이라면 LlamaIndex 🦙 설치가 필요해요.
출처: 문서
본문
콜랩 노트북에서 열고 있다면 LlamaIndex 설치가 필요할 거예요 🦙.
%pip install llama-index-vector-stores-faiss
!pip install llama-index
Faiss 인덱스 만들기 (Creating a Faiss Index)
import logging
import sys
logging.basicConfig(stream=sys.stdout, level=logging.INFO)
logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))
import faiss
# dimensions of text-ada-embedding-002
d = 1536
faiss_index = faiss.IndexFlatL2(d)
문서 로드, VectorStoreIndex 구축 (Load documents, build the VectorStoreIndex)
from llama_index.core import (
SimpleDirectoryReader,
load_index_from_storage,
VectorStoreIndex,
StorageContext,
)
from llama_index.vector_stores.faiss import FaissVectorStore
from IPython.display import Markdown, display
데이터 다운로드
!mkdir -p 'data/paul_graham/'
!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham/paul_graham_essay.txt'
# load documents
documents = SimpleDirectoryReader("./data/paul_graham/").load_data()
vector_store = FaissVectorStore(faiss_index=faiss_index)
storage_context = StorageContext.from_defaults(vector_store=vector_store)
index = VectorStoreIndex.from_documents(
documents, storage_context=storage_context
)
# save index to disk
index.storage_context.persist()
# load index from disk
vector_store = FaissVectorStore.from_persist_dir("./storage")
storage_context = StorageContext.from_defaults(
vector_store=vector_store, persist_dir="./storage"
)
index = load_index_from_storage(storage_context=storage_context)
인덱스 쿼리 (Query Index)
# set Logging to DEBUG for more detailed outputs
query_engine = index.as_query_engine()
response = query_engine.query("What did the author do growing up?")
display(Markdown(f"<b>{response}</b>"))
# set Logging to DEBUG for more detailed outputs
query_engine = index.as_query_engine()
response = query_engine.query(
"What did the author do after his time at Y Combinator?"
)
display(Markdown(f"<b>{response}</b>"))