Firestore 벡터 저장소

Firestore 벡터 저장소 (Firestore Vector Store)

이번엔 Google Cloud의 서버리스 문서 데이터베이스인 Firestore를 LlamaIndex의 벡터 저장소로 활용하는 방법을 배워볼게요. FirestoreVectorStore 클래스를 사용해 벡터를 저장하고 유사도 검색까지 할 수 있답니다. 시작 전에 Google Cloud 프로젝트와 Firestore API, 데이터베이스가 준비되어 있어야 해요.

출처: 문서

본문

Google Firestore (Native Mode)

Firestore 는 어떤 요구에도 확장되는 서버리스 문서 지향 데이터베이스입니다. Firestore의 Langchain 통합을 활용해 AI 기반 경험을 구축하도록 데이터베이스 애플리케이션을 확장해 보세요.

이 노트북에서는 Firestore를 사용해 벡터를 저장하고 FirestoreVectorStore 클래스로 쿼리하는 방법을 다룹니다.

시작하기 전에 (Before You Begin)

이 노트북을 실행하려면 다음을 수행해야 합니다.

이 노트북의 실행 환경에서 데이터베이스 접근이 확인된 후, 아래 값을 채우고 예제 스크립트를 실행하기 전에 해당 셀을 실행하세요.

라이브러리 설치 (Library Installation)

콜랩 노트북에서 열고 있다면 LlamaIndex 설치가 필요할 거예요 🦙. 이 노트북에서는 Google Generative AI 임베딩을 사용하기 위해 langchain-google-genai 도 함께 설치할 겁니다.

%pip install --quiet llama-index
%pip install --quiet llama-index-vector-stores-firestore llama-index-embeddings-huggingface

☁ Google Cloud 프로젝트 설정

이 노트북 내에서 Google Cloud 리소스를 활용할 수 있도록 Google Cloud 프로젝트를 설정합니다.

프로젝트 ID를 모른다면 다음을 시도해 보세요.

# @markdown Please fill in the value below with your Google Cloud project ID and then run the cell.


PROJECT_ID = "YOUR_PROJECT_ID"  # @param {type:"string"}


# Set the project id
!gcloud config set project {PROJECT_ID}

🔐 인증 (Authentication)

이 노트북에 로그인된 IAM 사용자로 Google Cloud에 인증해 Google Cloud 프로젝트에 접근합니다.

  • Colab으로 이 노트북을 실행하는 경우 아래 셀을 사용하고 계속 진행하세요.
  • Vertex AI Workbench를 사용하는 경우 설정 지침은 여기를 참고하세요.
from google.colab import auth


auth.authenticate_user()

기본 사용법 (Basic Usage)

FirestoreVectorStore 초기화 (Initialize FirestoreVectorStore)

FirestoreVectorStore 를 사용하면 Firestore에 데이터를 로드하고 쿼리할 수 있습니다.

# @markdown Please specify a source for demo purpose.
COLLECTION_NAME = "test_collection"

from llama_index.core import SimpleDirectoryReader


# Load documents and build index
documents = SimpleDirectoryReader(
    "../../examples/data/paul_graham"
).load_data()

from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from llama_index.core import Settings


# Set the embedding model, this is a local model
embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-small-en-v1.5")

from llama_index.core import VectorStoreIndex
from llama_index.core import StorageContext, ServiceContext


from llama_index.vector_stores.firestore import FirestoreVectorStore


# Create a Firestore vector store
store = FirestoreVectorStore(collection_name=COLLECTION_NAME)


storage_context = StorageContext.from_defaults(vector_store=store)
service_context = ServiceContext.from_defaults(
    llm=None, embed_model=embed_model
)


index = VectorStoreIndex.from_documents(
    documents, storage_context=storage_context, service_context=service_context
)

/var/folders/mh/cqn7wzgs3j79rbg243_gfcx80000gn/T/ipykernel_29666/1668628626.py:10: DeprecationWarning: Call to deprecated class method from_defaults. (ServiceContext is deprecated, please use `llama_index.settings.Settings` instead.) -- Deprecated since version 0.10.0.
  service_context = ServiceContext.from_defaults(llm=None, embed_model=embed_model)




LLM is explicitly disabled. Using MockLLM.

FirestoreVectorStore 를 사용해 저장된 벡터에 대해 유사도 검색을 수행할 수 있습니다. 이는 유사한 문서나 텍스트를 찾는 데 유용합니다.

query_engine = index.as_query_engine()
res = query_engine.query("What did the author do growing up?")
print(str(res.source_nodes[0].text))

None
What I Worked On


February 2021


Before college the two main things I worked on, outside of school, were writing and programming. I didn't write essays. I wrote what beginning writers were supposed to write then, and probably still are: short stories. My stories were awful. They had hardly any plot, just characters with strong feelings, which I imagined made them deep.


The first programs I tried writing were on the IBM 1401 that our school district used for what was then called "data processing." This was in 9th grade, so I was 13 or 14. The school district's 1401 happened to be in the basement of our junior high school, and my friend Rich Draves and I got permission to use it. It was like a mini Bond villain's lair down there, with all these alien-looking machines — CPU, disk drives, printer, card reader — sitting up on a raised floor under bright fluorescent lights.


The language we used was an early version of Fortran. You had to type programs on punch cards, then stack them in the card reader and press a button to load the program into memory and run it. The result would ordinarily be to print something on the spectacularly loud printer.


I was puzzled by the 1401. I couldn't figure out what to do with it. And in retrospect there's not much I could have done with it. The only form of input to programs was data stored on punched cards, and I didn't have any data stored on punched cards. The only other option was to do things that didn't rely on any input, like calculate approximations of pi, but I didn't know enough math to do anything interesting of that type. So I'm not surprised I can't remember any programs I wrote, because they can't have done much. My clearest memory is of the moment I learned it was possible for programs not to terminate, when one of mine didn't. On a machine without time-sharing, this was a social as well as a technical error, as the data center manager's expression made clear.


With microcomputers, everything changed. Now you could have a computer sitting right in front of you, on a desk, that could respond to your keystrokes as it was running instead of just churning through a stack of punch cards and then stopping. [1]


The first of my friends to get a microcomputer built it himself. It was sold as a kit by Heathkit. I remember vividly how impressed and envious I felt watching him sitting in front of it, typing programs right into the computer.


Computers were expensive in those days and it took me years of nagging before I convinced my father to buy one, a TRS-80, in about 1980. The gold standard then was the Apple II, but a TRS-80 was good enough. This was when I really started programming. I wrote simple games, a program to predict how high my model rockets would fly, and a word processor that my father used to write at least one book. There was only room in memory for about 2 pages of text, so he'd write 2 pages at a time and then print them out, but it was a lot better than a typewriter.


Though I liked programming, I didn't plan to study it in college. In college I was going to study philosophy, which sounded much more powerful. It seemed, to my naive high school self, to be the study of the ultimate truths, compared to which the things studied in other fields would be mere domain knowledge. What I discovered when I got to college was that the other fields took up so much of the space of ideas that there wasn't much left for these supposed ultimate truths. All that seemed left for philosophy were edge cases that people in other fields felt could safely be ignored.


I couldn't have put this into words when I was 18. All I knew at the time was that I kept taking philosophy courses and they kept being boring. So I decided to switch to AI.


AI was in the air in the mid 1980s, but there were two things especially that made me want to work on it: a novel by Heinlein called The Moon is a Harsh Mistress, which featured an intelligent computer called Mike, and a PBS documentary that showed Terry Winograd using SHRDLU. I haven't tried rereading The Moon is a Harsh Mistress, so I don't know how well it has aged, but when I read it I was drawn entirely into its world.

filters 인자를 지정하면 검색 결과에 사전 필터링(pre-filtering)을 적용할 수 있습니다.

from llama_index.core.vector_stores.types import (
    MetadataFilters,
    ExactMatchFilter,
    MetadataFilter,
)


filters = MetadataFilters(
    filters=[MetadataFilter(key="author", value="Paul Graham")]
)
query_engine = index.as_query_engine(filters=filters)
res = query_engine.query("What did the author do growing up?")
print(str(res.source_nodes[0].text))

더 알아보기 (Learn more)