Dragonfly와 벡터 저장소
Dragonfly와 벡터 저장소 (Dragonfly and Vector Store)
이 노트북에서는 Dragonfly를 벡터 저장소로 활용하는 방법을 빠르게 실습해 볼게요. Dragonfly는 Redis API와 호환되는 초고속 인메모리 저장소라서, 익숙한 Redis 클라이언트 위에서 LlamaIndex 벡터 저장소를 그대로 사용할 수 있답니다. 콜랩 노트북이라면 먼저 LlamaIndex 🦙 설치가 필요해요.
출처: 문서
본문
이 노트북에서는 Dragonfly를 벡터 저장소로 사용하는 간단한 데모를 보여드리겠습니다.
콜랩 노트북에서 열고 있다면 LlamaIndex 설치가 필요할 거예요 🦙.
%pip install -U llama-index llama-index-vector-stores-redis llama-index-embeddings-cohere llama-index-embeddings-openai
import os
import getpass
import sys
import logging
import textwrap
import warnings
warnings.filterwarnings("ignore")
logging.basicConfig(stream=sys.stdout, level=logging.INFO)
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.vector_stores.redis import RedisVectorStore
Dragonfly 시작하기
Dragonfly를 시작하는 가장 쉬운 방법은 Dragonfly 도커 이미지를 사용하거나, Dragonfly Cloud 데모 인스턴스에 빠르게 가입하는 것입니다.
이 튜토리얼의 모든 단계를 따라 하려면 아래와 같이 이미지를 실행하세요.
터미널 창
docker run -d -p 6379:6379 --name dragonfly docker.dragonflydb.io/dragonflydb/dragonfly
OpenAI 설정
먼저 OpenAI API 키를 추가해 보겠습니다. 이 키는 임베딩과 ChatGPT 사용을 위해 OpenAI에 접근할 수 있게 해 줍니다.
oai_api_key = getpass.getpass("OpenAI API Key:")
os.environ["OPENAI_API_KEY"] = oai_api_key
데이터 다운로드
!mkdir -p 'data/paul_graham/'
!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham/paul_graham_essay.txt'
--2025-06-30 14:41:20-- https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.111.133, 185.199.108.133, 185.199.110.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.111.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 75042 (73K) [text/plain]
Saving to: ‘data/paul_graham/paul_graham_essay.txt’
data/paul_graham/pa 100%[===================>] 73.28K --.-KB/s in 0.04s
2025-06-30 14:41:20 (2.00 MB/s) - ‘data/paul_graham/paul_graham_essay.txt’ saved [75042/75042]
데이터셋 읽어오기
여기서는 Paul Graham의 에세이 모음을 사용해 텍스트를 임베딩으로 변환하고, 벡터 저장소에 저장한 뒤, LLM QnA 루프에서 사용할 컨텍스트를 찾아 쿼리해 볼 겁니다.
# load documents
documents = SimpleDirectoryReader("./data/paul_graham").load_data()
print(
"Document ID:",
documents[0].id_,
"Document Filename:",
documents[0].metadata["file_name"],
)
Document ID: a5cae17c-27eb-411e-8967-fb6ef98bcdcf Document Filename: paul_graham_essay.txt
기본 벡터 저장소 초기화
이제 문서가 준비되었으니, 기본(default) 설정으로 벡터 저장소를 초기화할 수 있습니다. 이렇게 하면 벡터를 Dragonfly에 저장하고 실시간 검색을 위한 인덱스를 만들 수 있습니다.
from llama_index.core import StorageContext
from redis import Redis
# create a client connection
redis_client = Redis.from_url("redis://localhost:6379")
# create the vector store wrapper
vector_store = RedisVectorStore(redis_client=redis_client, overwrite=True)
# load storage context
storage_context = StorageContext.from_defaults(vector_store=vector_store)
# build and load index from documents and storage context
index = VectorStoreIndex.from_documents(
documents, storage_context=storage_context
)
14:41:29 llama_index.vector_stores.redis.base INFO Using default RedisVectorStore schema.
14:41:31 httpx INFO HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
14:41:31 llama_index.vector_stores.redis.base INFO Added 22 documents to index llama_index
기본 벡터 저장소 쿼리
이제 데이터가 인덱스에 저장되었으니 인덱스에 질문을 던질 수 있습니다.
인덱스는 이 데이터를 LLM의 지식 기반으로 사용합니다. as_query_engine() 의 기본 설정은 OpenAI 임베딩과 GPT 언어 모델을 사용합니다. 따라서 커스텀 또는 로컬 언어 모델을 선택하지 않는 한 OpenAI 키가 필요합니다.
아래에서 인덱스에 대한 검색과 LLM을 이용한 전체 RAG를 테스트해 보겠습니다.
query_engine = index.as_query_engine()
retriever = index.as_retriever()
result_nodes = retriever.retrieve("What did the author learn?")
for node in result_nodes:
print(node)
14:41:40 httpx INFO HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
14:41:40 llama_index.vector_stores.redis.base INFO Querying index llama_index with query *=>[KNN 2 @vector $vector AS vector_distance] RETURN 5 id doc_id text _node_content vector_distance SORTBY vector_distance ASC DIALECT 2 LIMIT 0 2
14:41:40 llama_index.vector_stores.redis.base INFO Found 2 results for query with id ['llama_index/vector_f12d31cc-d154-4ae2-9511-81a1e0b2c185', 'llama_index/vector_a67c3af9-14cc-45fd-a2dd-142753a61d79']
Node ID: f12d31cc-d154-4ae2-9511-81a1e0b2c185
Text: What I Worked On February 2021 Before college the two main
things I worked on, outside of school, were writing and programming. I
didn't write essays. I wrote what beginning writers were supposed to
write then, and probably still are: short stories. My stories were
awful. They had hardly any plot, just characters with strong feelings,
which I ...
Score: 0.819
Node ID: a67c3af9-14cc-45fd-a2dd-142753a61d79
Text: In the summer of 2016 we moved to England. We wanted our kids to
see what it was like living in another country, and since I was a
British citizen by birth, that seemed the obvious choice. We only
meant to stay for a year, but we liked it so much that we still live
there. So most of Bel was written in England. In the fall of 2019,
Bel was final...
Score: 0.815
response = query_engine.query("What did the author learn?")
print(textwrap.fill(str(response), 100))
14:41:44 httpx INFO HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
14:41:44 llama_index.vector_stores.redis.base INFO Querying index llama_index with query *=>[KNN 2 @vector $vector AS vector_distance] RETURN 5 id doc_id text _node_content vector_distance SORTBY vector_distance ASC DIALECT 2 LIMIT 0 2
14:41:44 llama_index.vector_stores.redis.base INFO Found 2 results for query with id ['llama_index/vector_f12d31cc-d154-4ae2-9511-81a1e0b2c185', 'llama_index/vector_a67c3af9-14cc-45fd-a2dd-142753a61d79']
14:41:45 httpx INFO HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
The author learned that philosophy courses in college were boring to him, leading him to switch his
focus to studying AI.
result_nodes = retriever.retrieve("What was a hard moment for the author?")
for node in result_nodes:
print(node)
14:41:47 httpx INFO HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
14:41:47 llama_index.vector_stores.redis.base INFO Querying index llama_index with query *=>[KNN 2 @vector $vector AS vector_distance] RETURN 5 id doc_id text _node_content vector_distance SORTBY vector_distance ASC DIALECT 2 LIMIT 0 2
14:41:47 llama_index.vector_stores.redis.base INFO Found 2 results for query with id ['llama_index/vector_8c02f420-3cfc-4da6-859b-97469872ef46', 'llama_index/vector_f12d31cc-d154-4ae2-9511-81a1e0b2c185']
Node ID: 8c02f420-3cfc-4da6-859b-97469872ef46
Text: HN was no doubt good for YC, but it was also by far the biggest
source of stress for me. If all I'd had to do was select and help
founders, life would have been so easy. And that implies that HN was a
mistake. Surely the biggest source of stress in one's work should at
least be something close to the core of the work. Whereas I was like
someone ...
Score: 0.804
Node ID: f12d31cc-d154-4ae2-9511-81a1e0b2c185
Text: What I Worked On February 2021 Before college the two main
things I worked on, outside of school, were writing and programming. I
didn't write essays. I wrote what beginning writers were supposed to
write then, and probably still are: short stories. My stories were
awful. They had hardly any plot, just characters with strong feelings,
which I ...
Score: 0.802
response = query_engine.query("What was a hard moment for the author?")
print(textwrap.fill(str(response), 100))
14:41:51 httpx INFO HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
14:41:51 llama_index.vector_stores.redis.base INFO Querying index llama_index with query *=>[KNN 2 @vector $vector AS vector_distance] RETURN 5 id doc_id text _node_content vector_distance SORTBY vector_distance ASC DIALECT 2 LIMIT 0 2
14:41:51 llama_index.vector_stores.redis.base INFO Found 2 results for query with id ['llama_index/vector_8c02f420-3cfc-4da6-859b-97469872ef46', 'llama_index/vector_f12d31cc-d154-4ae2-9511-81a1e0b2c185']
14:41:52 httpx INFO HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
Dealing with urgent problems related to Hacker News (HN) was a significant source of stress for the
author.
index.vector_store.delete_index()
14:41:55 llama_index.vector_stores.redis.base INFO Deleting index llama_index
커스텀 인덱스 스키마 사용하기
대부분의 사용 사례에서는 기본 인덱스 설정과 사양을 커스터마이즈할 수 있어야 합니다. 예를 들어 활성화하고 싶은 특정 메타데이터 필터를 정의할 때 유용합니다.
Dragonfly에서는 인덱스 스키마 객체를 정의(파일 또는 dict에서)하고 이를 벡터 저장소 클라이언트 래퍼에 전달하기만 하면 됩니다.
이 예제에서는 다음과 같이 진행합니다.
- 임베딩 모델을 Cohere로 전환
- 문서에
updated_at타임스탬프라는 추가 메타데이터 필드 추가 - 기존
file_name메타데이터 필드 인덱싱
from llama_index.core.settings import Settings
from llama_index.embeddings.cohere import CohereEmbedding
# set up Cohere Key
co_api_key = getpass.getpass("Cohere API Key:")
Settings.embed_model = CohereEmbedding(api_key=co_api_key)
from redisvl.schema import IndexSchema
custom_schema = IndexSchema.from_dict(
{
# customize basic index specs
"index": {
"name": "paul_graham",
"prefix": "essay",
"key_separator": ":",
},
# customize fields that are indexed
"fields": [
# required fields for llamaindex
{"type": "tag", "name": "id"},
{"type": "tag", "name": "doc_id"},
{"type": "text", "name": "text"},
# custom metadata fields
{"type": "numeric", "name": "updated_at"},
{"type": "tag", "name": "file_name"},
# custom vector field definition for cohere embeddings
{
"type": "vector",
"name": "vector",
"attrs": {
"dims": 1024,
"algorithm": "hnsw",
"distance_metric": "cosine",
},
},
],
}
)
custom_schema.index
IndexInfo(name='paul_graham', prefix='essay', key_separator=':', storage_type=<StorageType.HASH: 'hash'>)
custom_schema.fields
{'id': TagField(name='id', type=<FieldTypes.TAG: 'tag'>, path=None, attrs=TagFieldAttributes(sortable=False, separator=',', case_sensitive=False, withsuffixtrie=False)),
'doc_id': TagField(name='doc_id', type=<FieldTypes.TAG: 'tag'>, path=None, attrs=TagFieldAttributes(sortable=False, separator=',', case_sensitive=False, withsuffixtrie=False)),
'text': TextField(name='text', type=<FieldTypes.TEXT: 'text'>, path=None, attrs=TextFieldAttributes(sortable=False, weight=1, no_stem=False, withsuffixtrie=False, phonetic_matcher=None)),
'updated_at': NumericField(name='updated_at', type=<FieldTypes.NUMERIC: 'numeric'>, path=None, attrs=NumericFieldAttributes(sortable=False)),
'file_name': TagField(name='file_name', type=<FieldTypes.TAG: 'tag'>, path=None, attrs=TagFieldAttributes(sortable=False, separator=',', case_sensitive=False, withsuffixtrie=False)),
'vector': HNSWVectorField(name='vector', type='vector', path=None, attrs=HNSWVectorFieldAttributes(dims=1024, algorithm=<VectorIndexAlgorithm.HNSW: 'HNSW'>, datatype=<VectorDataType.FLOAT32: 'FLOAT32'>, distance_metric=<VectorDistanceMetric.COSINE: 'COSINE'>, initial_cap=None, m=16, ef_construction=200, ef_runtime=10, epsilon=0.01))}
from datetime import datetime
def date_to_timestamp(date_string: str) -> int:
date_format: str = "%Y-%m-%d"
return int(datetime.strptime(date_string, date_format).timestamp())
# iterate through documents and add new field
for document in documents:
document.metadata["updated_at"] = date_to_timestamp(
document.metadata["last_modified_date"]
)
vector_store = RedisVectorStore(
schema=custom_schema, # provide customized schema
redis_client=redis_client,
overwrite=True,
)
storage_context = StorageContext.from_defaults(vector_store=vector_store)
# build and load index from documents and storage context
index = VectorStoreIndex.from_documents(
documents, storage_context=storage_context
)
14:42:26 httpx INFO HTTP Request: POST https://api.cohere.com/v2/embed "HTTP/1.1 200 OK"
14:42:26 httpx INFO HTTP Request: POST https://api.cohere.com/v2/embed "HTTP/1.1 200 OK"
14:42:27 httpx INFO HTTP Request: POST https://api.cohere.com/v2/embed "HTTP/1.1 200 OK"
14:42:27 llama_index.vector_stores.redis.base INFO Added 22 documents to index paul_graham
벡터 저장소를 쿼리하고 메타데이터로 필터링하기
이제 추가 메타데이터가 Dragonfly에 인덱싱되었으니, 필터를 사용한 몇 가지 쿼리를 시도해 보겠습니다.
from llama_index.core.vector_stores import (
MetadataFilters,
MetadataFilter,
ExactMatchFilter,
)
retriever = index.as_retriever(
similarity_top_k=3,
filters=MetadataFilters(
filters=[
ExactMatchFilter(key="file_name", value="paul_graham_essay.txt"),
MetadataFilter(
key="updated_at",
value=date_to_timestamp("2023-01-01"),
operator=">=",
),
MetadataFilter(
key="text",
value="learn",
operator="text_match",
),
],
condition="and",
),
)
result_nodes = retriever.retrieve("What did the author learn?")
for node in result_nodes:
print(node)
14:42:37 httpx INFO HTTP Request: POST https://api.cohere.com/v2/embed "HTTP/1.1 200 OK"
14:42:37 llama_index.vector_stores.redis.base INFO Querying index paul_graham with query ((@file_name:{paul_graham_essay\.txt} @updated_at:[1672524000 +inf]) @text:(learn))=>[KNN 3 @vector $vector AS vector_distance] RETURN 5 id doc_id text _node_content vector_distance SORTBY vector_distance ASC DIALECT 2 LIMIT 0 3
14:42:37 llama_index.vector_stores.redis.base INFO Found 3 results for query with id ['essay:30148f62-13c6-4edb-b09f-1cf3054c5c98', 'essay:054f9488-83c7-4bf6-a408-9ef17eea0446', 'essay:608adb71-a995-489d-81dc-0deab7bbe656']
Node ID: 30148f62-13c6-4edb-b09f-1cf3054c5c98
Text: If he even knew about the strange classes I was taking, he never
said anything. So now I was in a PhD program in computer science, yet
planning to be an artist, yet also genuinely in love with Lisp hacking
and working away at On Lisp. In other words, like many a grad student,
I was working energetically on multiple projects that were not my
the...
Score: 0.404
Node ID: 054f9488-83c7-4bf6-a408-9ef17eea0446
Text: I wanted to go back to RISD, but I was now broke and RISD was
very expensive, so I decided to get a job for a year and then return
to RISD the next fall. I got one at a company called Interleaf, which
made software for creating documents. You mean like Microsoft Word?
Exactly. That was how I learned that low end software tends to eat
high end so...
Score: 0.396
Node ID: 608adb71-a995-489d-81dc-0deab7bbe656
Text: All that seemed left for philosophy were edge cases that people
in other fields felt could safely be ignored. I couldn't have put
this into words when I was 18. All I knew at the time was that I kept
taking philosophy courses and they kept being boring. So I decided to
switch to AI. AI was in the air in the mid 1980s, but there were two
things...
Score: 0.394
문서 또는 인덱스를 완전히 삭제하기
때로는 문서나 전체 인덱스를 삭제하는 것이 유용할 수 있습니다. 이는 delete 와 delete_index 메서드를 사용해 수행할 수 있습니다.
document_id = documents[0].doc_id
document_id
print("Number of documents before deleting", redis_client.dbsize())
vector_store.delete(document_id)
print("Number of documents after deleting", redis_client.dbsize())
하지만 인덱스 자체는 여전히 존재합니다(연결된 문서는 없지만).
vector_store.index_exists()
# now lets delete the index entirely
# this will delete all the documents and the index
vector_store.delete_index()
print("Number of documents after deleting", redis_client.dbsize())