Qdrant Cloud 추론을 사용한 하이브리드 검색

Qdrant Cloud 추론을 사용한 하이브리드 검색 (tutorials-basics-cloud-inference-hybrid-search)

시간: 30분	난이도: 중급
클라우드 추론으로 하이브리드 의미 검색 엔진 만들기

이 튜토리얼에서는 Qdrant Cloud의 내장 추론(inference) 기능을 이용해 하이브리드 의미 검색 엔진을 만드는 과정을 살펴볼게요. 여기서 배울 내용은 다음과 같아요.

• 로컬 모델을 실행하지 않고도 클라우드 추론으로 데이터를 자동 임베딩하는 방법
• 밀집 의미 임베딩(dense semantic embedding)과 희소 BM25 키워드를 결합하는 방법
• RRF(Reciprocal Rank Fusion, 상호 순위 융합)로 하이브리드 검색을 수행해 가장 관련성 높은 결과를 얻는 방법

출처: Qdrant 공식문서

클라이언트 초기화

먼저 Qdrant Cloud 계정과 전용 유료 클러스터를 만든 뒤 Qdrant 클라이언트를 초기화해요. 클라우드 추론을 활성화하려면 cloud_inference를 True로 설정하면 됩니다.

from qdrant_client import QdrantClient

client = QdrantClient(
    "xyz-example.cloud-region.cloud-provider.cloud.qdrant.io",
    api_key="<paste-your-api-key-here>",
    cloud_inference=True,
    timeout=30,
)

import { QdrantClient } from "@qdrant/js-client-rest";

const client = new QdrantClient({
  url: 'https://xyz-example.qdrant.io:6333',
  apiKey: '<paste-your-api-key-here>',
});

use qdrant_client::Qdrant;

let client = Qdrant::from_url("https://xyz-example.qdrant.io:6334")
    .api_key("<paste-your-api-key-here>")
    .build()
    .unwrap();

import io.qdrant.client.QdrantClient;
import io.qdrant.client.QdrantGrpcClient;

QdrantClient client = new QdrantClient(
    QdrantGrpcClient.newBuilder("xyz-example.qdrant.io", 6334, true)
        .withApiKey("<paste-your-api-key-here>")
        .build()
);

using Qdrant.Client;

var client = new QdrantClient(
    host: "xyz-example.cloud-region.cloud-provider.cloud.qdrant.io",
    https: true,
    apiKey: "<paste-your-api-key-here>"
);

import "github.com/qdrant/go-client/qdrant"

client, err := qdrant.NewClient(&qdrant.Config{
    Host:   "xyz-example.cloud-region.cloud-provider.cloud.qdrant.io",
    Port:   6334,
    APIKey: "<paste-your-api-key-here>",
    UseTLS: true,
})

컬렉션 생성

Qdrant는 벡터와 관련 메타데이터를 컬렉션(collection)에 저장해요. 컬렉션은 생성할 때 벡터 파라미터를 설정해야 해요. 여기서는 희소 벡터에 BM25, 밀집 벡터에 all-minilm-l6-v2를 사용하는 컬렉션을 구성할 거예요.

BM25는 역문서 빈도(IDF, Inverse Document Frequency)를 사용해 여러 문서에 등장하는 흔한 용어의 가중치를 낮추고, 검색에 더 판별적인 희귀 용어의 중요도를 높여요. bm25_sparse_vector라는 이름의 희소 벡터 구성에서 이 옵션을 켜면 Qdrant가 IDF 계산을 처리해 줍니다.

from qdrant_client import QdrantClient, models

client.create_collection(
    collection_name="{collection_name}",
    vectors_config={
        "dense_vector": models.VectorParams(
            size=384,
            distance=models.Distance.COSINE,
        )
    },
    sparse_vectors_config={
        "bm25_sparse_vector": models.SparseVectorParams(
            modifier=models.Modifier.IDF  # Enable Inverse Document Frequency
        )
    },
)

client.createCollection("{collection_name}", {
  vectors: {
    dense_vector: { size: 384, distance: "Cosine" },
  },
  sparse_vectors: {
    bm25_sparse_vector: { modifier: "idf" }, // Enable Inverse Document Frequency
  },
});

use qdrant_client::qdrant::{
    CreateCollectionBuilder, Distance, Modifier, SparseVectorParamsBuilder,
    SparseVectorsConfigBuilder, VectorParamsBuilder, VectorsConfigBuilder,
};

let mut vector_config = VectorsConfigBuilder::default();
vector_config.add_named_vector_params(
    "dense_vector",
    VectorParamsBuilder::new(384, Distance::Cosine),
);
let mut sparse_vectors_config = SparseVectorsConfigBuilder::default();
sparse_vectors_config.add_named_vector_params(
    "bm25_sparse_vector",
    SparseVectorParamsBuilder::default().modifier(Modifier::Idf), // Enable Inverse Document Frequency
);
client
    .create_collection(
        CreateCollectionBuilder::new("{collection_name}")
            .vectors_config(vector_config)
            .sparse_vectors_config(sparse_vectors_config),
    )
    .await?;

import io.qdrant.client.grpc.Collections.CreateCollection;
import io.qdrant.client.grpc.Collections.Distance;
import io.qdrant.client.grpc.Collections.Modifier;
import io.qdrant.client.grpc.Collections.SparseVectorConfig;
import io.qdrant.client.grpc.Collections.SparseVectorParams;
import io.qdrant.client.grpc.Collections.VectorParams;
import io.qdrant.client.grpc.Collections.VectorParamsMap;
import io.qdrant.client.grpc.Collections.VectorsConfig;
import java.util.Map;

client.createCollectionAsync(
    CreateCollection.newBuilder()
        .setCollectionName("{collection_name}")
        .setVectorsConfig(
            VectorsConfig.newBuilder()
                .setParamsMap(
                    VectorParamsMap.newBuilder()
                        .putAllMap(Map.of(
                            "dense_vector",
                            VectorParams.newBuilder()
                                .setSize(384)
                                .setDistance(Distance.Cosine)
                                .build()
                        ))
                )
                .build()
        )
        .setSparseVectorsConfig(
            SparseVectorConfig.newBuilder()
                .putMap(
                    "bm25_sparse_vector",
                    SparseVectorParams.newBuilder()
                        .setModifier(Modifier.Idf)
                        .build()
                )
                .build()
        )
        .build()
).get();

await client.CreateCollectionAsync(
    collectionName: "{collection_name}",
    vectorsConfig: new VectorParamsMap {
        Map = {
            ["dense_vector"] = new VectorParams {
                Size = 384,
                Distance = Distance.Cosine,
            },
        }
    },
    sparseVectorsConfig: new SparseVectorConfig {
        Map = {
            ["bm25_sparse_vector"] = new() {
                Modifier = Modifier.Idf, // Enable Inverse Document Frequency
            }
        }
    }
);

client.CreateCollection(context.Background(), &qdrant.CreateCollection{
    CollectionName: "{collection_name}",
    VectorsConfig: qdrant.NewVectorsConfigMap(map[string]*qdrant.VectorParams{
        "dense_vector": {
            Size:     384,
            Distance: qdrant.Distance_Cosine,
        },
    }),
    SparseVectorsConfig: qdrant.NewSparseVectorsConfig(map[string]*qdrant.SparseVectorParams{
        "bm25_sparse_vector": {
            Modifier: qdrant.Modifier_Idf.Enum(),
        },
    }),
})

데이터 추가

이제 샘플 문서와 관련 메타데이터, 그리고 각 문서의 포인트 ID를 추가할 수 있어요. 여기서는 miriad/miriad-4.4M 데이터셋의 샘플을 사용합니다.

qa_id	paper_id	question	year	venue	specialty	passage_text
38_77498699_0_1	77498699	What are the clinical features of relapsing polychondritis?	2006	Internet Journal of Otorhinolaryngology	Rheumatology	A 45-year-old man presented with painful swelling…
38_77498699_0_2	77498699	What treatments are available for relapsing polychondritis?	2006	Internet Journal of Otorhinolaryngology	Rheumatology	Patient showed improvement after treatment with…
38_88124321_0_3	88124321	How is Takayasu arteritis diagnosed?	2015	Journal of Autoimmune Diseases	Rheumatology	A 32-year-old woman with fatigue and limb pain…

데이터셋 전체를 넣지는 않고, 데모를 위해 처음 100개 항목만 가져올게요.

from qdrant_client.http.models import PointStruct, Document
from datasets import load_dataset
import uuid

dense_model = "sentence-transformers/all-minilm-l6-v2"
bm25_model = "qdrant/bm25"

ds = load_dataset("miriad/miriad-4.4M", split="train[0:100]")
points = []
for idx, item in enumerate(ds):
    passage = item["passage_text"]
    point = PointStruct(
        id=uuid.uuid4().hex,  # use unique string ID
        payload=item,
        vector={
            "dense_vector": Document(text=passage, model=dense_model),
            "bm25_sparse_vector": Document(text=passage, model=bm25_model),
        },
    )
    points.append(point)

client.upload_points(
    collection_name="{collection_name}",
    points=points,
    batch_size=8,
)

import { randomUUID } from "crypto";

const denseModel = "sentence-transformers/all-minilm-l6-v2";
const bm25Model = "qdrant/bm25";

// NOTE: loadDataset is a user-defined function.
// Implement it to handle dataset loading as needed.
const dataset = loadDataset("miriad/miriad-4.4M", "train[0:100]");
const points = dataset.map((item) => {
  const passage = item.passage_text;
  return {
    id: randomUUID().toString(),
    vector: {
      dense_vector: { text: passage, model: denseModel },
      bm25_sparse_vector: { text: passage, model: bm25Model },
    },
  };
});
await client.upsert("{collection_name}", { points });

use qdrant_client::qdrant::{Document, NamedVectors, PointStruct, UpsertPointsBuilder};
use qdrant_client::Payload;
use uuid::Uuid;

let dense_model = "sentence-transformers/all-minilm-l6-v2";
let bm25_model = "qdrant/bm25";

// NOTE: load_dataset is a user-defined function.
// Implement it to handle dataset loading as needed.
let dataset: Vec<_> = load_dataset("miriad/miriad-4.4M", "train[0:100]");
let points: Vec<PointStruct> = dataset
    .iter()
    .map(|item| {
        let passage = item["passage_text"].as_str().unwrap();
        let vectors = NamedVectors::default()
            .add_vector("dense_vector", Document::new(passage, dense_model))
            .add_vector("bm25_sparse_vector", Document::new(passage, bm25_model));
        let payload = Payload::try_from(item.clone()).unwrap();
        PointStruct::new(Uuid::new_v4().to_string(), vectors, payload)
    })
    .collect();

client.upsert_points(UpsertPointsBuilder::new("{collection_name}", points)).await?;

import static io.qdrant.client.PointIdFactory.id;
import static io.qdrant.client.VectorFactory.vector;
import static io.qdrant.client.VectorsFactory.namedVectors;
import io.qdrant.client.grpc.Points.Document;
import io.qdrant.client.grpc.Points.PointStruct;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.UUID;

String denseModel = "sentence-transformers/all-minilm-l6-v2";
String bm25Model = "qdrant/bm25";

// NOTE: loadDataset is a user-defined function.
// Implement it to handle dataset loading as needed.
List<Map<String, String>> dataset = loadDataset("miriad/miriad-4.4M", "train[0:100]");
List<PointStruct> points = new ArrayList<>();
for (Map<String, String> item : dataset) {
    String passage = item.get("passage_text");
    PointStruct point = PointStruct.newBuilder()
        .setId(id(UUID.randomUUID()))
        .setVectors(namedVectors(Map.of(
            "dense_vector",
            vector(Document.newBuilder().setText(passage).setModel(denseModel).build()),
            "bm25_sparse_vector",
            vector(Document.newBuilder().setText(passage).setModel(bm25Model).build())
        )))
        .build();
    points.add(point);
}
client.upsertAsync("{collection_name}", points).get();

var denseModel = "sentence-transformers/all-minilm-l6-v2";
var bm25Model = "qdrant/bm25";

// NOTE: LoadDataset is a user-defined function.
// Implement it to handle dataset loading as needed.
var dataset = LoadDataset("miriad/miriad-4.4M", "train[0:100]");
var points = new List<PointStruct>();
foreach (var item in dataset) {
    var passage = item["passage_text"].ToString();
    var point = new PointStruct {
        Id = Guid.NewGuid(),
        Vectors = new Dictionary<string, Vector> {
            ["dense_vector"] = new Document { Text = passage, Model = denseModel },
            ["bm25_sparse_vector"] = new Document { Text = passage, Model = bm25Model }
        },
    };
    points.Add(point);
}
await client.UpsertAsync(collectionName: "{collectionName}", points: points);

denseModel := "sentence-transformers/all-minilm-l6-v2"
bm25Model := "qdrant/bm25"

// NOTE: loadDataset is a user-defined function.
// Implement it to handle dataset loading as needed.
dataset := loadDataset("miriad/miriad-4.4M", "train[0:100]")
points := make([]*qdrant.PointStruct, 0, 100)
for _, item := range dataset {
    passage := item["passage_text"]
    point := &qdrant.PointStruct{
        Id: qdrant.NewID(uuid.New().String()),
        Vectors: qdrant.NewVectorsMap(map[string]*qdrant.Vector{
            "dense_vector":         qdrant.NewVectorDocument(&qdrant.Document{Text: passage, Model: denseModel}),
            "bm25_sparse_vector":   qdrant.NewVectorDocument(&qdrant.Document{Text: passage, Model: bm25Model}),
        }),
    }
    points = append(points, point)
}
_, err = client.Upsert(context.Background(), &qdrant.UpsertPoints{
    CollectionName: "{collection_name}",
    Points:         points,
})

입력 쿼리 설정

샘플 쿼리를 만들어 볼게요.

query_text = "What is relapsing polychondritis?"

let query_text = "What is relapsing polychondritis?";

let query_text = "What is relapsing polychondritis?";

String queryText = "What is relapsing polychondritis?";

var queryText = "What is relapsing polychondritis?";

queryText := "What is relapsing polychondritis?"

벡터 검색 실행

여기서는 의미적으로 관련 있는 결과를 검색해 줄 질문을 물어볼 거예요. 최종 결과는 RRF(Reciprocal Rank Fusion)를 사용한 재랭킹으로 얻어집니다.

results = client.query_points(
    collection_name="{collection_name}",
    prefetch=[
        models.Prefetch(
            query=models.Document(text=query_text, model=dense_model),
            using="dense_vector",
            limit=5,
        ),
        models.Prefetch(
            query=models.Document(text=query_text, model=bm25_model),
            using="bm25_sparse_vector",
            limit=5,
        ),
    ],
    query=models.FusionQuery(fusion=models.Fusion.RRF),
    limit=5,
    with_payload=True,
)
print(results.points)

const results = await client.query("{collection_name}", {
  prefetch: [
    {
      query: { text: queryText, model: denseModel },
      using: "dense_vector",
    },
    {
      query: { text: queryText, model: bm25Model },
      using: "bm25_sparse_vector",
    },
  ],
  query: { fusion: "rrf" },
});

use qdrant_client::qdrant::{
    Document, Fusion, PrefetchQueryBuilder, Query, QueryPointsBuilder,
};

let dense_prefetch = PrefetchQueryBuilder::default()
    .query(Query::new_nearest(Document::new(query_text, dense_model)))
    .using("dense_vector")
    .build();
let bm25_prefetch = PrefetchQueryBuilder::default()
    .query(Query::new_nearest(Document::new(query_text, bm25_model)))
    .using("bm25_sparse_vector")
    .build();
let query_request = QueryPointsBuilder::new("{collection_name}")
    .add_prefetch(dense_prefetch)
    .add_prefetch(bm25_prefetch)
    .query(Query::new_fusion(Fusion::Rrf))
    .with_payload(true)
    .build();
let results = client.query(query_request).await?;

import static io.qdrant.client.QueryFactory.fusion;
import static io.qdrant.client.QueryFactory.nearest;
import io.qdrant.client.grpc.Points.Document;
import io.qdrant.client.grpc.Points.Fusion;
import io.qdrant.client.grpc.Points.PrefetchQuery;
import io.qdrant.client.grpc.Points.QueryPoints;

PrefetchQuery densePrefetch = PrefetchQuery.newBuilder()
    .setQuery(nearest(Document.newBuilder().setText(queryText).setModel(denseModel).build()))
    .setUsing("dense_vector")
    .build();
PrefetchQuery bm25Prefetch = PrefetchQuery.newBuilder()
    .setQuery(nearest(Document.newBuilder().setText(queryText).setModel(bm25Model).build()))
    .setUsing("bm25_sparse_vector")
    .build();
QueryPoints request = QueryPoints.newBuilder()
    .setCollectionName("{collection_name}")
    .addPrefetch(densePrefetch)
    .addPrefetch(bm25Prefetch)
    .setQuery(fusion(Fusion.RRF))
    .build();
client.queryAsync(request).get();

await client.QueryAsync(
    collectionName: "{collection_name}",
    prefetch: new List<PrefetchQuery> {
        new() {
            Query = new Document { Text = queryText, Model = bm25Model },
            Using = "bm25_sparse_vector",
            Limit = 5
        },
        new() {
            Query = new Document { Text = queryText, Model = denseModel },
            Using = "dense_vector",
            Limit = 5
        }
    },
    query: Fusion.Rrf,
    limit: 5
);

prefetch := []*qdrant.PrefetchQuery{
    {
        Query: qdrant.NewQueryDocument(&qdrant.Document{Text: queryText, Model: bm25Model}),
        Using: qdrant.PtrOf("bm25_sparse_vector"),
    },
    {
        Query: qdrant.NewQueryDocument(&qdrant.Document{Text: queryText, Model: denseModel}),
        Using: qdrant.PtrOf("dense_vector"),
    },
}
client.Query(context.Background(), &qdrant.QueryPoints{
    CollectionName: "{collection_name}",
    Prefetch:       prefetch,
    Query:          qdrant.NewQueryFusion(qdrant.Fusion_RRF),
})

의미 검색 엔진은 관련성 순서로 가장 유사한 결과를 검색해 줍니다.

[ScoredPoint(id='9968a760-fbb5-4d91-8549-ffbaeb3ebdba', version=0, score=14.545895, payload={'text': "Relapsing Polychondritis is a rare..."}, vector=None, shard_key=None, order_value=None)]

다음 단계

하이브리드 검색 결과 품질은 더 비싸지만 더 높은 품질의 모델로 결과를 재랭킹하면 개선할 수 있어요. 자세한 내용은 재랭킹을 사용한 하이브리드 검색 튜토리얼에서 확인하세요.

더 알아보기 (Learn more)

• 재랭킹을 사용한 하이브리드 검색 — late interaction 모델로 결과 재랭킹
• Qdrant Cloud 추론 — 클라우드 기반 임베딩
• 하이브리드 쿼리 — RRF와 하이브리드 검색 심화
• Qdrant 공식 문서 홈