멀티모달 검색
멀티모달 검색 (tutorials-basics-multimodal-search)
| 시간: 15분 | 난이도: 초급 | 출력물: GitHub |  |
|---|
개요 (Overview)
여러분은 서로 다른 유형의 데이터를 조합할 때 정보를 더 효과적으로 이해하고 공유하는 경우가 많아요. 위안 음식의 맛은 어린 시절의 기억을 떠올리게 하죠. 어떤 노래는 긴 문단 대신 "팜팜 짝짝" 소리만으로 설명될 수 있어요. 이모지와 스티커는 말보다 더 빠르게 감정이나 복잡한 생각을 표현할 수 있어요.
텍스트, 이미지, 비디오, 오디오 같은 데이터의 모달리티(modality)들은 다양한 조합으로 의미 검색 애플리케이션의 훌륭한 사용 사례를 만들어요.
벡터 데이터베이스는 모달리티에 구애받지 않기(modality-agnostic) 때문에 이런 애플리케이션을 구축하는 데 아주 적합해요.
이 튜토리얼은 이미지와 텍스트 두 가지 모달리티를 다뤄요. **의미적 간극(semantic gap)**을 메워 주는 임베딩 모델만 선택하면, 어떤 모달리티 조합으로도 의미 검색 애플리케이션을 만들 수 있어요.
**의미적 간극(semantic gap)**이란 밝기 같은 저수준 특성과 귀여움 같은 고수준 개념 사이의 차이를 말해요.
Cohere Embed 4.0은 예를 들어 멀티모달 및 다국어 임베딩을 위해 만들어졌고, 100개 이상의 언어를 지원해요. 이 튜토리얼에서는 모델을 직접 실행하지 않고 Qdrant Cloud Inference를 통해 호출해요. 그래서 Qdrant가 임베딩을 생성하고 한 번에 컬렉션에 저장해요.
설정 (Setup)
클라이언트를 설치해요:
pip install qdrant-client
npm install @qdrant/js-client-rest
cargo add qdrant-client
// build.gradle
dependencies {
implementation("io.qdrant:client:+") // specify the desired version
}
dotnet add package Qdrant.Client
go get github.com/qdrant/go-client
데이터셋 (Dataset)
데모를 간단하게 만들기 위해, 이 튜토리얼은 이미지와 캡션으로 이루어진 아주 작은 데이터셋을 사용해요.
튜토리얼 이미지를 다운로드해서, 코드나 노트북과 같은 폴더에 images라는 이름의 폴더에 넣어 주세요.
Qdrant 연결 (Connect to Qdrant)
- Cloud Inference가 활성화된 Qdrant 클라이언트 객체를 생성해요.
Qdrant Cloud Free Tier Cluster를 사용할 거예요. 무료 클러스터를 만들고, 연결된 API 키와 엔드포인트 URL을 저장한 뒤 Qdrant 클라이언트를 인스턴스화하세요.
cloud_inference=True로 설정하면 Qdrant가 임베딩을 생성해 줘요:
import os
from qdrant_client import QdrantClient, models
client = QdrantClient(
url=os.getenv("QDRANT_URL"),
api_key=os.getenv("QDRANT_API_KEY"),
cloud_inference=True,
)
const client = new QdrantClient({
url: process.env.QDRANT_URL,
apiKey: process.env.QDRANT_API_KEY,
});
let client = Qdrant::from_url(&std::env::var("QDRANT_URL")?)
.api_key(std::env::var("QDRANT_API_KEY")?)
.build()?;
QdrantClient client =
new QdrantClient(
QdrantGrpcClient.newBuilder(QDRANT_URL, 6334, true)
.withApiKey(QDRANT_API_KEY)
.build());
var client = new QdrantClient(
host: QDRANT_URL,
https: true,
apiKey: QDRANT_API_KEY
);
client, err := qdrant.NewClient(&qdrant.Config{
Host: QDRANT_URL,
APIKey: QDRANT_API_KEY,
UseTLS: true,
})
- 데이터셋과 이미지 인코딩 도우미 함수를 정의해요. Cloud Inference는 이미지를 base64 데이터 URL로 받아들이므로, 업로드하기 전에 각 파일을 변환해야 해요:
import base64
def image_to_base64_url(image_path: str) -> str:
prefix = "data:image/png;base64"
with open(image_path, "rb") as image_file:
return prefix + "," + base64.b64encode(image_file.read()).decode("utf-8")
documents = [
{"caption": "An image about plane emergency safety.", "image": "images/image-1.png"},
{"caption": "An image about airplane components.", "image": "images/image-2.png"},
{"caption": "An image about COVID safety restrictions.", "image": "images/image-3.png"},
{"caption": "A confidential image about UFO sightings.", "image": "images/image-4.png"},
{"caption": "An image about unusual footprints on Aralar 2011.", "image": "images/image-5.png"},
]
function imageToBase64Url(imagePath: string): string {
const prefix = "data:image/png;base64";
const imageBuffer = readFileSync(imagePath);
return `${prefix},${imageBuffer.toString("base64")}`;
}
const documents = [
{ caption: "An image about plane emergency safety.", image: "images/image-1.png" },
{ caption: "An image about airplane components.", image: "images/image-2.png" },
{ caption: "An image about COVID safety restrictions.", image: "images/image-3.png" },
{ caption: "A confidential image about UFO sightings.", image: "images/image-4.png" },
{ caption: "An image about unusual footprints on Aralar 2011.", image: "images/image-5.png" },
];
fn image_to_base64_url(image_path: &str) -> anyhow::Result<String> {
let prefix = "data:image/png;base64";
let bytes = std::fs::read(image_path)?;
Ok(format!("{prefix},{}", BASE64_STANDARD.encode(bytes)))
}
struct Doc {
caption: &'static str,
image: &'static str,
}
let documents = vec![
Doc { caption: "An image about plane emergency safety.", image: "images/image-1.png" },
Doc { caption: "An image about airplane components.", image: "images/image-2.png" },
Doc { caption: "An image about COVID safety restrictions.", image: "images/image-3.png" },
Doc { caption: "A confidential image about UFO sightings.", image: "images/image-4.png" },
Doc { caption: "An image about unusual footprints on Aralar 2011.", image: "images/image-5.png" },
];
static class Doc {
final String caption;
final String image;
Doc(String caption, String image) {
this.caption = caption;
this.image = image;
}
}
static String imageToBase64Url(String imagePath) throws Exception {
String prefix = "data:image/png;base64";
byte[] bytes = Files.readAllBytes(Path.of(imagePath));
return prefix + "," + Base64.getEncoder().encodeToString(bytes);
}
static List<Doc> documents = List.of(
new Doc("An image about plane emergency safety.", "images/image-1.png"),
new Doc("An image about airplane components.", "images/image-2.png"),
new Doc("An image about COVID safety restrictions.", "images/image-3.png"),
new Doc("A confidential image about UFO sightings.", "images/image-4.png"),
new Doc("An image about unusual footprints on Aralar 2011.", "images/image-5.png")
);
static string ImageToBase64Url(string imagePath)
{
string prefix = "data:image/png;base64";
byte[] bytes = File.ReadAllBytes(imagePath);
return $"{prefix},{Convert.ToBase64String(bytes)}";
}
var documents = new[]
{
new { Caption = "An image about plane emergency safety.", Image = "images/image-1.png" },
new { Caption = "An image about airplane components.", Image = "images/image-2.png" },
new { Caption = "An image about COVID safety restrictions.", Image = "images/image-3.png" },
new { Caption = "A confidential image about UFO sightings.", Image = "images/image-4.png" },
new { Caption = "An image about unusual footprints on Aralar 2011.", Image = "images/image-5.png" },
};
type Doc struct {
Caption string
Image string
}
func imageToBase64Url(imagePath string) (string, error) {
prefix := "data:image/png;base64"
bytes, err := os.ReadFile(imagePath)
if err != nil {
return "", err
}
return fmt.Sprintf("%s,%s", prefix, base64.StdEncoding.EncodeToString(bytes)), nil
}
var documents = []Doc{
{Caption: "An image about plane emergency safety.", Image: "images/image-1.png"},
{Caption: "An image about airplane components.", Image: "images/image-2.png"},
{Caption: "An image about COVID safety restrictions.", Image: "images/image-3.png"},
{Caption: "A confidential image about UFO sightings.", Image: "images/image-4.png"},
{Caption: "An image about unusual footprints on Aralar 2011.", Image: "images/image-5.png"},
}
- 캡션이 있는 이미지를 위한 컬렉션을 생성해요.
COLLECTION_NAME = "multimodal-embeddings"
if not client.collection_exists(COLLECTION_NAME):
client.create_collection(
collection_name=COLLECTION_NAME,
vectors_config={
"image": models.VectorParams(size=512, distance=models.Distance.COSINE),
"text": models.VectorParams(size=512, distance=models.Distance.COSINE),
}
)
const collectionName = "multimodal-embeddings";
if (!(await client.collectionExists(collectionName)).exists) {
await client.createCollection(collectionName, {
vectors: {
image: { size: 512, distance: "Cosine" },
text: { size: 512, distance: "Cosine" },
},
});
}
let collection_name = "multimodal-embeddings";
if !client.collection_exists(collection_name).await? {
let mut vectors = VectorsConfigBuilder::default();
vectors.add_named_vector_params("image", VectorParamsBuilder::new(512, Distance::Cosine));
vectors.add_named_vector_params("text", VectorParamsBuilder::new(512, Distance::Cosine));
client
.create_collection(CreateCollectionBuilder::new(collection_name).vectors_config(vectors))
.await?;
}
String collectionName = "multimodal-embeddings";
if (!client.collectionExistsAsync(collectionName).get()) {
client.createCollectionAsync(
CreateCollection.newBuilder()
.setCollectionName(collectionName)
.setVectorsConfig(
VectorsConfig.newBuilder()
.setParamsMap(
VectorParamsMap.newBuilder()
.putMap(
"image",
VectorParams.newBuilder()
.setSize(512)
.setDistance(Distance.Cosine)
.build())
.putMap(
"text",
VectorParams.newBuilder()
.setSize(512)
.setDistance(Distance.Cosine)
.build())
.build()))
.build()
).get();
}
string collectionName = "multimodal-embeddings";
if (!await client.CollectionExistsAsync(collectionName))
{
await client.CreateCollectionAsync(
collectionName: collectionName,
vectorsConfig: new VectorParamsMap
{
Map =
{
["image"] = new VectorParams { Size = 512, Distance = Distance.Cosine },
["text"] = new VectorParams { Size = 512, Distance = Distance.Cosine },
}
}
);
}
collectionName := "multimodal-embeddings"
exists, err := client.CollectionExists(context.Background(), collectionName)
if !exists {
client.CreateCollection(context.Background(), &qdrant.CreateCollection{
CollectionName: collectionName,
VectorsConfig: qdrant.NewVectorsConfigMap(
map[string]*qdrant.VectorParams{
"image": {
Size: 512,
Distance: qdrant.Distance_Cosine,
},
"text": {
Size: 512,
Distance: qdrant.Distance_Cosine,
},
},
),
})
}
컬렉션을 만들 때 주의할 점: 이미지 벡터와 텍스트 벡터를 각각 image, text라는 이름의 벡터로 만들었어요. 둘 다 크기 512, 거리 지표는 코사인 유사도(COSINE)를 사용해요.
데이터를 Qdrant에 업로드 (Upload Data to Qdrant)
캡션이 있는 이미지를 컬렉션에 업로드해요. 각 이미지와 캡션은 Cloud Inference를 통해 Cohere Embed 4.0으로 임베딩되고, point로 저장돼요.
헤더로 Cohere API 키를 전달하고, 각 벡터를 models.Document(텍스트용) 또는 models.Image(이미지용)로 설명하면서 Cohere 모델 이름과 원하는 출력 차원을 지정해요:
from qdrant_client.context_headers import headers
cohere_api_key = os.getenv("COHERE_API_KEY")
with headers({"cohere-api-key": cohere_api_key}):
client.upsert(
collection_name=COLLECTION_NAME,
points=[
models.PointStruct(
id=idx,
vector={
"text": models.Document(
text=doc["caption"],
model="cohere/embed-v4.0",
options={"output_dimension": 512},
),
"image": models.Image(
image=image_to_base64_url(doc["image"]),
model="cohere/embed-v4.0",
options={"output_dimension": 512},
),
},
payload=doc
)
for idx, doc in enumerate(documents)
]
)
const cohereApiKey = process.env.COHERE_API_KEY!;
await withHeaders({ "cohere-api-key": cohereApiKey }, () =>
client.upsert(collectionName, {
points: documents.map((doc, idx) => ({
id: idx,
vector: {
text: { text: doc.caption, model: "cohere/embed-v4.0", options: { output_dimension: 512 } },
image: { image: imageToBase64Url(doc.image), model: "cohere/embed-v4.0", options: { output_dimension: 512 } },
},
payload: doc,
})),
})
);
let cohere_api_key = std::env::var("COHERE_API_KEY")?;
let mut options: HashMap<String, Value> = HashMap::new();
options.insert("output_dimension".to_string(), 512i64.into());
let mut points = Vec::new();
for (idx, doc) in documents.iter().enumerate() {
let vectors = NamedVectors::default()
.add_vector(
"text",
DocumentBuilder::new(doc.caption, "cohere/embed-v4.0")
.options(options.clone())
.build(),
)
.add_vector(
"image",
ImageBuilder::new_from_base64(image_to_base64_url(doc.image)?, "cohere/embed-v4.0")
.options(options.clone())
.build(),
);
points.push(PointStruct::new(
idx as u64,
vectors,
[
("caption", doc.caption.into()),
("image", doc.image.into()),
],
));
}
client
.with_header("cohere-api-key", &cohere_api_key)
.upsert_points(UpsertPointsBuilder::new(collection_name, points))
.await?;
String cohereApiKey = System.getenv("COHERE_API_KEY");
Context ctx = RequestHeaders.withHeader(
Context.current(), "cohere-api-key", cohereApiKey);
List<PointStruct> points = new java.util.ArrayList<>();
for (int idx = 0; idx < documents.size(); idx++) {
Doc doc = documents.get(idx);
points.add(
PointStruct.newBuilder()
.setId(io.qdrant.client.PointIdFactory.id(idx))
.setVectors(
namedVectors(
Map.of(
"text",
vector(
Document.newBuilder()
.setText(doc.caption)
.setModel("cohere/embed-v4.0")
.putOptions("output_dimension", value(512))
.build()),
"image",
vector(
Image.newBuilder()
.setImage(value(imageToBase64Url(doc.image)))
.setModel("cohere/embed-v4.0")
.putOptions("output_dimension", value(512))
.build()))))
.putAllPayload(
Map.of(
"caption", value(doc.caption),
"image", value(doc.image)))
.build());
}
ctx.call(() -> client.upsertAsync(collectionName, points).get());
string cohereApiKey = Environment.GetEnvironmentVariable("COHERE_API_KEY")!;
var points = documents.Select((doc, idx) => new PointStruct
{
Id = (ulong)idx,
Vectors = new Dictionary<string, Vector>
{
["text"] = new Document
{
Text = doc.Caption,
Model = "cohere/embed-v4.0",
Options = { ["output_dimension"] = 512 },
},
["image"] = new Image
{
Image_ = ImageToBase64Url(doc.Image),
Model = "cohere/embed-v4.0",
Options = { ["output_dimension"] = 512 },
},
},
Payload = { ["caption"] = doc.Caption, ["image"] = doc.Image }
}).ToList();
using (RequestHeaders.Use("cohere-api-key", cohereApiKey))
await client.UpsertAsync(collectionName: collectionName, points: points);
cohereApiKey := os.Getenv("COHERE_API_KEY")
ctx := qdrant.WithHeader(context.Background(), "cohere-api-key", cohereApiKey)
points := make([]*qdrant.PointStruct, len(documents))
for idx, doc := range documents {
imageUrl, err := imageToBase64Url(doc.Image)
points[idx] = &qdrant.PointStruct{
Id: qdrant.NewIDNum(uint64(idx)),
Vectors: qdrant.NewVectorsMap(map[string]*qdrant.Vector{
"text": qdrant.NewVectorDocument(&qdrant.Document{
Text: doc.Caption,
Model: "cohere/embed-v4.0",
Options: qdrant.NewValueMap(map[string]any{
"output_dimension": 512,
}),
}),
"image": qdrant.NewVectorImage(&qdrant.Image{
Image: qdrant.NewValueString(imageUrl),
Model: "cohere/embed-v4.0",
Options: qdrant.NewValueMap(map[string]any{
"output_dimension": 512,
}),
}),
}),
Payload: qdrant.NewValueMap(map[string]any{
"caption": doc.Caption,
"image": doc.Image,
}),
}
}
client.Upsert(ctx, &qdrant.UpsertPoints{
CollectionName: collectionName,
Points: points,
})
검색 (Search)
텍스트 → 이미지 (Text-to-Image)
"Plane components" 쿼리에 어떤 이미지가 돌아오는지 확인해 봐요. 업로드할 때와 같은 방식으로 쿼리를 models.Document로 감싸면, Cloud Inference가 같은 모델로 이를 임베딩해요:
from PIL import Image
with headers({"cohere-api-key": cohere_api_key}):
payload = client.query_points(
collection_name=COLLECTION_NAME,
query=models.Document(
text="Plane components",
model="cohere/embed-v4.0",
options={"output_dimension": 512},
),
using="image",
with_payload=["image"],
limit=1
).points[0].payload
Image.open(payload["image"])
const textToImageResults = await withHeaders({ "cohere-api-key": cohereApiKey }, () =>
client.query(collectionName, {
query: { text: "Plane components", model: "cohere/embed-v4.0", options: { output_dimension: 512 } },
using: "image",
with_payload: ["image"],
limit: 1,
})
);
console.log(textToImageResults.points[0].payload!.image);
let results = client
.with_header("cohere-api-key", &cohere_api_key)
.query(
QueryPointsBuilder::new(collection_name)
.query(Query::new_nearest(
DocumentBuilder::new("Plane components", "cohere/embed-v4.0")
.options(options.clone())
.build(),
))
.using("image")
.with_payload(true)
.limit(1),
)
.await?;
println!("{:?}", results.result[0].payload.get("image"));
var results = ctx.call(() -> client.queryAsync(
QueryPoints.newBuilder()
.setCollectionName(collectionName)
.setQuery(
nearest(
Document.newBuilder()
.setText("Plane components")
.setModel("cohere/embed-v4.0")
.putOptions("output_dimension", value(512))
.build()))
.setUsing("image")
.setWithPayload(enable(true))
.setLimit(1)
.build()
).get());
System.out.println(results.get(0).getPayloadMap().get("image"));
IReadOnlyList<ScoredPoint> results;
using (RequestHeaders.Use("cohere-api-key", cohereApiKey))
results = await client.QueryAsync(
collectionName: collectionName,
query: new Document
{
Text = "Plane components",
Model = "cohere/embed-v4.0",
Options = { ["output_dimension"] = 512 },
},
usingVector: "image",
payloadSelector: true,
limit: 1
);
Console.WriteLine(results[0].Payload["image"]);
results, err := client.Query(ctx, &qdrant.QueryPoints{
CollectionName: collectionName,
Query: qdrant.NewQueryNearest(
qdrant.NewVectorInputDocument(&qdrant.Document{
Text: "Plane components",
Model: "cohere/embed-v4.0",
Options: qdrant.NewValueMap(map[string]any{
"output_dimension": 512,
}),
}),
),
Using: qdrant.PtrOf("image"),
WithPayload: qdrant.NewWithPayloadInclude("image"),
Limit: qdrant.PtrOf(uint64(1)),
})
fmt.Println(results[0].Payload["image"])
결과 (Response):
다국어 검색 (Multilingual Search)
이제 Cohere Embed 4.0이 지원하는 30개 이상의 언어 중 하나인 이탈리아어로 같은 쿼리를 실행해 결과를 비교해 봐요:
with headers({"cohere-api-key": cohere_api_key}):
payload = client.query_points(
collection_name=COLLECTION_NAME,
query=models.Document(
text="Componenti di un aereo",
model="cohere/embed-v4.0",
options={"output_dimension": 512},
),
using="image",
with_payload=["image"],
limit=1
).points[0].payload
Image.open(payload["image"])
const multilingualResults = await withHeaders({ "cohere-api-key": cohereApiKey }, () =>
client.query(collectionName, {
query: { text: "Componenti di un aereo", model: "cohere/embed-v4.0", options: { output_dimension: 512 } },
using: "image",
with_payload: ["image"],
limit: 1,
})
);
console.log(multilingualResults.points[0].payload!.image);
let results = client
.with_header("cohere-api-key", &cohere_api_key)
.query(
QueryPointsBuilder::new(collection_name)
.query(Query::new_nearest(
DocumentBuilder::new("Componenti di un aereo", "cohere/embed-v4.0")
.options(options.clone())
.build(),
))
.using("image")
.with_payload(true)
.limit(1),
)
.await?;
println!("{:?}", results.result[0].payload.get("image"));
results = ctx.call(() -> client.queryAsync(
QueryPoints.newBuilder()
.setCollectionName(collectionName)
.setQuery(
nearest(
Document.newBuilder()
.setText("Componenti di un aereo")
.setModel("cohere/embed-v4.0")
.putOptions("output_dimension", value(512))
.build()))
.setUsing("image")
.setWithPayload(enable(true))
.setLimit(1)
.build()
).get());
System.out.println(results.get(0).getPayloadMap().get("image"));
using (RequestHeaders.Use("cohere-api-key", cohereApiKey))
results = await client.QueryAsync(
collectionName: collectionName,
query: new Document
{
Text = "Componenti di un aereo",
Model = "cohere/embed-v4.0",
Options = { ["output_dimension"] = 512 },
},
usingVector: "image",
payloadSelector: true,
limit: 1
);
Console.WriteLine(results[0].Payload["image"]);
results, err = client.Query(ctx, &qdrant.QueryPoints{
CollectionName: collectionName,
Query: qdrant.NewQueryNearest(
qdrant.NewVectorInputDocument(&qdrant.Document{
Text: "Componenti di un aereo",
Model: "cohere/embed-v4.0",
Options: qdrant.NewValueMap(map[string]any{
"output_dimension": 512,
}),
}),
),
Using: qdrant.PtrOf("image"),
WithPayload: qdrant.NewWithPayloadInclude("image"),
Limit: qdrant.PtrOf(uint64(1)),
})
fmt.Println(results[0].Payload["image"])
결과 (Response):
이미지 → 텍스트 (Image-to-Text)
이번에는 이 이미지에서 시작해 역방향 검색을 실행해 봐요:
이미지를 models.Image로 임베딩하고, 텍스트 벡터 사이에서만 검색해요:
with headers({"cohere-api-key": cohere_api_key}):
payload = client.query_points(
collection_name=COLLECTION_NAME,
query=models.Image(
image=image_to_base64_url("images/image-2.png"),
model="cohere/embed-v4.0",
options={"output_dimension": 512},
),
using="text",
with_payload=["caption"],
limit=1
).points[0].payload
print(payload["caption"])
const imageToTextResults = await withHeaders({ "cohere-api-key": cohereApiKey }, () =>
client.query(collectionName, {
query: { image: imageToBase64Url("images/image-2.png"), model: "cohere/embed-v4.0", options: { output_dimension: 512 } },
using: "text",
with_payload: ["caption"],
limit: 1,
})
);
console.log(imageToTextResults.points[0].payload!.caption);
let results = client
.with_header("cohere-api-key", &cohere_api_key)
.query(
QueryPointsBuilder::new(collection_name)
.query(Query::new_nearest(
ImageBuilder::new_from_base64(
image_to_base64_url("images/image-2.png")?,
"cohere/embed-v4.0",
)
.options(options.clone())
.build(),
))
.using("text")
.with_payload(true)
.limit(1),
)
.await?;
println!("{:?}", results.result[0].payload.get("caption"));
results = ctx.call(() -> client.queryAsync(
QueryPoints.newBuilder()
.setCollectionName(collectionName)
.setQuery(
nearest(
Image.newBuilder()
.setImage(value(imageToBase64Url("images/image-2.png")))
.setModel("cohere/embed-v4.0")
.putOptions("output_dimension", value(512))
.build()))
.setUsing("text")
.setWithPayload(enable(true))
.setLimit(1)
.build()
).get());
System.out.println(results.get(0).getPayloadMap().get("caption"));
using (RequestHeaders.Use("cohere-api-key", cohereApiKey))
results = await client.QueryAsync(
collectionName: collectionName,
query: new Image
{
Image_ = ImageToBase64Url("images/image-2.png"),
Model = "cohere/embed-v4.0",
Options = { ["output_dimension"] = 512 },
},
usingVector: "text",
payloadSelector: true,
limit: 1
);
Console.WriteLine(results[0].Payload["caption"]);
queryImageUrl, err := imageToBase64Url("images/image-2.png")
results, err = client.Query(ctx, &qdrant.QueryPoints{
CollectionName: collectionName,
Query: qdrant.NewQueryNearest(
qdrant.NewVectorInputImage(&qdrant.Image{
Image: qdrant.NewValueString(queryImageUrl),
Model: "cohere/embed-v4.0",
Options: qdrant.NewValueMap(map[string]any{
"output_dimension": 512,
}),
}),
),
Using: qdrant.PtrOf("text"),
WithPayload: qdrant.NewWithPayloadInclude("caption"),
Limit: qdrant.PtrOf(uint64(1)),
})
fmt.Println(results[0].Payload["caption"])
결과 (Response):
'An image about airplane components.'
다음 단계 (Next Steps)
이미지와 텍스트만의 멀티모달 검색만으로도 많은 사용 사례를 지원해요: 전자상거래, 미디어 관리, 콘텐츠 추천, 감정 인식, 생물의학 이미지 검색, 수화 음성 전사 등이 그 예시예요.
원하는 제품의 사진과 "베이지색으로" 같은 구체적인 텍스트 요구사항을 모두 가진 쇼핑객을 생각해 보세요. 텍스트나 이미지만으로 검색할 수도 있고, **늦은 융합(late fusion)**을 통해 임베딩을 결합할 수도 있어요 (벡터를 합치고 가중치를 주는 방식은 놀랍게도 잘 작동해요).
두 모달리티를 Discovery Search와 결합하면, 어떤 모달리티도 단독으로는 찾지 못했을 결과를 표면화할 수도 있어요.
벡터 검색과 유사도 학습에 대해 이야기하고, 실험하고, 즐기는 Discord 커뮤니티에 참여해 보세요!