k-NN 쿼리 explain
k-NN 쿼리 explain
knn 쿼리에서 점수가 어떻게 계산되고 정규화되며 결합되는지 이해하고 싶을 때 explain 파라미터를 사용해요. 이 글에서는 explain을 사용하는 방법과 응답에 포함되는 상세 설명을 살펴봐요.
출처: 문서
본문
3.0 버전에서 도입되었어요. knn 쿼리에서 점수가 어떻게 계산되고 정규화되며 결합되는지 이해하려면 explain 파라미터를 사용하면 돼요. 이 파라미터를 활성화하면 각 검색 결과에 대한 점수 산정 과정의 상세 정보를 볼 수 있어요. 여기에는 사용된 점수 정규화 기법, 서로 다른 점수가 결합되는 방식, 개별 하위 쿼리 점수의 계산 과정이 모두 드러나요. 이렇게 전반적인 통찰을 얻으면 knn 쿼리 결과를 더 쉽게 이해하고 최적화할 수 있어요. explain에 대한 자세한 내용은 Explain API를 참고하세요. explain은 리소스와 시간 측면 모두에서 비용이 많이 드는 작업이에요. 운영(production) 클러스터에서는 문제 해결(troubleshooting) 목적으로 가급적 드물게 사용하는 것을 권장해요. Faiss 엔진에 대해 완전한 knn 쿼리를 실행할 때 URL에 explain 파라미터를 제공할 수 있어요. 문법은 다음과 같아요.
GET {index}/_search?explain=true
POST {index}/_search?explain=true
Lucene 엔진을 사용하는 모든 유형의 쿼리에 대한 k-NN 검색에서 explain은 Faiss 엔진에서처럼 상세한 설명을 반환하지 않아요. explain 파라미터는 Faiss 엔진을 사용하는 다음 유형의 k-NN 검색에서 동작해요.
- 근사(Approximate) k-NN 검색
- 정확 검색을 포함한 근사 k-NN 검색
- 디스크 기반(Disk-based) 검색
- 효율적인 필터링을 적용한 k-NN 검색
- 방사형(Radial) 검색
- term 쿼리와 함께 사용하는 k-NN 검색 중첩(nested) 필드에 대한 k-NN 검색에서 explain은 다른 검색에서처럼 상세한 설명을 반환하지 않아요. explain 파라미터는 쿼리 파라미터로 제공할 수 있어요.
GET my-knn-index/_search?explain=true
{
"query": {
"knn": {
"my_vector": {
"vector": [2, 3, 5, 7],
"k": 2
}
}
}
}
또는 요청 본문(request body)에 explain 파라미터를 넣을 수도 있어요.
GET my-knn-index/_search
{
"query": {
"knn": {
"my_vector": {
"vector": [2, 3, 5, 7],
"k": 2
}
}
},
"explain": true
}
근사 k-NN 검색 예제
{
"took": 216038,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 2,
"relation": "eq"
},
"max_score": 88.4,
"hits": [
{
"_shard": "[my-knn-index-1][0]",
"_node": "VHcyav6OTsmXdpsttX2Yug",
"_index": "my-knn-index-1",
"_id": "5",
"_score": 88.4,
"_source": {
"my_vector1": [
2.5,
3.5,
5.5,
7.4
],
"price": 8.9
},
"_explanation": {
"value": 88.4,
"description": "the type of knn search executed was Approximate-NN",
"details": [
{
"value": 88.4,
"description": "the type of knn search executed at leaf was Approximate-NN with vectorDataType = FLOAT, spaceType = innerproduct where score is computed as `-rawScore + 1` from:",
"details": [
{
"value": -87.4,
"description": "rawScore, returned from FAISS library",
"details": []
}
]
}
]
}
},
{
"_shard": "[my-knn-index-1][0]",
"_node": "VHcyav6OTsmXdpsttX2Yug",
"_index": "my-knn-index-1",
"_id": "2",
"_score": 84.7,
"_source": {
"my_vector1": [
2.5,
3.5,
5.6,
6.7
],
"price": 5.5
},
"_explanation": {
"value": 84.7,
"description": "the type of knn search executed was Approximate-NN",
"details": [
{
"value": 84.7,
"description": "the type of knn search executed at leaf was Approximate-NN with vectorDataType = FLOAT, spaceType = innerproduct where score is computed as `-rawScore + 1` from:",
"details": [
{
"value": -83.7,
"description": "rawScore, returned from FAISS library",
"details": []
}
]
}
]
}
}
]
}
}
정확 검색을 포함한 근사 k-NN 검색 예제
{
"took": 87,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 2,
"relation": "eq"
},
"max_score": 84.7,
"hits": [
{
"_shard": "[my-knn-index-1][0]",
"_node": "MQVux8dZRWeznuEYKhMq0Q",
"_index": "my-knn-index-1",
"_id": "7",
"_score": 84.7,
"_source": {
"my_vector2": [
2.5,
3.5,
5.6,
6.7
],
"price": 5.5
},
"_explanation": {
"value": 84.7,
"description": "the type of knn search executed was Approximate-NN",
"details": [
{
"value": 84.7,
"description": "the type of knn search executed at leaf was Exact with spaceType = INNER_PRODUCT, vectorDataType = FLOAT, queryVector = [2.0, 3.0, 5.0, 6.0]",
"details": []
}
]
}
},
{
"_shard": "[my-knn-index-1][0]",
"_node": "MQVux8dZRWeznuEYKhMq0Q",
"_index": "my-knn-index-1",
"_id": "8",
"_score": 82.2,
"_source": {
"my_vector2": [
4.5,
5.5,
6.7,
3.7
],
"price": 4.4
},
"_explanation": {
"value": 82.2,
"description": "the type of knn search executed was Approximate-NN",
"details": [
{
"value": 82.2,
"description": "the type of knn search executed at leaf was Exact with spaceType = INNER_PRODUCT, vectorDataType = FLOAT, queryVector = [2.0, 3.0, 5.0, 6.0]",
"details": []
}
]
}
}
]
}
디스크 기반 검색 예제
{
"took" : 4,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 1,
"relation" : "eq"
},
"max_score" : 381.0,
"hits" : [
{
"_shard" : "[my-vector-index][0]",
"_node" : "pLaiqZftTX-MVSKdQSu7ow",
"_index" : "my-vector-index",
"_id" : "9",
"_score" : 381.0,
"_source" : {
"my_vector_field" : [
9.5,
9.5,
9.5,
9.5,
9.5,
9.5,
9.5,
9.5
],
"price" : 8.9
},
"_explanation" : {
"value" : 381.0,
"description" : "the type of knn search executed was Disk-based and the first pass k was 100 with vector dimension of 8, over sampling factor of 5.0, shard level rescoring enabled",
"details" : [
{
"value" : 381.0,
"description" : "the type of knn search executed at leaf was Approximate-NN with spaceType = HAMMING, vectorDataType = FLOAT, queryVector = [1.5, 2.5, 3.5, 4.5, 5.5, 6.5, 7.5, 8.5]",
"details" : [ ]
}
]
}
}
]
}
}
효율적인 필터링을 적용한 k-NN 검색 예제
{
"took" : 51,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 2,
"relation" : "eq"
},
"max_score" : 0.8620689,
"hits" : [
{
"_shard" : "[products-shirts][0]",
"_node" : "9epk8WoFT8yvnUI0tAaJgQ",
"_index" : "products-shirts",
"_id" : "8",
"_score" : 0.8620689,
"_source" : {
"item_vector" : [
2.4,
4.0,
3.0
],
"size" : "small",
"rating" : 8
},
"_explanation" : {
"value" : 0.8620689,
"description" : "the type of knn search executed was Approximate-NN",
"details" : [
{
"value" : 0.8620689,
"description" : "the type of knn search executed at leaf was Exact since filteredIds = 2 is less than or equal to K = 10 with spaceType = L2, vectorDataType = FLOAT, queryVector = [2.0, 4.0, 3.0]",
"details" : [ ]
}
]
}
},
{
"_shard" : "[products-shirts][0]",
"_node" : "9epk8WoFT8yvnUI0tAaJgQ",
"_index" : "products-shirts",
"_id" : "6",
"_score" : 0.029691212,
"_source" : {
"item_vector" : [
6.4,
3.4,
6.6
],
"size" : "small",
"rating" : 9
},
"_explanation" : {
"value" : 0.029691212,
"description" : "the type of knn search executed was Approximate-NN",
"details" : [
{
"value" : 0.029691212,
"description" : "the type of knn search executed at leaf was Exact since filteredIds = 2 is less than or equal to K = 10 with spaceType = L2, vectorDataType = FLOAT, queryVector = [2.0, 4.0, 3.0]",
"details" : [ ]
}
]
}
}
]
}
}
방사형 검색 예제
GET my-knn-index/_search?explain=true
{
"query": {
"knn": {
"my_vector": {
"vector": [7.1, 8.3],
"max_distance": 2
}
}
}
}
{
"took" : 376529,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 2,
"relation" : "eq"
},
"max_score" : 0.98039204,
"hits" : [
{
"_shard" : "[knn-index-test][0]",
"_node" : "c9b4aPe4QGO8eOtb8P5D3g",
"_index" : "knn-index-test",
"_id" : "1",
"_score" : 0.98039204,
"_source" : {
"my_vector" : [
7.0,
8.2
],
"price" : 4.4
},
"_explanation" : {
"value" : 0.98039204,
"description" : "the type of knn search executed was Radial with the radius of 2.0",
"details" : [
{
"value" : 0.98039204,
"description" : "the type of knn search executed at leaf was Approximate-NN with vectorDataType = FLOAT, spaceType = l2 where score is computed as `1 / (1 + rawScore)` from:",
"details" : [
{
"value" : 0.020000057,
"description" : "rawScore, returned from FAISS library",
"details" : [ ]
}
]
}
]
}
},
{
"_shard" : "[knn-index-test][0]",
"_node" : "c9b4aPe4QGO8eOtb8P5D3g",
"_index" : "knn-index-test",
"_id" : "3",
"_score" : 0.9615384,
"_source" : {
"my_vector" : [
7.3,
8.3
],
"price" : 19.1
},
"_explanation" : {
"value" : 0.9615384,
"description" : "the type of knn search executed was Radial with the radius of 2.0",
"details" : [
{
"value" : 0.9615384,
"description" : "the type of knn search executed at leaf was Approximate-NN with vectorDataType = FLOAT, spaceType = l2 where score is computed as `1 / (1 + rawScore)` from:",
"details" : [
{
"value" : 0.040000115,
"description" : "rawScore, returned from FAISS library",
"details" : [ ]
}
]
}
]
}
}
]
}
}
term 쿼리와 함께 사용하는 k-NN 검색 예제
GET my-knn-index/_search?explain=true
{
"query": {
"bool": {
"should": [
{
"knn": {
"my_vector2": { // vector field name
"vector": [2, 3, 5, 6],
"k": 2
}
}
},
{
"term": {
"price": "4.4"
}
}
]
}
}
}
{
"took" : 51,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 2,
"relation" : "eq"
},
"max_score" : 84.7,
"hits" : [
{
"_shard" : "[my-knn-index-1][0]",
"_node" : "c9b4aPe4QGO8eOtb8P5D3g",
"_index" : "my-knn-index-1",
"_id" : "7",
"_score" : 84.7,
"_source" : {
"my_vector2" : [
2.5,
3.5,
5.6,
6.7
],
"price" : 5.5
},
"_explanation" : {
"value" : 84.7,
"description" : "sum of:",
"details" : [
{
"value" : 84.7,
"description" : "the type of knn search executed was Approximate-NN",
"details" : [
{
"value" : 84.7,
"description" : "the type of knn search executed at leaf was Approximate-NN with vectorDataType = FLOAT, spaceType = innerproduct where score is computed as `-rawScore + 1` from:",
"details" : [
{
"value" : -83.7,
"description" : "rawScore, returned from FAISS library",
"details" : [ ]
}
]
}
]
}
]
}
},
{
"_shard" : "[my-knn-index-1][0]",
"_node" : "c9b4aPe4QGO8eOtb8P5D3g",
"_index" : "my-knn-index-1",
"_id" : "8",
"_score" : 83.2,
"_source" : {
"my_vector2" : [
4.5,
5.5,
6.7,
3.7
],
"price" : 4.4
},
"_explanation" : {
"value" : 83.2,
"description" : "sum of:",
"details" : [
{
"value" : 82.2,
"description" : "the type of knn search executed was Approximate-NN",
"details" : [
{
"value" : 82.2,
"description" : "the type of knn search executed at leaf was Approximate-NN with vectorDataType = FLOAT, spaceType = innerproduct where score is computed as `-rawScore + 1` from:",
"details" : [
{
"value" : -81.2,
"description" : "rawScore, returned from FAISS library",
"details" : [ ]
}
]
}
]
},
{
"value" : 1.0,
"description" : "price:[1082969293 TO 1082969293]",
"details" : [ ]
}
]
}
}
]
}
}
응답 본문 필드 (Response body fields)
Field Description
explanation: explanation 객체는 다음 필드를 포함해요. - value: 계산 결과를 담아요. - description: 어떤 종류의 계산이 수행되었는지 설명해요. 점수 정규화의 경우 description 필드에는 사용된 정규화 또는 결합 기법과 해당 점수가 포함돼요. - details: 수행된 하위 계산을 보여줘요.