Neural sparse ANN 쿼리 explain
Neural sparse ANN 쿼리 explain
neural sparse 근사 최근접 이웃(ANN) 쿼리에서 점수가 어떻게 계산되는지 이해하려면 explain 파라미터를 사용해요. 이 글에서는 explain의 사용 방법과 점수 산정 과정을 살펴봐요.
출처: 문서
본문
Introduced 3.5 neural sparse 근사 최근접 이웃(ANN) 쿼리에서 점수가 어떻게 계산되는지 이해하려면 explain 파라미터를 사용하면 돼요. 이 파라미터를 활성화하면 각 검색 결과에 대한 점수 산정 과정의 상세 정보를 볼 수 있어요. 여기에는 쿼리 토큰 가지치기(pruning), 양자화된 내적(dot product) 계산, 양자화 재조정(rescaling), 필터 적용이 모두 포함돼요. 이렇게 전반적인 통찰을 얻으면 neural sparse ANN 쿼리 결과를 더 쉽게 이해하고 최적화할 수 있어요. explain에 대한 자세한 내용은 Explain API를 참고하세요. explain은 리소스와 시간 측면 모두에서 비용이 많이 드는 작업이에요. 운영(production) 클러스터에서는 문제 해결(troubleshooting) 목적으로 가급적 드물게 사용하는 것을 권장해요. neural sparse ANN 쿼리를 실행할 때 URL에 explain 파라미터를 제공할 수 있어요. 문법은 다음과 같아요.
GET {index}/_search?explain=true
POST {index}/_search?explain=true
explain 파라미터는 다음 유형의 neural sparse ANN 검색에서 동작해요.
- 기본 neural sparse ANN 검색
- 필터가 적용된 neural sparse ANN 검색 explain 파라미터는 쿼리 파라미터로 제공할 수 있어요.
GET my-sparse-index/_search?explain=true
{
"query": {
"neural_sparse": {
"sparse_embedding": {
"query_tokens": {
"7001": 6.25,
"3509": 5.57
},
"method_parameters": {
"k": 5,
"top_n": 6
}
}
}
}
}
또는 요청 본문(request body)에 explain 파라미터를 넣을 수도 있어요.
GET my-sparse-index/_search
{
"query": {
"neural_sparse": {
"sparse_embedding": {
"query_tokens": {
"7001": 6.25,
"3509": 5.57
},
"method_parameters": {
"k": 5,
"top_n": 3
}
}
}
},
"explain": true
}
예제: 기본 neural sparse ANN 검색
GET my-sparse-index/_search?explain=true
{
"query": {
"neural_sparse": {
"sparse_embedding": {
"query_tokens": {
"13723": 0.75,
"9266": 0.61,
"2078": 0.35,
"2365": 0.41
},
"method_parameters": {
"k": 10,
"top_n": 6
}
}
}
}
}
{
"took": 15,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 1,
"relation": "eq"
},
"max_score": 16.059792,
"hits": [
{
"_shard": "[my-sparse-index][0]",
"_node": "iId6ipt-SCWA-7vHLh981Q",
"_index": "my-sparse-index",
"_id": "1",
"_score": 16.059792,
"_source": {
"sparse_embedding": {
"13723": 3.16,
"9266": 2.85,
"2078": 1.09,
"2365": 0.22
}
},
"_explanation": {
"value": 16.059792,
"description": "sparse_ann score for doc 94830 in field 'sparse_embedding'",
"details": [
{
"value": 6,
"description": "query token pruning: kept top 6 of 33 tokens",
"details": []
},
{
"value": 21756,
"description": "raw dot product score (quantized): 21756",
"details": [
{
"value": 9494,
"description": "token '13723' contribution: query_weight=47 * doc_weight=202",
"details": []
},
{
"value": 7098,
"description": "token '9266' contribution: query_weight=39 * doc_weight=182",
"details": []
},
{
"value": 1540,
"description": "token '2078' contribution: query_weight=22 * doc_weight=70",
"details": []
},
{
"value": 364,
"description": "token '2365' contribution: query_weight=26 * doc_weight=14",
"details": []
}
]
},
{
"value": 0.0007381776,
"description": "quantization rescaling: 1.0000 * 3.00 * 16.00 / 255 / 255 = 0.000738",
"details": [
{
"value": 1,
"description": "original boost: 1.0000",
"details": []
},
{
"value": 3,
"description": "ceiling_ingest (quantization parameter): 3.00",
"details": []
},
{
"value": 16,
"description": "ceiling_search (quantization parameter): 16.00",
"details": []
},
{
"value": 255,
"description": "MAX_UNSIGNED_BYTE_VALUE: 255",
"details": []
}
]
}
]
}
}
]
}
}
예제: 필터가 적용된 neural sparse ANN 검색
GET hotels-index/_search?explain=true
{
"query": {
"neural_sparse": {
"name_embedding": {
"query_tokens": {
"7001": 6.25,
"3509": 5.57
},
"method_parameters": {
"k": 5,
"filter": {
"range": {
"rating": {
"gte": 8,
"lte": 10
}
}
}
}
}
}
}
}
{
"took": 8,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 4,
"relation": "eq"
},
"max_score": 70.55404,
"hits": [
{
"_shard": "[hotels-index][0]",
"_node": "iId6ipt-SCWA-7vHLh981Q",
"_index": "hotels-index",
"_id": "8",
"_score": 70.55404,
"_source": {
"parking": true,
"name": "Crystal Beach Resort",
"rating": 9,
"name_embedding": {
"3509": 5.5722017,
"6121": 6.5081306,
"7001": 6.25483
}
},
"_explanation": {
"value": 70.55404,
"description": "sparse_ann score for doc 7 in field 'name_embedding'",
"details": [
{
"value": 2,
"description": "query token pruning: kept all 2 tokens (no pruning occurred)",
"details": []
},
{
"value": 17921,
"description": "raw dot product score (quantized): 17921",
"details": [
{
"value": 10000,
"description": "token '7001' contribution: query_weight=100 * doc_weight=100",
"details": []
},
{
"value": 7921,
"description": "token '3509' contribution: query_weight=89 * doc_weight=89",
"details": []
}
]
},
{
"value": 0.003936948,
"description": "quantization rescaling: 1.0000 * 16.00 * 16.00 / 255 / 255 = 0.003937",
"details": [
{
"value": 1,
"description": "original boost: 1.0000",
"details": []
},
{
"value": 16,
"description": "ceiling_ingest (quantization parameter): 16.00",
"details": []
},
{
"value": 16,
"description": "ceiling_search (quantization parameter): 16.00",
"details": []
},
{
"value": 255,
"description": "MAX_UNSIGNED_BYTE_VALUE: 255",
"details": []
}
]
},
{
"value": 1,
"description": "document passed filter with exact search mode (filter matched 4 documents
## Response body fields
The following table describes the fields in the explanation response.
Field Description
explanation The explanation object contains the following fields:
- value: Contains the calculation result.
- description: Explains what type of calculation was performed.
- details: Shows any subcalculations performed.
### Explanation components
The details array in the explanation contains the following components based on the neural sparse ANN scoring process.
Component Description
Query token pruning Shows the number of query tokens retained after pruning based on the top_n parameter. If no pruning occurs, indicates that all tokens were kept.
Raw dot product score The quantized dot product score before rescaling. Contains nested details showing each token’s contribution as query_weight * doc_weight.
Quantization rescaling Explains how the raw quantized score is converted to the final float score using the formula: boost * ceiling_ingest * ceiling_search / 255 / 255. Contains details about each parameter used in the calculation.
Filter explanation (When filters are applied) Shows filter criteria and search mode. Indicates whether exact search mode was used when the number of filtered documents is fewer than or equal to k.
### Quantization parameters
Neural sparse ANN search uses unsigned byte quantization to reduce memory usage and improve search performance. The quantization rescaling section includes the following parameters.
Parameter Description
original boost The boost value applied to the query. Default is 1.0.
ceiling_ingest The quantization ceiling parameter used during document ingestion.
ceiling_search The quantization ceiling parameter used during search.
MAX_UNSIGNED_BYTE_VALUE The maximum value for unsigned byte quantization (255).
During ingestion, float token weights are converted to unsigned bytes (0–255) by dividing each weight by ceiling_ingest and scaling to the byte range. Similarly, during search, query token weights are quantized using ceiling_search. The raw dot product is computed using these quantized byte values. To recover the approximate original score, the result is rescaled using the formula final_score = raw_quantized_score * boost * ceiling_ingest * ceiling_search / 255 / 255. The ceiling parameters determine the maximum weight value that can be represented without clipping—weights exceeding the ceiling are capped at 255.
## Next steps
For more information about neural sparse ANN search, see Neural sparse ANN search.
For more information about the Explain API, see Explain API.
For information about filtering in neural sparse ANN search, see Filtering in neural sparse ANN search.
Example: Basic neural sparse ANN search
Example: Neural sparse ANN search with filter
Response body fieldsExplanation components
Quantization parameters
Next steps
## 더 알아보기 (Learn more)
- [OpenSearch 공식 문서](https://docs.opensearch.org/latest/query-dsl/specialized/neural-sparse/neural-sparse-explain/)