Semantic search using text chunking
출처: 문서
본문
텍스트 청킹을 사용한 의미 검색
이 튜토리얼에서는 OpenSearch 2.19 이상에서 텍스트 청킹을 사용하여 긴 문서에 대해 의미 검색을 실행하는 방법을 알려줘요.
이 튜토리얼에서는 다음 OpenSearch 구성 요소를 사용할 거예요:
your_ 접두사로 시작하는 플레이스홀더를 자신의 값으로 바꾸세요.
{: .note}
1단계: 임베딩 모델 생성
이 튜토리얼에서는 Amazon Bedrock Titan Text Embeddings 모델을 사용합니다.
Python을 사용하는 경우 opensearch-py-ml 클라이언트 CLI를 사용하여 Amazon Bedrock Titan 임베딩 커넥터를 생성하고 모델을 테스트할 수 있어요. CLI는 많은 구성 단계를 자동화하여 설정을 더 빠르게 하고 오류 가능성을 줄여줘요. CLI 사용에 대한 자세한 내용은 CLI 문서를 참조하세요.
{: .tip}
자체 관리 OpenSearch를 사용하는 경우 블루프린트를 사용하여 모델을 생성하세요.
Amazon OpenSearch Service를 사용하는 경우 이 Python 노트북을 사용하여 모델을 생성하세요. 또는 이 튜토리얼을 따라 수동으로 커넥터를 생성할 수 있어요.
1.1단계: 커넥터 생성
커넥터를 생성하려면 다음 요청을 전송하세요. 이 튜토리얼에서는 ML 추론 프로세서를 사용할 거라서 커넥터에 전처리 또는 후처리 함수를 지정할 필요가 없어요:
POST _plugins/_ml/connectors/_create
{
"name": "Amazon Bedrock Connector: embedding",
"description": "The connector to bedrock Titan embedding model",
"version": 1,
"protocol": "aws_sigv4",
"parameters": {
"region": "us-west-2",
"service_name": "bedrock",
"model": "amazon.titan-embed-text-v2:0",
"dimensions": 1024,
"normalize": true,
"embeddingTypes": ["float"]
},
"credential": {
"access_key": "your_aws_access_key",
"secret_key": "your_aws_secret_key",
"session_token": "your_aws_session_token"
},
"actions": [
{
"action_type": "predict",
"method": "POST",
"url": "https://bedrock-runtime.${parameters.region}.amazonaws.com/model/${parameters.model}/invoke",
"headers": {
"content-type": "application/json",
"x-amz-content-sha256": "required"
},
"request_body": "{ \"inputText\": \"${parameters.inputText}\", \"dimensions\": ${parameters.dimensions}, \"normalize\": ${parameters.normalize}, \"embeddingTypes\": ${parameters.embeddingTypes} }"
}
]
}
{% include copy-curl.html %}
응답에는 커넥터 ID가 포함돼요:
{
"connector_id": "vhR15JQBLopfJ2xsx9p5"
}
커넥터 ID를 기록해 두세요. 다음 단계에서 사용할 거예요.
1.2단계: 모델 등록
모델을 등록하려면 다음 요청을 전송하세요:
POST _plugins/_ml/models/_register?deploy=true
{
"name": "Bedrock embedding model",
"function_name": "remote",
"description": "Bedrock text embedding model v2",
"connector_id": "vhR15JQBLopfJ2xsx9p5"
}
{% include copy-curl.html %}
응답에는 모델 ID가 포함돼요:
{
"task_id": "xRR35JQBLopfJ2xsO9pU",
"status": "CREATED",
"model_id": "xhR35JQBLopfJ2xsO9pr"
}
모델 ID를 기록해 두세요. 다음 단계에서 사용할 거예요.
1.3단계: 모델 테스트
모델을 테스트하려면 다음 요청을 전송하세요:
POST /_plugins/_ml/models/xhR35JQBLopfJ2xsO9pr/_predict
{
"parameters": {
"inputText": "hello world"
}
}
{% include copy-curl.html %}
응답에는 모델이 생성한 임베딩이 포함돼요:
{
"inference_results": [
{
"output": [
{
"name": "response",
"dataAsMap": {
"embedding": [
-0.020442573353648186,...
],
"embeddingsByType": {
"float": [
-0.020442573353648186, ...
]
},
"inputTextTokenCount": 3.0
}
}
],
"status_code": 200
}
]
}
2단계: 수집 파이프라인 생성
많은 텍스트 임베딩 모델에는 입력 크기 제한이 있어요. Amazon Titan Text Embeddings V2 모델은 최대 8,192개의 텍스트 토큰을 지원해요. 긴 문서를 처리하려면 문서를 더 작은 청크로 분할하고 각 청크를 모델에 전송해야 해요. 텍스트 청킹 프로세서는 원본 문서를 더 작은 조각으로 분할하고, ML 추론 프로세서는 각 청크에 대한 임베딩을 생성해요. 두 프로세서를 모두 포함하는 수집 파이프라인을 생성하려면 다음 요청을 전송하세요:
PUT _ingest/pipeline/bedrock-text-embedding-pipeline
{
"description": "ingest reviews, generate embedding, and format chunks",
"processors": [
{
"text_chunking": {
"algorithm": {
"fixed_token_length": {
"token_limit": 100,
"overlap_rate": 0.2,
"tokenizer": "standard"
}
},
"field_map": {
"passage_text": "passage_chunk"
}
}
},
{
"foreach": {
"field": "passage_chunk",
"processor": {
"set": {
"field": "_ingest._value",
"value": {
"text": "{{_ingest._value}}"
}
}
}
}
},
{
"foreach": {
"field": "passage_chunk",
"processor": {
"ml_inference": {
"model_id": "xhR35JQBLopfJ2xsO9pr",
"input_map": [
{
"inputText": "_ingest._value.text"
}
],
"output_map": [
{
"_ingest._value.embedding": "embedding"
}
]
}
}
}
}
]
}
{% include copy-curl.html %}
파이프라인을 테스트하려면 다음 요청을 전송하세요:
POST _ingest/pipeline/bedrock-text-embedding-pipeline/_simulate
{
"docs": [
{
"_index": "testindex",
"_id": "1",
"_source":{
"passage_text": "Ingest pipelines\nAn ingest pipeline is a sequence of processors that are applied to documents as they are ingested into an index. Each processor in a pipeline performs a specific task, such as filtering, transforming, or enriching data.\n\nProcessors are customizable tasks that run in a sequential order as they appear in the request body. This order is important, as each processor depends on the output of the previous processor. The modified documents appear in your index after the processors are applied.\n\nOpenSearch ingest pipelines compared to OpenSearch Data Prepper\nOpenSeach ingest pipelines run within the OpenSearch cluster, whereas OpenSearch Data Prepper is an external component that runs on the OpenSearch cluster.\n\nOpenSearch ingest pipelines perform actions on indexes and are preferred for use cases involving pre-processing simple datasets, machine learning (ML) processors, and vector embedding processors. OpenSearch ingest pipelines are recommended for simple data pre-processing and small datasets.\n\nOpenSearch Data Prepper is recommended for any data processing tasks it supports, particularly when dealing with large datasets and complex data pre-processing requirements. It streamlines the process of transferring and fetching large datasets while providing robust capabilities for intricate data preparation and transformation operations. Refer to the OpenSearch Data Prepper documentation for more information.\n\nOpenSearch ingest pipelines can only be managed using Ingest API operations.\n\nPrerequisites\nThe following are prerequisites for using OpenSearch ingest pipelines:\n\nWhen using ingestion in a production environment, your cluster should contain at least one node with the node roles permission set to ingest. For information about setting up node roles within a cluster, see Cluster Formation.\nIf the OpenSearch Security plugin is enabled, you must have the cluster_manage_pipelines permission to manage ingest pipelines.\nDefine a pipeline\nA pipeline definition describes the sequence of an ingest pipeline and can be written in JSON format. An ingest pipeline consists of the following:\n\n{\n \"description\" : \"...\"\n \"processors\" : [...]\n}\nRequest body fields\nField\tRequired\tType\tDescription\nprocessors\tRequired\tArray of processor objects\tA component that performs a specific data processing task as the data is being ingested into OpenSearch.\ndescription\tOptional\tString\tA description of the ingest pipeline.\n"
}
}
]
}
{% include copy-curl.html %}
응답에는 청크로 분할되고 각 청크에 대한 임베딩이 포함된 처리된 문서가 표시돼요:
{
"docs": [
{
"doc": {
"_index": "testindex",
"_id": "1",
"_source": {
"passage_text": """Ingest pipelines
An ingest pipeline is a sequence of processors that are applied to documents as they are ingested into an index. Each processor in a pipeline performs a specific task, such as filtering, transforming, or enriching data.
Processors are customizable tasks that run in a sequential order as they appear in the request body. This order is important, as each processor depends on the output of the previous processor. The modified documents appear in your index after the processors are applied.
OpenSearch ingest pipelines compared to OpenSearch Data Prepper
OpenSeach ingest pipelines run within the OpenSearch cluster, whereas OpenSearch Data Prepper is an external component that runs on the OpenSearch cluster.
OpenSearch ingest pipelines perform actions on indexes and are preferred for use cases involving pre-processing simple datasets, machine learning (ML) processors, and vector embedding processors. OpenSearch ingest pipelines are recommended for simple data pre-processing and small datasets.
OpenSearch Data Prepper is recommended for any data processing tasks it supports, particularly when dealing with large datasets and complex data pre-processing requirements. It streamlines the process of transferring and fetching large datasets while providing robust capabilities for intricate data preparation and transformation operations. Refer to the OpenSearch Data Prepper documentation for more information.
OpenSearch ingest pipelines can only be managed using Ingest API operations.
Prerequisites
The following are prerequisites for using OpenSearch ingest pipelines:
When using ingestion in a production environment, your cluster should contain at least one node with the node roles permission set to ingest. For information about setting up node roles within a cluster, see Cluster Formation.
If the OpenSearch Security plugin is enabled, you must have the cluster_manage_pipelines permission to manage ingest pipelines.
Define a pipeline
A pipeline definition describes the sequence of an ingest pipeline and can be written in JSON format. An ingest pipeline consists of the following:
{
"description" : "..."
"processors" : [...]
}
Request body fields
Field Required Type Description
processors Required Array of processor objects A component that performs a specific data processing task as the data is being ingested into OpenSearch.
description Optional String A description of the ingest pipeline.
""",
"passage_chunk": [
{
"text": """Ingest pipelines\nAn ingest pipeline is a sequence of processors that are applied to documents as they are ingested into an index. Each processor in a pipeline performs a specific task, such as filtering, transforming, or enriching data.\n\nProcessors are customizable tasks that run in a sequential order as they appear in the request body. This order is important, as each processor depends on the output of the previous processor. The modified documents appear in your index after the processors are applied.\n\nOpenSearch ingest pipelines compared to OpenSearch Data Prepper\nOpenSeach ingest pipelines run within the OpenSearch cluster, whereas OpenSearch Data Prepper is an external component that runs on the OpenSearch cluster.\n\nOpenSearch ingest pipelines perform actions on indexes and are preferred for use cases involving pre-processing simple datasets, machine learning (ML) processors, and vector embedding processors. OpenSearch ingest pipelines are recommended for simple data pre-processing and small datasets.\n\nOpenSearch Data Prepper is recommended for any data processing tasks it supports, particularly when dealing with large datasets and complex data pre-processing requirements. It streamlines the process of transferring and fetching large datasets while providing robust capabilities for intricate data preparation and transformation operations. Refer to the OpenSearch """,
"embedding": [
0.04044651612639427,
...
]
},
{
"text": """tasks it supports, particularly when dealing with large datasets and complex data pre-processing requirements. It streamlines the process of transferring and fetching large datasets while providing robust capabilities for intricate data preparation and transformation operations. Refer to the OpenSearch Data Prepper documentation for more information.\n\nOpenSearch ingest pipelines can only be managed using Ingest API operations.\n\nPrerequisites\nThe following are prerequisites for using OpenSearch ingest pipelines:\n\nWhen using ingestion in a production environment, your cluster should contain at least one node with the node roles permission set to ingest. For information about setting up node roles within a cluster, see Cluster Formation.\nIf the OpenSearch Security plugin is enabled, you must have the cluster_manage_pipelines permission to manage ingest pipelines.\nDefine a pipeline\nA pipeline definition describes the sequence of an ingest pipeline and can be written in JSON format. An ingest pipeline consists of the following:\n\n{\n \"description\" : \"...\"\n \"processors\" : [...]\n}\nRequest body fields\nField\tRequired\tType\tDescription\nprocessors\tRequired\tArray of processor objects\tA component that performs a specific data processing task as the data is being ingested into OpenSearch.\ndescription\tOptional\tString\tA description of the ingest pipeline.\n""",
"embedding": [
0.02055041491985321,
...
]
}
]
},
"_ingest": {
"_value": null,
"timestamp": "2025-02-08T07:49:43.484543119Z"
}
}
}
]
}
3단계: 인덱스 생성 및 데이터 수집
벡터 인덱스를 생성하려면 다음 요청을 전송하세요:
PUT opensearch_docs
{
"settings": {
"index.knn": true,
"default_pipeline": "bedrock-text-embedding-pipeline"
},
"mappings": {
"properties": {
"passage_chunk": {
"type": "nested",
"properties": {
"text": {
"type": "text"
},
"embedding": {
"type": "knn_vector",
"dimension": 1024
}
}
},
"passage_text": {
"type": "text"
}
}
}
}
{% include copy-curl.html %}
인덱스에 테스트 데이터를 수집하세요:
POST _bulk
{"index": {"_index": "opensearch_docs"}}
{"passage_text": "Ingest pipelines\nAn ingest pipeline is a sequence of processors that are applied to documents as they are ingested into an index. Each processor in a pipeline performs a specific task, such as filtering, transforming, or enriching data.\n\nProcessors are customizable tasks that run in a sequential order as they appear in the request body. This order is important, as each processor depends on the output of the previous processor. The modified documents appear in your index after the processors are applied.\n\nOpenSearch ingest pipelines compared to OpenSearch Data Prepper\nOpenSeach ingest pipelines run within the OpenSearch cluster, whereas OpenSearch Data Prepper is an external component that runs on the OpenSearch cluster.\n\nOpenSearch ingest pipelines perform actions on indexes and are preferred for use cases involving pre-processing simple datasets, machine learning (ML) processors, and vector embedding processors. OpenSearch ingest pipelines are recommended for simple data pre-processing and small datasets.\n\nOpenSearch Data Prepper is recommended for any data processing tasks it supports, particularly when dealing with large datasets and complex data pre-processing requirements. It streamlines the process of transferring and fetching large datasets while providing robust capabilities for intricate data preparation and transformation operations. Refer to the OpenSearch Data Prepper documentation for more information.\n\nOpenSearch ingest pipelines can only be managed using Ingest API operations.\n\nPrerequisites\nThe following are prerequisites for using OpenSearch ingest pipelines:\n\nWhen using ingestion in a production environment, your cluster should contain at least one node with the node roles permission set to ingest. For information about setting up node roles within a cluster, see Cluster Formation.\nIf the OpenSearch Security plugin is enabled, you must have the cluster_manage_pipelines permission to manage ingest pipelines.\nDefine a pipeline\nA pipeline definition describes the sequence of an ingest pipeline and can be written in JSON format. An ingest pipeline consists of the following:\n\n{\n \"description\" : \"...\"\n \"processors\" : [...]\n}\nRequest body fields\nField\tRequired\tType\tDescription\nprocessors\tRequired\tArray of processor objects\tA component that performs a specific data processing task as the data is being ingested into OpenSearch.\ndescription\tOptional\tString\tA description of the ingest pipeline.\n"}
{"index": {"_index": "opensearch_docs"}}
{"passage_text": "Monitors\nProactively monitor your data in OpenSearch with features available in Alerting and Anomaly Detection. For example, you can pair Anomaly Detection with Alerting to ensure that you’re notified as soon as an anomaly is detected. You can do this by setting up a detector to automatically detect outliers in your streaming data and monitors to alert you through notifications when data exceeds certain thresholds.\n\nMonitor types\nThe Alerting plugin provides the following monitor types:\n\nper query: Runs a query and generates alert notifications based on the matching criteria. See Per query monitors for information about creating and using this monitor type.\nper bucket: Runs a query that evaluates trigger criteria based on aggregated values in the dataset. See Per bucket monitors for information about creating and using this monitor type.\nper cluster metrics: Runs API requests on the cluster to monitor its health. See Per cluster metrics monitors for information about creating and using this monitor type.\nper document: Runs a query (or multiple queries combined by a tag) that returns individual documents that match the alert notification trigger condition. See Per document monitors for information about creating and using this monitor type.\ncomposite monitor: Runs multiple monitors in a single workflow and generates a single alert based on multiple trigger conditions. See Composite monitors for information about creating and using this monitor type.\nThe maximum number of monitors you can create is 1,000. You can change the default maximum number of alerts for your cluster by updating the plugins.alerting.monitor.max_monitors setting using the cluster settings API."}
{"index": {"_index": "opensearch_docs"}}
{"passage_text": "Search pipelines\nYou can use search pipelines to build new or reuse existing result rerankers, query rewriters, and other components that operate on queries or results. Search pipelines make it easier for you to process search queries and search results within OpenSearch. Moving some of your application functionality into an OpenSearch search pipeline reduces the overall complexity of your application. As part of a search pipeline, you specify a list of processors that perform modular tasks. You can then easily add or reorder these processors to customize search results for your application.\n\nTerminology\nThe following is a list of search pipeline terminology:\n\nSearch request processor: A component that intercepts a search request (the query and the metadata passed in the request), performs an operation with or on the search request, and returns the search request.\nSearch response processor: A component that intercepts a search response and search request (the query, results, and metadata passed in the request), performs an operation with or on the search response, and returns the search response.\nSearch phase results processor: A component that runs between search phases at the coordinating node level. A search phase results processor intercepts the results retrieved from one search phase and transforms them before passing them to the next search phase.\nProcessor: Either a search request processor or a search response processor.\nSearch pipeline: An ordered list of processors that is integrated into OpenSearch. The pipeline intercepts a query, performs processing on the query, sends it to OpenSearch, intercepts the results, performs processing on the results, and returns them to the calling application, as shown in the following diagram.\n"}
{% include copy-curl.html %}
문서가 제대로 처리되었는지 확인하려면 인덱스를 검색하여 생성된 청크와 임베딩을 확인하세요:
GET opensearch_docs/_search
{% include copy-curl.html %}
4단계: ML 추론 프로세서를 사용한 검색
입력 텍스트를 임베딩으로 변환하는 ML 추론 프로세서가 포함된 검색 파이프라인을 생성하세요:
PUT _search/pipeline/bedrock_semantic_search_pipeline
{
"request_processors": [
{
"ml_inference": {
"model_id": "xhR35JQBLopfJ2xsO9pr",
"input_map": [
{
"inputText": "ext.ml_inference.params.text"
}
],
"output_map": [
{
"ext.ml_inference.params.vector": "embedding"
}
]
}
}
]
}
{% include copy-curl.html %}
의미 검색을 실행하려면 다음 템플릿 쿼리를 사용하세요:
GET opensearch_docs/_search?search_pipeline=bedrock_semantic_search_pipeline
{
"query": {
"template": {
"nested": {
"path": "passage_chunk",
"query": {
"knn": {
"passage_chunk.embedding": {
"vector": "${ext.ml_inference.params.vector}",
"k": 5
}
}
}
}
}
},
"ext": {
"ml_inference": {
"params": {
"text": "What's OpenSearch ingest pipeline"
}
}
},
"_source": {
"excludes": [
"passage_chunk"
]
},
"size": 1
}
{% include copy-curl.html %}
파이프라인은 inputText를 ext.ml_inference.params.text에 매핑해요. 입력 처리 중에 파이프라인은 검색 요청의 ext.ml_inference.params.text 경로에서 값을 검색해요. 이 예에서 이 경로의 값은 "What's OpenSearch ingest pipeline"이며, 이 값은 inputText 매개변수로 모델에 전달돼요.
검색 중에 검색 쿼리는 "vector": "${ext.ml_inference.params.vector}"를 참조해요. 이 벡터 값은 초기 검색 요청에 제공되지 않아요. 대신 ML 추론 프로세서가 Amazon Bedrock Titan Embeddings 모델을 호출하여 생성해요. 모델은 검색 텍스트에서 임베딩 벡터를 생성하고 벡터를 ext.ml_inference.params.vector에 저장해요. 그런 다음 OpenSearch는 이 생성된 벡터를 사용하여 유사한 문서를 찾아요:
{
"took": 398,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 3,
"relation": "eq"
},
"max_score": 0.78014797,
"hits": [
{
"_index": "opensearch_docs",
"_id": "rj2T5JQBg4dihuRifxJT",
"_score": 0.78014797,
"_source": {
"passage_text": """Ingest pipelines
An ingest pipeline is a sequence of processors that are applied to documents as they are ingested into an index. Each processor in a pipeline performs a specific task, such as filtering, transforming, or enriching data.
Processors are customizable tasks that run in a sequential order as they appear in the request body. This order is important, as each processor depends on the output of the previous processor. The modified documents appear in your index after the processors are applied.
OpenSearch ingest pipelines compared to OpenSearch Data Prepper
OpenSeach ingest pipelines run within the OpenSearch cluster, whereas OpenSearch Data Prepper is an external component that runs on the OpenSearch cluster.
OpenSearch ingest pipelines perform actions on indexes and are preferred for use cases involving pre-processing simple datasets, machine learning (ML) processors, and vector embedding processors. OpenSearch ingest pipelines are recommended for simple data pre-processing and small datasets.
OpenSearch Data Prepper is recommended for any data processing tasks it supports, particularly when dealing with large datasets and complex data pre-processing requirements. It streamlines the process of transferring and fetching large datasets while providing robust capabilities for intricate data preparation and transformation operations. Refer to the OpenSearch Data Prepper documentation for more information.
OpenSearch ingest pipelines can only be managed using Ingest API operations.
Prerequisites
The following are prerequisites for using OpenSearch ingest pipelines:
When using ingestion in a production environment, your cluster should contain at least one node with the node roles permission set to ingest. For information about setting up node roles within a cluster, see Cluster Formation.
If the OpenSearch Security plugin is enabled, you must have the cluster_manage_pipelines permission to manage ingest pipelines.
Define a pipeline
A pipeline definition describes the sequence of an ingest pipeline and can be written in JSON format. An ingest pipeline consists of the following:
{
"description" : "..."
"processors" : [...]
}
Request body fields
Field Required Type Description
processors Required Array of processor objects A component that performs a specific data processing task as the data is being ingested into OpenSearch.
description Optional String A description of the ingest pipeline.
"""
}
}
]
}
}