문자별 분할 - 텍스트 스플리터 통합 가이드

문자별 분할 - 텍스트 스플리터 통합 가이드

문자 기반 분할은 텍스트 분할에서 가장 단순한 접근 방식이에요. 지정된 문자 시퀀스(기본값: "\n\n")를 사용해 텍스트를 나누고, 청크 길이는 문자 수로 측정합니다.

핵심 사항:

  1. 텍스트 분할 방법: 주어진 문자 구분자로 분할.
  2. 청크 크기 측정 방법: 문자 수로 측정.

다음 중에서 선택할 수 있습니다:

  • .splitText: 일반 문자열 청크를 반환.
  • .createDocuments: LangChain Document 객체를 반환하며, 다운스트림 작업을 위해 메타데이터를 보존해야 할 때 유용.
npm install @langchain/textsplitters

pnpm install @langchain/textsplitters

yarn add @langchain/textsplitters

bun add @langchain/textsplitters

import { CharacterTextSplitter } from "@langchain/textsplitters";
import { readFileSync } from "fs";

// Example: read a long document
const stateOfTheUnion = readFileSync("state_of_the_union.txt", "utf8");

const splitter = new CharacterTextSplitter({
    separator: "\n\n",
    chunkSize: 1000,
    chunkOverlap: 200,
});
const texts = splitter.createDocuments([{ pageContent: stateOfTheUnion }]);
console.log(texts[0]);

Document {
    pageContent: 'Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans.  \n\nLast year COVID-19 kept us apart. This year we are finally together again. \n\nTonight, we meet as Democrats Republicans and Independents. But most importantly as Americans. \n\nWith a duty to one another to the American people to the Constitution. \n\nAnd with an unwavering resolve that freedom will always triumph over tyranny. \n\nSix days ago, Russia’s Vladimir Putin sought to shake the foundations of the free world thinking he could make it bend to his menacing ways. But he badly miscalculated. \n\nHe thought he could roll into Ukraine and the world would roll over. Instead he met a wall of strength he never imagined. \n\nHe met the Ukrainian people. \n\nFrom President Zelenskyy to every Ukrainian, their fearlessness, their courage, their determination, inspires the world.'
}

.createDocuments를 사용해 각 문서와 연관된 메타데이터를 출력 청크로 전파하세요:

const metadatas = [{"document": 1}, {"document": 2}]
const documents = splitter.createDocuments(
    [{ pageContent: stateOfTheUnion }, { pageContent: stateOfTheUnion }],
    { metadatas: metadatas }
);
console.log(documents[0]);

Document {
    pageContent: 'Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans.  \n\nLast year COVID-19 kept us apart. This year we are finally together again. \n\nTonight, we meet as Democrats Republicans and Independents. But most importantly as Americans. \n\nWith a duty to one another to the American people to the Constitution. \n\nAnd with an unwavering resolve that freedom will always triumph over tyranny. \n\nSix days ago, Russia’s Vladimir Putin sought to shake the foundations of the free world thinking he could make it bend to his menacing ways. But he badly miscalculated. \n\nHe thought he could roll into Ukraine and the world would roll over. Instead he met a wall of strength he never imagined. \n\nHe met the Ukrainian people. \n\nFrom President Zelenskyy to every Ukrainian, their fearlessness, their courage, their determination, inspires the world.',
    metadata: {'document': 1}
}

.splitText를 사용해 문자열 콘텐츠를 직접 얻으세요:

splitter.splitText(stateOfTheUnion)[0]

'Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans.  \n\nLast year COVID-19 kept us apart. This year we are finally together again. \n\nTonight, we meet as Democrats Republicans and Independents. But most importantly as Americans. \n\nWith a duty to one another to the American people to the Constitution. \n\nAnd with an unwavering resolve that freedom will always triumph over tyranny. \n\nSix days ago, Russia’s Vladimir Putin sought to shake the foundations of the free world thinking he could make it bend to his menacing ways. But he badly miscalculated. \n\nHe thought he could roll into Ukraine and the world would roll over. Instead he met a wall of strength he never imagined. \n\nHe met the Ukrainian people. \n\nFrom President Zelenskyy to every Ukrainian, their fearlessness, their courage, their determination, inspires the world.'

출처: 문서

더 알아보기 (Learn more)