코드 텍스트 분할

코드 텍스트 분할 (CodeSplitter)

코드를 RAG 파이프라인에서 다룰 때는 일반 텍스트와 다른 분할이 필요해요. RecursiveCharacterTextSplitter는 프로그래밍 언어별로 유용한 구분자 목록을 미리 내장하고 있어서, 특정 언어에 맞춰 코드를 나눌 수 있어요. 이 페이지에서는 지원 언어 목록과 각 언어에서의 실제 사용 예시를 함께 볼게요.

출처: 공식문서

지원 언어

지원 언어는 langchain_text_splitters.Language 열거형에 저장돼요. 여기에는 다음과 같은 언어들이 포함돼요.

"cpp",
"go",
"java",
"kotlin",
"js",
"ts",
"php",
"proto",
"python",
"rst",
"ruby",
"rust",
"scala",
"swift",
"markdown",
"latex",
"html",
"sol",
"csharp",
"cobol",
"c",
"lua",
"perl",
"haskell"

특정 언어의 구분자 목록을 보려면 열거형 값을 get_separators_for_language에 전달하면 돼요.

RecursiveCharacterTextSplitter.get_separators_for_language

특정 언어에 맞춰진 분할기를 만들려면 열거형 값을 from_language에 전달하면 돼요.

RecursiveCharacterTextSplitter.from_language

설치

pip install -qU langchain-text-splitters

지원 언어 전체 목록 확인

from langchain_text_splitters import (
    Language,
    RecursiveCharacterTextSplitter,
)

Language 열거형의 전체 값을 보려면:

[e.value for e in Language]

['cpp',
 'go',
 'java',
 'kotlin',
 'js',
 'ts',
 'php',
 'proto',
 'python',
 'rst',
 'ruby',
 'rust',
 'scala',
 'swift',
 'markdown',
 'latex',
 'html',
 'sol',
 'csharp',
 'cobol',
 'c',
 'lua',
 'perl',
 'haskell',
 'elixir',
 'powershell',
 'visualbasic6']

특정 언어에 쓰이는 구분자도 확인할 수 있어요.

RecursiveCharacterTextSplitter.get_separators_for_language(Language.PYTHON)

['\nclass ', '\ndef ', '\n\tdef ', '\n\n', '\n', ' ', '']

Python

Python 분할기 예시예요. from_language에 Language.PYTHON을 전달하고 청크 크기를 지정하면, 코드의 의미 단위(함수 정의 등)를 기준으로 나눠져요.

PYTHON_CODE = """
def hello_world():
    print("Hello, World!")

# Call the function
hello_world()
"""
python_splitter = RecursiveCharacterTextSplitter.from_language(
    language=Language.PYTHON, chunk_size=50, chunk_overlap=0
)
python_docs = python_splitter.create_documents([PYTHON_CODE])
python_docs

[Document(metadata={}, page_content='def hello_world():\n    print("Hello, World!")'),
 Document(metadata={}, page_content='# Call the function\nhello_world()')]

JS

JS_CODE = """
function helloWorld() {
  console.log("Hello, World!");
}

// Call the function
helloWorld();
"""

js_splitter = RecursiveCharacterTextSplitter.from_language(
    language=Language.JS, chunk_size=60, chunk_overlap=0
)
js_docs = js_splitter.create_documents([JS_CODE])
js_docs

[Document(metadata={}, page_content='function helloWorld() {\n  console.log("Hello, World!");\n}'),
 Document(metadata={}, page_content='// Call the function\nhelloWorld();')]

TS

TS_CODE = """
function helloWorld(): void {
  console.log("Hello, World!");
}

// Call the function
helloWorld();
"""

ts_splitter = RecursiveCharacterTextSplitter.from_language(
    language=Language.TS, chunk_size=60, chunk_overlap=0
)
ts_docs = ts_splitter.create_documents([TS_CODE])
ts_docs

[Document(metadata={}, page_content='function helloWorld(): void {'),
 Document(metadata={}, page_content='console.log("Hello, World!");\n}'),
 Document(metadata={}, page_content='// Call the function\nhelloWorld();')]

Markdown

마크다운은 제목 단위로 나뉘는 게 자연스러워요. 주의할 점은 코드 블록이 중간에 잘리지 않도록 헤딩 기준으로 나눈다는 거예요.

markdown_text = """
# 🦜️🔗 LangChain

⚡ Building applications with LLMs through composability ⚡

## What is LangChain?

# Hopefully this code block isn't split
LangChain is a framework for...

As an open-source project in a rapidly developing field, we are extremely open to contributions.

"""

md_splitter = RecursiveCharacterTextSplitter.from_language(
    language=Language.MARKDOWN, chunk_size=60, chunk_overlap=0
)
md_docs = md_splitter.create_documents([markdown_text])
md_docs

[Document(metadata={}, page_content='# 🦜️🔗 LangChain'),
 Document(metadata={}, page_content='⚡ Building applications with LLMs through composability ⚡'),
 Document(metadata={}, page_content='## What is LangChain?'),
 Document(metadata={}, page_content="# Hopefully this code block isn't split"),
 Document(metadata={}, page_content='LangChain is a framework for...'),
 Document(metadata={}, page_content='As an open-source project in a rapidly developing field, we'),
 Document(metadata={}, page_content='are extremely open to contributions.')]

LaTeX

latex_text = """
\documentclass{article}

\begin{document}

\maketitle

\section{Introduction}
Large language models (LLMs) are a type of machine learning model that can be trained on vast amounts of text data to generate human-like language. In recent years, LLMs have made significant advances in a variety of natural language processing tasks, including language translation, text generation, and sentiment analysis.

\subsection{History of LLMs}
The earliest LLMs were developed in the 1980s and 1990s, but they were limited by the amount of data that could be processed and the computational power available at the time. In the past decade, however, advances in hardware and software have made it possible to train LLMs on massive datasets, leading to significant improvements in performance.

\subsection{Applications of LLMs}
LLMs have many applications in industry, including chatbots, content creation, and virtual assistants. They can also be used in academia for research in linguistics, psychology, and computational linguistics.

\end{document}

"""

latex_splitter = RecursiveCharacterTextSplitter.from_language(
    language=Language.MARKDOWN, chunk_size=60, chunk_overlap=0
)
latex_docs = latex_splitter.create_documents([latex_text])
latex_docs

[Document(metadata={}, page_content='\\documentclass{article}\n\n\x08egin{document}\n\n\\maketitle'),
 Document(metadata={}, page_content='\\section{Introduction}'),
 Document(metadata={}, page_content='Large language models (LLMs) are a type of machine learning'),
 Document(metadata={}, page_content='model that can be trained on vast amounts of text data to'),
 Document(metadata={}, page_content='generate human-like language. In recent years, LLMs have'),
 Document(metadata={}, page_content='made significant advances in a variety of natural language'),
 Document(metadata={}, page_content='processing tasks, including language translation, text'),
 Document(metadata={}, page_content='generation, and sentiment analysis.'),
 Document(metadata={}, page_content='\\subsection{History of LLMs}'),
 Document(metadata={}, page_content='The earliest LLMs were developed in the 1980s and 1990s,'),
 Document(metadata={}, page_content='but they were limited by the amount of data that could be'),
 Document(metadata={}, page_content='processed and the computational power available at the'),
 Document(metadata={}, page_content='time. In the past decade, however, advances in hardware and'),
 Document(metadata={}, page_content='software have made it possible to train LLMs on massive'),
 Document(metadata={}, page_content='datasets, leading to significant improvements in'),
 Document(metadata={}, page_content='performance.'),
 Document(metadata={}, page_content='\\subsection{Applications of LLMs}'),
 Document(metadata={}, page_content='LLMs have many applications in industry, including'),
 Document(metadata={}, page_content='chatbots, content creation, and virtual assistants. They'),
 Document(metadata={}, page_content='can also be used in academia for research in linguistics,'),
 Document(metadata={}, page_content='psychology, and computational linguistics.'),
 Document(metadata={}, page_content='\\end{document}')]

HTML

html_text = """
<!DOCTYPE html>
<html>
    <head>
        <title>🦜️🔗 LangChain</title>
        <style>
            body {
                font-family: Arial, sans-serif;
            }
            h1 {
                color: darkblue;
            }
        </style>
    </head>
    <body>
        <div>
            <h1>🦜️🔗 LangChain</h1>
            <p>⚡ Building applications with LLMs through composability ⚡</p>
        </div>
        <div>
            As an open-source project in a rapidly developing field, we are extremely open to contributions.
        </div>
    </body>
</html>
"""

html_splitter = RecursiveCharacterTextSplitter.from_language(
    language=Language.HTML, chunk_size=60, chunk_overlap=0
)
html_docs = html_splitter.create_documents([html_text])
html_docs

[Document(metadata={}, page_content='<!DOCTYPE html>\n<html>'),
 Document(metadata={}, page_content='<head>\n        <title>🦜️🔗 LangChain</title>'),
 Document(metadata={}, page_content='<style>\n            body {\n                font-family: Aria'),
 Document(metadata={}, page_content='l, sans-serif;\n            }\n            h1 {'),
 Document(metadata={}, page_content='color: darkblue;\n            }\n        </style>\n    </head'),
 Document(metadata={}, page_content='>'),
 Document(metadata={}, page_content='<body>'),
 Document(metadata={}, page_content='<div>\n            <h1>🦜️🔗 LangChain</h1>'),
 Document(metadata={}, page_content='<p>⚡ Building applications with LLMs through composability ⚡'),
 Document(metadata={}, page_content='</p>\n        </div>'),
 Document(metadata={}, page_content='<div>\n            As an open-source project in a rapidly dev'),
 Document(metadata={}, page_content='eloping field, we are extremely open to contributions.'),
 Document(metadata={}, page_content='</div>\n    </body>\n</html>')]

Solidity

SOL_CODE = """
pragma solidity ^0.8.20;
contract HelloWorld {
   function add(uint a, uint b) pure public returns(uint) {
       return a + b;
   }
}
"""

sol_splitter = RecursiveCharacterTextSplitter.from_language(
    language=Language.SOL, chunk_size=128, chunk_overlap=0
)
sol_docs = sol_splitter.create_documents([SOL_CODE])
sol_docs

[Document(metadata={}, page_content='pragma solidity ^0.8.20;'),
 Document(metadata={}, page_content='contract HelloWorld {\n   function add(uint a, uint b) pure public returns(uint) {\n       return a + b;\n   }\n}')]

C#

C_CODE = """
using System;
class Program
{
    static void Main()
    {
        int age = 30; // Change the age value as needed

        // Categorize the age without any console output
        if (age < 18)
        {
            // Age is under 18
        }
        else if (age >= 18 && age < 65)
        {
            // Age is an adult
        }
        else
        {
            // Age is a senior citizen
        }
    }
}
"""
c_splitter = RecursiveCharacterTextSplitter.from_language(
    language=Language.CSHARP, chunk_size=128, chunk_overlap=0
)
c_docs = c_splitter.create_documents([C_CODE])
c_docs

[Document(metadata={}, page_content='using System;'),
 Document(metadata={}, page_content='class Program\n{\n    static void Main()\n    {\n        int age = 30; // Change the age value as needed'),
 Document(metadata={}, page_content='// Categorize the age without any console output\n        if (age < 18)\n        {\n            // Age is under 18'),
 Document(metadata={}, page_content='}\n        else if (age >= 18 && age < 65)\n        {\n            // Age is an adult\n        }\n        else\n        {'),
 Document(metadata={}, page_content='// Age is a senior citizen\n        }\n    }\n}')]

기타 지원 언어

위에서 소개한 언어 외에도 Haskell, PHP, PowerShell, Visual Basic 6 같은 언어를 지원해요. 모두 RecursiveCharacterTextSplitter.from_language에 해당 언어 값을 넘겨 동일한 방식으로 사용할 수 있어요. 예를 들어 Haskell은:

HASKELL_CODE = """
main :: IO ()
main = do
    putStrLn "Hello, World!"
-- Some sample functions
add :: Int -> Int -> Int
add x y = x + y
"""
haskell_splitter = RecursiveCharacterTextSplitter.from_language(
    language=Language.HASKELL, chunk_size=50, chunk_overlap=0
)
haskell_docs = haskell_splitter.create_documents([HASKELL_CODE])
haskell_docs

[Document(metadata={}, page_content='main :: IO ()'),
 Document(metadata={}, page_content='main = do\n    putStrLn "Hello, World!"\n-- Some'),
 Document(metadata={}, page_content='sample functions\nadd :: Int -> Int -> Int\nadd x y'),
 Document(metadata={}, page_content='= x + y')]

더 알아보기 (Learn more)

• Text splitters 개요 — 분할기의 종류와 선택 기준
• 재귀 문자 텍스트 분할 — 코드 분할기의 기반이 되는 분할기
• 문자 단위 텍스트 분할 — 가장 단순한 분할 방식
• 토큰 단위 텍스트 분할 — 언어 모델 토큰 제한에 맞춘 분할