비동기 LLM 스트리밍

비동기 LLM 스트리밍 (Async LLM Streaming)

vLLM의 AsyncLLM(V1 엔진)으로 오프라인 추론에서 토큰 단위 스트리밍을 수행하는 최소 예제입니다. DELTA 모드 스트리밍을 사용해 생성되는 새 토큰만 받아 출력하는 핵심 패턴을 보여줍니다.

출처: 문서

본문

소스: https://github.com/vllm-project/vllm/blob/main/examples/deployment/async_llm_streaming.py

이 스크립트는 AsyncLLM(V1 엔진)으로 오프라인 추론에서 토큰 단위 스트리밍을 하는 예제입니다. 생성되는 새 토큰을 받는 DELTA 모드 스트리밍을 시연합니다.

# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Simple example demonstrating streaming offline inference with AsyncLLM (V1 engine).

This script shows the core functionality of vLLM's AsyncLLM engine for streaming
token-by-token output in offline inference scenarios. It demonstrates DELTA mode
streaming where you receive new tokens as they are generated.

Usage:
    python examples/deployment/async_llm_streaming.py
"""

import asyncio

from vllm import SamplingParams
from vllm.engine.arg_utils import AsyncEngineArgs
from vllm.sampling_params import RequestOutputKind
from vllm.v1.engine.async_llm import AsyncLLM

async def stream_response(engine: AsyncLLM, prompt: str, request_id: str) -> None:
    """Stream response from AsyncLLM and display tokens as they arrive.

    This function demonstrates the core streaming pattern:
    1. Create SamplingParams with DELTA output kind
    2. Call engine.generate() and iterate over the async generator
    3. Print new tokens as they arrive
    4. Handle the finished flag to know when generation is complete
    """
    print(f"\n🚀 Prompt: {prompt!r}")
    print("💬 Response: ", end="", flush=True)

    # Configure sampling parameters for streaming
    sampling_params = SamplingParams(
        max_tokens=100,
        temperature=0.8,
        top_p=0.95,
        seed=42,  # For reproducible results
        output_kind=RequestOutputKind.DELTA,  # Get only new tokens each iteration
    )

    try:
        # Stream tokens from AsyncLLM
        async for output in engine.generate(
            request_id=request_id, prompt=prompt, sampling_params=sampling_params
        ):
            # Process each completion in the output
            for completion in output.outputs:
                # In DELTA mode, we get only new tokens generated since last iteration
                new_text = completion.text
                if new_text:
                    print(new_text, end="", flush=True)

            # Check if generation is finished
            if output.finished:
                print("\n✅ Generation complete!")
                break

    except Exception as e:
        print(f"\n❌ Error during streaming: {e}")
        raise

async def main():
    print("🔧 Initializing AsyncLLM...")

    # Create AsyncLLM engine with simple configuration
    engine_args = AsyncEngineArgs(
        model="meta-llama/Llama-3.2-1B-Instruct",
        enforce_eager=True,  # Faster startup for examples
    )
    engine = AsyncLLM.from_engine_args(engine_args)

    try:
        # Example prompts to demonstrate streaming
        prompts = [
            "The future of artificial intelligence is",
            "In a galaxy far, far away",
            "The key to happiness is",
        ]

        print(f"🎯 Running {len(prompts)} streaming examples...")

        # Process each prompt
        for i, prompt in enumerate(prompts, 1):
            print(f"\n{'=' * 60}")
            print(f"Example {i}/{len(prompts)}")
            print(f"{'=' * 60}")

            request_id = f"stream-example-{i}"
            await stream_response(engine, prompt, request_id)

            # Brief pause between examples
            if i < len(prompts):
                await asyncio.sleep(0.5)

        print("\n🎉 All streaming examples completed!")

    finally:
        # Always clean up the engine
        print("🔧 Shutting down engine...")
        engine.shutdown()

if __name__ == "__main__":
    try:
        asyncio.run(main())
    except KeyboardInterrupt:
        print("\n🛑 Interrupted by user")

핵심 스트리밍 패턴은 다음과 같습니다.

1. SamplingParams에서 output_kind=RequestOutputKind.DELTA를 설정해 각 iteration마다 새 토큰만 받습니다.
2. engine.generate(request_id=..., prompt=..., sampling_params=...)를 호출해 async generator를 순회합니다.
3. 도착하는 새 토큰(completion.text)을 출력합니다.
4. output.finished 플래그로 생성 완료를 감지합니다.

AsyncLLM은 반드시 engine.shutdown()으로 정리해야 합니다. 유휴 모델 예시에서는 enforce_eager=True를 써서 시작을 빠르게 합니다.

더 알아보기 (Learn more)

• Online Serving — vllm serve 서버
• SamplingParams의 output_kind — DELTA/ACCUMULATE 등 출력 완료 방식