OpenAI 호환 서버 (OpenAI-Compatible Server)

OpenAI 호환 서버 (OpenAI-Compatible Server)

vLLM은 OpenAI의 Completions API와 Chat API를 비롯한 HTTP 서버를 제공해요. 이 기능 덕분에 모델을 서빙해서 HTTP 클라이언트로 바로 상호작용할 수 있죠. 별도로 프레임워크를 붙일 필요 없이, OpenAI SDK만 있어도 vLLM 서버를 호출할 수 있다는 점이 핵심이에요.

API 키 인증은 모든 엔드포인트를 보호하지 않아요

--api-key 옵션(또는 VLLM_API_KEY 환경 변수)은 /v1, /v2, /inference 경로 접두사 아래의 요청만 인증해요. 같은 HTTP 서버의 다른 엔드포인트는 인증되지 않는데, 특히 /invocations는 /v1 엔드포인트와 동일한 추론 기능을 노출하니 주의해야 해요. 그러니까 --api-key만으로 vLLM을 보호하려 하면 안 돼요. 보호·비보호 엔드포인트 전체 목록과 권장 보안 조치(예: 리버스 프록시 뒤에 배포)는 API Key Authentication Limitations 문서를 참고하세요.

지원하는 API

현재 지원하는 OpenAI API는 다음과 같아요.

Completions API

터미널에서 vLLM을 설치한 뒤, vllm serve 명령으로 서버를 시작하면 돼요. (vLLM의 Docker 이미지를 써도 됩니다.)

vllm serve NousResearch/Meta-Llama-3-8B-Instruct \
  --dtype auto \
  --api-key token-abc123

서버를 호출하려면 편집기에서 HTTP 클라이언트를 쓰는 스크립트를 만들고, 모델에 보낼 메시지를 담은 뒤 실행하면 돼요. 아래는 공식 OpenAI Python 클라이언트를 쓴 예시 스크립트예요.

from openai import OpenAI
client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="token-abc123",
)

completion = client.chat.completions.create(
    model="NousResearch/Meta-Llama-3-8B-Instruct",
    messages=[
        {"role": "user", "content": "Hello!"},
    ],
)

print(completion.choices[0].message)

팁: vLLM은 OpenAI가 지원하지 않는 일부 파라미터도 지원해요. 예를 들어 top_k가 그렇죠. 이런 파라미터는 OpenAI 클라이언트의 extra_body 파라미터로 전달할 수 있어요. 즉 top_k는 extra_body={"top_k": 50}처럼 넣으면 돼요.

중요: 기본적으로 서버는 Hugging Face 모델 저장소에 generation_config.json이 있으면 이를 적용해요. 즉 특정 샘플링 파라미터의 기본값이 모델 제작자가 권장한 값으로 덮어써질 수 있다는 뜻이에요. 이 동작을 끄려면 서버를 시작할 때 --generation-config vllm을 전달하세요.

추가 파라미터 (Extra Parameters)

vLLM은 OpenAI API에 없는 파라미터 세트를 지원해요. 이걸 쓰려면 OpenAI 클라이언트에서 추가 파라미터로 넘기거나, HTTP를 직접 호출한다면 JSON 페이로드에 바로 병합하면 돼요.

completion = client.chat.completions.create(
    model="NousResearch/Meta-Llama-3-8B-Instruct",
    messages=[
        {"role": "user", "content": "Classify this sentiment: vLLM is wonderful!"},
    ],
    extra_body={
        "structured_outputs": {"choice": ["positive", "negative"]},
    },
)

추가 HTTP 헤더 (Extra HTTP Headers)

X-Request-Id HTTP 요청 헤더는 --enable-request-id-headers로 활성화할 수 있어요.

completion = client.chat.completions.create(
    model="NousResearch/Meta-Llama-3-8B-Instruct",
    messages=[
        {"role": "user", "content": "Classify this sentiment: vLLM is wonderful!"},
    ],
    extra_headers={
        "x-request-id": "sentiment-classification-00001",
    },
)
print(completion._request_id)

completion = client.completions.create(
    model="NousResearch/Meta-Llama-3-8B-Instruct",
    prompt="A robot may not injure a human being",
    extra_headers={
        "x-request-id": "completion-test",
    },
)
print(completion._request_id)

Completions, Chat Completions, Responses API는 X-Vllm-Priority 요청 헤더도 지원해요. 이 값은 정수여야 하고, JSON 요청 본문의 priority 값을 덮어써요. 0이 아닌 우선순위는 서버가 우선순위 스케줄링을 사용해야 해요.

completion = client.chat.completions.create(
    model="NousResearch/Meta-Llama-3-8B-Instruct",
    messages=[{"role": "user", "content": "Hello!"}],
    extra_headers={"X-Vllm-Priority": "-10"},
)

API 레퍼런스

Completions API

vLLM의 Completions API는 OpenAI의 Completions API와 호환돼요. 공식 OpenAI Python 클라이언트로 상호작용할 수 있어요.

코드 예시: examples/basic/online_serving/openai_completion_client.py

지원하는 샘플링 파라미터

샘플링 파라미터가 지원돼요.

    use_beam_search: bool = False
    top_k: int | None = None
    min_p: float | None = None
    repetition_penalty: float | None = None
    length_penalty: float = 1.0
    stop_token_ids: list[int] | None = []
    include_stop_str_in_output: bool = False
    ignore_eos: bool = False
    min_tokens: int = 0
    skip_special_tokens: bool = True
    spaces_between_special_tokens: bool = True
    truncate_prompt_tokens: Annotated[int, Field(ge=-1, le=_INT64_MAX)] | None = None
    truncation_side: Literal["left", "right"] | None = Field(
        default=None,
        description=(
            "Which side to truncate from when truncate_prompt_tokens is active. "
            "'right' keeps the first N tokens. "
            "'left' keeps the last N tokens."
        ),
    )
    allowed_token_ids: list[int] | None = None
    prompt_logprobs: int | None = None
    logprob_token_ids: list[int] | None = Field(
        default=None,
        description=(
            "Specific vocab token IDs to return logprobs for at each generated "
            "position, in addition to the sampled token. More efficient than "
            "requesting the full vocab when only a small fixed label set is "
            "needed (e.g. multilabel "
            "scoring where each label corresponds to a known vocab id). When "
            "set, this explicit token selection takes precedence over the "
            "natural top-k selected by `logprobs`. Requires `logprobs` to be "
            "set."
        ),
    )
    bad_words: list[str] = Field(default_factory=list)

추가 파라미터

    prompt_embeds: bytes | list[bytes] | None = None
    add_special_tokens: bool = Field(
        default=True,
        description=(
            "If true (the default), special tokens (e.g. BOS) will be added to "
            "the prompt."
        ),
    )
    response_format: AnyResponseFormat | None = Field(
        default=None,
        description=(
            "Similar to chat completion, this parameter specifies the format "
            "of output. Only {'type': 'json_object'}, {'type': 'json_schema'}"
            ", {'type': 'structural_tag'}, or {'type': 'text' } is supported."
        ),
    )
    structured_outputs: StructuredOutputsParams | None = Field(
        default=None,
        description="Additional kwargs for structured outputs",
    )
    priority: int = Field(
        default=0,
        ge=_INT64_MIN,
        le=_INT64_MAX,
        description=(
            "The priority of the request (lower means earlier handling; "
            "default: 0). Any priority other than 0 will raise an error "
            "if the served model does not use priority scheduling."
        ),
    )
    request_id: str = Field(
        default_factory=random_uuid,
        description=(
            "The request_id related to this request. If the caller does "
            "not set it, a random_uuid will be generated. This id is used "
            "through out the inference process and return in response."
        ),
    )
    session_id: str | None = Field(
        default=None,
        description=(
            "Stable session identity shared by related requests. Unlike "
            "request_id, this value is expected to remain stable across "
            "multiple requests in the same conversation or agent session."
        ),
    )

    return_tokens_as_token_ids: bool | None = Field(
        default=None,
        description=(
            "If specified with 'logprobs', tokens are represented "
            " as strings of the form 'token_id:{token_id}' so that tokens "
            "that are not JSON-encodable can be identified."
        ),
    )
    return_token_ids: bool | None = Field(
        default=None,
        description=(
            "If specified, the result will include token IDs alongside the "
            "generated text. In streaming mode, prompt_token_ids is included "
            "only in the first chunk, and token_ids contains the delta tokens "
            "for each chunk. This is useful for debugging or when you "
            "need to map generated text back to input tokens."
        ),
    )
    routed_experts_prompt_start: int = Field(
        default=0,
        ge=0,
        description="Skip the first N prompt tokens from returned routed-expert data.",
    )
    return_token_offsets: bool | None = Field(
        default=False,
        description=(
            "If true, return char-level (start, end) offsets for each "
            "token relative to the tokenized source string in the "
            "`token_offsets` field of the rendered response. Only "
            "supported on the `/v1/completions/render` and "
            "`/v1/chat/completions/render` endpoints; ignored on regular "
            "generation endpoints. Honored only for Fast (Rust-backed) "
            "tokenizers; otherwise `token_offsets` is null. For chat "
            "requests, offsets are relative to the templated prompt "
            "string (after applying the chat template). Multimodal "
            "inputs and pre-tokenized inputs always yield null."
        ),
    )

    cache_salt: str | None = Field(
        default=None,
        min_length=1,
        max_length=1024,
        description=(
            "If specified, the prefix cache will be salted with the provided "
            "string to prevent an attacker to guess prompts in multi-user "
            "environments. The salt should be random, protected from "
            "access by 3rd parties, and long enough to be "
            "unpredictable (e.g., 43 characters base64-encoded, corresponding "
            "to 256 bit)."
        ),
    )

    kv_transfer_params: dict[str, Any] | None = Field(
        default=None,
        description="KVTransfer parameters used for disaggregated serving.",
    )

    ec_transfer_params: dict[str, Any] | None = Field(
        default=None,
        description=(
            "ECTransfer parameters used for encoder-cache disaggregated serving."
        ),
    )

    vllm_xargs: dict[str, str | int | float | list[str | int | float]] | None = Field(
        default=None,
        description=(
            "Additional request parameters with (list of) string or "
            "numeric values, used by custom extensions."
        ),
    )

    repetition_detection: RepetitionDetectionParams | None = Field(
        default=None,
        description="Parameters for detecting repetitive N-gram patterns "
        "in output tokens. If such repetition is detected, generation will "
        "be ended early. LLMs can sometimes generate repetitive, unhelpful "
        "token patterns, stopping only when they hit the maximum output length "
        "(e.g. 'abcdabcdabcd...' or '\\emoji \\emoji \\emoji ...'). This feature "
        "can detect such behavior and terminate early, saving time and tokens.",
    )

    thinking_token_budget: ThinkingTokenBudget = Field(
        default=None,
        description=(
            "Maximum number of tokens allowed for thinking operations "
            "(reasoning models). Non-negative integer sets the limit; "
            "-1 means unlimited (treated as unset)."
        ),
    )

    stream_interval: Annotated[int, Field(ge=1)] | None = Field(
        default=None,
        description=(
            "Number of tokens to batch into each streamed chunk. Raises the "
            "server's `--stream-interval` for this request. Values below the "
            "server setting are clamped up to it. The first and last chunks "
            "are always sent immediately. Ignored for non-streaming requests."
        ),
    )

Chat API

vLLM의 Chat API는 OpenAI의 Chat Completions API와 호환돼요. 공식 OpenAI Python 클라이언트로 상호작용할 수 있어요.

Vision- 및 Audio 관련 파라미터를 모두 지원해요. 자세한 내용은 Multimodal Inputs 가이드를 참고하세요.

지원하는 샘플링 파라미터

샘플링 파라미터가 지원돼요.

    use_beam_search: bool = False
    top_k: int | None = None
    min_p: float | None = None
    repetition_penalty: float | None = None
    length_penalty: float = 1.0
    stop_token_ids: list[int] | None = []
    include_stop_str_in_output: bool = False
    ignore_eos: bool = False
    min_tokens: int = 0
    skip_special_tokens: bool = True
    spaces_between_special_tokens: bool = True
    truncate_prompt_tokens: Annotated[int, Field(ge=-1, le=_INT64_MAX)] | None = None
    truncation_side: Literal["left", "right"] | None = Field(
        default=None,
        description=(
            "Which side to truncate from when truncate_prompt_tokens is active. "
            "'right' keeps the first N tokens. "
            "'left' keeps the last N tokens."
        ),
    )
    prompt_logprobs: int | None = None
    logprob_token_ids: list[int] | None = Field(
        default=None,
        description=(
            "Specific vocab token IDs to return logprobs for at each generated "
            "position, in addition to the sampled token. More efficient than "
            "`top_logprobs=-1` when only a small fixed label set is needed "
            "(e.g. multilabel scoring "
            "where each label corresponds to a known vocab id). When set, "
            "this explicit token selection takes precedence over the natural "
            "top-k selected by `top_logprobs`. Requires `logprobs=True`."
        ),
    )
    allowed_token_ids: list[int] | None = None
    bad_words: list[str] = Field(default_factory=list)

추가 파라미터

    echo: bool = Field(
        default=False,
        description=(
            "If true, the new message will be prepended with the last message "
            "if they belong to the same role."
        ),
    )
    add_generation_prompt: bool = Field(
        default=True,
        description=(
            "If true, the generation prompt will be added to the chat template. "
            "This is a parameter used by chat template in tokenizer config of the "
            "model."
        ),
    )
    continue_final_message: bool = Field(
        default=False,
        description=(
            "If this is set, the chat will be formatted so that the final "
            "message in the chat is open-ended, without any EOS tokens. The "
            "model will continue this message rather than starting a new one. "
            'This allows you to "prefill" part of the model\'s response for it. '
            "Cannot be used at the same time as `add_generation_prompt`."
        ),
    )
    add_special_tokens: bool = Field(
        default=False,
        description=(
            "If true, special tokens (e.g. BOS) will be added to the prompt "
            "on top of what is added by the chat template. "
            "For most models, the chat template takes care of adding the "
            "special tokens so this should be set to false (as is the "
            "default)."
        ),
    )
    documents: list[dict[str, str]] | None = Field(
        default=None,
        description=(
            "A list of dicts representing documents that will be accessible to "
            "the model if it is performing RAG (retrieval-augmented generation)."
            " If the template does not support RAG, this argument will have no "
            "effect. We recommend that each document should be a dict containing "
            '"title" and "text" keys.'
        ),
    )
    chat_template: str | None = Field(
        default=None,
        description=(
            "A Jinja template to use for this conversion. "
            "As of transformers v4.44, default chat template is no longer "
            "allowed, so you must provide a chat template if the tokenizer "
            "does not define one."
        ),
    )
    chat_template_kwargs: dict[str, Any] | None = Field(
        default=None,
        description=(
            "Additional keyword args to pass to the template renderer. "
            "Will be accessible by the chat template."
        ),
    )
    media_io_kwargs: dict[str, dict[str, Any]] | None = Field(
        default=None,
        description=(
            "Additional kwargs to pass to the media IO connectors, "
            "keyed by modality. Merged with engine-level media_io_kwargs."
        ),
    )
    mm_processor_kwargs: dict[str, Any] | None = Field(
        default=None,
        description=("Additional kwargs to pass to the HF processor."),
    )
    structured_outputs: StructuredOutputsParams | None = Field(
        default=None,
        description="Additional kwargs for structured outputs",
    )
    priority: int = Field(
        default=0,
        ge=_INT64_MIN,
        le=_INT64_MAX,
        description=(
            "The priority of the request (lower means earlier handling; "
            "default: 0). Any priority other than 0 will raise an error "
            "if the served model does not use priority scheduling."
        ),
    )
    request_id: str = Field(
        default_factory=random_uuid,
        description=(
            "The request_id related to this request. If the caller does "
            "not set it, a random_uuid will be generated. This id is used "
            "through out the inference process and return in response."
        ),
    )
    session_id: str | None = Field(
        default=None,
        description=(
            "Stable session identity shared by related requests. Unlike "
            "request_id, this value is expected to remain stable across "
            "multiple requests in the same conversation or agent session."
        ),
    )

    return_tokens_as_token_ids: bool | None = Field(
        default=None,
        description=(
            "If specified with 'logprobs', tokens are represented "
            " as strings of the form 'token_id:{token_id}' so that tokens "
            "that are not JSON-encodable can be identified."
        ),
    )
    return_token_ids: bool | None = Field(
        default=None,
        description=(
            "If specified, the result will include token IDs alongside the "
            "generated text. In streaming mode, prompt_token_ids is included "
            "only in the first chunk, and token_ids contains the delta tokens "
            "for each chunk. This is useful for debugging or when you "
            "need to map generated text back to input tokens."
        ),
    )
    routed_experts_prompt_start: int = Field(
        default=0,
        ge=0,
        description="Skip the first N prompt tokens from returned routed-expert data.",
    )
    return_token_offsets: bool | None = Field(
        default=False,
        description=(
            "If true, return char-level (start, end) offsets for each "
            "token relative to the tokenized source string in the "
            "`token_offsets` field of the rendered response. Only "
            "supported on the `/v1/completions/render` and "
            "`/v1/chat/completions/render` endpoints; ignored on regular "
            "generation endpoints. Honored only for Fast (Rust-backed) "
            "tokenizers; otherwise `token_offsets` is null. For chat "
            "requests, offsets are relative to the templated prompt "
            "string (after applying the chat template). Multimodal "
            "inputs and pre-tokenized inputs always yield null."
        ),
    )
    return_prompt_text: bool | None = Field(
        default=None,
        description=(
            "If true, the response will include ``prompt_text`` containing the "
            "prompt string produced by chat templating. In streaming mode it "
            "is sent only on the first chunk. This is useful for inspecting "
            "exactly what was fed into the model."
        ),
    )

    return_assistant_tokens_mask: bool = Field(
        default=False,
        description=(
            "If true, the /render response will include an "
            "``assistant_tokens_mask`` field — a per-token list of 0/1 "
            "values indicating which tokens were assistant-generated. "
            "Requires the chat template to use ``{% generation %}`` "
            "tags.  When the template does not support it, "
            "``assistant_tokens_mask`` will be ``null``."
        ),
    )

    cache_salt: str | None = Field(
        default=None,
        min_length=1,
        max_length=1024,
        description=(
            "If specified, the prefix cache will be salted with the provided "
            "string to prevent an attacker to guess prompts in multi-user "
            "environments. The salt should be random, protected from "
            "access by 3rd parties, and long enough to be "
            "unpredictable (e.g., 43 characters base64-encoded, corresponding "
            "to 256 bit)."
        ),
    )

    kv_transfer_params: dict[str, Any] | None = Field(
        default=None,
        description="KVTransfer parameters used for disaggregated serving.",
    )

    ec_transfer_params: dict[str, Any] | None = Field(
        default=None,
        description=(
            "ECTransfer parameters used for encoder-cache disaggregated serving."
        ),
    )

    vllm_xargs: dict[str, str | int | float | list[str | int | float]] | None = Field(
        default=None,
        description=(
            "Additional request parameters with (list of) string or "
            "numeric values, used by custom extensions."
        ),
    )

    repetition_detection: RepetitionDetectionParams | None = Field(
        default=None,
        description="Parameters for detecting repetitive N-gram patterns "
        "in output tokens. If such repetition is detected, generation will "
        "be ended early. LLMs can sometimes generate repetitive, unhelpful "
        "token patterns, stopping only when they hit the maximum output length "
        "(e.g. 'abcdabcdabcd...' or '\\emoji \\emoji \\emoji ...'). This feature "
        "can detect such behavior and terminate early, saving time and tokens.",
    )

    stream_interval: Annotated[int, Field(ge=1)] | None = Field(
        default=None,
        description=(
            "Number of tokens to batch into each streamed chunk. Raises the "
            "server's `--stream-interval` for this request. Values below the "
            "server setting are clamped up to it. The first and last chunks "
            "are always sent immediately. Ignored for non-streaming requests."
        ),
    )

Responses API

vLLM의 Responses API는 OpenAI의 Responses API와 호환돼요. 공식 OpenAI Python 클라이언트로 상호작용할 수 있어요.

코드 예시: examples/tool_calling/openai_responses_client_with_tools.py

요청 객체의 추가 파라미터

    request_id: str = Field(
        default_factory=lambda: f"resp_{random_uuid()}",
        description=(
            "The request_id related to this request. If the caller does "
            "not set it, a random_uuid will be generated. This id is used "
            "through out the inference process and return in response."
        ),
    )
    session_id: str | None = Field(
        default=None,
        description=(
            "Stable session identity shared by related requests. Unlike "
            "request_id, this value is expected to remain stable across "
            "multiple requests in the same conversation or agent session."
        ),
    )
    media_io_kwargs: dict[str, dict[str, Any]] | None = Field(
        default=None,
        description=(
            "Additional kwargs to pass to the media IO connectors, "
            "keyed by modality. Merged with engine-level media_io_kwargs."
        ),
    )
    mm_processor_kwargs: dict[str, Any] | None = Field(
        default=None,
        description=("Additional kwargs to pass to the HF processor."),
    )
    priority: int = Field(
        default=0,
        ge=_INT64_MIN,
        le=_INT64_MAX,
        description=(
            "The priority of the request (lower means earlier handling; "
            "default: 0). Any priority other than 0 will raise an error "
            "if the served model does not use priority scheduling."
        ),
    )
    cache_salt: str | None = Field(
        default=None,
        min_length=1,
        max_length=1024,
        description=(
            "If specified, the prefix cache will be salted with the provided "
            "string to prevent an attacker to guess prompts in multi-user "
            "environments. The salt should be random, protected from "
            "access by 3rd parties, and long enough to be "
            "unpredictable (e.g., 43 characters base64-encoded, corresponding "
            "to 256 bit)."
        ),
    )

    enable_response_messages: bool = Field(
        default=False,
        description=(
            "Dictates whether or not to return messages as part of the "
            "response object. Currently only supported for non-background."
        ),
    )
    # similar to input_messages / output_messages in ResponsesResponse
    # we take in previous_input_messages (ie in harmony format)
    # this cannot be used in conjunction with previous_response_id
    # TODO: consider supporting non harmony messages as well
    previous_input_messages: list[OpenAIHarmonyMessage | dict] | None = None
    structured_outputs: StructuredOutputsParams | None = Field(
        default=None,
        description="Additional kwargs for structured outputs",
    )

    repetition_penalty: float | None = None
    seed: int | None = Field(None, ge=_INT64_MIN, le=_INT64_MAX)
    stop: StopParam = []
    ignore_eos: bool = False
    vllm_xargs: dict[str, str | int | float | list[str | int | float]] | None = Field(
        default=None,
        description=(
            "Additional request parameters with (list of) string or "
            "numeric values, used by custom extensions."
        ),
    )
    kv_transfer_params: dict[str, Any] | None = Field(
        default=None,
        description="KVTransfer parameters used for disaggregated serving.",
    )
    ec_transfer_params: dict[str, Any] | None = Field(
        default=None,
        description=(
            "ECTransfer parameters used for encoder-cache disaggregated serving."
        ),
    )
    chat_template_kwargs: dict[str, Any] | None = Field(
        default=None,
        description=(
            "Additional keyword args to pass to the chat template renderer. "
            "Will be accessible by the template."
        ),
    )

응답 객체의 추가 파라미터

    # These are populated when enable_response_messages is set to True
    # NOTE: custom serialization is needed
    # see serialize_input_messages and serialize_output_messages
    input_messages: ResponseInputOutputMessage | None = Field(
        default=None,
        description=(
            "If enable_response_messages, we can show raw token input to model."
        ),
    )
    output_messages: ResponseInputOutputMessage | None = Field(
        default=None,
        description=(
            "If enable_response_messages, we can show raw token output of model."
        ),
    )

참고: 원본 URL https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html은 현재 https://docs.vllm.ai/en/latest/serving/online_serving/openai_compatible_server/로 리다이렉트됩니다. 본 문서는 리다이렉트된 최신 경로의 내용을 기준으로 작성됐어요.