Skip to content

Renderer APIs

Our renderer API is designed to disaggregate the render phase(preprocessing) and enable a token-in / token-out API server.

  • GPU-less deployment of frontend: Allow preprocessing (tokenization, MM input processing) and postprocessing (detokenization, tool call parsing, reasoning parsing) to run without GPU.
  • Disaggregated tokenization: Support use cases such as llm-d, Dynamo, and custom frontends that need to leverage vLLM's preprocessing logic without running the full inference engine.
  • Tokens-in / tokens-out engine: Make the engine a pure token-in / token-out service, decoupled from request preprocessing.

The dedicated vllm launch render server always exposes the /render and /derender endpoints.

Scale-out endpoints, including /render, /derender, and /inference/v1/generate, are disabled by default on a standard inference server. To expose them with vllm serve, opt in explicitly:

vllm serve <model> --enable-scale-out

API Reference

Get Responses prompt token IDs

Use /v1/responses/render to get prompt token IDs before choosing a model replica. Rendering applies prompt construction and preprocessing without running inference.

The Responses render endpoint uses the same prompt construction as /v1/responses and returns one token-in GenerateRequest. It is stateless: inline history is supported, but previous_response_id is not. Callers must resolve stored response state and include the resulting history in the request before rendering.

Configure the renderer and generation workers with the same model, tokenizer, chat template, and preprocessing options. Render the full request, including instructions, history, tools, and any template or truncation options, so the returned IDs reflect the prompt the model will receive.

For multimodal requests, the GenerateRequest contains the model-processed multimodal payload, which can be substantially larger than the source image or video. The caller must forward that payload unchanged to the generation service and provision transport limits and memory accordingly.

For example, start a standard inference server with scale-out endpoints enabled:

vllm serve meta-llama/Llama-3.1-8B-Instruct --enable-scale-out

Send a Responses request and use jq to extract the token_ids field:

curl --fail --silent --show-error http://localhost:8000/v1/responses/render \
    -H "Content-Type: application/json" \
    -d '{
        "model": "meta-llama/Llama-3.1-8B-Instruct",
        "input": "Explain prefix caching in one sentence.",
        "max_output_tokens": 32
    }' | jq '.token_ids'

To count the rendered prompt tokens for this text request, replace the jq filter with '.token_ids | length'. The endpoint returns the full GenerateRequest; jq filters the response on the client. Keep the complete response if you will forward it to /inference/v1/generate, including any multimodal features.

If the server has --api-key or VLLM_API_KEY configured, add -H "Authorization: Bearer <api-key>" to the request. See API key authentication limitations for the existing authentication boundaries.

For the post processing counterpart that turns generated token IDs back into OpenAI compatible responses, see the Derenderer APIs.

Generate Output Logprobs

/inference/v1/generate is a token in / token out API, so its output logprobs identify tokens by integer ID rather than by the OpenAI string token. With sampling_params.logprobs set and output_mode: "tokens" (the default), each choice carries a GenerateLogProbs; output_mode: "text" returns decoded ChatCompletionLogProbs instead (see token in, token out):

{
  "logprobs": {
    "content": [
      {
        "token_id": 262,
        "logprob": -0.10,
        "rank": 1,
        "top_logprobs": [
          {"token_id": 262, "logprob": -0.10, "rank": 1},
          {"token_id": 257, "logprob": -1.20, "rank": 2}
        ]
      }
    ]
  }
}
  • content has one entry per generated token, in generation order.
  • top_logprobs is a list, not a dict: JSON turns dict keys into strings and the ordering would be implicit. It follows the engine's order: the sampled token first, then the remaining candidates in rank order. With non-greedy sampling the sampled token can sit outside the top k (for example ranks [5, 1, 2] at logprobs=2); it then takes one of the logprobs slots and the rank-k candidate is left out, as on the OpenAI endpoints.
  • rank is the token's rank in the vocabulary distribution (1 = most likely) on every entry, the sampled one included; a top-k candidate's rank is its top-k position. The list is not sorted by it, so sort by rank if you need rank order. It is null when the engine could not rank the token (a NaN logprob, which is sent as -9999.0).
  • There is no token or bytes field. The generate server has no tokenizer; derender fills those in when it converts the response to the OpenAI shapes.
  • prompt_logprobs on the same response is unchanged (list[dict[int, Logprob] | None]).

Changed in this release

Output logprobs used to be ChatCompletionLogProbs with every token written as a "token_id:N" placeholder string, and bytes set to the UTF-8 bytes of that placeholder by the Rust frontend but left unset by the Python one. Clients that read only content[i].logprob are unaffected. Clients that parsed the placeholder should read content[i].token_id instead. return_tokens_as_token_ids on /v1/chat/completions and /v1/completions is unchanged: it is a user-facing OpenAI option and still uses the token_id:N format.

Multimodal Render Features

Multimodal render responses include a features object with per-modality hashes, placeholder ranges, and serialized processor data. When the model exposes placeholder-metadata or keep_on_cpu fields (for example image_grid_thw), the response also includes mm_metadata. Each mm_metadata entry is a base64-encoded MultiModalKwargsItem containing only those fields, not encoder inputs such as pixel_values.

The arrays in mm_hashes, mm_placeholders, kwargs_data, and mm_metadata use the same per-modality item order. Downstream workers should split those fields:

  • Encode requests keep kwargs_data.
  • Prefill requests may omit kwargs_data and send mm_metadata only when ec_transfer_params is also set, so embeddings are loaded by the EC connector. Omitting kwargs_data without ec_transfer_params is rejected.
  • Legacy clients that ignore mm_metadata and keep sending kwargs_data continue to work.

Example

The example below shows how a disaggregated encode / prefill coordinator can split a multimodal render response. The render step returns both kwargs_data (encoder tensors plus metadata) and mm_metadata (metadata only). Encode keeps the full payload; prefill drops kwargs_data after the EC connector has published embeddings.

import httpx

MODEL = "Qwen/Qwen3-VL-2B-Instruct"
RENDER = "http://localhost:8100"  # vllm launch render ...
ENCODE = "http://localhost:8200"  # encode worker
PREFILL = "http://localhost:8300"  # prefill worker

chat_request = {
    "model": MODEL,
    "messages": [
        {
            "role": "user",
            "content": [
                {"type": "image_url", "image_url": {"url": "<data-url>"}},
                {"type": "text", "text": "Describe this image."},
            ],
        }
    ],
}

with httpx.Client(timeout=120.0) as client:
    # 1. Render: preprocess into token IDs and multimodal features.
    render_response = client.post(
        f"{RENDER}/v1/chat/completions/render", json=chat_request
    ).json()

    features = render_response["features"]
    # features["kwargs_data"]["image"][0]  -> pixel_values + image_grid_thw
    # features["mm_metadata"]["image"][0]  -> image_grid_thw only

    # 2. Encode: send full kwargs_data so the encoder can run vision towers.
    encode_response = client.post(
        f"{ENCODE}/inference/v1/generate",
        json={
            "token_ids": render_response["token_ids"],
            "features": {
                "mm_hashes": features["mm_hashes"],
                "mm_placeholders": features["mm_placeholders"],
                "kwargs_data": features["kwargs_data"],
            },
            "sampling_params": {"max_tokens": 1},
        },
    ).json()
    ec_transfer_params = encode_response["ec_transfer_params"]

    # 3. Prefill: omit kwargs_data; load embeddings via EC connector.
    prefill_response = client.post(
        f"{PREFILL}/inference/v1/generate",
        json={
            "token_ids": render_response["token_ids"],
            "features": {
                "mm_hashes": features["mm_hashes"],
                "mm_placeholders": features["mm_placeholders"],
                "mm_metadata": features["mm_metadata"],
            },
            "ec_transfer_params": ec_transfer_params,
            "sampling_params": {"max_tokens": 64},
        },
    ).json()

print(prefill_response["choices"][0]["token_ids"])

Single-process clients can keep passing the full render response to /inference/v1/generate unchanged; mm_metadata is optional and ignored when kwargs_data is present.

Payload shape

Render response

/v1/chat/completions/render returns both kwargs_data and mm_metadata. The arrays share the same per-modality item order. Base64 blobs are truncated below for readability.

{
  "token_ids": [151644, 872],
  "features": {
    "mm_hashes": {"image": ["abc123..."]},
    "mm_placeholders": {"image": [{"offset": 0, "length": 256}]},
    "kwargs_data": {
      "image": ["<base64 MultiModalKwargsItem: pixel_values + image_grid_thw>"]
    },
    "mm_metadata": {
      "image": ["<base64 MultiModalKwargsItem: image_grid_thw only>"]
    }
  }
}

Forward kwargs_data to the encode worker. Keep mm_metadata for prefill.

Prefill request

Prefill omits kwargs_data and sends mm_metadata with ec_transfer_params from the encode response:

{
  "token_ids": [151644, 872],
  "features": {
    "mm_hashes": {"image": ["abc123..."]},
    "mm_placeholders": {"image": [{"offset": 0, "length": 256}]},
    "mm_metadata": {
      "image": ["<base64 MultiModalKwargsItem: image_grid_thw only>"]
    }
  },
  "ec_transfer_params": {
    "ec_items": [{"mm_hash": "abc123...", "peer_host": "10.0.0.1"}]
  },
  "sampling_params": {"max_tokens": 64}
}