k3s-llmd-lab · measured on a 3-node cluster

llm-d Request Flow

What happens between curl and a token. The gateway does not choose a backend — it pauses mid-request to ask a scheduler that reads the prompt, knows which pod holds its cached prefix, and may decide the request is worth splitting across two machines. Step through it below.

client OPENAI API agentgateway GATEWAY API · :80 forwarded to decode · x-prefiller-host-port INFERENCEPOOL · SELECTOR app=sim pd-sidecar :8000 → :8200 decode × 3 AGENT-2 · BANDWIDTH prefill × 3 AGENT-1 · COMPUTE prefill request KV cache · modelled, not moved response streams back ENDPOINT PICKER · EXT_PROC :9002 tokenizer TOKEN-PRODUCER P/D decider SPLIT, OR NOT prefill profile PREFIX ×3 · QUEUE ×1 decode profile PREFIX ×3 · QUEUE ×2 · KV ×2 headers + body endpoints to use render VLLM TOKENIZER · NO WEIGHTS exact token ids same tokenizer · block hashing Prometheus + GRAFANA vllm:* llm_d_epp_* · authenticated ollama QWEN2.5:0.5B · REAL /ollama · no picker
request path only on a split continuous, not per-request

The whole path

Everything, at rest

Playing automatically · click any step to take over · every figure came off the running cluster