Inside vLLM: Anatomy of a High-Throughput LLM Inference System深入 vLLM：高吞吐量 LLM 推理系统剖析

In this post, I'll gradually introduce all of the core system components and advanced features that make up a modern high-throughput LLM inference system. In particular I'll be doing a breakdown of how vLLM [1] works.在这篇文章中，我将逐步介绍构成现代高吞吐量 LLM 推理系统的核心系统组件和高级功能。特别地，我将详细拆解 vLLM [1] 的工作原理。

This post is the first in a series. It starts broad and then layers in detail (following an inverse-pyramid approach) so you can form an accurate high-level mental model of the complete system without drowning in minutiae.本文是该系列的第一篇。它将采用“倒金字塔”式的方法，从宏观概述逐步深入细节，帮助你在不陷入琐碎细节的情况下，建立起对整个系统准确的高层认知模型。

Later posts will dive into specific subsystems.后续文章将深入探讨各个具体的子系统。

This post is structured into five parts:本文分为五个部分：

1. LLM engine & engine core: fundamentals of vLLM (scheduling, paged attention, continuous batching, etc.) LLM 引擎与引擎核心：vLLM 的基础（调度、分页注意力、连续批处理等）
2. Advanced features: chunked prefill, prefix caching, guided & speculative decoding, disaggregated P/D高级功能：分块预填充（Chunked Prefill）、前缀缓存、引导式解码与投机解码、解耦式 P/D（预填充/解码）
3. Scaling up: from single-GPU to multi-GPU execution扩展：从单 GPU 到多 GPU 执行
4. Serving layer: distributed / concurrent web scaffolding服务层：分布式/并发 Web 框架
5. Benchmarks and auto-tuning: measuring latency and throughput 基准测试与自动调优：测量延迟和吞吐量

LLM Engine & Engine CoreLLM 引擎与引擎核心

The LLM engine is the fundamental building block of vLLM. On its own, it already enables high-throughput inference - but only in an offline setting. You can't serve it to customers over the web yet.LLM 引擎是 vLLM 的基本构建模块。它本身就能实现高吞吐量推理，但仅限于离线环境。目前还无法通过 Web 为客户提供服务。

We'll use the following offline inference snippet as our running example (adapted from basic.py).我们将使用以下离线推理代码片段作为贯穿全文的示例（改编自 basic.py）。

from vllm import LLM, SamplingParams

prompts = [
    "Hello, my name is",
    "The president of the United States is",
]

sampling_params = SamplingParams(temperature=0.8, top_p=0.95)

def main():
    llm = LLM(model="TinyLlama/TinyLlama-1.1B-Chat-v1.0")

    outputs = llm.generate(prompts, sampling_params)

if __name__ == "__main__":
    main()

This configuration is:此配置为：

• offline (no web/distributed system scaffolding)离线（无 Web/分布式系统框架）
• synchronous (all execution happens in a single blocking process)同步（所有执行都在单个阻塞进程中进行）
• single-GPU (no data/model/pipeline/expert parallelism; DP/TP/PP/EP = 1)单 GPU（无数据/模型/流水线/专家并行；DP/TP/PP/EP = 1）
• using standard transformer [2] (supporting hybrid models like Jamba requires a more complex hybrid KV-cache memory allocator)使用标准 Transformer [2]（支持 Jamba 等混合模型需要更复杂的混合 KV 缓存内存分配器）

From here, we'll gradually build up to an online, async, multi-GPU, multi-node inference system - but still serving a standard transformer.在此基础上，我们将逐步构建一个在线、异步、多 GPU、多节点的推理系统，但仍用于服务标准 Transformer。

In this example we do two things, we:在此示例中，我们主要做两件事：

1. Instantiate an engine实例化一个引擎
2. Call generate on it to sample from the given prompts调用 generate 方法对给定的提示词进行采样

Let's start analyzing the constructor.让我们从构造函数开始分析。

LLM Engine constructorLLM 引擎构造函数

The main components of the engine are:引擎的主要组件包括：

• vLLM config (contains all of the knobs for configuring model, cache, parallelism, etc.)vLLM 配置（包含用于配置模型、缓存、并行度等的所有参数）
• processor (turns raw inputs → EngineCoreRequests via validation, tokenization, and processing)处理器（通过验证、分词和处理，将原始输入转换为 EngineCoreRequests）
• engine core client (in our running example we're using InprocClient which is basically == EngineCore; we'll gradually build up to DPLBAsyncMPClient which allows serving at scale)引擎核心客户端（在我们的示例中，我们使用 InprocClient，它基本等同于 EngineCore；我们将逐步演进到支持大规模服务的 DPLBAsyncMPClient）
• output processor (converts raw EngineCoreOutputs → RequestOutput that the user sees)输出处理器（将原始 EngineCoreOutputs 转换为用户可见的 RequestOutput）

Engine core itself is made up of several sub components:引擎核心本身由几个子组件组成：

• Model Executor (drives forward passes on the model, we're currently dealing with UniProcExecutor which has a single Worker process on a single GPU). We'll gradually build up to MultiProcExecutor which supports multiple GPUs模型执行器（驱动模型的前向传播，目前我们处理的是在单个 GPU 上具有单个 Worker 进程的 UniProcExecutor）。我们将逐步演进到支持多 GPU 的 MultiProcExecutor。
• Structured Output Manager (used for guided decoding - we'll cover this later)结构化输出管理器（用于引导式解码——稍后讨论）
• Scheduler (decides which requests go into the next engine step) - it further contains:
  1. policy setting - it can be either FCFS (first come first served) or priority (higher priority requests are served first)策略设置——可以是 FCFS（先来先服务）或优先级（高优先级请求优先）
  2. waiting and running queues等待队列和运行队列
  3. KV cache manager - the heart of paged attention [3]KV 缓存管理器——分页注意力 [3] 的核心

The KV-cache manager maintains a free_block_queue - a pool of available KV-cache blocks (often on the order of hundreds of thousands, depending on VRAM size and block size). During paged attention, the blocks serve as the indexing structure that map tokens to their computed KV cache blocks.KV 缓存管理器维护一个 free_block_queue——一个可用 KV 缓存块池（通常根据 VRAM 大小和块大小，规模在数十万量级）。在分页注意力机制中，这些块作为索引结构，将 Token 映射到它们计算出的 KV 缓存块。

During model executor construction, a Worker object is created, and three key procedures are executed. (Later, with MultiProcExecutor, these same procedures run independently on each worker process across different GPUs.)在模型执行器构建期间，会创建一个 Worker 对象并执行三个关键过程。（稍后，使用 MultiProcExecutor 时，这些相同的过程会在不同 GPU 上的每个 Worker 进程中独立运行。）

1. Init device:
   • Assign a CUDA device (e.g. "cuda:0") to the worker and check that the model dtype is supported (e.g. bf16)为 Worker 分配 CUDA 设备（例如 "cuda:0"）并检查模型数据类型是否受支持（例如 bf16）
   • Verify enough VRAM is available, given the requested gpu_memory_utilization (e.g. 0.8 → 80% of total VRAM)验证是否有足够的 VRAM 可用，基于请求的 gpu_memory_utilization（例如 0.8 -> 总 VRAM 的 80%）
   • Set up distributed settings (DP / TP / PP / EP, etc.)设置分布式配置（DP / TP / PP / EP 等）
   • Instantiate a model_runner (holds the sampler, KV cache, and forward-pass buffers such as input_ids, positions, etc.)实例化 model_runner（持有采样器、KV 缓存以及 input_ids、positions 等前向传播缓冲区）
   • Instantiate an InputBatch object (holds CPU-side forward-pass buffers, block tables for KV-cache indexing, sampling metadata, etc.)实例化 InputBatch 对象（持有 CPU 端前向传播缓冲区、KV 缓存索引的块表、采样元数据等）
2. Load model:
   • Instantiate the model architecture实例化模型架构
   • Load the model weights加载模型权重
   • Call model.eval() (PyTorch's inference mode)调用 model.eval()（PyTorch 的推理模式）
   • Optional: call torch.compile() on the model可选：对模型调用 torch.compile()
3. Initialize KV cache
   • Get per-layer KV-cache spec. Historically this was always FullAttentionSpec (homogeneous transformer), but with hybrid models (sliding window, Transformer/SSM like Jamba) it became more complex (see Jenga [5])获取各层的 KV 缓存规格。历史上这始终是 FullAttentionSpec（同构 Transformer），但随着混合模型（滑动窗口、Transformer/SSM 如 Jamba）的出现，它变得更加复杂（参见 Jenga [5]）
   • Run a dummy/profiling forward pass and take a GPU memory snapshot to compute how many KV cache blocks fit in available VRAM运行一次模拟/性能分析前向传播，并获取 GPU 内存快照，以计算可用 VRAM 中能容纳多少 KV 缓存块
   • Allocate, reshape and bind KV cache tensors to attention layers分配、重塑并将 KV 缓存张量绑定到注意力层
   • Prepare attention metadata (e.g. set the backend to FlashAttention) later consumed by kernels during the fwd pass准备注意力元数据（例如将后端设置为 FlashAttention），供后续前向传播期间的内核使用
   • Unless --enforce-eager is provided, for each of warmup batch sizes do a dummy run and capture CUDA graphs. CUDA graphs record the whole sequence of GPU work into a DAG. Later during fwd pass we launch/replay pre-baked graphs and cut on kernel launch overhead and thus improve latency.除非提供了 --enforce-eager，否则对于每个预热批次大小，都会进行模拟运行并捕获 CUDA Graph。CUDA Graph 将整个 GPU 工作序列记录为 DAG。随后在前向传播期间，我们启动/重放预先构建的 Graph，从而减少内核启动开销并改善延迟。

I've abstracted away many low-level details here — but these are the core pieces I'll introduce now, since I'll reference them repeatedly in the following sections.这里我抽象了许多底层细节——但这些是我现在要介绍的核心部分，因为我在后续章节中会反复引用它们。

Now that we have the engine initialized let's proceed to the generate function.

Generate functionGenerate 函数

The first step is to validate and feed requests into the engine. For each prompt we:第一步是验证并将请求输入引擎。对于每个提示词，我们：

1. Create a unique request ID and capture its arrival time创建一个唯一的请求 ID 并记录其到达时间
2. Call an input preprocessor that tokenizes the prompt and returns a dictionary containing prompt, prompt_token_ids, and a type (text, tokens, embeds, etc.)调用输入预处理器，对提示词进行分词，并返回一个包含提示词、prompt_token_ids 和类型（文本、Token、嵌入等）的字典
3. Pack this info into an EngineCoreRequest, adding priority, sampling params, and other metadata将此信息打包到 EngineCoreRequest 中，添加优先级、采样参数和其他元数据
4. Pass the request into the engine core, which wraps it in a Request object and sets its status to WAITING. This request is then added to the scheduler's waiting queue (append if FCFS, or heap-push if priority)将请求传递给引擎核心，它将其包装在 Request 对象中并将状态设置为 WAITING。然后将此请求添加到调度器的等待队列中（如果是 FCFS 则追加，如果是优先级则堆推入）

At this point the engine has been fed and execution can begin. In the synchronous engine example, these initial prompts are the only ones we'll process — there's no mechanism to inject new requests mid-run. In contrast, the asynchronous engine supports this (aka continuous batching [6]): after each step, both new and old requests are considered.此时引擎已被填充，执行可以开始。在同步引擎示例中，这些初始提示词是我们处理的唯一内容——没有在运行中注入新请求的机制。相比之下，异步引擎支持这一点（即连续批处理 [6]）：在每一步之后，新旧请求都会被考虑在内。

Next, as long as there are requests to process, the engine repeatedly calls its step() function. Each step has three stages:接下来，只要有要处理的请求，引擎就会重复调用其 step() 函数。每一步都有三个阶段：

1. Schedule: select which requests to run in this step (decode, and/or (chunked) prefill)调度：选择在这一步中运行哪些请求（解码，和/或（分块）预填充）
2. Forward pass: run the model and sample tokens前向传播：运行模型并采样 Token
3. Postprocess: append sampled token IDs to each Request, detokenize, and check stop conditions. If a request is finished, clean up (e.g. return its KV-cache blocks to free_block_queue) and return the output early后处理：将采样的 Token ID 追加到每个 Request，反分词，并检查停止条件。如果请求完成，则进行清理（例如将 KV 缓存块返回给 free_block_queue）并提前返回输出

Next, we'll examine scheduling in more detail.接下来，我们将更详细地检查调度。

Scheduler调度器

There are two main types of workloads an inference engine handles:推理引擎处理的负载主要有两种类型：

1. Prefill requests — a forward pass over all prompt tokens. These are usually compute-bound (threshold depends on hardware and prompt length). At the end, we sample a single token from the probability distribution of the final token's position.预填充（Prefill）请求——对所有提示词 Token 进行前向传播。这些通常是计算密集型的（阈值取决于硬件和提示词长度）。最后，我们从最后一个 Token 位置的概率分布中采样一个 Token。
2. Decode requests — a forward pass over just the most recent token. All earlier KV vectors are already cached. These are memory-bandwidth-bound, since we still need to load all LLM weights (and KV caches) just to compute one token.解码（Decode）请求——仅对最近的一个 Token 进行前向传播。所有之前的 KV 向量都已缓存。这些是内存带宽受限的，因为我们仍然需要加载所有 LLM 权重（和 KV 缓存）来计算一个 Token。

The V1 scheduler can mix both types of requests in the same step, thanks to smarter design choices. In contrast, the V0 engine could only process either prefill or decode at once.得益于更智能的设计选择，V1 调度器可以在同一步骤中混合两种类型的请求。相比之下，V0 引擎一次只能处理预填充或解码。

The scheduler prioritizes decode requests — i.e. those already in the running queue. For each such request it:

1. Computes the number of new tokens to generate (not always 1, due to speculative decoding and async scheduling — more on that later).计算要生成的新 Token 数量（由于投机解码和异步调度，不一定是 1 个——稍后详述）。
2. Calls the KV-cache manager's allocate_slots function (details below).调用 KV 缓存管理器的 allocate_slots 函数（详情见下文）。
3. Updates the token budget by subtracting the number of tokens from step 1.通过减去第 1 步中的 Token 数量来更新 Token 预算。

After that, it processes prefill requests from the waiting queue, it:

1. Retrieves the number of computed blocks (returns 0 if prefix caching is disabled — we'll cover that later).检索已计算的块数量（如果禁用了前缀缓存则返回 0——稍后讨论）。
2. Calls the KV-cache manager's allocate_slots function.调用 KV 缓存管理器的 allocate_slots 函数。
3. Pops the request from waiting and moves it to running, setting its status to RUNNING.将请求从等待队列中弹出并移至运行队列，将其状态设置为 RUNNING。
4. Updates the token budget.更新 Token 预算。

Let's now look at what allocate_slots does, it:

1. Computes number of blocks — determines how many new KV-cache blocks (n) must be allocated. Each block stores 16 tokens by default. For example, if a prefill request has 17 new tokens, we need ceil(17/16) = 2 blocks.计算块数量——确定必须分配多少新的 KV 缓存块 (n)。每个块默认存储 16 个 Token。例如，如果预填充请求有 17 个新 Token，我们需要 ceil(17/16) = 2 个块。
2. Checks availability — if there aren't enough blocks in the manager's pool, exit early. Depending on whether it's a decode or prefill request, the engine may attempt recompute preemption (swap preemption was supported in V0) by evicting low-priority requests (calling kv_cache_manager.free which returns KV blocks to block pool), or it might skip scheduling and continue execution.检查可用性——如果管理器池中没有足够的块，则提前退出。根据是解码还是预填充请求，引擎可能会尝试重新计算抢占（V0 支持交换抢占），通过驱逐低优先级请求（调用 kv_cache_manager.free 将 KV 块返回给块池），或者可能会跳过调度并继续执行。
3. Allocates blocks — via the KV-cache manager's coordinator, fetches the first n blocks from the block pool (the free_block_queue doubly linked list mentioned earlier). Stores to req_to_blocks, the dictionary mapping each request_id to its list of KV-cache blocks.分配块——通过 KV 缓存管理器的协调器，从块池（前面提到的 free_block_queue 双向链表）中获取前 n 个块。存储到 req_to_blocks，这是一个将每个 request_id 映射到其 KV 缓存块列表的字典。

We're finally ready to do a forward pass!

Run forward pass运行前向传播

We call model executor's execute_model, which delegates to the Worker, which in turn delegates to the model runner.我们调用模型执行器的 execute_model，它委托给 Worker，Worker 再委托给模型运行器。

Here are the main steps:主要步骤如下：

1. Update states — prune finished requests from input_batch; update misc fwd pass related metadata (e.g., KV cache blocks per request that will be used to index into paged KV cache memory).更新状态——从 input_batch 中剔除已完成的请求；更新与前向传播相关的杂项元数据（例如，每个请求将用于索引分页 KV 缓存内存的 KV 缓存块）。
2. Prepare inputs — copy buffers from CPU→GPU; compute positions; build slot_mapping (more on that in example); construct attention metadata.准备输入——将缓冲区从 CPU 复制到 GPU；计算位置；构建 slot_mapping；构建注意力元数据。
3. Forward pass — run the model with custom paged attn kernels. All sequences are flattened and concatenated into one long "super sequence". Position indices and attention masks ensure each sequence only attends to its own tokens, which enables continuous batching without right-padding.前向传播——使用自定义分页注意力内核运行模型。所有序列被展平并连接成一个长的“超级序列”。位置索引和注意力掩码确保每个序列只关注自己的 Token，这使得无需右填充（Right-padding）即可实现连续批处理。
4. Gather last-token states — extract hidden states for each sequence's final position and compute logits.收集最后一个 Token 的状态——提取每个序列最终位置的隐藏状态并计算 Logits。
5. Sample — sample tokens from computed logits as dictated by the sampling config (greedy, temperature, top-p, top-k, etc.).采样——按照采样配置（贪婪、温度、Top-p、Top-k 等）从计算出的 Logits 中采样 Token。

Forward-pass step itself has two execution modes:前向传播步骤本身有两种执行模式：

1. Eager mode — run the standard PyTorch forward pass when eager execution is enabled.Eager 模式——启用 Eager 执行时运行标准的 PyTorch 前向传播。
2. "Captured" mode — execute/replay a pre-captured CUDA Graph when eager is not enforced (remember we captured these during engine construction in the initialize KV cache procedure).“Captured”模式——当未强制使用 Eager 时，执行/重放预先捕获的 CUDA Graph（记住我们在引擎构建期间的初始化 KV 缓存过程中捕获了这些）。

Here is a concrete example that should make continuous batching and paged attention clear:这是一个具体的示例，应该能让你清楚地理解连续批处理和分页注意力：

Advanced Features — extending the core engine logic高级功能——扩展核心引擎逻辑

With the basic engine flow in place, we can now look at the advanced features.有了基本的引擎流程，我们现在可以看看高级功能了。

We've already discussed preemption, paged attention, and continuous batching.我们已经讨论了抢占、分页注意力和连续批处理。

Next, we'll dive into:接下来，我们将深入探讨：

1. Chunked prefill分块预填充
2. Prefix caching前缀缓存
3. Guided decoding (through grammar-constrained finite-state machines)引导式解码（通过语法约束的有限状态机）
4. Speculative decoding投机解码
5. Disaggregated P/D (prefill/decoding)解耦式 P/D（预填充/解码）

Chunked prefill

Chunked prefill is a technique for handling long prompts by splitting their prefill step into smaller chunks. Without it, we could end up with a single very long request monopolizing one engine step disallowing other prefill requests to run. That would postpone all other requests and increase their latency.分块预填充是一种通过将预填充步骤拆分为较小块来处理长提示词的技术。如果没有它，我们可能会遇到一个非常长的请求垄断一个引擎步骤，从而不允许其他预填充请求运行。这将推迟所有其他请求并增加它们的延迟。

For example, let each chunk contain n (=8) tokens, labeled with lowercase letters separated by "-". A long prompt P could look like x-y-z, where z is an incomplete chunk (e.g. 2 toks). Executing the full prefill for P would then take ≥ 3 engine steps (> can happen if it's not scheduled for execution in one of the steps), and only in the last chunked prefill step would we sample one new token.例如，让每个块包含 n (=8) 个 Token，用“-”分隔的小写字母标记。一个长提示词 P 可能看起来像 x-y-z，其中 z 是一个不完整的块（例如 2 个 Token）。为 P 执行完整的预填充将需要 ≥ 3 个引擎步骤（如果它没有在其中一个步骤中被调度执行，则可能会更多），并且只有在最后一个分块预填充步骤中，我们才会采样一个新的 Token。

Here is that same example visually:这是该示例的视觉呈现：

Implementation is straightforward: cap the number of new tokens per step. If the requested number exceeds long_prefill_token_threshold, reset it to exactly that value. The underlying indexing logic (described earlier) takes care of the rest.实现很简单：限制每一步的新 Token 数量。如果请求的数量超过 long_prefill_token_threshold，则将其重置为该精确值。底层的索引逻辑（前面描述过）负责处理其余部分。

In vLLM V1, you enable chunked prefill by setting long_prefill_token_threshold to a positive integer. (Technically, it can happen irrespective of this, if the prompt length exceeds the token budget we truncate it and run a chunked prefill.)在 vLLM V1 中，你可以通过将 long_prefill_token_threshold 设置为正整数来启用分块预填充。（从技术上讲，如果提示词长度超过 Token 预算，我们截断它并运行分块预填充，无论是否设置该参数，这都会发生。）

Prefix Caching前缀缓存

To explain how prefix caching works, let's take the original code example and tweak it a bit:为了解释前缀缓存的工作原理，让我们对原始代码示例进行一些微调：

from vllm import LLM, SamplingParams

long_prefix = "<a piece of text that is encoded into more than block_size tokens>"

prompts = [
    "Hello, my name is",
    "The president of the United States is",
]

sampling_params = SamplingParams(temperature=0.8, top_p=0.95)

def main():
    llm = LLM(model="TinyLlama/TinyLlama-1.1B-Chat-v1.0")

    outputs = llm.generate(long_prefix + prompts[0], sampling_params)
    outputs = llm.generate(long_prefix + prompts[1], sampling_params)

if __name__ == "__main__":
    main()

Prefix caching avoids recomputing tokens that multiple prompts share at the beginning - hence prefix.前缀缓存避免了重新计算多个提示词在开头共享的 Token——因此得名“前缀”。

The crucial piece is the long_prefix: it's defined as any prefix longer than a KV-cache block (16 tokens by default). To simplify our example let's say long_prefix has exactly length n x block_size (where n ≥ 1).关键部分是 long_prefix：它被定义为任何长于 KV 缓存块（默认 16 个 Token）的前缀。为了简化示例，假设 long_prefix 的长度正好是 n x block_size（其中 n ≥ 1）。

Without prefix caching, each time we process a new request with the same long_prefix, we'd recompute all n x block_size tokens.如果没有前缀缓存，每次我们处理具有相同 long_prefix 的新请求时，我们都会重新计算所有 n x block_size 个 Token。

With prefix caching, those tokens are computed once (their KVs stored in KV cache paged memory) and then reused, so only the new prompt tokens need processing. This speeds up prefill requests (though it doesn't help with decode).有了前缀缓存，这些 Token 只计算一次（它们的 KV 存储在 KV 缓存分页内存中）然后被重用，因此只需要处理新的提示词 Token。这加快了预填充请求的速度（尽管对解码没有帮助）。

How does this work in vLLM?这在 vLLM 中是如何工作的？

During the first generate call, in the scheduling stage, inside kv_cache_manager.get_computed_blocks, the engine invokes hash_request_tokens:在第一次 generate 调用期间，在调度阶段，在 kv_cache_manager.get_computed_blocks 内部，引擎调用 hash_request_tokens：

1. This function splits the long_prefix + prompts[0] into 16-token chunks.此函数将 long_prefix + prompts[0] 拆分为 16 个 Token 的块。
2. For each complete chunk, it computes a hash (using either the built-in hash or SHA-256, which is slower but has fewer collisions). The hash combines the previous block's hash, the current tokens, and optional metadata.对于每个完整块，它计算一个哈希值（使用内置哈希或 SHA-256，后者较慢但冲突较少）。该哈希结合了前一个块的哈希、当前 Token 和可选元数据。
optional metadata includes: MM hash, LoRA ID, cache salt (injected into hash of the first block ensures only requests with this cache salt can reuse blocks).可选元数据包括：MM 哈希、LoRA ID、缓存盐（注入到第一个块的哈希中，确保只有具有此缓存盐的请求才能重用块）。
3. Each result is stored as a BlockHash object containing both the hash and its token IDs. We return a list of block hashes.每个结果都存储为一个 BlockHash 对象，包含哈希及其 Token ID。我们返回一个块哈希列表。

The list is stored in self.req_to_block_hashes[request_id].该列表存储在 self.req_to_block_hashes[request_id] 中。

Next, the engine calls find_longest_cache_hit to check if any of these hashes already exist in cached_block_hash_to_block. On the first request, no hits are found.接下来，引擎调用 find_longest_cache_hit 来检查这些哈希是否已存在于 cached_block_hash_to_block 中。在第一个请求中，找不到匹配项。

Then we call allocate_slots which calls coordinator.cache_blocks, which associates the new BlockHash entries with allocated KV blocks and records them in cached_block_hash_to_block.然后我们调用 allocate_slots，它调用 coordinator.cache_blocks，将新的 BlockHash 条目与分配的 KV 块关联，并记录在 cached_block_hash_to_block 中。

Afterwards, the forward pass will populate KVs in paged KV cache memory corresponding to KV cache blocks that we allocated above.之后，前向传播将填充分页 KV 缓存内存中的 KV，对应于我们上面分配的 KV 缓存块。

On a second generate call with the same prefix, steps 1-3 repeat, but now find_longest_cache_hit finds matches for all n blocks (via linear search). The engine can reuse those KV blocks directly.在具有相同前缀的第二次 generate 调用中，步骤 1-3 重复，但现在 find_longest_cache_hit 找到了所有 n 个块的匹配项（通过线性搜索）。引擎可以直接重用这些 KV 块。

If the original request were still alive, the reference count for those blocks would increment (e.g. to 2). In this example, the first request has already completed, so the blocks were freed back to the pool and their reference counts set back to 0. Because we were able to retrieve them from cached_block_hash_to_block we know they're valid (the logic of the KV cache manager is setup in such a way), so we just remove them from free_block_queue again.如果原始请求仍然存活，这些块的引用计数将增加（例如增加到 2）。在此示例中，第一个请求已经完成，因此块被释放回池中，引用计数重置为 0。因为我们能够从 cached_block_hash_to_block 中检索它们，所以我们知道它们是有效的（KV 缓存管理器的逻辑就是这样设置的），所以我们只是再次将它们从 free_block_queue 中移除。

And that's the gist of prefix caching: don't recompute prefixes you've already seen — just reuse their KV cache!这就是前缀缓存的要点：不要重新计算你已经见过的前缀——直接重用它们的 KV 缓存！

Prefix caching is enabled by default. To disable it: enable_prefix_caching = False.前缀缓存默认启用。要禁用它：enable_prefix_caching = False。

Guided Decoding (FSM)引导式解码 (FSM)

Guided decoding is a technique where, at each decoding step, the logits are constrained by a grammar-based finite state machine. This ensures that only tokens allowed by the grammar can be sampled.引导式解码是一种技术，在每个解码步骤中，Logits 都受到基于语法的有限状态机的约束。这确保了只有语法允许的 Token 才能被采样。

It's a powerful setup: you can enforce anything from regular grammars (Chomsky type-3, e.g. arbitrary regex patterns) all the way up to context-free grammars (type-2, which cover most programming languages).这是一个强大的设置：你可以强制执行从正则语法（Chomsky 3 型，例如任意正则表达式）一直到上下文无关语法（2 型，涵盖大多数编程语言）的任何内容。

To make this less abstract, let's start with the simplest possible example, building on our earlier code:为了让它不那么抽象，让我们从最简单的例子开始，基于我们之前的代码：

from vllm import LLM, SamplingParams
from vllm.sampling_params import GuidedDecodingParams

prompts = [
    "This sucks",
    "The weather is beautiful",
]

guided_decoding_params = GuidedDecodingParams(choice=["Positive", "Negative"])
sampling_params = SamplingParams(guided_decoding=guided_decoding_params)

def main():
    llm = LLM(model="TinyLlama/TinyLlama-1.1B-Chat-v1.0")

    outputs = llm.generate(prompts, sampling_params)

if __name__ == "__main__":
    main()

In the toy example I gave (assume character-level tokenization): at prefill, the FSM masks logits so only "P" or "N" are viable. If "P" is sampled, the FSM moves to the "Positive" branch; next step only "o" is allowed, and so on.在我给出的玩具示例中（假设字符级分词）：在预填充时，FSM 掩盖 Logits，因此只有“P”或“N”是可行的。如果采样到“P”，FSM 移动到“Positive”分支；下一步只允许“o”，依此类推。

How this works in vLLM:这在 vLLM 中是如何工作的：

1. At LLM engine construction, a StructuredOutputManager is created; it has access to the tokenizer and maintains a _grammar_bitmask tensor.在 LLM 引擎构建时，创建了一个 StructuredOutputManager；它有权访问分词器并维护一个 _grammar_bitmask 张量。
2. When adding a request, its status is set to WAITING_FOR_FSM and grammar_init selects the backend compiler (e.g., xgrammar [7]; note that backends are 3rd party code).添加请求时，其状态设置为 WAITING_FOR_FSM，grammar_init 选择后端编译器（例如 xgrammar [7]；注意后端是第三方代码）。
3. The grammar for this request is compiled asynchronously.此请求的语法被异步编译。
4. During scheduling, if the async compile has completed, the status switches to WAITING and request_id is added to structured_output_request_ids; otherwise it's placed in skipped_waiting_requests to retry on next engine step.在调度期间，如果异步编译已完成，状态切换为 WAITING，request_id 被添加到 structured_output_request_ids；否则它被放置在 skipped_waiting_requests 中，以便在下一个引擎步骤重试。
5. After the scheduling loop (still inside scheduling), if there are FSM requests, the StructuredOutputManager asks the backend to prepare/update _grammar_bitmask.在调度循环之后（仍在调度内部），如果有 FSM 请求，StructuredOutputManager 会要求后端准备/更新 _grammar_bitmask。
6. After the forward pass produces logits, xgr_torch_compile's function expands the bitmask to vocab size (32x expansion ratio because we use 32 bit integers) and masks disallowed logits to –∞.在前向传播产生 Logits 后，xgr_torch_compile 的函数将位掩码扩展到词汇表大小（32 倍扩展比，因为我们使用 32 位整数），并将禁止的 Logits 掩盖为 –∞。
7. After sampling the next token, the request's FSM is advanced via accept_tokens. Visually we move to the next state on the FSM diagram.采样下一个 Token 后，请求的 FSM 通过 accept_tokens 推进。视觉上，我们移动到 FSM 图上的下一个状态。

Step 6 deserves further clarification.第 6 步值得进一步澄清。

If vocab_size = 32, _grammar_bitmask is a single integer; its binary representation encodes which tokens are allowed ("1") vs disallowed ("0"). For example, "101…001" expands to a length-32 array [1, 0, 1, …, 0, 0, 1]; positions with 0 get logits set to –∞. For larger vocabularies, multiple 32-bit words are used and expanded/concatenated accordingly. The backend (e.g., xgrammar) is responsible for producing these bit patterns using the current FSM state.如果 vocab_size = 32，_grammar_bitmask 是一个整数；其二进制表示编码了哪些 Token 是允许的（“1”）与禁止的（“0”）。例如，“101…001”扩展为长度为 32 的数组 [1, 0, 1, …, 0, 0, 1]；位置为 0 的 Logits 设置为 –∞。对于更大的词汇表，使用多个 32 位字并相应地扩展/连接。后端（例如 xgrammar）负责使用当前的 FSM 状态生成这些位模式。

Here is an even simpler example with vocab_size = 8 and 8-bit integers (for those of you who like my visuals):这是一个 vocab_size = 8 和 8 位整数的更简单示例（给喜欢我图示的读者）：

You can enable this in vLLM by passing in a desired guided_decoding config.你可以通过传入所需的 guided_decoding 配置在 vLLM 中启用此功能。

Speculative Decoding投机解码

In autoregressive generation, each new token requires a forward pass of the large LM. This is expensive — every step reloads and applies all model weights just to compute a single token! (assuming batch size == 1, in general it's B)在自回归生成中，每个新 Token 都需要大型 LM 的前向传播。这很昂贵——每一步都重新加载并应用所有模型权重，仅仅为了计算一个 Token！（假设 batch size == 1，通常为 B）。

Speculative decoding [8] speeds this up by introducing a smaller draft LM. The draft proposes k tokens cheaply. But we don't ultimately want to sample from the smaller model — it's only there to guess candidate continuations. The large model still decides what's valid.投机解码 [8] 通过引入一个较小的草稿 LM 加速了这一点。草稿模型廉价地提出 k 个 Token。但我们最终不想从较小的模型中采样——它只是为了猜测候选续写。大模型仍然决定什么是有效的。

Here are the steps:步骤如下：

1. Draft: run the small model on the current context and propose k tokens草稿：在当前上下文上运行小模型并提出 k 个 Token。
2. Verify: run the large model once on context + k draft tokens. This produces probabilities for those k positions plus one extra (so we get k+1 candidates)验证：在上下文 + k 个草稿 Token 上运行大模型一次。这为那 k 个位置加上一个额外的位置产生概率（所以我们得到 k+1 个候选）。
3. Accept/reject: going from left to right over the k draft tokens:
   • If the large model's probability for the draft token ≥ the draft's probability, accept it如果大模型对草稿 Token 的概率 ≥ 草稿的概率，则接受它。
   • Otherwise, accept it with probability p_large(token)/p_draft(token)否则，以 p_large(token)/p_draft(token) 的概率接受它。
   • Stop at the first rejection, or accept all k draft tokens.在第一次拒绝时停止，或接受所有 k 个草稿 Token。
   • If all k draft tokens are accepted, also sample the extra (k+1)-th token "for free" from the large model (we already computed that distribution).如果所有 k 个草稿 Token 都被接受，也从大模型中“免费”采样额外的 (k+1)-th Token（我们已经计算了该分布）。
   • If there was a rejection create a new rebalanced distribution at that position (p_large - p_draft, clamp min at 0, normalize to sum to 1) and sample the last token from it.如果发生了拒绝，在该位置创建一个新的重新平衡分布（p_large - p_draft，最小值设为 0，归一化为总和为 1）并从中采样最后一个 Token。

Why this works: Although we use the small model to propose candidates, the accept/reject rule guarantees that in expectation the sequence is distributed exactly as if we had sampled token by token from the large model. This means speculative decoding is statistically equivalent to standard autoregressive decoding — but potentially much faster, since a single large-model pass can yield up to k+1 tokens.为什么这有效：虽然我们使用小模型提出候选，但接受/拒绝规则保证了在期望上，序列的分布与我们从大模型中逐个 Token 采样完全相同。这意味着投机解码在统计上等同于标准的自回归解码——但可能快得多，因为单次大模型传递可以产生多达 k+1 个 Token。

vLLM V1 does not support the LLM draft model method, instead it implements faster—but less accurate—proposal schemes: n-gram, EAGLE [9], and Medusa [10].vLLM V1 不支持 LLM 草稿模型方法，而是实现了更快但不太准确的提议方案：n-gram、EAGLE [9] 和 Medusa [10]。

One-liners on each:每个方案的一句话介绍：

1. n-gram: take the last prompt_lookup_max tokens; find a prior match in the sequence; if found, propose the k tokens that followed that match; otherwise decrement the window and retry down to prompt_lookup_minn-gram：获取最后的 prompt_lookup_max 个 Token；在序列中找到之前的匹配；如果找到，提出匹配后的 k 个 Token；否则减少窗口并重试，直到 prompt_lookup_min。
The current implementation returns k tokens after the first match. It feels more natural to introduce a recency bias and reverse the search direction? (i.e. last match)
2. Eagle: perform "model surgery" on the large LM—keep embeddings and LM head, replace the transformer stack with a lightweight MLP; fine-tune that as a cheap draftEagle：对大模型进行“模型手术”——保留嵌入和 LM 头，用轻量级 MLP 替换 Transformer 堆栈；将其微调为廉价的草稿。
3. Medusa: train auxiliary linear heads on top (embeddings before LM head) of the large model to predict the next k tokens in parallel; use these heads to propose tokens more efficiently than running a separate small LMMedusa：在大模型顶部（LM 头之前的嵌入）训练辅助线性头，以并行预测接下来的 k 个 Token；使用这些头比运行单独的小 LM 更有效地提出 Token。

Here's how to invoke speculative decoding in vLLM using ngram as the draft method:

from vllm import LLM, SamplingParams

prompts = [
    "Hello, my name is",
    "The president of the United States is",
]

sampling_params = SamplingParams(temperature=0.8, top_p=0.95)

speculative_config={
    "method": "ngram",
    "prompt_lookup_max": 5,
    "prompt_lookup_min": 3,
    "num_speculative_tokens": 3,
}

def main():
    llm = LLM(model="TinyLlama/TinyLlama-1.1B-Chat-v1.0", speculative_config=speculative_config)

    outputs = llm.generate(prompts, sampling_params)

if __name__ == "__main__":
    main()

How does this work in vLLM?

Setup (during engine construction):设置（在引擎构建期间）：

1. Init device: create a drafter (draft model, e.g., NgramProposer) and a rejection_sampler (parts of it are written in Triton).初始化设备：创建一个草稿器（草稿模型，例如 NgramProposer）和一个拒绝采样器（部分代码用 Triton 编写）。
2. Load model: load draft model weights (no-op for n-gram).加载模型：加载草稿模型权重（n-gram 为空操作）。

After that in the generate function (assume we get a brand new request):之后在 generate 函数中（假设我们得到了一个全新的请求）：

1. Run the regular prefill step with the large model.用大模型运行常规预填充步骤。
2. After the forward pass and standard sampling, call propose_draft_token_ids(k) to sample k draft tokens from the draft model.在前向传播和标准采样后，调用 propose_draft_token_ids(k) 从草稿模型中采样 k 个草稿 Token。
3. Store these in request.spec_token_ids (update the request metadata).将这些存储在 request.spec_token_ids 中（更新请求元数据）。
4. On the next engine step, when the request is in the running queue, add len(request.spec_token_ids) to the "new tokens" count so allocate_slots reserves sufficient KV blocks for the fwd pass.在下一个引擎步骤中，当请求处于运行队列时，将 len(request.spec_token_ids) 加到“新 token”计数中，以便 allocate_slots 为前向传播预留足够的 KV 块。
5. Copy spec_token_ids into input_batch.token_ids_cpu to form (context + draft) tokens.将 spec_token_ids 复制到 input_batch.token_ids_cpu 中，以构成（上下文 + 草稿）token。
6. Compute metadata via _calc_spec_decode_metadata (this copies over tokens from input_batch.token_ids_cpu, prepares logits, etc.), then run a large-model forward pass over the draft tokens.通过 _calc_spec_decode_metadata 计算元数据（这会从 input_batch.token_ids_cpu 复制 token、准备 logits 等），然后对草稿 token 运行大模型前向传播。
7. Instead of regular sampling from logits, use the rejection_sampler to accept/reject left-to-right and produce output_token_ids.使用 rejection_sampler 而不是从 logits 进行常规采样，以从左到右进行接受/拒绝判断并生成 output_token_ids。
8. Repeat steps 2-7 until a stop condition is met.重复步骤 2-7，直到满足停止条件。

The best way to internalize this is to fire up your debugger and step through the codebase, but this section hopefully gives you a taste for it. This as well:

Disaggregated P/D解耦式 P/D（预填充/解码）

I've already previously hinted at the motivation behind disaggregated P/D (prefill/decode).我之前已经暗示过解耦式 P/D（预填充/解码）背后的动机。

Prefill and decode have very different performance profiles (compute-bound vs. memory-bandwidth-bound), so separating their execution is a sensible design. It gives tighter control over latency — both TTFT (time-to-first-token) and ITL (inter-token latency) — more on this in the benchmarking section.预填充（Prefill）和解码（Decode）具有截然不同的性能特征（计算密集型 vs. 内存带宽密集型），因此将它们的执行过程分离开来是一种明智的设计。它能更精准地控制延迟——包括 TTFT（首字延迟）和 ITL（token 间延迟）——基准测试部分会有更多相关介绍。

In practice, we run N vLLM prefill instances and M vLLM decode instances, autoscaling them based on the live request mix. Prefill workers write KV to a dedicated KV-cache service; decode workers read from it. This isolates long, bursty prefill from steady, latency-sensitive decode.在实践中，我们运行 N 个 vLLM 预填充实例和 M 个 vLLM 解码实例，并根据实时请求组合对它们进行自动扩缩容。预填充 worker 将 KV 写入专用的 KV 缓存服务；解码 worker 则从中读取。这可以将长且突发性的预填充任务与稳定、对延迟敏感的解码任务隔离开来。

How does this work in vLLM?

For clarity, the example below relies on SharedStorageConnector, a debugging connector implementation used to illustrate the mechanics.为清晰起见，下文示例依赖于 SharedStorageConnector，这是一种用于演示机制的调试连接器实现。

We launch 2 vLLM instances (GPU 0 for prefill and GPU 1 for decode), and then transfer the KV cache between them:我们启动 2 个 vLLM 实例（GPU 0 用于预填充，GPU 1 用于解码），然后在它们之间传输 KV 缓存：

import os
import time
from multiprocessing import Event, Process
import multiprocessing as mp

from vllm import LLM, SamplingParams
from vllm.config import KVTransferConfig

prompts = [
    "Hello, my name is",
    "The president of the United States is",
]

def run_prefill(prefill_done):
  os.environ["CUDA_VISIBLE_DEVICES"] = "0"

  sampling_params = SamplingParams(temperature=0, top_p=0.95, max_tokens=1)

  ktc=KVTransferConfig(
      kv_connector="SharedStorageConnector",
      kv_role="kv_both",
      kv_connector_extra_config={"shared_storage_path": "local_storage"},
  )

  llm = LLM(model="TinyLlama/TinyLlama-1.1B-Chat-v1.0", kv_transfer_config=ktc)
  llm.generate(prompts, sampling_params)

  prefill_done.set()  # notify decode instance that KV cache is ready

  # To keep the prefill node running in case the decode node is not done;
  # otherwise, the script might exit prematurely, causing incomplete decoding.
  try:
      while True:
          time.sleep(1)
  except KeyboardInterrupt:
      print("Script stopped by user.")

def run_decode(prefill_done):
  os.environ["CUDA_VISIBLE_DEVICES"] = "1"

  sampling_params = SamplingParams(temperature=0, top_p=0.95)

  ktc=KVTransferConfig(
      kv_connector="SharedStorageConnector",
      kv_role="kv_both",
      kv_connector_extra_config={"shared_storage_path": "local_storage"},
  )

  llm = LLM(model="TinyLlama/TinyLlama-1.1B-Chat-v1.0", kv_transfer_config=ktc)

  prefill_done.wait()  # block waiting for KV cache from prefill instance

  # Internally it'll first fetch KV cache before starting the decoding loop
  outputs = llm.generate(prompts, sampling_params)

if __name__ == "__main__":
  prefill_done = Event()
  prefill_process = Process(target=run_prefill, args=(prefill_done,))
  decode_process = Process(target=run_decode, args=(prefill_done,))

  prefill_process.start()
  decode_process.start()

  decode_process.join()
  prefill_process.terminate()

These are the steps in vLLM:以下是 vLLM 中的步骤：

1. Instantiation — During engine construction, connectors are created in two places:
   • Inside the worker's init device procedure (under init worker distributed environment function), with role "worker".在 worker 的初始化设备过程中（位于 init worker distributed environment 函数下），角色为“worker”。
   • Inside the scheduler constructor, with role "scheduler".在调度器构造函数中，角色为“scheduler”。
2. Cache lookup — When the scheduler processes prefill requests from the waiting queue (after local prefix-cache checks), it calls connector's get_num_new_matched_tokens. This checks for externally cached tokens in the KV-cache server. Prefill always sees 0 here; decode may have a cache hit. The result is added to the local count before calling allocate_slots.缓存查找 — 当调度器处理等待队列中的预填充请求时（在本地前缀缓存检查之后），它会调用连接器的 get_num_new_matched_tokens。这会检查 KV 缓存服务器中是否存在外部缓存的 token。预填充在此处总是看到 0；而解码可能会命中缓存。结果在调用 allocate_slots 之前被加到本地计数中。
3. State update — The scheduler then calls connector.update_state_after_alloc, which records requests that had a cache (no-op for prefill).状态更新 — 调度器随后调用 connector.update_state_after_alloc，记录有缓存的请求（预填充无此操作）。
4. Meta build — At the end of scheduling, the scheduler calls meta = connector.build_connector_meta:
   • Prefill adds all requests with is_store=True (to upload KV).预填充添加所有 is_store=True 的请求（用于上传 KV）。
   • Decode adds requests with is_store=False (to fetch KV).解码添加 is_store=False 的请求（用于获取 KV）。
5. Context manager — Before the forward pass, the engine enters a KV-connector context manager:
   • On enter: kv_connector.start_load_kv is called. For decode, this loads KV from the external server and injects it into paged memory. For prefill, it's a no-op.进入时：调用 kv_connector.start_load_kv。对于解码，这会从外部服务器加载 KV 并注入到分页内存中。对于预填充，这是空操作。
   • On exit: kv_connector.wait_for_save is called. For prefill, this blocks until KV is uploaded to the external server. For decode, it's a no-op.退出时：调用 kv_connector.wait_for_save。对于预填充，这会阻塞直到 KV 上传到外部服务器。对于解码，这是空操作。

Here is a visual example:这是一个直观的示例：

From UniprocExecutor to MultiProcExecutor从 UniprocExecutor 到 MultiProcExecutor

With the core techniques in place, we can now talk about scaling up.掌握了核心技术后，我们现在可以谈谈如何进行扩展。

Suppose your model weights no longer fit into a single GPU's VRAM.假设你的模型权重不再能装进单个 GPU 的显存中。

The first option is to shard the model across multiple GPUs on the same node using tensor parallelism (e.g., TP=8). If the model still doesn't fit, the next step is pipeline parallelism across nodes.第一个选择是使用张量并行（例如 TP=8）将模型分片到同一节点的多个 GPU 上。如果模型仍然装不下，下一步就是跨节点的流水线并行。

At this stage, we need multiple GPU processes (workers) and an orchestration layer to coordinate them. That's exactly what MultiProcExecutor provides.在这一阶段，我们需要多个 GPU 进程（worker）和一个编排层来协调它们。这正是 MultiProcExecutor 所提供的。

How this works in vLLM:

1. MultiProcExecutor initializes an rpc_broadcast_mq message queue (implemented with shared memory under the hood).MultiProcExecutor 初始化一个 rpc_broadcast_mq 消息队列（底层使用共享内存实现）。
2. The constructor loops over world_size (e.g. TP=8 ⇒ world_size=8) and spawns a daemon process for each rank via WorkerProc.make_worker_process.构造函数遍历 world_size（例如 TP=8 ⇒ world_size=8），并通过 WorkerProc.make_worker_process 为每个 rank 派生一个守护进程。
3. For each worker, the parent first creates a reader and writer pipe.对于每个 worker，父进程首先创建一个读取管道和一个写入管道。
4. The new process runs WorkerProc.worker_main, which instantiates a worker (going through the same "init device", "load model", etc. as in UniprocExecutor).新进程运行 WorkerProc.worker_main，它实例化一个 worker（经历与 UniprocExecutor 相同的“初始化设备”、“加载模型”等过程）。
5. Each worker determines whether it is the driver (rank 0 in the TP group) or a regular worker. Every worker sets up two queues:
   • rpc_broadcast_mq (shared with the parent) for receiving work.rpc_broadcast_mq（与父进程共享）用于接收工作。
   • worker_response_mq for sending responses back.worker_response_mq 用于发送响应。
6. During initialization, each child sends its worker_response_mq handle to the parent via the pipe. Once all are received, the parent unblocks — this completes coordination.初始化期间，每个子进程通过管道将 worker_response_mq 句柄发送给父进程。一旦全部接收完毕，父进程解除阻塞——协调完成。
7. Workers then enter a busy loop, blocking on rpc_broadcast_mq.dequeue. When a work item arrives, they execute it (just like in UniprocExecutor, but now with TP/PP-specific partitioned work). Results are sent back through worker_response_mq.enqueue.Workers 进入忙等待循环，阻塞在 rpc_broadcast_mq.dequeue 上。当工作项到达时，它们执行该项（就像在 UniprocExecutor 中一样，但现在带有 TP/PP 特定的分区工作）。结果通过 worker_response_mq.enqueue 发回。
8. At runtime, when a request arrives, MultiProcExecutor enqueues it into rpc_broadcast_mq (non-blocking) for all children workers. It then waits on the designated output rank's worker_response_mq.dequeue to collect the final result.运行时，当请求到达时，MultiProcExecutor 将其非阻塞地加入 rpc_broadcast_mq，发送给所有子 worker。然后它在指定输出 rank 的 worker_response_mq.dequeue 上等待以收集最终结果。

From the engine's perspective, nothing has changed — all of this multiprocessing complexity is abstracted away through a call to model executor's execute_model.从引擎的角度看，没有任何变化——所有这些多进程的复杂性都通过调用模型执行器的 execute_model 被抽象掉了。

• In the UniProcExecutor case: execute_model directly leads to calling execute_model on the worker在 UniProcExecutor 的情况下：execute_model 直接导致调用 worker 上的 execute_model。
• In the MultiProcExecutor case: execute_model indirectly leads to calling execute_model on each worker through rpc_broadcast_mq在 MultiProcExecutor 的情况下：execute_model 通过 rpc_broadcast_mq 间接导致调用每个 worker 上的 execute_model。

At this point, we can run models that are as large as resources allow using the same engine interface.至此，我们可以使用相同的引擎接口运行资源允许范围内的任意大模型。

The next step is to scale out: enable data parallelism (DP > 1) replicating the model across nodes, add a lightweight DP coordination layer, introduce load balancing across replicas, and place one or more API servers in front to handle incoming traffic.下一步是向外扩展：启用数据并行（DP > 1）在节点间复制模型，添加轻量级 DP 协调层，引入跨副本的负载均衡，并在前端放置一个或多个 API 服务器来处理传入流量。

Distributed system serving vLLM分布式 vLLM 服务系统

There are many ways to set up serving infrastructure, but to stay concrete, here's one example: suppose we have two H100 nodes and want to run four vLLM engines across them.搭建服务基础设施的方法有很多，为了具体说明，举个例子：假设我们有两个 H100 节点，想要在它们上面运行四个 vLLM 引擎。

If the model requires TP=4, we can configure the nodes like this.如果模型需要 TP=4，我们可以这样配置节点。

On the first node, run the engine in headless mode (no API server) with the following arguments:在第一个节点上，以无头模式（无 API 服务器）运行引擎，参数如下：

vllm serve <model-name>
  --tensor-parallel-size 4
  --data-parallel-size 4
  --data-parallel-size-local 2
  --data-parallel-start-rank 0
  --data-parallel-address <master-ip>
  --data-parallel-rpc-port 13345
  --headless

and run that same command on the other node with few tweaks:在另一个节点上运行相同的命令，只需少量调整：

• no --headless去掉 --headless
• modify DP start rank修改 DP 起始 rank

vllm serve <model-name>
  --tensor-parallel-size 4
  --data-parallel-size 4
  --data-parallel-size-local 2
  --data-parallel-start-rank 2
  --data-parallel-address <master-ip>
  --data-parallel-rpc-port 13345

How does this work in VLLM?这在 VLLM 中是如何工作的？

On the headless server node在无头服务器节点上

On the headless node, a CoreEngineProcManager launches 2 processes (per --data-parallel-size-local) each running EngineCoreProc.run_engine_core. Each of these functions creates a DPEngineCoreProc (the engine core) and then enters its busy loop.在无头节点上，CoreEngineProcManager 启动 2 个进程（根据 --data-parallel-size-local），每个进程运行 EngineCoreProc.run_engine_core。这些函数中的每一个都会创建一个 DPEngineCoreProc（引擎核心）并进入忙等待循环。

DPEngineCoreProc initializes its parent EngineCoreProc (child of EngineCore), which:DPEngineCoreProc 初始化其父级 EngineCoreProc（EngineCore 的子类），它会：

1. Creates an input_queue and output_queue (queue.Queue).创建一个 input_queue 和 output_queue（queue.Queue）。
2. Performs an initial handshake with the frontend on the other node using a DEALER ZMQ socket (async messaging lib), and receives coordination address info.使用 DEALER ZMQ 套接字（异步消息库）与另一节点的后端进行初始握手，并接收协调地址信息。
3. Initializes DP group (e.g. using NCCL backend).初始化 DP 组（例如使用 NCCL 后端）。
4. Initializes the EngineCore with MultiProcExecutor (TP=4 on 4 GPUs as described earlier).使用 MultiProcExecutor 初始化 EngineCore（如前所述，在 4 个 GPU 上 TP=4）。
5. Creates a ready_event (threading.Event).创建一个 ready_event（threading.Event）。
6. Starts an input deamon thread (threading.Thread) running process_input_sockets(…, ready_event). Similarly starts an output thread.启动一个输入守护线程（threading.Thread）运行 process_input_sockets(…, ready_event)。同样启动一个输出线程。
7. Still in the main thread, waits on ready_event until all input threads across all 4 processes (spanning the 2 nodes) have completed the coordination handshake finally executing ready_event.set().仍在主线程中，等待 ready_event，直到所有 4 个进程（跨越 2 个节点）的所有输入线程都完成了协调握手，最终执行 ready_event.set()。
8. Once unblocked, sends a "ready" message to the frontend with metadata (e.g., num_gpu_blocks available in paged KV cache memory).一旦解除阻塞，向前端发送一条“就绪”消息，附带元数据（例如，分页 KV 缓存内存中可用的 GPU 块数）。
9. The main, input, and output threads then enter their respective busy loops.主线程、输入线程和输出线程随后进入各自的忙等待循环。

TL;DR: We end up with 4 child processes (one per DP replica), each running a main, input, and output thread. They complete a coordination handshake with the DP coordinator and frontend, then all three threads per process run in steady-state busy loops.简而言之：我们最终得到了 4 个子进程（每个 DP 副本一个），每个进程运行一个主线程、一个输入线程和一个输出线程。它们与 DP 协调器和前端完成协调握手，然后每个进程的三个线程都在稳态的忙等待循环中运行。

Current steady state:当前稳态：

• Input thread — blocks on the input socket until a request is routed from the API server; upon receipt, it decodes the payload, enqueues a work item via input_queue.put_nowait(...), and returns to blocking on the socket.输入线程 — 阻塞在输入套接字上，直到从 API 服务器路由来一个请求；收到后，它解码有效负载，通过 input_queue.put_nowait(...) 加入工作项，并返回到阻塞状态。
• Main thread — wakes on input_queue.get(...), feeds the request to the engine; MultiProcExecutor runs the forward pass and enqueues results to output_queue.主线程 — 在 input_queue.get(...) 上唤醒，将请求馈送给引擎；MultiProcExecutor 运行前向传播并将结果加入 output_queue。
• Output thread — wakes on output_queue.get(...), sends the result back to the API server, then resumes blocking.输出线程 — 在 output_queue.get(...) 上唤醒，将结果发回 API 服务器，然后恢复阻塞。

Additional mechanics:其他机制：

• DP wave counter — the system tracks "waves"; when all engines become idle they quiesce, and the counter increments when new work arrives (useful for coordination/metrics).DP 波次计数器 — 系统跟踪“波次”；当所有引擎变为空闲时它们会静默，当新工作到达时计数器递增（用于协调/指标）。
• Control messages — the API server can send more than just inference requests (e.g., aborts and utility/control RPCs).控制消息 — API 服务器不仅可以发送推理请求（例如，中止和实用/控制 RPC）。
• Dummy steps for lockstep — if any DP replica has work, all replicas execute a forward step; replicas without requests perform a dummy step to participate in required synchronization points (avoids blocking the active replica).同步步骤的虚拟步骤 — 如果任何 DP 副本有工作，所有副本执行一个前向步骤；没有请求的副本执行虚拟步骤以参与必要的同步点（避免阻塞活动副本）。

Now for the second part, what happens on the API server node?

On the API server node在 API 服务器节点上

We instantiate an AsyncLLM object (an asyncio wrapper around the LLM engine). Internally this creates a DPLBAsyncMPClient (data-parallel, load-balancing, asynchronous, multiprocessing client).我们实例化一个 AsyncLLM 对象（LLM 引擎的 asyncio 包装器）。在内部，它创建一个 DPLBAsyncMPClient（数据并行、负载均衡、异步、多进程客户端）。

Inside the parent class of MPClient, the launch_core_engines function runs and:在 MPClient 的父类内部，launch_core_engines 函数运行并：

1. Creates the ZMQ addresses used for the startup handshake (as seen on the headless node).创建启动握手所使用的 ZMQ 地址（如无头节点所示）。
2. Spawns a DPCoordinator process.派生一个 DPCoordinator 进程。
3. Creates a CoreEngineProcManager (same as on the headless node).创建一个 CoreEngineProcManager（与无头节点相同）。

Inside AsyncMPClient (child of MPClient), we:在 AsyncMPClient（MPClient 的子类）内部，我们：

1. Create an outputs_queue (asyncio.Queue).创建一个 outputs_queue（asyncio.Queue）。
2. We create an asyncio task process_outputs_socket which communicates (through the output socket) with output threads of all 4 DPEngineCoreProc and writes into outputs_queue.创建一个 asyncio 任务 process_outputs_socket，它通过输出套接字与所有 4 个 DPEngineCoreProc 的输出线程通信，并写入 outputs_queue。
3. Subsequently one more asyncio task output_handler from AsyncLLM reads from this queue and finally sends out information to the create_completion function.随后，AsyncLLM 的另一个 asyncio 任务 output_handler 从该队列读取，并最终将信息发送到 create_completion 函数。

Inside DPAsyncMPClient we create an asyncio task run_engine_stats_update_task which communicates with DP coordinator.在 DPAsyncMPClient 内部，我们创建一个 asyncio 任务 run_engine_stats_update_task，它与 DP 协调器通信。

The DP coordinator mediates between the frontend (API server) and backend (engine cores). It:DP 协调器在前端（API 服务器）和后端（引擎核心）之间进行中介。它：

• Periodically sends load-balancing info (queue sizes, waiting/running requests) to the frontend's run_engine_stats_update_task.定期向前端的 run_engine_stats_update_task 发送负载均衡信息（队列大小、等待/运行中的请求）。
• Handles SCALE_ELASTIC_EP commands from the frontend by dynamically changing the number of engines (only works with Ray backend).通过动态更改引擎数量来处理来自前端的 SCALE_ELASTIC_EP 命令（仅适用于 Ray 后端）。
• Sends START_DP_WAVE events to the backend (when triggered by frontend) and reports wave-state updates back.向后端发送 START_DP_WAVE 事件（由前端触发）并报告波次状态更新。

To recap, the frontend (AsyncLLM) runs several asyncio tasks (remember: concurrent, not parallel):总结一下，前端（AsyncLLM）运行多个 asyncio 任务（记住：是并发，不是并行）：

• A class of tasks handles input requests through the generate path (each new client request spawns a new asyncio task).一类任务通过生成路径处理输入请求（每个新客户端请求都会产生一个新的 asyncio 任务）。
• Two tasks (process_outputs_socket, output_handler) process output messages from the underlying engines.两个任务（process_outputs_socket, output_handler）处理来自底层引擎的输出消息。
• One task (run_engine_stats_update_task) maintains communication with the DP coordinator: sending wave triggers, polling LB state, and handling dynamic scaling requests.一个任务（run_engine_stats_update_task）维持与 DP 协调器的通信：发送波次触发器、轮询负载均衡状态以及处理动态扩缩容请求。

Finally, the main server process creates a FastAPI app and mounts endpoints such as OpenAIServingCompletion and OpenAIServingChat, which expose /completion, /chat/completion, and others. The stack is then served via Uvicorn.最后，主服务器进程创建一个 FastAPI 应用并挂载如 OpenAIServingCompletion 和 OpenAIServingChat 等端点，它们暴露 /completion, /chat/completion 等接口。整个栈通过 Uvicorn 提供服务。

So, putting it all together, here's the full request lifecycle!综上所述，这就是完整的请求生命周期！

You send from your terminal:你从终端发送：

curl -X POST http://localhost:8000/v1/completions -H "Content-Type: application/json" -d '{
  "model": "TinyLlama/TinyLlama-1.1B-Chat-v1.0",
  "prompt": "The capital of France is",
  "max_tokens": 50,
  "temperature": 0.7
}'

What happens next:接下来发生的事：

1. The request hits OpenAIServingCompletion's create_completion route on the API server.请求到达 API 服务器上 OpenAIServingCompletion 的 create_completion 路由。
2. The function tokenizes the prompt asynchronously, and prepares metadata (request ID, sampling params, timestamp, etc.).该函数异步对提示词进行分词，并准备元数据（请求 ID、采样参数、时间戳等）。
3. It then calls AsyncLLM.generate, which follows the same flow as the synchronous engine, eventually invoking DPAsyncMPClient.add_request_async.然后它调用 AsyncLLM.generate，遵循与同步引擎相同的流程，最终调用 DPAsyncMPClient.add_request_async。
4. This in turn calls get_core_engine_for_request, which does load balancing across engines based on the DP coordinator's state (picking the one that has minimal score / lowest load: score = len(waiting) * 4 + len(running)).这进而调用 get_core_engine_for_request，它根据 DP 协调器的状态在引擎之间进行负载均衡（选择分数最小/负载最低的引擎：分数 = 等待数 * 4 + 运行数）。
5. The ADD request is sent to the chosen engine's input_socket.ADD 请求被发送到所选引擎的 input_socket。
6. At that engine:
   • Input thread — unblocks, decodes data from the input socket, and places a work item on the input_queue for the main thread.输入线程 — 解除阻塞，从输入套接字解码数据，并将工作项放入主线程的 input_queue。
   • Main thread — unblocks on input_queue, adds the request to the engine, and repeatedly calls engine_core.step(), enqueueing intermediate results to output_queue until a stop condition is met.主线程 — 在 input_queue 上解除阻塞，将请求添加到引擎，并重复调用 engine_core.step()，将中间结果加入 output_queue，直到满足停止条件。
   Reminder: step() calls the scheduler, model executor (which in turn can be MultiProcExecutor!), etc. We have already seen this!提醒：step() 会调用调度器、模型执行器（它本身可能是 MultiProcExecutor！）等。我们已经见过这些了！
   • Output thread — unblocks on output_queue and sends results back through the output socket.输出线程 — 在 output_queue 上解除阻塞，并通过输出套接字将结果发回。
7. Those results trigger the AsyncLLM output asyncio tasks (process_outputs_socket and output_handler), which propagate tokens back to FastAPI's create_completion route.这些结果触发 AsyncLLM 的输出 asyncio 任务（process_outputs_socket 和 output_handler），将 token 传回 FastAPI 的 create_completion 路由。
8. FastAPI attaches metadata (finish reason, logprobs, usage info, etc.) and returns a JSONResponse via Uvicorn to your terminal!FastAPI 附加元数据（完成原因、logprobs、使用情况信息等），并通过 Uvicorn 返回 JSONResponse 到你的终端！

And just like that, your completion came back — the whole distributed machinery hidden behind a simple curl command! :) So much fun!!!就这样，你的补全结果回来了——整个分布式机制都隐藏在一个简单的 curl 命令之后！:) 太有趣了！！！

Benchmarks and auto-tuning - latency vs throughput基准测试与自动调优 - 延迟 vs 吞吐量

So far we've been analyzing the "gas particles" — the internals of how requests flow through the engine/system. Now it's time to zoom out and look at the system as a whole, and ask: how do we measure the performance of an inference system?到目前为止，我们一直在分析“气体粒子”——请求如何在引擎/系统中流动的内部细节。现在是时候拉远视角，审视整个系统，并提出问题：我们如何衡量推理系统的性能？

At the highest level there are two competing metrics:在最高层级，有两个相互竞争的指标：

1. Latency — the time from when a request is submitted until tokens are returned延迟 — 从提交请求到返回 token 所需的时间
2. Throughput — the number of tokens/requests per second the system can generate/process吞吐量 — 系统每秒可以生成/处理的 token/请求数量

Latency matters most for interactive applications, where users are waiting on responses.延迟对于交互式应用最为重要，因为用户在等待响应。

Throughput matters in offline workloads like synthetic data generation for pre/post-training runs, data cleaning/processing, and in general - any type of offline batch inference jobs.吞吐量在离线工作负载中很重要，例如预训练/后训练运行的合成数据生成、数据清洗/处理，以及一般的任何类型的离线批量推理作业。

Before explaining why latency and throughput compete, let's define a few common inference metrics:在解释为什么延迟和吞吐量会产生竞争之前，先定义几个常见的推理指标：

Metric	Definition
TTFT (time to first token)	Time from request submission until the first output token is received
ITL (inter-token latency)	Time between two consecutive tokens (e.g., from token i-1 to token i)
TPOT (time per output token)	The average ITL across all output tokens in a request
Latency / E2E (end-to-end latency)	Total time to process a request, i.e. TTFT + sum of all ITLs, or equivalently the time between submitting request and receiving the last output token
Throughput	Total tokens processed per second (input, output, or both), or alternatively requests per second
Goodput	Throughput that meets service-level objectives (SLOs) such as max TTFT, TPOT, or e2e latency. For example, only tokens from requests meeting those SLOs are counted

Here is a simplified model explaining the competing nature of these 2 metrics.这是一个解释这两个指标竞争性质的简化模型。

The tradeoff becomes clear when looking at how batch size B affects a single decode step. As B ↓ toward 1, ITL drops: there's less work per step and the token isn't "competing" with others. As B ↑ toward infinity, ITL rises because we do more FLOPs per step—but throughput improves (until we hit peak perf) because weight I/O is amortized across more tokens.当我们观察批大小 B 如何影响单个解码步骤时，权衡变得清晰。随着 B ↓ 趋向 1，ITL 下降：每步的工作量减少，且 token 不会与其他 token“竞争”。随着 B ↑ 趋向无穷大，ITL 上升，因为我们每步执行的 FLOPs 更多——但吞吐量提高（直到达到峰值性能），因为权重 I/O 被分摊到了更多 token 上。

A roofline model helps with understanding here: below a saturation batch B_sat, the step time is dominated by HBM bandwidth (streaming weights layer-by-layer into on-chip memory), so step latency is nearly flat—computing 1 vs 10 tokens can take a similar time. Beyond B_sat, the kernels become compute-bound and step time grows roughly with B; each extra token adds to ITL.屋顶线（Roofline）模型有助于理解这一点：在饱和批大小 B_sat 以下，步骤时间由 HBM 带宽主导（将权重逐层流式传输到片上内存），因此步骤延迟几乎是平坦的——计算 1 个与 10 个 token 可能花费的时间相似。超过 B_sat 后，内核变为计算密集型，步骤时间随 B 大致增长；每个额外的 token 都会增加 ITL。

How to benchmark in vLLM如何在 vLLM 中进行基准测试

vLLM provides a vllm bench {serve,latency,throughput} CLI that wraps vllm / benchmarks / {server,latency,throughput}.py.vLLM 提供了一个 vllm bench {serve,latency,throughput} CLI，它封装了 vllm / benchmarks / {server,latency,throughput}.py。

Here is what the scripts do:脚本的作用如下：

• latency — uses a short input (default 32 tokens) and samples 128 output tokens with a small batch (default 8). It runs several iterations and reports e2e latency for the batch.latency — 使用短输入（默认 32 token）并以小批次（默认 8）采样 128 个输出 token。它运行多次迭代并报告批次的端到端延迟。
• throughput — submits a fixed set of prompts (default: 1000 ShareGPT samples) all at once (aka as QPS=Inf mode), and reports input/output/total tokens and requests per second across the run.throughput — 一次性提交一组固定的提示词（默认：1000 个 ShareGPT 样本）（即 QPS=Inf 模式），并报告运行期间的输入/输出/总 token 数和每秒请求数。
• serve — Launches a vLLM server and simulates a real-world workload by sampling request inter-arrival times from a Poisson (or more generally, Gamma) distribution. It sends requests over a time window, measures all the metrics we’ve discussed, and can optionally enforce a server-side max concurrency (via a semaphore, e.g. limiting the server to 64 concurrent requests).serve — 启动一个 vLLM 服务器，并通过从泊松（或更通用的伽马）分布中采样请求到达间隔时间来模拟真实世界的工作负载。它在一段时间内发送请求，测量我们讨论过的所有指标，并可选择强制执行服务器端最大并发数（通过信号量，例如限制服务器为 64 个并发请求）。

Here is an example of how you can run the latency script:

vllm bench latency
  --model <model-name>
  --input-tokens 32
  --output-tokens 128
  --batch-size 8

There is also an auto-tune script that drives the serve benchmark to find argument settings that meet target SLOs (e.g., "maximize throughput while keeping p99 e2e < 500 ms"), returning a suggested config.还有一个自动调优脚本，它驱动 serve 基准测试以找到满足目标 SLO 的参数设置（例如，“在保持 p99 e2e < 500 ms 的同时最大化吞吐量”），并返回建议的配置。

Epilogue结语

We began with the basic engine core (UniprocExecutor), added advanced features like speculative decoding and prefix caching, scaled up to MultiProcExecutor (with TP/PP > 1), and finally scaled out, wrapped everything in the asynchronous engine and distributed serving stack—closing with how to measure system performance.我们从基础引擎核心（UniprocExecutor）开始，添加了推测解码和前缀缓存等高级功能，扩展到 MultiProcExecutor（TP/PP > 1），最后向外扩展，将所有内容封装在异步引擎和分布式服务栈中——并以如何衡量系统性能作为结束。

vLLM also includes specialized handling that I've skipped. E.g.:vLLM 还包含我跳过的专业处理。例如：

• Diverse hardware backends: TPUs, AWS Neuron (Trainium/Inferentia), etc.多样化的硬件后端：TPU、AWS Neuron (Trainium/Inferentia) 等。
• Architectures/techniques: MLA, MoE, encoder-decoder (e.g., Whisper), pooling/embedding models, EPLB, m-RoPE, LoRA, ALiBi, attention-free variants, sliding-window attention, multimodal LMs, and state-space models (e.g., Mamba/Mamba-2, Jamba)架构/技术：MLA, MoE, 编码器-解码器（例如 Whisper）、池化/嵌入模型、EPLB、m-RoPE、LoRA、ALiBi、无注意力变体、滑动窗口注意力、多模态 LM 和状态空间模型（例如 Mamba/Mamba-2, Jamba）。
• TP/PP/SPTP/PP/SP
• Hybrid KV-cache logic (Jenga), more complex sampling methods like beam sampling, and more混合 KV 缓存逻辑（Jenga）、更复杂的采样方法（如束搜索）等。
• Experimental: async scheduling实验性：异步调度

The nice thing is that most of these are orthogonal to the main flow described above—you can almost treat them like "plugins" (in practice there's some coupling, of course).好消息是，其中大多数与上述主流程是正交的——你几乎可以把它们当作“插件”来对待（当然，在实践中会有一些耦合）。

I love understanding systems. Having said that, the resolution definitely suffered at this altitude. In the next posts I'll zoom in on specific subsystems and get into the nitty-gritty details.我热爱理解系统。话虽如此，在这个高度上，分辨率确实有所下降。在接下来的文章中，我将聚焦于特定的子系统并深入探讨细节。

Acknowledgements致谢

A huge thank you to Hyperstack for providing me with H100s for my experiments over the past year!非常感谢 Hyperstack 在过去一年中为我的实验提供 H100！

Thanks to Nick Hill (core vLLM contributor, RedHat), Mark Saroufim (PyTorch), Kyle Krannen (NVIDIA, Dynamo), and Ashish Vaswani for reading pre-release version of this blog post and providing feedback!感谢 Nick Hill（vLLM 核心贡献者，RedHat）、Mark Saroufim（PyTorch）、Kyle Krannen（NVIDIA, Dynamo）和 Ashish Vaswani 阅读本文的预发布版本并提供反馈！

References参考文献

1. vLLM https://github.com/vllm-project/vllmvLLM https://github.com/vllm-project/vllm
2. "Attention Is All You Need", https://arxiv.org/abs/1706.03762"Attention Is All You Need", https://arxiv.org/abs/1706.03762
3. "Efficient Memory Management for Large Language Model Serving with PagedAttention", https://arxiv.org/abs/2309.06180"Efficient Memory Management for Large Language Model Serving with PagedAttention", https://arxiv.org/abs/2309.06180
4. "DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model", https://arxiv.org/abs/2405.04434"DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model", https://arxiv.org/abs/2405.04434
5. "Jenga: Effective Memory Management for Serving LLM with Heterogeneity", https://arxiv.org/abs/2503.18292"Jenga: Effective Memory Management for Serving LLM with Heterogeneity", https://arxiv.org/abs/2503.18292
6. "Orca: A Distributed Serving System for Transformer-Based Generative Models", https://www.usenix.org/conference/osdi22/presentation/yu"Orca: A Distributed Serving System for Transformer-Based Generative Models", https://www.usenix.org/conference/osdi22/presentation/yu
7. "XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models", https://arxiv.org/abs/2411.15100"XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models", https://arxiv.org/abs/2411.15100
8. "Accelerating Large Language Model Decoding with Speculative Sampling", https://arxiv.org/abs/2302.01318"Accelerating Large Language Model Decoding with Speculative Sampling", https://arxiv.org/abs/2302.01318
9. "EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty", https://arxiv.org/abs/2401.15077"EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty", https://arxiv.org/abs/2401.15077
10. "Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads", https://arxiv.org/abs/2401.10774"Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads", https://arxiv.org/abs/2401.10774
11. LMCache, https://github.com/LMCache/LMCacheLMCache, https://github.com/LMCache/LMCache