In-House LLM Serving at NetflixNetflix 的内部大语言模型（LLM）服务
By AI Platform’s Model Runtime team and Inference team作者：AI 平台模型运行时（Model Runtime）团队与推理（Inference）团队
Introduction引言
Most organizations consume LLMs through hosted APIs. Netflix went further — we run the full stack ourselves, from model deployment through inference, inside our existing production environment rather than a separate ML silo. Some of those decisions weren’t obvious, and a few revealed their trade-offs only under production load.大多数机构通过托管 API 使用大语言模型。Netflix 更进一步——我们自行运行整个技术栈，从模型部署到推理，均在现有的生产环境中完成，而非将其隔离在独立的机器学习孤岛中。其中一些决策并非显而易见，少数决策甚至只有在生产负载下才会显现出其权衡之处。
This post focuses on the choices where alternatives were seriously considered: engine selection, model packaging, API surface design, deployment strategy, and output constraints enforcement. The goal is to share not just what was built, but why — and what production revealed that the design phase didn’t anticipate.本文重点探讨我们经过慎重考虑后所做的选择：引擎选择、模型打包、API 界面设计、部署策略以及输出约束强制执行。我们的目标不仅是分享我们构建了什么，还要分享为什么要这样构建，以及生产环境揭示了哪些设计阶段未能预见的问题。
Architecture Overview架构概述
Member-scale ML at Netflix is fronted by a unified JVM-based serving system that handles the end-to-end flow for downstream consumers: routing and A/B test logic, candidate generation, feature fetching, inference, post-processing, and logging at each stage. Both real-time and cached batch paths are supported. Figure 1 shows the two ways callers reach inference today: the gRPC path through this serving system and a direct HTTP path used by newer LLM-driven applications.Netflix 的会员级机器学习服务由一个统一的、基于 JVM 的服务系统支撑，该系统处理下游消费者的端到端流程：路由与 A/B 测试逻辑、候选生成、特征获取、推理、后处理以及各阶段的日志记录。系统同时支持实时路径和缓存批处理路径。图 1 展示了目前调用者进行推理的两种方式：通过该服务系统的 gRPC 路径，以及较新的 LLM 驱动型应用所使用的直接 HTTP 路径。
Where inference runs depends on the model. Small CPU models run in-process, avoiding remote-call overhead. Larger models need GPUs — the serving system handles pre- and post-processing locally but delegates inference to a remote service, Model Scoring Service (MSS). MSS is the shared inference backend, supporting XGBoost, TensorFlow, PyTorch, and LLMs behind a unified interface, with NVIDIA Triton Inference Server underneath managing model loading, batching, and GPU scheduling.推理运行的位置取决于模型。小型 CPU 模型在进程内运行，避免了远程调用的开销。大型模型需要 GPU——服务系统在本地处理预处理和后处理，但将推理委托给远程服务：模型评分服务（Model Scoring Service，简称 MSS）。MSS 是共享的推理后端，通过统一接口支持 XGBoost、TensorFlow、PyTorch 和大语言模型，底层由 NVIDIA Triton 推理服务器管理模型加载、批处理和 GPU 调度。
On top of Triton sits a Java control plane that handles deployment, versioning, health checking, autoscaling, and multi-region rollout. Model authors package their artifacts and configure the deployment; the control plane provisions GPU instances, configures Triton, and orchestrates zero-downtime upgrades.在 Triton 之上，有一个 Java 控制平面负责处理部署、版本控制、健康检查、自动伸缩和多区域发布。模型作者打包其工件并配置部署；控制平面负责预置 GPU 实例、配置 Triton 并编排零停机升级。
Design Decisions and Implementation设计决策与实现
Four decisions shape this platform — engine, packaging, API surface, and rollout — presented in dependency order, since each one constrains the next.四项决策塑造了该平台——引擎、打包、API 界面和发布——按依赖顺序呈现，因为每一项决策都会制约下一项。
vLLM as the Paved-Path EnginevLLM 作为“铺路”引擎
The platform was originally built on TensorRT-LLM, a performant inference engine at the time and already integrated with Triton — the compute backend in use within MSS.该平台最初基于 TensorRT-LLM 构建，这在当时是一款高性能推理引擎，且已与 MSS 中使用的计算后端 Triton 集成。
By summer 2025, two things had shifted: open-source engines had largely closed the performance gap with specialized stacks, and our workload mix had broadened to include embedding generation, prefill-only inference for ranking and retrieval, autoregressive decoding, and custom models with non-trivial per-step constraint logic. We re-benchmarked against this mix and selected vLLM as our paved-path engine on operational fit:到了 2025 年夏季，情况发生了两点变化：开源引擎在很大程度上缩小了与专用技术栈的性能差距；我们的工作负载组合也变得更加广泛，包括嵌入（embedding）生成、用于排序和检索的仅预填充（prefill-only）推理、自回归解码，以及具有复杂分步约束逻辑的定制模型。我们针对这种组合重新进行了基准测试，并基于运营契合度选择了 vLLM 作为我们的“铺路”引擎：
- Loads custom model architectures without a multi-step compilation pipeline — faster iteration on non-standard models.无需多步编译流水线即可加载自定义模型架构——对非标准模型的迭代速度更快。
- Extensibility hooks for custom decoding logic — necessary for the constrained-decoding work described later.提供用于自定义解码逻辑的扩展钩子——这对于后文所述的受限解码工作至关重要。
- Debuggability — easier to inspect failures and intermediate state than with a compiled engine in earlier TensorRT-LLM.可调试性——比早期 TensorRT-LLM 的编译引擎更容易检查故障和中间状态。
- Familiarity — many ML practitioners were already using vLLM in research, which cut the research-to-production handoff cost.熟悉度——许多机器学习从业者已在研究中使用 vLLM，这降低了从研究到生产的移交成本。
Integrating vLLM into Triton将 vLLM 集成到 Triton 中
With vLLM picked, the next decision was how to package models for it. Triton supports two ways, and the choice has significant implications for maintainability — specifically, how tightly model artifacts are coupled to frontend upgrades.选定 vLLM 后，下一个决策是如何为其打包模型。Triton 支持两种方式，该选择对可维护性有重大影响——具体而言，即模型工件与前端升级的耦合紧密程度。
- Python backend. The author defines explicit input/output tensor specs at packaging time. These specs are frozen in the artifact and must match what the third-party vendor’s frontend’s request builder expects, so every frontend upgrade that touches I/O specs requires a coordinated change to packaging code; otherwise, requests fail at runtime.Python 后端。作者在打包时定义明确的输入/输出张量规范。这些规范在工件中被冻结，必须与第三方供应商前端的请求构建器预期相匹配，因此每次涉及 I/O 规范的前端升级都需要对打包代码进行协调更改；否则，请求将在运行时失败。
- vLLM backend. The artifact is just a JSON config pointing to the model weights and tokenizer. Triton’s vLLM backend reads this config and generates I/O tensor specs dynamically at deployment time — the author never defines them. Models and frontend evolve independently.vLLM 后端。工件仅是一个指向模型权重和分词器的 JSON 配置。Triton 的 vLLM 后端读取此配置并在部署时动态生成 I/O 张量规范——作者无需定义它们。模型和前端可以独立演进。
The vLLM backend is the architecturally correct default. Two things bit us in production:vLLM 后端是架构上正确的默认选择。但在生产中，有两点让我们吃了苦头：
- Triton/vLLM version mismatch. Triton’s vLLM backend is compiled against a specific vLLM API surface. When the two drift — for example, Triton 25.09 importing vllm.engine.metrics, a module removed in vLLM 0.11.2 — the backend fails to load entirely. The platform has to pin compatible versions when baking the service image, and prevent model authors from overriding the vLLM version at packaging time.Triton/vLLM 版本不匹配。Triton 的 vLLM 后端是针对特定的 vLLM API 界面编译的。当两者产生偏差时——例如 Triton 25.09 导入了 vllm.engine.metrics（该模块在 vLLM 0.11.2 中已被移除）——后端将完全无法加载。平台必须在构建服务镜像时锁定兼容版本，并防止模型作者在打包时覆盖 vLLM 版本。
- Custom model logic. The vLLM backend expects a standard HuggingFace-compatible model and handles the full inference lifecycle. Models needing custom preprocessing, postprocessing, or non-standard execution — ensemble pipelines, custom tokenization — must use the Python backend, which gives full control over execute(). This escape hatch will likely remain necessary for a subset of models.自定义模型逻辑。vLLM 后端期望使用标准的 HuggingFace 兼容模型并处理完整的推理生命周期。需要自定义预处理、后处理或非标准执行（如集成流水线、自定义分词）的模型必须使用 Python 后端，这提供了对 execute() 的完全控制。对于部分模型而言，这个“逃生舱”可能仍是必需的。
Ecosystem-Compatible HTTP Frontend生态系统兼容的 HTTP 前端
With engine and packaging settled, the next question is how callers reach the system. A key design goal of our system was that LLM models should NOT be special snowflakes. Every model — XGBoost ensemble or large-scale LLMs — is scored via the same gRPC call, so we reuse the same client libraries, health checking, and deployment pipelines. Given that the OpenAI-compatible API interface has become the de facto interface for the LLM ecosystem — inference engines, orchestration frameworks, evaluation tools, and client libraries all speak it — so we expose the OpenAI-compatible API as an additional frontend alongside gRPC.引擎和打包确定后，下一个问题是调用者如何访问系统。我们系统的一个关键设计目标是：大语言模型不应是“特殊雪花”。每个模型——无论是 XGBoost 集成还是大规模大语言模型——都通过相同的 gRPC 调用进行评分，因此我们复用了相同的客户端库、健康检查和部署流水线。鉴于 OpenAI 兼容的 API 已成为大语言模型生态系统的事实标准界面（推理引擎、编排框架、评估工具和客户端库都支持它），我们除了 gRPC 之外，还额外暴露了 OpenAI 兼容的 API 作为前端。
The payoff shows up in the experimentation-to-production path: graduating from a hosted model to a fine-tuned self-hosted one — for quality, latency, cost, or data privacy — is nearly seamless. Same API, minimal code changes.其收益体现在从实验到生产的路径上：从托管模型过渡到微调后的自托管模型（为了质量、延迟、成本或数据隐私）几乎是无缝的。API 相同，代码改动极小。
Behind the API, the implementation reuses NVIDIA’s Triton OpenAI-compatible frontend. It starts an embedded Triton server, wraps it in a TritonLLMEngine that converts request schemas into Triton inference requests, and serves responses through FastAPI. KServe HTTP/gRPC frontends are enabled alongside, so the same Triton instance remains accessible to the Java control plane over gRPC. Adopting Triton’s frontend directly exposed one gap: response_format — accepted by the schema — was silently dropped before reaching vLLM, so that a caller requesting JSON output proceeded without guided decoding constraints and could receive malformed JSON with no error surfaced by the platform. We git-subtreed and patched the frontend to translate response_format into vLLM’s guided decoding parameters at request time.在 API 背后，实现复用了 NVIDIA 的 Triton OpenAI 兼容前端。它启动一个嵌入式 Triton 服务器，将其封装在 TritonLLMEngine 中，将请求模式转换为 Triton 推理请求，并通过 FastAPI 提供响应。KServe HTTP/gRPC 前端也同时启用，因此同一个 Triton 实例仍可通过 gRPC 被 Java 控制平面访问。直接采用 Triton 前端暴露了一个缺口：response_format（模式所接受的参数）在到达 vLLM 之前被静默丢弃，导致请求 JSON 输出的调用者在没有受限解码的情况下继续执行，可能会收到格式错误的 JSON，且平台不会发出错误提示。我们通过 git subtree 引入并修补了该前端，以便在请求时将 response_format 转换为 vLLM 的受限解码参数。
Deployment Strategies部署策略
With API surface and engine in place, the question that remains is how new versions roll out without dropping requests. GPU deployments take longer to bring up than CPU services, and the I/O schema may change between model versions — adding a coordination problem on top. The platform offers two strategies:API 界面和引擎就位后，剩下的问题是如何在新版本发布时避免丢弃请求。GPU 部署的启动时间比 CPU 服务更长，且 I/O 模式可能在模型版本之间发生变化——这增加了协调难度。平台提供两种策略：
- Red-Black deploys a new version alongside the current one. Once the new instance passes health checks, traffic shifts in phases — the new version scales up while the old scales down at the same rate. If any step fails, the system triggers an atomic rollback. Red-Black is the right choice when the model interface is stable. Production revealed a coordination gap when a new version requires an I/O schema change (e.g., new tensor dimensions): the upstream consumer can’t update its config until the new model is fully live, so it inevitably sends “old” requests to a “new” deployment during the migration window, and those fail.红蓝部署（Red-Black）在当前版本旁部署新版本。一旦新实例通过健康检查，流量将分阶段切换——新版本扩容的同时，旧版本以相同速率缩容。如果任何步骤失败，系统会触发原子回滚。当模型界面稳定时，红蓝部署是正确的选择。生产环境揭示了一个协调缺口：当新版本需要 I/O 模式变更（例如新的张量维度）时，上游消费者无法在模型完全上线前更新其配置，因此在迁移窗口期间，它不可避免地会向“新”部署发送“旧”请求，从而导致失败。
- Versioned solves that gap by maintaining an independent deployment for every (modelId, modelVersion) pair. Multiple versions serve simultaneously, decoupling model deployment from consumer updates: the consumer waits for the new version to be fully ready before switching its config, while the old version keeps serving legacy traffic. The platform cleans up older deployments after inactivity but always preserves the latest. The trade-off is a temporary increase in GPU cost during the transition overlap.版本化部署（Versioned）通过为每一对 (modelId, modelVersion) 维护独立的部署来解决该缺口。多个版本同时运行，使模型部署与消费者更新解耦：消费者等待新版本完全就绪后再切换配置，而旧版本继续处理遗留流量。平台会在不活跃一段时间后清理旧部署，但始终保留最新版本。其权衡在于过渡重叠期间 GPU 成本的暂时增加。
We recommend embedding variable configurations (e.g., tensor shapes) directly into the inference model to make it version-agnostic, so it can use the cheaper Red-Black path. Versioned is reserved for the rare cases where a breaking interface change is unavoidable.我们建议将变量配置（如张量形状）直接嵌入推理模型中，使其实现版本无关，从而可以使用更经济的红蓝部署路径。版本化部署仅保留用于无法避免破坏性界面变更的极少数情况。
Operational Notes运营说明
Beyond those four decisions, two operational details are worth flagging — both hit production gaps the design phase didn’t anticipate.除了上述四项决策外，还有两个运营细节值得注意——它们都触及了设计阶段未预见到的生产缺口。
Boot sequence启动顺序
Bringing a vLLM-on-Triton instance up involves several coordinated steps before the gRPC port opens. Two are non-routine.启动一个 vLLM-on-Triton 实例涉及在 gRPC 端口打开前进行多个协调步骤。其中两项是非例行操作。
- Model caching. Downloading large LLMs directly from S3 or Hugging Face at startup is slow enough to inflate cold-start latency past what schedulers tolerate. We materialize models on Amazon FSx at the time of model announcement, so warm starts hit a high-performance file system instead of object storage.模型缓存。在启动时直接从 S3 或 Hugging Face 下载大型大语言模型速度太慢，会使冷启动延迟超过调度程序的容忍度。我们在模型发布时将模型具体化到 Amazon FSx 上，因此热启动时会访问高性能文件系统，而非对象存储。
- Embedded vs standalone Triton. When consumers need the OpenAI-compatible API, Triton runs as an embedded server inside the OpenAI-compatible frontend process; otherwise, it runs standalone. This is configured per-deployment at packaging time.嵌入式与独立 Triton。当消费者需要 OpenAI 兼容 API 时，Triton 作为嵌入式服务器在 OpenAI 兼容前端进程内运行；否则，它以独立模式运行。这是在打包时针对每个部署进行配置的。
The rest of the boot sequence is mechanical: extracting the model package, installing custom vLLM plugins via Python entry_points, cleaning the Prometheus multiprocess directory, and gating the gRPC port until the engine is ready.启动顺序的其余部分是机械性的：提取模型包、通过 Python entry_points 安装自定义 vLLM 插件、清理 Prometheus 多进程目录，并在引擎就绪前阻塞 gRPC 端口。
Unified metrics endpoint统一指标端点
The Prometheus cleanup above hints at a wider observability gap. vLLM writes metrics to PROMETHEUS_MULTIPROC_DIR as .db files; Triton reports server-level metrics through its own Prometheus endpoint. Neither is aware of the other, and Triton’s built-in bridge surfaces only 9 of 40+ vLLM metrics — missing critical ones like token throughput, KV cache utilization, and prefix cache hit rates.上述 Prometheus 的清理工作暗示了一个更广泛的可观测性缺口。vLLM 将指标以 .db 文件形式写入 PROMETHEUS_MULTIPROC_DIR；Triton 通过其自身的 Prometheus 端点报告服务器级指标。两者互不感知，且 Triton 内置的桥接器仅能呈现 40 多个 vLLM 指标中的 9 个——缺失了诸如令牌吞吐量、KV 缓存利用率和前缀缓存命中率等关键指标。
We added a lightweight HTTP proxy that merges both into a single /metrics endpoint: it fetches Triton metrics via HTTP, reads vLLM metrics from disk using Prometheus’s MultiProcessCollector, and returns the combined output. Existing dashboards and alerts work without modification.我们添加了一个轻量级 HTTP 代理，将两者合并到一个单一的 /metrics 端点中：它通过 HTTP 获取 Triton 指标，使用 Prometheus 的 MultiProcessCollector 从磁盘读取 vLLM 指标，并返回合并后的输出。现有的仪表板和警报无需修改即可正常工作。
Deep-Dive: Constrained Decoding at Scale深度解析：大规模受限解码
Some Netflix production workloads rely heavily on fine-grained control over token generation. Rather than applying business logic after inference — paying for invalid generations, then retrying or repairing — we push constraints inside the decode loop, so the model generates outputs that are compliant by construction. We implement this via vLLM’s custom logits processor interface, modeling each constraint as a state machine that evolves with the generated token history and emits token-eligibility masks at each step. Each request gets its own configured processor, since different requests apply different rules.Netflix 的一些生产工作负载严重依赖于对令牌生成的细粒度控制。我们没有在推理后应用业务逻辑（这需要为无效生成付费，然后重试或修复），而是将约束推入解码循环中，使模型生成的输出在构建时即符合要求。我们通过 vLLM 的自定义 Logits 处理器接口实现这一点，将每个约束建模为一个状态机，该状态机随生成的令牌历史演进，并在每一步发出令牌合格掩码。每个请求都有其自己的配置处理器，因为不同的请求应用不同的规则。
Getting this to scale ran across two engine versions: we initially deployed on vLLM V0 (V1 had feature gaps), then migrated to V1 in Q4 2025 once it matured. The two subsections that follow are the before-and-after.实现这一规模化跨越了两个引擎版本：我们最初部署在 vLLM V0 上（V1 当时功能有缺失），待 V1 在 2025 年第四季度成熟后迁移至 V1。以下两个小节分别是实现前后的对比。
Why the first implementation didn’t scale为什么最初的实现无法扩展
Our initial pure-Python implementation worked functionally but hit a scaling bottleneck. In vLLM V0, custom logits processors run per-request: the GPU produces logits for the whole batch, the CPU copies them across and waits for the transfer, and then constraint logic runs sequentially for each request — sequentially because the GIL prevents Python from parallelizing the per-request work. CPU time in logit processing therefore grows linearly with batch size, hitting tail latencies. End-to-end latency becomes CPU-bound even though the model’s forward pass is batched efficiently on GPU. It’s a bottleneck invisible in single-request benchmarks that only surfaces under realistic concurrency. Figure 2 makes the serial pattern visible.我们最初的纯 Python 实现功能上可行，但遇到了扩展瓶颈。在 vLLM V0 中，自定义 Logits 处理器按请求运行：GPU 为整个批次生成 Logits，CPU 将其复制过来并等待传输，然后为每个请求顺序运行约束逻辑——由于全局解释器锁（GIL）的存在，Python 无法并行化按请求处理的工作。因此，Logit 处理的 CPU 时间随批次大小线性增长，导致长尾延迟。尽管模型的正向传播在 GPU 上高效批处理，但端到端延迟最终受限于 CPU。这是一个在单请求基准测试中不可见、仅在真实并发下才会显现的瓶颈。图 2 展示了这种串行模式。
vLLM V1 enabled a batch-level designvLLM V1 实现了批处理级设计
The structural fix arrived in vLLM V1, which moved logits processing to batch level. We rewrote our custom processor to operate on batch-level data structures, computing masks across many requests together, and reimplemented the hot path in C++ with multi-threading to step around the GIL. The V1 API requires explicit tracking of batch membership changes via update_state(batch_update) — more complex than V0’s per-request interface, but necessary to maintain correct state in a dynamically evolving batch. Figure 3 shows logits processing time staying flat as batch size grows.结构性修复出现在 vLLM V1 中，它将 Logits 处理移至批处理级别。我们重写了自定义处理器以操作批处理级数据结构，共同计算多个请求的掩码，并使用多线程在 C++ 中重新实现了热路径，从而绕过了 GIL。V1 API 要求通过 update_state(batch_update) 显式跟踪批次成员变更——这比 V0 的按请求接口更复杂，但对于在动态演进的批次中保持正确状态是必要的。图 3 显示了 Logits 处理时间随批次大小增长保持平稳。
Operational hardening运营强化
Now, performance was no longer the bottleneck. But stateful constraint logic in the decode loop introduced two issues the design phase didn’t anticipate:现在，性能不再是瓶颈。但在解码循环中引入有状态的约束逻辑引发了两个设计阶段未预见的问题：
- Partial prefills. V1 performs chunked prefilling, so a request can be prefilled over multiple engine steps. BatchUpdate lacks the granularity to tell whether a request was fully or only partially prefilled, so we added internal tracking.部分预填充（Partial prefills）。V1 执行分块预填充，因此请求可以在多个引擎步骤中完成预填充。BatchUpdate 缺乏区分请求是完全预填充还是部分预填充的粒度，因此我们添加了内部跟踪。
- Preemption. Under memory pressure, vLLM may evict a partially completed request’s KV cache and reschedule it later with a different prompt and output token list. This breaks the state machine’s assumption that the output token list grows monotonically. We detect when the token history shrinks between decode steps, reset the state machine, and reinitialize from the new prompt.抢占（Preemption）。在内存压力下，vLLM 可能会驱逐部分完成的请求的 KV 缓存，并在稍后使用不同的提示词和输出令牌列表重新调度它。这打破了状态机关于输出令牌列表单调增长的假设。我们检测到令牌历史在解码步骤之间缩短时，会重置状态机，并从新的提示词重新初始化。
Wrap up总结
We set out to build an LLM serving platform for broad production ML requirements — low latency, deep customization, and integration with existing infrastructure. The result is a system on vLLM and Triton, unified behind a consistent API, designed to give ML practitioners a fast path from experimentation to production.我们着手构建一个满足广泛生产机器学习需求的大语言模型服务平台——低延迟、深度定制以及与现有基础设施的集成。最终成果是一个基于 vLLM 和 Triton 的系统，统一在一致的 API 之后，旨在为机器学习从业者提供从实验到生产的快速路径。
The lessons were often in the details — version pinning, silent API gaps, packaging trade-offs — but addressing them has made the platform meaningfully more robust and the developer experience smoother. Next investments reflect where we expect friction:经验教训往往存在于细节中——版本锁定、静默 API 缺口、打包权衡——但解决这些问题使平台变得更加稳健，开发体验也更加顺畅。接下来的投资方向反映了我们预期的摩擦点：
- System prompt compression to reduce prompt length without sacrificing quality.系统提示词压缩，以在不牺牲质量的前提下减少提示词长度。
- Asynchronous scheduling of vLLM V1.vLLM V1 的异步调度。
- Vectorized logits processors that run as fused GPU kernels instead of CPU code.作为融合 GPU 内核而非 CPU 代码运行的向量化 Logits 处理器。
- Lower-precision model variants to decrease memory footprint and increase throughput.低精度模型变体，以降低内存占用并提高吞吐量。
We’ll continue working closely with the open-source community as this space evolves.随着该领域的演进，我们将继续与开源社区密切合作。
Contributions贡献
This system is the result of close collaboration and contributions from many teams within the AI Platform org at Netflix. In particular, Liping Peng designed and developed the model packaging workflow and drove the integration of Triton and vLLM with MSS to enable a unified pathway for serving LLMs. Hakan Baba, Nicolas Hortiguera, and ZQ Zhang led GPU capacity planning, system performance tuning, application integration and observability, as well as A/B test readiness and operational excellence efforts for all production models. Santino Ramos enabled vLLM for production models and optimized constrained decoding performance. Binh Tang developed the initial version of custom model serving and benchmarked different LLM serving frameworks. Lanxi Huang and Daneo Zhang built the serving development tools to enable user self-service. Lingyi Liu drove the overall system architecture and core technical decisions. Abhishek Agrawal and Shaojing Li provide management leadership to ensure alignment, prioritization and execution.该系统是 Netflix AI 平台组织内多个团队密切合作与贡献的成果。特别是 Liping Peng 设计并开发了模型打包工作流，并推动了 Triton 和 vLLM 与 MSS 的集成，从而实现了大语言模型服务的统一路径。Hakan Baba、Nicolas Hortiguera 和 ZQ Zhang 领导了所有生产模型的 GPU 容量规划、系统性能调优、应用集成与可观测性，以及 A/B 测试准备和卓越运营工作。Santino Ramos 为生产模型启用了 vLLM 并优化了受限解码性能。Binh Tang 开发了自定义模型服务的初始版本，并对不同的大语言模型服务框架进行了基准测试。Lanxi Huang 和 Daneo Zhang 构建了服务开发工具，以实现用户自助服务。Lingyi Liu 推动了整体系统架构和核心技术决策。Abhishek Agrawal 和 Shaojing Li 提供了管理领导力，以确保对齐、优先级排序和执行。
Acknowledgements致谢
This work heavily leverages open-source ML libraries, such as Triton, vLLM and PyTorch, etc. We’re especially grateful to the teams and contributors from the community. We also thank our partner teams in Netflix AI for Member Systems for their close collaborations and innovation on the modeling side.这项工作大量利用了开源机器学习库，如 Triton、vLLM 和 PyTorch 等。我们特别感谢社区的团队和贡献者。我们也感谢 Netflix AI for Member Systems 的合作伙伴团队，感谢他们在模型侧的密切合作与创新。
