Migrating LiteLLM to Rust - Building the Fastest and Litest AI Gateway将 LiteLLM 迁移到 Rust - 构建最快且最轻量的 AI 网关
Last Updated: June 2026最近更新：2026年6月
Over the past year, we have heard the same thing from our users and our community: they want the fastest, most lightweight AI gateway they can run. We have heard you. We are addressing it by moving LiteLLM to Rust, and committing to sub-1ms overhead with a sub-100MB memory binary you can deploy. By the end of this migration, you will get a pure Rust server that can serve 100% of your AI traffic, with every hot path operation, including auth and rate limiting, running in Rust.在过去的一年里，我们从用户和社区听到同一个声音：他们想要运行最快、最轻量的 AI 网关。我们听到了。我们通过将 LiteLLM 移植到 Rust，并承诺实现亚毫秒级开销、低于 100MB 内存的二进制文件来解决这个需求。迁移完成后，您将获得一个纯 Rust 服务器，能够处理您 100% 的 AI 流量，所有热点路径操作，包括认证和限流，都将在 Rust 中运行。
We are opening an early beta and want to work directly with teams who care about a fast, lightweight gateway. If that is you, sign up here and we will get you testing the Rust gateway in your own stack, with a direct line to our team.我们正在开放早期测试版，想直接与关注快速、轻量网关的团队合作。如果这正是您，请在此注册，我们将让您在自己的技术栈中测试 Rust 网关，并提供直接与我们团队沟通的渠道。
The reason it matters: under real load, CPU and memory climb with concurrency, and pods get OOM-killed at the worst time. Today the LiteLLM Python proxy peaks around 359MB of memory under load, and that cost multiplies across every pod, region, and retry you run.这很重要的原因：在真实负载下，CPU 和内存会随并发度上升，Pod 会在最糟糕的时刻被 OOM 杀掉。如今 LiteLLM 的 Python 代理在负载下内存峰值约为 359MB，这个成本会在每个 Pod、每个地区以及每次重试时成倍增长。
We are already seeing the payoff in benchmarks. The Rust gateway serves about 15x the throughput (453 to 6,782 requests per second) on about 11x less memory (359MB to 32MB), and cuts per-request overhead from about 7.5ms on the Python path to about 0.05ms, well under the 1ms we commit to.我们已经在基准测试中看到回报。Rust 网关的吞吐量约为 Python 的 15 倍（每秒 453 请求提升至 6,782 请求），内存使用约为 11 倍更少（359MB 降至 32MB），每请求开销从 Python 路径的约 7.5ms 降至约 0.05ms，远低于我们承诺的 1ms。
What you get
You deploy a single Rust binary. It uses about 65MB of memory, gateway overhead stays under 1ms, and nothing in your setup changes: same config.yaml, same database, same client API, same providers. You keep LiteLLM's coverage of 100+ LLM providers behind one OpenAI-compatible API, with /chat/completions, /messages, /responses, and every other LLM endpoint LiteLLM supports today, now as the fastest and most lightweight LLM gateway you can self-host.您只需部署一个 Rust 二进制文件。它大约占用 65MB 内存，网关开销保持在 1ms 以下，且您的部署无需任何更改：相同的 config.yaml、相同的数据库、相同的客户端 API、相同的供应商。您仍然可以通过一个兼容 OpenAI 的 API 访问 100 多家 LLM 供应商，支持 /chat/completions、/messages、/responses 以及 LiteLLM 当前支持的所有其他 LLM 端点，现在它是您可以自托管的最快、最轻量的 LLM 网关。
This is not a v2 and not a rewrite. There is no new major version to migrate to and nothing for you to change. The runtime under the hot path gets faster and lighter while your config stays exactly where it is.这不是 v2，也不是一次重写。没有新的主版本需要迁移，也无需您做任何更改。热点路径的运行时会变得更快更轻，而您的配置保持不变。
We ship this the careful way. Each route moves to Rust only after it passes our full parity and end-to-end test suite, and it runs in production before the next route starts. Stability is the priority, and we target zero regressions on every release.我们以谨慎的方式发布。每条路由仅在通过完整的等价性和端到端测试套件后才迁移到 Rust，并且在下一个路由开始之前已经在生产环境运行。稳定性是首要任务，我们的目标是在每次发布中实现零回归。
How fast is the LiteLLM gateway? A throughput, overhead, and memory benchmark
Per-request overhead. We built a small harness: a mock upstream, a thin Rust forwarding gateway (axum), the same forwarding path running through LiteLLM today (litellm.acompletion over uvicorn), and a load client that times each request in microseconds. At 10 concurrent clients against the same mock, the Rust gateway adds about 0.05ms of overhead per request; the LiteLLM Python path adds about 7.5ms. That is roughly 150x lower, and well under the 1ms we commit to.每请求开销。我们构建了一个小型测试工具：一个模拟上游、一个轻量 Rust 转发网关（axum）、今天通过 LiteLLM 运行的相同转发路径（litellm.acompletion 通过 uvicorn），以及一个以微秒计时的负载客户端。在相同的模拟环境下，10 个并发客户端时，Rust 网关每请求额外开销约 0.05ms；LiteLLM Python 路径约为 7.5ms。大约低 150 倍，远低于我们承诺的 1ms。
Sustained load. Against the current LiteLLM Python proxy on the same /v1/responses workload at 50 concurrent clients, the Rust path served about 15x the throughput on about 11x less memory.持续负载。在相同的 /v1/responses 工作负载下，使用 50 个并发客户端，Rust 路径的吞吐量约为 Python 代理的 15 倍，内存使用约为 11 倍更少。
| Per-request overhead | Throughput under load | Peak memory under load | |
|---|---|---|---|
| Rust gateway | ~0.05ms | 6,782 req/s | 31.7MB |
| LiteLLM (Python) | ~7.5ms | 453 req/s | 358.9MB |
The overhead harness (mock, gateway, load client) is checked in next to this post under benchmark/, and the summarized numbers are in rust_proxy_benchmark_results.csv, so you can reproduce the sub-1ms result. This measures the gateway forwarding path (request transform, forwarding, response handling), not a full production workload.该开销测试工具（模拟、网关、负载客户端）已在本帖旁的 benchmark/ 目录下提交，汇总数据在 rust_proxy_benchmark_results.csv 中，您可以复现亚毫秒级结果。此测量的是网关转发路径（请求转换、转发、响应处理），而非完整的生产工作负载。
What stays the same
Nothing you depend on changes. The migration is invisible from the outside:您依赖的任何东西都不会改变。迁移对外部是透明的：
- Your Python SDK keeps the exact same interface; the same calls now run on Rust bindings underneath.您的 Python SDK 保持完全相同的接口；相同的调用现在在 Rust 绑定下运行。
- Your
config.yamlis unchanged.您的 config.yaml 未改变。 - Your database and schema are unchanged.您的数据库和模式未改变。
- Your client API and request/response shapes are unchanged.您的客户端 API 以及请求/响应结构未改变。
- Your providers, routing, and keys are unchanged.您的供应商、路由和密钥未改变。
You get lower memory and lower overhead, and you do nothing to get it.您将获得更低的内存占用和更低的开销，而无需做任何操作。
How the migration works
If you just want the outcome, you have it above. The rest of this post is for engineers who want to see how we move the gateway to Rust without breaking anything.如果您只想要结果，上面已经给出。本文其余部分面向想了解我们如何在不破坏任何功能的前提下将网关迁移到 Rust 的工程师。
The core idea is a clean split. We build one Rust core that only transforms data: it turns your request into a provider request, turns the provider response back, transforms stream chunks, counts tokens, and normalizes errors. It never opens a socket, reads a secret, or writes to your database. The host process does all of that. That separation is what lets us put Rust into production without rewriting the server, because Python keeps doing the I/O while Rust takes over the translation.核心思路是清晰的拆分。我们构建了一个仅负责数据转换的 Rust 核心：它将您的请求转换为供应商请求，将供应商响应转换回来，转换流块，统计 token，规范化错误。它从不打开套接字、读取密钥或写入数据库，这些都由宿主进程完成。这种分离让我们能够在不重写服务器的情况下将 Rust 投入生产，因为 Python 仍负责 I/O，而 Rust 接管翻译工作。
One route at a time, proven in production
We never flip a whole endpoint at once. For each route we prove one provider first, roll it out to every provider on that route, and only then start the next route. The smallest, lowest-risk route goes first.我们从不一次性切换整个端点。对于每条路由，我们先验证一个供应商，随后在该路由的所有供应商上推广，只有在此之后才开始下一个路由。风险最小、最安全的路由最先迁移。
The repeating cadence inside Stage 1阶段 1 内部的重复节奏
In Stage 1 the server does not change shape. Python still serves traffic and does the I/O, but hands translation to the Rust core through a flag-gated binding, per provider. A parity check enforces identical output before any provider turns on, and if the flag is off the existing Python path runs unchanged.在阶段 1 中，服务器形态不变。Python 仍然提供流量并完成 I/O，但通过标记门控绑定将翻译交给 Rust 核心，每个供应商都有对应的绑定。等价性检查在任何供应商开启前强制输出完全相同，若标记关闭，则现有的 Python 路径保持不变。
Stage 1 · Rust core, driven by the Python SDK阶段 1 · Rust 核心，由 Python SDK 驱动
The routes move in order of risk:路由按风险顺序迁移：
- OCR first. Start with Mistral OCR, the smallest surface: no streaming, tiny schema, few params. Once it matches the Python output byte for byte in production, roll out to all OCR providers, then move the route into the Rust core. Integration risk is retired here before any larger endpoint moves.先 OCR。先从 Mistral OCR 开始，表面最小：无流式、结构简洁、参数少。生产环境中字节对字节匹配后，推广至所有 OCR 供应商，再将路由迁入 Rust 核心。这样在更大端点迁移前就已消除集成风险。
/v1/messagesnext. This adds streaming: SSE parsing, chunk emission, usage accounting, token cost. One provider first, then all, then the route into Rust.接下来是 /v1/messages。这会加入流式：SSE 解析、块发射、使用计量、token 成本。先一个供应商，然后全部，最后迁入 Rust。/chat/completionsafter that. The largest surface, taken on only once streaming is proven: tools, function calling, multimodal, and the full optional-param matrix.随后是 /chat/completions。最大表面，仅在流式验证后才处理：工具、函数调用、多模态以及完整的可选参数矩阵。- Major providers. Azure, then Bedrock, then Vertex, by traffic volume. Auth-coupled providers get signed headers from the host (boto3 / google-auth first, native Rust later). Long-tail providers keep running on Python.主要供应商。先 Azure，再 Bedrock，随后 Vertex，按流量排序。与认证耦合的供应商先使用宿主提供的签名头（boto3 / google-auth），随后迁移到原生 Rust。长尾供应商继续在 Python 上运行。
Onto a Rust server
Once the routes run on Rust, the router moves too: routing, fallbacks, retries, and cooldowns, with state in Redis. Then the server itself moves in two steps.路由迁移到 Rust 后，路由器也迁移：路由、回退、重试和冷却，状态保存在 Redis 中。随后服务器本身分两步迁移。
Stage 2 → Stage 3 · onto a server阶段 2 → 阶段 3 · 迁移到服务器
- FastAPI as a thin shell. FastAPI still terminates HTTP and runs auth, rate-limit, and callbacks, but the entire forwarding path is a single call into Rust.FastAPI 仍作为轻量外壳。FastAPI 仍然终止 HTTP 并运行认证、限流和回调，但整个转发路径仅通过一次调用进入 Rust。
- Pure Rust server. A native server (axum / hyper) runs the forwarding path with no Python on the hot path. Your custom Python plugins (auth, guardrails, callbacks, SSO) keep working in an optional sidecar, so nothing breaks. We roll it out with shadow traffic and a percentage cutover.纯 Rust 服务器。原生服务器（axum / hyper）在热点路径上运行转发路径，不再有 Python。您的自定义 Python 插件（认证、守护规则、回调、SSO）仍在可选的 sidecar 中运行，保证不破坏任何功能。我们将通过影子流量和分批切换进行发布。
The end state is a pure Rust data plane. Customer Python plugins keep running in the sidecar, so it is non-breaking. Removing Python entirely would require porting plugins to a Rust or WASM interface, which is a breaking change we are deferring.最终状态是纯 Rust 数据平面。客户的 Python 插件继续在 sidecar 中运行，因此迁移是非破坏性的。要完全移除 Python，需要将插件移植到 Rust 或 WASM 接口，这是一项我们暂缓的破坏性改动。
Why this order
- The OCR route retires integration risk on the smallest surface.OCR 路由在最小表面上消除集成风险。
/v1/messagesretires streaming risk before the largest parameter set./v1/messages 在最大参数集之前消除流式风险。/chat/completionsis taken on only after streaming is proven./chat/completions 仅在流式验证后才进行。- By the time the server moves, the core, providers, and router are already running in production through the SDK, so the server work is mostly plumbing.等服务器迁移时，核心、供应商和路由已经通过 SDK 在生产环境运行，服务器工作主要是管道化。
Every step ships to real users before the next begins, with the parity check as the gate.每一步在真实用户中发布后才进入下一步，等价性检查是唯一的闸门。
Timeline
We move one function at a time, smallest first, and only after each step passes our test suite.我们一次迁移一个功能，先小后大，并且每一步都必须通过我们的测试套件。
| Target | What moves to Rust |
|---|---|
| Aug 15, 2026 | litellm.ocr() for Mistral, then all of litellm.ocr(), then the /ocr route |
| Sep 1, 2026 | Same pattern for /messages, then /chat/completions |
| Sep 15, 2026 | The router: load balancing, fallbacks, retries, cooldowns |
| Dec 1, 2026 | The full server: FastAPI thin shell, then pure Rust (axum) |
Frequently asked questions
Is LiteLLM the fastest LLM gateway?
That is the goal of this work. With the Rust hot path, LiteLLM targets sub-1ms gateway overhead and a sub-100MB binary, matching compiled-language gateways while keeping coverage of 100+ providers behind one OpenAI-compatible API. In our benchmark the Rust gateway adds about 0.05ms of overhead per request, versus about 7.5ms for the LiteLLM Python path today, and serves 6,782 requests per second under load at 31.7MB peak memory. Gateway overhead is usually a small fraction of total model latency, so it matters most for high-throughput, low-latency workloads like classification and embeddings at scale.这就是本工作的目标。通过 Rust 热点路径，LiteLLM 目标是实现亚毫秒级网关开销和低于 100MB 的二进制文件，匹配编译语言网关的性能，同时保持 100+ 供应商通过单一兼容 OpenAI 的 API 的覆盖率。在我们的基准中，Rust 网关每请求额外开销约 0.05ms，而 LiteLLM Python 路径约为 7.5ms，且在 31.7MB 峰值内存下实现每秒 6,782 请求。网关开销通常只占模型总延迟的一小部分，因此在高吞吐、低延迟的分类和嵌入等工作负载中尤为重要。
Is LiteLLM slow?
Gateway latency and throughput depend on how you deploy the proxy: worker count, concurrency settings, and whether logging callbacks run on the hot path. Tuned, the Python proxy serves production traffic across hundreds of providers today. Moving the hot path to Rust pushes the floor lower still: in our reproducible benchmark the Rust gateway adds about 0.05ms of overhead per request, versus about 7.5ms for the LiteLLM Python path, and serves 6,782 requests per second at 31.7MB peak memory.网关的延迟和吞吐量取决于您如何部署代理：工作进程数量、并发设置以及日志回调是否在热点路径上运行。经过调优后，Python 代理已经能够在数百家供应商的生产流量中运行。将热点路径迁移到 Rust 进一步压低底线：在我们的可复现基准中，Rust 网关每请求额外开销约 0.05ms，而 Python 路径约为 7.5ms，峰值内存 31.7MB 时每秒处理 6,782 请求。
Is LiteLLM limited by the Python GIL?
The GIL only matters for CPU-bound work on the request path, and the gateway is mostly I/O. LiteLLM scales today by running multiple workers. The Rust migration removes the question for the hot path: request transforms, streaming, and routing run in the Rust core and router, outside the GIL, with no first-party Python on the forwarding path in the end state.GIL 只在请求路径的 CPU 密集型工作中起作用，而网关主要是 I/O。LiteLLM 通过运行多个工作进程实现横向扩展。Rust 迁移消除了热点路径的 GIL 限制：请求转换、流式和路由在 Rust 核心和路由器中运行，最终状态下转发路径不再有任何官方 Python 代码。
How much memory does the LiteLLM gateway use?
The Python proxy peaked at 358.9MB under our load test. The Rust end state targets roughly 65MB. Lower, bounded memory is the main reason for this work: it reduces the high-CPU and OOM failures that show up under concurrent load.Python 代理在我们的负载测试中峰值为 358.9MB。Rust 最终状态目标约为 65MB。更低且受限的内存是此工作的主要动因：它降低了高 CPU 使用和 OOM 失效的风险。
Are these benchmarks reproducible?
Yes. The overhead harness (a mock upstream, a thin Rust gateway, and a load client that times each request in microseconds) is checked in under benchmark/, alongside the summarized CSV. Same upstream and payload for both runtimes; the only variable is Python versus Rust.是的。开销测试工具（模拟上游、轻量 Rust 网关以及以微秒计时的负载客户端）已在 benchmark/ 目录下提交，并附有汇总的 CSV。两种运行时使用相同的上游和负载，仅变量是 Python 与 Rust 的对比。
Will the Rust gateway be a breaking change?
No. Config, database schema, and the client API contract stay the same. The runtime under the hot path changes gradually, route by route, behind passing parity and end-to-end tests.不。配置、数据库模式以及客户端 API 合约保持不变。热点路径的运行时会逐步、按路由迁移，在通过等价性和端到端测试后才切换。
We are hiring Rust engineers
We are building this with a small team and looking for Rust engineers who want to work on the hot path of an AI gateway that serves 100+ providers. If that sounds like you, come build it with us.我们正在用一个小团队构建此项目，寻找想在服务 100+ 供应商的 AI 网关热点路径上工作的 Rust 工程师。如果这听起来像您，欢迎加入我们一起构建。
References
- How Datadog migrated their static analyzer from Java to RustDatadog 如何将其静态分析器从 Java 迁移到 Rust
- How GitGuardian migrated the heart of their platform to RustGitGuardian 如何将平台核心迁移到 Rust
- LiteLLM AI Gateway, full feature overviewLiteLLM AI 网关，完整功能概览
- Load balancing and routing across 100+ LLM providers在 100+ LLM 供应商之间进行负载均衡和路由