We are pleased to share a research preview of OpenRL, a new open-source project coming out of GKE Labs. OpenRL is a self-hosted training API for fine-tuning LLMs on your own Kubernetes cluster.我们很高兴分享 OpenRL 的研究预览，这是 GKE Labs 推出的全新开源项目。OpenRL 是一个自托管的训练 API，用于在您自己的 Kubernetes 集群上微调大语言模型（LLM）。
Why we built it我们为何构建它
If you look at agentic RL on LLMs, it is incredibly easy to get bogged down in system complexity. To run a single RL loop, you have to coordinate a dozen different things: selecting and cleaning datasets, choosing RL environments, debugging training loops, managing reward signals, handling inference mismatches, allocating hardware, and managing infrastructure. Picture looks something like this:如果你观察基于 LLM 的代理式强化学习（RL），会发现系统复杂性极易让人陷入困境。要运行一次 RL 循环，需要协调十几件事：选择并清洗数据集、选择 RL 环境、调试训练循环、管理奖励信号、处理推理不匹配、分配硬件以及管理基础设施。整体图景大致如下：
Each of these is a hard problem. But what makes it more complex is how tightly AI research and infrastructure concerns are mixed together in today's tooling and frameworks. 每一项都是难题。但更复杂的是，今天的工具和框架中 AI 研究与基础设施关注点紧密交织在一起。
We believe decoupling the infrastructure from AI research can make these problems more tractable so that infrastructure engineers and AI researchers can independently tackle them. We have seen this pattern with Kubernetes where Kubernetes abstracted out the infrastructure and made application developers and SREs life easier.我们相信，将基础设施与 AI 研究解耦可以让这些问题更易处理，从而让基础设施工程师和 AI 研究员能够各自独立攻克。我们在 Kubernetes 上已经看到这种模式：Kubernetes 抽象了底层基础设施，使应用开发者和 SRE 的工作更轻松。
So, can you abstract out post training infrastructure? We believe so and drew huge inspiration/validation from Tinker (from Thinking Machines). The Tinker APIs for post training hit that Goldilocks zone where it hides all the post training infrastructure behind four key APIs:那么，后训练基础设施能被抽象出来吗？我们相信可以，并且从 Tinker（Thinking Machines 的项目）中获得了巨大的灵感和验证。Tinker 的后训练 API 正好落在金丝雀区间，它将所有后训练基础设施隐藏在四个关键 API 背后：
So the end result of this abstraction is that AI Researchers get full flexibility on their RL loop and infrastructure engineers can focus on scaling, orchestration, and reliability. OpenRL allows you to run the same training APIs but on your own infrastructure. And this decoupling has other interesting benefits.这种抽象的最终结果是，AI 研究员在 RL 循环上拥有完整的灵活性，而基础设施工程师则可以专注于扩展、编排和可靠性。OpenRL 让您可以在自己的基础设施上运行相同的训练 API。这种解耦还带来了其他有趣的好处。
Sharing GPUs共享 GPU
Traditional RL loops are strictly sequential. The trainer waits for the sampler to finish rollouts, the sampler waits for the environment to score rewards (which is often bound by slow CPU/network tasks), and the whole loop sits blocked. Your expensive GPUs spend a lot of time doing nothing. The abstraction allows running multiple RL jobs and allows infrastructure engineers to pack the training/sampling steps to utilize more of their GPUs. The graph below shows the GPU consumption in OpenRL for running one, two, and three RL jobs concurrently. 传统的 RL 循环是严格顺序的。训练器要等采样器完成 rollout，采样器要等环境打分奖励（通常受限于慢速 CPU/网络任务），整个循环被阻塞。昂贵的 GPU 大部分时间处于空闲。抽象化后可以同时运行多个 RL 任务，基础设施工程师可以将训练/采样步骤打包，以更充分利用 GPU。下图展示了在 OpenRL 中并行运行 1、2、3 个 RL 任务时的 GPU 使用情况。
Better UX更好的用户体验
Once you separate out the infrastructure behind the APIs, you start to see the gains in user experience of developing the RL loop because AI researchers no longer have to wrangle the complex python dependencies like cuda. When you are doing R&D, you do not have to run the RL loop directly on the machines with GPUs, you can simply run your RL loop on your Mac pointing to the training APIs running on a Kubernetes cluster/VMs.一旦将基础设施抽象到 API 背后，您会发现开发 RL 循环的用户体验大幅提升，因为 AI 研究员不再需要处理复杂的 Python 依赖（如 cuda）。在研发阶段，您无需在配备 GPU 的机器上直接运行 RL 循环，只需在 Mac 上运行 RL 循环并指向运行在 Kubernetes 集群/虚拟机上的训练 API 即可。
Autoresearch自动化研究
We believe that frontier AI research will get more and more automated in the future and abstracting out infrastructure as a building block is key to that. To demonstrate that, we added an autoresearch recipe inspired heavily by karpathy's work. The recipe demonstrates how to conduct parallel experiments to conduct parameter sweep, and improve the reward signal for our text-to-sql recipe for Gemma models.我们相信，前沿 AI 研究未来会越来越自动化，而将基础设施抽象为可构建块是关键。为此，我们加入了一个受 Karpathy 工作深度启发的自动化研究配方。该配方演示了如何并行实验、进行参数扫掠，并提升我们针对 Gemma 模型的文本到 SQL 配方的奖励信号。
What OpenRL is notOpenRL 并非以下内容
- A managed service. OpenRL is self-hosted and not a managed service. We aim to make it easy for users to deploy and operate it on their Kubernetes clusters.托管服务。OpenRL 是自托管的，而非托管服务。我们致力于让用户能够轻松在自己的 Kubernetes 集群上部署和运行它。
- An RL framework. OpenRL gives AI researchers full control over their RL loop.RL 框架。OpenRL 为 AI 研究员提供对其 RL 循环的完整控制权。
Get started快速开始
We have made it easy to run OpenRL on your Mac, Nvidia GPUs, or on GKE. This allows you to test your RL loop on Mac and when you are ready to scale, you can point the RL loop to the OpenRL endpoint running in the GKE cluster.我们已经简化了在 Mac、Nvidia GPU 或 GKE 上运行 OpenRL 的流程。这让您可以在 Mac 上测试 RL 循环，准备好扩展时，只需将 RL 循环指向运行在 GKE 集群中的 OpenRL 端点即可。
Try out our text-to-SQL example for teaching the latest Gemma model SQL here: guides. 在此处尝试我们的文本到 SQL 示例，以教学最新的 Gemma 模型 SQL：guides。
One of the benefits of a Tinker compatible endpoint is that you can use Tinker-Cookbook with OpenRL. Tinker-cookbook is one of the best resources for post training infrastructure for RL.Tinker 兼容端点的一个好处是您可以将 Tinker-Cookbook 与 OpenRL 一起使用。Tinker-cookbook 是后训练 RL 基础设施的最佳资源之一。
Future steps未来计划
We have started with a simple architecture focussing on LoRA fine-tuning and plan to evolve the project in the coming months, so please give it a try and share your feedback. A few things we are very excited to work on:我们从一个聚焦 LoRA 微调的简易架构起步，计划在接下来的几个月中不断演进该项目，请务必尝试并反馈。我们非常期待开展的几项工作包括：
- Full parameter fine-tuning全参数微调
- Multitenancy (simultaneous RL on different types of base models)多租户（在不同类型的基础模型上同时进行 RL）
Acknowledgement致谢
We have been inspired by the work done by various open source projects in AI communities, so huge thank you to Thinking Machines, vLLM, PyTorch, prime-rl, verl, SkyRL, and llm-d.我们受到了 AI 社区众多开源项目的启发，特别感谢 Thinking Machines、vLLM、PyTorch、prime-rl、verl、SkyRL 和 llm-d。
