PyTorch: a reference languagePyTorch：一种参考语言

A reference implementation is a simplified but complete version of a system that trades performance in return for clarity. We might then say a reference “language” is the fabric of APIs and conventions from which these implementations are cut. At first glance, PyTorch obviously is a reference language: it is, after all, commonly called the lingua franca of modern deep learning. But upon a closer look, there is confusion:参考实现（Reference implementation）是系统的一种简化但完整的版本，它以牺牲性能为代价换取清晰度。由此我们可以说，参考“语言”就是由这些实现所构成的 API 和规范体系。乍看之下，PyTorch 显然是一种参考语言：毕竟它常被称为现代深度学习的通用语（lingua franca）。但仔细观察，这里存在一些困惑：

• Reference implementations usually aren’t deployed to production. But I do my training jobs with PyTorch!参考实现通常不会部署到生产环境。但我确实是用 PyTorch 来进行训练任务的！

• Everyone’s writing kernels with kernel DSLs. What is the role of PyTorch if it’s just gluing kernels together?每个人都在用内核领域特定语言（Kernel DSL）编写内核。如果 PyTorch 仅仅是将这些内核粘合在一起，那么它的角色又是什么呢？

• AI coding will eventually mean that any stack can be rewritten from scratch; why does being written in PyTorch matter?AI 编程最终意味着任何技术栈都可以从零重写；那么，基于 PyTorch 编写还有什么意义呢？

So for me, recently, an unusually clarifying perspective has been to think of PyTorch as playing a dual role: as both the reference language and the implementation language. When the scale is not too large or the compiler is working well, the reference implementation can ship to production. But increasingly, I think it will be more and more natural to think of the reference implementation as a software artifact that stands apart from the actual production implementation, by which we can verify the correctness of the production implementation. One implementation to research in, one implementation to scale with, and one verifier to, in the darkness, bind them.因此，对我个人而言，最近有一个非常有启发性的视角：将 PyTorch 视为扮演双重角色——既是参考语言，也是实现语言。当规模不大或编译器运行良好时，参考实现可以直接投入生产。但我认为，未来将参考实现视为独立于实际生产实现的软件工件会变得越来越自然，我们可以通过它来验证生产实现的正确性。一套用于研究的实现，一套用于扩展的实现，以及一个在暗处将它们绑定在一起的验证器。

The clearest demonstration of this is in the modern usage of kernel DSLs. The traditional, compiler-maximalist view argues that end users should write implementations of NN modules using a high level API (e.g., a Numpy/PyTorch-style API) which a compiler then determines how to compile into an optimized form. But for the most important operations like matrix multiplies and attention, it is not easy for compilers to guarantee peak performance; the proliferation of kernel DSLs has made it dramatically simpler for people to achieve optimal performance by explicitly spelling out tiling and data movement. Does this eliminate the high level API? Usually not: it’s pretty useful to have a reference implementation in plain PyTorch, and most kernel authors will maintain one in parallel with the optimized kernel, verifying correctness with numerical tests.这一点在内核 DSL 的现代用法中体现得最为明显。传统的“编译器至上”观点认为，终端用户应使用高级 API（例如 Numpy/PyTorch 风格的 API）编写神经网络模块的实现，然后由编译器决定如何将其编译为优化形式。但对于矩阵乘法和注意力机制等最关键的操作，编译器很难保证达到峰值性能；内核 DSL 的激增使得人们通过显式指定分块（tiling）和数据移动来获得最佳性能变得简单得多。这是否消除了高级 API 的必要性？通常不会：保留一份纯 PyTorch 的参考实现非常有用，大多数内核编写者会将其与优化后的内核并行维护，并通过数值测试来验证正确性。

In the same way kernel DSLs have changed how production implementations of operators can be written, I think coding agents change the way production implementations of train steps can be written. Traditionally, we think of autograd as a core part of PyTorch’s value proposition, because it guarantees you will get correct derivatives. However, at scale, the implicit backwards graph becomes an albatross around one’s neck: the majority of your compute is hidden away, with no opportunity to interact with it with normal debugging tools or apply fusions to it in the same way you can do it in eager forwards code. With a compiler, it is possible to modify the backwards graph with, e.g., a pattern match, but this is brittle and a less nice experience than just swapping a call from a reference implementation to a hand-written kernel. This is not a new observation: the now defunct Tangent library was built on the proposition that source-to-source automatic differentiation could be useful.正如内核 DSL 改变了算子生产实现的编写方式一样，我认为编程智能体（Coding agents）也在改变训练步骤生产实现的编写方式。传统上，我们将自动微分（autograd）视为 PyTorch 核心价值主张的一部分，因为它保证了你能得到正确的导数。然而，在大规模场景下，隐式的反向图反而成了沉重的负担：大部分计算被隐藏起来，无法像在即时（eager）前向代码中那样使用常规调试工具进行交互，也无法对其应用融合（fusion）优化。虽然可以通过编译器使用模式匹配等方式修改反向图，但这很脆弱，且体验远不如直接将调用从参考实现切换为手写内核来得好。这并非新发现：现已废弃的 Tangent 库最初就是基于“源到源自动微分（source-to-source automatic differentiation）可能很有用”这一理念构建的。

The new recipe looks like this. Keep the traditional PyTorch autograd-friendly code as the reference implementation. Use LLMs to generate an explicit forward-backward version of the code, which can be optimized separately from the reference implementation. Unlike pattern matching, you never have to worry about your optimizations failing to apply. The cost is that the reference and the real implementation can diverge: we need a verifier that shows us they are equivalent. This verifier can be implemented simply with a bitwise equivalence test, or implemented as some sort of graph capture and structural equivalence, in the tradition of translation validation. To ensure the verifier works when one side has a fusion the other doesn’t, you only need to provide a reference implementation of the fusion (an inverse pattern match, if you will!).新的方案如下：保留传统的 PyTorch 自动微分友好型代码作为参考实现。利用大模型（LLM）生成代码的显式前向-反向版本，并将其与参考实现分开进行优化。与模式匹配不同，你无需担心优化无法应用。其代价是参考实现与实际实现可能会产生分歧：我们需要一个验证器来证明它们是等价的。这个验证器可以简单地通过位级等价测试来实现，或者按照翻译验证（translation validation）的传统，实现为某种图捕获和结构等价性检查。为了确保当一方存在融合而另一方没有时验证器仍能工作，你只需要提供该融合的参考实现（也可以说是一种反向模式匹配！）即可。

I am not going to claim that this recipe is right for everyone. It turns out PyTorch, the reference language, is a pretty good executable spec, and at the end of the day what really matters is how quickly you get the experimental results you need. But, having spent a lot of my time recently thinking about what it means for PyTorch to excel at frontier training–and in particular whether or not it is necessary for PyTorch to disrupt itself as scaling continues–I feel that this perspective helps bridge the old and the new. An open question Horace He posed last year was this: “How can we get all of the control of eager-mode execution with some of the conveniences of graph-level abstraction?” I think this recipe is a pretty promising answer, and PyTorch continues to be at the center of it.我并不是说这个方案适合所有人。事实证明，作为参考语言的 PyTorch 是一个相当不错的可执行规范，归根结底，真正重要的是你能多快获得所需的实验结果。但我最近花了很多时间思考 PyTorch 在前沿训练中脱颖而出意味着什么——特别是随着规模不断扩大，PyTorch 是否需要自我颠覆——我觉得这个视角有助于连接过去与未来。Horace He 去年提出了一个悬而未决的问题：“我们如何才能在获得即时模式执行的所有控制力的同时，又享受到图级抽象的部分便利？”我认为这个方案是一个非常有前景的答案，而 PyTorch 将继续处于这一进程的核心。