How ast-grep Rewrote Tree-sitter in Rust and Made It 30% Faster ast-grep 如何用 Rust 重写 Tree-sitter 并实现 30% 的性能提升

Part 1 of 4 — the complete adventure共 4 部分 —— 完整的探索之旅

ast-grep rewrote Tree-sitter's C core in Rust, with AI writing the code. The new core is faster at parsing, faster at reading the completed tree, and faster in ast-grep itself. (The title's “30%” is the parser-only number; end-to-end, ast-grep runs about 22% faster.)ast-grep 使用 AI 辅助，将 Tree-sitter 的 C 语言核心重写为 Rust。新的核心在解析速度、读取完整树的速度以及 ast-grep 本身的运行速度上都有所提升。（标题中的“30%”仅指解析器层面的数据；端到端来看，ast-grep 的运行速度提升了约 22%。）

Source repository: HerringtonDarkholme/tree-sitter.源码仓库：HerringtonDarkholme/tree-sitter。

Two quick introductions before the numbers. ast-grep — the structural code-search tool this blog belongs to — searches code by syntax rather than by text, so every file it touches must first become a syntax tree. Tree-sitter is the parser framework that builds that tree: you give it a grammar definition, and it generates a fast parser for that language. Born in the editor world, it now powers an enormous ecosystem of grammars and tools.在公布数据前，先做两个简短的介绍。ast-grep 是本博客所属的结构化代码搜索工具，它通过语法而非文本来搜索代码，因此它处理的每个文件都必须先转换为语法树。Tree-sitter 是构建该树的解析器框架：你只需提供语法定义，它就能为该语言生成一个快速解析器。它诞生于编辑器领域，如今已支撑起一个庞大的语法和工具生态系统。

Performance and peak RSS. Throughput is normalized so the unmodified C build (“C / normal”) scores 100; higher is better. RSS is peak resident memory, and the raw-parsing row shows it as a range because it varies across the benchmark's language fixtures. The outline row is ast-grep's real workload: parse every file in a repository, then walk each completed tree to extract a structural outline.关于性能和峰值常驻内存（RSS）。吞吐量已进行归一化处理，将未修改的 C 语言构建版本（“C / normal”）设为 100；数值越高越好。RSS 为峰值常驻内存，原始解析行显示为一个范围，因为其在基准测试的各语言测试用例中会有所波动。大纲行是 ast-grep 的实际工作负载：解析仓库中的每个文件，然后遍历每棵完整的树以提取结构大纲。

Benchmark	C / normal	Rust	Difference
Raw parsing	Throughput: 100 RSS: 8.48–21.41 MiB	Throughput: 129.74 RSS: 8.42–25.70 MiB	+29.74% throughput +20.0% RSS upper bound
Tree traversal	Throughput: 100 RSS: 20.38 MiB	Throughput: 110.16 RSS: 22.20 MiB	+10.16% throughput +8.9% RSS
Complete ast-grep outline	User CPU: 1.233 s RSS: 26.52 MiB	User CPU: 0.960 s RSS: 34.43 MiB	−22.2% user CPU +29.8% RSS

Rust won every parser and traversal fixture, and ast-grep produced exactly the same outline. Memory is the tradeoff: the Rust build uses about 8 MiB more in the ast-grep run. On the much larger TypeScript stress corpus — the TypeScript compiler repository's test-baseline tree, this project's memory torture test — it peaks at 91.2 MiB. That figure is a triumph, not a confession: earlier in the project, the same corpus peaked above 1 GiB.Rust 版本在所有解析和遍历测试中均胜出，且 ast-grep 生成的大纲完全一致。内存是需要权衡的地方：Rust 构建版本在 ast-grep 运行中多使用了约 8 MiB 内存。在规模更大的 TypeScript 压力测试语料库（即 TypeScript 编译器仓库的测试基准树，也是本项目极其严苛的内存测试）上，峰值达到 91.2 MiB。这个数字是胜利的象征，而非妥协：在项目早期，同一语料库的峰值曾超过 1 GiB。

The result is not a 1:1 replacement for upstream Tree-sitter. It is a narrower runtime built for AI coding agents that analyze complete file snapshots:最终成果并非对上游 Tree-sitter 的 1:1 替代，而是一个专为 AI 编程代理设计的、更精简的运行时，旨在分析完整的文件快照：

• Existing generated languages and parsers remain compatible.现有的已生成语言和解析器保持兼容。
• Native loading of WebAssembly-compiled languages and incremental old-tree reuse were removed.移除了 WebAssembly 编译语言的本地加载功能以及旧树的增量复用功能。
• Compatibility still requires many raw pointers and unsafe blocks.兼容性仍需要大量原始指针和 unsafe 代码块。

That boundary keeps the grammar ecosystem useful for agentic coding while removing editor-specific machinery the target workload does not need.这一边界使语法生态系统既能服务于代理式编程，又去除了目标工作负载中不需要的编辑器专用机制。

That is the ending. Getting there was another matter.这就是结局。而实现它的过程则是另一回事。

Why Rewrite Tree-sitter? 为什么要重写 Tree-sitter？

Every serious ast-grep performance investigation eventually arrived at the same place: Tree-sitter.每一次针对 ast-grep 的深入性能调查，最终都会指向同一个地方：Tree-sitter。

ast-grep could make its rules faster. It could prune work, cache configuration, and avoid visiting irrelevant syntax. But every file still had to become a syntax tree first, and Tree-sitter built that tree. The parser was both the foundation and, increasingly, the ceiling.ast-grep 可以优化自身的规则执行速度。它可以裁剪任务、缓存配置并避免访问无关的语法。但每个文件仍然必须先变成语法树，而 Tree-sitter 正是构建这棵树的工具。解析器既是基础，也日益成为了性能的上限。

I had dreamed about rewriting or deeply optimizing it for years. The dream usually lasted until I opened the runtime. There was a mature C implementation, binary compatibility, external scanners, error recovery, incremental parsing, ambiguous grammars, several language bindings, and a small matter of not breaking the enormous grammar ecosystem built on top of all of it.多年来，我一直梦想着重写或深度优化它。但这个梦想通常在我打开运行时代码时就破灭了。那里有成熟的 C 语言实现、二进制兼容性、外部扫描器、错误恢复、增量解析、歧义语法、多种语言绑定，还有一个不能破坏建立在这一切之上的庞大语法生态系统的难题。

For one person, this was not a weekend project. It was a Herculean task wearing a header file.对于一个人来说，这不是一个周末就能完成的项目。这简直是一项披着头文件外衣的“海格力斯式”艰巨任务。

So nothing happened.所以，什么也没发生。

Then AI-assisted rewrite attempts started appearing everywhere—Bun, pgrust, and Roc among them. They did not prove that rewriting Tree-sitter was wise, make the runtime smaller, or make parser theory less strange. They showed that the experiment had become cheap enough for one person to attempt, giving me enough leverage to ask the unreasonable question and get an answer before the decade ended.后来，AI 辅助重写开始在各处涌现——包括 Bun、pgrust 和 Roc 等项目。它们并没有证明重写 Tree-sitter 是明智之举，也没有让运行时变小或让解析理论变得简单。但它们证明了这种实验的成本已经低到个人可以尝试，这给了我足够的筹码去提出那个不合理的问题，并在十年结束前得到答案。

So I instructed ChatGPT to rewrite Tree-sitter's C core in Rust. The project moved from a compatibility-first translation, through a fast but unreadable optimization attempt, to a simpler runtime and real parser gains—only to discover that a faster parser could still make ast-grep slower.于是，我指示 ChatGPT 将 Tree-sitter 的 C 语言核心重写为 Rust。项目从最初的兼容性优先翻译，经历了快速但难以阅读的优化尝试，最终走向了更简单的运行时和真正的解析性能提升——却发现更快的解析器有时反而会让 ast-grep 变慢。

The rest of this post follows that journey: what worked, what had to be reverted, and what it took to turn a parser benchmark win into an application win.本文的其余部分将回顾这段旅程：哪些方法有效，哪些必须回滚，以及如何将解析器基准测试的胜利转化为应用层面的胜利。

Tree-sitter's Parsing Architecture Tree-sitter 的解析架构

Tree-sitter takes source code and produces a syntax tree. Each supported language begins as a grammar definition that Tree-sitter compiles into generated parsing tables and lexer code; when this series says “generated languages,” “generated grammars,” or “generated tables,” it means those artifacts. At runtime, a lexer turns characters into tokens such as identifier, +, and number. The parser then uses the generated table and a stack to decide what each token means.Tree-sitter 接收源代码并生成语法树。每种支持的语言都始于一个语法定义，Tree-sitter 将其编译为生成的解析表和词法分析器代码；当本文提到“生成的语言”、“生成的语法”或“生成的表”时，指的就是这些产物。在运行时，词法分析器将字符转换为标识符、+ 和数字等标记。随后，解析器利用生成的表和一个栈来确定每个标记的含义。

Most of the time, the table requests one of two operations:大多数情况下，表会请求执行以下两种操作之一：

• shift: consume a token and push it onto the parser stack;移进（shift）：消耗一个标记并将其压入解析器栈；
• reduce: recognize that several syntax pieces form a larger grammar rule, replace them with one parent, and continue.归约（reduce）：识别出多个语法片段组成了一个更大的语法规则，用一个父节点替换它们，然后继续。

If every table entry had one valid answer, the parser could follow one history with one stack. That is the ordinary LR case. Programming-language grammars occasionally have genuine conflicts: more than one action may remain valid until additional input reveals which interpretation survives.如果每个表项只有一个有效答案，解析器就可以通过一个栈来跟踪一条历史记录。这就是普通的 LR 情况。编程语言语法偶尔会出现真正的冲突：在更多输入揭示哪种解释正确之前，可能存在多种有效的操作。

Tree-sitter therefore uses generalized LR (GLR). It can follow several histories at once while sharing their common past in a graph-structured stack. Think of one road that can briefly fork, then merge again. The graph is necessary when the grammar is ambiguous. It is considerably less charming when the parser constructs graph machinery for a road that remains straight.因此，Tree-sitter 使用了通用 LR (GLR)。它可以在图结构栈中共享公共历史的同时，跟踪多条历史记录。想象一条道路可能会短暂分叉，然后再汇合。当语法存在歧义时，这种图结构是必要的。但当解析器为本来笔直的道路构建图结构机制时，其效率就大打折扣了。

The other central object is the subtree. A shifted token becomes a leaf; a reduction combines children into an internal syntax node. These values are created during parsing, shared across stack histories, published as the final tree, traversed by ast-grep, and eventually released. Optimizing only their birth while ignoring the rest of that lifetime would later produce a rather expensive lesson.另一个核心对象是子树。移进的标记成为叶子节点；归约操作将子节点组合成内部语法节点。这些值在解析过程中创建，在栈历史记录间共享，作为最终树发布，被 ast-grep 遍历，并最终释放。如果只优化它们的创建过程而忽略其整个生命周期，后来会付出相当昂贵的代价。

That is enough parser theory for the overview.关于解析器理论的概述就到这里。

First Step: Preserve C Behavior in Rust 第一步：在 Rust 中保持 C 语言的行为

The first goal was not elegance. It was parity.首要目标不是优雅，而是对等。

The rewrite contract was deliberately conservative:重写的契约被特意设定得非常保守：

• Use the existing tests as the behavioral oracle. A plausible Rust implementation was not enough; it had to produce the same trees, recovery behavior, navigation results, and public API effects.使用现有测试作为行为准则。仅有一个合理的 Rust 实现是不够的；它必须产生相同的树、恢复行为、导航结果以及公共 API 效果。
• Test the ecosystem, not only handwritten examples. Existing generated grammars and external scanners had to keep working without regeneration or source changes.测试整个生态系统，而不仅仅是手写的示例。现有的生成语法和外部扫描器必须在无需重新生成或修改源码的情况下正常工作。
• Preserve the binary interface (ABI). Generated language tables, public C functions, layouts, symbols, and calling conventions remained compatible while the implementation behind them changed language.保留二进制接口 (ABI)。在实现语言变更的同时，生成的语言表、公共 C 函数、内存布局、符号和调用约定必须保持兼容。
• Translate before redesigning. The first Rust version intentionally resembled the C control flow so parity failures had a bounded search area.先翻译，再重构。第一个 Rust 版本刻意模仿了 C 语言的控制流，以便在出现对等性错误时能锁定排查范围。

In one sentence: preserve everything the ecosystem can observe, then make the inside replaceable.一言以蔽之：保留生态系统所能感知的一切，然后让内部结构变得可替换。

I instructed ChatGPT to translate the runtime one part at a time: basic utilities, tree storage, lexing, the parser stack, tree navigation, and finally the parser loop. The agent read the C and Rust code, wrote patches, fixed compiler errors, ran tests, and investigated mismatches. I supplied the goals, constraints, objections, and decisions. The existing implementation and test suite supplied the answer key.我指示 ChatGPT 分部分翻译运行时：基础工具、树存储、词法分析、解析器栈、树导航，最后是解析循环。AI 读取 C 和 Rust 代码，编写补丁，修复编译器错误，运行测试并调查不匹配之处。我提供目标、约束、异议和决策。现有的实现和测试套件提供了答案对照表。

This distinction matters. I did not personally type a heroic Rust port and then ask AI to polish the comments. The implementation, profiling, instrumentation, and much of the experimental code were produced by the agent under my direction. Without AI, this project would still be an idea I occasionally mentioned before wisely changing the subject.这一点至关重要。我并没有亲自编写一个英勇的 Rust 移植版，然后让 AI 润色注释。实现、性能分析、插桩以及大部分实验性代码都是在我的指导下由 AI 生成的。没有 AI，这个项目至今仍只会是我偶尔提及、随后又明智地转移话题的一个想法。

The C core had become Rust. It compiled. It passed the tests. Existing grammars could use it. This was already the kind of result that had looked impossible when the project was sitting on the shelf.C 语言核心变成了 Rust。它能编译，通过了测试，现有语法也能使用它。这在项目搁置时看起来是不可能完成的结果。

Naturally, I immediately asked for more.当然，我立刻要求做得更多。

Why the First Optimization Attempt Failed 为什么第一次优化尝试失败了？

The request was pure vibe coding — one line typed into a /goal command. The process behind it was more careful than that: I directed ChatGPT to use proper profiling tools, understand the runtime's data layouts and ownership, and look for algorithmic changes instead of merely polishing individual instructions. The benchmarks duly crossed the requested line. Then I opened the code and could not follow it. Layers of overlapping AI-generated optimizations sat on top of a mechanical C-to-Rust translation, and soon the parser began segfaulting: no friendly Rust panic, no failed assertion, the process simply left. A parser that is twenty percent faster and occasionally disappears is not an optimization. It is a benchmark with a jump scare.那次请求纯粹是“氛围编程”——在 /goal 命令中输入的一行指令。其背后的过程比这谨慎得多：我指导 ChatGPT 使用专业的性能分析工具，理解运行时的内存布局和所有权，并寻找算法层面的改动，而不是仅仅润色单条指令。基准测试数据确实达到了目标。但当我打开代码时，我已经无法理解它了。层层叠叠的 AI 生成优化覆盖在机械的 C 转 Rust 翻译之上，解析器很快开始出现段错误（segfault）：没有友好的 Rust panic，没有断言失败，进程直接消失了。一个快了 20% 但偶尔会崩溃的解析器不是优化，而是带有“惊吓”效果的基准测试。

I reverted the optimization work entirely. The twenty percent went with it, and the performance the project eventually reached came later, from the clean, layered work described below. Part 2 tells this story in full. What matters here is how it reversed the project: I stopped asking ChatGPT to make the pile faster and started asking it to make the system explainable.我完全回滚了优化工作。那 20% 的性能随之而去，项目最终达到的性能来自于下文描述的简洁、分层的优化。第 2 部分将完整讲述这个故事。这里重要的是它如何扭转了项目方向：我不再要求 ChatGPT 让代码跑得更快，而是要求它让系统变得可解释。

Second Step: Reduce Scope and Improve Readability 第二步：缩小范围并提高可读性

The cleanup had two parts:清理工作分为两部分：

1. delete features and representations outside the target product;删除目标产品范围之外的功能和表示；
2. turn the retained C-style Rust into code whose ownership and control flow could be reasoned about locally.将保留下来的 C 风格 Rust 代码转变为所有权和控制流可以局部推理的代码。

Neither step promised a heroic benchmark. Both were prerequisites for trusting the next one.这两步都没有承诺带来惊人的基准测试结果，但它们是信任后续工作的先决条件。

Remove Incremental Parsing from the Target Runtime 从目标运行时中移除增量解析

At first, “rewrite Tree-sitter” implied preserving every feature. Then the target workload forced a better question: preserving it for whom?起初，“重写 Tree-sitter”意味着保留所有功能。但目标工作负载迫使我提出了一个更好的问题：这是为谁保留的？

Upstream Tree-sitter is famously useful inside editors. A human inserts one character, deletes two more, and expects highlighting to update before the next frame. Incremental parsing lets the runtime reuse the old tree and rebuild only the affected region. In that world, reparsing the complete file after every keystroke is needless work.上游 Tree-sitter 在编辑器中非常有用。用户插入一个字符、删除两个字符，期望高亮在下一帧之前更新。增量解析允许运行时复用旧树并仅重建受影响区域。在那个世界里，每次按键后重新解析整个文件是多余的。

That was not the world of this branch. ast-grep and the AI coding-agent tools I cared about operate on complete file snapshots: an agent reads a file, analyzes or rewrites it, and asks the tool to process the new snapshot. There is no editor-owned syntax tree advancing one keystroke at a time. A fresh parse is not a degraded fallback; it is the normal operation.但这并非此分支的场景。ast-grep 和我所关注的 AI 编程代理工具操作的是完整的文件快照：代理读取文件、分析或重写它，并要求工具处理新的快照。这里没有编辑器拥有的、随按键逐步推进的语法树。全新解析并非降级退路，而是正常操作。

So I decided to remove incremental old-tree reuse and instructed ChatGPT to do it. The public parameter remains for compatibility, but this runtime parses fresh. The machinery for finding and reusing pieces of the old tree — and it reached into a surprising number of core structures — disappeared from the hot implementation; Part 2 inventories exactly what went.因此，我决定移除旧树的增量复用，并指示 ChatGPT 执行此操作。公共参数为了兼容性得以保留，但此运行时始终进行全新解析。查找和复用旧树片段的机制——它涉及了大量核心结构——从热点实现中消失了；第 2 部分详细盘点了移除的内容。

A second, separate scope cut rode along with it: native loading of Wasm-compiled grammars. Tree-sitter can compile a grammar to WebAssembly and load it at runtime — a capability distinct from the browser Wasm build, which stayed. Native tools set this project's performance target, and runtime Wasm grammar loading was not part of that workload.第二个独立的范围裁剪随之而来：移除 Wasm 编译语法的本地加载功能。Tree-sitter 可以将语法编译为 WebAssembly 并在运行时加载——这与浏览器 Wasm 构建不同（后者得以保留）。本地工具设定了本项目的性能目标，而运行时 Wasm 语法加载并非该工作负载的一部分。

This is not a proposal that upstream Tree-sitter should abandon incremental parsing. It is a product decision for a narrower runtime aimed at file-at-a-time analysis and agent tooling. If this branch returns to interactive editor use, the decision must be reopened. That was the rule: delete only behind a declared boundary, never because a feature happened to be inconvenient.这并非建议上游 Tree-sitter 放弃增量解析。这是一个针对以文件为单位的分析和代理工具的更精简运行时的产品决策。如果此分支回归到交互式编辑器使用场景，该决策必须重新评估。这就是规则：只在明确的边界内删除，绝不因为功能碰巧不方便就删除。

Deletion turned out to be the first real optimization technique: remove work whose use case has already left the building.删除成为了第一个真正的优化技术：移除那些使用场景已经不存在的工作。

Refactor the Retained Runtime for Maintainability 重构保留的运行时以提升可维护性

A line-by-line translation is only readable if the reader already knows the original line by line. I instructed ChatGPT to break the monolithic, pointer-heavy port into more idiomatic internal Rust—without making the ABI-facing types “idiomatic” in ways that would break existing grammars.逐行翻译只有在读者已经逐行了解原始代码时才可读。我指示 ChatGPT 将庞大且充斥指针的移植代码拆分为更地道的 Rust 内部结构——同时不以破坏现有语法的方式将面向 ABI 的类型“地道化”。

The cleanup was less a redesign than a long series of small promotions. Internal raw-pointer parameters became references or slices wherever their lifetimes were local and provable; a sentinel pointer meaning “no node here” became an honest Option. Out-parameters turned into return values where the C ABI did not demand them, large modules split by responsibility so mutation sat next to the state it changed, and the dense tricks that had to stay — compact indexes into the tree, pointer arithmetic — were hidden behind narrow, named operations. Where compatibility was real, the code stayed deliberately ugly: generated-language layouts and exported functions remained C-shaped, because another binary had already committed to that shape.这次清理与其说是重构，不如说是一系列小规模的优化提升。内部原始指针参数在生命周期可证明且局部时，变为了引用或切片；表示“此处无节点”的哨兵指针变成了诚实的 Option。输出参数在 C ABI 不强制要求时变为了返回值，大型模块按职责拆分，使变异操作紧邻其修改的状态，而必须保留的密集技巧（如树的紧凑索引、指针算术）则被隐藏在狭窄、命名明确的操作之后。在必须兼容的地方，代码刻意保持“丑陋”：生成的语言布局和导出函数保持 C 形状，因为其他二进制文件已经承诺了该形状。

That cleanup made questions answerable: Who owns this piece of the tree? Can this reference survive when its storage grows? Why does one reduction create a temporary parser state only to delete it immediately?这种清理使得问题变得可回答：谁拥有这部分树？当存储空间增长时，此引用能存活吗？为什么一次归约操作会创建一个临时解析器状态却又立即删除它？

The important output was not prettier syntax. It was a runtime organized well enough that a segfault, invariant failure, or suspicious allocation had an address in the architecture.重要的产出不是更漂亮的语法，而是一个组织得足够好的运行时，使得段错误、不变量失败或可疑的内存分配在架构中都有迹可循。

GLR and Memory-Layout Optimizations GLR 和内存布局优化

Once I could understand the runtime, I directed ChatGPT back toward reduction — the “reduce” from the architecture section above, and the operation the parser performs constantly.一旦我能理解运行时，我就指示 ChatGPT 回到归约（reduce）——即上文架构部分提到的操作，也是解析器不断执行的操作。

That small operation touches both major data structures. It removes the children from the parser's working stack, then stores them under a new parent in the syntax tree. The profile showed that Tree-sitter was doing much more work around those children than the ordinary case required.这个小操作同时涉及两个主要数据结构。它从解析器的工作栈中移除子节点，然后将它们存储在语法树的一个新父节点下。性能分析显示，Tree-sitter 在这些子节点周围执行的工作远超普通情况所需。

The successful changes eventually fell into four simple principles:成功的改动最终归纳为四个简单的原则：

• Avoid work for uncommon cases. About 99% of the observed parser stack was one straight path, so the parser no longer builds a graph until the input actually forks. Work needed mainly for editing is also kept out of a fresh parse when possible.避免处理罕见情况。观察到的解析器栈中约 99% 是一条直线，因此在输入真正分叉之前，解析器不再构建图结构。主要用于编辑的工作在可能的情况下也从全新解析中剔除。
• Make allocation cheap, and indexes small. Asking the general-purpose allocator for every internal syntax node is expensive. An arena obtains a growing block and serves many nodes from it. Separately, compact indexes reduce the bytes moved between the parser stack and the tree.降低分配成本，减小索引体积。向通用分配器请求每个内部语法节点代价高昂。Arena 分配器获取一个增长的块，并从中提供许多节点。此外，紧凑的索引减少了在解析器栈和树之间移动的字节数。
• Do repeated work once. The parser prepares common grammar lookups ahead of time, and the tree reader avoids looking up the same children repeatedly.重复工作只做一次。解析器提前准备好常见的语法查找，树读取器避免重复查找相同的子节点。
• Give the simplest cases a short path. One parser action is handled directly, and ordinary ASCII input avoids the full character-decoding path. The complete fallback remains available whenever the simple path does not apply.为最简单的情况提供快速路径。单次解析操作直接处理，普通的 ASCII 输入避免了完整的字符解码路径。在简单路径不适用时，完整的后备方案仍然可用。

ordinary parse: keep one stack path
real ambiguity: switch to the full graph

The 99% number described the problem, but not the solution. I directed ChatGPT (in research mode rather than coding mode, this time) to review academic work on generalized parsers, and the result pointed to an older lesson: do not build the general structure until the input actually needs it. The arena was a separate idea and required its own experiments. The linear stack avoided graph bookkeeping; the arena reduced calls to the general allocator. Making the indexes smaller was yet another layout choice, and it did not become faster automatically. Several versions lost before the pieces paid for themselves.99% 的数字描述了问题，但不是解决方案。我指示 ChatGPT（这次是在研究模式而非编码模式下）回顾关于通用解析器的学术工作，结果指向了一个古老的教训：在输入真正需要之前，不要构建通用结构。Arena 是一个独立的想法，需要自己的实验。线性栈避免了图结构簿记；Arena 减少了对通用分配器的调用。减小索引是另一种布局选择，它不会自动变快。在这些改动产生价值之前，有几个版本是失败的。

The important result: ordinary parsing stopped paying the full price of rare ambiguity and of separate allocation for every internal syntax node.重要的结果是：普通解析不再需要为罕见的歧义和每个内部语法节点的独立分配支付全部代价。

For a moment, the parser benchmarks looked excellent.那一刻，解析器的基准测试看起来非常出色。

Then I asked ChatGPT to build ast-grep against it.然后，我让 ChatGPT 构建 ast-grep 来测试它。

End-to-End Performance Findings 端到端性能发现

The agent ran that binary across opencode, a real TypeScript repository. At that point, the parser-only benchmark put this Rust implementation roughly 30% ahead of the C runtime.AI 在 opencode（一个真实的 TypeScript 仓库）上运行了该二进制文件。此时，仅解析器的基准测试显示这个 Rust 实现比 C 运行时快了约 30%。

The application was slower.但应用程序变慢了。

How could a parser become thirty percent faster while the application became slower?解析器快了 30%，应用程序怎么反而变慢了？

The parser benchmark had reused one parser. ast-grep created parsers for thousands of files, then walked every completed tree to extract its outline. The benchmark had measured only the middle of that journey.解析器基准测试复用了同一个解析器。而 ast-grep 为数千个文件创建解析器，然后遍历每棵完成的树以提取大纲。基准测试只测量了那段旅程的中间部分。

The first arena reserved a huge virtual memory region every time a parser was created. It did not immediately consume all that physical memory, which had made the design look harmless. Across a repository, however, that reservation happened thousands of times, and each one cost real work: a fresh round of reserve-and-release syscalls, plus the page-fault and page-table churn behind them, repeated for every file. That churn — not parsing — was the CPU regression on the opencode corpus. I directed ChatGPT to replace the reservation with an ordinary small allocation that grew only when needed.第一个 Arena 每次创建解析器时都会预留一块巨大的虚拟内存区域。它没有立即消耗所有物理内存，这使得设计看起来无害。然而，在整个仓库中，这种预留发生了数千次，每次都伴随着实际的工作：一轮新的预留和释放系统调用，以及其背后的缺页中断和页表抖动，且对每个文件重复。这种抖动——而非解析本身——是 opencode 语料库上 CPU 回归的原因。我指示 ChatGPT 将预留替换为仅在需要时增长的普通小额分配。

That removed one problem and exposed another: memory. On the separate TypeScript stress corpus, measured with one ast-grep worker throughout, the arena's early growth strategy kept old blocks alive and pushed peak memory to 1.04 GiB. Getting from there to the final 91.2 MiB took several rounds of arena surgery — including one twist where the memory I kept telling ChatGPT to reclaim turned out not to be the memory actually being wasted. Part 4 has the traces and the culprit; I will not spoil the reveal here.这消除了一个问题，却暴露了另一个：内存。在单独的 TypeScript 压力测试语料库上，使用单个 ast-grep 工作进程测量时，Arena 的早期增长策略保留了旧块，将峰值内存推高至 1.04 GiB。从那里降至最终的 91.2 MiB 经历了多轮 Arena 手术——包括一个转折：我一直让 ChatGPT 回收的内存，结果证明并不是实际浪费的内存。第 4 部分有跟踪记录和罪魁祸首；我不会在这里剧透。

The completed tree had one more surprise. Compact indexes helped while building it, but ast-grep had to look them up while reading it. Some parser time had simply moved into tree traversal. ChatGPT changed the tree reader to look up each group of children once instead of repeatedly, recovering that cost.已完成的树还有另一个惊喜。紧凑索引在构建时很有帮助，但 ast-grep 在读取时必须查找它们。一些解析时间只是转移到了树遍历中。ChatGPT 修改了树读取器，使其一次性查找每组子节点，而不是重复查找，从而挽回了这部分成本。

Those repairs — ordinary allocation instead of a per-parser virtual-memory ceremony, and a tree reader that resolves each group of children once — closed the regression, and together with a later round of parser-side tuning they became the numbers at the top of this post: the outline run finishing with 22.2% less user CPU than the C build. The end-to-end failure did not invalidate the parser work. It invalidated the old meaning of “worked.” From then on, a performance result needed to cover parsing, memory, reading the tree, and the complete application lifecycle.这些修复——使用普通分配代替每个解析器的虚拟内存仪式，以及一次性解析每组子节点的树读取器——消除了性能回归，再加上后续一轮解析器端的调优，最终得到了本文开头的数字：大纲运行时的用户 CPU 消耗比 C 构建版本减少了 22.2%。端到端的失败并没有否定解析器的工作，而是否定了“工作”的旧定义。从那时起，性能结果必须涵盖解析、内存、读取树以及完整的应用程序生命周期。

There was no single magic patch behind these results. Some changes saved parser time, some prevented a memory disaster, and others recovered time while reading the completed tree. This overview keeps the principles that connect the individual experiments; the detailed posts take them apart.这些结果背后没有单一的“魔法补丁”。一些改动节省了解析时间，一些防止了内存灾难，另一些在读取已完成树时挽回了时间。本概述保留了连接各个实验的原则；详细文章则对它们进行了拆解。

Lessons from the AI-Assisted Rewrite AI 辅助重写的经验教训

At the beginning, AI increased the amount of code a single instruction could set in motion. That was enough to make the rewrite possible and nowhere near enough to make it good.起初，AI 增加了单条指令所能驱动的代码量。这足以使重写成为可能，但远不足以使其变得优秀。

The early loop looked like this:早期的循环看起来像这样：

/goal improve the perf by 20%
        -> a great deal of plausible code
        -> a confusing benchmark
        -> another plausible patch

Later it looked like this:后来它看起来像这样：

find the expensive work
        -> explain why it happens
        -> change one mechanism
        -> compare with the previous Rust revision
        -> test the complete application
        -> retain, revise, or reject

ChatGPT did not gradually become infallible. I gradually learned enough about the runtime to give it narrower problems, challenge bundled assumptions, and demand evidence at the correct boundary. The segfaulting early speedup, the arena's memory explosion, and the slower application all arrived behind reasonable-looking code and encouraging local results. Profiles and tests had to catch what both of us missed.ChatGPT 并没有逐渐变得绝对正确。是我逐渐了解了运行时，从而能够给它更具体的问题，挑战捆绑在一起的假设，并在正确的边界处要求证据。导致段错误的早期提速、Arena 的内存爆炸以及变慢的应用程序，都是在看起来合理的代码和令人鼓舞的局部结果背后出现的。性能分析和测试必须捕捉到我们两人都遗漏的东西。

By the end, the collaboration had found its proper division of labor. ChatGPT could explore implementation space at a pace I could never match by hand. My job was to keep narrowing the question until profiles, invariants, and end-to-end controls could answer it. Speed made the expedition possible; evidence decided which parts returned.到最后，这种协作找到了恰当的分工。ChatGPT 能够以我手工无法比拟的速度探索实现空间。我的工作是不断缩小问题范围，直到性能分析、不变量和端到端控制能够回答它。速度使探索成为可能；证据决定了哪些部分得以保留。

That is the whole adventure. The remaining posts slow it down:这就是整个冒险。剩下的文章将放慢节奏：

1. Rewriting Tree-sitter's C Core in Rust: Migration and Compatibility covers the migration itself, the /goal run and its revert, what this branch deleted, and how the compatibility boundary survived.《将 Tree-sitter 的 C 核心重写为 Rust：迁移与兼容性》涵盖了迁移本身、/goal 运行及其回滚、此分支删除的内容以及兼容性边界是如何存活下来的。
2. Improving Tree-sitter's GLR Algorithm and Memory Layout explains why the parser built a graph for an almost-always-linear workload, then follows the many decisions hidden inside “use an arena.”《改进 Tree-sitter 的 GLR 算法与内存布局》解释了为什么解析器会为几乎总是线性的工作负载构建图结构，并追踪了隐藏在“使用 Arena”背后的许多决策。
3. Optimizing Tree-sitter for End-to-End ast-grep Performance contains the memory traces, the traversal investigation, the benchmark rules — and a further round of parser-side optimization (lookup indexes, single-action dispatch) driven by the application profile.《为端到端 ast-grep 性能优化 Tree-sitter》包含了内存跟踪、遍历调查、基准测试规则，以及由应用程序性能分析驱动的进一步解析器端优化（查找索引、单动作分发）。

The short version is that AI gave me the chance to move a load-bearing wall. The rest of the project was discovering, one benchmark at a time, everything else that wall had been holding up.简而言之，AI 给了我移动承重墙的机会。项目的其余部分就是通过一次次基准测试，去发现那面墙原本支撑着的一切。