Julien DelangeJulien Delange
Kassen QianKassen Qian
Rosa TrieuRosa Trieu
When Codiga joined Datadog last year, we integrated our static analyzer product with Datadog’s infrastructure to release Datadog Code Analysis. Codiga’s static analyzer was written in Java and supported code analysis for Python, JavaScript, and TypeScript. It initially used ANother Tool for Language Recognition (ANTLR) for generating an abstract syntax tree (AST) of each language supported by the product.当 Codiga 去年加入 Datadog 时，我们将静态分析产品与 Datadog 基础设施集成，推出了 Datadog 代码分析。Codiga 的静态分析器使用 Java 编写，支持 Python、JavaScript 和 TypeScript 的代码分析。它最初使用 ANother Tool for Language Recognition（ANTLR）为产品支持的每种语言生成抽象语法树（AST）。
Shortly after the acquisition, we worked to expand support to additional languages, but ran into two major issues:收购后不久，我们着手扩展对其他语言的支持，但遇到了两个主要问题：
With our existing tools and flow, some languages were only partially supported使用现有工具和流程时，某些语言仅得到部分支持
Parsing was slower than we needed it to be, especially since analysis time compounded on larger repositories with thousands of files解析速度慢于预期，尤其是在大型仓库（数千个文件）中，分析时间会叠加
To our surprise, these shortcomings were quickly addressed by migrating from ANTLR to Tree-sitter, a popular parsing library to parse source code and build its tree. We found that Tree-sitter has a vibrant open source community that contributes to many languages in full, and that parsing code with it is much faster because the grammar is implemented in C. However, Tree-sitter support of Java was limited at the time - it had the most robust support for C and Rust.令我们惊讶的是，这些不足通过将 ANTLR 迁移到 Tree-sitter——一种流行的源码解析库——得以快速解决。我们发现 Tree-sitter 拥有活跃的开源社区，完整支持多种语言，并且由于语法在 C 中实现，解析速度更快。然而，当时 Tree-sitter 对 Java 的支持有限——它对 C 和 Rust 的支持最为完善。
We found ourselves at a fork in the road: should we continue with a static analyzer in Java, trading performance for predictability, or take the risk of a costly and possibly failed rewrite to Rust in hopes of making static analysis much faster?我们面临抉择：是继续使用 Java 的静态分析器，以可预测性换取性能，还是冒着成本高且可能失败的风险，将其重写为 Rust，以期显著提升静态分析速度？
In this blog post, the first in a series that explores the technical challenges and decisions behind our static analyzer, we’ll cover:在本博客文章（系列首篇，探讨我们静态分析器背后的技术挑战和决策）中，我们将介绍：
Why we chose to optimize for performance为何我们选择优化性能
The architecture of the Datadog static analyzerDatadog 静态分析器的架构
The benefits of migrating from Java to Rust从 Java 迁移到 Rust 的收益
Our experience learning Rust for an efficient migration我们学习 Rust 进行高效迁移的经验
An initial tradeoff between control and performance在控制与性能之间的初始权衡
Let’s start with an important difference between Codiga and Datadog:让我们先来看 Codiga 与 Datadog 的一个重要区别：
At Datadog, users natively run the static analyzer on their own CI instance, which sends the results to Datadog. This approach gives customers more control over the execution of the static analysis process, which has major advantages in terms of security and performance. But it can also become a significant problem in CI environments that are restricted in resources. For example, when using a GitHub Actions runner sized at two cores and 7 GB of RAM, we struggled to get the analyzer to scan fast enough - medium-sized repositories (a few thousand files) should be fully scanned in less than three minutes, but instead, it took five with Datadog.在 Datadog，用户在自己的 CI 实例上本地运行静态分析器，结果发送到 Datadog。此方式让客户对静态分析过程拥有更高的控制权，在安全性和性能方面具有显著优势。但在资源受限的 CI 环境中也可能成为瓶颈。例如，在使用两核 7 GB RAM 的 GitHub Actions Runner 时，我们难以让分析器足够快——中等规模仓库（几千个文件）应在三分钟内完成扫描，但实际用了五分钟。
At Codiga, most of our analysis tools were already running on large instances. The analysis was processed on our servers: developers could trigger a push on their repository and the analysis would automatically be done for them. When we operated our static analyzer at Codiga, we controlled its execution and could allocate resources as needed. We also had the execution environment and the underlying JVM fine-tuned to squeeze the most performance from our existing infrastructure.在 Codiga，我们的大多数分析工具已经运行在大型实例上。分析在我们的服务器上完成：开发者推送代码后，分析会自动执行。我们能够控制静态分析器的执行，并根据需要分配资源。我们还对执行环境和底层 JVM 进行调优，以从现有基础设施中挤出最大性能。
In addition, Codiga’s analyzer required a recent Java Virtual Machine (JVM 17+). This was a new configuration constraint, and customers may already have had a JVM in their CI environment that could have conflicted with the one used to run our static analyzer.此外，Codiga 的分析器需要最近的 Java 虚拟机（JVM 17+）。这是一项新的配置约束，客户的 CI 环境中可能已经存在 JVM，可能与我们运行分析器的 JVM 冲突。
Thus, continuing with the static analyzer in Java presented a number of unpalatable drawbacks for customers: none were impossible to overcome, but all solutions presented limited upside. Should we push forward or take a different route altogether?因此，继续使用 Java 的静态分析器会给客户带来多种不理想的缺点：虽然都不是不可克服的，但所有解决方案的收益都有限。我们是继续前进，还是彻底走另一条路？
To answer this question, we need to understand how the static analyzer works.要回答这个问题，需要先了解静态分析器的工作原理。
Architecture of the Datadog static analyzerDatadog 静态分析器的架构
Our static analyzer is architected in layers. The main components are the parsing part that builds the abstract syntax tree (AST), and the execution part that manipulates the AST, reports code violations, and offers fixes.我们的静态分析器采用分层架构。主要组件包括构建抽象语法树（AST）的解析层，以及操作 AST、报告代码违规并提供修复的执行层。
These two components of the analyzer rely on third-party dependencies:这两个组件依赖于第三方库：
AST build: To build the AST, we use Tree-sitter. There was an initial Java library for Tree-sitter, but it was limited in functionality. (For example, it did not support Tree-sitter pattern matching, which is mandatory for optimizing the speed of the analyzer.) The Tree-sitter core libraries are implemented in Rust.AST 构建：我们使用 Tree-sitter 来构建 AST。最初有一个 Java 版的 Tree-sitter 库，但功能受限。（例如，它不支持 Tree-sitter 的模式匹配，而模式匹配对提升分析器速度至关重要。）Tree-sitter 的核心库是用 Rust 实现的。
JavaScript execution: Our static analysis rules are written in JavaScript. In order to execute these rules, we need a JavaScript runtime to execute the JavaScript code of our rules. The JavaScript execution layer was implemented using the polyglot features of GraalVM.JavaScript 执行：我们的静态分析规则使用 JavaScript 编写。为执行这些规则，需要一个 JavaScript 运行时。JavaScript 执行层是基于 GraalVM 的多语言特性实现的。
From a user experience perspective, we wanted the analyzer’s speed and performance to be optimized to run scans as quickly as possible. This is why we placed an emphasis on Tree-sitter’s pattern matching and fast rule execution.从用户体验角度出发，我们希望分析器的速度和性能得到最大优化，以尽可能快速完成扫描。这也是我们强调 Tree-sitter 的模式匹配和快速规则执行的原因。
From Java to Rust从 Java 到 Rust
Why Rust? Out of all the alternatives to the existing Java implementation of the analyzer, Rust made the most sense given our dependency on Tree-sitter and its level of support for Rust.为什么选择 Rust？在所有可替代现有 Java 实现的方案中，考虑到我们对 Tree-sitter 的依赖以及 Rust 对其的支持程度，Rust 是最合适的选择。
A successful migration from Java to Rust would mean that we could migrate all features while maintaining feature parity. We established a set of tests that would verify the migration was working: a scan with the new Rust version had to produce the same results as its predecessor in Java. This meant that the same program would produce the same violations when scanned by both versions, and the execution time of the new version is at least as fast as the old version (no performance regressions).成功的 Java 到 Rust 迁移意味着我们可以在保持功能完整的前提下迁移所有特性。我们制定了一套测试，用于验证迁移是否成功：新 Rust 版的扫描必须产生与旧 Java 版相同的结果。这意味着同一程序在两个版本扫描时应报告相同的违规，并且新版本的执行时间至少不慢于旧版本（即没有性能回退）。
Migrating the parser component of our analyzer was the easy part. Rust is a first-class citizen in the Tree-sitter ecosystem; using the Rust library of Tree-sitter brought everything out of the box. From this perspective, migrating from Java to Rust was even better for us because we did not need to maintain our own Java library for Tree-sitter.迁移解析器组件是最容易的部分。Rust 在 Tree-sitter 生态中是头等公民；使用 Rust 版的 Tree-sitter 库即可开箱即用。从这个角度看，从 Java 迁移到 Rust 对我们更有利，因为我们不再需要维护自己的 Java Tree-sitter 库。
This migration not only enabled a seamless integration of Rust and Tree-sitter features, but also enhanced our static analyzer’s overall performance. We observed that the migration tripled our performance and resulted in a tenfold reduction in memory usage, which is crucial for optimizing operations in resource-constrained environments like CI systems with limited RAM and CPU resources.此迁移不仅实现了 Rust 与 Tree-sitter 功能的无缝集成，还提升了静态分析器的整体性能。我们观察到性能提升了三倍，内存使用降低了十倍，这对在资源受限的 CI 环境（内存、CPU 限制）中优化运行至关重要。
For the JavaScript runtime used to execute the code for static analysis rules, we migrated from GraalVM to deno. Deno is a simple JavaScript runtime library integrated in Rust powered by v8. We only integrated the deno-core crate that included the JavaScript support without the underlying runtime. We did this for security reasons: static analysis rules need to execute JavaScript code but do not need to do any operation on disk nor communicate over the network.对于执行静态分析规则代码的 JavaScript 运行时，我们将 GraalVM 替换为 deno。Deno 是一个基于 V8、用 Rust 实现的简易 JavaScript 运行时库。我们仅集成了 deno-core crate，其中包含 JavaScript 支持但不包含底层运行时。这样做是出于安全考虑：静态分析规则需要执行 JavaScript 代码，但不需要磁盘操作或网络通信。
Other components of the analyzer had to be migrated as well. The following table summarizes the mapping between each core function and the library used to implement it in Java or Rust.分析器的其他组件也需要迁移。下表概述了每个核心功能在 Java 与 Rust 中对应的实现库。
| Function | Java Library | Rust Library |
|---|---|---|
| Building AST | Tree-sitter (custom library) | Tree-sitter (vanilla support) |
| Parsing YAML | snakeyaml | serde_yaml |
| HTTP client | JVM standard classes | reqwest |
| Concurrent execution | Java Future framework | rayon |
Learning Rust for an efficient migration学习 Rust 进行高效迁移
Datadog Static Analysis was our first Rust project, so we planned for what we expected to be a large undertaking. Not only did we need to migrate our codebase, but we needed to learn Rust from scratch. The initial plan accounted for three months for learning Rust and migrating the entirety of our static analysis tooling from Java to Rust.Datadog 静态分析是我们的第一个 Rust 项目，因此我们为可能的大工程做好了计划。不仅要迁移代码库，还要从零开始学习 Rust。最初计划预留三个月用于学习 Rust 并将全部静态分析工具从 Java 迁移到 Rust。
Going by our previous experiences, we had planned to give ourselves at least a few weeks to become familiar with new concepts (such as the borrow checker) and understand how we would map the existing codebase onto Rust concepts. To our surprise, we gained a firm grasp of the language and a clear idea of how our codebase would be mapped onto Rust within 10 days.根据以往经验，我们本打算给自己至少几周时间熟悉新概念（如借用检查器），并弄清如何将现有代码映射到 Rust。出乎意料的是，我们在 10 天内就掌握了语言要点，并明确了代码映射方案。
Migrating the entire codebase happened even faster than we had anticipated. Within a month, the entire code analysis infrastructure was migrated from Java to Rust, and all customers were running on the new Rust analyzer.整个代码库的迁移比预期更快。一个月内，全部代码分析基础设施已从 Java 完全迁移到 Rust，所有客户均已使用新的 Rust 分析器。
During the process, the hardest part for us was to clearly understand the following concepts:在此过程中，最难理解的概念包括：
The Copy trait vs. Clone trait and when to explicitly clone a value. The Rust ownership feature prevents memory errors and requires an understanding of how values are assigned and copied. Understanding the copy and clone trait are very important if you want to write efficient Rust code.Copy trait 与 Clone trait 以及何时显式克隆值。Rust 的所有权机制防止内存错误，需要了解值的赋值与复制方式。掌握 copy 与 clone trait 对编写高效 Rust 代码至关重要。
The borrow checker and passing references instead of passing the value. We copied values across function calls too often. In the majority of these cases, we could have simply passed a reference value and avoided any cloning.借用检查器以及传递引用而非传值。我们在函数调用间复制值的次数过多。在大多数情况下，只需传递引用即可避免克隆。
Parallelization and concurrency. The rayon library simplifies a lot of operations and prevents developers from implementing their own concurrency mechanisms.并行化与并发。rayon 库简化了大量操作，避免开发者自行实现并发机制。
Once we understood these new concepts, we found that the Rust ecosystem is really efficient. From IDE plugins to CI/CD integration (available directly as GitHub actions), there are a lot of tools that help you write efficient Rust code.弄清这些新概念后，我们发现 Rust 生态非常高效。从 IDE 插件到 CI/CD 集成（直接提供 GitHub Actions），有大量工具帮助编写高效的 Rust 代码。
What helped us the most was to adopt strict coding standards from the beginning:对我们帮助最大的做法是从一开始就采用严格的编码规范：
Linting (rust-clippy) and code formatting (rustfmt) checker at every push在每次推送时使用 lint（rust-clippy）和代码格式化（rustfmt）检查
Adding unit and integration tests as much as we could to prevent any regression尽可能添加单元测试和集成测试，以防止回归
Checking all tests using a pre-commit push and in our CI/CD pipeline在 pre-commit 推送和 CI/CD 流水线中运行所有测试
While having strict guidelines from the beginning made the learning curve steeper (especially because every code style issue detected by rust-clippy and rustfmt had to be addressed before committing our code), it ensured that we maintained high code quality and prevented any performance regressions that could have impacted our customers.虽然从一开始就制定严格指南让学习曲线更陡（因为每个 rust-clippy 和 rustfmt 检测到的代码风格问题都必须在提交前解决），但这确保了我们保持高代码质量，避免了可能影响客户的性能回退。
Enhanced performance in the IDEIDE 中的性能提升
As we mentioned above, our internal tests showed that after migrating, on average, analysis time for a repository had been reduced by a factor of three, which means developers now had faster feedback on their pull requests and platform teams reduced compute time spent in their CI environment.正如前文所述，我们的内部测试显示，迁移后单个仓库的分析时间平均缩短了三倍，这意味着开发者在 Pull Request 上获得更快的反馈，平台团队在 CI 环境中消耗的计算时间也相应减少。
In addition, the new static analyzer is available as a binary and does not need the JVM to run, which reduces setup time on every commit. For example, there’s no need to install a JVM in the CI environment before running the analysis, which saves multiple seconds at every CI run.此外，新静态分析器以二进制形式提供，无需 JVM 即可运行，从而减少每次提交的设置时间。例如，在 CI 环境中无需先安装 JVM 再运行分析，每次 CI 执行即可节省数秒。
Removing our dependency on the JVM and speeding up the analysis enabled us to embed the analyzer directly into the IDE. The very same lightweight and fast analyzer that runs in your CI/CD pipelines simultaneously reports coding errors and suggests fixes in your IDE in real time, tightening the feedback loop for the quality and security of the code you’re writing.摆脱对 JVM 的依赖并加速分析后，我们得以将分析器直接嵌入 IDE。相同的轻量高速分析器既可在 CI/CD 流水线中运行，又能实时在 IDE 中报告代码错误并提供修复建议，进一步紧密了代码质量与安全性的反馈闭环。
What’s next展望未来
If you’re considering migrating your codebase to Rust like us, we advise starting with smaller, non-critical components of your system to become familiar with Rust’s syntax, tooling, and compilation model.如果你也考虑像我们一样将代码库迁移到 Rust，建议先从系统中较小、非关键的组件入手，以熟悉 Rust 的语法、工具链和编译模型。
As the Rust ecosystem continues to grow, we’ve found that specialized tools and libraries needed for Java are not as readily available in Rust. This required us to explore alternatives and invest in developing in-house solutions.随着 Rust 生态的不断壮大，我们发现许多 Java 所依赖的专用工具和库在 Rust 中尚未成熟。这迫使我们探索替代方案并投入开发内部解决方案。
In our next post, we will discuss our experiences within the Tree-sitter ecosystem, detailing our contributions and our collaborative efforts with the open source community.在下一篇文章中，我们将分享在 Tree-sitter 生态中的经验，详细介绍我们的贡献以及与开源社区的合作。
Excited about high-performance Rust? Datadog is hiring.对高性能 Rust 感兴趣？Datadog 正在招聘。
Resources资源
Initial Datadog static analyzer (Java version)最初的 Datadog 静态分析器（Java 版）
Current Datadog static analyzer (Rust version)当前的 Datadog 静态分析器（Rust 版）
