Zak van der Merwe has spent his entire career at AWS building control planes. First for EC2 and now for DSQL. On the surface, the control plane looks quite boring: it records what should exist and reconciles that with what actually does. Nobody leaves school dreaming of building one, but Zak will be the first to tell you that if you like solving hard problems in distributed systems, there are few better places to be. It’s where many of those hard problems converge, and where the decisions you make determine whether a service survives its own growth.Zak van der Merwe 的整个职业生涯几乎都在 AWS 构建控制平面，先是为 EC2 构建，现在则是为 DSQL 构建。表面上看，控制平面似乎相当枯燥：它记录应该存在的事物，并将其与实际存在的事物进行协调。没人会在毕业时梦想着去构建控制平面，但 Zak 会第一个告诉你：如果你喜欢解决分布式系统中的难题，这里是最好的去处之一。许多难题都在这里汇聚，而你所做的决策将决定一项服务能否在自身增长中存活下来。

If you’ve been following Marc Brooker’s and Marc Bowes’s writing on DSQL, this is a great companion piece that pulls back the curtain and shows what it means to build a database that was designed from the start with control plane engineers in mind.如果你一直在关注 Marc Brooker 和 Marc Bowes 关于 DSQL 的文章，那么这篇文章将是一个绝佳的配套读物。它为你揭开了帷幕，展示了从一开始就为控制平面工程师而设计的数据库意味着什么。

–W

I’ve been working at AWS for nearly fourteen years, and for almost all of that time I’ve been building control planes. It’s not the kind of career anyone maps out for themselves. Nobody leaves university thinking “I want to spend the next decade making sure the bookkeeping layer of a cloud service stays up.” But here I am, and I think the reason I’m still here is that control planes turn out to be where many of the interesting problems live, even if it takes a while to see that clearly.我在 AWS 工作了近十四年，几乎所有时间都在构建控制平面。这并不是那种人们会为自己规划的职业道路。没人会从大学毕业时想：“我想花接下来十年的时间，确保云服务的记账层保持正常运行。”但我确实走到了这一步，我认为我之所以留在这里，是因为控制平面恰恰是许多有趣问题的发源地，尽管这一点可能需要一段时间才能看清。

Before Amazon, I worked at a telecoms company in Cape Town where we had maybe ten servers, all in a room in the back of the office, and every single one had a name. You’d SSH into them, you’d share them with your colleagues, and if something went wrong you could walk over and deal with it. That was my entire mental model of what it meant to run infrastructure. Servers were things you knew individually, took care of deliberately, and could reason about as a set because there were few enough to fit in your head.在加入亚马逊之前，我曾在开普敦的一家电信公司工作。那时我们大约有十台服务器，都放在办公室后面的房间里，每一台都有自己的名字。你会通过 SSH 登录它们，与同事共享使用，如果出了问题，你可以直接走过去处理。这就是我当时对运行基础设施的全部认知模型：服务器是那些你逐一了解、精心呵护、并且因为数量少到可以装进大脑而能够进行逻辑推理的对象。

I mention this not because it’s an unusual background but because it was so common less than two decades ago, and I think that’s what makes it worth saying out loud. Maybe your version is a small Kubernetes cluster or a handful of RDS instances where you can visualize the whole thing, you can name the parts, and when something breaks you know which part broke. That feeling of knowing your infrastructure is comfortable, and it makes the next part of the story genuinely hard to describe, because what happened when I joined EC2 was that that feeling just evaporated.我提到这一点并不是因为这种背景很罕见，而是因为它在不到二十年前非常普遍，我认为这值得大声说出来。也许你现在的版本是一个小型 Kubernetes 集群或几台 RDS 实例，你可以直观地看到整个系统，能叫出各个组件的名字，当某处损坏时，你也知道是哪一部分出了问题。那种掌控基础设施的感觉很舒适，这使得故事的下一部分变得非常难以描述，因为当我加入 EC2 时，那种感觉瞬间消失了。

Honestly, when I started, I didn’t really understand how EC2 worked. I kept trying to map it back to what I knew. If I launch an instance and the underlying server dies, what happens? Does my VM somehow get teleported onto another host? How does the cloud create this illusion that hardware failures don’t matter? I couldn’t square any of it with what I knew about running software.老实说，刚开始时，我并不真正理解 EC2 是如何工作的。我一直试图用我已知的知识去套用它。如果我启动一个实例，而底层的服务器挂了，会发生什么？我的虚拟机是会自动“传送”到另一台主机上吗？云是如何创造出“硬件故障无关紧要”这种错觉的？我无法将这一切与我所知的软件运行方式联系起来。

My first job at EC2 was health-checking the fleet, pinging every server and trying to figure out if it was healthy or not, and what I found was the opposite of magic. Things were failing constantly. Hosts were going down, hardware misbehaving, disks dying. I had seen the underbelly of EC2 and it was chaotic. My mental model had gone from “servers are precious things you protect” to “everything is on fire all the time.”我在 EC2 的第一份工作是检查机群的健康状况，通过 ping 每台服务器来判断它们是否健康。我发现的情况与魔法恰恰相反：故障无处不在。主机不断宕机，硬件行为异常，磁盘不断损坏。我看到了 EC2 的底层，那里简直是一团糟。我的认知模型从“服务器是需要保护的珍贵之物”转变为“一切都在随时起火”。

It took a while to shake that feeling, but what I would eventually come to realize was that these failures were tiny drops in an enormous ocean of things working fine. The system was just operating at a scale where failures were a constant, a statistical certainty rather than an emergency. And the thing that made it possible to run a service at that scale without a human responding to every failure, the thing keeping everything humming, was the control plane.我花了一些时间才摆脱这种感觉，但我最终意识到，这些故障只是正常运行的巨大海洋中的几滴水。系统只是运行在一种故障成为常态的规模上，这是一种统计学上的必然，而非紧急情况。而正是控制平面，让在无需人工响应每一次故障的情况下运行这种规模的服务成为可能，它让一切保持平稳运转。

One way or another, my years at AWS have been spent working on control planes. Every AWS service has one, and I like to think of them as our unsung heroes. The better they work, the less anyone notices them. They’re the reason you don’t have to name your servers, and the reason that when hardware fails, you as a customer never have to deal with it. I’ve gotten to build control planes for two major AWS services: EC2, and DSQL. They’re nearly a decade apart, yet the hard lessons from building one led directly to the design of the other, and that’s the story I want to tell today.无论如何，我在 AWS 的岁月都是在构建控制平面。每个 AWS 服务都有一个控制平面，我喜欢把它们看作我们默默无闻的英雄。它们工作得越好，就越没人注意到它们。它们是你无需为服务器命名的原因，也是当硬件发生故障时，你作为客户永远不必为此操心的原因。我有幸为两个主要的 AWS 服务构建过控制平面：EC2 和 DSQL。它们相隔近十年，但构建前者所积累的惨痛教训直接指导了后者的设计，这就是我今天想讲的故事。

What is a control plane anyway? 到底什么是控制平面？

At this point, I probably owe you a better explanation of what I mean by control plane and why I think they’re interesting. I’ll use EC2 as an example, because that’s where I learned most of what I know.此时，我大概欠你一个更好的解释，说明我所说的控制平面是什么，以及为什么我认为它们很有趣。我将以 EC2 为例，因为那是我学习到最多知识的地方。

The way I think about it is that every service has a data plane and a control plane. The data plane is the set of core capabilities, the raw computing power, the hardware, the networking. The control plane is the conduit between those capabilities and customers. It’s the thing that takes what exists physically in a data center and presents it to you in a format you can actually consume and get value from. Without the control plane, you’d be back to SSH-ing into named servers in a closet somewhere. With it, you can spin up a thousand machines with an API call and never think about where they live.我的理解是，每个服务都有一个数据平面和一个控制平面。数据平面是核心能力的集合，包括原始计算能力、硬件和网络。控制平面则是这些能力与客户之间的桥梁。它将数据中心物理存在的事物，以一种你可以真正使用并从中获取价值的格式呈现给你。没有控制平面，你又会回到那个需要 SSH 登录到某个储藏室里命名服务器的老路上去。有了它，你可以通过一个 API 调用启动一千台机器，而无需考虑它们住在哪里。

EC2 involves thousands of engineers and more features than anyone can keep track of, and yet the control plane, conceptually… is pretty simple. Stripped down, EC2 lets you rent a virtual machine (VM) in the cloud, and the control plane’s job is to set up and tear down these VMs for you.EC2 涉及成千上万名工程师和多到没人能记得过来的功能，但从概念上讲，它的控制平面……其实很简单。归根结底，EC2 让你可以在云端租用一台虚拟机（VM），而控制平面的工作就是为你创建和销毁这些虚拟机。

I like the analogy of a thermostat, because it’s constantly measuring the temperature, it knows where things need to be, and it’s always nudging the system in the right direction. That’s what our control plane does. It’s a continuous loop, watching the state of the world, comparing it to what should be true, and correcting the difference. When you launch a VM, the control plane records that a VM should exist, finds a physical server in the right data center, sets up the image, configures networking, and launches it. Later, if that server disappears for any reason, the control plane notices and updates its records to reflect reality. It’s always reconciling what is with what should be.我喜欢用恒温器来做类比，因为它不断测量温度，知道系统应该处于什么状态，并始终将系统引导至正确的方向。我们的控制平面就是这样工作的。它是一个持续的循环，观察世界的状态，将其与应有的状态进行比较，并修正差异。当你启动一台虚拟机时，控制平面会记录下“应该存在一台虚拟机”，找到正确数据中心里的物理服务器，设置镜像，配置网络，然后启动它。后来，如果该服务器因任何原因消失，控制平面会注意到这一点并更新记录以反映现实。它始终在协调“实际状态”与“应有状态”。

One thing the team talked about constantly, almost to the point where it became a mantra, was that no matter what happens to the control plane, VMs that are already running need to keep working. We call this static stability, and it sounds obvious because of course running VMs should keep running. But at scale, obvious things are the hardest to protect, because every new feature, every change, every dependency is a chance to accidentally violate that guarantee. Maintaining it is the difference between an outage where customers can’t launch new resources and an outage where everything stops. Both are bad, but the second is catastrophically worse. The fact that EC2 was statically stable gave me some comfort in my early days.团队一直在讨论一件事，几乎成了口头禅：无论控制平面发生什么，已经在运行的虚拟机必须保持工作。我们称之为“静态稳定性”。这听起来很显而易见，因为运行中的虚拟机当然应该继续运行。但在大规模环境下，显而易见的事情往往最难保护，因为每一个新功能、每一次变更、每一个依赖项，都有可能意外破坏这一保证。维持这种稳定性，是导致“客户无法启动新资源”的故障与“一切都停止运行”的故障之间的区别。两者都很糟糕，但后者是灾难性的。EC2 能够保持静态稳定性，这让我在早期的日子里感到些许安慰。

The EC2 team has done a phenomenal job making bad days rare. But understanding what bad days look like shaped a lot of what I know about building control planes.EC2 团队在减少糟糕状况方面做得非常出色。但了解糟糕状况的样子，塑造了我对构建控制平面的许多认知。

Living inside the control plane 生活在控制平面之内

To understand how bad days start, it helps to know how the control plane stores state. At the heart of EC2’s control plane there is a relational database. When customers call the RunInstances API to launch a VM, the most critical thing that happens is that the control plane writes a row into its database: customer X now has VM Y. That’s when the API can safely return.要理解糟糕的情况是如何开始的，了解控制平面如何存储状态很有帮助。EC2 控制平面的核心是一个关系型数据库。当客户调用 RunInstances API 启动虚拟机时，发生的最关键的事情是控制平面在数据库中写入一行记录：客户 X 现在拥有虚拟机 Y。只有这样，API 才能安全返回。

In reality, a single RunInstances request triggers hundreds or thousands of internal API calls between micro and macro-services. Many of these services have their own databases recording their own state. It’s hard to exaggerate how complex this has grown over the years, but at the very bottom of all that complexity, there is a MySQL database, and what’s in that database is supposed to match reality.实际上，一个 RunInstances 请求会触发微服务和宏服务之间数百甚至数千次内部 API 调用。其中许多服务都有自己的数据库来记录各自的状态。很难夸大这些年来它变得多么复杂，但在所有复杂性的最底层，是一个 MySQL 数据库，而数据库中的内容理应与现实相符。

The simplest way things went wrong was also the scariest. Sometimes the primary database server just died. Our solution was a hot standby, a backup server continuously replicating from the primary, ideally only milliseconds behind. When the primary failed, we’d cut over to the standby and it could limit the outage to seconds. The team earned that through years of operational practice, building tooling, writing runbooks, training on-call engineers to execute the switchover under pressure. But seconds of outage still meant pagers getting lit up at 3am and asking humans to make decisions with incomplete information. We kept asking ourselves whether the architecture could take humans out of that loop entirely.最简单也最可怕的故障方式是：主数据库服务器直接挂了。我们的解决方案是热备，即一个备份服务器持续从主服务器复制数据，理想情况下仅有几毫秒的延迟。当主服务器故障时，我们会切换到备用服务器，这可以将故障时间限制在几秒钟内。团队通过多年的运营实践赢得了这一点，包括构建工具、编写运行手册、培训值班工程师在压力下执行切换。但几秒钟的停机仍然意味着凌晨 3 点传呼机响起，要求人工在信息不全的情况下做出决策。我们不断问自己，架构是否能完全将人类从这个循环中剔除。

The slower, more chronic problem was making sure our MySQL database kept up with business growth. This is pretty frustrating when you think about it, because the data plane does all the heavy lifting, like downloading VM images, configuring networking, running workloads, while the database is just keeping track of what exists. Every instance we launched meant more inserts, more updates, and more reads against the database, and eventually the bookkeeper couldn’t keep up with the workers.另一个更缓慢、更长期的挑战是确保我们的 MySQL 数据库跟上业务增长。当你想到这一点时，会觉得非常沮丧，因为数据平面承担了所有繁重的工作，比如下载虚拟机镜像、配置网络、运行工作负载，而数据库只是在记录存在的东西。我们启动的每一个实例都意味着更多的插入、更多的更新和更多的数据库读取，最终，记账员跟不上工人的速度了。

So we introduced more servers replicating from the primary and used these as read replicas. Many of the EC2 APIs don’t make any changes, they just describe the state of your current resources (how many VMs do you have, and so on). We sent traffic for these read-only APIs to our new read replicas and this massively reduced the load on our primary database server. This is standard practice for any team trying to scale up a relational database. Incidentally, this fleet of read replicas is why the EC2 API is eventually consistent, and as Marc Brooker has written, this puts an unfortunate cognitive load on our customers. It’s something we wanted to do better with DSQL, which we’ll get to in a bit.因此，我们引入了更多从主服务器复制数据的服务器，并将它们用作只读副本。许多 EC2 API 不会做任何更改，它们只是描述你当前资源的状态（比如你有多少台虚拟机等）。我们将这些只读 API 的流量发送到新的只读副本，这极大地减轻了主数据库服务器的负载。对于任何试图扩展关系型数据库的团队来说，这是标准做法。顺便说一句，这组只读副本就是 EC2 API 最终一致性的原因，正如 Marc Brooker 所写，这给我们的客户带来了不必要的认知负担。这是我们希望在 DSQL 中做得更好的地方，稍后我们会谈到。

Read replicas bought us time, but every write still funneled through a single primary server, and eventually we had to shard the database. The first phase of this was visible to customers as we split each AWS region into multiple availability zones (AZs), each with their own independent control plane and separate MySQL databases. This helped with both scaling and availability, since zones fail independently and the blast radius of any single failure shrinks. It also became a fundamental building block that allows AWS customers to build architectures resilient to the loss of a single AZ. The second phase was internal: we sharded each zone into what we call cells. Both of these projects took years of engineering time because they required changes across many services. Every place in the codebase that talks to the database has to know which shard to route to. Simple lookups by primary key are straightforward, but anything else, such as joins across data that doesn’t align with your sharding boundaries, gets much trickier. Even the simplest decisions have consequences at this level. Do you shard by account or by resource? Different services choose differently depending on their access patterns, and there’s no universally right answer.只读副本为我们争取了时间，但每一次写入仍然汇聚到单一的主服务器，最终我们不得不对数据库进行分片。这一过程的第一阶段对客户可见，我们将每个 AWS 区域拆分为多个可用区（AZ），每个可用区都有独立的控制平面和独立的 MySQL 数据库。这既有助于扩展又提高了可用性，因为区域故障是独立的，任何单一故障的影响范围都会缩小。这也成为了一个基础构建块，允许 AWS 客户构建能够抵御单个可用区故障的架构。第二阶段是内部的：我们将每个区域进一步分片为我们称为“单元”（cells）的结构。这两个项目都花费了数年的工程时间，因为它们需要在许多服务中进行更改。 codebase 中每一个与数据库通信的地方都必须知道路由到哪个分片。按主键进行的简单查找很直接，但其他任何操作，例如跨越与分片边界不一致的数据进行连接查询，都会变得非常棘手。在这个层面，即使是最简单的决策也会产生后果。你是按账户分片还是按资源分片？不同的服务根据其访问模式会有不同的选择，没有普遍正确的答案。

There is also a human cost to all of this that I don’t think we talk about enough. In those early years, we didn’t have the automation to handle a lot of what a modern control plane just takes care of. When a security vulnerability was discovered and the whole fleet needed to be patched, we didn’t have a system that could say “go update every host at a safe rate.” We would literally recruit the whole team, subdivide all the hosts, and assign shifts. Everyone in the Cape Town office would get a chunk. Go update every one of your hosts, report status. That’s what life looks like without a mature control plane, and it’s the kind of thing that doesn’t scale. You can patch a fleet of a few hundred hosts that way. You cannot patch a fleet of millions that way. The control plane is what eventually got humans out of that loop entirely.这一切还有一个我们讨论得不够多的人力成本。在早些年，我们没有自动化工具来处理现代控制平面所能处理的许多事情。当发现安全漏洞需要修补整个机群时，我们没有一个系统能说“以安全的速度更新每一台主机”。我们不得不动员整个团队，将所有主机细分并分配轮班。开普敦办公室的每个人都会分到一部分。去更新你的每一台主机，然后报告状态。这就是没有成熟控制平面时的生活，这种事情是无法扩展的。你可以用这种方式修补几百台主机，但你无法用这种方式修补数百万台。最终是控制平面让人们完全脱离了这个循环。

If you’ve lived through this progression, the scaling cliffs, the read replica tradeoffs, the sharding projects that always take longer than you think they will, you know it’s a long and painful road, and it’s one that every team building a successful service backed by a relational database eventually walks.如果你经历过这个过程——那些扩展的悬崖、只读副本的权衡、总是比预期耗时更长的分片项目——你就会知道这是一条漫长而痛苦的道路，这也是每个由关系型数据库支撑的成功服务团队最终都要走的路。

Searching for Database Xanadu 寻找数据库的“香格里拉”

After a decade working on EC2, I formed some strong opinions on what my ideal database looks like. It scales with my business without heroics. It is highly available with no downtime for updates, and no servers to babysit. My ideal database lets me leverage the power of the relational data model to model my domain and write software more productively.在 EC2 工作了十年后，我对理想数据库的样子形成了一些强烈的看法。它能随业务增长而扩展，无需英雄式的救火；它高度可用，更新时无需停机，也无需照顾服务器；最理想的数据库能让我利用关系型数据模型的力量来建模领域，并更高效地编写软件。

As it turns out, in the early 2020s, a group of experienced engineers on the databases side of AWS were thinking about exactly how to build this type of database. These engineers were expats from services like EC2 and had felt the pain of operating relational databases firsthand. They were also looking at the lessons learned operating massive scale serverless databases like DynamoDB and dreaming up ways to apply them to relational databases.事实证明，在 2020 年代初，AWS 数据库领域的一群资深工程师正在思考如何构建这种类型的数据库。这些工程师是从 EC2 等服务转岗过来的，亲身感受过操作关系型数据库的痛苦。他们同时也在审视操作 DynamoDB 等大规模无服务器数据库所学到的经验，并构思如何将其应用于关系型数据库。

They wanted to do for databases what EC2 and really Lambda did to servers. If you operate a traditional database with a “head node” you are in the world of “servers with names” like I was before joining EC2. The ideal database would free you from thinking about “databases with names”. Instead, it would have a control plane that takes care of all of that for you so that you can just think about your database as a logical endpoint that’s always available while it scales up and down.他们想为数据库做 EC2（实际上是 Lambda）为服务器所做的事情。如果你操作一个带有“头节点”的传统数据库，你就处于我加入 EC2 之前那种“有名字的服务器”的世界里。理想的数据库将让你不再需要考虑“有名字的数据库”。相反，它会有一个为你处理所有这些事务的控制平面，这样你只需将数据库视为一个逻辑端点，它在扩展时始终可用。

Sometime around 2021, this project really started to pick up steam. We’d figured out an architecture which seemed to deliver on this promise of the ideal database. I got the opportunity to join the team and start building its control plane. This service would launch in GA as Amazon Aurora DSQL in 2025.大约在 2021 年，这个项目真正开始加速。我们找到了一种似乎能实现这种理想数据库承诺的架构。我有机会加入团队并开始构建它的控制平面。该服务于 2025 年作为 Amazon Aurora DSQL 正式发布。

Let’s quickly revisit the major pain points that EC2 went through and see how life is different on DSQL—especially for control plane builders.让我们快速回顾一下 EC2 经历的主要痛点，看看 DSQL 的生活有何不同——特别是对于控制平面构建者而言。

In DSQL, there isn’t one server running your database. DSQL spins up a Firecracker micro-VM per connection, which means every connection is its own small head node. If one fails, only that single connection is affected rather than your whole application. Nobody gets paged, no one has to decide to cut over. I don’t manage standbys anymore, because the architecture has removed humans from that painful loop entirely.在 DSQL 中，没有单一的服务器在运行你的数据库。DSQL 为每个连接启动一个 Firecracker 微型虚拟机，这意味着每个连接都有自己的小型头节点。如果一个连接失败，只会影响该连接，而不会影响整个应用程序。没人会被传呼，没人需要决定切换。我不再需要管理备用服务器，因为架构已经完全将人类从那个痛苦的循环中移除。

Scaling reads was another problem we spent years on at EC2, adding replicas by hand and accepting eventual consistency as the cost. DSQL adds read replicas automatically, and in fact this is one of the primary jobs of the control plane that I helped build. If your application suddenly sees a spike in read traffic, DSQL handles it, and the reads are strongly consistent, always. After years of telling customers “try again in a moment,” this property still blows my mind. It fundamentally simplifies the architecture of any control plane built on DSQL, and it removes that cognitive tax from the developers using the APIs those control planes expose.扩展读取是我们在 EC2 上花费数年解决的另一个问题，我们通过手动添加副本并接受最终一致性作为代价。DSQL 会自动添加只读副本，事实上，这是我参与构建的控制平面的主要工作之一。如果你的应用程序突然遇到读取流量激增，DSQL 会处理它，而且读取始终是强一致性的。在告诉客户“稍后再试”多年后，这一特性仍然让我感到震撼。它从根本上简化了任何基于 DSQL 构建的控制平面的架构，并消除了使用这些控制平面 API 的开发者的认知负担。

And then there’s sharding, which was availability zones and cells at EC2 and took us years. When you build AWS control planes for major new services, you have to anticipate that sharding will become necessary, and experience has shown that it’s cheaper to do it from the start than to retrofit it later. This is an ugly dilemma, because you’re extending your time to market on a speculative future problem, and when delivery timelines get tight, I’ve seen many teams give up on sharding just to ship. DSQL removes that dilemma because it automatically partitions your workload and you don’t have to think about it. You can use all the Postgres goodies you’re used to, complex transactions, multi-table joins, secondary indexes, while knowing your database is going to scale with your needs. Many new AWS control planes over the last decade were built on DynamoDB for this same reason, but DSQL offers a world with fewer compromises. You get the scalability of DynamoDB with the relational programming model that developers actually prefer to work with.还有分片问题，在 EC2 上这涉及可用区和单元，花费了我们数年时间。当你为重要的新服务构建 AWS 控制平面时，必须预料到分片是必要的，经验表明，从一开始就做分片比事后补救更便宜。这是一个丑陋的困境，因为你在一个推测性的未来问题上延长了上市时间，当交付期限紧迫时，我见过许多团队为了交付而放弃分片。DSQL 消除了这个困境，因为它会自动对你的工作负载进行分区，你根本无需考虑它。你可以使用你习惯的所有 Postgres 功能——复杂的事务、多表连接、二级索引——同时确信你的数据库会随着你的需求而扩展。过去十年中，许多新的 AWS 控制平面都是出于同样的原因构建在 DynamoDB 之上，但 DSQL 提供了一个更少妥协的世界。你既能获得 DynamoDB 的可扩展性，又能拥有开发者更偏爱的关系型编程模型。

“Self-hosting” “自托管”

When it came time to choose a database for the DSQL control plane, we chose DSQL. A team that runs on its own product feels every rough edge before its customers do, but getting there meant taking on the same circular dependency we’d faced at EC2: a control plane can’t depend on the thing it controls.当需要为 DSQL 控制平面选择数据库时，我们选择了 DSQL。一个运行在自己产品上的团队会在客户之前感受到每一个粗糙的边缘，但要做到这一点，意味着我们要承担在 EC2 时面临的同样的循环依赖：控制平面不能依赖于它所控制的事物。

We’ve seen two significant benefits from the decision to “self-host”. As customers adopt DSQL, they are creating thousands of databases, and the control plane is continuously scaling their databases up and down based on usage, often very rapidly. All of this customer activity creates “bookkeeping” work for the DSQL control plane, and the amount of this work grows with DSQL adoption. Since the DSQL control plane runs on DSQL, our bookkeeping database scales up to keep up with this increase in demand with minimal work from the team.我们从“自托管”的决定中看到了两个显著的好处。随着客户采用 DSQL，他们正在创建成千上万个数据库，控制平面会根据使用情况持续地对这些数据库进行自动扩展，通常非常迅速。所有这些客户活动都会为 DSQL 控制平面产生“记账”工作，而这项工作量会随着 DSQL 的采用而增长。由于 DSQL 控制平面运行在 DSQL 之上，我们的记账数据库能够以极少的人力投入，自动扩展以满足这种需求的增长。

The other benefit is in how we deal with availability zone outages. DSQL was designed from the ground up to survive single zone failures, but just because a zone is down doesn’t mean that customer workloads stop scaling or that customers stop creating databases. In my EC2 days, zone failures were fire storms as control plane databases died and pagers went off. For the DSQL control plane, these unfortunate bad days are much less painful because the DSQL control plane’s database remains available which allows the control plane to keep doing its critical work that ensures customer databases keep chugging along.另一个好处在于我们如何应对可用区故障。DSQL 从设计之初就考虑了抵御单区域故障，但区域停机并不意味着客户的工作负载会停止扩展，也不意味着客户会停止创建数据库。在我的 EC2 时期，区域故障如同火灾风暴，控制平面数据库崩溃，传呼机响个不停。对于 DSQL 控制平面而言，这些不幸的糟糕日子痛苦得多，因为 DSQL 控制平面的数据库保持可用，这使得控制平面能够继续执行其关键工作，确保客户的数据库保持平稳运行。

Taking off the rose-tinted glasses 摘下玫瑰色的眼镜

If you’re still with me, you’re probably thinking to yourself: “what’s the catch?”如果你读到这里，你可能在想：“有什么陷阱吗？”

As a relatively new service, there are features that we just don’t support yet. Some of these are gaps that we’re actively filling. Others are more nuanced, and we want to take our time to make sure we build the right thing. A good example is foreign key constraints. Foreign key constraints are a classic database feature that can be very useful and aren’t fundamentally hard to implement. However, foreign keys can also be dangerous at scale. We want to get this right, and that takes time.作为一个相对较新的服务，确实有一些功能我们还不支持。其中一些是我们正在积极填补的空白。另一些则更微妙，我们希望花时间确保构建正确的东西。一个很好的例子是外键约束。外键约束是一个经典的数据库功能，非常有用且在实现上并没有根本性的困难。然而，外键在大规模环境下也可能很危险。我们想把这一点做好，这需要时间。

One of the advantages of running Postgres on a single node is that it maintains the working set in memory, and cached reads are insanely fast. Real architectures are more complicated though. For example, a control plane using Postgres would run across multiple availability zones and put a connection multiplexing proxy in front of the database. These are necessary steps for availability and scale, but they increase latency. When you build on DSQL, you don’t need to manage these things yourself. You get good (though not quite single-node Postgres good) latency that remains consistent as your application scales. This is exactly what I want as a control plane builder. Yes, I want fast, but I care even more about predictable latency as my application scales.在单节点上运行 Postgres 的一个优势是它将工作集保存在内存中，缓存读取速度极快。但现实的架构更复杂。例如，使用 Postgres 的控制平面会跨越多个可用区运行，并在数据库前面放置一个连接多路复用代理。这些是可用性和扩展性的必要步骤，但它们增加了延迟。当你基于 DSQL 构建时，你不需要自己管理这些。你获得的是良好的延迟（虽然不如单节点 Postgres 那么快），并且随着应用程序的扩展，这种延迟保持一致。这正是我作为控制平面构建者所想要的。是的，我想要快，但我更关心随着应用程序扩展，延迟是否可预测。

It’s also worth being honest about where things stand for control plane builders at AWS. Migrating something like EC2’s control plane onto DSQL would take years even if we started today, and that’s okay. The ten-odd years I spent on the EC2 control plane taught me that the work that matters most tends to measure its impact in years, not quarters.坦诚地看待 AWS 控制平面构建者的现状也是值得的。即使我们今天开始，将 EC2 的控制平面迁移到 DSQL 上也需要数年时间，这没关系。我在 EC2 控制平面上度过的十年左右的时间教会了我，最重要的工作往往以“年”而非“季度”来衡量其影响力。

Looking around corners 审视未来

We’ve spent most of this post deep in database scaling and life support. It’s a familiar shape for a lot of engineering stories. The problems we faced at EC2, how to go faster without breaking things, how to spend more of our time on the things that matter to customers, how to coordinate across a team that grew from a handful of people to thousands, and how to keep the system reliable while the ground shifted underneath us, are the same problems every engineering organization runs into as it scales. They are close cousins of the problems that produced Amazon’s original distributed computing manifesto back in 1998, and my own focus narrowed over the years to a single version of them, which was how to let individual teams fully own a piece of EC2 and move fast on their most urgent problems without expensive coordination, all while the product still felt like one coherent thing to a customer.我们在这篇文章中花了大部分篇幅深入讨论数据库扩展和生命支持。这在很多工程故事中都很常见。我们在 EC2 面临的问题——如何在不破坏系统的情况下加快速度，如何将更多时间花在对客户重要的事情上，如何在从几个人增长到数千人的团队中进行协调，以及如何在基础不断变动的情况下保持系统可靠——与任何工程组织在扩展时遇到的问题是一样的。它们是 1998 年亚马逊原始分布式计算宣言所产生问题的近亲，而我多年来的关注点缩小到了其中的一个版本：如何让各个团队完全拥有 EC2 的一部分，并在最紧迫的问题上快速行动，而无需昂贵的协调，同时让产品在客户眼中仍然像一个连贯的整体。

When I look at the broader industry today, I see echoes of that same pressure playing out at a scale I did not expect, because the arrival of agentic coding has driven the cost of writing software down to almost nothing, and that pushes the hard part of the work somewhere else. When code is cheap, the bottleneck moves to judgment, to figuring out what to build, how to ship it safely, and how to anticipate what your customers will need before they ask. That is the same shift a good control plane makes for the people who build on it, taking the invisible work of keeping infrastructure alive off their plate so they can spend their attention on their customers, only now it is happening to software development as a whole, and even a single-person team feels the need to scale out.当我审视当今更广泛的行业时，我看到了同样的压力在以我未曾预料的规模上演。因为智能编码（agentic coding）的出现将编写软件的成本降到了几乎为零，这使得工作的难点转移到了其他地方。当代码变得廉价时，瓶颈就转移到了判断力上：弄清楚构建什么、如何安全地交付它，以及如何在客户提出需求之前预见他们的需求。这与优秀的控制平面为在其上构建的人们所带来的转变是一样的——将维持基础设施运行的隐形工作从他们的盘子里拿走，这样他们就可以把注意力花在客户身上。现在，这种情况正在发生在整个软件开发领域，甚至是一个人的团队也感受到了扩展的需求。

I am not going to pretend I know what building software will look like a year from now, because we are in the middle of a remodel and the walls are still open. What I do know is that it is much easier to move fast when you are standing on a foundation that will not crack under you, and that the problems worth spending a career on have always been the ones that need your judgment rather than your ability to keep the bookkeeping layer from falling over. My hope is that DSQL gives the next generation of builders that foundation, and gives them back the time to go look around corners for their customers, which is the part I always wished we had more room for at EC2.我不会假装我知道一年后的软件构建会是什么样子，因为我们正处于重塑之中，墙壁还是敞开的。我所知道的是，当你站在一个不会在你脚下开裂的地基上时，前进会容易得多；而且值得投入职业生涯去解决的问题，一直是那些需要你的判断力，而不是你需要去防止记账层崩溃的问题。我希望 DSQL 能为下一代构建者提供那个地基，让他们有时间去为客户探索未来，而这正是我一直希望在 EC2 时能有更多空间去做的事情。

And as Werner says: “Now, go build.”正如 Werner 所说：“现在，去构建吧。”