For as long as most of us have been building with data, the systems we reach for — databases, query engines, data warehouses — have, at any appreciable scale, been separate systems. As a community, we’ve generated a lot of healthy arguments about their design along the way. Single host, clustered, or distributed, whether data should all live in memory, whether throughput or latency was the thing that mattered most, but almost all of them have been big systems that live on the other side of a wire. And that’s changing, because the relative costs of compute, memory, and network on a single machine are not the constraints they once were, and a lot of the work we used to send away no longer needs to leave the application.在大多数人从事数据构建工作的这段时间里，我们所接触的系统——数据库、查询引擎、数据仓库——在达到一定规模时，往往都是相互独立的。作为一个社区，我们在此过程中围绕它们的设计进行了许多有益的争论。无论是单机、集群还是分布式，数据是应该全部存放在内存中，还是吞吐量或延迟更为重要，这些几乎都是运行在网络另一端的庞大系统。但这种情况正在改变，因为单台机器上计算、内存和网络的相对成本已不再像过去那样成为制约因素，我们过去需要发送出去处理的许多工作，现在已无需离开应用程序。

In this post, Andy Warfield explains how databases like DuckDB are enabling a new way to build with data, why they matter right now, and how they complement the work we’ve been doing in S3 (e.g., S3 Files, S3 Tables, S3 Vectors). And most importantly, why DuckLabs, the team behind DuckDB, is joining AWS.在这篇文章中，Andy Warfield 解释了像 DuckDB 这样的数据库如何开启了一种全新的数据构建方式，为什么它们在当下如此重要，以及它们如何与我们在 S3 领域所做的工作（例如 S3 Files、S3 Tables、S3 Vectors）形成互补。最重要的是，他还说明了 DuckDB 背后的团队 DuckLabs 为何加入 AWS。

Enjoy.请尽情阅读。

—W

A (somewhat uncomfortably) long time ago in a British pub 很久以前（有点尴尬地）在一家英国酒吧里

Possibly even more than other places, evenings at the pub are an integral part of grad school in the UK. I still remember in those conversations that some of my math and physics friends would poke fun at the computer scientists: “any discipline that needs to put science in its name probably isn’t a science.” I’m sure that I said some uncomfortable things back to them, but there was a grain of truth to what they were saying. It was a weird grain of truth though because it was grating enough to stick with me, and the more I thought about it, the more I realized that by their definition, not being a science was possibly the thing that I loved the most about working on computer systems.在英国读研期间，酒吧里的夜晚是生活中不可或缺的一部分，这或许比在其他地方更为普遍。我至今仍记得，在那些交谈中，我的一些数学和物理学朋友会调侃计算机科学家：“任何需要在名字里加上‘科学’二字的学科，大概都不是真正的科学。”我敢肯定，我也曾反唇相讥，说了一些让他们感到不快的话，但他们的话确实有几分道理。这道理虽然听起来有些刺耳，却让我记忆犹新。我思考得越多，就越意识到，按照他们的定义，这种“非科学”的特质恰恰是我热爱计算机系统工作的原因所在。

Here’s what I mean: where the physical sciences explore the world as it is and look for deeper and deeper understandings of an immovable, inevitable truth, computer science — and especially computer systems — has always been about finding the most elegant solution to a problem in the face of changing “physics”. The relative differences between things, like the speed of memory compared to the speed of the network, the richness of software abstractions relative to available compute power, and so on. It’s an art trying to engineer elegant compromises, to do your best in a world of invariants that don’t stay still over time. And that even the success of your own system can wind up pushing you into a different scale of operation, and a different set of constraints than the ones you started out with.我的意思是：物理科学探索的是客观世界，寻求对某种不可动摇、必然真理的深入理解；而计算机科学——尤其是计算机系统——一直致力于在不断变化的“物理法则”下，找到解决问题的最优雅方案。这些“物理法则”指的是事物间的相对差异，比如内存速度与网络速度的对比、软件抽象的丰富程度与可用计算能力的对比等等。在不变的定律随时间推移而失效的世界中，努力设计出优雅的折中方案，并做到极致，这本身就是一种艺术。甚至你自身系统的成功，最终都可能将你推向与起步时完全不同的运行规模和约束条件。

There are loads of really neat examples of these tradeoffs shaping systems design. The Berkeley Network of Workstations (NOW) project was one of the first systems research projects that I remember really capturing my imagination as a computer science student. NOW capitalized on relatively inexpensive compute and the emerging bandwidth of commodity networks to build interesting scalable distributed systems. Later, the graduate work I was involved in with the Xen hypervisor took advantage of the opposite trend: Xen used the increasing abundance of CPU, memory, and network capabilities on individual servers to build isolated virtual machines and allowed the same hardware to be used by mutually untrusting tenants. Database design has always been about making the most of the current hardware capabilities. To pick a single example, at CWI in Amsterdam, the MonetDB and X100 work took advantage of the fact that the bottleneck in query processing had quietly moved off the disk and into the CPU — cache behaviour, branch prediction, superscalar pipelines — to rebuild query execution around batches of values small enough to stay in cache. All of these systems are interesting because of how they reimagined design in the face of changing constraints and evolving physics.有许多绝佳的例子展示了这些权衡如何塑造了系统设计。伯克利工作站网络 (NOW) 项目是我作为计算机系学生时，第一个真正让我着迷的系统研究项目。NOW 利用了相对廉价的计算能力和新兴的商用网络带宽，构建了有趣且可扩展的分布式系统。后来，我参与的 Xen 虚拟机管理程序的研究生工作利用了相反的趋势：Xen 利用单个服务器上日益丰富的 CPU、内存和网络能力来构建隔离的虚拟机，并允许互不信任的租户使用相同的硬件。数据库设计一直致力于充分利用当前的硬件能力。举个例子，在阿姆斯特丹的 CWI，MonetDB 和 X100 的工作利用了这样一个事实：查询处理的瓶颈已经悄然从磁盘转移到了 CPU 上——缓存行为、分支预测、超标量流水线——从而围绕足以驻留在缓存中的小批量数据重新构建了查询执行过程。所有这些系统之所以有趣，正是因为它们如何在不断变化的约束和演进的物理法则面前，重新构想了设计。

A fun bit of this is that these types of constraints tend to run in cycles. A lot of the virtualization ideas that we explored in Xen were established in the 60s on IBM mainframes. Despite wildly different form factors between massive mainframes and commodity servers, the common theme at those two moments in time was that CPU was abundant and finer-grained isolation was desirable. This cyclical nature of systems design also means that as physics evolve and new designs emerge there’s often an initial reaction along the lines of “we did all this in the 70s,” but then a bit of a more nuanced realization that we are coming back to old ideas with fresh eyes and the benefit of other lessons we’ve learned since the last time.有趣的是，这类约束往往呈现周期性。我们在 Xen 中探索的许多虚拟化思想，早在 60 年代的 IBM 大型机上就已经确立了。尽管庞大的大型机和商用服务器在外形上大相径庭，但这两个时间点的共同主题都是：CPU 资源充足，且更细粒度的隔离是非常可取的。系统设计的这种周期性也意味着，随着物理法则的演进和新设计的出现，人们往往会产生“我们在 70 年代就已经做过这些了”的初步反应，但随后会产生一种更微妙的认识：我们正带着全新的视角和自上次尝试以来所学到的经验教训，重新审视这些旧思想。

One place where the physics have changed an awful lot and have wildly influenced system design is in data processing. Processing data, especially for reasonably sophisticated queries, turns into a very complicated systems problem very quickly. There’s a direct tension between the complexity of the query and the scale of the data being processed, which is that data dependencies between data in different parts of a data set make parallelization difficult and inefficient because you need more and more fast memory to work across those dependent bits of data, and parallelization, either because of distribution or concurrency control, inevitably means doing stuff that complicates and slows down that memory. For this reason, processing tasks are always easiest and most efficient to deal with on a single, fast computer, BUT, the volume of data that we deal with just keeps growing and growing. And so, when we need to process really big data sets and the constrained resource (the physics of the day) is our ability to read it fast enough – where the NIC or the disk on a single server just can’t even make it through the volume of data we want to process in, say, a single day — we tend to reach for distributed processing tools. We find ways to partition the processing task, we spread it out across a lot of parallel servers, we do as much as we can in partitioned parallel execution, and then we combine the results (hopefully as a much more compact set of intermediate data with a lot less work required) into a single answer.物理法则发生了巨大变化并对系统设计产生深远影响的一个领域是数据处理。处理数据，尤其是处理相当复杂的查询，很快就会变成一个非常复杂的系统问题。查询的复杂性与所处理数据规模之间存在直接的张力：数据集中不同部分之间的数据依赖关系使得并行化变得困难且低效，因为你需要越来越多的高速内存来处理这些相互依赖的数据片段；而并行化（无论是由于分布式还是并发控制）不可避免地意味着要进行一些使内存处理变得复杂和缓慢的操作。因此，处理任务在单台快速计算机上总是最简单、最高效的，但是，我们处理的数据量却在不断增长。所以，当我们确实需要处理超大数据集，而受限资源（当下的物理法则）是我们读取数据的速度——即单台服务器的网卡或磁盘甚至无法处理我们在一天内想要处理的数据量时——我们往往会求助于分布式处理工具。我们寻找各种方法来划分处理任务，将其分散到大量并行服务器上，在分区并行执行中尽可能多地完成工作，然后将结果（希望是一组更紧凑、工作量更小的中间数据）合并成一个单一的答案。

Limited I/O bandwidth to large datasets was certainly the physics of the day in the early 2000s, and so scale-out processing was the core idea behind both Google’s work on MapReduce, and later behind Spark’s Resilient Distributed Datasets (RDDs). These systems were all conceived in the face of two invariants (from that time): the data sets that we wanted to process were getting pretty darned big, and the single host commodity network interface was comparatively small. These systems solved a really important problem in working with data and have had enormous influence on how we do data processing at almost every scale today.对大型数据集有限的 I/O 带宽无疑是 21 世纪初的物理法则，因此横向扩展处理成了 Google MapReduce 工作以及后来 Spark 弹性分布式数据集 (RDD) 的核心思想。这些系统都是在那个时代两个不变的约束下构思出来的：我们想要处理的数据集变得非常庞大，而单机商用网络接口却相对狭窄。这些系统解决了处理数据时一个非常重要的问题，并对我们今天在几乎所有规模下的数据处理方式产生了巨大的影响。

I’ve always found two aspects of these distributed query processing systems to be remarkable. First, they innovated on the developer ergonomics of data processing. I think we take this a little bit for granted sometimes, but MapReduce, as an example, observed that a pretty simple pattern from functional programming (database people will pedantically point out that it should actually be called Map/Group-by-and-aggregate, but it’s a little less catchy) could sneakily force developers to articulate data processing tasks in a way that made the opportunity for parallelism very, very explicit. Spark took this even further, extending the idea of lazily evaluated operator chaining (and eventually dataframes) into something that was relatively easy to program, but could then be passed behind the scenes to a query planner and scheduler in order to decompose the developer’s analysis code into a collection of tasks that could be dispatched onto a distributed set of computers. These systems and interfaces did a really good job at helping developers ignore the fact that their code was running on the other side of a wire. They took the physical necessity of having a remote, distributed system, and helped the developer work through the constraints of that system without having to think about splitting up their query, sending stuff over the network, dealing with failures, and so on.我一直觉得这些分布式查询处理系统有两个方面非常了不起。首先，它们在数据处理的开发者工效学上进行了创新。我认为我们有时对此有点理所当然，但以 MapReduce 为例，它观察到函数式编程中一个非常简单的模式（数据库专家会迂腐地指出它实际上应该被称为 Map/Group-by-and-aggregate，尽管它听起来没那么顺口）可以巧妙地强制开发者以一种使并行化机会非常明确的方式来表述数据处理任务。Spark 更进一步，将惰性求值的算子链（以及后来的数据帧）扩展为一种相对容易编程的东西，然后可以在后台传递给查询规划器和调度程序，以便将开发者的分析代码分解为可以分发到分布式计算机集合上的任务集合。这些系统和接口在帮助开发者忽略代码运行在网络另一端这一事实方面做得非常出色。它们承担了拥有远程分布式系统的物理必要性，并帮助开发者在不考虑拆分查询、通过网络发送数据、处理故障等问题的情况下，克服了该系统的约束。

The second thing I find fascinating is the way these systems tend to think about performance. The tradeoff that they’ve often made, and it’s a very reasonable one, is to engineer for arbitrarily high parallelism and throughput. They are comfortable paying an up-front cost for planning jobs, for shipping tasks around, and the other overheads of distribution because the throughput benefit of parallelism was so meaningful. The ability to drive throughput, and the availability of enormous amounts of parallel compute in cloud environments, has meant that these systems have tended to achieve that throughput by adding additional computers to the processing job rather than by being lean.我发现的第二件令人着迷的事情是这些系统思考性能的方式。它们经常做出的权衡（这是一个非常合理的权衡）是为任意高的并行度和吞吐量进行工程设计。它们乐于为规划作业、传输任务以及分布式带来的其他开销支付前期成本，因为并行化带来的吞吐量收益非常显著。驱动吞吐量的能力以及云环境中海量并行计算资源的可用性，意味着这些系统往往通过增加额外的计算机来完成处理作业，而不是通过精简自身来实现吞吐量。

I mean this much more as an observation than as a criticism of these systems, because they were building for their own physics. The fact that they were distributed systems meant that queries would always be sent to run remotely – we were always talking to a server at the other end of a wire, and so we built ergonomic developer abstractions that made that manageable and set the appropriate expectations. If we needed more throughput or to deal with larger data sets, we could just run jobs on even bigger clusters. But it’s interesting to think about how this necessarily remote structure created its own physics for analytics developers: developers learned to expect latency in running jobs and they grew to depend on having a remote cluster to be able to do meaningful processing.我这样说是为了观察而非批评这些系统，因为它们是在为自己的物理法则进行构建。它们是分布式系统这一事实意味着查询总是被发送到远程运行——我们总是在与网络另一端的服务器对话，因此我们构建了符合人体工程学的开发者抽象，使这种管理变得可控，并设定了适当的预期。如果我们需要更高的吞吐量或处理更大的数据集，我们只需在更大的集群上运行作业。但有趣的是，这种必然的远程结构是如何为分析开发者创造了属于它自己的物理法则：开发者学会了预期运行作业时的延迟，并且变得依赖于拥有一个远程集群来完成有意义的处理。

“You can have a second computer once you’ve shown you know how to use the first one.” –Paul Barham“一旦你证明自己知道如何使用第一台计算机，你就可以拥有第二台。”——Paul Barham

While all of this distributed query processing work has been evolving, something has been happening to the physics of the systems that motivated them. Nineteen years ago, a then-new m1.xlarge (the beefiest EC2 instance on AWS at the time) had 15 GB of RAM, 4 virtual cores, and roughly 1 Gb/s of network connectivity. Today, a single m8g.48xlarge has about 50× more memory, 50× more cores, and 50× more network bandwidth. The servers that we work on today have more parallelism and aggregate I/O bandwidth than the clusters that many of us first ran Hadoop and Spark on. Heck, even the MacBook Pro that I’m writing on right now has 3-5x the CPU cores and RAM, about 40x the memory bandwidth and over 100x the I/O bandwidth of that m1.xlarge. And yes, the data sets have grown too, but the thing about data set growth is that it happens along a distribution. The largest data sets have become exponentially larger, but those are the tail. Many data sets scale with very human things: the size of a business, the number of customers it serves, or the number of bank transactions a person makes in a day. This growing hardware advantage relative to data size has driven a resurgence of interest in extremely efficient single-host engines.在所有这些分布式查询处理工作不断演进的同时，驱动它们的系统物理法则也在发生变化。十九年前，当时全新的 m1.xlarge（当时 AWS 上最强大的 EC2 实例）拥有 15 GB 内存、4 个虚拟核心和大约 1 Gb/s 的网络连接。今天，单个 m8g.48xlarge 拥有大约 50 倍的内存、50 倍的核心数和 50 倍的网络带宽。我们今天使用的服务器拥有比我们许多人最初运行 Hadoop 和 Spark 的集群更多的并行性和总 I/O 带宽。见鬼，即使是我现在正在写作的 MacBook Pro，其 CPU 核心数和内存也是那台 m1.xlarge 的 3-5 倍，内存带宽约为 40 倍，I/O 带宽超过 100 倍。是的，数据集也增长了，但数据集增长的特点是它沿着某种分布发生。最大的数据集呈指数级增长，但那些只是极少数。许多数据集随着非常人性化的因素而扩展：企业的规模、服务的客户数量，或者一个人一天内进行的银行交易次数。这种相对于数据规模不断增长的硬件优势，激发了人们对极其高效的单机引擎的重新关注。

In 2015, Frank McSherry, Michael Isard, and Derek Murray wrote a wonderfully irreverent paper, “Scalability! But at what COST?,” about exactly this topic, poking a bit of fun at the (in)efficiency of large distributed processing systems. They took some common data processing tasks, built a good single-threaded version, and compared it to the scale a distributed framework needed to match that performance. It was an enjoyable and surprising read because they showed, for example, that a well-optimized implementation running on a single thread could beat distributed graph systems running on 128 cores, and that it took 512 cores before the distributed version finally pulled ahead. The paper was punctuated by the fact that the authors all worked on distributed data processing systems themselves, and so it wasn’t so much a poke at distributed designs as it was a call to pay attention to per-core efficiency.2015 年，Frank McSherry、Michael Isard 和 Derek Murray 写了一篇非常离经叛道的论文《可扩展性！但代价是什么？》(Scalability! But at what COST?)，专门讨论了这个话题，并对大型分布式处理系统的（低）效率进行了一番调侃。他们选取了一些常见的数据处理任务，构建了一个优秀的单线程版本，并将其与分布式框架达到相同性能所需的规模进行了比较。这是一篇令人愉快且令人惊讶的阅读材料，因为他们展示了，例如，在单线程上运行的优化良好的实现可以击败在 128 个核心上运行的分布式图形系统，并且分布式版本需要 512 个核心才能最终领先。这篇论文的重点在于作者本人都从事分布式数据处理系统的工作，因此与其说是对分布式设计的抨击，不如说是呼吁关注单核效率。

Their work resonated with (and I think continues to influence) performance-leaning systems people, and I was reminded of it in one of my first conversations with Hannes Mühleisen and Mark Raasveldt, the creators of DuckDB — when over pints of beer in Amsterdam, they quoted Paul Barham’s epigraph in the COST paper: “You can have a second computer once you’ve shown you know how to use the first one.”他们的工作引起了（我认为并将继续影响）倾向于性能的系统人员的共鸣，我在与 DuckDB 的创建者 Hannes Mühleisen 和 Mark Raasveldt 的第一次谈话中就想到了这一点——当时在阿姆斯特丹喝着啤酒，他们引用了 COST 论文中 Paul Barham 的题词：“一旦你证明自己知道如何使用第一台计算机，你就可以拥有第二台。”

If it walks like a duck… 如果它走起来像鸭子……

I’m not saying the COST paper was the specific motivation for DuckDB, but it captured a sentiment that Hannes and Mark clearly agreed with. They’d both been researchers at CWI, in the same lab that produced the MonetDB and X100 work I mentioned earlier, so they came at analytics with a deep interest in ideas like vectorization and the goal of getting everything they could out of a single CPU. When they started the project in 2018, that focus on efficiency led them somewhere different from distributed systems, toward an analytics engine that runs as a library, in the same address space as the application that wants to process data. Their 2019 SIGMOD demo paper used the popularity of SQLite as an example of how effective it could be to package an engine to be embedded inside applications and argued for the potential value of an efficient analytics-focused data processing library.我并不是说 COST 论文是 DuckDB 的具体动机，但它捕捉到了 Hannes 和 Mark 显然认同的一种情绪。他们都曾是 CWI 的研究员，就在我前面提到的产生 MonetDB 和 X100 工作的同一个实验室，所以他们带着对向量化等思想的浓厚兴趣以及从单个 CPU 中获得一切可能性能的目标进入了分析领域。当他们在 2018 年启动该项目时，这种对效率的关注将他们引向了与分布式系统不同的方向，即一个作为库运行的分析引擎，与想要处理数据的应用程序处于同一个地址空间。他们 2019 年的 SIGMOD 演示论文以 SQLite 的流行作为例子，说明了将引擎打包嵌入到应用程序中是多么有效，并论证了一个高效的以分析为中心的数据处理库的潜在价值。

Running as an embedded library was a clever reimagining of where the “work” happens, working in-situ on the same in-memory data structures you already use for your tables, and caring as much about its own overhead as about the queries it runs. The research project led to a wildly popular open-source engine, called DuckDB. (Fun aside: “Duck” was because the team felt that database projects tended to use a lot of names that emphasized hyper-awesome-ultra-performance and they found it grating. Shortly before starting the project, Hannes had a pet duck named Wilbur, possibly the most influential duck in the history of computer science.) Importantly, the fact that it’s structured as an embedded library doesn’t mean that it has to always be in the client application. The library form factor just means that it doesn’t have to exist as an external service, usually on the other end of the wire. Instead, the library turns into a building block that can be placed at the appropriate point(s) in the software stack to be effective. The most extreme version of that is that the engine compiles to WebAssembly and runs entirely inside a browser tab, which you can see for yourself at https://shell.duckdb.org/.作为嵌入式库运行是对“工作”发生地点的巧妙重构，在与你已经用于表的内存数据结构相同的位置进行原位工作，并且像关注它运行的查询一样关注它自身的开销。该研究项目产生了一个广受欢迎的开源引擎，名为 DuckDB。（顺便提一下：“Duck”是因为团队觉得数据库项目往往使用很多强调超级-牛逼-极致-性能的名字，他们觉得这很刺耳。在启动项目前不久，Hannes 养了一只名叫 Wilbur 的宠物鸭，这可能是计算机科学史上最有影响力的鸭子。）重要的是，它被构建为嵌入式库并不意味着它必须始终在客户端应用程序中。库的形式意味着它不必作为外部服务存在，通常不需要在网络的另一端。相反，库变成了可以放置在软件栈适当位置的构建块，从而发挥作用。最极端的版本是该引擎编译为 WebAssembly 并完全在浏览器标签页中运行，你可以在 https://shell.duckdb.org/ 上亲自查看。

The S3 team started to see exactly this sort of application-driven use in our own customer workloads. Application builders were embedding DuckDB directly to work with their data, and that pattern has only accelerated since. It’s what led me to reach out to the DuckLabs team while we were working on S3 Tables. We had decided to extend S3 with a first-class tabular storage primitive built on Iceberg, because some of our largest Spark customers were adopting Iceberg and wanted storage that was an intentional fit for tabular data. Having watched this excitement around DuckDB in a different set of our customers, I wanted to be sure S3 Tables would be a great fit for the DuckDB users as well. And that’s how I found myself sitting with Hannes and Mark, over those pints in Amsterdam.S3 团队开始在我们自己的客户工作负载中看到这种应用程序驱动的使用方式。应用程序构建者直接嵌入 DuckDB 来处理他们的数据，而这种模式此后只增不减。这促使我在我们致力于 S3 Tables 时联系了 DuckLabs 团队。我们决定扩展 S3，引入一个基于 Iceberg 构建的一流表格存储原语，因为我们一些最大的 Spark 客户正在采用 Iceberg，并希望存储能够真正适合表格数据。在观察到我们不同客户群对 DuckDB 的这种热情后，我想确保 S3 Tables 对 DuckDB 用户来说也是一个绝佳的选择。这就是我发现自己与 Hannes 和 Mark 在阿姆斯特丹一起喝啤酒的原因。

…and quacks like a duck ……而且叫起来像鸭子

We spent a bunch of time talking about systems in those early conversations and I think we found that across S3 and DuckDB, we shared a lot of the same views toward building software, enabling developers, and working with data. Hannes has described their goal as allowing “anyone to work with data confidently”, a framing I absolutely love, because of the emphasis Hannes puts on the “confidently” part when he says it. We quickly agreed that AWS would become their customer and sponsor their Iceberg extension, with a goal of broadening support for Iceberg outside of the Spark landscape and making it as simple as possible to build applications over S3 Tables. Since then, that extension has grown to be a very mature implementation of both the Iceberg v2 and v3 specifications, and has motivated the async I/O support that is about to arrive in the 2.0 release. In building async I/O, their team took the goal of being able to saturate the NIC while scanning tables from S3 – it’s a really nice bit of performance work. And the extension has proven to be pretty popular with the DuckDB community, receiving over 800K downloads a week.在那些早期的谈话中，我们花了很多时间讨论系统，我想我们发现，在 S3 和 DuckDB 之间，我们在构建软件、赋能开发者和处理数据方面有着许多相同的观点。Hannes 将他们的目标描述为让“任何人都能自信地处理数据”，我非常喜欢这个表述，因为 Hannes 在说这句话时强调了“自信地”这一部分。我们很快达成共识，AWS 将成为他们的客户并赞助他们的 Iceberg 扩展，目标是在 Spark 生态系统之外扩大对 Iceberg 的支持，并使在 S3 Tables 之上构建应用程序变得尽可能简单。从那时起，该扩展已经成长为对 Iceberg v2 和 v3 规范非常成熟的实现，并推动了即将在 2.0 版本中推出的异步 I/O 支持。在构建异步 I/O 时，他们的团队设定了在从 S3 扫描表时能够饱和网卡的目标——这是一项非常出色的性能工作。而且该扩展在 DuckDB 社区中非常受欢迎，每周下载量超过 80 万次。

Over the past two years (give or take), our teams have worked more and more closely together and have broadened the areas where we’ve collaborated. We explored AWS Lambda as a matched primitive for launching DuckDB queries quickly, and we smoothed the integrations between DuckDB and the AWS database and analytics engines through DuckDB’s evolving ATTACH and CONNECT commands.在过去的两年左右时间里，我们的团队合作得越来越紧密，并拓宽了协作领域。我们探索了 AWS Lambda 作为快速启动 DuckDB 查询的匹配原语，并通过 DuckDB 不断发展的 ATTACH 和 CONNECT 命令，平滑了 DuckDB 与 AWS 数据库和分析引擎之间的集成。

The DuckDB community has taught us that when the engine is just a library, the line between building an application and analyzing its data gets a lot blurrier. You reach for the same engine when you’re prototyping against a pile of CSV files, when you add a feature to your application that needs to aggregate something, when you build a second application against the same data, and when you eventually want to report on all of it. It’s the same SQL and the same engine at every step, which means you aren’t making a decision on day one about what your data is eventually going to become.DuckDB 社区教会了我们，当引擎只是一个库时，构建应用程序和分析其数据之间的界限变得模糊多了。当你针对一堆 CSV 文件进行原型设计时、当你为应用程序添加需要聚合某些内容的特性时、当你针对相同数据构建第二个应用程序时，以及当你最终想要报告所有这些内容时，你都会使用同一个引擎。每一步都是相同的 SQL 和相同的引擎，这意味着你不需要在第一天就决定你的数据最终会变成什么样。

None of which displaces the need for the distributed approaches we already have. When a job genuinely needs a thousand machines, it needs a thousand machines, and approaches that made that tractable aren’t going anywhere. What’s changed is that an enormous amount of the data work people actually do never needed a cluster in the first place, and now it doesn’t have one.所有这些都不会取代我们已经拥有的分布式方法的需求。当一项工作确实需要一千台机器时，它就需要一千台机器，而使之可行的那些方法也不会消失。改变的是，人们实际上所做的海量数据工作根本不需要集群，而现在它也不再需要集群了。

The library form factor and focus on portability also means that a single, efficient code base can live at many places in the stack and result in improvements “raising all ships.” Somewhere along the way I started describing DuckDB to people as “the glibc of structured data”: a lean, unglamorous, ubiquitous dependency that a great deal of software links against and almost nobody has to think about. DuckDB’s extensibility and its ability to be used across so many data formats, existing database interfaces, and in application form factors ranging from browsers to server-side engine code make it applicable to a huge range of problems. The shift I find most interesting isn’t really about where the engine runs — it’s that “analytics” is becoming less of a separate activity that happens to data somewhere else, and more of something you do continuously as you build.库的形式和对可移植性的关注也意味着，一个单一、高效的代码库可以存在于栈中的许多地方，并带来“水涨船高”的改进。在某个过程中，我开始向人们描述 DuckDB 为“结构化数据的 glibc”：一个精简、朴实、无处不在的依赖项，大量软件与之链接，而几乎没有人需要去考虑它。DuckDB 的可扩展性及其跨越如此多数据格式、现有数据库接口以及从浏览器到服务器端引擎代码的应用程序形式的能力，使其适用于极其广泛的问题。我发现最有趣的转变并不是关于引擎在哪里运行——而是“分析”正在变得不再是发生在其他地方的数据的独立活动，而更多地成为你在构建过程中持续进行的事情。

…it must be a duck ……那它一定是一只鸭子

Hannes and Mark’s vision is so aligned with the way that we think about data at Amazon that we wanted to give them the broadest possible opportunity to set the agenda not just for DuckDB, but for how analytics should be built and delivered in the cloud. So, after many conversations (not all in pubs, I swear), we jointly made the decision to have the DuckLabs team join AWS (and we start working together this week).Hannes 和 Mark 的愿景与我们在亚马逊思考数据的方式高度一致，因此我们希望给他们提供最广泛的机会，不仅为 DuckDB，而且为如何在云中构建和交付分析设定议程。因此，在经过多次谈话（我发誓，并非都在酒吧里）后，我们共同决定让 DuckLabs 团队加入 AWS（我们本周开始合作）。

From an AWS perspective, we want to continue to evolve DuckDB to be the most natural tool that developers, and increasingly agents, reach for when they work with structured data. We’re already using it internally for dashboards, CLI tooling, in-server accelerators, and bridges between systems, and many of our customers use it even more broadly than that. We want to support the project’s evolution in a way that preserves exactly what makes it great: the incredible community, the remarkable development velocity that the team has maintained over the past eight years, and the broad and growing set of application domains where DuckDB is effective.从 AWS 的角度来看，我们希望继续发展 DuckDB，使其成为开发者（以及越来越多的智能体）在处理结构化数据时最自然的工具。我们已经在内部将其用于仪表板、CLI 工具、服务器内加速器和系统间的桥梁，而我们的许多客户使用它的范围甚至更广。我们希望以一种保留其伟大之处的方式支持该项目的演进：令人难以置信的社区、团队在过去八年中保持的惊人开发速度，以及 DuckDB 有效的广泛且不断增长的应用程序领域。

DuckLabs is joining AWS as a subsidiary, and the DuckDB project will continue to operate as an open source project under the stewardship of the DuckDB Foundation, developed by the DuckLabs team, and be available under an MIT license. Hannes and Mark have talked about this evolution in their own words in a blog post on the DuckLabs blog. The DuckLabs team will continue to operate as they do today from their current office in Amsterdam, where Hannes’s architect brother has designed much of the office’s decor. As part of the broader AWS team, we’ll all work to accelerate DuckDB’s already incredible velocity.DuckLabs 将作为子公司加入 AWS，DuckDB 项目将继续在 DuckDB 基金会的管理下作为一个开源项目运行，由 DuckLabs 团队开发，并根据 MIT 许可证提供。Hannes 和 Mark 在 DuckLabs 博客上的一篇文章中用他们自己的话谈到了这一演进。DuckLabs 团队将继续像今天一样在他们位于阿姆斯特丹的现有办公室工作，Hannes 的建筑师兄弟为该办公室设计了大部分装饰。作为更广泛的 AWS 团队的一部分，我们将共同努力，加速 DuckDB 本已令人难以置信的发展速度。

The SIGMOD paper I mentioned has this enjoyably pragmatic tone: “none of DuckDB’s components is revolutionary in its own regard. Instead, we combined methods and algorithms from the state of the art that were best suited for our use cases.” This humble position of always being willing to learn and use what’s best, and of innovating where necessary really characterizes the project and its community, and I think it also resonates with the engineers that build our data and analytics services, and our customers who build on AWS. Systems evolve as the physics continue to shift, and the embedded engine isn’t the end of that story so much as it is a powerful approach that helps carry us forward into new territory. I’m looking forward to seeing where this team takes us.我提到的 SIGMOD 论文有着这种令人愉快的务实基调：“DuckDB 的组件在各自方面都没有革命性。相反，我们将最适合我们用例的现有技术和算法结合在一起。”这种始终愿意学习和使用最好的东西，并在必要时进行创新的谦逊立场，真正体现了该项目及其社区的特点，我认为它也与构建我们数据和分析服务的工程师，以及在 AWS 上进行构建的客户产生了共鸣。随着物理法则的不断转移，系统也在不断演进，嵌入式引擎并不是这个故事的终点，而是一种强大的方法，帮助我们走向新的领域。我很期待看到这个团队将带我们去向何方。