SQLite vs DuckDB on the same $16 box: every cliff moved 100x在同一台 16 美元的服务器上对比 SQLite 与 DuckDB：所有性能瓶颈均提升了 100 倍

TL;DR简而言之

I spent six days working on and running a benchmark of observability data on DuckDB. I've done this with SQLite in the last blog post and I really wanted to see how DuckDB compares on a cheap CCX13 Hetzner instance. The way I've done the measurements is by implementing DuckDB as a valid storage engine for Traceway and then running its benchmarking suite. The benchmarks show that DuckDB's columnar engine is able to query 100x more data points without any major backend changes. The write throughput is also 3x to 15x higher. The result is that you can now self-host the full OTel stack that can handle a large data volume on a pretty small server. Keep reading to find out how I've done the measurements!我花了六天时间在 DuckDB 上运行可观测性数据的基准测试。在上一篇博文中，我曾对 SQLite 做过同样的测试，我非常想看看 DuckDB 在 Hetzner 的廉价 CCX13 实例上表现如何。我的测量方法是将 DuckDB 作为 Traceway 的有效存储引擎，然后运行其基准测试套件。测试结果表明，DuckDB 的列式引擎无需对后端进行重大修改，即可查询多出 100 倍的数据点。写入吞吐量也提高了 3 到 15 倍。这意味着你现在可以在一台相当小的服务器上自托管完整的 OTel 栈，并处理大量数据。请继续阅读，了解我是如何进行这些测量的！

This is what came back:测试结果如下：

	SQLite (post 2)	DuckDB (this post)	Change
Metrics writes	61,712 pts/sec	254,242 pts/sec	4x
Spans writes	30,508 spans/sec	95,737 spans/sec	3x
Logs writes	4,877 rec/sec	75,225 rec/sec	15x
Metrics read cliff	1M rows (3.85 s median)	100M rows (3.0 s median)	100x
Spans read cliff	100k rows (2.27 s median)	10M rows (902 ms median)	100x
Logs read cliff	100k rows (114 ms median)	10M rows (85 ms median)	100x

A read cliff, throughout this series, is the largest table size at which real dashboard pages still load: the median of three endpoint probes at or under 5 seconds, none timing out. It earns the name because of what sits one 10x step past it: queries don't slow down, they stop coming back.在本系列中，“读取瓶颈”（read cliff）指的是仪表盘页面仍能正常加载的最大表大小：即三次端点探测的中位数在 5 秒或以内，且没有超时。之所以称之为“瓶颈”，是因为一旦超过这个阈值（通常是 10 倍量级），查询就不会变慢，而是直接不再返回结果。

Each signal's read cliff sits exactly 100x further out on DuckDB than on SQLite, at equal or better latency, on identical hardware. I checked that symmetry against the raw JSON twice before I believed it. The box also ingested a billion metric points in under an hour into 10.8 GB of disk and stayed up, a scale the SQLite benchmark never got within 100x of. Logs, the signal post 2 said to keep off SQLite entirely, are DuckDB's biggest win.在相同硬件上，每个信号在 DuckDB 上的读取瓶颈恰好比 SQLite 远 100 倍，且延迟相同或更优。在相信这个结果之前，我对着原始 JSON 数据核对了两次。这台机器还在不到一小时内摄入了十亿个指标点，占用 10.8 GB 磁盘空间且运行稳定，这是 SQLite 基准测试从未达到过的规模（差距超过 100 倍）。日志处理是 DuckDB 的最大优势，而这正是第 2 篇博文中建议完全避开 SQLite 的场景。

What's actually being compared实际对比的内容

Post 2 ended promising a ClickHouse comparison, and that post is still coming. But every time I sat down to build it, the same question got in front of me: before reaching for a client-server OLAP database, with its own container, its own memory appetite, its own failure modes, how far does an embedded one go? Traceway ships a DuckDB telemetry backend as an opt-in build, same single binary, same deployment story as the SQLite build, different storage engine under the telemetry tables. If you self-host on one cheap box, these two builds are the actual decision in front of you, and I couldn't find anyone who had published numbers for it. So I ran post 2's entire methodology against the DuckDB build and put the results side by side.第 2 篇博文结尾曾承诺进行 ClickHouse 对比，该文章后续会发布。但每当我着手构建它时，都会面临同一个问题：在引入一个需要独立容器、占用更多内存、具有特定故障模式的客户端-服务器 OLAP 数据库之前，嵌入式数据库究竟能走多远？Traceway 提供了一个可选的 DuckDB 遥测后端构建版本，它与 SQLite 构建版本使用相同的二进制文件，部署方式也相同，只是遥测表下的存储引擎不同。如果你在一台廉价服务器上进行自托管，这两个版本就是你实际面临的选择，而我找不到任何相关的公开数据。因此，我针对 DuckDB 版本运行了第 2 篇博文中的整套方法论，并将结果进行了并列对比。

Here's the plan. Methodology first, including what changed in the harness since post 2 and one fix in the backend itself that these numbers depend on. Then writes, all three signals against their SQLite baselines. Then reads, same shape. Then the billion-row run, what queries cost past the cliffs, the fleet math for both builds, and which build I would actually run.计划如下：首先是方法论，包括自第 2 篇博文以来测试工具的变化，以及后端中一个至关重要的修复。接着是写入测试，对比三种信号在 SQLite 基准下的表现。然后是读取测试，采用相同的形式。最后是十亿行数据的运行测试，分析瓶颈之后的查询成本、两种构建版本的集群规模计算，以及我最终会选择运行哪个版本。

How I measured测量方法

The setup is post 2's, so I'll keep it short: a system under test (SUT) running the Traceway binary and a separate load generator on a private link, both in Hetzner's Nuremberg datacenter, OTLP, the OpenTelemetry wire protocol, over gzipped protobuf, uniform-random data in bounded ranges. Traceway built from commit 14b4aa6e, Ubuntu 24.04, retention off during the bench.测试环境沿用第 2 篇博文的设置，简述如下：被测系统（SUT）运行 Traceway 二进制文件，并在私有链路连接的另一台机器上运行负载生成器，两者均位于 Hetzner 的纽伦堡数据中心。使用 OTLP（OpenTelemetry 有线协议），通过压缩的 protobuf 传输，数据为有界范围内的均匀随机数据。Traceway 基于 commit 14b4aa6e 构建，操作系统为 Ubuntu 24.04，测试期间关闭了数据留存策略。

Two scenarios per signal. The throughput ramp: batch-size ramp at fixed rate, then a request-rate ramp at the winning batch (this time with a finer rate ladder, 1 through 25 req/sec, because the interesting cliffs turned out to sit between post 2's coarser steps), then a small-batch high-rate ramp shaped like an SDK fleet. A step passes at under 5% errors and at least 70% of target rate achieved. The read-probe: fill the table to 1M, 10M, 100M, 1B, then 5B rows, and at each level load three real dashboard endpoints per signal, with a level passing when the median is at or under 5 s and nothing hits the 6 s timeout. The 5-second bar is generous on purpose and post 2's discussion of that still applies unchanged.每个信号进行两种场景测试。吞吐量阶梯测试：在固定速率下增加批处理大小，然后在最优批处理大小下增加请求速率（这次使用了更细的速率梯度，从每秒 1 到 25 个请求，因为有趣的瓶颈出现在第 2 篇博文较粗的步长之间），最后进行模拟 SDK 集群的小批量高频测试。如果错误率低于 5% 且达到目标速率的 70% 以上，则视为通过该步骤。读取探测：将表填充至 100 万、1000 万、1 亿、10 亿，然后是 50 亿行，在每个级别加载每个信号的三个真实仪表盘端点。当三个端点的中位数延迟在 5 秒或以内，且没有触发 6 秒超时限制时，视为通过该级别。5 秒的门槛是刻意设定的，第 2 篇博文中对此的讨论依然适用。

Three things differ from post 2's harness, all disclosed:与第 2 篇博文的测试工具相比，有三处不同，均在此披露：

1. The loadgen grew. Post 2 generated load from a second CCX13, which SQLite's ceilings never stressed. On my first DuckDB spans run, that loadgen box died mid-ramp while the database sat at 0% errors, the traffic generator broke before the thing it was breaking. The loadgen is now a CCX23. The SUT, the box every number describes, is unchanged.负载生成器升级了。第 2 篇博文使用第二台 CCX13 作为负载生成器，但 SQLite 的上限从未对其造成压力。在我第一次运行 DuckDB 的 spans（跨度）测试时，负载生成器在测试中途崩溃了，而数据库的错误率仍为 0%。流量生成器在数据库崩溃前先挂了。现在负载生成器已更换为 CCX23。被测系统（SUT）保持不变。
2. The digestion gate. DuckDB checkpoints its write-ahead log after ingest stops; probe during that and you measure a busy engine, not a query. The read-probe now polls the backend's deep-health endpoint until the database and WAL file sizes are stable, records the wait per level (5 seconds at every level in the final runs), then probes. SQLite reports no engine gauges and skips the wait, so its post 2 numbers were produced identically.消化门控（Digestion gate）。DuckDB 会在摄入停止后检查预写日志（WAL）；如果在此时进行探测，测量的是繁忙的引擎而非查询性能。现在的读取探测会轮询后端的深度健康检查端点，直到数据库和 WAL 文件大小稳定，记录每个级别的等待时间（最终运行中每个级别等待 5 秒），然后再进行探测。SQLite 不提供引擎指标且跳过了等待，因此其第 2 篇博文的数据是以相同方式产生的。
3. Only accepted rows count. The fill counts items confirmed by 2xx responses, not items attempted, which matters once the backend learns to refuse work. Fills overshoot their targets, at the smallest level by as much as 2.6x, so I quote actual row counts wherever the difference matters.仅计算已接受的行数。填充计数基于 2xx 响应确认的项目，而非尝试发送的项目。一旦后端开始拒绝处理，这一点至关重要。填充量会超过目标值，在最小级别甚至达到 2.6 倍，因此我在差异显著的地方引用实际的行数。

Caveats. These are single-shot runs, one throughput run and one read ladder per signal. Post 2 promised medians of three reps starting with this post, and I'm walking that back one more time: the fix-and-rerun cycle below consumed the budget, and I'd rather publish single-shot numbers with that admission than sit on them. The key cells reproduced across the debugging runs (100M metrics reads landed between 2.7 s and 3.2 s across four runs on different days), but the protocol is single-shot. "Passing" still means three named endpoints on freshly written data on an otherwise idle box; read-under-ingest remains future work.注意事项。这些是一次性运行测试，每个信号仅进行一次吞吐量测试和一次读取阶梯测试。第 2 篇博文曾承诺从本文开始使用三次重复测试的中位数，但我再次撤回了这一承诺：修复和重测循环消耗了预算，我宁愿在承认这一点的前提下发布单次运行数据，也不愿压着数据不发。关键单元格在调试运行中可以复现（在不同日期的四次运行中，1 亿指标读取时间在 2.7 秒到 3.2 秒之间），但协议确实是单次运行。“通过”仍然意味着在空闲机器上对新写入的数据进行三个指定端点的测试；在摄入过程中进行读取测试留待以后完成。

One more disclosure, this one in the backend rather than the harness. My first DuckDB runs produced numbers I could not trust: the box kept dying mid-benchmark, and the hunt led to a real bug in my own ingest path, which had no admission control and let a sustained burst of fat batches run the process out of memory. I fixed it with an ingest gate that caps concurrent processing and answers overload with a 503 plus Retry-After (the commit linked above), and reran everything. All numbers in this post are from the fixed backend; the pre-fix results still visible in the repo's history predate it and understate DuckDB, spans by 23% and logs by almost half, because the crashes were cutting the ramps short. SQLite never surfaced the bug, its slower insert path errors early under pressure, which is post 2's 5 req/sec logs wall doing its job.再披露一点，这是关于后端而非测试工具的。我最初的 DuckDB 运行结果不可信：机器在基准测试中途不断崩溃。排查发现是我自己的摄入路径中存在一个真正的 Bug，它没有准入控制，导致持续的突发大批量数据耗尽了进程内存。我通过一个摄入门控修复了这个问题，该门控限制了并发处理，并以 503 加 Retry-After 响应过载（链接见上方的 commit），然后重新运行了一切。本文中的所有数据均来自修复后的后端；仓库历史中可见的修复前结果均已过时，且严重低估了 DuckDB 的性能（spans 低估了 23%，日志低估了近一半），因为崩溃缩短了测试过程。SQLite 从未出现过这个 Bug，其较慢的插入路径在压力下会提前报错，这正是第 2 篇博文中每秒 5 个请求的日志墙发挥的作用。

Writes: 4x, 3x, 15x写入：4 倍、3 倍、15 倍

Signal	SQLite	DuckDB	Winning shape (DuckDB)	Lowest failing step
Metrics	61,712/sec	254,242/sec	batch 16384 × 20 req/sec, p50 3.2 s	22.5 req/sec, 12.2% errors
Spans	30,508/sec	95,737/sec	batch 16384 × 7.5 req/sec, p50 1.2 s	8.75 req/sec, 9.7% errors
Logs	4,877/sec	75,225/sec	batch 16384 × 5 req/sec, p50 3.0 s	5.625 req/sec, 11.4% errors

Going in, my quiet bar for this table was "double SQLite and I'll call it a win," and the one cell I braced for was logs. Post 2's starkest result was logs writing at 4,877/sec into a wall at 5 req/sec so sharp that 1.5x more requests took the error rate from 0% to 98%. That wall is the cell that moved most: the DuckDB build writes logs at fifteen times SQLite's rate, and the wall became a slope of polite refusals. The signal with the lowest SQLite ceiling and the highest real-world volume is the one the columnar engine helps most, which is either obvious in hindsight or backwards from everything post 2 trained me to expect, and I held the second view until this table existed.测试开始前，我心里的及格线是“达到 SQLite 的两倍就算赢”，而我最担心的是日志。第 2 篇博文最显著的结果是日志写入在每秒 5 个请求时撞墙，且 1.5 倍的额外请求就让错误率从 0% 飙升至 98%。这堵墙是变动最大的单元格：DuckDB 构建版本写入日志的速度是 SQLite 的 15 倍，且那堵“墙”变成了一个温和的拒绝坡度。列式引擎对 SQLite 上限最低、实际数据量最大的信号帮助最大，这在事后看来显而易见，但在测试前我完全持相反观点。

The "lowest failing step" column matters as much as the headline column, and lowest is the right word: the ladder bisects between the last passing and first failing rate, so those are the smallest rates that fail, not the first failures in wall-clock order. Every failure in the column is the backend refusing work rather than falling over: the run logs show 503s from the admission gate, the archived JSON shows zero insert failures and zero dropped rows behind them, and the container finished all three runs with zero restarts. That is the same graceful-cliff behavior SQLite showed in post 2, and DuckDB only exhibits it because of the fix I disclosed above; before it, this table could not be produced at all.“最低失败步骤”列与标题列同样重要：“最低”是指该速率是测试中首次出现失败的步骤。表中的每一次失败都是后端在拒绝工作而非崩溃：运行日志显示来自准入网关的 503 错误，存档的 JSON 显示插入失败为零，丢失行数为零，容器在三次运行中均未重启。这与 SQLite 在第 2 篇博文中表现出的优雅瓶颈行为相同，而 DuckDB 之所以能做到，全靠我上述的修复；在此之前，这个表根本无法生成。

The bottom of each ramp holds one more first for the series: the SDK-fleet shape, hundreds of small batches per second rather than a few fat ones, which post 2 could only measure on SQLite. DuckDB sustains 53,091 metric points/sec, 36,577 spans/sec, and 30,442 log records/sec at batch 100. My early attempts never produced these numbers, the pre-fix backend died before that phase of the ramp could run, so seeing them appear at all was the first sign the reruns were measuring something the crashes had been eating.每个测试坡度的底部还有本系列首次出现的内容：SDK 集群形态，即每秒数百个小批次而非少量大批次，这是第 2 篇博文只能在 SQLite 上测量的。DuckDB 在 100 批次下维持了每秒 53,091 个指标点、36,577 个 spans 和 30,442 条日志记录。我早期的尝试从未得到这些数字，修复前的后端在测试阶段结束前就崩溃了，所以看到这些数字出现，是我意识到重测终于测量到之前被崩溃掩盖的真实性能的第一个信号。

Reads: every cliff, 100x later读取：所有瓶颈均推迟 100 倍

Writes were the appetizer; reads are why anyone reaches for a columnar engine, and they're where I had the most to lose. Post 2's most-quoted line was about read cliffs, the last row count where the dashboard still loads, and if DuckDB only matched SQLite here, the whole build would be a curiosity. Before running it I told myself I'd be satisfied if each signal's cliff moved 10x.写入只是开胃菜；读取才是人们选择列式引擎的原因，也是我损失最大的地方。第 2 篇博文中最常被引用的是关于“读取瓶颈”的论述，即仪表盘仍能加载的最大行数。如果 DuckDB 在这里只能与 SQLite 持平，那么整个构建版本就只是个新奇玩意儿。测试前我告诉自己，如果每个信号的瓶颈能提升 10 倍，我就满足了。

Fill level	Metrics	Spans	Logs
1M	146 ms	139 ms	27 ms
10M	382 ms	902 ms	85 ms
100M	3.0 s, passes	over 60 s, fails	fails unevenly: body search 3.5 s, trace-id 2 ms, severity filter over 60 s
1B	fill succeeds, queries over 60 s	not reached	not reached

Each cell is the median of three real endpoint probes after fill and digestion; bold marks each signal's last passing level. Two footnotes on the table itself. The levels are fill targets and the fills run over: the 1M level actually holds 1.7M rows for spans and logs and 2.6M for metrics, and every level above lands within 16% of target, so the exact row counts are in the JSON. And the over-60-s durations come from a second ladder run with the probe timeout raised to 60 seconds, which gets its own section below; the standard probe gives up at 6 s. I got 100x per signal, not 10x: SQLite's last passing levels were 1M, 100k, and 100k, and all three moved by the same factor of one hundred. Same endpoints, same thresholds.每个单元格都是填充和消化后三次真实端点探测的中位数；粗体标记了每个信号的最后一个通过级别。关于表格的两点说明：级别是填充目标，实际填充量会超出：100 万级别实际上包含 170 万行 spans 和日志，以及 260 万行指标，所有级别均在目标值的 16% 以内，确切行数在 JSON 中。超过 60 秒的持续时间来自第二次阶梯测试，该测试将探测超时提高到了 60 秒，详情见下文；标准探测在 6 秒时放弃。我最终每个信号都提升了 100 倍，而不是 10 倍：SQLite 的最后一个通过级别分别是 100 万、10 万和 10 万，而这三个级别都提升了相同的 100 倍。端点相同，阈值相同。

Metrics reads scale gently: 146 ms at 1M, 382 ms at 10M, 3.0 s at 100M. A hundred times the rows costs twenty times the latency, the scan gets cheaper per row as the table grows. The level SQLite barely passed is a hundredth of the level DuckDB passes with the same margin.指标读取扩展平稳：100 万行时 146 毫秒，1000 万行时 382 毫秒，1 亿行时 3.0 秒。行数增加 100 倍，延迟仅增加 20 倍，随着表变大，每行的扫描成本反而降低了。SQLite 勉强通过的级别，只是 DuckDB 以同样余量通过级别的百分之一。

Spans pay for their percentiles, same as they did on SQLite, just later. This is the page doing the paying, shown on my local instance with the loadgen's spans:Spans 的延迟与 SQLite 一样，依然取决于百分位数，只是发生得更晚。这是产生延迟的页面，显示的是我在本地实例上使用负载生成器的 spans：

Every row in that table is a P50/P95/P99 aggregation over the spans behind it, and the stacked latency chart above it is a second aggregation. Those two queries cost 902 ms at 10M rows and blow past a full minute at 100M. Post 2 watched the same two pages die between 100k and 1M.该表中的每一行都是其背后 spans 的 P50/P95/P99 聚合，其上方的堆叠延迟图是第二次聚合。这两个查询在 1000 万行时耗时 902 毫秒，在 1 亿行时超过一分钟。第 2 篇博文观察到同样的两个页面在 10 万到 100 万行之间崩溃。

Logs are the numbers I trusted least and checked hardest, an early bad run had me convinced for a full day that DuckDB couldn't read logs at all, and the truth turned out to be post 2's most interesting pattern repeated at ten times the rows. Here's the page in question, on my local instance with the loadgen's log records:日志是我最不信任、核对最严的数据。早期的糟糕运行曾让我整整一天确信 DuckDB 根本无法读取日志，但事实证明，这只是第 2 篇博文中观察到的最有趣模式在 10 倍行数下的重复。这是相关页面，显示的是我在本地实例上使用负载生成器的日志记录：

At their passing levels logs are the fastest dashboard of the three (85 ms median at 10M), and their failure at 100M is uneven in the same way SQLite's was at 1M: the trace-ID lookup still answers in 2 ms, the body search in 3.5 s, and only the severity filter, which matches and ranks a large slice of 100M rows, dies. The pages an on-call actually opens mid-incident, find this trace, search this error, remain usable at 100M rows; the full-table scan does not. Retention policy should still follow that split. The split just moved 100x.在通过的级别上，日志是三个仪表盘中最快的一个（1000 万行时中位数 85 毫秒），它们在 1 亿行时的失败与 SQLite 在 100 万行时的情况如出一辙：trace-ID 查询在 2 毫秒内响应，正文搜索在 3.5 秒内完成，只有匹配并排序大量 1 亿行数据的严重性过滤器崩溃了。值班人员在中途处理事故时真正打开的页面（查找此 trace，搜索此错误）在 1 亿行时依然可用；全表扫描则不然。留存策略应遵循这种划分。只是这个划分点现在推迟了 100 倍。

A billion rows on an 80 GB disk80 GB 磁盘上的十亿行数据

I put the 1B level on the ladder expecting to write a paragraph about how it died, because every earlier attempt at it had ended the same way: container gone, level unreadable. This time the final fill step alone ran for 52 minutes, adding its last 900M points at a sustained 287k points/sec with the gate shedding the excess, and brought the table to 1,001,472,000 rows in 10.8 GB on disk, about 10.8 bytes per point after columnar compression, under an hour of total ingest across the ladder. The box stayed up. Health checks answered the whole way through, and post-fill digestion took 5 seconds. Nothing in posts 1 or 2 gets within 100x of this level; on the fixed backend a billion rows is just a big table.我将 10 亿行级别放入测试阶梯，原本期望写一段关于它是如何崩溃的段落，因为之前的所有尝试都以容器消失、级别不可读告终。这一次，仅最终填充步骤就运行了 52 分钟，以每秒 287k 点的持续速度添加了最后 9 亿个点（网关丢弃了多余数据），最终在磁盘上以 10.8 GB 的空间存储了 1,001,472,000 行数据，列式压缩后平均每点约 10.8 字节，整个阶梯测试的总摄入时间不到一小时。机器保持稳定。健康检查全程通过，填充后的消化过程耗时 5 秒。第 1 或第 2 篇博文中的任何内容都未达到此级别的 100 分之一；在修复后的后端上，十亿行只是一个大表而已。

A big table you cannot look at: all three dashboard queries ran past 60 seconds at 1B, so the level fails, and 5B was never attempted since the ladder stops at the first failure. I left it that way on purpose. A 5B fill would prove the disk holds 54 GB of points, and I already know nothing could read them. The practical summary of the ladder's top: this box can store a billion metric points, and it can show you a hundred million of them.一个你无法查看的大表：所有三个仪表盘查询在 10 亿行时都超过了 60 秒，因此该级别失败，而 50 亿行测试从未尝试，因为阶梯测试在第一次失败时就停止了。我刻意保持了这种状态。50 亿行的填充将证明磁盘可以容纳 54 GB 的数据点，但我已经知道没有任何东西能读取它们。阶梯测试顶部的实际总结是：这台机器可以存储十亿个指标点，并且可以向你展示其中的一亿个。

How slow is slow past the cliff瓶颈之后有多慢

Post 2 could only say "it timed out" about anything past a cliff. This time I reran the entire ladder with a 60-second probe threshold to put real durations on the failing cells, and there were no durations to put: every failing query was still running when the new cap cut it off. Metrics at 1B, still running at 61 s. Spans aggregations at 100M, still running at 61 s. The logs severity filter at 100M, still running at 61 s while the body search finished in 3.2 s beside it.第 2 篇博文对于瓶颈之后的任何情况只能说“超时了”。这次我用 60 秒的探测阈值重新运行了整个阶梯测试，以给失败的单元格标注真实的持续时间，结果发现根本没有持续时间可言：当新的上限切断查询时，每个失败的查询仍在运行。10 亿行的指标查询在 61 秒时仍在运行。1 亿行的 Spans 聚合在 61 秒时仍在运行。1 亿行的日志严重性过滤器在 61 秒时仍在运行，而旁边的正文搜索在 3.2 秒内就完成了。

So the cliffs are not slopes on either database. Queries go from a second or three to more than a minute across a single 10x step in rows, the signature of an aggregation outgrowing the 4 GB memory budget and spilling. Post 2 found the same absence on SQLite at one hundredth the scale, and it is why I keep publishing cliffs instead of curves: there is no slow-but-usable band above a cliff to plan around.所以，在两种数据库上，瓶颈都不是平缓的斜坡。查询时间从一三秒直接跳跃到超过一分钟，这正是聚合操作超出 4 GB 内存预算并发生溢出的标志。第 2 篇博文在 SQLite 上以百分之一的规模发现了同样的现象，这就是我坚持发布“瓶颈”而不是“曲线”的原因：瓶颈之上不存在可以规划的“缓慢但可用”的区间。

The 60-second run also caught a smaller flaw for the fix list: a cancelled dashboard query does not promptly release its read-pool connection, so after the two spans aggregations timed out, a 3 ms exceptions query queued behind them for 52 seconds. It affected no passing number, and it goes on the same list the admission gate came from.60 秒的运行测试还捕获了修复列表中的一个小缺陷：取消的仪表盘查询不会立即释放其读取池连接，因此在两个 Spans 聚合超时后，一个 3 毫秒的异常查询在队列中排队等待了 52 秒。它没有影响任何通过的数据，我已将其加入准入网关所在的同一修复列表中。

What fits comfortably, on each build每个构建版本能舒适处理的规模

Cliffs are ceilings; what matters day to day is how far under them a real fleet sits. The small-fleet shape from post 2, ten backends emitting 50 spans/sec, 200 log records/sec, and 10 metric points/sec each, held against both builds:瓶颈是上限；日常工作中重要的是真实集群在瓶颈之下运行多远。第 2 篇博文中的小型集群模型——10 个后端，每个每秒发出 50 个 spans、200 条日志记录和 10 个指标点——在两个构建版本上的表现如下：

	SQLite envelope used	DuckDB envelope used
Spans (500/sec)	1.7%	0.5%
Logs (2,000/sec)	40%	2.7%
Metrics (100/sec)	0.2%	0.04%

Writes were already a non-issue on SQLite for everything except logs, and on DuckDB they stop being a conversation entirely. Retention is still the real knob, with new units. At this fleet's rates, logs reach their last passing read level (10M rows) in about 85 minutes and the failing level in about 14 hours; spans reach 10M in 5.5 hours; metrics reach 100M in 11 days. On SQLite, post 2 measured the logs window at 50 seconds. An incident-sized working set, hours of spans and logs, days of metrics, fits on the right side of every DuckDB cliff with an ordinary retention job, which is the thing the SQLite build could not offer.写入在 SQLite 上除了日志外本来就不是问题，而在 DuckDB 上，写入甚至都不再是讨论的话题。留存策略仍然是真正的调节旋钮，只是单位变了。按照此集群的速率，日志在约 85 分钟内达到其最后一个通过读取级别（1000 万行），在约 14 小时内达到失败级别；Spans 在 5.5 小时内达到 1000 万行；指标在 11 天内达到 1 亿行。在 SQLite 上，第 2 篇博文测得的日志窗口仅为 50 秒。一个事故规模的工作集，数小时的 spans 和日志，数天的指标，在普通留存作业下完全处于 DuckDB 每个瓶颈的右侧，这是 SQLite 构建版本无法提供的。

What I deliberately didn't measure我刻意没有测量的内容

• Read under concurrent write. Fill, digest, settle, probe. Stricter isolation than post 2, and still not what a dashboard experiences at 3 PM. Top of the future-work list.并发写入下的读取。填充、消化、沉降、探测。比第 2 篇博文更严格的隔离，但仍然不是下午 3 点仪表盘所经历的真实情况。这是未来工作列表的首位。
• Mixed-signal load. Each signal had the box to itself; a real deployment writes all three at once and these ceilings do not simply add.混合信号负载。每个信号独占机器；真实部署会同时写入所有三种信号，这些上限不能简单相加。
• Result correctness. I timed responses, I did not diff their contents.结果正确性。我测量了响应时间，没有对比它们的内容差异。
• Variance. Single-shot, as admitted in the methodology, with the median-of-three protocol still owed.方差。如方法论中所述，这是单次运行，仍欠缺三次中值协议。
• Tuning. Stock config, meaning what the benchmark compose file ships by default: 4 GB DuckDB memory cap, 256 MB checkpoint threshold, no schema or query changes. The 1B read failure smells like it wants a memory-budget experiment, and that is a deliberate cliffhanger.调优。默认配置，即 benchmark compose 文件默认提供的：4 GB DuckDB 内存上限，256 MB 检查点阈值，无 schema 或查询更改。10 亿行的读取失败看起来像是需要进行内存预算实验，这是一个刻意留下的悬念。
• ClickHouse. The comparison post 2 promised is still owed, and it gets its own post on these same ladders rather than a section squeezed in here.ClickHouse。第 2 篇博文承诺的对比仍然欠着，它将会有自己独立的文章，使用同样的阶梯测试，而不是在这里强行塞入一个章节。
• Durability over weeks. An hour-long billion-row fill says nothing about month three on the same disk.数周的持久性。长达一小时的十亿行填充无法说明在同一磁盘上运行三个月后的情况。

What surprised me令我惊讶的事

I expected the reads to win, that's what columnar storage is for, and I set my bar at 10x per signal. All three delivered 100x. But the thing that will stay with me from this week is that the database was never the hard part: every wrong-looking number in six days of benchmarking traced back to my own ingest path, not the engine. DuckDB, whenever my code got out of its way, was boring in the best way a database can be boring.我预期读取性能会获胜，毕竟那是列式存储的用途，我设定的目标是每个信号提升 10 倍。结果三个信号都提升了 100 倍。但本周最让我难忘的是，数据库本身从来不是难题：六天基准测试中每一个看起来异常的数据，最终都追溯到我自己的摄入路径，而不是引擎本身。DuckDB，只要我的代码不挡路，它就表现得非常“无聊”——这是数据库所能达到的最好状态。

The other surprise was logs, again. Post 2's verdict was "keep logs off SQLite," and logs turned out to be the strongest single argument for the DuckDB build: 15x on writes, 100x on the read cliff, with the incident-workflow pages still answering at 100M rows.另一个惊喜再次是日志。第 2 篇博文的结论是“不要在 SQLite 上存日志”，而日志最终成了支持 DuckDB 构建版本的最大理由：写入提升 15 倍，读取瓶颈提升 100 倍，且事故处理页面在 1 亿行数据时依然能响应。

The verdict结论

Post 1 claimed a $16 box could run your observability stack. Post 2 tested the claim and returned it with an exception: not logs, and mind the read cliffs. This post retires the exception. On the same box, the DuckDB build writes every signal faster than a small fleet emits, serves dashboards at row counts the SQLite build cannot approach, and turns logs from the signal post 2 told you to keep off the box into the best result in the dataset. A billion metric points cost 10.8 GB of disk. I came into the week hoping to double post 2's numbers; the smallest improvement in the table is 3x.第 1 篇博文声称 16 美元的机器可以运行你的可观测性栈。第 2 篇博文测试了这一说法并给出了例外：日志不行，且要注意读取瓶颈。本文撤销了这一例外。在同一台机器上，DuckDB 构建版本写入每个信号的速度都快于小型集群的发出速度，能够处理 SQLite 构建版本无法触及的行数，并将第 2 篇博文建议避开的日志变成了数据集中的最佳结果。十亿个指标点仅占用 10.8 GB 磁盘空间。我本周开始时希望将第 2 篇博文的数据翻倍；结果表格中最小的提升也是 3 倍。

So if you self-host Traceway on a single machine, run the DuckDB build (docker-compose.duckdb.yml in the repo brings it up). One config note before you do: that compose file ships a conservative 2 GB DuckDB memory cap and leaves the checkpoint threshold at DuckDB's 16 MB default, while the benchmark ran on 4 GB and 256 MB, so set DUCKDB_MEMORY_LIMIT and DUCKDB_CHECKPOINT_THRESHOLD to match if you want these numbers. I would only reach for the SQLite build in two situations, and both are real. If you need a binary that compiles anywhere Go does, SQLite wins: the DuckDB build needs CGO, Go's C bridge, and glibc, so its container image is Debian rather than Alpine. And if your volume lives comfortably under post 2's numbers, SQLite's storage engine doing all its work inline, no deferred checkpoints, no digestion window to wait out, remains the simplest thing that works. The moment logs matter, or retention past an hour matters, the comparison stops being close.因此，如果你在单台机器上自托管 Traceway，请运行 DuckDB 版本（仓库中的 docker-compose.duckdb.yml 可直接启动）。在操作前有一个配置说明：该 compose 文件默认设置了保守的 2 GB DuckDB 内存上限，并将检查点阈值保持在 DuckDB 默认的 16 MB，而我的基准测试是在 4 GB 和 256 MB 下运行的，如果你想获得这些数据，请相应设置 DUCKDB_MEMORY_LIMIT 和 DUCKDB_CHECKPOINT_THRESHOLD。我只会在两种情况下选择 SQLite 版本，且都是现实情况：如果你需要一个能在任何 Go 支持的平台上编译的二进制文件，SQLite 胜出：DuckDB 构建版本需要 CGO、Go 的 C 桥接和 glibc，因此其容器镜像是 Debian 而非 Alpine。此外，如果你的数据量远低于第 2 篇博文的数字，SQLite 的存储引擎内联工作、无延迟检查点、无需等待消化窗口，依然是最简单有效的选择。但一旦日志变得重要，或者需要超过一小时的留存时间，这种对比就不再接近了。

One asterisk belongs on the whole table: these numbers exist because the benchmark first found a crash bug in my ingest path, and I fixed it before rerunning everything. If you run the DuckDB build, run a version with the admission gate. The engine was never the problem. The code in front of it was, and finding that out is the most useful thing this comparison produced.整个表格有一个星号：这些数据之所以存在，是因为基准测试首先发现了我摄入路径中的一个崩溃 Bug，我在重新运行一切之前修复了它。如果你运行 DuckDB 版本，请确保运行带有准入网关的版本。引擎从来不是问题所在。它前面的代码才是，找出这一点是本次对比产生的最有价值的成果。

Raw data + workflow原始数据 + 工作流

Everything in this post, runnable from the repo:本文中的一切均可从仓库中运行：

• Raw JSON lives in benchmarks/blog/post-3-data/: throughput, read-probe, and 60-second diagnostic JSONs, one per signal.原始 JSON 位于 benchmarks/blog/post-3-data/：包含吞吐量、读取探测和 60 秒诊断 JSON，每个信号一个。
• The workflow: benchmark-hardware.yml. Throughput: 29828430297. Read-probe: 29838394873. 60-second diagnostic: 29848976962. The pre-fix runs the methodology disclosure mentions: 29734966806 and 29815312404.工作流：benchmark-hardware.yml。吞吐量：29828430297。读取探测：29838394873。60 秒诊断：29848976962。方法论披露中提到的修复前运行：29734966806 和 29815312404。
• The fix: commit 14b4aa6e, the ingest admission gate.修复：commit 14b4aa6e，摄入准入网关。
• Try Traceway: self-host in five minutes.尝试 Traceway：五分钟完成自托管。

Questions, pushback, or "your number is wrong because X": jstojiljkovic941@gmail.com, or find me on GitHub.如有问题、反对意见或“你的数据因为 X 而错误”：jstojiljkovic941@gmail.com，或在 GitHub 上找我。

Next post: the one post 2 promised. ClickHouse on the same box, same loadgen, same ladders, to find out what the full client-server stack buys you over the embedded engines and what it costs to get it.下一篇博文：第 2 篇博文承诺的那篇。在同一台机器、同样的负载生成器、同样的阶梯测试下对比 ClickHouse，看看完整的客户端-服务器栈能比嵌入式引擎多带来什么，以及为此付出的代价。