A DuckDB Database with No Data in It一个不包含任何数据的 DuckDB 数据库

The DuckDB team
2026-10-07 | 6 min

TL;DR: A DuckDB database file does not have to hold any data. It can store nothing but view definitions that point at Parquet files on object storage, so a few hundred kilobytes behave like a shared catalog over a whole data lake. This post shows how to build one, host it, and attach it read-only.TL;DR:DuckDB 数据库文件不必存储任何数据。它只需存储指向对象存储中 Parquet 文件的视图定义,几百 KB 的文件大小就能像共享目录一样管理整个数据湖。本文将介绍如何构建、托管此类数据库,并以只读方式进行挂载。

A DuckDB database file does not have to contain any data. It can store nothing but view definitions that point at files stored somewhere else, such as Parquet on object storage. The file stays a few hundred kilobytes no matter how large the data it describes, and anyone who attaches it sees a set of named relations that are ready to query.DuckDB 数据库文件不必包含任何数据。它可以仅存储指向其他位置(如对象存储中的 Parquet 文件)的视图定义。无论其描述的数据有多大,该文件始终保持在几百 KB 左右,任何挂载该文件的人都能看到一组可供查询的命名关系。

Nikolas Goebel described this idea in DuckDB doesn't need data⁠, which argues that modern databases have separated where data is stored from how it is read, to the point that most databases no longer need to hold any data of their own. From that perspective, DuckDB becomes a browser for the data cloud, where you access a dataset through a URL rather than a copy on your own disk.Nikolas Goebel 在《DuckDB doesn't need data》一文中阐述了这一理念,他认为现代数据库已经将数据存储位置与读取方式分离,以至于大多数数据库不再需要持有自己的任何数据。从这个角度来看,DuckDB 变成了数据云的浏览器,你通过 URL 而非本地磁盘副本访问数据集。

In this blog post, we turn that idea into a small, shareable catalog.在本文中,我们将这一理念转化为一个小型、可共享的目录。

The Idea 核心理念

A DuckDB table stores its rows inside the database file. A view stores only the text of a query. When you read the view, DuckDB runs that query and returns the current result. If the query reads a remote Parquet file, the data lives in object storage and the view holds only the instruction for how to read it.DuckDB 表将行存储在数据库文件中。而视图仅存储查询文本。当你读取视图时,DuckDB 会运行该查询并返回当前结果。如果查询读取的是远程 Parquet 文件,数据本身位于对象存储中,而视图仅包含如何读取数据的指令。

A database file made entirely of such views contains no rows of its own. It is a catalog: a set of named, ready-to-query relations created over data that stays in its original location and format. You can publish that catalog as a curated semantic layer over a data lake, where the view definitions encode the joins, filters, and column names that consumers should use, without copying or moving the underlying data.完全由此类视图组成的数据库文件本身不包含任何行。它是一个目录:一组针对原始位置和格式的数据所创建的、可直接查询的命名关系。你可以将该目录发布为数据湖之上的精选语义层,其中视图定义编码了消费者应使用的连接、过滤和列名,而无需复制或移动底层数据。

Building a Catalog of Views 构建视图目录

The examples use DuckDB's public train services dataset, which records one row for every stop a Dutch railway train makes. The data comes from the open datasets published by the Rijden de Treinen (Are the trains running?) application⁠, and you can read it straight from its URL.示例使用了 DuckDB 的公共列车服务数据集,该数据集记录了荷兰火车每次停靠的信息。数据来自 Rijden de Treinen(火车运行了吗?)应用程序发布的开放数据集,你可以直接通过其 URL 读取。

First, install the httpfs extension so that DuckDB can read over the network. This only needs to be run once.首先,安装 httpfs 扩展,以便 DuckDB 能够通过网络进行读取。此操作只需运行一次。

INSTALL httpfs;

Now create a persistent database file and define a view over the remote data. Use a fully qualified location, a remote URL or an absolute path, so that the view resolves regardless of the client's working directory.现在创建一个持久化的数据库文件,并针对远程数据定义一个视图。请使用完全限定的位置,即远程 URL 或绝对路径,以确保无论客户端的工作目录如何,视图都能正确解析。

ATTACH 'rail_catalog.duckdb' AS rail;
USE rail;

CREATE VIEW services AS
    SELECT *
    FROM 'https://blobs.duckdb.org/train_services.parquet';

The catalog now exposes a services relation that reads from the remote file whenever it is queried.该目录现在公开了一个 services 关系,每次查询时都会从远程文件读取数据。

SELECT departure_time, station_name, type
FROM rail.services
LIMIT 3;
departure_time station_name type
2023-05-15 00:00:00 Rotterdam Centraal Intercity
2023-05-15 00:13:00 Delft Intercity
2023-05-15 00:29:00 Den Haag HS Intercity

Because the file stores only the view definition, it stays small no matter how large the referenced dataset grows. The train_services.parquet file holds 380,959 rows, yet the catalog that reads it is a few hundred kilobytes.由于文件仅存储视图定义,因此无论引用的数据集增长到多大,文件大小始终很小。train_services.parquet 文件包含 380,959 行数据,而读取它的目录仅有几百 KB。

Hosting the Catalog 托管目录

Upload rail_catalog.duckdb to any location that DuckDB can read over the network, for example an s3:// bucket or an https:// endpoint. The same bucket that already holds the Parquet files is a convenient place to put it.将 rail_catalog.duckdb 上传到 DuckDB 可通过网络读取的任何位置,例如 s3:// 存储桶或 https:// 端点。将其放在已经存储 Parquet 文件的同一个存储桶中就很方便。

Sharing the catalog then means sharing one URL instead of a list of individual file paths. The recipient does not need to know how many files back each view or where they are stored, only the address of the catalog.共享目录意味着只需共享一个 URL,而不是一长串独立的文件路径列表。接收者无需了解每个视图对应多少个文件或它们存储在哪里,只需知道目录的地址即可。

Attaching the Catalog Read-Only 以只读方式挂载目录

Consumers attach the catalog read-only and query the views as if they were local tables. Each query reads the current data from the referenced files, so the result reflects whatever is in object storage at that moment.消费者以只读方式挂载目录,并像查询本地表一样查询这些视图。每次查询都会从引用的文件中读取当前数据,因此结果反映了对象存储中那一刻的状态。

ATTACH 'https://example.com/rail_catalog.duckdb' AS rail (READ_ONLY);

SELECT station_name, count(*) AS calls
FROM rail.services
GROUP BY station_name
ORDER BY calls DESC
LIMIT 3;
station_name calls
Utrecht Centraal 7663
Amsterdam Centraal 7591
Zwolle 5013

Consumers need the httpfs extension and, for s3:// sources, credentials that grant access to the underlying data. DuckDB reads only the columns and row groups a query needs, so a filtered or aggregated query over a large remote dataset does not download the whole thing.消费者需要安装 httpfs 扩展;对于 s3:// 源,还需要授予访问底层数据权限的凭据。DuckDB 只会读取查询所需的列和行组,因此对大型远程数据集进行过滤或聚合查询时,不会下载整个数据集。

Surviving Changes to the Data Layout 应对数据布局的变化

Because a query reads the view rather than the files directly, the publisher can change the files without changing the query. The consumer runs the same query before and after.由于查询读取的是视图而非直接读取文件,发布者可以在不更改查询的情况下更改文件。消费者在变更前后运行的是相同的查询。

Suppose the single Parquet file becomes too large and the publisher splits it into one file per year. Updating the view to read a directory of files is a one-line change in the catalog, and consumers notice nothing.假设单个 Parquet 文件变得过大,发布者将其拆分为按年份存储的文件。更新视图以读取文件目录在目录中只需一行代码的修改,而消费者对此毫无察觉。

CREATE OR REPLACE VIEW services AS
    SELECT *
    FROM read_parquet('s3://my-bucket/rail/services/year=*/*.parquet');

A view can also keep the schema stable when the files change. If a later batch of files renames station_name to station, the view can map the new column back to the name consumers already use.当文件发生变化时,视图还可以保持模式(Schema)的稳定。如果后续一批文件将 station_name 重命名为 station,视图可以将新列映射回消费者已经使用的名称。

CREATE OR REPLACE VIEW services AS
    SELECT
        * EXCLUDE (station),
        station AS station_name
    FROM read_parquet('s3://my-bucket/rail/services/*.parquet');

If you repartition the files or move them to a new bucket, you only have to update the catalog. Anyone querying it keeps running the same queries as before.如果你重新分区文件或将其移动到新的存储桶,只需更新目录即可。查询它的任何人都可以继续运行与之前相同的查询。

When to Use It 适用场景

Use this pattern when you want to share a dataset without sending the recipient a long list of individual file paths, and when the data is already in a format DuckDB reads well, such as Parquet. Readers get a set of named relations to query, and the publisher can change the file layout without breaking those queries.当你希望共享数据集而无需向接收者发送冗长的文件路径列表,且数据已经处于 DuckDB 可良好读取的格式(如 Parquet)时,请使用此模式。读者可以获得一组可查询的命名关系,且发布者可以在不破坏这些查询的情况下更改文件布局。

Note the following limitations:请注意以下限制:

  • Consumers must be able to access both the catalog file and every data source that its views reference.消费者必须能够同时访问目录文件及其视图引用的所有数据源。
  • Connections over HTTPS and the S3 API are read-only, so the catalog cannot be modified in place. To change a view, edit a local copy and upload it again.通过 HTTPS 和 S3 API 的连接是只读的,因此目录无法在原地修改。若要更改视图,请编辑本地副本并重新上传。
  • Query performance depends on the network and the layout of the referenced files. The tips for querying remote files apply.查询性能取决于网络状况和引用文件的布局。查询远程文件的相关建议同样适用。

Conclusion 结论

A DuckDB file that holds only views is a small catalog you can share as a single URL. The data stays in its original location, the publisher can change how it is stored, and the reader gets a stable set of relations to query.仅包含视图的 DuckDB 文件是一个小型目录,你可以将其作为单个 URL 进行共享。数据保留在原始位置,发布者可以更改其存储方式,而读者则获得了一组稳定的可查询关系。

For the full reference, see the Share a DuckDB Database That Only Stores Views guide.有关完整参考,请参阅《Share a DuckDB Database That Only Stores Views》指南。