Introducing S3-compatible storage support for Iceberg tables in Snowflake Horizon Catalog

AI is only as good as the data it can reach. This is why teams are adopting Apache Iceberg™: to access data in place from their preferred engines and models.AI 的能力取决于它所能获取的数据。这就是团队采用 Apache Iceberg™ 的原因：以便从他们偏好的引擎和模型中就地访问数据。

But landing data in Iceberg isn’t enough. You also have to securely manage access to those tables from every engine. This is where Snowflake Horizon Catalog comes in. Built on Apache Polaris™, it leverages open security mechanisms based on the Iceberg REST Catalog API to grant query-specific secure access to engines like Spark, Trino, Flink, PyIceberg, DuckDB, and more for both read and write operations. This means engines can get the same level of access to these tables as Snowflake's own query engine does.但仅仅将数据存入 Iceberg 是不够的。您还必须安全地管理从各个引擎对这些表的访问。这就是 Snowflake Horizon Catalog 的用武之地。它构建于 Apache Polaris™ 之上，利用基于 Iceberg REST Catalog API 的开放安全机制，为 Spark、Trino、Flink、PyIceberg、DuckDB 等引擎授予针对特定查询的安全访问权限，涵盖读写操作。这意味着这些引擎可以获得与 Snowflake 自身查询引擎同等级别的表访问权限。

This setup has worked for Amazon S3, Azure Storage, and Google Cloud Storage out of the box. However, it did not work for any other storage that speaks the S3 wire protocol without being AWS. Until now, MinIO, Ceph, Wasabi, Backblaze B2, Cloudflare R2, IDrive e2, or a self-hosted cluster in your own data center could not participate in this architecture pattern. The problem isn’t about how Snowflake writes these tables. Rather, other engines couldn't read the file paths in the Iceberg metadata.这种设置对于 Amazon S3、Azure Storage 和 Google Cloud Storage 来说是开箱即用的。然而，它并不适用于除 AWS 之外任何使用 S3 线协议的存储。在此之前，MinIO、Ceph、Wasabi、Backblaze B2、Cloudflare R2、IDrive e2 或您自己数据中心内的自托管集群都无法参与这种架构模式。问题不在于 Snowflake 如何写入这些表，而在于其他引擎无法读取 Iceberg 元数据中的文件路径。

This ends today. With this private preview, Snowflake Horizon managed Iceberg tables on S3-compatible storage are now readable and writable from any compatible engine.今天，这一切都将改变。通过此次私有预览版，Snowflake Horizon 管理的 S3 兼容存储上的 Iceberg 表现在可以被任何兼容的引擎读取和写入。

Why this matters为何这很重要

Plenty of data cannot move into public cloud storage, often for reasons that have nothing to do with technology: a data residency requirement, a sovereign cloud mandate, an existing on-premises investment, or an egress bill that makes a copy uneconomic. Until now, those teams had to choose between keeping the data where it belongs or getting multi-engine access to it.大量数据无法迁移到公有云存储中，通常原因与技术无关：数据驻留要求、主权云授权、现有的本地投资，或者会导致复制成本不经济的出口费用。在此之前，这些团队必须在“将数据保留在应有的位置”和“实现对数据的多引擎访问”之间做出选择。

Horizon Catalog can already govern those tables and query with Snowflake’s compute. Engines outside Snowflake couldn’t access them.Horizon Catalog 已经能够管理这些表并使用 Snowflake 的计算资源进行查询。但 Snowflake 之外的引擎无法访问它们。

Now they can access these tables too.现在，它们也可以访问这些表了。

Horizon as a translation layer作为转换层的 Horizon

Horizon Catalog is designed to completely abstract storage location and type for engines. The result is a consistent experience for your teams regardless of where the data is stored. It works by first identifying the storage location by where it lives, not just how to talk to it. This works because commercial AWS, government regions, sovereign clouds, and S3-compatible systems each carry a distinct identity inside Snowflake. All your preferred engines see is a pointer to the table’s metadata file.Horizon Catalog 旨在为引擎完全抽象化存储位置和类型。其结果是，无论数据存储在何处，您的团队都能获得一致的体验。它的工作原理是首先识别存储位置，而不仅仅是识别如何与存储进行通信。这是可行的，因为商业 AWS、政府区域、主权云和 S3 兼容系统在 Snowflake 内部都拥有独特的标识。您所有偏好的引擎所看到的只是一个指向表元数据文件的指针。

Powering this experience is Horizon Catalog's built-in storage intelligence layer. For every table, it resolves the full access contract an engine needs. Part of that contract is the table's location, in the form that the reader expects. Azure resolves to abfss://, Google Cloud Storage to gs://, and S3-compatible storage to standard s3://. Snowflake's internal storage model isn’t exposed in the open metadata, so standard Iceberg engines can work with these tables without Snowflake-specific support.支撑这种体验的是 Horizon Catalog 内置的存储智能层。对于每一张表，它都会解析引擎所需的完整访问契约。该契约的一部分是表的位置，以读取器期望的格式呈现。Azure 解析为 abfss://，Google Cloud Storage 解析为 gs://，而 S3 兼容存储则解析为标准的 s3://。Snowflake 的内部存储模型不会暴露在开放元数据中，因此标准的 Iceberg 引擎无需 Snowflake 的特定支持即可使用这些表。

[The two storage models Horizon translates between][Horizon 在其间进行转换的两种存储模型]

Your storage stays the same. Your configuration stays the same. Your S3-compatible table now behaves like any other Iceberg table.您的存储保持不变。您的配置保持不变。您的 S3 兼容表现在表现得就像任何其他 Iceberg 表一样。

Horizon: The catalog for universal governance, including diverse storage typesHorizon：实现通用治理的目录，涵盖多种存储类型

Interoperability is only useful if it does not cost you your controls. Your existing access controls come along with these tables. The same roles and grants apply. The catalog remains the single place access is decided. Operations external engines perform through the Iceberg REST Catalog APIs land in Access History next to your Snowflake SQL activity, giving these tables, including from external engines, one audit trail.互操作性只有在不牺牲控制权的前提下才有用。您现有的访问控制会随这些表一同迁移。相同的角色和授权依然适用。目录仍然是决定访问权限的唯一场所。外部引擎通过 Iceberg REST Catalog API 执行的操作会出现在“访问历史记录”（Access History）中，与您的 Snowflake SQL 活动并列，从而为这些表（包括来自外部引擎的访问）提供统一的审计追踪。

Updating your existing tables更新现有表

Tables predating this feature have their older metadata files written with the internal scheme. They need a one-time metadata update before external engines can access them. During this preview, our team runs this update for you on request. Tables created after S3-compatible storage support is enabled for your account need no additional work.在此功能推出前创建的表，其旧版元数据文件是使用内部方案编写的。在外部引擎访问它们之前，需要进行一次性的元数据更新。在此预览期间，我们的团队可应要求为您执行此更新。在您的账户启用 S3 兼容存储支持后创建的表无需进行额外操作。

Two things to know about this update. First, Horizon Catalog makes it easy to see which tables need it: until a table is updated, the catalog returns a clear error instead of metadata the engine can't resolve, so you get a clear error up front rather than a scan that fails partway through. Second, the update rebuilds only the Iceberg metadata layer, which means rewriting a few metadata files without moving any data. Snowflake Time Travel is unaffected.关于此更新有两点需要了解。首先，Horizon Catalog 可以轻松识别哪些表需要更新：在表更新之前，目录会返回一个明确的错误，而不是引擎无法解析的元数据，因此您会预先得到明确的错误提示，而不是在扫描过程中途失败。其次，更新仅重建 Iceberg 元数据层，这意味着只需重写少量元数据文件，而无需移动任何数据。Snowflake 时间旅行（Time Travel）功能不受影响。

How it works工作原理

Point Horizon Table at an S3-compatible volume:将 Horizon Table 指向 S3 兼容卷：

The engine connects with your endpoint and keys. Here with PyIceberg:引擎使用您的端点和密钥进行连接。以下是使用 PyIceberg 的示例：

Spark, Trino and others take the same three properties under their own configuration prefixes.Spark、Trino 和其他引擎在各自的配置前缀下使用相同的三个属性。

What's coming next未来展望

Vended credentials: Today, the engine manages its own bucket access because it interacts with S3-compatible volumes using static keys and secrets that we cannot scope down. Even though some providers issue short-lived credentials, they force us to use different APIs. To solve this, we are actively building a way to vend scoped credentials for each storage platform.分发凭证（Vended credentials）：目前，引擎管理其自身的存储桶访问权限，因为它使用我们无法缩小范围的静态密钥和机密信息与 S3 兼容卷进行交互。尽管一些提供商会颁发短期凭证，但它们迫使我们使用不同的 API。为了解决这个问题，我们正在积极构建一种为每个存储平台分发受限凭证的方法。

Get Started开始使用

S3-compatible storage support is in private preview. If you have Iceberg data on MinIO, Ceph, Cloudflare R2, or a cluster in your own data center, talk to your account team about switching it on.S3 兼容存储支持目前处于私有预览阶段。如果您在 MinIO、Ceph、Cloudflare R2 或您自己数据中心的集群上有 Iceberg 数据，请联系您的客户团队以开启此功能。

Apache Iceberg™ is a trademark of the Apache Software Foundation in the United States and/or other countries.Apache Iceberg™ 是 Apache 软件基金会在美国和/或其他国家的商标。

Forward-looking statements前瞻性陈述

This article contains forward-looking statements, including about our future product offerings, and are not commitments to deliver any product offerings. Actual results and offerings may differ and are subject to known and unknown risk and uncertainties. See our latest 10-Q for more information.本文包含前瞻性陈述，包括关于我们未来产品供应的陈述，这些陈述并非交付任何产品供应的承诺。实际结果和供应情况可能会有所不同，并受已知和未知的风险及不确定性影响。有关更多信息，请参阅我们最新的 10-Q 报告。