Blog/Data Engineering/Managing Business Logic Interoperability in Multi-Engine Lakehouses
Jul 28, 2026/28 min readData Engineering

Managing Business Logic Interoperability in Multi-Engine Lakehouses

A CFO gets different revenue figures from two dashboards, even though both pointed at the same Iceberg table. This isn't always because the data drifted, but rather, because the two dashboards point to different engines. "Recognized revenue" was defined in a SQL view in each engine, and the definitions quietly drifted or contained a difference in how each engine handles nulls in a CASE expression. No error, no alert. Just two different answers from the same source of truth, and no clear owner for the reconciliation.一位首席财务官(CFO)从两个仪表板中得到了不同的收入数据,尽管两者都指向同一个 Iceberg 表。这并不总是因为数据发生了漂移,而是因为这两个仪表板指向了不同的引擎。每个引擎中的 SQL 视图都定义了“已确认收入”,但这些定义在悄然发生漂移,或者在各引擎处理 CASE 表达式中的空值(null)的方式上存在差异。没有错误,也没有警报。仅仅是来自同一个事实来源的两个不同答案,而且没有明确的负责人来处理对账。

The series breaks down multi-engine lakehouse architectures into three interoperability dimensions: data, business logic and governance. Solving for interoperability requires ecosystem solutions that are vendor-neutral and provide customers with optionality at each of these dimensions. Apache Iceberg™ for data interoperability (and some minor efforts to address business logic interoperability) and Apache Polaris™ for governance interoperability have emerged as widely adopted open source projects with diverse communities. For business logic interoperability holistically, however, no open source project has reached the same level of adoption as those two, though progress is being made with Apache Ossie (incubating), formerly known as Open Semantic Interchange (OSI), which has seen vendor participation more than double in the past year.本系列文章将多引擎湖仓架构拆解为三个互操作性维度:数据、业务逻辑和治理。解决互操作性问题需要厂商中立的生态系统解决方案,并为客户在每个维度上提供选择权。Apache Iceberg™(用于数据互操作性,以及一些解决业务逻辑互操作性的微小努力)和 Apache Polaris™(用于治理互操作性)已成为拥有多元化社区、被广泛采用的开源项目。然而,在业务逻辑互操作性方面,尚未有开源项目达到与前两者同等的采用水平,尽管 Apache Ossie(孵化中,前身为 Open Semantic Interchange,简称 OSI)正在取得进展,该项目在过去一年中厂商参与度翻了一番。

While business logic doesn't carry the same stakes as governance interoperability, where gaps can mean compliance violations and data breaches, it's still where organizations are most likely to get caught off-guard as they plan their lakehouse and AI deployments. These gaps are mostly not acceptable in an enterprise environment. The good news is that viable workarounds exist today, and for most organizations, those workarounds are enough to architect around the gaps while the ecosystem catches up.虽然业务逻辑不像治理互操作性那样涉及合规违规和数据泄露等高风险问题,但它仍然是组织在规划湖仓和 AI 部署时最容易措手不及的地方。在企业环境中,这些差距大多是不可接受的。好消息是,目前存在可行的变通方案,对于大多数组织而言,这些方案足以在生态系统赶上之前绕过这些差距进行架构设计。

What a multi-engine lakehouse is really doing is unbundling a data warehouse. The data interoperability deep dive covers the storage layer of that unbundling, which is in reasonable shape. But a data warehouse/data processing system was never just storage; it was also business logic entities like views, user-defined functions (UDFs), stored procedures and metric definitions that turned raw tables into business answers.多引擎湖仓实际上所做的是对数据仓库进行解耦。数据互操作性的深度解析涵盖了这种解耦的存储层,该层目前状态良好。但数据仓库/数据处理系统从来不仅仅是存储;它还包含视图、用户定义函数(UDF)、存储过程和指标定义等业务逻辑实体,正是这些实体将原始表转化为了业务答案。

Once you decouple compute engines from a previously tightly coupled compute engine and storage engine, all that business logic needs to either live somewhere securely accessible and engine-agnostic or be replicated independently in every engine that accesses the data. This overview provides a deep dive into existing mechanisms for managing business logic in multi-engine lakehouse environments and practical guidance for how to architect with today's solutions as you look to the future when interoperable business logic is fully solved.一旦将计算引擎从之前紧密耦合的计算和存储引擎中分离出来,所有这些业务逻辑要么必须存在于某个安全可访问且与引擎无关的地方,要么必须在访问数据的每个引擎中独立复制。本概述深入探讨了在多引擎湖仓环境中管理业务逻辑的现有机制,并为在未来彻底解决互操作性业务逻辑问题之前,如何利用当今的解决方案进行架构设计提供了实用指导。

Where we are today现状

There is no complete, widely adopted standard for sharing business logic across engines in production. Not for lack of attempts or progress: Iceberg Views, Iceberg UDFs, Substrait and Ossie all exist in various stages of maturity. None have achieved the cross-engine adoption needed to be a reliable architectural dependency today. For anyone making production architecture decisions right now, the practical effect is the same as having no standard at all, though the seeds being planted should factor into long-term planning.目前还没有一个完整且被广泛采用的标准来在生产环境中跨引擎共享业务逻辑。这并非缺乏尝试或进展:Iceberg 视图、Iceberg UDF、Substrait 和 Ossie 都处于不同的成熟阶段。但没有一个项目实现了作为可靠架构依赖项所需的跨引擎采用率。对于目前正在制定生产架构决策的任何人来说,其实际效果等同于完全没有标准,尽管所播下的种子应纳入长期规划中。

It's worth framing the scope of this problem accurately. The majority of query volume in most data platforms hits tables directly: SELECT against Iceberg tables where data interoperability (covered in the data interoperability deep dive) is sufficient. Business logic interoperability matters specifically for the workloads that route through or include views, UDFs, metric definitions and stored procedures. While that may be a smaller share of total query volume, it's where the highest-stakes business answers live. The CFO's revenue dashboard, the regulatory report, the executive KPI that drives a board decision: Those are the workloads that encode business logic, and those are the ones affected by the gaps described in this post.准确界定此问题的范围非常重要。大多数数据平台中,大部分查询量直接作用于表:即针对 Iceberg 表执行 SELECT 操作,此时数据互操作性(在数据互操作性深度解析中涵盖)已足够。业务逻辑互操作性主要针对那些通过或包含视图、UDF、指标定义和存储过程的工作负载。虽然这在总查询量中可能占比很小,但却是最高风险业务答案的所在。CFO 的收入仪表板、监管报告、驱动董事会决策的执行 KPI:这些都是编码了业务逻辑的工作负载,也是受本文所述差距影响的工作负载。

The consequence is straightforward. Any metric, view, UDF or transformation logic defined in one engine must be manually redefined in every other engine that accesses the same data. There's no synchronization mechanism and no feedback loop that tells you when definitions have drifted. This isn't a hypothetical problem. Companies like Airbnb and LinkedIn have built entire internal frameworks to address it: Minerva at Airbnb for metric consistency and Coral at LinkedIn for cross-engine SQL translation. But those organizations have large dedicated engineering teams that can sustain these systems indefinitely. For most organizations, that's not a realistic path; the ongoing maintenance burden alone tends to outweigh the initial build cost.后果显而易见。在某一个引擎中定义的任何指标、视图、UDF 或转换逻辑,都必须在访问相同数据的每一个其他引擎中手动重新定义。没有同步机制,也没有反馈回路来告知定义何时发生了漂移。这并非假设性问题。Airbnb 和 LinkedIn 等公司已经构建了完整的内部框架来解决此问题:Airbnb 的 Minerva 用于指标一致性,LinkedIn 的 Coral 用于跨引擎 SQL 翻译。但这些组织拥有庞大的专业工程团队,可以无限期地维护这些系统。对于大多数组织而言,这不是一条现实的路径;仅持续的维护负担往往就超过了最初的构建成本。

For governance interoperability, tools like Immuta and Privacera exist to centralize policy definitions and push them to multiple engines. No equivalent exists for business logic. Semantic layer platforms like Cube and AtScale come closest: You define metrics once, and they generate engine-specific SQL at query time. But consumers must query through the tool for the consistency guarantee to hold; anything that bypasses it doesn't benefit, and the "sit-in-the-middle" architecture runs into the same scaling and single-point-of-failure problems that have historically limited data virtualization, even when they are distributed systems. Transformation tools like dbt and SQLMesh take a different approach. They generate per-engine SQL at build time from a single model definition, which is closer to "define once, deploy to many," but they only cover the transformations managed through the tool. We cover both in detail later in this post.对于治理互操作性,Immuta 和 Privacera 等工具可以集中定义策略并将其推送到多个引擎。但业务逻辑领域不存在等效工具。Cube 和 AtScale 等语义层平台最接近这一目标:你只需定义一次指标,它们会在查询时生成特定于引擎的 SQL。但消费者必须通过该工具进行查询才能保证一致性;任何绕过它的行为都无法从中受益,而且这种“中间件”架构会遇到历史上限制数据虚拟化扩展和单点故障的问题,即使它们是分布式系统。dbt 和 SQLMesh 等转换工具采取了不同的方法。它们在构建时从单一模型定义生成每个引擎的 SQL,这更接近“一次定义,多处部署”,但它们只涵盖通过该工具管理的转换。我们将在本文稍后详细介绍这两者。

The problem gets worse in decentralized organizations where different business units build their own logic independently on shared data sets, each internally consistent but unaware when their definitions diverge from each other.在去中心化的组织中,这个问题会变得更糟,因为不同的业务部门在共享数据集上独立构建自己的逻辑,虽然各自内部保持一致,但却无法察觉定义何时与其他部门产生了分歧。

The pragmatic answer for today: For high-stakes business logic (for example, finance, regulatory and executive dashboards), either physicalize the logic via a pipeline that computes the result of the logic and writes the results of that computation to a table so all engines inherently get the same answer, or centralize authoritative definitions into a single, enterprise-ready engine capable of handling highly concurrent workloads. Multi-engine reads on shared Iceberg tables are reasonable for ad hoc analysis; multi-engine definitions of "recognized revenue" are not, unless the maintenance burden is worth it for a given definition. This isn't a retreat from the lakehouse vision. Your data is still in Iceberg, portable and open. You're just being deliberate about where authoritative logic lives while the ecosystem catches up.针对当下的务实回答是:对于高风险业务逻辑(例如财务、监管和执行仪表板),要么通过管道将逻辑物理化,即计算逻辑结果并将计算结果写入表,以便所有引擎本质上都能得到相同的答案;要么将权威定义集中到一个能够处理高并发工作负载的单一企业级引擎中。针对共享 Iceberg 表的多引擎读取对于即席分析是合理的;但除非维护负担对于特定定义来说是值得的,否则不建议使用多引擎定义“已确认收入”。这并非放弃湖仓愿景。你的数据依然在 Iceberg 中,是可移植且开放的。你只是在生态系统赶上之前,审慎地决定权威逻辑的存放位置。

Here's a summary table of what we'll go through in this blog:以下是本文将讨论内容的摘要表:

Mechanism What it covers State today Pragmatic guidance
Iceberg Views Named SQL queries stored in the catalog; supports multiple dialect representations Spec live; Spark/Trino ship single-dialect support; multi-dialect authoring limited to Java API; no official transpilation stance or tool Use as a catalog-level registry; don't rely on for cross-engine consistency without verification; physicalize or designate a single authoritative engine for high-stakes logic
Iceberg UDFs SQL UDF metadata format in the catalog; same architecture as views Spec published; no major engine has shipped support Treat as engine-locked for now; route critical UDF workloads through a single engine
Apache Ossie (incubating) Metric and dimension definitions; YAML-based, vendor-neutral; AI context support 50+ orgs in working group; spec live under Apache 2.0; converters in progress; incubation underway at the Apache Software Foundation Most credible path for metric consistency long-term; in the meantime, centralize authoritative definitions in a single engine or semantic layer
dbt Semantic Models / SQLMesh Metric and transformation definitions as code; generate per-engine SQL at build time Mature; widely deployed Useful for managing consistency within transformations they own; doesn't address behavioral divergence at query time; secondary to architectural mitigations
sqlglot SQL transpilation across 30+ dialects; handles null ordering, type casting, function semantics Mature open source; actively maintained Use for migrations; covers a lot of semantic ground, but not all; note that you take on operations and accountability
Substrait Cross-language spec for serializing relational algebra DuckDB, Arrow, DataFusion support; not much support outside of those Right long-term concept; the project isn't very mature from an adoption perspective, so don't treat it as a production dependency until the engines you use actually adopt it
Stored procedures Engine-proprietary procedural languages; no shared abstraction exists No community spec; no portability mechanism; no clear path Accept as engine-locked; route those workloads through a single engine; refactor to declarative SQL or externalize the logic to a shared tool

What exists but doesn't fully solve the business logic problem现有但未能完全解决业务逻辑问题的方案

Before evaluating the tools and specs the community has built, it's worth establishing why cross-engine business logic consistency is hard for the workloads that depend on it, which is certainly not all workloads. The behavioral differences across SQL engines are the root cause; everything that follows in this section is an attempt to work around them.在评估社区构建的工具和规范之前,有必要先确定为什么对于依赖它的工作负载(当然并非所有工作负载)而言,跨引擎业务逻辑的一致性很难实现。各 SQL 引擎的行为差异是根本原因;本节后续内容都是为了规避这些差异所做的尝试。

Cross-engine behavioral differences跨引擎行为差异

The most fundamental challenge for business logic interoperability isn't that engines use different SQL syntax; syntax differences are well-understood and tools like sqlglot or custom tooling can translate between them. The harder problem is that engines with identical syntax produce different results because they implement SQL semantics differently. These aren't bugs; each engine is internally consistent. The problem is that "correct" is engine-specific, and two engines can both be right by their own standards while producing different answers from the same query on the same data. These divergences exist regardless of the underlying table format; they're properties of the compute engines, not the storage layer.业务逻辑互操作性最根本的挑战不在于引擎使用不同的 SQL 语法;语法差异是众所周知的,sqlglot 或自定义工具等工具可以在它们之间进行翻译。更棘手的问题是,语法相同的引擎因为 SQL 语义实现方式不同而产生不同的结果。这些并非 Bug;每个引擎内部都是一致的。问题在于“正确”是特定于引擎的,两个引擎可能根据各自的标准都是正确的,但在对相同数据执行相同查询时却产生不同的答案。无论底层表格式如何,这些差异都存在;它们是计算引擎的属性,而非存储层的属性。

A few concrete examples:几个具体的例子:

  • ORDER BY with null values: Spark defaults to nulls first for ascending order; DuckDB and Trino default to nulls last. A query that sorts users by last_login_date ascending to find inactive accounts will place users who have never logged in (null) at the top on Spark and at the bottom on DuckDB or Trino, changing which users appear in the first page of results.带有空值的 ORDER BY:Spark 在升序排序时默认空值在前;DuckDB 和 Trino 则默认空值在后。一个按 last_login_date 升序对用户进行排序以查找非活跃账户的查询,在 Spark 上会将从未登录过的用户(null)排在最前面,而在 DuckDB 或 Trino 上则排在最后,从而改变了出现在第一页结果中的用户。
  • Type casting: Spark's behavior depends on whether ANSI mode is enabled; with ANSI mode off (the historical default; Spark 3.x defaults to ANSI mode off; Spark 4.0 changed this default, though many production deployments remain on 3.x), CAST('a' AS INT) silently returns null, while DuckDB and Trino throw an error on invalid casts by default (both provide TRY_CAST as the null-returning alternative). A pipeline that casts user input and filters on the result will silently include rows with null values on Spark (ANSI off) that would have been excluded on DuckDB or Trino.类型转换:Spark 的行为取决于是否启用了 ANSI 模式;在 ANSI 模式关闭时(历史默认值;Spark 3.x 默认关闭 ANSI 模式;Spark 4.0 更改了此默认值,尽管许多生产部署仍停留在 3.x 版本),CAST('a' AS INT) 会静默返回 null,而 DuckDB 和 Trino 默认情况下会对无效转换抛出错误(两者都提供 TRY_CAST 作为返回 null 的替代方案)。一个对用户输入进行转换并根据结果进行过滤的管道,在 Spark(ANSI 关闭)上会静默包含空值行,而在 DuckDB 或 Trino 上这些行会被排除。
  • Timestamp handling: The timezone that an engine assumes when resolving timestamp values adds another layer. How timezone resolution interacts with session configuration and how daily or weekly aggregations behave across DST boundaries all vary.时间戳处理:引擎在解析时间戳值时假设的时区增加了另一层复杂性。时区解析如何与会话配置交互,以及每日或每周聚合如何跨越夏令时(DST)边界,这些情况各不相同。

These differences tend to surface in production on edge cases rather than in testing on clean data.这些差异往往在生产环境的边缘案例中浮现,而不是在干净数据的测试中。

Figure 1: An example where the same query against the same table produces different results, depending on what engine executed the query.
Figure 1: An example where the same query against the same table produces different results, depending on what engine executed the query.

 

SQL is where most of the business logic portability discussion focuses, but it's not the only language in play. UDFs can be written in Python, Java or Scala with no cross-engine portability mechanism; stored procedures depend on engine-proprietary procedural languages with no equivalents elsewhere. Both are covered in dedicated sections later in this post.SQL 是大多数业务逻辑可移植性讨论的焦点,但它并不是唯一涉及的语言。UDF 可以用 Python、Java 或 Scala 编写,且没有跨引擎可移植性机制;存储过程依赖于引擎专有的过程语言,在其他地方没有等效项。这两者将在本文稍后的专门章节中介绍。

The governance deep dive covers fine-grained access control (FGAC) enforcement approaches in detail. The primitive to carry forward from there is that SQL behavioral divergence can affect who sees what data, not just whether a dashboard number is right. One consequence that extends beyond analytics: FGAC policies (row filters, column masks) are typically SQL expressions. For simple predicates on standard operators (<, >=, !=), all major engines execute them the same way. But filters involving type casting, null handling or function calls enter the territory where engines can produce different results, meaning the security boundary itself varies by engine. A row filter that evaluates differently on Engine B than Engine A means users may see different data depending on which engine they query through, with no error to flag the discrepancy.治理深度解析详细涵盖了细粒度访问控制(FGAC)的执行方法。从中引申出的基本原则是,SQL 行为差异不仅会影响仪表板数字的正确性,还会影响谁能看到什么数据。除了分析之外,另一个后果是:FGAC 策略(行过滤器、列掩码)通常是 SQL 表达式。对于标准运算符(<, >=, !=)上的简单谓词,所有主流引擎的执行方式相同。但涉及类型转换、空值处理或函数调用的过滤器则进入了引擎可能产生不同结果的领域,这意味着安全边界本身因引擎而异。如果一个行过滤器在引擎 B 上的评估结果与引擎 A 不同,则意味着用户根据其查询的引擎不同,可能会看到不同的数据,且没有任何错误来标记这种差异。

Iceberg viewsIceberg 视图

With that behavioral variance as context, let's look at what the Iceberg community has built to share business logic across engines. The Iceberg view spec defines a metadata object for storing named SQL queries in the catalog alongside tables. Each view version contains one or more representations: a {type: "sql", sql: "...", dialect: "..."} tuple that records the SQL text and the dialect it was written in. A view can store multiple representations for different dialects, so in principle an engine can look for a representation matching its own dialect. The view is versioned, discoverable via the REST Catalog API and appears as a table-like object to downstream consumers.在了解了这些行为差异作为背景后,让我们看看 Iceberg 社区为在引擎间共享业务逻辑所做的工作。Iceberg 视图规范定义了一个元数据对象,用于在目录中与表一起存储命名的 SQL 查询。每个视图版本包含一个或多个表示形式:一个 {type: "sql", sql: "...", dialect: "..."} 元组,记录了 SQL 文本及其编写所用的方言。视图可以为不同方言存储多种表示形式,因此原则上引擎可以寻找与其自身方言匹配的表示形式。视图具有版本控制,可通过 REST Catalog API 发现,并对下游消费者呈现为类似表的对象。

// simplified — the full spec includes schema, default catalog/namespace, and other fields
{
  "version-id": 2,
  "representations": [
    { "type": "sql", "sql": "SELECT date_add(order_date, 30) ...", "dialect": "spark" },
    { "type": "sql", "sql": "SELECT date_add('day', 30, order_date) ...", "dialect": "trino" }
  ],
  ...
}

Iceberg views are a real step forward from having no shared view mechanism at all. It also stops well short of solving the problem. The spec is a metadata container: It stores what you give it, with no translation layer, no behavioral normalization and no mechanism to verify that multiple dialect representations actually produce the same results. If you store a Spark SQL representation and a Trino SQL representation of the same view, keeping them semantically equivalent as both the underlying data and the business logic evolve is entirely your responsibility.Iceberg 视图是相对于没有共享视图机制的一大进步。但它距离解决问题还有很大差距。该规范是一个元数据容器:它存储你提供的内容,没有翻译层,没有行为规范化,也没有机制来验证多种方言表示形式是否确实产生相同的结果。如果你存储了同一视图的 Spark SQL 和 Trino SQL 表示形式,那么随着底层数据和业务逻辑的演变,保持它们在语义上等效完全是你自己的责任。

The deeper issue, covered in detail in the cross-engine behavioral differences section above, is that even identical SQL text can produce different results across engines. The behavioral divergences described there (null ordering, type casting, timestamp handling) all apply directly to Iceberg views. A view stored in one dialect and executed by another engine is subject to every one of those differences.更深层次的问题(如上文跨引擎行为差异章节所述)在于,即使相同的 SQL 文本在不同引擎上也可能产生不同的结果。那里描述的行为差异(空值排序、类型转换、时间戳处理)都直接适用于 Iceberg 视图。以一种方言存储并由另一种引擎执行的视图,会受到所有这些差异的影响。

The community is actively developing an Iceberg materialized view spec with the same metadata-container architecture, specifying a SQL dialect, as well as adding a data dimension to the problem. The materialized output is a physical Iceberg table, so you inherit both the business logic and data interoperability issues.社区正在积极开发 Iceberg 物化视图规范,采用相同的元数据容器架构,指定 SQL 方言,并为该问题增加了数据维度。物化输出是一个物理 Iceberg 表,因此你同时继承了业务逻辑和数据互操作性问题。

The community is actively working on closing these gaps. The Iceberg view spec already supports storing multiple dialect representations per view, but today that's practically only accessible via the Java ViewBuilder API. Engines like Spark and Trino only write a single dialect through SQL. Making multi-dialect authoring easier via PyIceberg and engine-level SQL support is an ongoing discussion, but the conversation in the community has evolved. The community has also recently merged a structured expressions spec that represents value expressions as structured signatures rather than dialect-specific SQL text, which is covered in the "Open problems and active work" section below.社区正在积极努力弥补这些差距。Iceberg 视图规范已经支持为每个视图存储多种方言表示形式,但目前实际上只能通过 Java ViewBuilder API 访问。Spark 和 Trino 等引擎仅通过 SQL 写入单一方言。通过 PyIceberg 和引擎级 SQL 支持使多方言编写变得更容易是一个持续的讨论话题,但社区的对话已经演变。社区最近还合并了一个结构化表达式规范,该规范将值表达式表示为结构化签名,而不是特定于方言的 SQL 文本,这将在下文“开放问题与正在进行的工作”章节中介绍。

Pragmatic guidance务实指导

Iceberg views are worth using as a catalog-level registry for view definitions; having view metadata in the catalog rather than scattered across engine-specific catalogs is a genuine organizational improvement. But don't rely on them for cross-engine consistency without additional discipline. If a view encodes high-stakes business logic, either: (a) physicalize it as a table so all engines read the same precomputed result; or (b) designate one engine as the authoritative source for that view, and treat the Iceberg view as documentation rather than an executable contract across engines. If you do maintain multiple dialect representations, build cross-engine regression tests that compare results on representative data and run them on a schedule; the drift is silent and gradual.Iceberg 视图值得作为视图定义的目录级注册表使用;将视图元数据放在目录中而不是分散在特定于引擎的目录中,确实是组织上的一大改进。但在没有额外约束的情况下,不要依赖它们来实现跨引擎的一致性。如果视图编码了高风险业务逻辑,请采取以下措施之一:(a) 将其物理化为表,以便所有引擎读取相同的预计算结果;或 (b) 指定一个引擎作为该视图的权威来源,并将 Iceberg 视图视为文档,而不是跨引擎的可执行契约。如果你确实维护多种方言表示形式,请构建跨引擎回归测试,在代表性数据上比较结果并按计划运行;漂移是无声且渐进的。

Iceberg UDFsIceberg UDF

The Iceberg community has also specified a SQL UDF metadata format that follows the same architectural pattern as views: Store the function body, its SQL dialect, input/output types and versioning metadata in the catalog. Like views, UDFs can store multiple representations for different dialects. The spec is published, but no major engine has shipped support yet; this is earlier in its lifecycle than Iceberg views.Iceberg 社区还指定了一种 SQL UDF 元数据格式,遵循与视图相同的架构模式:在目录中存储函数体、其 SQL 方言、输入/输出类型和版本控制元数据。与视图一样,UDF 可以为不同方言存储多种表示形式。规范已经发布,但目前还没有主流引擎提供支持;这比 Iceberg 视图处于生命周期的更早阶段。

The more structural challenge is that the Iceberg UDF spec only covers SQL UDFs. In practice, many UDFs that encode critical business logic are written in Python, Java or Scala. SQL UDFs have the same dialect divergence problem as SQL views: The function body is SQL text, and the same behavioral differences (type casting, null handling, function semantics) apply. Non-SQL UDFs add another dimension. Python UDFs have relatively broad support across engines (Snowpark, Spark, Trino, DuckDB all support them), though the registration mechanisms and runtime environments differ. But other languages are less portable. Scala UDFs, for example, have no equivalent in Trino or DuckDB. For non-SQL UDFs to become portable via the Iceberg spec, the spec would first need to support non-SQL languages as representation types, and the community would need to converge on a common invocation contract across engines. That's a harder problem than SQL dialect translation, though in many cases the core function logic is already portable; it's the wiring around it (type declarations, serialization, available libraries) that isn't.更具结构性的挑战在于,Iceberg UDF 规范仅涵盖 SQL UDF。在实践中,许多编码关键业务逻辑的 UDF 是用 Python、Java 或 Scala 编写的。SQL UDF 具有与 SQL 视图相同的方言分歧问题:函数体是 SQL 文本,且适用相同的行为差异(类型转换、空值处理、函数语义)。非 SQL UDF 增加了另一个维度。Python UDF 在各引擎间具有相对广泛的支持(Snowpark、Spark、Trino、DuckDB 都支持它们),尽管注册机制和运行时环境有所不同。但其他语言的可移植性较差。例如,Scala UDF 在 Trino 或 DuckDB 中没有等效项。要使非 SQL UDF 通过 Iceberg 规范实现可移植,规范首先需要支持非 SQL 语言作为表示类型,并且社区需要在引擎间达成通用的调用契约。这是一个比 SQL 方言翻译更棘手的问题,尽管在许多情况下核心函数逻辑本身是可移植的;不可移植的是其周围的连接部分(类型声明、序列化、可用库)。

Pragmatic guidance

Treat UDFs as engine-locked for now. If a UDF encodes critical business logic, that's one of the stronger arguments for routing that workload through a single engine rather than attempting cross-engine portability. Where necessary, physicalize the logic via a pipeline that writes results to a table so all engines inherently get the same answer.目前请将 UDF 视为引擎锁定的。如果 UDF 编码了关键业务逻辑,这是将该工作负载路由通过单一引擎而不是尝试跨引擎可移植性的更有力论据之一。必要时,通过将结果写入表的管道物理化逻辑,以便所有引擎本质上都能得到相同的答案。

Semantic layer and Apache Ossie (incubating)语义层与 Apache Ossie(孵化中)

Before covering Ossie specifically, it's worth disambiguating a term: Semantic layer. Semantic layer means different things to different people. Some use it to mean a metric definitions layer. Some use it to mean a BI modeling layer (LookML, Cube, AtScale). Some use it to mean the full stack of business logic on top of raw tables. For this post, the functional pieces that a "semantic layer" typically refers to are covered individually: View-based transformations are covered in the Iceberg views section above; access policies are covered in the governance deep dive; and metric and dimension definitions are covered below. What matters is the specific interoperability problem for each, not the umbrella term.在专门介绍 Ossie 之前,有必要澄清一个术语:语义层。语义层对不同的人意味着不同的含义。有些人用它来指代指标定义层。有些人用它来指代 BI 建模层(LookML、Cube、AtScale)。有些人用它来指代原始表之上的全栈业务逻辑。对于本文而言,“语义层”通常指代的各个功能部分已单独介绍:基于视图的转换在上面的 Iceberg 视图章节中介绍;访问策略在治理深度解析中介绍;指标和维度定义在下文介绍。重要的是每个部分的具体互操作性问题,而不是这个总称。

Ossie is the most significant initiative addressing semantic layer definition portability. With contributions since its inception from Snowflake, alongside Salesforce, dbt Labs and others in 2025, the working group has grown to include more than 50 organizations: Databricks, Cube, AtScale, Dremio, Starburst, Collibra, Oracle and many others have joined since founding. The breadth of participation signals genuine industry alignment, not a single-vendor initiative.Ossie 是解决语义层定义可移植性最重要的倡议。自 2025 年成立以来,Snowflake 以及 Salesforce、dbt Labs 和其他公司做出了贡献,工作组已发展到包括 50 多个组织:Databricks、Cube、AtScale、Dremio、Starburst、Collibra、Oracle 等许多公司自成立以来已加入。参与的广泛性标志着真正的行业一致,而非单一厂商的倡议。

The spec is live under Apache 2.0 licensing: a declarative YAML-based standard that defines semantic models, data sets, metrics, dimensions, relationships and AI context in a vendor-neutral format. Metric expressions support dialect annotations, so a metric can carry its calculation in ANSI SQL (or other dialects) alongside the semantic metadata. The initiative has a public roadmap with active working groups developing metric semantics (aggregation, grain, cumulative metrics), catalog integration (including an Apache Polaris converter) and ontology/conceptual interoperability. Future roadmap efforts cover a semantic query language and reference engine, and AI-native context for grounded query generation. Converters for Snowflake, dbt, Apache Polaris and Salesforce exist. Snowflake, for one, already offers bidirectional support: the SYSTEM$CREATE_SEMANTIC_VIEW_FROM_OSSIE_YAML stored procedure (in public preview at time of publication) builds a native semantic view from an Ossie model, and the SYSTEM$READ_OSSIE_YAML_FROM_SEMANTIC_VIEW function (public preview at publication time) exports an existing semantic view back to Ossie YAML, so a definition can round-trip between the open spec and a governed engine object. The initiative has already transitioned to foundation-led governance as an incubating project in the Apache Software Foundation.该规范在 Apache 2.0 许可下发布:这是一个基于 YAML 的声明式标准,以厂商中立的格式定义语义模型、数据集、指标、维度、关系和 AI 上下文。指标表达式支持方言注释,因此指标可以在语义元数据旁边携带其 ANSI SQL(或其他方言)计算方式。该倡议有一个公开的路线图,设有积极的工作组来开发指标语义(聚合、粒度、累计指标)、目录集成(包括 Apache Polaris 转换器)以及本体/概念互操作性。未来的路线图工作涵盖语义查询语言和参考引擎,以及用于接地查询生成的 AI 原生上下文。目前已存在针对 Snowflake、dbt、Apache Polaris 和 Salesforce 的转换器。以 Snowflake 为例,它已经提供了双向支持:SYSTEM$CREATE_SEMANTIC_VIEW_FROM_OSSIE_YAML 存储过程(发布时处于公开预览阶段)可从 Ossie 模型构建原生语义视图,而 SYSTEM$READ_OSSIE_YAML_FROM_SEMANTIC_VIEW 函数(发布时处于公开预览阶段)可将现有的语义视图导出回 Ossie YAML,因此定义可以在开放规范和受治理的引擎对象之间往返。该倡议已经转为以基金会为导向的治理,成为 Apache 软件基金会的孵化项目。

What Ossie covers directly addresses the CFO-gets-different-numbers scenario from the current state assessment: If "recognized revenue" is defined once in an Ossie model and consumed by every tool in the stack, the definition doesn't drift. That's the highest-value slice of business logic interoperability.Ossie 所涵盖的内容直接解决了当前状态评估中提到的“CFO 得到不同数字”的场景:如果“已确认收入”在 Ossie 模型中定义一次并被堆栈中的每个工具消费,那么定义就不会发生漂移。这是业务逻辑互操作性中价值最高的部分。

What Ossie does not cover today: UDFs, stored procedures, procedural logic and transformation logic. Metrics are important, arguably the highest-value category, but they're one category. Organizations that adopt Ossie will have metric consistency across tools but still face the drift problem for everything else discussed in this post. The scope limitation is deliberate and appropriate for a first initiative; it's a step in the right direction that should be taken into account when planning your architecture.Ossie 目前未涵盖的内容:UDF、存储过程、过程逻辑和转换逻辑。指标很重要,可以说是价值最高的类别,但它们只是一个类别。采用 Ossie 的组织将在工具间实现指标一致性,但在本文讨论的其他所有内容上仍将面临漂移问题。这种范围限制是审慎且适合作为首个倡议的;这是朝着正确方向迈出的一步,在规划架构时应将其考虑在内。

Pragmatic guidance

Ossie defines the standard, but a standard still has to be implemented and governed somewhere, and that requirement holds, no matter whose tools you use: Metric and dimension definitions need to live as first-class, governed objects, defined once at the source, so an AI agent asked for "ARR" resolves against a single authoritative definition instead of grabbing whichever revenue field it lands on, and every consumer, BI tool or agent, inherits that one definition rather than reimplementing it. You can build this yourself, but it means standing up and maintaining a governed definition store that every engine and tool resolves against, and keeping it in sync as definitions change. If you'd rather not, Snowflake provides this capability out of the box through Horizon Context. Definitions live as native, governed semantic objects and because Snowflake semantic views are easily round-tripped to and from Ossie YAML (see stored procedure and function above), adopting it now doesn't wall you off from the open standard as it matures.Ossie 定义了标准,但标准仍然需要在某个地方实现和治理,无论你使用谁的工具,这一要求都成立:指标和维度定义需要作为一等公民的受治理对象,在源头定义一次,以便 AI 代理询问“ARR”时能根据单一权威定义进行解析,而不是抓取它碰巧落入的任何收入字段;并且每个消费者、BI 工具或代理都继承该定义,而不是重新实现它。你可以自己构建,但这意味着要建立并维护一个受治理的定义存储,每个引擎和工具都据此解析,并在定义变更时保持同步。如果你不想这样做,Snowflake 通过 Horizon Context 开箱即用地提供了此功能。定义作为原生的、受治理的语义对象存在,并且由于 Snowflake 语义视图可以轻松地与 Ossie YAML 进行往返转换(见上文存储过程和函数),因此现在采用它并不会在你需要时将你与成熟的开放标准隔绝。

For organizations operating across multiple engines or prioritizing cross-vendor portability long-term, Apache Ossie (incubating) is the most promising path and worth tracking closely given the breadth of industry participation.对于在多个引擎上运营或长期优先考虑跨厂商可移植性的组织,Apache Ossie(孵化中)是最有前途的路径,鉴于行业参与的广泛性,值得密切关注。

In the meantime, the pragmatic approach remains the same as the broader guidance in this post: Centralize authoritative metric definitions in a single engine or semantic layer, and treat other consumers as downstream, or physicalize the logic. dbt and SQLMesh can also help here by managing metric definitions as code and generating per-engine SQL from a single source, though they're limited to the transformations they manage and don't solve behavioral divergence. For organizations evaluating semantic layer platforms (Cube, AtScale and others in the Ossie working group), watch for native Ossie spec support as a signal of long-term portability.与此同时,务实的方法与本文更广泛的指导保持一致:将权威指标定义集中在单一引擎或语义层中,并将其他消费者视为下游,或者将逻辑物理化。dbt 和 SQLMesh 也可以通过将指标定义作为代码进行管理并从单一源生成每个引擎的 SQL 来提供帮助,尽管它们仅限于其管理的转换,且无法解决行为分歧。对于正在评估语义层平台(Cube、AtScale 以及 Ossie 工作组中的其他平台)的组织,请关注对原生 Ossie 规范的支持,以此作为长期可移植性的信号。

Open problems and active work开放问题与正在进行的工作

The previous section covered what exists today and where workarounds are needed. This section covers the problems that don't have shipped solutions yet: Some have active community work with real momentum (transpilation, intermediate representations); some are recognized gaps with no clear path forward (stored procedures); and some are lower-priority known issues that will likely be the last to get addressed (the remaining long tail).前一节涵盖了当前存在的方案以及需要变通的地方。本节涵盖尚未发布解决方案的问题:有些方案有积极的社区工作且势头强劲(转译、中间表示);有些是公认的差距且没有明确的前进路径(存储过程);还有一些是优先级较低的已知问题,很可能是最后得到解决的部分(剩余的长尾问题)。

Transpilation and intermediate representations转译与中间表示

The long-term solution to SQL behavioral divergence is likely some form of canonical representation. Instead of maintaining N copies of the same logic in N dialects, store one version and adapt it at query time or build time. Two approaches are being pursued: syntactic transpilation (translate SQL text between dialects) and intermediate representations (store logic in an engine-agnostic format that gets compiled to engine-specific SQL).SQL 行为分歧的长期解决方案很可能是某种形式的规范化表示。与其在 N 个方言中维护 N 份相同逻辑的副本,不如存储一个版本并在查询时或构建时进行适配。目前正在探索两种方法:语法转译(在方言之间翻译 SQL 文本)和中间表示(以与引擎无关的格式存储逻辑,然后编译为特定于引擎的 SQL)。

Figure 2: A visual representation of the concept of the single intermediate representation along with adapters for engines.
Figure 2: A visual representation of the concept of the single intermediate representation along with adapters for engines.

 

sqlglot is an open source Python SQL parser and transpiler with support for 30+ dialects. It handles function name differences, quoting conventions, type mappings and many syntax transformations, and it emits warnings rather than silently producing wrong SQL when it can't translate cleanly. sqlglot goes beyond syntax in several important cases:sqlglot 是一个开源的 Python SQL 解析器和转译器,支持 30 多种方言。它处理函数名称差异、引用约定、类型映射和许多语法转换,并且在无法干净地翻译时会发出警告,而不是静默产生错误的 SQL。sqlglot 在几个重要案例中超越了语法层面:

  • Transpiling ORDER BY col ASC from Spark to DuckDB correctly adds an explicit NULLS FIRST clause to preserve Spark's null ordering default将 ORDER BY col ASC 从 Spark 转译到 DuckDB 时,正确添加了显式的 NULLS FIRST 子句,以保留 Spark 的空值排序默认值
  • CAST('a' AS INT) from Spark (where invalid casts return null by default) becomes TRY_CAST on DuckDB (where CAST throws)来自 Spark 的 CAST('a' AS INT)(默认情况下无效转换返回 null)在 DuckDB(CAST 会抛出错误)上变为 TRY_CAST
  • CONCAT calls get rewritten to preserve each engine's null propagation behaviorCONCAT 调用被重写以保留每个引擎的空值传播行为

These are genuine semantic translations, not just keyword swaps. That said, there are categories of divergence that no syntactic transpiler can address: decimal precision behavior (the same division on the same types produces different precision across engines), division-by-zero semantics (null vs. infinity vs. error) and implicit-type coercion rules that are embedded in engine runtimes rather than expressed in SQL text. For migrations, build-time SQL generation and reducing the manual work needed to stand up multi-dialect pipelines, sqlglot is a useful tool in that space. But it doesn't eliminate the need to test cross-engine equivalence for critical business logic. For high-stakes business logic, transpilation is a tool on the path to a good architecture, not a substitute for one. Authoritative definitions still need to live somewhere deliberate: a single engine or physicalized as tables.这些是真正的语义翻译,而不仅仅是关键字交换。话虽如此,仍有几类分歧是任何语法转译器都无法解决的:十进制精度行为(相同类型上的相同除法在不同引擎上产生不同的精度)、除以零的语义(null 与无穷大与错误)以及嵌入在引擎运行时而非 SQL 文本中表达的隐式类型强制转换规则。对于迁移、构建时 SQL 生成以及减少建立多方言管道所需的手动工作,sqlglot 是该领域的一个有用工具。但它并不能消除为关键业务逻辑测试跨引擎等效性的需求。对于高风险业务逻辑,转译是通往良好架构路径上的一个工具,而不是替代品。权威定义仍然需要存放在某个审慎的地方:单一引擎或物理化为表。

LinkedIn's Coral takes a different approach. It is a Java-based SQL translation and rewrite engine that uses an intermediate representation (based on Apache Calcite's relational algebra) to translate between HiveQL, Spark SQL and Trino dialects. LinkedIn operates it in production across its data platform, demonstrating that IR-based translation works at large data volumes for a specific set of dialects, particularly when maintained by a large engineering organization. The limitation, however, is that the supported dialect set is narrow, and extending it requires deep familiarity with Calcite's framework. While Coral is open source, it's not a turnkey solution for most teams.LinkedIn 的 Coral 采取了不同的方法。它是一个基于 Java 的 SQL 翻译和重写引擎,使用中间表示(基于 Apache Calcite 的关系代数)在 HiveQL、Spark SQL 和 Trino 方言之间进行翻译。LinkedIn 在其整个数据平台的生产环境中运行它,证明了基于 IR 的翻译在特定方言集的大数据量下是有效的,特别是当由大型工程组织维护时。然而,局限性在于支持的方言集较窄,且扩展它需要深入了解 Calcite 的框架。虽然 Coral 是开源的,但对于大多数团队来说,它不是开箱即用的解决方案。

The Substrait project is working toward a broader version of this concept: a cross-language specification for serializing relational algebra that any engine could produce or consume. If business logic were stored as Substrait plans rather than SQL text, dialect differences would be handled by engine-specific adapters rather than text translation. In practice, adoption has been uneven. Apache Arrow's Acero and DataFusion (as a producer and consumer), and Ibis (as a producer) have native Substrait support, but the major query engines that dominate Iceberg lakehouse deployments don't have native Substrait integration. Until the engines you actually use support it, Substrait is a promising architecture rather than a production dependency.Substrait 项目正致力于实现此概念的更广泛版本:一种用于序列化关系代数的跨语言规范,任何引擎都可以生产或消费。如果业务逻辑存储为 Substrait 计划而不是 SQL 文本,方言差异将由特定于引擎的适配器处理,而不是通过文本翻译。在实践中,采用率一直不均衡。Apache Arrow 的 Acero 和 DataFusion(作为生产者和消费者)以及 Ibis(作为生产者)具有原生 Substrait 支持,但主导 Iceberg 湖仓部署的主要查询引擎没有原生的 Substrait 集成。在你实际使用的引擎支持它之前,Substrait 是一种有前途的架构,而不是生产依赖。

Within Iceberg itself, the recently merged expressions spec (Iceberg PR #16652) is a narrower take on the same idea. Instead of raw SQL text, it represents expressions as standardized expression signatures. The structure is engine-agnostic, but the spec deliberately stops there. It doesn't say how the defined expressions actually behave, leaving that to each engine. It standardizes the shape of an expression, not its implementation behavior. That makes it foundational plumbing (scalar expressions, not full queries) that view and UDF definitions could eventually build on, but not a fix for the behavioral divergence described above.在 Iceberg 内部,最近合并的表达式规范(Iceberg PR #16652)是同一概念的较窄版本。它不使用原始 SQL 文本,而是将表达式表示为标准化的表达式签名。该结构与引擎无关,但规范刻意止步于此。它没有说明定义的表达式实际上如何表现,而是留给每个引擎去处理。它标准化了表达式的形状,而不是其实现行为。这使其成为基础性的管道(标量表达式,而非完整查询),视图和 UDF 定义最终可以基于此构建,但它并不能修复上述行为分歧。

A fourth, semantic-layer-level variant is on Ossie's roadmap: a reference engine plus an Ossie-to-SQL compiler and a cross-implementation conformance suite, so a single semantic model definition could be compiled to each engine's dialect with consistent interpretation. It is scoped to Ossie-modeled semantic queries (not arbitrary SQL, views or UDFs) and is on the roadmap rather than shipped, but it applies the same canonical-representation idea at the semantic layer.Ossie 的路线图上还有第四种语义层级别的变体:参考引擎加上 Ossie 到 SQL 的编译器和跨实现的一致性套件,这样单个语义模型定义就可以以一致的解释编译为每个引擎的方言。它的范围仅限于 Ossie 建模的语义查询(而非任意 SQL、视图或 UDF),且处于路线图阶段而非已发布,但它在语义层应用了相同的规范表示概念。

There's also an accountability problem that applies to all transpilation and IR approaches. In a single-engine world, when a query produces the wrong answer, there's one vendor to call. In a multi-engine world with a translation layer in between, a wrong result could be the fault of the source engine, the target engine or the translation layer. Debugging cross-engine result differences becomes a cross-vendor support problem: slow to resolve, ambiguous in ownership and costly when it surfaces in a downstream business decision. The governance deep dive covers why this accountability gap is particularly consequential for security policies.此外还有一个适用于所有转译和 IR 方法的问责问题。在单一引擎世界中,当查询产生错误答案时,只有一个供应商可以联系。在中间有翻译层的多引擎世界中,错误的结果可能是源引擎、目标引擎或翻译层的过错。调试跨引擎结果差异成为跨供应商支持问题:解决缓慢、责任模糊,且在下游业务决策中出现时代价高昂。治理深度解析涵盖了为什么这种问责差距对安全策略尤为重要。

Pragmatic guidance

sqlglot covers more ground than most teams expect: Use it for migrations, build-time SQL generation and as a first pass when standing up multi-dialect pipelines. For the divergences it handles (null ordering, type casting, null propagation in functions), it genuinely reduces the manual work. For the divergences it can't (precision, division-by-zero, implicit coercion), cross-engine regression tests on representative data are the backstop. Watch Substrait adoption across the specific engines in your architecture; it's a sound long-term concept but not yet a dependency you can rely on. For critical business logic, the architectural guidance from the current state assessment still applies: Centralize authoritative definitions in a single engine or physicalize the results rather than depending entirely on a translation layer.sqlglot 涵盖的范围比大多数团队预期的要广:将其用于迁移、构建时 SQL 生成以及作为建立多方言管道时的第一步。对于它能处理的分歧(空值排序、类型转换、函数中的空值传播),它确实减少了手动工作。对于它无法处理的分歧(精度、除以零、隐式强制转换),在代表性数据上进行的跨引擎回归测试是后盾。关注你架构中特定引擎对 Substrait 的采用情况;这是一个合理的长期概念,但还不是你可以依赖的生产依赖。对于关键业务逻辑,来自当前状态评估的架构指导依然适用:将权威定义集中在单一引擎中,或物理化结果,而不是完全依赖翻译层。

Stored procedures and procedural logic存储过程与过程逻辑

Unlike views, UDFs and metrics, stored procedures don't have an Iceberg spec equivalent, a community proposal or even a widely discussed path forward. This isn't an oversight; it's a reflection of how deeply engine-coupled stored procedures are. Views and UDFs are at least SQL expressions that can be stored and potentially translated. Stored procedures are full programs: loops, conditionals, exception handling, transaction control, cursor management, dynamic SQL and calls to engine-specific system functions, all written in engine-proprietary procedural languages (PL/pgSQL, Snowflake Scripting, Spark's programmatic APIs). There's no shared abstraction because there's no shared surface area to abstract over.与视图、UDF 和指标不同,存储过程没有 Iceberg 规范等效项、社区提案,甚至没有被广泛讨论的前进路径。这并非疏忽;它反映了存储过程与引擎的耦合程度有多深。视图和 UDF 至少是可以存储并可能被翻译的 SQL 表达式。存储过程是完整的程序:循环、条件判断、异常处理、事务控制、游标管理、动态 SQL 以及对特定引擎系统函数的调用,全部用引擎专有的过程语言(PL/pgSQL、Snowflake Scripting、Spark 的编程 API)编写。没有共享抽象,因为没有可供抽象的共享表面。

In a single-engine world, stored procedures are where critical multi-step business workflows live: end-of-month financial close processes, data quality enforcement routines, complex ETL orchestration, conditional transformation logic that can't be expressed as a single SQL statement. In a multi-engine world, these are entirely engine-locked with no portability path. Transpilation tools like sqlglot operate on SQL expressions, not procedural control flow. Intermediate representations (IRs), like Substrait, model relational algebra, not imperative logic. Even if every other category of business logic described in this post achieved full cross-engine portability tomorrow, stored procedures would remain tied to the engine they were written for.在单一引擎世界中,存储过程是关键多步业务工作流的所在地:月底财务结算流程、数据质量强制执行例程、复杂的 ETL 编排、无法表示为单个 SQL 语句的条件转换逻辑。在多引擎世界中,这些完全被锁定在引擎中,没有可移植性路径。sqlglot 等转译工具操作的是 SQL 表达式,而不是过程控制流。像 Substrait 这样的中间表示(IR)建模的是关系代数,而不是指令式逻辑。即使本文描述的其他所有类别的业务逻辑明天都实现了完全的跨引擎可移植性,存储过程仍将与编写它们的引擎绑定。

Stored procedures are also where complex governance logic sometimes lives: multi-step anonymization workflows, conditional access decisions that go beyond what a single-row filter expression can handle, audit logging routines, to name a few. When that logic is engine-locked, the compliance implications extend beyond analytics consistency into security posture; the governance deep dive covers this in detail.存储过程有时也是复杂治理逻辑的所在地:多步匿名化工作流、超出单行过滤器表达式处理能力的条件访问决策、审计日志例程等等。当逻辑被锁定在引擎中时,合规性影响会超出分析一致性,延伸到安全态势;治理深度解析对此进行了详细介绍。

Pragmatic guidance

Accept that stored procedures are engine-locked and plan accordingly. If a critical business workflow lives in a stored procedure, that's one of the strongest arguments for routing that workload through a single engine. Where possible, refactor procedural logic into declarative SQL (which has a portability path, even if an imperfect one) or externalize the orchestration layer to tools like Airflow, Dagster or Prefect, which coordinate workflow steps without depending on engine-specific procedural languages.接受存储过程被引擎锁定这一事实,并据此进行规划。如果关键业务工作流存在于存储过程中,这是将该工作负载路由通过单一引擎的更有力论据之一。尽可能将过程逻辑重构为声明式 SQL(它有一条可移植性路径,即使是不完美的路径),或者将编排层外包给 Airflow、Dagster 或 Prefect 等工具,这些工具可以在不依赖特定引擎过程语言的情况下协调工作流步骤。

The remaining long tail剩余的长尾问题

Even if views, UDFs, metrics and stored procedures all had cross-engine solutions tomorrow, a set of lower-priority business logic categories would remain engine-specific. Triggers and event-driven logic have no portability mechanism and no community proposal addressing them. Scheduled tasks and job orchestration defined inside an engine (as opposed to external orchestrators like Airflow) are tied to that engine's scheduler semantics. Materialized view refresh logic and maintenance schedules are engine-specific by definition. Session state and context functions used in business logic (CURRENT_USER(), CURRENT_ROLE(), session variables) return engine-specific values with no cross-engine equivalence guarantee, which matters when those functions appear in access control policies or audit logic. None of these are typically the source of the "two dashboards, different numbers" problem, which is why they're lower priority than the categories covered earlier. But they are worth acknowledging: Any architecture assessment that only accounts for views, UDFs and metrics will undercount the total surface area of engine-coupled business logic.即使视图、UDF、指标和存储过程明天都有了跨引擎解决方案,仍有一组优先级较低的业务逻辑类别将保持特定于引擎的状态。触发器和事件驱动逻辑没有可移植性机制,也没有社区提案解决它们。在引擎内部定义的计划任务和作业编排(相对于 Airflow 等外部编排器)与该引擎的调度程序语义绑定。物化视图刷新逻辑和维护计划在定义上就是特定于引擎的。业务逻辑中使用的会话状态和上下文函数(CURRENT_USER()、CURRENT_ROLE()、会话变量)返回特定于引擎的值,且没有跨引擎等效性保证,当这些函数出现在访问控制策略或审计逻辑中时,这很重要。这些通常都不是导致“两个仪表板,不同数字”问题的根源,这就是为什么它们比前面介绍的类别优先级低的原因。但值得承认的是:任何只考虑视图、UDF 和指标的架构评估,都会低估引擎耦合业务逻辑的总表面积。

Diagnostic questions诊断问题

If you're evaluating or already operating a multi-engine lakehouse, these are the questions worth asking before you're in too deep and discover the answers the hard way:如果你正在评估或已经运营多引擎湖仓,在陷入太深并以惨痛方式发现答案之前,值得问自己这些问题:

  1. For any business logic you can't physicalize as an output table (active users, recognized revenue, churn), how do you detect when definitions diverge across engines, and what's your reconciliation process?对于任何无法物理化为输出表的业务逻辑(活跃用户、已确认收入、流失率),你如何检测定义何时在引擎间发生分歧,以及你的对账流程是什么?
  2. Which engine's business logic definition is the explicit, documented system of record for high-stakes metrics (finance, regulatory reporting, executive dashboards) — and is that decision enforced architecturally or just assumed?哪个引擎的业务逻辑定义是高风险指标(财务、监管报告、执行仪表板)的明确、已记录的记录系统——这一决策是通过架构强制执行的,还是仅仅是假设的?
  3. When two engines produce different results from the same Iceberg table, what is your escalation path? If neither engine vendor can reproduce the issue independently and a translation layer sits in between, have you mapped that cross-vendor accountability chain before you're in production — and do you know who owns the debugging when an Iceberg view produces a wrong result due to dialect differences?当两个引擎从同一个 Iceberg 表产生不同结果时,你的升级路径是什么?如果两个引擎厂商都无法独立重现该问题,且中间存在翻译层,你是否在投入生产前就映射了这种跨厂商问责链——并且你知道当 Iceberg 视图因方言差异产生错误结果时,谁负责调试吗?
  4. When you need to update a business logic definition (for example, how "churn" or "MRR" is calculated), what is your process for deploying that change consistently across all engines at the same time — and what's your rollback plan if one engine fails validation after another has already gone live?当你需要更新业务逻辑定义(例如,“流失”或“MRR”是如何计算的)时,你如何在所有引擎中同时一致地部署该变更的流程是什么——如果一个引擎在另一个引擎已经上线后验证失败,你的回滚计划是什么?
  5. Have you audited the SQL behavioral differences between your engines, including null ordering, type casting behavior (ANSI mode, TRY_CAST availability) and timestamp resolution?你是否审计了引擎之间的 SQL 行为差异,包括空值排序、类型转换行为(ANSI 模式、TRY_CAST 可用性)和时间戳解析?
  6. How does each engine in your architecture handle timestamp storage, timezone resolution and DST transitions — and have you specifically tested daily and weekly aggregations that cross a DST boundary on a global data set?你的架构中每个引擎如何处理时间戳存储、时区解析和 DST 过渡——你是否专门测试过在全球数据集上跨越 DST 边界的每日和每周聚合?

Conclusion结论

Business logic interoperability is the least mature of the three pillars, but it's not a dead end. Most query volume in a typical data platform hits tables directly, where data interoperability is sufficient; the gaps described in this post affect a smaller but disproportionately high-stakes slice of workloads. The building blocks are in progress: Iceberg views and UDFs provide catalog-level metadata containers, Apache Ossie (incubating) is converging on metric definition portability with broad industry participation, and transpilation tools cover a good amount of ground.业务逻辑互操作性是这三大支柱中最不成熟的一项,但它并非死胡同。在典型的数据平台中,大多数查询量直接命中表,此时数据互操作性已足够;本文所述的差距影响的是一小部分工作负载,但这些工作负载的重要性却不成比例地高。相关构建模块正在推进中:Iceberg 视图和 UDF 提供了目录级的元数据容器,Apache Ossie(孵化中)正通过广泛的行业参与在指标定义的可移植性上达成共识,而转译工具也已覆盖了相当多的场景。

The problem is that none of these are sufficient on their own today, and the pragmatic path remains architectural: Physicalize high-stakes logic as tables; keep authoritative logic definitions in a single engine, where the cons of physicalization outweigh the pros; and treat cross-engine logic consistency as something you verify rather than assume. The data is portable, but the business logic isn't yet. Treat that as a known constraint, and architect around it deliberately.问题在于,目前这些方案单独使用都不够充分,务实的路径依然在于架构设计:将高风险逻辑物化为表;在物理化弊大于利的情况下,将权威的逻辑定义保留在单一引擎中;并将跨引擎逻辑一致性视为一种需要验证而非预设的特性。数据是可移植的,但业务逻辑尚不可移植。请将其视为一项已知约束,并围绕它进行审慎的架构设计。

To learn more, check out An Architect's Guide to Multi-Engine Lakehouses: What's Solved and What Isn't, which is a brief summary of all three pillars, and the other deep investigations into data interoperability and governance interoperability.欲了解更多信息,请参阅《架构师的多引擎湖仓指南:已解决与未解决的问题》(An Architect's Guide to Multi-Engine Lakehouses: What's Solved and What Isn't),其中简要总结了上述三大支柱,并包含了关于数据互操作性和治理互操作性的其他深度研究。

Share this post

Subscribe to our blog newsletter

Get the best, coolest and latest delivered to your inbox each week

Where Data Does More