Introduction: The evolution of Grab’s Data Lake引言：Grab 数据湖的演进

At Grab’s scale, managing petabytes of data across billions of S3 objects demands more than a storage layer. It demands a robust architectural primitive that supports the high-concurrency needs of a modern “Lakehouse.” Our goal is full storage-compute separation, leveraging S3 as an elastic foundation for both near-real-time metrics and large-scale batch transformations.在 Grab 的规模下，管理跨越数十亿个 S3 对象的 PB 级数据，需要的不仅仅是一个存储层。我们需要一个强大的架构原语，以支持现代“湖仓一体”（Lakehouse）的高并发需求。我们的目标是实现存储与计算的完全分离，利用 S3 作为近实时指标处理和大规模批处理转换的弹性基础。

For years, the vast majority of our tables were Hive Parquet, managed through the Hive Metastore with a directory-based layout. This model served us well, but as data volume grew, the directory-and-metastore approach became the limiting factor. We are now transitioning to a table-centric architecture built on modern table formats, treating data as a first-class primitive to ensure consistency and performance across our internal data transformation platforms: Slide, which powers batch transformations, and Hugo, which handles online-to-data-lake ingestion. Along the way, we also built the UnifiedSparkCatalog, a unified Spark catalog that hides table-format differences from users entirely, which we are open-sourcing alongside this post.多年来，我们绝大多数表都是 Hive Parquet 格式，并通过带有目录布局的 Hive Metastore 进行管理。这种模式曾非常有效，但随着数据量的增长，基于目录和元数据存储的方法成了瓶颈。我们目前正在转向基于现代表格式的以表为中心的架构，将数据视为一等公民原语，以确保我们内部数据转换平台的一致性和性能。这些平台包括：负责批处理转换的 Slide，以及处理从线上到数据湖摄取的 Hugo。在此过程中，我们还构建了 UnifiedSparkCatalog，这是一个统一的 Spark 目录，能够对用户完全屏蔽表格式的差异。我们将随本文一同开源该项目。

The catalyst for change: Challenges with Hive Parquet变革的催化剂：Hive Parquet 面临的挑战

For years, Hive Parquet was the backbone of our Data Lake, representing the vast majority of our tables. However, as data volume scaled, the architectural limitations of directory-based storage became apparent. We identified four primary bottlenecks:多年来，Hive Parquet 一直是我们数据湖的基石，占据了我们绝大多数的表。然而，随着数据规模的扩展，基于目录的存储架构局限性日益凸显。我们发现了四个主要瓶颈：

  • Catalog latency: The Hive Metastore (HMS) became a centralized failure point. High concurrency during metadata access led to O(n) listing overhead, where query planning time scaled linearly with partition count, crippling throughput.目录延迟：Hive Metastore (HMS) 成为了单点故障。元数据访问期间的高并发导致了 O(n) 的列表开销，查询规划时间随分区数量线性增加，严重制约了吞吐量。
  • The small file problem: The directory layout left us with severe file fragmentation. Certain Machine Learning (ML) datasets had an average file size under 1 MB, with thousands of files in each partition. At this scale, the overhead of S3 object listing and metadata request latency drove up Application Programming Interface (API) costs and slowed scan operations.小文件问题：目录布局导致了严重的文件碎片化。某些机器学习 (ML) 数据集的平均文件大小不到 1 MB，每个分区中有数千个文件。在这种规模下，S3 对象列表的开销和元数据请求延迟推高了应用程序编程接口 (API) 的成本，并拖慢了扫描操作。
  • Operational toil: Data engineers faced constant manual overhead for partition registration. Without native ACID support (no native UPSERT or DELETE), teams relied on complex workarounds to manage data changes carefully.运维繁琐：数据工程师面临着持续的手动分区注册开销。由于缺乏原生的 ACID 支持（没有原生的 UPSERT 或 DELETE），团队不得不依赖复杂的变通方法来谨慎管理数据变更。
  • The broken information loop: A fundamental disconnect existed between the catalog and storage. Because the HMS, not the storage layer, was treated as the source of truth, direct S3 modifications frequently left the catalog stale and out of sync with the actual state on disk.信息回路断裂：目录与存储之间存在根本性的脱节。由于 HMS（而非存储层）被视为事实来源，直接对 S3 进行的修改经常导致目录过时，与磁盘上的实际状态不同步。

Why Iceberg? Strategic alignment and future-proofing为什么选择 Iceberg？战略协同与面向未来

We evaluated several open table formats before selecting Apache Iceberg as our default. The deciding factors came down to community governance, engine compatibility, and long-term flexibility.在选择 Apache Iceberg 作为默认格式之前，我们评估了多种开放表格式。最终的决定因素在于社区治理、引擎兼容性以及长期的灵活性。

Recent industry momentum, including growing cloud-native support for Iceberg, further validates this direction. We are positioning Grab to be format-agnostic in the long term, but Iceberg provides the most mature foundation today.近期行业的发展势头，包括云原生对 Iceberg 日益增长的支持，进一步验证了这一方向。我们正将 Grab 定位为长期格式无关，但 Iceberg 提供了目前最成熟的基础。

Comparison of Legacy Hive Parquet and Apache Iceberg

Adopting Iceberg at scale大规模采用 Iceberg

Migrating an established lake is not a flag flip. Our challenge was rolling out Iceberg across a lake that was overwhelmingly Hive Parquet, queried by many engines and teams, without breaking the downstream consumers that depended on those tables. Rather than converting everything at once, we moved the highest-value tables first. The efficiency gains across our production workloads have been substantial. Here are representative examples:迁移一个成熟的数据湖并非一蹴而就。我们面临的挑战是在一个绝大多数仍是 Hive Parquet 格式、且被众多引擎和团队查询的数据湖中推广 Iceberg，同时不能破坏依赖这些表的下游消费者。我们没有选择一次性全部转换，而是优先迁移了高价值表。生产工作负载的效率提升非常显著，以下是几个典型案例：

  • Query performance via Z-ordering: On a high-traffic navigation dataset, we achieved roughly a 10x improvement in query runtime. Z-ordering co-locates rows with similar values across specified dimensions, enabling Trino to leverage data skipping and min/max statistics to prune irrelevant files during query planning. This reduced query runtime from 70 seconds to 6 seconds.通过 Z-ordering 提升查询性能：在一个高流量的导航数据集上，我们将查询运行时间缩短了约 10 倍。Z-ordering 将跨指定维度的相似值行存放在一起，使 Trino 能够在查询规划期间利用数据跳过（data skipping）和最小/最大统计信息来过滤掉无关文件。这使查询运行时间从 70 秒缩短到了 6 秒。
  • S3 API cost reduction: For a heavily queried operations table, daily S3 API costs were reduced by up to 95% with no changes to the queries themselves. Larger file sizes and the elimination of expensive object listing during query planning drove most of the savings.降低 S3 API 成本：对于一个被频繁查询的运营表，在不更改任何查询的情况下，每日 S3 API 成本降低了高达 95%。文件大小的增加以及查询规划期间昂贵的对象列表操作的消除，带来了大部分成本节省。
  • Compute savings: For a dataset used in funnel analysis, we reduced cluster resource usage by approximately half. A separate ML feature pipeline also improved feature freshness for downstream models.计算资源节省：对于一个用于漏斗分析的数据集，我们将集群资源使用量减少了约一半。一个独立的机器学习特征流水线也提升了下游模型的特征时效性。

The UnifiedSparkCatalog: Making mixed formats transparentUnifiedSparkCatalog：让混合格式变得透明

Migrating to Iceberg solved our storage and metadata problems, but it surfaced a new one at the developer-experience layer. Modern table formats like Delta, Iceberg, and Hudi each implement their own custom catalog that extends Spark’s SessionCatalog. In a standard Spark runtime, only one catalog implementation can be set as the default spark_catalog. Supporting additional formats requires explicit catalog declarations, meaning users must reference tables with format-specific prefixes like iceberg_catalog.schema.table or delta_catalog.schema.table.迁移到 Iceberg 解决了我们的存储和元数据问题，但在开发者体验层面又引出了新问题。Delta、Iceberg 和 Hudi 等现代表格式各自实现了扩展 Spark SessionCatalog 的自定义目录。在标准的 Spark 运行时中，只能将一个目录实现设置为默认的 spark_catalog。支持其他格式需要显式声明目录，这意味着用户必须使用特定的格式前缀来引用表，例如 iceberg_catalog.schema.table 或 delta_catalog.schema.table。

With Iceberg, Delta, Hudi, and Hive tables now coexisting and tables actively migrating between formats, this created two problems: engineers had to know the underlying format of every table they queried, and any format migration silently broke every downstream query that hardcoded a prefix.随着 Iceberg、Delta、Hudi 和 Hive 表的共存，以及表在不同格式间的频繁迁移，这导致了两个问题：工程师必须了解所查询的每一张表的底层格式，且任何格式迁移都会静默破坏所有硬编码了前缀的下游查询。

The UnifiedSparkCatalog is our answer. It is a unified Spark catalog that abstracts the complexity of working with mixed table formats so users never need to think about which format a table uses. We took inspiration from Trino’s Table Redirection, a feature that transparently points a query at the right connector when a table’s format differs from the catalog it was queried through. Our Spark equivalent works as follows:UnifiedSparkCatalog 是我们的解决方案。它是一个统一的 Spark 目录，抽象了处理混合表格式的复杂性，因此用户无需考虑表所使用的具体格式。我们借鉴了 Trino 的表重定向（Table Redirection）功能，该功能在表的格式与查询目录不匹配时，能透明地将查询指向正确的连接器。我们 Spark 版本的实现原理如下：

How it works工作原理

  1. Table detection: The catalog loads metadata from the Hive Metastore.表检测：目录从 Hive Metastore 加载元数据。
  2. Format identification: A TableTypeDetector utility identifies the format based on metadata properties (e.g., the provider field) or path-based inference.格式识别：TableTypeDetector 工具根据元数据属性（例如 provider 字段）或基于路径的推断来识别格式。
  3. Operation routing: The catalog delegates the operation to the correct format-specific catalog (Iceberg’s SparkCatalog, Delta’s DeltaCatalog, etc.) without requiring any prefix from the user.操作路由：目录将操作委托给正确的特定格式目录（如 Iceberg 的 SparkCatalog、Delta 的 DeltaCatalog 等），而无需用户添加任何前缀。

Key design decisions关键设计决策

  • Lazy initialization: Catalogs for each format are initialized only when first needed, reducing startup overhead. If a format’s JAR is missing from the classpath, initialization continues gracefully. The catalog simply skips that format rather than failing the entire session.延迟初始化：每种格式的目录仅在首次需要时才进行初始化，从而减少启动开销。如果类路径中缺少某种格式的 JAR 包，初始化将优雅地继续——目录只会跳过该格式，而不会导致整个会话失败。
  • Naming as spark_catalog: The catalog reports its name as spark_catalog because Spark treats this name specially for legacy Hive Data Manipulation Language (DML) operations. Many internal Spark code paths check for this exact name to determine whether to use Hive-compatible logic for inserts, updates, and deletes. Using any other name would break legacy Hive table operations.命名为 spark_catalog：该目录将其名称报告为 spark_catalog，因为 Spark 会针对传统的 Hive 数据操作语言 (DML) 对该名称进行特殊处理。许多内部 Spark 代码路径会检查此确切名称，以决定是否对插入、更新和删除操作使用 Hive 兼容逻辑。使用任何其他名称都会破坏传统的 Hive 表操作。
  • Catalog reuse: Before creating a new catalog instance, the system checks whether one already exists in Spark’s catalog manager. This preserves compatibility with plugins like OpenLineage, which inspect catalog class types for lineage extraction.目录重用：在创建新的目录实例之前，系统会检查 Spark 的目录管理器中是否已存在该实例。这保持了与 OpenLineage 等插件的兼容性，因为这些插件会检查目录类类型以进行血缘提取。
  • Fallback behavior: If a table is not found in the expected format-specific catalog, the system falls back to the base session catalog, ensuring robust behavior for standard Hive tables.回退行为：如果未在预期的特定格式目录中找到表，系统会回退到基础会话目录，从而确保标准 Hive 表的稳健运行。

We are open-sourcing UnifiedSparkCatalog alongside this blog post. The code and documentation are available here.我们随本文一同开源了 UnifiedSparkCatalog。代码和文档可在此处获取。

Lessons learned and overcoming hurdles经验教训与克服障碍

Scaling Iceberg across a large ecosystem revealed several technical nuances:在大型生态系统中扩展 Iceberg 揭示了几个技术细节：

  • Hive lock contention: We encountered “zombie locks” in the HMS that blocked commits. We traced this to a low read timeout on the metastore side under high load. Adjusting retry intervals and increasing the timeout resolved the issue.Hive 锁竞争：我们在 HMS 中遇到了阻塞提交的“僵尸锁”。经排查，这是由于高负载下元数据存储端的读取超时时间过短所致。调整重试间隔并增加超时时间解决了该问题。
  • Timestamp handling: Spark 3.4 introduced TIMESTAMP_NTZ (no time zone), while Iceberg defaults to TIMESTAMP_LTZ (local time zone). This caused compatibility issues with legacy Hive views. We resolved it through a custom migration workflow and targeted patches to our Trino deployment to ensure consistent casting.时间戳处理：Spark 3.4 引入了 TIMESTAMP_NTZ（无时区），而 Iceberg 默认使用 TIMESTAMP_LTZ（本地时区）。这导致了与旧版 Hive 视图的兼容性问题。我们通过自定义迁移工作流和针对 Trino 部署的定向补丁，确保了转换的一致性。
  • Storage tier costs: Generating Iceberg metadata involves reading historical data, which can trigger a one-time cost spike as files move between S3 storage tiers. To manage this, we prioritize migrations based on a table’s scan frequency and API operation costs rather than migrating the entire lake at once.存储层成本：生成 Iceberg 元数据涉及读取历史数据，这可能会在文件在 S3 存储层之间移动时触发一次性成本激增。为了管理这一点，我们根据表的扫描频率和 API 操作成本来确定迁移优先级，而不是一次性迁移整个数据湖。

Conclusion: The road ahead结论：未来之路

Apache Iceberg is now foundational to Grab’s data strategy. It is the default format for Slide and Hugo, and adoption is expanding across our compute platforms.Apache Iceberg 现已成为 Grab 数据战略的基石。它是 Slide 和 Hugo 的默认格式，且其采用范围正在我们的计算平台中不断扩大。

Looking forward, we are experimenting with Storage Partitioned Joins to eliminate shuffle stages in Spark and monitoring the Apache XTable project to maintain interoperability between formats. Our journey does not end with adoption. We will continue contributing back to the ecosystem, starting with the upcoming release of the UnifiedSparkCatalog.展望未来，我们正在尝试使用存储分区连接（Storage Partitioned Joins）来消除 Spark 中的 Shuffle 阶段，并密切关注 Apache XTable 项目以保持格式间的互操作性。我们的旅程并未因采用而结束。我们将继续回馈生态系统，首先就是即将发布的 UnifiedSparkCatalog。

Acknowledgments: This journey was made possible by the dedicated efforts of the Data Engineering, Infrastructure, and Search & Personalization teams at Grab.致谢：这一旅程离不开 Grab 数据工程、基础设施以及搜索与个性化团队的辛勤付出。

Join us加入我们

Grab is Southeast Asia’s leading superapp, serving over 900 cities across eight countries (Cambodia, Indonesia, Malaysia, Myanmar, the Philippines, Singapore, Thailand, and Vietnam). Through a single platform, millions of users access mobility, delivery, and digital financial services, including ride-hailing, food delivery, payments, lending, and digital banking via GXS Bank and GXBank. Founded in 2012, Grab’s mission is to drive Southeast Asia forward by creating economic empowerment for everyone while delivering sustainable financial performance and positive social impact.Grab 是东南亚领先的超级应用，服务于八个国家（柬埔寨、印度尼西亚、马来西亚、缅甸、菲律宾、新加坡、泰国和越南）的 900 多个城市。通过单一平台，数百万用户可以获取出行、配送和数字金融服务，包括网约车、外卖、支付、借贷，以及通过 GXS Bank 和 GXBank 提供的数字银行服务。Grab 成立于 2012 年，其使命是通过为每个人创造经济赋权，同时提供可持续的财务业绩和积极的社会影响，推动东南亚向前发展。

Powered by technology and driven by heart, our mission is to drive Southeast Asia forward by creating economic empowerment for everyone. If this mission speaks to you, join our team today!以科技为动力，以真诚为驱动，我们的使命是通过为每个人创造经济赋权来推动东南亚向前发展。如果这一使命引起了您的共鸣，欢迎立即加入我们的团队！