Your Kafka Infrastructure Just Became Optional你的 Kafka 基础设施现在可以成为可选项了

You’ve built the pipeline. Kafka cluster. Zookeeper (or KRaft if you’re current). Schema Registry. Kafka Connect with a Snowflake sink connector. An S3 landing zone. An ETL pipeline to pick up files and load them. Five separate systems, each with its own monitoring, scaling, failover, and billing, are required just to get an event from your application into a queryable table.你已经搭建好了这条流水线：Kafka 集群、Zookeeper（如果你还在用的话，或者 KRaft）、Schema Registry、带有 Snowflake Sink Connector 的 Kafka Connect，以及一个 S3 数据落地区，再加上一个用于提取文件并加载数据的 ETL 流水线。仅仅是为了将应用程序中的一个事件放入可查询的表中，就需要五个独立的系统，每个系统都有自己的监控、扩缩容、故障转移和计费需求。

At Snowflake Summit 2026, two announcements landed that make this architecture optional: Snowflake Openflow (now GA across AWS, Azure, and GCP) and Datastream, a Kafka-compatible streaming endpoint native to Snowflake that eliminates the middleware entirely.在 2026 年的 Snowflake 峰会上，两项公告的发布使得这种架构不再是必需的：Snowflake Openflow（现已在 AWS、Azure 和 GCP 上全面可用）以及 Datastream，后者是一个原生集成在 Snowflake 中、兼容 Kafka 的流式处理端点，彻底消除了中间件的需求。

Here’s what changes, why it matters, and when you should (and shouldn’t) rip out your Kafka cluster.以下是具体的变化、其重要性，以及你何时应该（或不应该）拆除你的 Kafka 集群。

The Problem: Real-Time Events Delayed by Infrastructure问题所在：基础设施导致实时事件延迟

The slide Snowflake showed at Summit tells the whole story:Snowflake 在峰会上展示的幻灯片说明了一切：

EVENT (App/IoT/Web)
  → 1. Kafka Cluster
  → 2. Zookeeper & Schema Registry
  → 3. Kafka Connect
  → 4. S3/GCS Landing Zone
  → 5. ETL Pipeline
  → SNOWFLAKE

Each hop adds latency, cost, and operational burden. Each is a failure domain. Each requires a team (or at least a team member) to keep running.每一次跳转都会增加延迟、成本和运维负担。每一个环节都是一个故障域。每一个环节都需要一个团队（或至少一名团队成员）来维护运行。

The real cost isn’t compute; it’s attention. Every incident page at 2 am for a Kafka consumer lag spike, every Schema Registry compatibility check that blocks a deploy, every Connect connector that silently falls behind — that’s engineering time spent on plumbing instead of product.真正的成本不在于计算资源，而在于人力注意力。凌晨两点因 Kafka 消费者滞后而触发的每一次告警、每一次阻塞部署的 Schema Registry 兼容性检查、每一个静默掉队的 Connect 连接器——这些都是本应用于产品研发却被浪费在维护管道上的工程时间。

The operational burden of each hop: Kafka Cluster (broker provisioning, partition rebalancing, disk, upgrades), Zookeeper/KRaft (quorum health, leader election, metadata drift), Schema Registry (compatibility, versioning, breaking-change coordination), Kafka Connect (configs, dead letter queues, offsets, plugins), Landing Zone (bucket policies, lifecycle rules, file accumulation), and ETL Pipeline (orchestration, retries, dedup, freshness checks).每个环节的运维负担：Kafka 集群（Broker 配置、分区重平衡、磁盘、升级）、Zookeeper/KRaft（仲裁健康状况、领导者选举、元数据漂移）、Schema Registry（兼容性、版本控制、重大变更协调）、Kafka Connect（配置、死信队列、偏移量、插件）、落地区（存储桶策略、生命周期规则、文件堆积）以及 ETL 流水线（编排、重试、去重、新鲜度检查）。

Six systems. Six sets of credentials. Six monitoring dashboards. Six teams that can say “not my problem” when data stops flowing.六个系统。六套凭据。六个监控仪表盘。当数据停止流动时，有六个团队可以推卸责任说“这不是我的问题”。

The Solution: Two Products, One Goal解决方案：两款产品，一个目标

Snowflake attacked this problem from two angles simultaneously:Snowflake 同时从两个角度解决了这个问题：

Openflow: The Integration PlatformOpenflow：集成平台

Openflow is Snowflake’s fully managed data integration service, built on Apache NiFi. It’s been in preview for a while and went GA across all three major clouds in 2025.Openflow 是 Snowflake 的全托管数据集成服务，基于 Apache NiFi 构建。它在预览阶段运行了一段时间，并于 2025 年在三大主流云平台上全面发布。

What it gives you:它为你提供了什么：

• Pre-built connectors for Kafka, MongoDB, BigQuery, Shopify, Veeva, Oracle, Salesforce, and dozens more针对 Kafka、MongoDB、BigQuery、Shopify、Veeva、Oracle、Salesforce 等数十种数据源的预构建连接器
• Managed infrastructure — runs in Snowpark Container Services (SPCS) or your own VPC (BYOC)托管基础设施——运行在 Snowpark Container Services (SPCS) 或你自己的 VPC (BYOC) 中
• Snowpipe Streaming High Performance as the ingestion backend — 10 GB/s throughput, as low as 5 second end-to-end latency以 Snowpipe Streaming 高性能引擎作为后端——吞吐量可达 10 GB/s，端到端延迟低至 5 秒
• Single Message Transforms — filter, enrich, and reshape data before it lands单条消息转换——在数据落地前进行过滤、丰富和重塑
• Supports structured and unstructured data — JSON, Avro, Protobuf, images, audio, video, sensor data支持结构化和非结构化数据——JSON、Avro、Protobuf、图像、音频、视频、传感器数据

For teams that still have Kafka (or MongoDB, or BigQuery) as a source, Openflow replaces the Connect + Landing Zone + ETL portion of the pipeline. Your Kafka cluster stays, but the five downstream hops collapse to one.对于那些仍然将 Kafka（或 MongoDB、BigQuery）作为数据源的团队，Openflow 可以替代流水线中 Connect + 落地区 + ETL 的部分。你的 Kafka 集群可以保留，但下游的五个跳转环节将缩减为一个。

Datastream: The Kafka-Compatible Native EndpointDatastream：兼容 Kafka 的原生端点

Datastream is the more radical play. It’s a Kafka-compatible streaming endpoint built directly into Snowflake. Your producers speak the Kafka wire protocol — but there’s no Kafka cluster on the receiving end.Datastream 是一个更彻底的方案。它是一个直接内置于 Snowflake 中的、兼容 Kafka 的流式处理端点。你的生产者使用 Kafka 网络协议发送数据——但接收端不再需要 Kafka 集群。

EVENT (App/IoT/Web)
  → DATASTREAM
  → Iceberg Tables / Snowflake Tables

Three components. One bill. Zero middleware.三个组件。一张账单。零中间件。

Your application produces events using any Kafka client library. Datastream accepts them natively, handles partitioning and ordering, and writes directly to Iceberg tables or Snowflake tables. No brokers to provision. No partitions to rebalance. No consumer groups to manage.你的应用程序使用任何 Kafka 客户端库产生事件。Datastream 原生接收这些事件，处理分区和排序，并直接写入 Iceberg 表或 Snowflake 表。无需配置 Broker，无需重平衡分区，无需管理消费者组。

How Openflow Connectors WorkOpenflow 连接器的工作原理

Openflow isn’t just a Kafka tool the Summit slide highlighted its broader connector ecosystem:Openflow 不仅仅是一个 Kafka 工具，峰会的幻灯片强调了其更广泛的连接器生态系统：

OpenFlow connectors, all generally available: Kafka (Apache Kafka, Confluent, AWS MSK), BigQuery (Google BigQuery), MongoDB (Atlas / self-hosted), Shopify (Shopify APIs), Veeva (Veeva Vault), Oracle (Oracle DB CDC), and Snowflake → Kafka (reverse ETL to Kafka topics).OpenFlow 连接器（均已全面可用）：Kafka（Apache Kafka、Confluent、AWS MSK）、BigQuery（Google BigQuery）、MongoDB（Atlas / 自托管）、Shopify（Shopify API）、Veeva（Veeva Vault）、Oracle（Oracle DB CDC）以及 Snowflake → Kafka（反向 ETL 到 Kafka 主题）。

Each connector uses the same deployment model:每个连接器都使用相同的部署模式：

1. Create an Openflow deployment (SPCS or BYOC)创建 Openflow 部署（SPCS 或 BYOC）
2. Create a runtime within that deployment在该部署内创建一个运行时环境
3. Deploy the connector to the runtime将连接器部署到该运行时环境
4. Configure source credentials and target table配置源凭据和目标表
5. Data flows数据开始流动

-- The Snowflake side: create the target table
CREATE OR REPLACE TABLE raw.events.kafka_events (
    record_metadata VARIANT,
    record_content VARIANT,
    _ingestion_time TIMESTAMP_NTZ DEFAULT CURRENT_TIMESTAMP()
);

-- Or use an Iceberg table for open format
CREATE OR REPLACE ICEBERG TABLE raw.events.kafka_events_iceberg (
    event_id STRING,
    event_type STRING,
    payload VARIANT,
    event_time TIMESTAMP_NTZ
)
CATALOG = 'SNOWFLAKE'
EXTERNAL_VOLUME = 'my_ext_volume'
BASE_LOCATION = 'events/';

The connector handles offset tracking, exactly-once delivery, and dead letter queue routing. You configure it through the Openflow canvas (a visual NiFi-based interface) or via APIs.连接器处理偏移量跟踪、精确一次交付和死信队列路由。你可以通过 Openflow 画布（基于 NiFi 的可视化界面）或 API 进行配置。

Authentication options: SASL (PLAIN, SCRAM-SHA-256, SCRAM-SHA-512, OAUTHBEARER), AWS MSK IAM, and mTLS. Covers every major Kafka deployment pattern.认证选项：SASL（PLAIN、SCRAM-SHA-256、SCRAM-SHA-512、OAUTHBEARER）、AWS MSK IAM 和 mTLS。涵盖了所有主要的 Kafka 部署模式。

Data formats: JSON (default), Avro, and Protobuf.数据格式：JSON（默认）、Avro 和 Protobuf。

Snowpipe Streaming: The Engine Under the HoodSnowpipe Streaming：核心引擎

Both OpenFlow connectors and Datastream use Snowpipe Streaming High Performance as their ingestion backend. Understanding this layer matters because it defines the performance envelope:OpenFlow 连接器和 Datastream 都使用 Snowpipe Streaming 高性能引擎作为其摄取后端。了解这一层非常重要，因为它决定了性能极限：

Snowpipe Streaming High Performance: throughput up to 10 GB/s per table; latency as low as 5 seconds end-to-end; exactly-once delivery via offset token tracking; ordered within each channel; targets Snowflake tables and Snowflake-managed Iceberg tables (v2 and v3); automatic schema evolution; serverless pricing by uncompressed GB ingested.Snowpipe Streaming 高性能引擎：每张表吞吐量高达 10 GB/s；端到端延迟低至 5 秒；通过偏移令牌跟踪实现精确一次交付；在每个通道内保持有序；支持写入 Snowflake 表和 Snowflake 管理的 Iceberg 表（v2 和 v3）；支持自动模式演进；按摄入的未压缩 GB 数据量进行无服务器计费。

The SDKs (Java, Python, Node.js) and REST API give you direct programmatic access to Snowpipe Streaming if you want to bypass both Kafka and Openflow entirely. For IoT edge devices or lightweight microservices, the REST API is the lightest path:如果你想完全绕过 Kafka 和 Openflow，SDK（Java、Python、Node.js）和 REST API 可以让你直接以编程方式访问 Snowpipe Streaming。对于物联网边缘设备或轻量级微服务，REST API 是最轻量的路径：

# Direct row-level ingestion via REST API
# No Kafka, no Openflow, no files — just HTTP
curl -X POST "https://<account>.snowflakecomputing.com/v1/streaming/channels/<channel>/rows" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '[
    {"event_id": "e-001", "event_type": "page_view", "ts": "2026-06-10T14:30:00Z"},
    {"event_id": "e-002", "event_type": "add_to_cart", "ts": "2026-06-10T14:30:01Z"}
  ]'

When to Use What如何选择

Not every team should rip out Kafka tomorrow. Here’s the decision matrix:并不是每个团队都应该立即拆除 Kafka。以下是决策矩阵：

Match the tool to your situation: Kafka serving multiple consumers → keep Kafka, use the Openflow Kafka connector; Kafka only feeding the warehouse → replace with Datastream; brand-new pipeline → Datastream or the Snowpipe Streaming SDK; replicating from MongoDB/BigQuery/Oracle → Openflow connectors; under 1000 events/sec from IoT/edge → Snowpipe Streaming REST API; publishing from Snowflake to Kafka → Openflow Snowflake-to-Kafka connector.根据情况匹配工具：Kafka 服务于多个消费者 → 保留 Kafka，使用 Openflow Kafka 连接器；Kafka 仅用于向仓库供数 → 用 Datastream 替换；全新的流水线 → 使用 Datastream 或 Snowpipe Streaming SDK；从 MongoDB/BigQuery/Oracle 复制数据 → 使用 Openflow 连接器；物联网/边缘设备每秒低于 1000 个事件 → 使用 Snowpipe Streaming REST API；从 Snowflake 发布数据到 Kafka → 使用 Openflow Snowflake 到 Kafka 连接器。

The Cost Equation成本核算

Let’s talk money. A typical mid-size Kafka deployment for warehouse ingestion:我们来谈谈钱。一个典型的中型 Kafka 仓库摄取部署：

Estimated monthly infrastructure cost: 3-broker Kafka cluster $1,200–$2,000; Zookeeper/KRaft $400–$600; Schema Registry $200–$400; Kafka Connect workers $600–$1,000; S3 storage + transfers $100–$500; ETL compute $500–$2,000 totaling roughly $3,000–$6,500/month, plus 10–20% of a senior engineer’s time (≈ $15K–$30K/month of attention).预估月度基础设施成本：3 节点 Kafka 集群 1,200–2,000 美元；Zookeeper/KRaft 400–600 美元；Schema Registry 200–400 美元；Kafka Connect 工作节点 600–1,000 美元；S3 存储 + 传输 100–500 美元；ETL 计算 500–2,000 美元，总计约 3,000–6,500 美元/月，外加高级工程师 10%–20% 的时间成本（约 1.5 万–3 万美元/月的注意力成本）。

Plus the hidden cost: 10–20% of a senior engineer’s time managing the above. At fully loaded cost, that’s $15K–$30K/month of engineering attention.此外还有隐性成本：高级工程师 10%–20% 的时间用于管理上述系统。按全额成本计算，这意味着每月 1.5 万至 3 万美元的工程精力投入。

Snowpipe Streaming pricing is based on throughput credits per uncompressed GB ingested. For a team ingesting 100 GB/day, the Snowflake-side cost is a fraction of the cost of the infrastructure it replaces. And it’s serverless: no clusters to right-size, no capacity planning.Snowpipe Streaming 的定价基于每摄入未压缩 GB 的吞吐量积分。对于每天摄入 100 GB 的团队来说，Snowflake 侧的成本仅为其所替代基础设施成本的一小部分。而且它是无服务器的：无需调整集群大小，无需容量规划。

The catch: if Kafka serves purposes beyond Snowflake ingestion (event sourcing, CQRS, cross-service communication), you can’t eliminate it. You can only simplify the Snowflake ingestion path.需要注意的是：如果 Kafka 的用途不仅仅是 Snowflake 摄取（例如事件溯源、CQRS、跨服务通信），你就不能消除它。你只能简化 Snowflake 的摄取路径。

Deployment Models: SPCS vs. BYOC部署模式：SPCS 与 BYOC

OpenFlow gives you two deployment options:OpenFlow 为你提供两种部署选项：

SPCS runs in Snowpark Container Services across AWS/Azure/GCP with native Snowflake auth and managed scaling, best for standard use cases and quick setup. BYOC runs in your own AWS VPC with Secrets Manager/Vault, PrivateLink and custom peering, and self-managed scaling, best for private sources, compliance, and data residency.SPCS 在 AWS/Azure/GCP 上的 Snowpark Container Services 中运行，具有原生 Snowflake 认证和托管扩缩容功能，最适合标准用例和快速设置。BYOC 在你自己的 AWS VPC 中运行，支持 Secrets Manager/Vault、PrivateLink 和自定义对等连接，并支持自管理扩缩容，最适合私有数据源、合规性要求和数据驻留需求。

For most teams, SPCS is the right starting point. BYOC makes sense when your data sources are in a private network that can’t be reached from Snowflake’s infrastructure, or when compliance requires data to never leave your VPC.对于大多数团队，SPCS 是合适的起点。当你的数据源位于无法从 Snowflake 基础设施访问的私有网络中，或者合规性要求数据不得离开你的 VPC 时，BYOC 才有意义。

What This Means for the Streaming Ecosystem这对流式处理生态意味着什么

Snowflake isn’t killing Kafka. Kafka remains the dominant event streaming platform for cross-service communication, event sourcing, and complex event processing.Snowflake 并没有要消灭 Kafka。Kafka 仍然是跨服务通信、事件溯源和复杂事件处理领域的主导流式处理平台。

What Snowflake is killing: the “Kafka as a conduit to the warehouse” pattern. The use case where Kafka exists primarily (or solely) to buffer events before loading them into an analytical store — that pattern is now redundant.Snowflake 要消灭的是：“将 Kafka 作为仓库管道”的模式。那种 Kafka 存在的主要（或唯一）目的仅仅是为了在将事件加载到分析存储之前进行缓冲的用例——这种模式现在已经多余了。

The industry is converging on a clear split:行业正在达成一种明确的分工：

Where each tool wins: service-to-service event communication → Kafka (or Pulsar, or Kinesis); event sourcing / CQRS → Kafka; complex event processing with stateful operators → Kafka Streams / Flink; getting events into your analytical warehouse → Snowpipe Streaming / Datastream / OpenFlow; CDC from databases into Snowflake → OpenFlow connectors.各工具的优势领域：服务间事件通信 → Kafka（或 Pulsar、Kinesis）；事件溯源 / CQRS → Kafka；带有有状态算子的复杂事件处理 → Kafka Streams / Flink；将事件导入分析仓库 → Snowpipe Streaming / Datastream / OpenFlow；从数据库到 Snowflake 的 CDC → OpenFlow 连接器。

The Kafka ecosystem won’t shrink. But Kafka clusters that exist only as expensive, over-engineered S3 writers — those are going away.Kafka 生态系统不会萎缩。但那些仅仅作为昂贵、过度设计的 S3 写入器的 Kafka 集群——它们将会消失。

Limitations (Because You Should Know Before You Commit)局限性（在投入使用前你应该了解）

• Datastream is a new announcement. Check Snowflake docs for current availability status and regional support before building production pipelines on it.Datastream 是新发布的。在基于它构建生产流水线之前，请查看 Snowflake 文档了解当前的可用性状态和区域支持情况。
• OpenFlow autoscaling is not supported for the Kafka connector. You set a fixed node count. Under-provision and you get backpressure.OpenFlow Kafka 连接器不支持自动扩缩容。你需要设置固定的节点数量。如果配置不足，将会产生背压。
• OpenFlow BYOC is AWS-only. If your sources are in Azure or GCP private networks, you need the SPCS deployment with external access integrations.OpenFlow BYOC 仅支持 AWS。如果你的数据源位于 Azure 或 GCP 的私有网络中，你需要使用带有外部访问集成功能的 SPCS 部署。
• Schema evolution for Iceberg tables is not supported in the OpenFlow Kafka connector. If your schemas change frequently, target Snowflake-native tables instead.OpenFlow Kafka 连接器不支持 Iceberg 表的模式演进。如果你的模式经常变化，请改用 Snowflake 原生表。
• Only one SPCS deployment per account. You separate workloads by creating multiple runtimes within that deployment, not multiple deployments.每个账户仅支持一个 SPCS 部署。你应该通过在部署内创建多个运行时环境来区分工作负载，而不是创建多个部署。
• Snowpipe Streaming latency is “as low as 5 seconds”; it is not guaranteed to be 5 seconds. Actual latency depends on ingestion volume, channel count, and table configuration.Snowpipe Streaming 的延迟是“低至 5 秒”；并不保证一定是 5 秒。实际延迟取决于摄入量、通道数量和表配置。
• Kafka connector requires Kafka 0.10.0.0+. Legacy Kafka versions are not supported.Kafka 连接器要求 Kafka 0.10.0.0+ 版本。不支持旧版 Kafka。

Quick Reference快速参考

The Bigger Picture宏观视角

Data infrastructure is consolidating. The era of “best-of-breed tool for every hop” created resilient but expensive architectures. The next era rewards platforms that can collapse multiple hops into one — without sacrificing reliability.数据基础设施正在整合。那个“为每个环节选择最佳工具”的时代创造了稳健但昂贵的架构。下一个时代将奖励那些能够将多个环节合并为一个——且不牺牲可靠性的平台。

Snowflake’s bet: if the warehouse can speak Kafka protocol natively, the warehouse becomes the streaming endpoint. No translation layer needed. No impedance mismatch. No six-system Rube Goldberg machine between an event and a query.Snowflake 的赌注是：如果仓库能够原生使用 Kafka 协议，那么仓库本身就成为了流式处理的端点。无需转换层。无需阻抗失配。事件与查询之间不再需要六个系统的鲁布·戈德堡机械装置。

The teams that will benefit most aren’t the ones with complex multi-consumer Kafka deployments. They’re the ones who built Kafka because it was the only reliable way to get real-time data into their warehouse — and now have a simpler option.受益最大的团队不是那些拥有复杂多消费者 Kafka 部署的团队，而是那些为了将实时数据导入仓库而不得不构建 Kafka（因为那是当时唯一可靠的方法）的团队——现在他们有了更简单的选择。

Audit your Kafka deployment this week. If more than 50% of your topics exist solely to feed your warehouse, OpenFlow, or Datastream, just made half your infrastructure redundant.本周审计一下你的 Kafka 部署。如果超过 50% 的主题仅仅是为了给仓库供数，那么 OpenFlow 或 Datastream 已经让你的基础设施有一半变得多余了。