Your Kafka Infrastructure Just Became Optional你的 Kafka 基础设施现在可以不再是必需品了

You’ve built the pipeline. Kafka cluster. Zookeeper (or KRaft if you’re current). Schema Registry. Kafka Connect with a Snowflake sink connector. An S3 landing zone. An ETL pipeline to pick up files and load them. Five separate systems, each with its own monitoring, scaling, failover, and billing, are required just to get an event from your application into a queryable table.你已经搭建好了流水线：Kafka 集群、Zookeeper（如果你用的是较旧版本，或者是 KRaft）、Schema Registry、带有 Snowflake Sink 连接器的 Kafka Connect，以及一个 S3 数据落地区。为了将一个事件从应用程序导入到可查询的表中，你需要维护五个独立的系统，每个系统都有自己的监控、扩缩容、故障转移和账单。

At Snowflake Summit 2026, two announcements landed that make this architecture optional: Snowflake Openflow (now GA across AWS, Azure, and GCP) and Datastream, a Kafka-compatible streaming endpoint native to Snowflake that eliminates the middleware entirely.在 2026 年的 Snowflake 峰会上，两项重磅发布让这种架构变得不再必要：Snowflake Openflow（现已在 AWS、Azure 和 GCP 上全面可用）以及 Datastream——这是 Snowflake 原生支持的 Kafka 兼容流式传输端点，彻底消除了中间件的需求。

Here’s what changes, why it matters, and when you should (and shouldn’t) rip out your Kafka cluster.以下是具体的变化、其重要性，以及你何时应该（或不应该）拆除现有的 Kafka 集群。

The Problem: Real-Time Events Delayed by Infrastructure问题所在：基础设施导致实时事件延迟

The slide Snowflake showed at Summit tells the whole story:Snowflake 在峰会上展示的这张幻灯片说明了一切：

EVENT (App/IoT/Web)
  → 1. Kafka Cluster
  → 2. Zookeeper & Schema Registry
  → 3. Kafka Connect
  → 4. S3/GCS Landing Zone
  → 5. ETL Pipeline
  → SNOWFLAKE

Each hop adds latency, cost, and operational burden. Each is a failure domain. Each requires a team (or at least a team member) to keep running.每一道工序都会增加延迟、成本和运维负担。每一道工序都是一个故障域，且都需要专门的团队（或至少一名团队成员）来维持运行。

The real cost isn’t compute; it’s attention. Every incident page at 2 am for a Kafka consumer lag spike, every Schema Registry compatibility check that blocks a deploy, every Connect connector that silently falls behind — that’s engineering time spent on plumbing instead of product.真正的成本不在于计算资源，而在于人力精力。凌晨两点因 Kafka 消费者滞后而触发的报警、阻塞部署的 Schema Registry 兼容性检查、默默掉队的 Connect 连接器——这些都是本该用于产品开发的时间，却被耗费在了处理这些管道琐事上。

The operational burden of each hop: Kafka Cluster (broker provisioning, partition rebalancing, disk, upgrades), Zookeeper/KRaft (quorum health, leader election, metadata drift), Schema Registry (compatibility, versioning, breaking-change coordination), Kafka Connect (configs, dead letter queues, offsets, plugins), Landing Zone (bucket policies, lifecycle rules, file accumulation), and ETL Pipeline (orchestration, retries, dedup, freshness checks).每一道工序的运维负担包括：Kafka 集群（Broker 配置、分区重平衡、磁盘管理、升级）、Zookeeper/KRaft（仲裁健康、领导者选举、元数据漂移）、Schema Registry（兼容性、版本控制、破坏性变更协调）、Kafka Connect（配置、死信队列、偏移量、插件）、落地区（存储桶策略、生命周期规则、文件堆积）以及 ETL 流水线（编排、重试、去重、新鲜度检查）。

Six systems. Six sets of credentials. Six monitoring dashboards. Six teams that can say “not my problem” when data stops flowing.六个系统，六套凭证，六个监控看板。当数据流中断时，有六个团队可以互相推诿说“这不是我的问题”。

The Solution: Two Products, One Goal解决方案：两款产品，一个目标

Snowflake attacked this problem from two angles simultaneously:Snowflake 同时从两个角度解决了这个问题：

Openflow: The Integration PlatformOpenflow：集成平台

Openflow is Snowflake’s fully managed data integration service, built on Apache NiFi. It’s been in preview for a while and went GA across all three major clouds in 2025.Openflow 是 Snowflake 基于 Apache NiFi 构建的全托管数据集成服务。它在预览阶段运行了一段时间，并于 2025 年在三大主流云平台上全面可用（GA）。

What it gives you:它为你提供：

• Pre-built connectors for Kafka, MongoDB, BigQuery, Shopify, Veeva, Oracle, Salesforce, and dozens more针对 Kafka、MongoDB、BigQuery、Shopify、Veeva、Oracle、Salesforce 等数十种数据源的预构建连接器
• Managed infrastructure — runs in Snowpark Container Services (SPCS) or your own VPC (BYOC)托管式基础设施——运行在 Snowpark 容器服务（SPCS）或你自己的 VPC（BYOC）中
• Snowpipe Streaming High Performance as the ingestion backend — 10 GB/s throughput, as low as 5 second end-to-end latency以 Snowpipe Streaming 高性能版作为摄取后端——支持 10 GB/s 的吞吐量，端到端延迟最低可达 5 秒
• Single Message Transforms — filter, enrich, and reshape data before it lands单消息转换——在数据落地前进行过滤、丰富和重塑
• Supports structured and unstructured data — JSON, Avro, Protobuf, images, audio, video, sensor data支持结构化和非结构化数据——JSON、Avro、Protobuf、图像、音频、视频、传感器数据等

For teams that still have Kafka (or MongoDB, or BigQuery) as a source, Openflow replaces the Connect + Landing Zone + ETL portion of the pipeline. Your Kafka cluster stays, but the five downstream hops collapse to one.对于仍将 Kafka（或 MongoDB、BigQuery）作为数据源的团队，Openflow 可以替代流水线中 Connect + 落地区 + ETL 的部分。你的 Kafka 集群可以保留，但原本的五个下游环节可以缩减为一个。

Datastream: The Kafka-Compatible Native EndpointDatastream：原生 Kafka 兼容端点

Datastream is the more radical play. It’s a Kafka-compatible streaming endpoint built directly into Snowflake. Your producers speak the Kafka wire protocol — but there’s no Kafka cluster on the receiving end.Datastream 是更激进的方案。它是一个直接内置于 Snowflake 中的 Kafka 兼容流式传输端点。你的生产者使用 Kafka 通信协议发送数据，但接收端不再需要 Kafka 集群。

EVENT (App/IoT/Web)
  → DATASTREAM
  → Iceberg Tables / Snowflake Tables

Three components. One bill. Zero middleware.三个组件，一份账单，零中间件。

Your application produces events using any Kafka client library. Datastream accepts them natively, handles partitioning and ordering, and writes directly to Iceberg tables or Snowflake tables. No brokers to provision. No partitions to rebalance. No consumer groups to manage.你的应用程序使用任何 Kafka 客户端库即可生产事件。Datastream 原生接收这些事件，处理分区和排序，并直接写入 Iceberg 表或 Snowflake 表。无需配置 Broker，无需重平衡分区，无需管理消费者组。

How Openflow Connectors WorkOpenflow 连接器的工作原理

Openflow isn’t just a Kafka tool the Summit slide highlighted its broader connector ecosystem:Openflow 不仅仅是一个 Kafka 工具，峰会幻灯片强调了其更广泛的连接器生态系统：

OpenFlow connectors, all generally available: Kafka (Apache Kafka, Confluent, AWS MSK), BigQuery (Google BigQuery), MongoDB (Atlas / self-hosted), Shopify (Shopify APIs), Veeva (Veeva Vault), Oracle (Oracle DB CDC), and Snowflake → Kafka (reverse ETL to Kafka topics).OpenFlow 连接器（均已全面可用）：Kafka（Apache Kafka、Confluent、AWS MSK）、BigQuery（Google BigQuery）、MongoDB（Atlas/自托管）、Shopify（Shopify API）、Veeva（Veeva Vault）、Oracle（Oracle DB CDC），以及 Snowflake → Kafka（反向 ETL 到 Kafka 主题）。

Each connector uses the same deployment model:每个连接器都使用相同的部署模型：

1. Create an Openflow deployment (SPCS or BYOC)创建 Openflow 部署（SPCS 或 BYOC）
2. Create a runtime within that deployment在部署中创建运行时
3. Deploy the connector to the runtime将连接器部署到运行时
4. Configure source credentials and target table配置源凭证和目标表
5. Data flows数据开始流动

-- The Snowflake side: create the target table
CREATE OR REPLACE TABLE raw.events.kafka_events (
    record_metadata VARIANT,
    record_content VARIANT,
    _ingestion_time TIMESTAMP_NTZ DEFAULT CURRENT_TIMESTAMP()
);

-- Or use an Iceberg table for open format
CREATE OR REPLACE ICEBERG TABLE raw.events.kafka_events_iceberg (
    event_id STRING,
    event_type STRING,
    payload VARIANT,
    event_time TIMESTAMP_NTZ
)
CATALOG = 'SNOWFLAKE'
EXTERNAL_VOLUME = 'my_ext_volume'
BASE_LOCATION = 'events/';

The connector handles offset tracking, exactly-once delivery, and dead letter queue routing. You configure it through the Openflow canvas (a visual NiFi-based interface) or via APIs.连接器负责处理偏移量跟踪、精确一次（Exactly-once）交付和死信队列路由。你可以通过 Openflow 画布（基于 NiFi 的可视化界面）或 API 进行配置。

Authentication options: SASL (PLAIN, SCRAM-SHA-256, SCRAM-SHA-512, OAUTHBEARER), AWS MSK IAM, and mTLS. Covers every major Kafka deployment pattern.身份验证选项：SASL（PLAIN、SCRAM-SHA-256、SCRAM-SHA-512、OAUTHBEARER）、AWS MSK IAM 和 mTLS。涵盖了所有主流的 Kafka 部署模式。

Data formats: JSON (default), Avro, and Protobuf.数据格式：JSON（默认）、Avro 和 Protobuf。

Snowpipe Streaming: The Engine Under the HoodSnowpipe Streaming：核心引擎

Both OpenFlow connectors and Datastream use Snowpipe Streaming High Performance as their ingestion backend. Understanding this layer matters because it defines the performance envelope:OpenFlow 连接器和 Datastream 都使用 Snowpipe Streaming 高性能版作为摄取后端。理解这一层很重要，因为它决定了性能上限：

Snowpipe Streaming High Performance: throughput up to 10 GB/s per table; latency as low as 5 seconds end-to-end; exactly-once delivery via offset token tracking; ordered within each channel; targets Snowflake tables and Snowflake-managed Iceberg tables (v2 and v3); automatic schema evolution; serverless pricing by uncompressed GB ingested.Snowpipe Streaming 高性能版：每表吞吐量高达 10 GB/s；端到端延迟低至 5 秒；通过偏移量令牌跟踪实现精确一次交付；在每个通道内保持顺序；支持 Snowflake 表和 Snowflake 管理的 Iceberg 表（v2 和 v3）；自动模式演进；按摄取的未压缩 GB 数进行无服务器定价。

The SDKs (Java, Python, Node.js) and REST API give you direct programmatic access to Snowpipe Streaming if you want to bypass both Kafka and Openflow entirely. For IoT edge devices or lightweight microservices, the REST API is the lightest path:如果你想完全绕过 Kafka 和 Openflow，SDK（Java、Python、Node.js）和 REST API 可以让你直接以编程方式访问 Snowpipe Streaming。对于物联网边缘设备或轻量级微服务，REST API 是最轻量的路径：

# Direct row-level ingestion via REST API
# No Kafka, no Openflow, no files — just HTTP
curl -X POST "https://<account>.snowflakecomputing.com/v1/streaming/channels/<channel>/rows" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '[
    {"event_id": "e-001", "event_type": "page_view", "ts": "2026-06-10T14:30:00Z"},
    {"event_id": "e-002", "event_type": "add_to_cart", "ts": "2026-06-10T14:30:01Z"}
  ]'

When to Use What如何选择

Not every team should rip out Kafka tomorrow. Here’s the decision matrix:并非每个团队都需要立即拆除 Kafka。以下是决策矩阵：

Match the tool to your situation: Kafka serving multiple consumers → keep Kafka, use the Openflow Kafka connector; Kafka only feeding the warehouse → replace with Datastream; brand-new pipeline → Datastream or the Snowpipe Streaming SDK; replicating from MongoDB/BigQuery/Oracle → Openflow connectors; under 1000 events/sec from IoT/edge → Snowpipe Streaming REST API; publishing from Snowflake to Kafka → Openflow Snowflake-to-Kafka connector.根据情况选择工具：如果 Kafka 服务于多个消费者 → 保留 Kafka，使用 Openflow Kafka 连接器；如果 Kafka 仅用于向数据仓库供数 → 用 Datastream 替换；如果是全新的流水线 → 使用 Datastream 或 Snowpipe Streaming SDK；如果从 MongoDB/BigQuery/Oracle 进行同步 → 使用 Openflow 连接器；如果物联网/边缘设备每秒数据量低于 1000 条 → 使用 Snowpipe Streaming REST API；如果将数据从 Snowflake 发布到 Kafka → 使用 Openflow Snowflake-to-Kafka 连接器。

The Cost Equation成本账

Let’s talk money. A typical mid-size Kafka deployment for warehouse ingestion:谈谈钱。一个典型的中型 Kafka 数据仓库摄取部署：

Estimated monthly infrastructure cost: 3-broker Kafka cluster $1,200–$2,000; Zookeeper/KRaft $400–$600; Schema Registry $200–$400; Kafka Connect workers $600–$1,000; S3 storage + transfers $100–$500; ETL compute $500–$2,000 totaling roughly $3,000–$6,500/month, plus 10–20% of a senior engineer’s time (≈ $15K–$30K/month of attention).预估月度基础设施成本：3 节点 Kafka 集群 1,200–2,000 美元；Zookeeper/KRaft 400–600 美元；Schema Registry 200–400 美元；Kafka Connect 工作节点 600–1,000 美元；S3 存储及传输 100–500 美元；ETL 计算资源 500–2,000 美元。总计约 3,000–6,500 美元/月，外加一名高级工程师 10%–20% 的时间成本（约合 1.5 万–3 万美元/月的精力投入）。

Plus the hidden cost: 10–20% of a senior engineer’s time managing the above. At fully loaded cost, that’s $15K–$30K/month of engineering attention.此外还有隐性成本：一名高级工程师 10%–20% 的时间用于管理上述系统。按全额成本计算，这相当于每月 1.5 万至 3 万美元的工程精力投入。

Snowpipe Streaming pricing is based on throughput credits per uncompressed GB ingested. For a team ingesting 100 GB/day, the Snowflake-side cost is a fraction of the cost of the infrastructure it replaces. And it’s serverless: no clusters to right-size, no capacity planning.Snowpipe Streaming 的定价基于每摄取未压缩 GB 的吞吐量积分。对于每天摄取 100 GB 的团队来说，Snowflake 端的成本仅为它所替代的基础设施成本的一小部分。而且它是无服务器的：无需调整集群大小，无需容量规划。

The catch: if Kafka serves purposes beyond Snowflake ingestion (event sourcing, CQRS, cross-service communication), you can’t eliminate it. You can only simplify the Snowflake ingestion path.代价是：如果 Kafka 除了 Snowflake 摄取之外还有其他用途（事件溯源、CQRS、跨服务通信），你就无法消除它。你只能简化 Snowflake 的摄取路径。

Deployment Models: SPCS vs. BYOC部署模型：SPCS vs. BYOC

OpenFlow gives you two deployment options:OpenFlow 提供两种部署选项：

SPCS runs in Snowpark Container Services across AWS/Azure/GCP with native Snowflake auth and managed scaling, best for standard use cases and quick setup. BYOC runs in your own AWS VPC with Secrets Manager/Vault, PrivateLink and custom peering, and self-managed scaling, best for private sources, compliance, and data residency.SPCS 运行在 AWS/Azure/GCP 上的 Snowpark 容器服务中，具有原生 Snowflake 身份验证和托管式扩缩容，最适合标准用例和快速设置。BYOC 运行在你自己的 AWS VPC 中，支持 Secrets Manager/Vault、PrivateLink 和自定义对等连接，并由你自行管理扩缩容，最适合私有数据源、合规性和数据驻留要求。

For most teams, SPCS is the right starting point. BYOC makes sense when your data sources are in a private network that can’t be reached from Snowflake’s infrastructure, or when compliance requires data to never leave your VPC.对于大多数团队，SPCS 是合适的起点。当数据源位于无法从 Snowflake 基础设施访问的私有网络中，或者合规性要求数据不得离开你的 VPC 时，BYOC 才有意义。

What This Means for the Streaming Ecosystem这对流处理生态系统意味着什么

Snowflake isn’t killing Kafka. Kafka remains the dominant event streaming platform for cross-service communication, event sourcing, and complex event processing.Snowflake 并没有要消灭 Kafka。Kafka 仍然是跨服务通信、事件溯源和复杂事件处理领域的主流流式传输平台。

What Snowflake is killing: the “Kafka as a conduit to the warehouse” pattern. The use case where Kafka exists primarily (or solely) to buffer events before loading them into an analytical store — that pattern is now redundant.Snowflake 要消灭的是：“将 Kafka 作为数据仓库管道”的模式。那种 Kafka 仅仅（或主要）是为了在加载到分析存储之前缓冲事件的用例——这种模式现在已经多余了。

The industry is converging on a clear split:行业正在形成明确的分工：

Where each tool wins: service-to-service event communication → Kafka (or Pulsar, or Kinesis); event sourcing / CQRS → Kafka; complex event processing with stateful operators → Kafka Streams / Flink; getting events into your analytical warehouse → Snowpipe Streaming / Datastream / OpenFlow; CDC from databases into Snowflake → OpenFlow connectors.各工具的优势领域：服务间事件通信 → Kafka（或 Pulsar、Kinesis）；事件溯源 / CQRS → Kafka；带状态算子的复杂事件处理 → Kafka Streams / Flink；将事件导入分析仓库 → Snowpipe Streaming / Datastream / OpenFlow；数据库 CDC 到 Snowflake → OpenFlow 连接器。

The Kafka ecosystem won’t shrink. But Kafka clusters that exist only as expensive, over-engineered S3 writers — those are going away.Kafka 生态系统不会萎缩。但那些仅仅作为昂贵且过度设计的 S3 写入器的 Kafka 集群，将会消失。

Limitations (Because You Should Know Before You Commit)局限性（在投入使用前你应该了解）

• Datastream is a new announcement. Check Snowflake docs for current availability status and regional support before building production pipelines on it.Datastream 是新发布的产品。在构建生产流水线之前，请查看 Snowflake 文档以了解当前的可用性状态和区域支持情况。
• OpenFlow autoscaling is not supported for the Kafka connector. You set a fixed node count. Under-provision and you get backpressure.OpenFlow 不支持 Kafka 连接器的自动扩缩容。你需要设置固定的节点数量。如果配置不足，会产生反压。
• OpenFlow BYOC is AWS-only. If your sources are in Azure or GCP private networks, you need the SPCS deployment with external access integrations.OpenFlow BYOC 仅支持 AWS。如果你的数据源位于 Azure 或 GCP 的私有网络中，你需要使用带有外部访问集成（External Access Integrations）的 SPCS 部署。
• Schema evolution for Iceberg tables is not supported in the OpenFlow Kafka connector. If your schemas change frequently, target Snowflake-native tables instead.OpenFlow Kafka 连接器不支持 Iceberg 表的模式演进。如果你的模式经常变化，请改用 Snowflake 原生表。
• Only one SPCS deployment per account. You separate workloads by creating multiple runtimes within that deployment, not multiple deployments.每个账户仅支持一个 SPCS 部署。你可以通过在同一部署中创建多个运行时来隔离工作负载，而不是创建多个部署。
• Snowpipe Streaming latency is “as low as 5 seconds”; it is not guaranteed to be 5 seconds. Actual latency depends on ingestion volume, channel count, and table configuration.Snowpipe Streaming 的延迟是“低至 5 秒”；并不保证一定是 5 秒。实际延迟取决于摄取量、通道数和表配置。
• Kafka connector requires Kafka 0.10.0.0+. Legacy Kafka versions are not supported.Kafka 连接器需要 Kafka 0.10.0.0+ 版本。不支持旧版本的 Kafka。

Quick Reference快速参考

The Bigger Picture宏观视角

Data infrastructure is consolidating. The era of “best-of-breed tool for every hop” created resilient but expensive architectures. The next era rewards platforms that can collapse multiple hops into one — without sacrificing reliability.数据基础设施正在整合。那个“为每一个环节选择最佳工具”的时代创造了稳健但昂贵的架构。下一个时代将奖励那些能够将多个环节合并为一个、且不牺牲可靠性的平台。

Snowflake’s bet: if the warehouse can speak Kafka protocol natively, the warehouse becomes the streaming endpoint. No translation layer needed. No impedance mismatch. No six-system Rube Goldberg machine between an event and a query.Snowflake 的赌注是：如果数据仓库能原生支持 Kafka 协议，那么仓库本身就是流处理的终点。无需转换层，无需阻抗失配，无需在事件和查询之间堆砌复杂的“鲁布·戈德堡机械”。

The teams that will benefit most aren’t the ones with complex multi-consumer Kafka deployments. They’re the ones who built Kafka because it was the only reliable way to get real-time data into their warehouse — and now have a simpler option.受益最大的团队并不是那些拥有复杂多消费者 Kafka 部署的团队。而是那些当初为了将实时数据导入仓库而不得不搭建 Kafka 的团队——现在他们有了更简单的选择。

Audit your Kafka deployment this week. If more than 50% of your topics exist solely to feed your warehouse, OpenFlow, or Datastream, just made half your infrastructure redundant.本周审计一下你的 Kafka 部署。如果超过 50% 的主题仅仅是为了向仓库供数，那么 OpenFlow 或 Datastream 已经让你的基础设施有一半变得多余了。