Cell-Based Architecture for Resilient Payment Systems 基于单元的弹性支付系统架构

The American Express core payments ecosystem is a global platform relied on by Card Members and partners around the world. Every day, it processes live payment transactions that require high availability, low latency, and predictable performance.美国运通核心支付生态系统是一个全球平台，受到全球持卡会员和合作伙伴的信赖。每天，它处理需要高可用性、低延迟和可预测性能的实时支付交易。

Resiliency is not an afterthought; it has been encoded into the system’s design from the beginning. Localized faults are contained within defined boundaries, and recovery is designed to be fast and predictable.弹性并非事后考虑；它从一开始就被编码到系统设计中。局部故障被限制在定义的边界内，恢复设计快速且可预测。

To achieve this, the platform is built around a cell-based architecture that isolates failures, maintains low-latency processing, and scales capacity without expanding the failure domain.为了实现这一点，该平台基于单元架构构建，该架构隔离故障、保持低延迟处理，并在不扩大故障域的情况下扩展容量。

This blog outlines the principles that guide this architecture and how they help us build a resilient payments latform at global scale.这篇博客概述了指导该架构的原则，以及它们如何帮助我们构建全球规模的弹性支付平台。

Core Payments Ecosystem核心支付生态系统

In 2018, we started a journey to modernize our core payments ecosystem. This platform processes live card and payment transactions and is mission-critical to our Card Members and partners.2018年，我们开始了现代化核心支付生态系统的旅程。该平台处理实时卡和支付交易，对我们的持卡会员和合作伙伴至关重要。

As we modernized the platform, resiliency remained a primary design requirement. We needed an architecture that could continue processing transactions reliably, even when individual components failed. This decision was heavily influenced by our historical design patterns, which predated the term “cell-based architecture,” but share many of the same principles.在现代化平台的过程中，弹性始终是首要设计需求。我们需要一种架构，即使在单个组件发生故障时也能继续可靠地处理交易。这一决定深受我们历史设计模式的影响，这些模式早于“基于单元的架构”这一术语，但共享许多相同的原则。

Our new platform targeted cloud-native technologies, which meant we needed to think differently about how we designed for resiliency and scalability.我们的新平台针对云原生技术，这意味着我们需要重新思考如何设计弹性和可扩展性。

In the next sections, we’ll discuss some of the design principles we follow in our core payments ecosystem and how they not only improve our ability to process payments reliably but also help us reduce latency and scale more easily.在接下来的部分中，我们将讨论核心支付生态系统遵循的一些设计原则，以及它们如何不仅提高我们可靠处理支付的能力，还帮助我们减少延迟并更轻松地扩展。

What is Cell-based Architecture?什么是基于单元的架构？

Cell-based architecture is an architecture pattern that has gained popularity in the cloud-native distributed systems space.基于单元的架构是一种在云原生分布式系统领域越来越流行的架构模式。

The idea behind the concept is to group related microservices, databases, and other components into independent instances called cells. Each cell is able to function independently without reliance on other cells.该概念背后的想法是将相关的微服务、数据库和其他组件分组到称为单元的独立实例中。每个单元能够独立运行，无需依赖其他单元。

In this diagram: Each cell contains its own services and data so a failure stays within that cell instead of spreading across the platform.在此图中：每个单元包含自己的服务和数据，因此故障停留在该单元内，而不是扩散到整个平台。

The primary benefit of cell-based architecture is reducing the blast radius of failures. With each cell being independent, if one cell experiences issues, it doesn’t impact the others. The trade-off is that cell-based architecture often increases management overhead and architectural complexity, as it requires careful design to ensure that cells are truly independent and that data is appropriately localized.基于单元的架构的主要好处是减少故障的爆炸半径。由于每个单元都是独立的，如果一个单元出现问题，不会影响其他单元。权衡之处在于，基于单元的架构通常会增加管理开销和架构复杂性，因为它需要精心设计以确保单元真正独立，并且数据适当本地化。

However, for mission-critical systems like payments, we find that the benefits of a reduced blast radius and improved resiliency outweigh the additional complexity.然而，对于像支付这样的关键任务系统，我们发现减少爆炸半径和提高弹性的好处超过了额外的复杂性。

We’ve also found that when implemented well, a cell-based architecture can help platforms reduce latency (by reducing external dependencies and network hops) and improve scaling by introducing additional independent cells.我们还发现，如果实施得当，基于单元的架构可以通过减少外部依赖和网络跳数来帮助平台降低延迟，并通过引入额外的独立单元来改善扩展。

How We Follow Cell-Based Architecture我们如何遵循基于单元的架构

Each instance of our core payments ecosystem is designed as a cell, which:核心支付生态系统的每个实例都被设计为一个单元，它：

• Is an independently deployable unit that can process payments on its own.是一个可独立部署的单元，可以自行处理支付。
• Has its own set of microservices, databases, and other components.拥有自己的一组微服务、数据库和其他组件。
• Is a single failure domain, meaning that if one cell experiences issues, it doesn’t cascade the failure beyond the cell boundary.是一个单一的故障域，意味着如果一个单元出现问题，故障不会级联到单元边界之外。
• Can be taken out of rotation for maintenance or in response to failures without impacting the overall system or requiring coordination with other cells.可以因维护或应对故障而退出轮换，而不会影响整个系统或需要与其他单元协调。
• Has no synchronous cross-cell dependencies in the critical path of processing transactions.在处理交易的关键路径中没有同步的跨单元依赖。

A cell is defined by its failure boundaries rather than a specific infrastructure construct. In practice, cells never span multiple regions—everything required to process transactions (DNS, databases, microservices, and supporting services) remains local within that boundary.单元由其故障边界定义，而不是特定的基础设施构造。在实践中，单元从不跨越多个区域——处理交易所需的一切（DNS、数据库、微服务和支持服务）都保持在该边界内本地化。

To achieve this, we follow a set of core principles that guide our design decisions and help us ensure that our cells are truly independent and resilient.为了实现这一点，我们遵循一套核心原则，指导我们的设计决策，并帮助我们确保单元真正独立且具有弹性。

Data and Processing Locality by Default默认的数据和处理本地化

Processing payments requires data: currency rates, merchant category codes, and so on. Some data is static, while some data changes with each transaction.处理支付需要数据：汇率、商户类别代码等。有些数据是静态的，而有些数据随每笔交易变化。

Static & Semi-Static Data Replication静态和半静态数据复制

For static or semi-static data like currency rates and merchant category codes, we replicate that data to each cell.对于静态或半静态数据，如汇率和商户类别代码，我们将这些数据复制到每个单元。

In this diagram: Reference data is pushed into every cell ahead of time so transaction processing never needs a synchronous lookup to a central source.在此图中：参考数据提前推送到每个单元，因此交易处理永远不需要同步查询中央源。

Rather than relying on a fall-through read to a centralized system of record during transaction processing, we pre-populate this data in each cell ahead of time. This keeps reference data local before transactions arrive, avoids cache-miss latency during processing, and preserves critical-path isolation.我们不在交易处理期间依赖回退读取到集中式记录系统，而是提前在每个单元中预填充这些数据。这使参考数据在交易到达之前保持本地化，避免处理期间的缓存未命中延迟，并保留关键路径隔离。

The replication work happens outside the transaction path, which lets us keep the data available locally without introducing synchronous cross-cell dependencies.复制工作在交易路径之外进行，这使我们能够保持数据本地可用，而无需引入同步的跨单元依赖。

Dynamic Data Routing动态数据路由

Not all data is static and not all data can be pre-populated. For more dynamic data (data that changes with each transaction), data replication may not be fast enough to ensure that every cell has the right data at the right time. We don’t want to route transactions to cells that don’t have the latest data, as that would increase latency and potentially lead to processing failures.并非所有数据都是静态的，也并非所有数据都可以预填充。对于更动态的数据（随每笔交易变化的数据），数据复制可能不够快，无法确保每个单元在正确的时间拥有正确的数据。我们不想将交易路由到没有最新数据的单元，因为这会增加延迟并可能导致处理失败。

Instead, we use deterministic routing to route transactions to the cell where the right data is already available. In a recent article, Migrating the Payment Network Twice with Zero Downtime, we introduced the Global Transaction Router, which is responsible for managing connectivity and routing transactions to the appropriate cell. It can do so because it understands just enough of the payment specifications to make routing decisions based on the transaction data.相反，我们使用确定性路由将交易路由到已有正确数据的单元。在最近的一篇文章《零停机迁移支付网络两次》中，我们介绍了全局交易路由器，它负责管理连接并将交易路由到适当的单元。它之所以能做到这一点，是因为它足够了解支付规范，可以根据交易数据做出路由决策。

For example, we may route transactions based on partner, market, or payment type; how we route depends on the payment transaction data and the use case, but the key is that we selectively route transactions to where they are needed when there is a need for strong data consistency across transactions.例如，我们可以根据合作伙伴、市场或支付类型路由交易；路由方式取决于支付交易数据和用例，但关键是在需要跨交易强数据一致性时，我们有选择地将交易路由到需要的地方。

In this diagram: The router sends a transaction to the cell that already has the authoritative dynamic state, while replication continues asynchronously outside the critical path.在此图中：路由器将交易发送到已经拥有权威动态状态的单元，而复制在关键路径之外异步继续。

We keep transaction processing localized by restricting microservice communication to pod-to-pod interactions within the cell’s Kubernetes network, ensuring all processing remains within the cell’s boundaries.我们通过将微服务通信限制在单元Kubernetes网络内的Pod到Pod交互，使交易处理保持本地化，确保所有处理保持在单元边界内。

To ensure failover data is synchronized across cells using message-based replication, that replication happens asynchronously outside the transaction path, so it doesn’t impact latency or availability.为了确保故障转移数据通过基于消息的复制在单元间同步，该复制在交易路径之外异步进行，因此不会影响延迟或可用性。

No in-flight transaction waits for replication to complete; if the latest state is required, the Global Transaction Router sends the transaction to the cell where that data is already authoritative or available.没有正在进行的交易等待复制完成；如果需要最新状态，全局交易路由器会将交易发送到该数据已经权威或可用的单元。

We only allow our microservices to talk to localized database instances. This keeps latency predictable and avoids unnecessary network hops, but it requires deliberate routing decisions.我们只允许微服务与本地化的数据库实例通信。这使延迟可预测，并避免不必要的网络跳数，但需要深思熟虑的路由决策。

By introducing deterministic routing at the edge, we can ensure that transactions are routed to the cell where the right data is already available.通过在边缘引入确定性路由，我们可以确保交易被路由到已有正确数据的单元。

Enforced Boundaries for Ingress and Egress强制入口和出口边界

Along with its routing capabilities, the Global Transaction Router also serves as a key enforcer of our “local only” processing.除了路由功能外，全局交易路由器还是我们“仅本地”处理的关键执行者。

Transactions must enter a cell through the Global Transaction Router; if a cell cannot process a transaction and that transaction needs to be rerouted to another cell, it must also go through the Global Transaction Router.交易必须通过全局交易路由器进入单元；如果一个单元无法处理交易，并且该交易需要重新路由到另一个单元，它也必须通过全局交易路由器。

In this way, the Global Transaction Router also serves as a payments mesh, connecting our cells globally.通过这种方式，全局交易路由器还充当支付网格，连接我们的全球单元。

In this diagram: All cross-cell traffic is funneled through the Global Transaction Router, which preserves strict cell boundaries.在此图中：所有跨单元流量都通过全局交易路由器汇集，从而保持严格的单元边界。

Preventing cross-cell dependencies becomes increasingly difficult as platforms grow.随着平台的发展，防止跨单元依赖变得越来越困难。

By tightly controlling cross-cell communication through the Global Transaction Router, we prevent cells from forming strong dependencies on each other, as they do not have the ability to communicate at all—only the Global Transaction Router can communicate across cells.通过全局交易路由器严格控制跨单元通信，我们防止单元之间形成强依赖，因为它们根本无法通信——只有全局交易路由器可以跨单元通信。

This enforcement occasionally results in duplicated services where shared implementations might otherwise seem simpler, but it preserves cell independence and improves latency by reducing cross-cell network hops.这种强制有时会导致服务重复，否则共享实现可能看起来更简单，但它保持了单元独立性，并通过减少跨单元网络跳数改善了延迟。

The same principle applies to observability. Each cell publishes logs, metrics, and traces to observability components localized within that cell first, so losing part of the observability stack only reduces visibility for that cell instead of the entire platform. We still aggregate observability data asynchronously to provide global dashboards, alerting, and fleet-wide analysis, but that aggregation remains outside the transaction’s critical path.同样的原则适用于可观测性。每个单元首先将日志、指标和跟踪发布到该单元内的本地化可观测性组件，因此丢失部分可观测性堆栈只会降低该单元的可见性，而不是整个平台。我们仍然异步聚合可观测性数据以提供全局仪表板、告警和整个机群分析，但该聚合保持在交易关键路径之外。

Cells Break in Isolation; Other Cells Replace Them单元独立故障；其他单元替换它们

Leveraging the ability to reroute transactions to other cells is a key part of our resiliency strategy.利用将交易重新路由到其他单元的能力是我们弹性策略的关键部分。

When failures occur, their impact stays contained within the affected cell, and transactions are automatically rerouted to a healthy cell where processing restarts.当发生故障时，其影响保持在受影响的单元内，交易自动重新路由到健康单元，处理重新开始。

We reroute not only new incoming transactions but also transactions that were already in-flight in the failing cell.我们不仅重新路由新的传入交易，还重新路由已在故障单元中正在进行的交易。

Our Payments Processing subsystem follows an orchestrated microservices architecture, where an orchestrator microservice manages the processing workflow and calls other microservices to perform specific tasks.我们的支付处理子系统遵循编排的微服务架构，其中编排器微服务管理工作流并调用其他微服务执行特定任务。

If a downstream service begins to fail, the orchestrator detects the failure, halts processing, and sends the transaction back to the Global Transaction Router to be rerouted to another cell.如果下游服务开始失败，编排器检测到故障，停止处理，并将交易发送回全局交易路由器以重新路由到另一个单元。

In this diagram: When a cell fails mid-flow, the transaction is rerouted and restarted in a healthy cell rather than resumed across cells.在此图中：当单元在流程中间失败时，交易被重新路由并在健康单元中重新开始，而不是跨单元恢复。

We do not attempt to resume partially processed transactions across cells. Instead, we restart transaction processing in another cell with the original transaction data.我们不会尝试跨单元恢复部分处理的交易。相反，我们在另一个单元中使用原始交易数据重新开始交易处理。

This restart is only safe while the transaction is still within the core payments ecosystem. Once a transaction has been sent to an external system (e.g., card issuer), we consider that a point of no return, and we don’t allow transactions to be rerouted after that point.这种重新启动仅在交易仍在核心支付生态系统内时是安全的。一旦交易已发送到外部系统（例如发卡行），我们将其视为不可返回点，并且在此之后不允许交易重新路由。

Card authorizations are structured so that the point of no return is toward the end of processing. If a transaction fails before the point of no return, we can safely reroute and restart processing without worrying about duplicate transactions or data consistency issues.卡授权被设计为不可返回点位于处理接近结束时。如果交易在不可返回点之前失败，我们可以安全地重新路由并重新开始处理，而无需担心重复交易或数据一致性问题。

For other payment types, we manage idempotency through transaction identifiers. Each transaction carries a unique transaction identifier that remains consistent across retries and reroutes. Downstream systems use these identifiers to detect and suppress duplicate requests, allowing retries and reroutes to be handled safely without introducing inconsistencies or duplicate transactions.对于其他支付类型，我们通过交易标识符管理幂等性。每笔交易携带一个唯一的交易标识符，在重试和重新路由时保持一致。下游系统使用这些标识符检测并抑制重复请求，从而安全地处理重试和重新路由，而不会引入不一致或重复交易。

The restart model emphasizes the importance of avoiding shared state between cells. Cross-cell shared state would introduce synchronization challenges and potential consistency issues, especially during failover scenarios. Communication failures between cells could impact the ability to process transactions globally, which we want to avoid at all costs for a payments system.重新启动模型强调了避免单元间共享状态的重要性。跨单元共享状态会引入同步挑战和潜在的一致性问题，尤其是在故障转移场景中。单元之间的通信故障可能影响全局处理交易的能力，对于支付系统，我们希望不惜一切代价避免这种情况。

In our architecture, cells are designed to be loosely coupled. Each cell has its own database clusters, and the microservices within a cell only communicate with the local database cluster.在我们的架构中，单元被设计为松散耦合。每个单元有自己的数据库集群，单元内的微服务仅与本地数据库集群通信。

When a cell fails, its impact stays confined to that cell, allowing other cells to continue processing transactions normally.当单元发生故障时，其影响被限制在该单元内，允许其他单元继续正常处理交易。

When rerouted, transactions are processed without reliance on state from the previous cell.重新路由时，交易的处理不依赖于前一个单元的状态。

At any point in time, a cell can be taken out of rotation. When a cell is taken out of rotation either automatically or manually, another cell takes its place. This does not have to be a binary cutover. As discussed in Migrating the Payment Network Twice with Zero Downtime, the Global Transaction Router can shift traffic between cells by percentage, allowing us to gradually drain a cell for maintenance, validate a recovering cell under partial load, or respond more safely during incidents.在任何时间点，单元都可以退出轮换。当单元自动或手动退出轮换时，另一个单元会取代它。这不必是二进制切换。如《零停机迁移支付网络两次》中所述，全局交易路由器可以按百分比在单元之间转移流量，使我们能够逐步排空单元进行维护、在部分负载下验证恢复中的单元，或在事件期间更安全地响应。

Minimal Dependencies at the Edge边缘的最小依赖

With the Global Transaction Router at the edge, it’s a critical service providing connectivity, routing, and resiliency. To ensure its availability, we aim to keep dependencies within this system as small as possible.由于全局交易路由器位于边缘，它是一个提供连接、路由和弹性的关键服务。为了确保其可用性，我们力求使该系统内的依赖尽可能小。

The closer to the edge, the fewer dependencies we aim for.越靠近边缘，我们追求的依赖越少。

But we don’t just reduce the dependencies; we also aim to keep them out of the critical path.但我们不仅减少依赖；我们还力求将它们排除在关键路径之外。

If our logging infrastructure becomes unavailable, we don’t want that to impact the ability to process transactions. We do this by using an asynchronous logger configured with a buffer truncation policy, so if the buffer is full, we drop logs instead of blocking transaction processing.如果我们的日志基础设施不可用，我们不希望这影响处理交易的能力。我们通过使用配置了缓冲区截断策略的异步记录器来实现这一点，因此如果缓冲区已满，我们会丢弃日志而不是阻塞交易处理。

If our configuration service becomes unavailable, we want to continue running with the last known configuration. For this, we maintain an in-memory configuration that is updated asynchronously, so if the configuration service becomes unavailable, we can continue running with the last known configuration until it becomes available again and we can pull the latest configuration.如果我们的配置服务不可用，我们希望继续使用最后已知的配置运行。为此，我们维护一个异步更新的内存配置，因此如果配置服务不可用，我们可以继续使用最后已知的配置运行，直到它再次可用并可以拉取最新配置。

In this diagram: The edge path stays thin and resilient by handling logging and configuration asynchronously instead of letting those dependencies block transactions.在此图中：边缘路径通过异步处理日志和配置而不是让这些依赖阻塞交易来保持精简和弹性。

Keeping dependencies out of the critical path reduces failure points. This requires deliberate trade-offs: accepting degraded non-critical functionality (logging, metrics) to preserve transaction processing.将依赖排除在关键路径之外减少了故障点。这需要深思熟虑的权衡：接受降级的非关键功能（日志、指标）以保留交易处理。

Summary总结

In distributed payments systems, resiliency isn’t achieved through monitoring and retries alone—it’s achieved by defining clear failure boundaries and enforcing them through design.在分布式支付系统中，弹性不仅仅通过监控和重试实现——而是通过定义清晰的故障边界并通过设计强制执行来实现。

By organizing our core payments ecosystem into isolated, independently recoverable cells, we transform major failures into controlled routing decisions. Locality, deterministic routing, idempotent processing, and strict boundary enforcement work together to ensure growth and change don’t increase risk.通过将核心支付生态系统组织成隔离、可独立恢复的单元，我们将重大故障转化为受控的路由决策。本地化、确定性路由、幂等处理和严格的边界强制执行共同确保增长和变化不会增加风险。

This discipline underpins our cell-based architecture, enabling us to operate a global payments platform with low latency and high resiliency—principles that continue shaping our evolution.这种纪律支撑着我们的基于单元的架构，使我们能够以低延迟和高弹性运营全球支付平台——这些原则继续塑造着我们的演进。