Introducing Meerkat: an experiment in global consensus隆重介绍 Meerkat：一项全球共识实验

Many internal services at Cloudflare need to read and modify the same control-plane state from across our 330+ global data centers. They need guarantees that different readers never see inconsistent state, and that the system remains available for writes even when some data centers or links fail.Cloudflare 的许多内部服务需要从我们全球 330 多个数据中心读取和修改相同的控制平面状态。它们需要确保不同的读取者永远不会看到不一致的状态，并且即使在某些数据中心或链路发生故障时，系统仍能保持写入可用性。

But Cloudflare’s network runs across the entire Internet, and the Internet is an unpredictable place. Servers and data centers go down. Queues fill up. Links and cables get cut. These conditions make it difficult to run a globally available data system that guarantees strong consistency (e.g., that all readers are guaranteed to read all prior writes) because hostile conditions hinder distributed system replicas’ ability to reliably synchronize data with one another.但 Cloudflare 的网络覆盖整个互联网，而互联网是一个不可预测的环境。服务器和数据中心会宕机，队列会填满，链路和电缆会被切断。这些情况使得运行一个保证强一致性（例如，确保所有读取者都能读取到所有先前的写入）的全球可用数据系统变得困难，因为恶劣的条件阻碍了分布式系统副本之间可靠地同步数据。

One way to synchronize data safely despite adverse network conditions is via a consensus algorithm, which allows a set of machines to agree on the same sequence of values, such as key-value store put and get operations, as long as a majority remains alive and able to communicate. 在不利的网络条件下安全同步数据的一种方法是使用共识算法，它允许一组机器就相同的值序列（例如键值存储的 put 和 get 操作）达成一致，只要大多数机器保持活跃且能够通信即可。

Unfortunately, commonly deployed consensus algorithms like Raft suffer in wide-area networks like Cloudflare’s because they rely on leaders and timeouts. The leader is the only replica allowed to make writes, and if it fails due to a crash or network degradation, the system becomes unavailable until some other replica times out and a new leader is elected. And these timeout values are hard to configure in networks with unpredictable latencies.遗憾的是，像 Raft 这样广泛部署的共识算法在 Cloudflare 这样的广域网中表现不佳，因为它们依赖于领导者和超时机制。领导者是唯一被允许进行写入的副本，如果它因崩溃或网络降级而失败，系统将变得不可用，直到其他副本超时并选出新的领导者。在延迟不可预测的网络中，这些超时值很难配置。

We have experienced multiple incidents caused by unavailable leaders in consensus-driven systems.我们曾多次经历过因共识驱动系统中的领导者不可用而引发的事故。

And so, for the past year, Cloudflare’s Research team has been building a new distributed consensus service called Meerkat powered by a consensus algorithm called QuePaxa, published in 2023 by Tennage & Băsescu et al. QuePaxa differs from Raft in that all replicas can perform writes at all times, and progress is never halted due to a timeout, which makes it well suited for Cloudflare’s network. We layer applications, like a transactional key-value store and leasing system, atop Meerkat’s consensus log. To our knowledge, this will be the first industrial deployment of QuePaxa at global scale.因此，在过去的一年里，Cloudflare 研究团队一直在构建一个名为 Meerkat 的新型分布式共识服务，它由 Tennage & Băsescu 等人在 2023 年发表的名为 QuePaxa 的共识算法提供支持。QuePaxa 与 Raft 的不同之处在于，所有副本都可以随时执行写入，且进度永远不会因为超时而中断，这使其非常适合 Cloudflare 的网络。我们在 Meerkat 的共识日志之上分层构建了事务性键值存储和租约系统等应用程序。据我们所知，这将是 QuePaxa 在全球范围内的首次工业级部署。

Meerkat is an experimental consensus service that is still in development. It’s being designed initially to manage small pieces of control plane state (e.g., leadership for replicated databases) and so it will be kept internal-only for the immediate future. This post introduces Meerkat and lays the groundwork for the Meerkat-related blog posts to come. Meerkat 是一项尚处于开发阶段的实验性共识服务。它最初的设计目的是管理小规模的控制平面状态（例如，副本数据库的领导权），因此在近期内它将仅供内部使用。本文将介绍 Meerkat，并为后续相关的博客文章奠定基础。

Copy linkWhat we need from a global control-plane data system复制链接我们对全球控制平面数据系统的需求

Many Cloudflare services read and write control-plane data, data that helps those services operate correctly, from multiple machines distributed all over the world. One example of control-plane data is placement information: where certain resources (like an AI model instance) are stored. Another example is leadership information: which machine is currently allowed to perform writes to a database. 许多 Cloudflare 服务需要从分布在世界各地的多台机器上读取和写入控制平面数据，这些数据有助于这些服务正确运行。控制平面数据的一个例子是放置信息：某些资源（如 AI 模型实例）存储在哪里。另一个例子是领导权信息：当前允许哪台机器对数据库执行写入操作。

Control-plane data must be both strongly consistent and accessible despite particular kinds of faults.控制平面数据必须既具备强一致性，又能在特定故障下保持可访问性。

In this section we precisely describe our consistency and fault tolerance requirements for a Cloudflare consensus service. We use a key-value store for a running example of an application running atop our consensus service, though other applications (e.g., distributed leases/locks) are possible.在本节中，我们将精确描述 Cloudflare 共识服务的各种一致性和容错要求。我们以运行在共识服务之上的键值存储为例，尽管其他应用程序（如分布式租约/锁）也是可行的。

Copy linkStrong consistency复制链接强一致性

A distributed data system’s consistency level describes what kinds of weird behavior the system is allowed to exhibit when it receives concurrent reads and writes. Consider a distributed key-value store that stores a single numeric value x = 6 across multiple nodes. Also consider the following sequence of writes. These writes are submitted to different nodes on a best-effort basis, and could arrive in any order: 分布式数据系统的一致性级别描述了当系统接收到并发读写时，允许表现出什么样的异常行为。考虑一个在多个节点上存储单个数值 x = 6 的分布式键值存储。再考虑以下写入序列。这些写入以尽力而为的方式提交给不同的节点，并且可能以任何顺序到达：

1. x = x + 1x = x + 1
2. x = x / 2x = x / 2

A system’s consistency level tells you what values of x a client might see when reading x after these writes. Consider the following sequence of operations and the possible execution orders under different consistency levels:系统的一致性级别告诉您，客户端在这些写入之后读取 x 时可能会看到什么值。考虑以下操作序列以及在不同一致性级别下可能的执行顺序：

In a weak consistency level, writes can be re-ordered. In a stronger consistency model, writes can’t be reordered, but reads can. In the strongest possible consistency level, the operations are ordered exactly as they occurred in real time. This property is called linearizability.在弱一致性级别中，写入可以重新排序。在更强的一致性模型中，写入不能重新排序，但读取可以。在最强的一致性级别中，操作的顺序与它们在现实时间中发生的顺序完全一致。此属性称为线性一致性。

At Cloudflare, many services want linearizability. Unlike weaker forms of consistency, linearizability relieves programmers from thinking about all the weird behaviors the data systems might exhibit. Instead, they can reason about the distributed system like they reason about local memory on a single-threaded machine: all reads after a write will see that write. For additional reading material on the dangers of weak consistency, check out this post by Marc Brooker.在 Cloudflare，许多服务需要线性一致性。与较弱的一致性形式不同，线性一致性使程序员无需考虑数据系统可能表现出的所有奇怪行为。相反，他们可以像在单线程机器上处理本地内存一样来推导分布式系统：写入后的所有读取都将看到该写入。有关弱一致性危险的更多阅读材料，请查看 Marc Brooker 的这篇文章。

(If you’re wondering, Meerkat’s key-value store also provides serializability, which we’ll write about in a future post.)（如果您感到好奇，Meerkat 的键值存储还提供可串行化，我们将在以后的文章中对此进行介绍。）

Copy linkFault tolerance复制链接容错性

A system’s level of fault tolerance describes what kinds of faults the system can handle before catastrophes happen. Catastrophes are typically violations of properties the system aims to uphold, e.g., that two consecutive reads without an intervening write for the same key never see different values, or that the system remains available for writes. The faults include network failures or delays, machine crashes, and machine restarts. A system will typically explicitly handle some faults but not others (you can’t handle all faults, as the universe could always reach heat-death). For example, some key-value stores might guarantee to remain available for writes as long as two-thirds of the machines in the system can communicate and don’t crash, but make no promises if a machine is compromised and starts sending malicious messages.系统的容错级别描述了系统在发生灾难前可以处理哪些类型的故障。灾难通常是系统旨在维护的属性遭到破坏，例如，同一键在两次连续读取之间没有中间写入，却看到了不同的值，或者系统无法保持写入可用性。故障包括网络故障或延迟、机器崩溃和机器重启。系统通常会显式处理某些故障，而不处理其他故障（您无法处理所有故障，因为宇宙终将走向热寂）。例如，某些键值存储可能保证只要系统中三分之二的机器能够通信且不崩溃，就保持写入可用，但如果机器被入侵并开始发送恶意消息，则不作任何承诺。

Our desired fault tolerance properties are as follows:我们期望的容错属性如下：

First, the data system should remain available for writes and reads from a client located in any of our data centers as long as the following are true:首先，只要满足以下条件，数据系统应保持对位于我们任何数据中心内的客户端的读写可用性：

1. A majority of the machines in our system are alive and can communicate with one another. (Formally, we tolerate f faults in a system of 2f + 1 machines).系统中的大多数机器处于活跃状态且能够相互通信。（形式上，我们在 2f + 1 台机器的系统中容忍 f 个故障）。
2. The client can contact any machine in the system that is connected to a majority of live machines.客户端可以联系系统中任何连接到大多数活跃机器的机器。

This means that a single failed machine, or network degradation on a single link, does not affect availability of the system. This property is not provided by Raft-based systems, as we’ll see later.这意味着单台机器故障或单条链路上的网络降级不会影响系统的可用性。正如我们稍后将看到的，基于 Raft 的系统不具备此属性。

Second, the data system remains correct as long as no actor in the system is actively malicious (and, of course, there are no bugs). We define correctness in terms of consensus safety later, but loosely speaking this means no two up-to-date machines will ever disagree about the world (e.g., one thinks that key1=1 while another thinks that key1=2).其次，只要系统中没有参与者是主动恶意的（当然，也没有 bug），数据系统就能保持正确。我们稍后会根据共识安全性来定义正确性，但简单来说，这意味着没有任何两台最新的机器会对世界状态产生分歧（例如，一台认为 key1=1，而另一台认为 key1=2）。

To summarize, the system must remain correct even if machines crash, machines restart, networks fail or degrade, data centers go down, and more (though we, like Raft-based systems, do not handle Byzantine faults).总之，即使机器崩溃、机器重启、网络失败或降级、数据中心宕机等，系统也必须保持正确（尽管我们和基于 Raft 的系统一样，不处理拜占庭故障）。

Copy linkIntroducing Meerkat复制链接隆重介绍 Meerkat

Meerkat is a consensus service upon which we can build applications that exhibit the above properties (strong consistency and fault tolerance) like a key-value (KV) store. To understand how Meerkat works, we first outline Meerkat’s general architecture, and then describe how Meerkat’s choice of consensus algorithm helps provide strong consistency and fault tolerance.Meerkat 是一个共识服务，我们可以在其上构建具有上述属性（强一致性和容错性）的应用程序，例如键值 (KV) 存储。为了理解 Meerkat 的工作原理，我们首先概述 Meerkat 的通用架构，然后描述 Meerkat 选择的共识算法如何帮助提供强一致性和容错性。

Developers of services using Meerkat request a cluster of Meerkat replicas. Each replica is connected to every other replica. Each replica participates in the consensus algorithm and can receive both reads and writes. The developer can specify which data centers are allowed to host their replicas, and Meerkat places them automatically.使用 Meerkat 的服务开发人员会请求一组 Meerkat 副本。每个副本都连接到其他所有副本。每个副本都参与共识算法，并且可以接收读写请求。开发人员可以指定哪些数据中心允许托管其副本，Meerkat 会自动进行放置。

To interact with their cluster, a developer’s client sends an application-specific request to any replica in the cluster. A single replica may host many kinds of applications, but the simplest one is a key-value store, so the simplest application-specific request type is a KV get or put. The replica responds to the request with an application-specific response (e.g., the records requested with the get). Note that KV reads (gets) are guaranteed to read up-to-date information.为了与集群交互，开发人员的客户端会向集群中的任何副本发送特定于应用程序的请求。单个副本可能托管多种类型的应用程序，但最简单的是键值存储，因此最简单的特定于应用程序的请求类型是 KV get 或 put。副本会以特定于应用程序的响应（例如，get 请求的记录）来响应请求。请注意，KV 读取 (gets) 保证能读取到最新信息。

Copy linkMeerkat’s log复制链接 Meerkat 的日志

Under the hood, the replica translates application requests (e.g., get and put) into log events. That replica distributes each log event to all other replicas using a consensus algorithm such that all replicas maintain the exact same log of events (in reality, a replica may lag behind, but shall never record different entries). These events are arbitrary — Meerkat’s core doesn’t care what’s in them. Meerkat applications care about log event contents. Each Meerkat replica “hosts” many Meerkat applications (e.g., key-value store) that read the log events and construct state. (Note that each replica belongs to exactly one cluster.)在底层，副本将应用程序请求（例如 get 和 put）转换为日志事件。该副本使用共识算法将每个日志事件分发给所有其他副本，以便所有副本维护完全相同的事件日志（实际上，副本可能会滞后，但绝不会记录不同的条目）。这些事件是任意的——Meerkat 的核心并不关心它们的内容。Meerkat 应用程序关心的是日志事件的内容。每个 Meerkat 副本都“托管”许多 Meerkat 应用程序（例如键值存储），这些应用程序读取日志事件并构建状态。（请注意，每个副本仅属于一个集群。）

For instance, the KV Meerkat application constructs an in-memory key-value store from the log events. So when a client sends a write like put k1 v1, the receiving replica places that write into a log event and distributes it to all replicas. If someone else subsequently writes put k1 v11 to a different replica, this event is also distributed to all replicas. Since all functioning replicas have the same log, those replicas can apply the operations in the log in sequence to construct the exact same state. Note that get requests also create distributed log events (for linearizability, as explained in the next section).例如，KV Meerkat 应用程序从日志事件中构建内存键值存储。因此，当客户端发送像 put k1 v1 这样的写入时，接收副本会将该写入放入日志事件并分发给所有副本。如果随后有人向不同的副本写入 put k1 v11，此事件也会分发给所有副本。由于所有正常运行的副本都有相同的日志，这些副本可以按顺序应用日志中的操作，从而构建完全相同的状态。请注意，get 请求也会创建分布式日志事件（为了线性一致性，如下一节所述）。

Here is an example of how a replica’s KV store is updated as it receives log events:以下是副本的 KV 存储在接收日志事件时如何更新的示例：

Copy linkHow Meerkat’s log enables strong consistency复制链接 Meerkat 的日志如何实现强一致性

Meerkat guarantees that if one client executes put k1 v1, a second client subsequently executes put k1 v11, and a third client subsequently executes get k1 (with a consistent read), they will always read v11. It guarantees this even if each request is submitted to a different replica, and those replicas are distributed randomly across the world. This is linearizability. To see how Meerkat guarantees this, we must examine Meerkat’s log in more detail.Meerkat 保证，如果第一个客户端执行 put k1 v1，第二个客户端随后执行 put k1 v11，第三个客户端随后执行 get k1（进行一致性读取），他们将始终读取到 v11。即使每个请求都提交给不同的副本，且这些副本随机分布在世界各地，它也能保证这一点。这就是线性一致性。要了解 Meerkat 如何保证这一点，我们必须更详细地检查 Meerkat 的日志。

The Meerkat log is a sequence of slots. A slot is a box that can contain an event or not. A slot that contains an event is called a decided slot. All slots in the log are decided except the last slot, which is currently being decided. One of Meerkat’s invariants is that if any two replicas decide on the value for a slot, those values are the same. In other words, no two replicas will ever disagree on the value of a decided slot (though one replica may think the last slot is empty while another does not). This property helps guarantee the desired properties we described in the previous section.Meerkat 日志是一系列槽位 (slots)。槽位是一个可以包含或不包含事件的容器。包含事件的槽位称为已决槽位 (decided slot)。日志中的所有槽位都是已决的，除了最后一个正在决定的槽位。Meerkat 的不变量之一是，如果任何两个副本对某个槽位的值达成一致，那么这些值就是相同的。换句话说，没有两个副本会对已决槽位的值产生分歧（尽管一个副本可能认为最后一个槽位是空的，而另一个则不然）。此属性有助于保证我们在上一节中描述的所需属性。

To decide on the value of the last (empty) slot in the log, Meerkat replicas run a distributed consensus algorithm. A consensus algorithm allows a set of machines communicating over a network to agree on a decided slot value. Our consensus algorithm works as long as a majority of replicas (more than half) are alive.为了决定日志中最后一个（空）槽位的值，Meerkat 副本运行分布式共识算法。共识算法允许通过网络通信的一组机器就已决槽位值达成一致。只要大多数副本（超过一半）处于活跃状态，我们的共识算法就能正常工作。

So if the log currently contains two entries, and a client submits put k1 v11 to a replica, that replica triggers a consensus algorithm for slot 3. But another client might have submitted put k1 v111 to a different replica for slot 3. The consensus algorithm ensures that only one such proposal for slot 3 wins out. Specifically, it ensures that at least a majority of replicas agree on the same proposal, deciding it for slot 3. The non-majority can never decide a different proposal, but might miss the fact that slot 3 has been decided at all. 因此，如果日志当前包含两个条目，并且客户端向某个副本提交 put k1 v11，该副本会触发槽位 3 的共识算法。但另一个客户端可能已经向不同的副本提交了槽位 3 的 put k1 v111。共识算法确保只有一个针对槽位 3 的提案获胜。具体来说，它确保至少大多数副本同意相同的提案，并将其确定为槽位 3 的值。少数派永远无法决定不同的提案，但可能根本不知道槽位 3 已经决定。

To see how this provides linearizability for our key-value store, consider a write followed by a read. One replica Z proposes put k1 v11 and this proposal is decided at slot 3 by a majority of replicas, but NOT replica Y. Subsequently, a reader executes get k1 on replica Y. Replica Y believes slot 3 is empty, so proposes get k1 at slot 3. Critically, a majority of replicas will not agree to place that event at slot 3, because that slot has already been decided. They will force replica Y to decide (by receiving older decisions) put k1 v11 in slot 3, and to propose get k1 for slot 4, thus linearizing the read after the write in the log. (And if that replica can’t contact a majority, it will be unable to complete the read.)为了了解这如何为我们的键值存储提供线性一致性，考虑一次写入后紧跟一次读取。副本 Z 提议 put k1 v11，该提案在槽位 3 被大多数副本决定，但副本 Y 除外。随后，读取者在副本 Y 上执行 get k1。副本 Y 认为槽位 3 是空的，因此提议在槽位 3 执行 get k1。关键在于，大多数副本不会同意将该事件放入槽位 3，因为该槽位已经决定了。它们会强制副本 Y 决定（通过接收较早的决定）槽位 3 中的 put k1 v11，并为槽位 4 提议 get k1，从而在日志中将读取操作线性化在写入之后。（如果该副本无法联系到大多数副本，它将无法完成读取。）

Copy linkHow Meerkat’s consensus algorithm provides higher availability than Raft复制链接 Meerkat 的共识算法如何提供比 Raft 更高的可用性

Deciding on log entries requires a distributed consensus algorithm. But which one? All valid consensus algorithms would provide the required consistency and correctness guarantees, but not all provide the same availability guarantees. 决定日志条目需要分布式共识算法。但选哪一个呢？所有有效的共识算法都能提供所需的一致性和正确性保证，但并非所有算法都能提供相同的可用性保证。

Specifically, many algorithms that rely on authoritative leaders do not provide our desired availability guarantees, because they can become unavailable when a single machine experiences issues. Consider Raft, one of the most well-known and probably the most implemented consensus algorithm. Raft relies on an authoritative leader: the only replica in the cluster that can drive consensus. As a result, all writes get forwarded to the leader. This design choice helps make Raft “understandable” and, coupled with leases, can make leader-served reads automatically linearizable (since they’re guaranteed to be up-to-date). But it also adds a single point of (temporary) failure.具体来说，许多依赖权威领导者的算法无法提供我们期望的可用性保证，因为当单台机器出现问题时，它们可能会变得不可用。考虑 Raft，它是最著名、可能也是实现最广泛的共识算法之一。Raft 依赖于权威领导者：集群中唯一能推动共识的副本。因此，所有写入都会转发给领导者。这种设计选择有助于使 Raft “易于理解”，并且结合租约，可以使领导者服务的读取自动实现线性一致性（因为它们保证是最新的）。但这也增加了一个（临时的）单点故障。

In general, there are two problems with authoritative leaders. First, if the leader goes down, the system becomes unavailable (all writes block) until a new leader is elected. This is unacceptable for Meerkat. Second, if the leader stays up but slows down, either because it is overloaded or there are network delays, then performance degrades. The leader is a bottleneck because there is no alternative way to perform writes. 总的来说，权威领导者存在两个问题。首先，如果领导者宕机，系统将变得不可用（所有写入阻塞），直到选出新领导者。这对 Meerkat 来说是不可接受的。其次，如果领导者保持在线但变慢（无论是由于过载还是网络延迟），性能就会下降。领导者是一个瓶颈，因为没有其他方式可以执行写入。

The first problem is exacerbated in wide-area networks. Consider that when a leader goes down, most algorithms choose a new leader using timeouts: if a non-leader replica hasn’t heard from the leader in some amount of time, they propose themselves as the leader. At that point, the old leader has been deposed, and the system cannot accept writes until a new leader has been elected. The problem is that when the timeout is shorter than the network delay between the original leader and that replica, replicas will constantly be timing out and thus blocking writes. And when the timeout is too long, the system reacts slowly to a failed leader, during which writes are also blocked. Plus, if multiple replicas propose themselves as leader at the same time, their “campaigns” can interfere with each other, causing them to constantly re-propose themselves as leader — all the while blocking writes. We have seen these exact issues with Cloudflare’s systems that use Raft because our wide-area network delays can and do vary wildly, making tuning timeouts especially difficult.第一个问题在广域网中会加剧。考虑一下，当领导者宕机时，大多数算法会使用超时机制选择新领导者：如果非领导者副本在一段时间内没有收到领导者的消息，它们就会提议自己为领导者。此时，旧领导者已被废黜，在选出新领导者之前，系统无法接受写入。问题在于，当超时时间短于原始领导者与该副本之间的网络延迟时，副本会不断超时从而阻塞写入。而当超时时间过长时，系统对故障领导者的反应又很慢，期间写入也会阻塞。此外，如果多个副本同时提议自己为领导者，它们的“竞选”会相互干扰，导致它们不断地重新提议自己为领导者——在此期间写入一直被阻塞。我们在使用 Raft 的 Cloudflare 系统中就看到了这些确切的问题，因为我们的广域网延迟可能会且确实会发生剧烈波动，这使得调整超时时间变得格外困难。

We chose a different consensus algorithm for Meerkat, called QuePaxa, that aims to avoid the “tyranny of timeouts” imposed by protocols like Raft. QuePaxa is a subtle protocol, but here are the highlights. A client can contact any replica, and that replica can drive consensus for the latest slot. There is a leader, but it is not required — its only advantage is that it can drive consensus with fewer round trips (one) than other replicas (3+). Critically, clients are free to contact multiple replicas concurrently for the same proposal, to increase the chance of the proposal being successful. Concurrent proposals do not destructively interfere:  replicas work together to decide one of the proposed values.我们为 Meerkat 选择了一种不同的共识算法，名为 QuePaxa，旨在避免像 Raft 这样的协议所带来的“超时暴政”。QuePaxa 是一个微妙的协议，但以下是其重点。客户端可以联系任何副本，该副本可以为最新的槽位推动共识。虽然存在领导者，但它并非必需——其唯一的优势在于，它推动共识所需的往返次数（一次）比其他副本（3 次以上）少。关键在于，客户端可以自由地为同一个提案同时联系多个副本，以增加提案成功的机会。并发提案不会产生破坏性干扰：副本会协同工作以决定提议值中的一个。

In short, QuePaxa has three advantages over Raft for our purposes:简而言之，对于我们的目的，QuePaxa 比 Raft 有三个优势：

1. Because there is no required leader, the system never becomes unavailable or degraded due to a single replica (the leader) being down, unavailable, or degraded. Clients can perform writes as long as they can contact some healthy replica (anywhere in the world). 由于不需要领导者，系统永远不会因为单个副本（领导者）宕机、不可用或降级而变得不可用或性能下降。只要客户端能联系到任何健康的副本（世界任何地方），就可以执行写入。
2. Because there is no leader, there are no leader elections that degrade the system. And concurrent proposals made by different replicas constructively interfere, unlike Raft’s leadership elections. This is ideal for Cloudflare’s network, in which latencies can vary wildly.由于没有领导者，就不会有降低系统性能的领导者选举。与 Raft 的领导者选举不同，不同副本发出的并发提案会产生建设性干扰。这对于延迟可能剧烈波动的 Cloudflare 网络来说是理想的选择。
3. QuePaxa was designed for a less reliable network environment (“asynchrony”), and for networks in which an imaginary adversary can launch targeted attacks on replica connections. The authors found that it maintains much higher (~10x) throughput than Raft and Multi-Paxos during such conditions. These conditions more accurately resemble our own network than the conditions other algorithms assume.QuePaxa 是为可靠性较低的网络环境（“异步”）以及可能存在虚构对手对副本连接发起针对性攻击的网络而设计的。作者发现，在这种条件下，它的吞吐量比 Raft 和 Multi-Paxos 高得多（约 10 倍）。这些条件比其他算法假设的条件更准确地模拟了我们自己的网络。

We will save the full description of QuePaxa for another post. Major shoutout to the authors of the QuePaxa paper for being available for feedback and questions about their work.我们将把 QuePaxa 的完整描述留到另一篇文章中。特别感谢 QuePaxa 论文的作者，他们随时为有关其工作的反馈和问题提供支持。

Copy linkAssessing Meerkat’s performance 复制链接评估 Meerkat 的性能

Meerkat has limitations. It is not designed to create general-purpose data systems like databases.Meerkat 有局限性。它不是为创建数据库等通用数据系统而设计的。

All consensus algorithms come with a cost: lots of round-trips. QuePaxa in particular takes one to three round trips (usually, although it can take more) between the initial proposer and a majority of replicas to decide on a proposal and add an event to the log. The difference is with the leader. It takes one if the leader is proposing (+ an extra broadcast to notify replicas of the decision) and three if a non-leader is proposing (+ extra broadcast). If multiple replicas make proposals at the same time, it can take more. These communication costs point to the important performance limitation of consensus algorithms in general: proposal decision latency is proportional to the latency between some majority of replicas. So if your replicas are far from one another, latency will increase — there’s no getting around that.所有共识算法都有代价：大量的往返通信。特别是 QuePaxa，在初始提议者和大多数副本之间需要一到三次往返（通常，尽管可能更多）来决定提案并将事件添加到日志中。区别在于领导者。如果领导者提议，则需要一次（外加一次广播以通知副本决定）；如果非领导者提议，则需要三次（外加额外广播）。如果多个副本同时进行提议，则可能需要更多。这些通信成本指出了共识算法在通用性能上的重要局限性：提案决定延迟与某些大多数副本之间的延迟成正比。因此，如果您的副本彼此相距遥远，延迟就会增加——这是无法避免的。

At first glance, it seems Meerkat’s write and read latency will be quite poor. Especially if all writes and reads (for consistency) must go through the log, and thus require so many round trips.乍一看，Meerkat 的读写延迟似乎会很差。特别是如果所有的读写（为了一致性）都必须经过日志，从而需要如此多的往返次数。

But there are a few ways to squeeze better performance out of Meerkat: 但有几种方法可以从 Meerkat 中挤出更好的性能：

1. Because developers have control over where their replicas live, they can choose to move replicas closer together, reducing round-trip latency (only applicable for services that don’t need truly global distribution).由于开发人员可以控制副本的位置，他们可以选择将副本移得更近，从而减少往返延迟（仅适用于不需要真正全球分布的服务）。
2. Writes can be batched. So if a replica receives 10 writes in a span of 10ms, it can place all of those in a single proposal, improving throughput.写入可以进行批处理。因此，如果副本在 10 毫秒内收到 10 次写入，它可以将所有这些写入放入单个提案中，从而提高吞吐量。
3. Not all reads must trigger a consensus round. If a developer is OK with reading stale (but never inconsistent) data, they can read from any replica’s local data.并非所有的读取都必须触发共识轮次。如果开发人员可以接受读取陈旧（但绝不不一致）的数据，他们可以从任何副本的本地数据中读取。
4. Multiple operations can be bundled into a single consensus round. For instance, our key-value store supports compare-and-swap-style writes in which writes execute only if a value has not changed since it was read. (In fact, it supports general transactions.)多个操作可以捆绑到单个共识轮次中。例如，我们的键值存储支持比较并交换 (compare-and-swap) 风格的写入，即仅当值自读取以来未发生更改时才执行写入。（事实上，它支持通用事务。）

Still, Meerkat’s fundamental latency limitations remain, especially when it is run at global scale, as it was designed to do. These limitations make it perfect, in the short term, for control plane information that is written infrequently but must remain consistent.尽管如此，Meerkat 的基本延迟限制依然存在，尤其是在按设计进行全球规模运行时。这些限制使其在短期内非常适合写入频率较低但必须保持一致的控制平面信息。

Copy linkWhat’s next复制链接接下来要做什么

Meerkat is not deployed to production, but we have run multiple proofs-of-concept with up to 50 replicas distributed around the world, to great success. Leaders in our proof-of-concept clusters constantly fail, and the cluster keeps operating with no increase in error-rate.Meerkat 尚未部署到生产环境，但我们已经运行了多个概念验证，最多有 50 个分布在世界各地的副本，并取得了巨大成功。我们概念验证集群中的领导者会不断失败，而集群在错误率没有增加的情况下持续运行。

We have a lot more to say about Meerkat. Over the course of the next year we’ll be writing Meerkat posts that discuss how QuePaxa really works, how we’re formally verifying some of our Rust implementation, how bootstrapping and cluster management works, how we find optimal replica placement, how we use deterministic simulation testing to find bugs, and more. We’ll also be preparing a manuscript for peer-review!关于 Meerkat，我们还有很多话要说。在接下来的一年里，我们将撰写有关 Meerkat 的文章，讨论 QuePaxa 的实际工作原理、我们如何对 Rust 实现进行形式化验证、引导和集群管理的工作方式、我们如何找到最佳副本放置位置、我们如何使用确定性模拟测试来发现 bug 等等。我们还将准备一份供同行评审的手稿！

Follow along on the Cloudflare Blog as Meerkat progresses, and check out more of our projects at Cloudflare Research.请关注 Cloudflare 博客以了解 Meerkat 的进展，并查看我们在 Cloudflare Research 的更多项目。