Distributed Locking: A Practical Guide分布式锁:实用指南
Was your data ever mysteriously overwritten? No? Think again. Have you noticed conflicting updates to the same data? Still nope? Lucky you! I had cases when my data changed, and it looked like Marty was travelling back to the future. Once it was correct, then it was wrong, then correct again. Most of the time, twin services were writing to the sample place.你的数据是否曾被莫名其妙地覆盖过?没有?再想想。你有没有注意到同一份数据出现了冲突的更新?还是没有?那你真幸运!我遇到过数据被更改的情况,看起来就像马蒂(Marty)在进行时空穿梭。数据一会儿正确,一会儿错误,然后又变回正确。大多数情况下,都是两个孪生服务在同一个地方进行写入操作。
In distributed systems, coordination is crucial. Whenever you have multiple processes (or services) that could update or read the same data simultaneously, you risk data corruption, race conditions, or unwanted duplicates. The popular solution is to use distributed locks - a mechanism ensuring only one process can operate on a resource at a time.在分布式系统中,协调至关重要。每当有多个进程(或服务)可能同时更新或读取同一份数据时,你就会面临数据损坏、竞态条件或产生不必要重复数据的风险。常见的解决方案是使用分布式锁——这是一种确保同一时间只有一个进程能操作特定资源的机制。
A distributed lock ensures that if one actor (node, service instance, etc.) changes a shared resource—like a database record, file, or external service—no other node can step in until the first node is finished. While the concept is straightforward, implementing it across multiple machines requires careful design and a robust failure strategy.分布式锁确保了如果一个参与者(节点、服务实例等)正在更改共享资源(如数据库记录、文件或外部服务),那么在第一个节点完成操作之前,其他任何节点都无法介入。虽然这个概念很简单,但在多台机器上实现它需要精心的设计和稳健的故障处理策略。
Today, we’ll try to discuss it and:今天,我们将探讨以下内容:
-
Explain why distributed locking is needed in real-world scenarios.解释为什么在现实场景中需要分布式锁。
-
Explore how popular tools and systems implement locks (Redis, ZooKeeper, databases, Kubernetes single-instance setups, etc.).探讨主流工具和系统(如 Redis、ZooKeeper、数据库、Kubernetes 单实例部署等)是如何实现锁的。
-
Discuss potential pitfalls—like deadlocks, lock expiration, and single points of failure—and how to address them.讨论潜在的陷阱——如死锁、锁过期和单点故障——以及如何解决这些问题。
By the end, you should have a decent grasp of distributed locks, enough to make informed decisions about whether (and how) to use them in your architecture.读完本文,你应该能对分布式锁有一个不错的理解,足以让你在架构中决定是否(以及如何)使用它们。
1. Why Distributed Locks Matter1. 为什么分布式锁很重要
When you scale an application to multiple machines or microservices, each one might update the same resource simultaneously. That can lead to writers overwriting each other. It’s the classic concurrency problem: who’s in charge of updating shared data? Without a strategy to coordinate these updates, you risk inconsistent results.当你将应用程序扩展到多台机器或多个微服务时,每个实例都可能同时更新同一资源。这可能导致写入操作相互覆盖。这是经典的并发问题:谁负责更新共享数据?如果没有协调这些更新的策略,你就会面临数据结果不一致的风险。
For example, you might have:例如,你可能会遇到:
-
Multiple Writers are updating the same table row. One process believes it has the latest data and changes, but another process overwrites it moments later.
That’s a common issue when updating read models. We need to ensure a single subscriber for a specific read model. When multiple microservices listen to an event stream but only one should update a secondary database, a lock ensures they don’t collide.多个写入者正在更新同一张表行。一个进程认为自己拥有最新的数据并进行了更改,但另一个进程随后立即将其覆盖。 在更新读模型时,这是一个常见问题。我们需要确保特定读模型只有一个订阅者。当多个微服务监听同一个事件流,但只有一个服务应该更新辅助数据库时,锁可以确保它们不会发生冲突。 -
At-Least-Once Messaging leading to repeated deliveries of the same message, with multiple consumers each processing it as if it were unique.“至少一次”(At-Least-Once)消息传递导致同一消息被重复投递,多个消费者将其视为唯一消息进行处理。
-
A Global Resource that can only support one writer at a time, such as a large file or an external API endpoint.全局资源,例如大文件或外部 API 端点,一次只能支持一个写入者。
-
Scheduled jobs: If you schedule a job on multiple nodes to ensure reliability, you still want only the first node that grabs the lock to run it, preventing duplicates.定时任务:如果你为了可靠性在多个节点上安排了同一个任务,你仍然希望只有第一个抢到锁的节点来执行它,从而防止重复执行。
A distributed lock is the simplest way to say, “Only one node can modify this resource until it’s finished.” Other nodes wait or fail immediately rather than attempting parallel writes.分布式锁最简单的定义就是:“在资源处理完成前,只允许一个节点修改它。”其他节点要么等待,要么立即失败,而不是尝试并行写入。
It can also be seen as Leader Election Lite: If you only need “one node at a time” for a particular task, a plain lock is sometimes enough - there is no need for a full-blown leader election framework.它也可以被视为简易版的领导者选举:如果你在特定任务中只需要“一次一个节点”,那么普通的锁有时就足够了——不需要动用完整的领导者选举框架。
Without locks, you can get unpredictable states - like a read model flipping from correct to incorrect or a file partially overwritten by multiple workers. Locks sacrifice a bit of parallelism for the certainty that no two nodes update the same resource simultaneously. In many cases, that’s the safest trade-off, especially if data correctness is paramount.没有锁,你可能会得到不可预测的状态——比如读模型在正确和错误之间反复切换,或者文件被多个工作进程部分覆盖。锁牺牲了一点并行性,以换取“没有两个节点会同时更新同一资源”的确定性。在许多情况下,这是最安全的权衡,尤其是当数据正确性至关重要时。
There are other ways to handle concurrency (like idempotent actions or write-ahead logs). Still, a distributed lock is often the most direct solution when you truly need to avoid simultaneous writes.还有其他处理并发的方法(如幂等操作或预写日志)。尽管如此,当你确实需要避免同时写入时,分布式锁往往是最直接的解决方案。
2. How Locks Typically Work2. 锁通常是如何工作的
-
Lock Request
A node determines it needs exclusive access to a shared resource and communicates with a lock manager (e.g., Redis, ZooKeeper, or a database) that supports TTL or ephemeral locks.锁请求 节点确定其需要对共享资源进行独占访问,并与支持 TTL(生存时间)或临时锁的锁管理器(如 Redis、ZooKeeper 或数据库)进行通信。 -
Acquire Attempt获取尝试
-
If no lock currently exists, the node creates one, for instance by setting a Redis key with a time-to-live or by creating an ephemeral zNode in ZooKeeper.如果当前不存在锁,节点将创建一个锁,例如通过设置带有 TTL 的 Redis 键,或在 ZooKeeper 中创建一个临时 zNode。
-
If another node is holding the lock, this node either waits, fails immediately, or retries, depending on your chosen policy.如果另一个节点持有该锁,当前节点将根据你选择的策略进行等待、立即失败或重试。
-
-
Critical Operation
After acquiring the lock, the node proceeds with the operation that must not be run in parallel—such as updating a database row, writing to a shared file, or sending a limited external API request.关键操作 在获取锁之后,节点继续执行不能并行运行的操作——例如更新数据库行、写入共享文件或发送受限的外部 API 请求。 -
Standard Release
Once finished, the node explicitly removes the lock record (e.g., deleting the Redis key or calling an unlock function). At this point, other contenders are free to acquire the lock.标准释放 一旦完成,节点会显式移除锁记录(例如删除 Redis 键或调用解锁函数)。此时,其他竞争者可以自由获取该锁。 -
Crash or Automatic Release
If the node fails or loses its session, the TTL expires, or the ephemeral mechanism detects the node’s absence. The lock manager then removes the lock, automatically merging into the same path as a standard release so a new node can acquire it. This avoids leaving the system stuck with a “zombie” lock that never gets freed.崩溃或自动释放 如果节点发生故障或失去会话,TTL 将过期,或者临时机制检测到节点不存在。锁管理器随后会移除该锁,自动进入与标准释放相同的路径,以便新节点可以获取它。这避免了系统陷入永远无法释放的“僵尸”锁状态。
The basic flow would look like:基本流程如下所示:
The general lock acquisition workflow通用的锁获取工作流
And the acquisition part with TTL handling:以及处理 TTL 的获取部分:
3. Tools for Distributed Locking3. 分布式锁工具
There are many tools for distributed locking; let’s check the most popular for certain categories.分布式锁工具有很多;让我们来看看特定类别中最流行的几种。
-
Redis (and clones): You store a “lock key” in Redis with a time limit attached. If the lock key doesn’t exist, you create it and claim the lock. Once your task finishes, you remove the key. If your process crashes, Redis removes the key after the time limit expires, so no one stays locked out forever. This method is quick and works well if you already rely on Redis, but you should be cautious about network splits or cluster issues.Redis(及其克隆版):你在 Redis 中存储一个带有时间限制的“锁键”。如果锁键不存在,你就创建它并声明锁。任务完成后,删除该键。如果进程崩溃,Redis 会在时间限制过期后移除该键,因此不会有人永远被锁在外面。这种方法速度快,如果你已经依赖 Redis,它工作得很好,但你应该警惕网络分区或集群问题。
-
ZooKeeper / etcd: These tools consistently keep data across a group of servers. You write a small record saying, “I own this lock,” and if you go offline, the system notices and removes your record automatically. ZooKeeper and etcd are often used for more advanced cluster coordination or leader election, so they’re more complex to set up than Redis. However, they offer strong consistency guarantees.ZooKeeper / etcd:这些工具在服务器组之间保持数据的一致性。你写入一条小记录说“我拥有这个锁”,如果你下线了,系统会注意到并自动移除你的记录。ZooKeeper 和 etcd 通常用于更高级的集群协调或领导者选举,因此它们的设置比 Redis 更复杂。然而,它们提供了强一致性保证。
-
Database Locks: If you have a single database (like Postgres or MySQL) where all your data lives, you can let the database control concurrency. You request a named lock, and the database blocks any other request for that same lock name until you release it. It’s convenient if you already use a single DB for everything, but it may not scale well across multiple databases or regions, and frequent locking could cause contention.数据库锁:如果你有一个单一数据库(如 Postgres 或 MySQL),且所有数据都存放在那里,你可以让数据库控制并发。你请求一个命名锁,数据库会阻塞对该锁名的任何其他请求,直到你释放它。如果你已经为所有内容使用单个数据库,这很方便,但它可能无法跨多个数据库或区域很好地扩展,且频繁锁定可能导致竞争。
-
Kubernetes Single-Instance: Instead of coordinating multiple replicas, you literally run only one replica of your service. Because there’s only one service process, you avoid concurrency altogether at the application level. It’s the simplest approach - no lock code is needed. The downside is that you lose parallel processing and can’t spin up more replicas for high availability or scaling.Kubernetes 单实例:与其协调多个副本,不如只运行一个服务副本。因为只有一个服务进程,你在应用程序层面完全避免了并发。这是最简单的方法——不需要锁代码。缺点是你失去了并行处理能力,且无法为了高可用性或扩展性启动更多副本。
Distributed locks all share a common goal: ensure only one node does a particular thing at any given time. However, each tool mentioned approaches the problem with distinct designs, strengths, and failover behaviours.分布式锁都有一个共同目标:确保在任何给定时间只有一个节点执行特定操作。然而,上述每个工具都以独特的设计、优势和故障转移行为来处理这个问题。
Let’s look at each tool’s big-picture purpose—why you’d even consider it—then move on to how it implements (or approximates) a lock. Lastly, let’s discuss a few technical details that matter once you start coding or troubleshooting.让我们看看每个工具的大局目标——你为什么会考虑它——然后深入探讨它如何实现(或近似实现)锁。最后,讨论一些一旦你开始编码或排查故障时需要注意的技术细节。
3.1 Redis (In-Memory Data Store with TTL-Based Locking)3.1 Redis(基于 TTL 锁的内存数据存储)
Redis runs all commands in a single thread, so there is no risk of two commands interfering partway through. It provides atomic commands to “claim” a key only if it doesn’t exist and supports a time-to-live (TTL). Redis allows multiple nodes to coordinate around a shared key space. Essentially, if one node sets a “lock key” successfully, that key signals to everyone else that the lock is taken.Redis 在单线程中运行所有命令,因此不存在两个命令在执行过程中相互干扰的风险。它提供了原子命令来“声明”一个键(仅在它不存在时),并支持生存时间(TTL)。Redis 允许多个节点围绕共享键空间进行协调。本质上,如果一个节点成功设置了一个“锁键”,该键就会向其他人发出信号,表明锁已被占用。
Redis runs all commands in a single thread, so there is no risk of two commands interfering partway through. If two clients try to create the same lock key simultaneously, Redis processes them in strict sequence: either the first wins or the second does, but never both.Redis 在单线程中运行所有命令,因此不存在两个命令在执行过程中相互干扰的风险。如果两个客户端试图同时创建同一个锁键,Redis 会严格按顺序处理它们:要么第一个成功,要么第二个成功,但绝不会两者同时成功。
To create a lock, you need to create a “lock key” using a Redis command, for instance:要创建锁,你需要使用 Redis 命令创建一个“锁键”,例如:
SET lockKey node123 NX EX 30That looks cryptic, but let’s see what’s happening behind the scenes:这看起来很晦涩,但让我们看看幕后发生了什么:
-
Atomic Key Creation with NX. The NX option in the
SETcommand means “set the key only if it does not already exist.” Redis checks the key’s existence and creates it if none is found. This is all done atomically, so if another client tries the same operation, one will succeed, and the other will fail.使用 NX 的原子键创建。SET 命令中的 NX 选项意味着“仅在键不存在时设置它”。Redis 会检查键的存在性,如果未找到则创建它。这一切都是原子完成的,因此如果另一个客户端尝试相同的操作,一个会成功,另一个会失败。 -
Time-to-Live with EX. The EX option (short for “expire after X seconds”) tells Redis to remove the key automatically after the specified number of seconds. If the client (the lock holder) crashes, once the EX timer hits zero, Redis deletes the key. This prevents the lock from staying locked forever.使用 EX 的生存时间。EX 选项(“expire after X seconds”的缩写)告诉 Redis 在指定的秒数后自动移除键。如果客户端(锁持有者)崩溃,一旦 EX 计时器达到零,Redis 就会删除该键。这防止了锁被永久锁定。
-
Lock Ownership. When a client sets the key (for example,
SET lockKey node123 NX EX 30), if Redis replies with OK, that client is considered to “own” the lock. If Redis replies with nil or an empty response, it means another client already owns it.锁所有权。当客户端设置键时(例如 SET lockKey node123 NX EX 30),如果 Redis 回复 OK,则认为该客户端“拥有”锁。如果 Redis 回复 nil 或空响应,则意味着另一个客户端已经拥有它。 -
Renewals. If you need more time, say your operation runs longer than the initial 30 seconds, you can extend the lock by calling something like
PEXPIRE lockKey newTTL, which resets the timer. Alternatively, you could run anotherSET lockKey node123 NX EX 30if you do it carefully (although the recommended pattern is to check that you still own the lock before renewing).续期。如果你需要更多时间,比如你的操作运行时间超过了最初的 30 秒,你可以通过调用类似 PEXPIRE lockKey newTTL 的命令来延长锁,这会重置计时器。或者,如果操作得当,你可以运行另一个 SET lockKey node123 NX EX 30(尽管推荐的模式是在续期前检查你是否仍然拥有锁)。 -
Releasing the Lock. Once finished, the client deletes the key, typically with
DEL lockKey. This frees the lock immediately so other nodes can acquire it. It’s crucial to verify that you are indeed the one who set the key (sometimes done by storing a unique value in the key, then checking it before deleting).释放锁。一旦完成,客户端通常会使用 DEL lockKey 删除键。这会立即释放锁,以便其他节点可以获取它。验证你确实是设置该键的人至关重要(有时通过在键中存储唯一值,然后在删除前检查它来实现)。 -
Handling Network Partitions. In a simple single-node Redis, a partition can lead to confusion if the lock holder is cut off but still thinks it holds the lock. On the other hand, Redis might expire the key after the TTL, and another client might acquire the lock. To address these edge cases in larger clusters, you can use the Redlock algorithm, which coordinates across multiple Redis instances to reduce the chance of conflicting lock ownership during network splits.处理网络分区。在简单的单节点 Redis 中,如果锁持有者被切断但仍认为自己持有锁,分区可能会导致混乱。另一方面,Redis 可能会在 TTL 过期后删除键,而另一个客户端可能会获取该锁。为了在更大的集群中解决这些边缘情况,你可以使用 Redlock 算法,该算法在多个 Redis 实例间进行协调,以减少网络分区期间冲突锁所有权的可能性。
This approach suffices for a “best effort” distributed lock in most daily scenarios. However, if your environment is prone to frequent network partitions or requires extremely strict guarantees, more advanced configurations or tools might be necessary.这种方法足以应付大多数日常场景中的“尽力而为”的分布式锁。然而,如果你的环境容易出现频繁的网络分区或需要极其严格的保证,可能需要更高级的配置或工具。
Choosing Redis may be a good option if you already run a Redis cluster for caching or need a lightweight solution that’s easy to integrate.如果你已经运行了用于缓存的 Redis 集群,或者需要一个易于集成的轻量级解决方案,选择 Redis 可能是一个不错的选择。
Performance is generally good, though absolute consistency under partitions can require more complex setups.性能通常很好,尽管在分区情况下的绝对一致性可能需要更复杂的设置。
3.2 ZooKeeper / etcd (Strongly Consistent Key-Value Stores with Ephemeral Nodes)3.2 ZooKeeper / etcd(具有临时节点的强一致性键值存储)
ZooKeeper and etcd each run as a cluster of nodes that keep data consistent across a majority of them (a quorum). When you write a small record saying, “I own this lock,” that record is replicated so every node in the cluster knows about it. This consistency helps avoid confusion if one node is temporarily out of sync.ZooKeeper 和 etcd 每个都作为节点集群运行,在大多数节点(法定人数)之间保持数据一致。当你写入一条小记录说“我拥有这个锁”时,该记录会被复制,以便集群中的每个节点都知道它。这种一致性有助于避免在某个节点暂时不同步时产生混乱。
source: https://youtube.com/watch?v=a-zatYN1Lx0来源:https://youtube.com/watch?v=a-zatYN1Lx0
You create something like /locks/lockKey in ZooKeeper or a key in etcd. If there is no record, you can successfully claim the lock. If it already exists, you either wait or fail, depending on your locking recipe.你在 ZooKeeper 中创建诸如 /locks/lockKey 之类的路径,或在 etcd 中创建一个键。如果没有记录,你可以成功声明锁。如果它已经存在,你根据你的加锁配方选择等待或失败。
Releasing Lock释放锁
When you’re done, you delete the ephemeral node or lease key, signalling that source is available. Since it’s ephemeral, the node is removed if you just disconnect cleanly. But if you do explicit cleanup, that’s more predictable.当你完成后,删除临时节点或租约键,表示资源可用。因为它是临时的,如果你干净地断开连接,节点就会被移除。但如果你进行显式清理,结果会更可预测。
Other clients can “watch” that lock path. If the lock holder crashes, ZooKeeper or etcd detects the session loss and removes the node, instantly notifying watchers that the lock is free again. This allows any waiting client to move in and grab the lock right away.其他客户端可以“监视”该锁路径。如果锁持有者崩溃,ZooKeeper 或 etcd 会检测到会话丢失并移除该节点,立即通知监视者锁已释放。这允许任何等待的客户端立即介入并抢占锁。
If a node is isolated from the quorum, ZooKeeper or etcd eventually considers that session dead and removes the ephemeral node. This auto-frees the lock. The newly updated cluster state reflects that the lock is available, so another node can pick it up.如果某个节点与法定人数隔离,ZooKeeper 或 etcd 最终会认为该会话已死并移除临时节点。这会自动释放锁。新更新的集群状态反映出锁可用,因此另一个节点可以获取它。
Why would you Choose ZooKeeper or etcd?你为什么要选择 ZooKeeper 或 etcd?
If your environment already depends on them for cluster metadata or leader election, reusing them for locks is natural. They replicate data across multiple servers, so updates to the lock state are consistent, reducing the risk of split-brain scenarios. The ephemeral mechanism automatically frees locks if a session dies, so you don’t end up with “zombie” locks after crashes.如果你的环境已经依赖它们进行集群元数据或领导者选举,重用它们进行加锁是很自然的。它们在多台服务器上复制数据,因此锁状态的更新是一致的,减少了脑裂场景的风险。临时机制会在会话死亡时自动释放锁,因此你不会在崩溃后留下“僵尸”锁。
They offer stronger consistency guarantees at the cost of heavier operational overhead compared to Redis. Setting up a ZooKeeper or etcd cluster is more complex than a single Redis node. They also come with higher operational overhead, so they’re often chosen by teams already using these tools for other cluster metadata.它们提供了更强的一致性保证,但代价是与 Redis 相比更高的操作开销。设置 ZooKeeper 或 etcd 集群比单个 Redis 节点更复杂。它们也伴随着更高的运维开销,因此通常由已经在其他集群元数据中使用这些工具的团队选择。
3.3 Database Advisory Locks (Using Your Existing Relational DB)3.3 数据库建议锁(使用你现有的关系型数据库)
Sometimes you’d rather avoid setting up additional infrastructure, like Redis or ZooKeeper, and simply rely on the single relational database you already use. Many SQL databases—PostgreSQL, MySQL, SQL Server, and others—provide built-in locking features that can help you coordinate concurrency directly in your existing environment. There are generally two ways to handle locks in a relational database:有时你宁愿避免设置额外的基础设施(如 Redis 或 ZooKeeper),而只是依赖你已经使用的单个关系型数据库。许多 SQL 数据库——PostgreSQL、MySQL、SQL Server 等——都提供内置的锁定功能,可以帮助你在现有环境中直接协调并发。在关系型数据库中处理锁通常有两种方式:
-
row-level locks (like
SELECT ... FOR UPDATE)行级锁(如 SELECT ... FOR UPDATE) -
advisory locks (like
pg_advisory_lockorGET_LOCK)建议锁(如 pg_advisory_lock 或 GET_LOCK)
They serve different needs, but both let you say, “I want exclusive access to something” using your existing database.它们服务于不同的需求,但都允许你使用现有的数据库说:“我想要某物的独占访问权”。
Below is Section 3.3 in a style matching the Redis or ZooKeeper/etcd examples. It keeps the same structure, including a “big picture” intro, a step-by-step explanation of how locks work, and a conclusion about when you’d choose this approach.以下是 3.3 节,风格与 Redis 或 ZooKeeper/etcd 示例相匹配。它保持了相同的结构,包括“大局”介绍、锁工作原理的分步解释,以及关于何时选择此方法的结论。
3.3 Database Locks (Monolithic or Single-DB Setup)3.3 数据库锁(单体或单数据库设置)
Sometimes, you’d rather avoid setting up extra infrastructure like Redis or ZooKeeper. If your entire application relies on a single relational database (e.g., PostgreSQL, MySQL, SQL Server), you can often leverage built-in locking features to manage concurrency. Many databases provide two main methods:有时,你宁愿避免设置像 Redis 或 ZooKeeper 这样的额外基础设施。如果你的整个应用程序依赖于单个关系型数据库(如 PostgreSQL、MySQL、SQL Server),你通常可以利用内置的锁定功能来管理并发。许多数据库提供两种主要方法:
-
Row-Level Locks (e.g.,
SELECT ... FOR UPDATE)行级锁(例如 SELECT ... FOR UPDATE) -
Advisory Locks (e.g.,
pg_advisory_lockin Postgres,GET_LOCKin MySQL,sp_getapplockin SQL Server)建议锁(例如 Postgres 中的 pg_advisory_lock,MySQL 中的 GET_LOCK,SQL Server 中的 sp_getapplock)
They serve different needs, but both let you say, “I want exclusive access to something,” using your existing DB.它们服务于不同的需求,但都允许你使用现有的数据库说:“我想要某物的独占访问权”。
Row-Level Locks with SELECT ... FOR UPDATE使用 SELECT ... FOR UPDATE 的行级锁
You can lock specific rows in a table by issuing something like:你可以通过发出类似以下的命令来锁定表中的特定行:
BEGIN;
SELECT * FROM locks WHERE lock_id = @loc_key FOR UPDATE;
/* make changes */
COMMIT;That looks cryptic, but here’s what’s happening.这看起来很晦涩,但以下是正在发生的事情。
You wrap the SQL command with the transaction (BEGIN/COMMIT). The query includes FOR UPDATE, which tells the database: “Lock this row so nobody else can modify it until I’m done.” If someone else tries to update the same row, they’ll have to wait.你用事务(BEGIN/COMMIT)包装 SQL 命令。查询包含 FOR UPDATE,它告诉数据库:“锁定这一行,这样在我完成之前没人能修改它。”如果其他人试图更新同一行,他们将不得不等待。
The lock lasts until you commit or roll back the transaction. At commit, the lock is released, letting other transactions proceed.锁持续到你提交或回滚事务为止。在提交时,锁被释放,允许其他事务继续。
You may also use regular rows instead of a dedicated locks table. It’s natural if your concurrency problem revolves around specific table rows. For others you need to define key that would represent the scope of locking你也可以使用常规行而不是专门的锁表。如果你的并发问题围绕特定的表行,这是很自然的。对于其他情况,你需要定义一个代表锁定范围的键。
Advisory Locks (e.g., pg_advisory_lock or GET_LOCK)建议锁(例如 pg_advisory_lock 或 GET_LOCK)
Instead of tying a lock to a specific row, you can lock an arbitrary identifier:与其将锁绑定到特定行,不如锁定一个任意标识符:
-- PostgreSQL
SELECT pg_advisory_lock(12345);
/* do something exclusive */
SELECT pg_advisory_unlock(12345);-- MySQL
SELECT GET_LOCK('readModel', 10);
/* do something exclusive */
SELECT RELEASE_LOCK('readModel');You pass an integer (12345) or string ('readModel') to request a lock from the database’s internal lock manager. If it’s “free”, you acquire it; if it’s already taken, you wait or fail (depending on the timeout).你传递一个整数(12345)或字符串('readModel')来从数据库的内部锁管理器请求锁。如果它是“空闲”的,你就获取它;如果它已被占用,你就等待或失败(取决于超时设置)。
Locks are session-bound. The lock belongs to your current DB session. The lock is freed automatically if that session ends due to a crash or normal exit. The downside is that the lock might remain locked if your session hangs around in a weird state.锁是会话绑定的。锁属于你当前的数据库会话。如果该会话因崩溃或正常退出而结束,锁会自动释放。缺点是如果你的会话处于某种奇怪的状态,锁可能会一直保持锁定。
Those locks can be much lighter than row-level locks (e.g. PostgreSQL Advisory Locks). Yet, some of them may require passing integers, so you may need to map the string to a number, which can require some consistent hashing algorithm.这些锁可以比行级锁轻得多(例如 PostgreSQL 建议锁)。然而,其中一些可能需要传递整数,因此你可能需要将字符串映射为数字,这可能需要某种一致性哈希算法。
If you want the reference implementations of PostgreSQL Advisory Locks, check mine in Pongo.如果你想要 PostgreSQL 建议锁的参考实现,请查看我在 Pongo 中的实现。
Why would you use database locks?你为什么要使用数据库锁?
In many cases, database locks require minimal setup. There’s no need to spin up Redis or ZooKeeper if you already trust a single relational DB for everything. You can use familiar SQL, which can benefit many developers. You get transaction integrations.在许多情况下,数据库锁需要最小的设置。如果你已经信任单个关系型数据库来处理所有事情,则无需启动 Redis 或 ZooKeeper。你可以使用熟悉的 SQL,这对许多开发人员都有好处。你还可以获得事务集成。
Still, they’re tight to a single database scope. They don’t help if you have multiple databases or multi-region replication.尽管如此,它们被限制在单个数据库范围内。如果你有多个数据库或多区域复制,它们就没用了。
The other downside is lock contention. A busy system constantly grabbing locks could block queries and reduce performance.另一个缺点是锁竞争。一个忙碌的系统不断抢占锁可能会阻塞查询并降低性能。
You also can get deadlocks if multiple transactions lock rows or advisory locks in different orders. Monitoring is crucial.如果多个事务以不同的顺序锁定行或建议锁,你还可能遇到死锁。监控至关重要。
3.4 Kubernetes Single-Instance Deployments (An Operational Constraint)3.4 Kubernetes 单实例部署(一种操作约束)
Sometimes, you don’t need a distributed lock at all—you just need to ensure there’s no possibility of concurrency. In Kubernetes, you can tell the cluster to run exactly one replica of your service. With only one pod, you don’t risk two pods writing to the same resource simultaneously. This approach is straightforward but also very limiting.有时,你根本不需要分布式锁——你只需要确保不存在并发的可能性。在 Kubernetes 中,你可以告诉集群只运行一个服务副本。只有一个 Pod,你就不会面临两个 Pod 同时写入同一资源的风险。这种方法很简单,但也非常有限。
How It Works in Terms of Locking它在锁定方面是如何工作的
-
No Actual Lock: Technically, you don’t lock anything. You simply rely on the fact that setting
replicas: 1in your Deployment or StatefulSet ensures the scheduler never spawns more instances. You specify exactly one replica for your service. Kubernetes never spawns a second one, so concurrency can’t occur at the application level.无实际锁:从技术上讲,你没有锁定任何东西。你只是依赖于在 Deployment 或 StatefulSet 中设置 replicas: 1 这一事实,确保调度程序永远不会生成更多实例。你为服务指定了精确的一个副本。Kubernetes 永远不会生成第二个副本,因此在应用程序层面不会发生并发。 -
One Pod, One Writer: If that single pod is alive, it’s the only one doing the work. If it crashes, Kubernetes restarts it, but you might have downtime.一个 Pod,一个写入者:如果那个单一的 Pod 活着,它就是唯一做这项工作的人。如果它崩溃,Kubernetes 会重启它,但你可能会遇到停机时间。
-
No Concurrency: Because there’s literally just one instance, you don’t need to coordinate or lock anything. You’re effectively sidestepping the whole locking problem无并发:因为实际上只有一个实例,你不需要协调或锁定任何东西。你实际上是在回避整个锁定问题。
-
No Parallelism or scaling: If you eventually need multiple pods, you’ll have to switch to an actual lock approach. If traffic spikes or you want high availability, you can’t spin up more pods without moving to a distributed locking mechanism.无并行或扩展:如果你最终需要多个 Pod,你必须切换到实际的锁方法。如果流量激增或你想要高可用性,在不转向分布式锁定机制的情况下,你无法启动更多 Pod。
Funnily enough, Kubernetes stores all cluster data in etcd. etcd keeps the entire cluster’s state: Deployments, Services, ConfigMaps, and so on. However, the cluster doesn’t perform ephemeral lock logic for a single-instance Deployment. It merely stores “replicas: 1” and lets the controller keep the Pod count in sync with that number.有趣的是,Kubernetes 将所有集群数据存储在 etcd 中。etcd 保存了整个集群的状态:Deployments、Services、ConfigMaps 等。然而,集群不会为单实例 Deployment 执行临时锁逻辑。它仅仅存储“replicas: 1”,并让控制器保持 Pod 数量与该数字同步。
Kubernetes doesn’t use the lock record as described before with Zookeeper/etcd section. The Deployment/Stateful Set controller periodically checks how many Pods are currently running for that Deployment. If there are zero (e.g., the Pod crashed), the controller starts a new Pod to reach the desired count of one. If there’s already one Pod, the controller does nothing—it matches the desired state.Kubernetes 不使用 ZooKeeper/etcd 章节中描述的锁记录。Deployment/Stateful Set 控制器会定期检查该 Deployment 当前运行了多少个 Pod。如果为零(例如 Pod 崩溃),控制器会启动一个新 Pod 以达到预期的 1 个。如果已经有一个 Pod,控制器什么也不做——它匹配了预期状态。
In Kubernetes, the mechanisms and controllers designed to manage the number of pods are quite robust, but in highly dynamic or unusual situations, there might be brief moments where conditions could lead to more than one pod being created temporarily. However, such occurrences are generally rare and usually quickly corrected by Kubernetes’ control loops. For instance:在 Kubernetes 中,旨在管理 Pod 数量的机制和控制器非常稳健,但在高度动态或异常的情况下,可能会有短暂的时刻,条件可能导致暂时创建了多个 Pod。然而,这种情况通常很少见,并且通常会被 Kubernetes 的控制循环迅速纠正。例如:
-
Rapid Scale-Up and Scale-Down: If you rapidly change the replicas count up and then back down (or vice versa), it might lead to the Kubernetes scheduler and the replica set controller temporarily creating more pods than desired. This can occur in highly dynamic environments where scripts or operators frequently adjust the replica counts.快速扩容和缩容:如果你快速更改副本数量,然后又改回去(反之亦然),这可能会导致 Kubernetes 调度程序和副本集控制器暂时创建比预期更多的 Pod。这可能发生在脚本或操作员频繁调整副本数量的高度动态环境中。
-
Controller Delays: Kubernetes controllers periodically reconcile the desired state with the actual state. If there’s a delay or lag in the controller’s loop (due to cluster load or other disruptions), you might briefly see more pods than expected.控制器延迟:Kubernetes 控制器定期协调预期状态与实际状态。如果控制器循环有延迟或滞后(由于集群负载或其他干扰),你可能会短暂看到比预期更多的 Pod。
-
Network Partitions or etcd Availability Issues: If parts of the cluster cannot communicate with each other or with etcd (where the desired state is stored), different nodes might make different decisions. For example, one node might not know that a new pod has already been scheduled elsewhere and might schedule another.网络分区或 etcd 可用性问题:如果集群的部分节点无法相互通信或无法与 etcd(存储预期状态的地方)通信,不同的节点可能会做出不同的决定。例如,一个节点可能不知道已经在其他地方调度了一个新 Pod,并可能调度另一个。
Why You’d Choose Single-Instance in Kubernetes?你为什么要选择 Kubernetes 单实例?
If you just need to enforce that there’s a single instance processing background job (read model handling, job processing) and you have Kubernetes set up, then it’s a decent choice. You remove the concurrency at all. You don’t spin up Redis or ZooKeeper just to handle locks.如果你只需要强制执行只有一个实例在处理后台作业(读模型处理、作业处理),并且你已经设置了 Kubernetes,那么这是一个不错的选择。你消除了所有并发。你不需要为了处理锁而启动 Redis 或 ZooKeeper。
Still, the race conditions can be dangerous for high traffic or important cases. It’s not fully reliable. So, it’s a decent choice for rarely Triggered Tasks: If you have a once-a-day job and don’t care about parallelism or availability, one pod might be enough.尽管如此,竞态条件对于高流量或重要情况仍然可能是危险的。它并不完全可靠。因此,对于偶尔触发的任务来说,这是一个不错的选择:如果你有一个每天一次的作业,并且不在乎并行性或可用性,那么一个 Pod 可能就足够了。
3.5 TLDR on locking tooling choice3.5 锁工具选择总结
Here’s my recommended decision-making scheme for locking mechanism:这是我推荐的锁定机制决策方案:
-
If you have a single relational DB handling all app state, advisory locks might suffice (and require no extra moving parts and minimal infrastructure). Many ready-made packages provide that. Inside modules locks or monolithic apps using a single DB instance are good use cases here. Beware: database locks are harder to scale across multiple DB nodes. They can become a contention point if locks are frequent.如果你有一个单一的关系型数据库处理所有应用状态,建议锁可能就足够了(不需要额外的移动部件,基础设施最小化)。许多现成的包都提供了这一点。模块内锁或使用单个数据库实例的单体应用是这里的良好用例。注意:数据库锁在多个数据库节点间更难扩展。如果锁频繁,它们可能成为竞争点。
-
If your environment already includes a Redis cluster for caching and you want a simpler ephemeral lock, Redis is a natural fit. If you want an application-level lock across multiple services, Redis is a decent candidate. Remember that it can bring a potential edge case under network partitions; multi-node setups may require Redlock for stronger safety.如果你的环境已经包含用于缓存的 Redis 集群,并且你想要一个更简单的临时锁,Redis 是一个自然的选择。如果你想要跨多个服务的应用级锁,Redis 是一个不错的候选者。记住,它在网络分区下可能会带来潜在的边缘情况;多节点设置可能需要 Redlock 以获得更强的安全性。
-
If concurrency is never desired or is completely out of scope for a specific microservice, AND you’re using already Kubernetes a single-instance Kubernetes approach may be acceptable. However, it clearly sacrifices high availability and scaling. It’s not really a distributed lock at all, just an operational limit of “one pod.”如果永远不需要并发,或者并发对于特定微服务完全不在范围内,并且你已经在使用 Kubernetes,那么单实例 Kubernetes 方法可能是可以接受的。然而,它显然牺牲了高可用性和扩展性。这根本不是分布式锁,只是“一个 Pod”的操作限制。
-
If your microservices need advanced coordination (like leader election, watchers, or strongly consistent state), consider ZooKeeper or etcd. It’s the most configurable option and the most scalable. Yet it’s heavier to operate than a simple single Redis or DB solution. It’s also another service to operate. And pretty complex one.如果你的微服务需要高级协调(如领导者选举、监视器或强一致性状态),请考虑 ZooKeeper 或 etcd。它是最可配置且可扩展性最强的选项。但它比简单的单个 Redis 或数据库解决方案更重。这也是另一个需要运维的服务。而且相当复杂。
4. Potential Pitfalls and Best Practices4. 潜在陷阱和最佳实践
4.1 Deadlocks and Lock Ordering4.1 死锁和锁排序
If your system needs to acquire multiple locks at once, you risk deadlocks (e.g., process A has Lock1 and wants Lock2; process B has Lock2 and wants Lock1). The best practice is to lock in a consistent order globally or use carefully designed transaction boundaries. I’ll show you next week how you could use queuing and single writer for that.如果你的系统需要同时获取多个锁,你就会面临死锁风险(例如,进程 A 持有 Lock1 并想要 Lock2;进程 B 持有 Lock2 并想要 Lock1)。最佳实践是全局以一致的顺序加锁,或使用精心设计的事务边界。下周我将向你展示如何使用队列和单一写入者来解决这个问题。
4.2 Lock Contention4.2 锁竞争
Locks serialize access. If too many services fight for the same lock, your system effectively becomes single-threaded. To avoid bottlenecks, lock only the smallest critical sections. If concurrency at some granularity is acceptable, consider sharding or partitioned locks.锁会序列化访问。如果太多的服务争夺同一个锁,你的系统实际上就变成了单线程。为了避免瓶颈,只锁定最小的关键部分。如果某种粒度下的并发是可以接受的,请考虑分片或分区锁。
4.3 Crash Handling (TTL, Ephemeral Sessions)4.3 崩溃处理(TTL,临时会话)
What if the lock holder dies mid-operation? This is the classic reason for TTL or ephemeral sessions:如果锁持有者在操作过程中死亡怎么办?这是 TTL 或临时会话的经典原因:
-
TTL: If the process doesn’t extend the TTL, the lock is auto-freed. Another node can proceed.TTL:如果进程不延长 TTL,锁会自动释放。另一个节点可以继续。
-
Ephemeral: The ephemeral node is removed if the session disappears (like in ZooKeeper).临时:如果会话消失,临时节点会被移除(如在 ZooKeeper 中)。
Without these, you risk permanent blocking if someone forgets to release or if the holder abruptly goes offline.没有这些,如果有人忘记释放或持有者突然离线,你就会面临永久阻塞的风险。
4.4 Single Point of Failure4.4 单点故障
If you rely on a single Redis instance or a single ZooKeeper node, your lock manager can fail. Always consider using a clustered or highly available setup, such as Redis with sentinel or cluster mode or a ZooKeeper ensemble of three or five nodes.如果你依赖单个 Redis 实例或单个 ZooKeeper 节点,你的锁管理器可能会失败。始终考虑使用集群或高可用性设置,例如带有 sentinel 或集群模式的 Redis,或者由三个或五个节点组成的 ZooKeeper 集合。
4.5 Clock Skew and Network Partitions4.5 时钟漂移和网络分区
Systems like Redlock try to handle partial failures, but no distributed lock can be 100% guaranteed if your network is severely partitioned (CAP theorem territory). You might end up with multiple holders, each believing they’re the only one. Proper design, timeouts, and conflict detection help reduce these edge cases.像 Redlock 这样的系统试图处理部分故障,但如果你的网络严重分区(CAP 定理领域),没有任何分布式锁能保证 100% 正确。你最终可能会有多个持有者,每个都认为自己是唯一的。适当的设计、超时和冲突检测有助于减少这些边缘情况。
Conclusion结论
Distributed locks are a fundamental tool for coordinating concurrency across systems. Whether you choose Redis (with TTL-based locks), ZooKeeper (ephemeral nodes), database locks, or even just run a single replica in Kubernetes, the core principle remains: only one party can operate on a resource at a time. Handling failures (crashed holders) is crucial, which is why ephemeral or time-based mechanisms are standard. Each approach has trade-offs in complexity, performance, and fault-tolerance.分布式锁是跨系统协调并发的基本工具。无论你选择 Redis(基于 TTL 的锁)、ZooKeeper(临时节点)、数据库锁,还是仅仅在 Kubernetes 中运行一个副本,核心原则保持不变:一次只有一个方能操作资源。处理故障(崩溃的持有者)至关重要,这就是为什么临时或基于时间的机制是标准配置的原因。每种方法在复杂性、性能和容错性方面都有权衡。
Key Takeaways for senior engineers and architects:给资深工程师和架构师的关键要点:
-
Try to avoid distributed locking if you can. Usually, they’re a good sanity checkpoint to validate why you got here and if you really need it.如果可以,尽量避免分布式锁定。通常,它们是一个很好的理智检查点,用来验证你为什么走到这一步,以及你是否真的需要它。
-
Use locks carefully: Over-locking can degrade concurrency, but it’s critical when data corruption or double-processing must be prevented.谨慎使用锁:过度锁定会降低并发性,但在必须防止数据损坏或重复处理时,它至关重要。
-
Choose the right store: Redis is often the simplest if you already run it. ZooKeeper suits advanced coordination. DB locks can be fine for single-region or monolithic architectures.选择正确的存储:如果你已经在使用 Redis,它通常是最简单的。ZooKeeper 适合高级协调。数据库锁对于单区域或单体架构可能很好。
-
Plan for failure: Implement TTL/ephemeral locks to avoid “zombie” or stuck locks.规划故障:实施 TTL/临时锁以避免“僵尸”或卡住的锁。
-
Zero concurrency?: Setting
replicas: 1in a StatefulSet or Deployment, concurrency is removed but multi-replica scaling is forfeited. It’s not a distributed lock—just an operational constraint.零并发?:在 StatefulSet 或 Deployment 中设置 replicas: 1,并发被消除了,但放弃了多副本扩展。这不是分布式锁——只是一个操作约束。 -
Understand trade-offs: If your scenario is straightforward, a lock-based single subscription might be simpler than full leader election. For advanced cluster coordination, leader election or a robust distributed approach might be more robust.理解权衡:如果你的场景很简单,基于锁的单一订阅可能比完整的领导者选举更简单。对于高级集群协调,领导者选举或稳健的分布式方法可能更可靠。
Distributed systems will always have complexities, but a well-implemented distributed lock (or a strategic single-instance approach) can tame the chaos of concurrency—keeping your data consistent and your architecture stable.分布式系统总会有复杂性,但一个实现良好的分布式锁(或策略性的单实例方法)可以驯服并发的混乱——保持数据一致并使架构稳定。
What are your experiences, use cases, and challenges with distributed locking?你在分布式锁定方面的经验、用例和挑战是什么?
And hey, all the best for Christmas if you celebrate it. If you don’t, try to also get the chance to rest a bit. And if you don’t want to, that’s fine, as long as you’re happy!嘿,如果你庆祝圣诞节,祝你圣诞快乐。如果不庆祝,也试着休息一下。如果你不想休息,那也没关系,只要你开心就好!
Cheers!干杯!
OskarOskar
p.s. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, and putting pressure on your local government or companies. You can also support Ukraine by donating, e.g. to the Ukraine humanitarian organisation, Ambulances for Ukraine or Red Cross.附:乌克兰仍处于残酷的俄罗斯入侵之下。许多乌克兰人受伤、无家可归,需要帮助。你可以通过多种方式提供帮助,例如直接帮助难民、传播意识,并向当地政府或公司施压。你也可以通过捐款来支持乌克兰,例如捐给乌克兰人道主义组织、乌克兰救护车组织或红十字会。