Your architecture diagram is not your resilience

This first article in our resilience series draws on a conversation with Mark Russinovich about how resilience is changing in the AI era and what it takes to continuously validate it at scale.本系列关于韧性的第一篇文章，源于与 Mark Russinovich 的一次对话，探讨了在人工智能时代韧性如何发生演变，以及在大规模环境下持续验证韧性需要具备哪些条件。

What surprises me most about resilience failures is how ordinary the drift is. A workload is deployed across availability zones, but a health probe still points to a single dependency. A database supports failover, but the application’s connection string is pinned to one region. Nothing looks broken. The architecture diagram still shows a resilient design even as the operational reality underneath it changes.关于韧性失效，最令我惊讶的是这种偏差（drift）是多么常见。工作负载部署在多个可用区，但健康探测仍指向单一依赖项；数据库支持故障转移，但应用程序的连接字符串却锁定在单一区域。表面上看一切正常，即使底层的运营现实已经改变，架构图依然显示出一种具有韧性的设计。

For years, resilience was something you set up once: configure disaster recovery, write a runbook, run the occasional failover test. That kept the lights on, but it treated resilience as a project with an end date rather than a property you maintain.多年来，韧性一直被视为一种“一次性设置”：配置灾难恢复、编写运行手册、偶尔进行故障转移测试。这虽然能维持系统运行，却将韧性当作了一个有截止日期的项目，而非一种需要持续维护的属性。

So, when an availability zone or a region has a bad day, the question is not whether a recovery plan exists on paper. It is whether resilience is still true today, and whether the team can prove it.因此，当某个可用区或区域出现故障时，问题不在于纸面上是否存在恢复计划，而在于韧性在今天是否依然有效，以及团队能否证明这一点。

The dependency that breaks a workload is also changing. On Microsoft’s FY26 Q4 earnings call, Satya Nadella put it plainly: “For whatever reason, if a given model goes away, then you can’t be left high and dry. You need to be able to still continue your cyber operations.”导致工作负载中断的依赖项也在发生变化。在微软 2026 财年第四季度财报电话会议上，Satya Nadella 直言不讳地指出：“无论出于何种原因，如果某个特定模型失效，你不能因此陷入困境。你需要确保能够继续进行网络运营。”

Traditional disaster recovery planning assumes the critical dependency is infrastructure. Increasingly it is an AI model, an inference endpoint, a retrieval pipeline, or a service operating under capacity constraints. A workload can be perfectly healthy from an infrastructure perspective and still fail its users because that dependency is unavailable, throttled, or economically impractical to run. That dependency rarely appears on the diagram at all.传统的灾难恢复规划假设关键依赖项是基础设施。而现在，它越来越多地表现为 AI 模型、推理端点、检索管道或处于容量限制下的服务。从基础设施角度看，工作负载可能非常健康，但由于该依赖项不可用、被限流或在经济上不可行，它依然无法为用户提供服务。而这种依赖项往往根本不会出现在架构图中。

There is a second shift underneath the first. An architecture diagram assumes a human drew it and a human will read it. Both of those assumptions are ending, and the dependencies it describes are no longer all deterministic. That changes what it means to know your estate is resilient.在这一变化之下，还存在着第二个转变。架构图假设是由人类绘制并由人类阅读的。这两个假设正在终结，且架构图所描述的依赖项也不再全部是确定性的。这改变了“了解你的资产具有韧性”的定义。

This is the first in a series on how we are helping customers move to resilience that is designed in, measured as the estate changes, and improved over time. We have written before about how to design a resilient workload. This is a different problem: knowing whether hundreds of workloads still match their design today and being able to prove it. It is where a large share of our roadmap investment is now going.这是我们系列文章的第一篇，旨在探讨如何帮助客户实现“设计内嵌、随资产变化而衡量、并随时间改进”的韧性。我们之前写过如何设计具有韧性的工作负载，而这面对的是一个不同的问题：了解数百个工作负载在今天是否仍符合其设计，并能够证明这一点。这也是我们目前路线图投资的重点方向。

Why resilience drifts为什么会出现韧性偏差

Resilience has always been a shared responsibility. We provide the availability zones, a secondary region of choice, and the replication primitives needed to support resiliency, and those do not drift. What drifts is the other half of the bargain: whether a given workload still uses them the way it was designed to, after a year of changes nobody flagged as risky. Change is where this concentrates. Across the industry, roughly 70 percent of cloud outages are related to change in some way—not dramatic failures, but ordinary modifications whose blast radius nobody re-evaluated.韧性始终是共同责任。我们提供可用区、可选的辅助区域以及支持韧性所需的复制原语，这些都不会产生偏差。真正产生偏差的是契约的另一半：在经历了一年多无人标记为风险的变更后，给定的工作负载是否仍按最初设计的方式使用这些资源。变更正是问题的集中点。在整个行业中，大约 70% 的云中断在某种程度上与变更有关——不是重大的故障，而是那些无人重新评估其影响范围的普通修改。

That is why change discipline matters as much as design. Internally, a change rolls out to a canary region first, then a pilot region, with bake times where health signals are watched before it goes any further. That practice came out of an incident of our own, and it is the same discipline the Well-Architected Framework describes as safe deployment practices.这就是为什么变更纪律与设计同等重要。在内部，变更会先部署到金丝雀区域，然后是试点区域，并设置“烘焙时间”（bake time），在进一步推广前观察健康信号。这种做法源于我们自己的一次事故，它与“架构完善框架”（Well-Architected Framework）中所述的安全部署实践是相同的准则。

Learn more about the Azure Well-Architected Framework了解更多关于 Azure 架构完善框架的信息

Disaster recovery is reactive by design. Teams set recovery objectives when a project ships, stand up replication, and then move on, with little ongoing visibility into whether those objectives still hold as the workload changes. Resilience is designed once and rarely revisited. Across a growing estate of availability zones and regions, the distance between the resilience that was designed and the resilience that actually exists widens quietly, and it usually surfaces only during an incident.灾难恢复在设计上是被动的。团队在项目交付时设定恢复目标，建立复制机制，然后就转向其他工作，很少持续关注随着工作负载的变化，这些目标是否依然有效。韧性往往是一次性设计，很少被重新审视。在不断扩大的可用区和区域资产中，已设计的韧性与实际存在的韧性之间的差距在悄然扩大，通常只有在发生事故时才会浮出水面。

We learned this the hard way. Mark tells the story of the storage change that came close to taking Azure down, and what it permanently changed about how we deploy.我们从惨痛的教训中得出了这一点。Mark 讲述了那个差点导致 Azure 下线的存储变更故事，以及它如何永久改变了我们的部署方式。

What a diagram can’t tell you架构图无法告诉你的事

A diagram is a claim about a system, made once, by someone reasoning about the system as they believed it to be. It is useful, and it is not evidence. There are four things it structurally cannot tell you.架构图是对系统的一种陈述，由某人基于其对系统的认知一次性做出。它很有用，但并非证据。从结构上讲，有四件事是它无法告诉你的。

Whether the goal is being met right now. A diagram has no timestamp. Health modeling does: described in the Well-Architected Framework and now available through health models in Azure Monitor, it represents an application as a hierarchy of its components and the signals underneath them, so health is expressed in terms the business recognizes rather than as a wall of resource-level metrics. Paired with service level indicators, it answers the only question that matters during an incident: is this application meeting its objective right now? The standard that matters is not our own. A service is only healthy if the customer thinks it is healthy—we can believe it is fine, but if the customer is not seeing a healthy service, we have a problem.目标是否正在实现。架构图没有时间戳。健康建模则不然：在架构完善框架中进行了描述，并可通过 Azure Monitor 中的健康模型实现，它将应用程序表示为组件及其底层信号的层次结构，因此健康状况是以业务可识别的术语表达的，而不是一堆资源级指标。结合服务水平指标（SLI），它能回答事故期间唯一重要的问题：该应用程序现在是否达到了目标？重要的标准不是我们自己的。只有当客户认为服务健康时，它才是健康的——我们可能认为没问题，但如果客户没有看到健康的服务，那就是我们的问题。
What “resilient” even means for this application. On a diagram, resilient is an adjective. A resiliency goal makes it a threshold you either meet or miss and defining it at the level of the application rather than resource by resource is what makes the answer meaningful.对于此应用程序，“韧性”究竟意味着什么。在架构图中，韧性是一个形容词。而韧性目标将其转化为一个要么达到、要么错过的阈值，并在应用程序层面而非逐个资源地定义它，这才是使答案具有意义的关键。
Whether the failover path actually works. Every diagram draws the arrow. Only a test proves it.故障转移路径是否真的有效。每张架构图都会画出箭头，但只有测试才能证明它有效。
The resources nobody drew. A diagram shows what someone remembered. Generated Infrastructure-as-Code covers every resource in the application. The difference between those two sets is where drift begins—and increasingly the reader on the other end is an agent rather than a person, working from the outcome you asked for instead of the picture you drew.无人绘制的资源。架构图显示的是某人记忆中的内容。生成的“基础设施即代码”（IaC）涵盖了应用程序中的每一个资源。这两组内容之间的差异就是偏差开始的地方——而且越来越多的阅读者是代理（agent）而非人类，它们根据你要求的预期结果而非你绘制的图表来工作。

This is how we run Azure. Rather than asking each team to declare what healthy means for their service, we standardized on service level indicators, then applied machine learning to observed behavior so that healthy is defined by what the service actually does rather than by what someone assumed it would do.这就是我们运行 Azure 的方式。我们没有要求每个团队声明其服务的健康定义，而是标准化了服务水平指标，并将机器学习应用于观察到的行为，从而使“健康”的定义由服务的实际运行情况决定，而不是由某人的假设决定。

This is the part I find most interesting. We wrote our documentation, our schemas, and our templates for people. Increasingly the thing reading them is an agent, working from the outcome you asked for rather than the picture you drew, and generating the resources itself. When the author and the reader are both machines, a diagram is no longer even the medium the decision is made in.这是我发现最有趣的部分。我们为人类编写了文档、模式和模板。但现在阅读它们的往往是代理，它们根据你要求的预期结果而非你绘制的图表来工作，并自行生成资源。当作者和读者都是机器时，架构图甚至不再是做出决策的媒介。

Mark on why we break our own services on purpose, what a game day actually tests, and why every failure mode we simulate is one that has really happened.Mark 谈到了为什么我们要故意破坏自己的服务，演练（game day）究竟测试了什么，以及为什么我们模拟的每一种故障模式都是真实发生过的。

When the dependency is probabilistic当依赖项具有概率性时

A model that disappears is the obvious risk. The subtler one is a model that answers, differently each time. It is a newer source of drift, and it does not behave like the old ones. Ask the same question of a model twice and you can get two different answers. That makes correctness harder to define, and it makes change harder to reason about: if you change the prompt, change the model, or change the harness and the skills around it, you have changed the software, and it deserves the same discipline as any other change.模型消失是一个显而易见的风险。更微妙的风险是模型每次回答的结果都不同。这是偏差的一个较新来源，且其行为与旧的偏差不同。对模型提出同一个问题两次，你可能会得到两个不同的答案。这使得正确性更难定义，也使得变更更难推断：如果你更改了提示词、模型或周边的工具链，你就更改了软件，它应当与任何其他变更一样受到同样的纪律约束。

Most teams smoke-test instead. It looks fine, so it ships. Evaluation is the step that gets skipped, and it is the one that matters: measuring the system against what you actually consider valuable, not simply whether it responded.大多数团队转而进行冒烟测试。看起来没问题，就发布了。评估是容易被跳过的步骤，但它才是最重要的：根据你真正认为有价值的内容来衡量系统，而不仅仅是看它是否做出了响应。

The first question is whether the system needs to be probabilistic at all. Stay as deterministic as you can and use AI where it earns its place, not everywhere. And where you can wrap a non-deterministic system in a deterministic check, do it. If an agent is only supposed to update dependency versions, have a second system verify that versions are the only thing that changed. Where a deterministic check is not possible, use adversarial review: a second agent whose whole job is to find what is wrong with the first one’s work.第一个问题是系统是否真的需要具备概率性。尽可能保持确定性，在 AI 能发挥作用的地方使用它，而不是到处使用。在可能的地方，用确定性检查包装非确定性系统。如果代理的任务只是更新依赖项版本，请让第二个系统验证版本是唯一发生变化的内容。在无法进行确定性检查的地方，使用对抗性审查：让第二个代理的全部工作就是找出第一个代理工作中的错误。

And accountability does not transfer. When someone deploys an agent, someone remains answerable for what it does.此外，责任不可转嫁。当某人部署代理时，仍需有人对其行为负责。

That applies to us as well. Our own incident triage system uses language models to answer which service is responsible for an incident—work that used to mean waking people up to argue over logs. It is genuinely faster, and it is still trust but verify.这一点同样适用于我们。我们自己的事故分类系统使用语言模型来回答哪个服务应对事故负责——这项工作过去意味着要把人叫醒来争论日志。它确实更快，但仍然遵循“信任但要验证”的原则。

Mark on why guardrails around AI should be deterministic wherever possible, and why the agent is never the one accountable.Mark 谈到了为什么 AI 周围的护栏应尽可能采用确定性，以及为什么代理永远不是承担责任的对象。

What this looks like in practice在实践中是什么样子的

That discipline has to hold wherever a workload runs, and the foundations are familiar. The reliability guidance in the Well-Architected Framework already says to design resilience in from the start, and the Cloud Adoption Framework describes the operating half: carrying that intent into how the estate is actually run and reviewed month after month. Where an estate spans global, national, and sovereign or regulated environments, the word resilient has to mean the same thing in each rather than being redefined at every boundary; our guidance on reliability and sovereignty covers where those constraints intersect.这种纪律必须在工作负载运行的任何地方得到坚持，其基础是大家所熟悉的。架构完善框架中的可靠性指南已经明确要求从一开始就设计韧性，而云采用框架（Cloud Adoption Framework）则描述了运营的一面：将这种意图贯彻到资产的实际运行和逐月审查中。当资产跨越全球、国家以及主权或受监管环境时，“韧性”一词在每个环境中必须具有相同的含义，而不是在每个边界被重新定义；我们关于可靠性和主权的指南涵盖了这些约束交叉的地方。

Your diagram shows three zones. It does not show that the health probe behind the load balancer resolves to one of them. Availability zones protect a workload from datacenter-level failures within a region, but only if compute, storage, and data tiers are genuinely spread across them.你的架构图显示了三个可用区。但它没有显示负载均衡器后面的健康探测只解析到其中一个。可用区保护工作负载免受区域内数据中心级故障的影响，但前提是计算、存储和数据层确实分布在这些可用区中。
Plan region resiliency for disaster recovery. Set explicit recovery objectives, an RTO and RPO, for each workload. Availability zones give you high availability inside a region, while a secondary region of choice provides a failover location when an entire region is affected. They answer different risks, and a resilient design is deliberate about both. In a regulated estate the choice narrows further, because a recovery region has to sit inside the same jurisdiction as the workload it protects.为灾难恢复规划区域韧性。为每个工作负载设置明确的恢复目标，即 RTO 和 RPO。可用区为你提供区域内的高可用性，而可选的辅助区域则在整个区域受影响时提供故障转移位置。它们应对的是不同的风险，而韧性设计应对两者都深思熟虑。在受监管的资产中，选择范围进一步缩小，因为恢复区域必须位于其所保护的工作负载所在的同一司法管辖区内。
Decide how much resilience is worth buying. Resilience is a cost decision as much as a design one. An application carrying 100 million dollars of revenue on a single day justifies an active-active topology across regions for that day; the same application may run in a single region with active-passive failover for the rest of the year. The right answer is deliberate, not maximal.决定投入多少成本来实现韧性。韧性既是一个成本决策，也是一个设计决策。一个在单日内承载 1 亿美元收入的应用程序，证明了在该日进行跨区域“主-主”拓扑的必要性；而同样的应用程序在一年中的其余时间里，可能只需在单区域内运行“主-备”故障转移即可。正确的答案是深思熟虑的，而非追求最大化。
Know your blast radius. Understand what each workload depends on and where its hidden single points of failure are, then keep that picture current as the application changes. Internally we do this as reliability threat modeling: the same discipline as security threat modeling, asking of each component what would happen if it failed, and what we would do about it.了解你的影响范围。了解每个工作负载依赖什么，以及其隐藏的单点故障在哪里，然后随着应用程序的变化保持该图表的更新。在内部，我们将其作为可靠性威胁建模来执行：这与安全威胁建模遵循相同的纪律，针对每个组件询问如果它发生故障会怎样，以及我们将如何应对。
Check the dependencies your recovery path itself relies on. A workload can be replicated correctly and still be unrecoverable: if its encryption keys live only in the primary region, they are gone precisely when a region outage means you need them. Recovery paths have dependencies too, and they are rarely on the diagram.检查恢复路径本身依赖的依赖项。工作负载可能被正确复制，但仍然无法恢复：如果其加密密钥仅存在于主区域，那么当区域中断需要它们时，它们正好丢失了。恢复路径也有依赖项，而且它们很少出现在架构图上。
Your diagram probably has no box for the AI dependency your application now relies on. Design for your application, workload, AI model, and service dependencies, not just infrastructure. Plan for graceful degradation and fallback so that if a critical dependency is deprecated, throttled, unavailable, or capacity-constrained, the workload continues to operate through an alternative path rather than failing outright.你的架构图可能没有为你现在依赖的 AI 依赖项留出位置。要为你的应用程序、工作负载、AI 模型和服务依赖项进行设计，而不仅仅是基础设施。规划优雅降级和回退方案，以便在关键依赖项被弃用、限流、不可用或容量受限时，工作负载能通过替代路径继续运行，而不是直接崩溃。

This is not theoretical. Carne Group, one of Europe’s largest independent third-party asset managers with one trillion dollars under management, rebuilt its estate on Azure with infrastructure-as-code landing zones precisely so resilience would be reproducible rather than remembered. Because the definition lives in code, their team can stand up a duplicate site in another region and, as Carne Group’s global technology lead Stéphane Bebrone puts it, “even in the event of a worst-case scenario, we could be back up and running more or less in the same day.” They are working toward an active-passive topology across regions and plan to use Chaos Studio to verify those failover paths on a schedule rather than on an incident. Under DORA, they have to be able to prove it, not assert it.这并非理论。Carne Group 是欧洲最大的独立第三方资产管理公司之一，管理着一万亿美元资产，他们利用“基础设施即代码”着陆区在 Azure 上重建了资产，从而使韧性变得可复现而非仅仅依靠记忆。由于定义存在于代码中，他们的团队可以在另一个区域建立一个副本站点，正如 Carne Group 全球技术负责人 Stéphane Bebrone 所言：“即使在最坏的情况下，我们也能在同一天左右恢复运行。”他们正致力于实现跨区域的“主-备”拓扑，并计划定期使用 Chaos Studio 来验证这些故障转移路径，而不是仅在事故发生时才测试。根据 DORA 法规，他们必须能够证明这一点，而不仅仅是断言。

Learn more about how Carne Group used Azure Site Recovery and Backup了解更多关于 Carne Group 如何使用 Azure Site Recovery 和 Backup 的信息

Closing the gap between intent and reality弥合意图与现实之间的差距

An architecture diagram is a statement of intent, not proof. It may show zone redundancy, regional failover, and protected dependencies, but only a test can determine whether those assumptions still hold. Assembling the underlying data into a clear view of application resiliency takes real effort and expertise. That is the customer’s half of the shared responsibility, and it is the gap Azure Infrastructure Resiliency Manager is built to help close. In public preview, its agentic-first experience helps teams start resilient, get resilient, and stay resilient.架构图是意图的陈述，而非证明。它可能显示了可用区冗余、区域故障转移和受保护的依赖项，但只有测试才能确定这些假设是否依然成立。将底层数据汇集成对应用程序韧性的清晰视图需要真正的努力和专业知识。这是客户在共同责任中应承担的一半，也是 Azure Infrastructure Resiliency Manager 旨在帮助弥合的差距。目前处于公开预览阶段，其以代理为先的体验可帮助团队实现“初始韧性、获取韧性、保持韧性”。

Start resilient. Define what resilient means for an application, set explicit resiliency goals, and use the Resiliency Agent to generate resiliency-aware Infrastructure-as-Code up front, so a new workload starts resilient instead of being corrected months later. Service Groups help teams represent an application as a logical group of Azure resources in the portal, making it easier to manage resiliency posture at the application level rather than resource by resource.初始韧性。定义应用程序的韧性含义，设定明确的韧性目标，并使用 Resiliency Agent 预先生成具有韧性意识的“基础设施即代码”，从而使新工作负载从一开始就具备韧性，而不是在几个月后才进行修正。服务组（Service Groups）可帮助团队在门户中将应用程序表示为 Azure 资源的逻辑组，从而更轻松地在应用程序层面而非逐个资源地管理韧性状态。
Get resilient. See where the estate stands against the intended design, identify resources that were never zone resilient or stopped being so after a change, and prioritize the gaps that matter first. Rather than correlating findings across several tools, teams get recommendations and generated Infrastructure-as-Code for supported fixes, so remediation can move through a pull request instead of becoming a separate project.获取韧性。查看资产与预期设计相比的情况，识别那些从未实现可用区韧性或在变更后失去韧性的资源，并优先处理最重要的差距。团队无需在多个工具间关联发现结果，而是直接获得修复建议和生成的“基础设施即代码”，从而通过拉取请求（pull request）进行修复，而无需将其变成一个单独的项目。
Stay resilient. Validate the assumptions behind the recovery path before a real outage does it for you. For workloads where the customer manages the compute, such as virtual machines, a zone-down drill can simulate the loss of an availability zone and show what actually happens to the application. For other services, teams can use failover validation, product-specific recovery capabilities, and fault injection through Azure Chaos Studio to test the right failure modes.保持韧性。在真正的中断发生前，验证恢复路径背后的假设。对于客户管理计算资源（如虚拟机）的工作负载，可用区关闭演练可以模拟可用区的丢失，并展示应用程序的实际运行情况。对于其他服务，团队可以使用故障转移验证、特定于产品的恢复功能以及通过 Azure Chaos Studio 进行的故障注入来测试正确的故障模式。

We will be candid about the gaps too. Consistent, self-service resiliency assessment across every workload and environment is not finished work, and saying so matters. Resiliency improves when teams can see the gaps, measure them, and systematically close them over time. It is also work that never quite ends. As services harden the failure rate drops, but the failures that remain get rarer and stranger; you approach perfection asymptotically without ever arriving.我们也会坦诚地面对差距。在每个工作负载和环境中实现一致的自助式韧性评估尚未完成，承认这一点很重要。当团队能够看到差距、衡量差距并随着时间推移系统地弥合差距时，韧性就会提高。这也是一项永无止境的工作。随着服务不断强化，故障率会下降，但剩下的故障会变得更罕见、更诡异；你是在无限接近完美，却永远无法真正抵达。

The bottom line: a diagram is not proof底线：架构图不是证明

Resilience is not a project you finish; it is a posture you maintain. The work is to design it in from the first architecture decision, then keep proving it as the estate changes, so drift is caught by a test rather than by an incident. All of it is anchored in the Azure Essentials frameworks, the Well-Architected Framework, and the Cloud Adoption Framework, so that resilient and sovereign carry the same meaning across every environment instead of being reinvented team by team.韧性不是一个可以完成的项目；它是一种需要维护的状态。工作重点是从第一个架构决策开始就将其设计进去，然后随着资产的变化不断证明它，从而通过测试而非事故来捕捉偏差。所有这些都植根于 Azure Essentials 框架、架构完善框架和云采用框架中，以确保“韧性”和“主权”在每个环境中都具有相同的含义，而不是由各个团队各自为政地重新发明。

None of this is a new framework. A diagram tells you what you intended; only a test tells you what you have. That was true when people drew the diagrams and read them. It matters more now that neither is reliably the case, and that some of what you depend on answers differently every time you ask. If resilience cannot be tested, it cannot be trusted.这些都不是什么新框架。架构图告诉你你的意图；只有测试才能告诉你你拥有什么。当人们手工绘制并阅读架构图时，这是事实。现在，当这两者都不再可靠，且你所依赖的部分内容每次询问时答案都不同时，这一点就显得更加重要。如果韧性无法测试，它就无法被信任。

In the next part of this series, we will look at how to measure that posture at scale.在本系列的下一部分中，我们将探讨如何大规模衡量这种状态。

The three clips above are drawn from a longer conversation covering the 2014 change that came close to taking Azure down, how we learned to measure health from the customer’s point of view, what breaks differently when a dependency is an AI model, and why the hardware in a datacenter fails every single day. Check out the full interview on the Azure Essentials YouTube channel.上述三个片段摘自一次更深入的对话，涵盖了 2014 年那次差点导致 Azure 下线的变更、我们如何学习从客户的角度衡量健康状况、当依赖项是 AI 模型时故障方式有何不同，以及为什么数据中心里的硬件每天都在发生故障。请在 Azure Essentials YouTube 频道观看完整采访。