AWS Architecture BlogAWS 架构博客
Architecting AI-powered resilience framework on AWS在 AWS 上构建 AI 驱动的弹性框架
When your production system goes down, you often discover the hard way that your resilience testing missed critical dependencies. Building an AI-powered resilience framework on AWS helps you find those weaknesses before your customers do.当您的生产系统宕机时,您往往会在艰难的过程中发现弹性测试遗漏了关键依赖。 在 AWS 上构建 AI 驱动的弹性框架可以帮助您在客户发现之前找到这些薄弱环节。
Your systems don’t fail because your infrastructure isn’t resilient. They fail because resilience is assumed, not proven. Every deployment introduces new dependencies, every configuration change creates untested paths, and every gap between design intent and runtime behavior is a risk waiting to surface. In a world where customers expect always-on availability, the cost of discovering these weaknesses in production isn’t just technical. It’s measured in revenue lost, trust eroded, and events that were entirely preventable.您的系统之所以失败,并不是因为基础设施不具弹性,而是因为弹性被假设而未得到验证。 每一次部署都会引入新的依赖,每一次配置更改都会产生未经测试的路径,而设计意图与运行时行为之间的每一个差距都是潜在风险。 在客户期望始终可用的世界里,发现这些弱点的成本不仅是技术层面的,更体现在收入损失、信任受损以及本可以完全避免的事件上。
In this post, you’ll learn how to architect and implement a five-layer AI-powered resilience framework that automatically discovers dependencies, generates targeted experiments, and integrates with your existing Continuous Integration/Continuous Deployment (CI/CD) pipelines. First, we’ll explore the key challenges in resilience testing. Then, we’ll walk through the five-layer architecture that solves these challenges. Finally, we’ll show you how to implement this, with phased rollout guidance for pilot, expansion, and organization-wide deployment.在本文中,您将学习如何构建并实现一个五层 AI 驱动的弹性框架,该框架能够自动发现依赖、生成有针对性的实验,并与您现有的持续集成/持续交付(CI/CD)流水线集成。 首先,我们将探讨弹性测试的关键挑战。 然后,我们将逐层讲解解决这些挑战的五层架构。 最后,我们将展示如何实现它,并提供针对试点、扩展和全组织部署的分阶段 rollout 指南。
Traditional resilience testing can take weeks because the information needed is naturally spread across architecture diagrams, runbooks, code repositories, and team knowledge that evolves with every deployment. Designing meaningful chaos experiments on top of that requires specialized expertise that most teams don’t have on hand. Here’s how this AI-powered framework discovers your infrastructure dependencies in hours and generates targeted experiments without requiring specialized knowledge.传统的弹性测试可能需要数周时间,因为所需信息自然分散在架构图、运行手册、代码仓库以及随每次部署而演进的团队知识中。 在此基础上设计有意义的混沌实验需要专业技能,而大多数团队并未具备。 本 AI 驱动的框架能够在数小时内发现您的基础设施依赖,并生成有针对性的实验,无需专业知识。
Combining AWS Resilience Hub, AWS Fault Injection Service, Amazon Bedrock AgentCore, and AWS Systems Manager work together to discover infrastructure dependencies in hours, tailor experiments to your specific architecture, and identify weaknesses before they affect customers.将 AWS Resilience Hub、AWS Fault Injection Service、Amazon Bedrock AgentCore 与 AWS Systems Manager 结合使用,可在数小时内发现基础设施依赖、根据您的特定架构定制实验,并在影响客户之前识别薄弱环节。
With the next generation of AWS Resilience Hub now providing native dependency discovery and generative AI-powered failure mode analysis, this framework extends those capabilities by automating experiment generation, embedding resilience testing into your CI/CD pipelines, and creating a continuous validation loop through custom AI agents hosted on Amazon Bedrock AgentCore.随着下一代 AWS Resilience Hub 现在提供原生依赖发现和生成式 AI 驱动的故障模式分析,该框架通过自动化实验生成、将弹性测试嵌入 CI/CD 流水线以及通过托管在 Amazon Bedrock AgentCore 上的自定义 AI 代理创建持续验证循环,进一步扩展了这些能力。
Key concepts关键概念
Before diving into the architecture, here are the key terms used throughout this post:在深入架构之前,以下是本文中使用的关键术语:
- Chaos engineering — The discipline of experimenting on a system to build confidence in its ability to withstand turbulent conditions in production.混沌工程——在系统上进行实验的学科,以建立对其在生产环境中承受动荡条件的信心。
- Mean time to resolution (MTTR) — The average time to restore service after a failure.平均恢复时间(MTTR)——故障后恢复服务的平均时间。
- Recovery Time Objective (RTO) — The maximum acceptable downtime for your application.恢复时间目标(RTO)——对您的应用程序可接受的最大停机时间。
- Recovery Point Objective (RPO) — The maximum acceptable data loss, measured in time.恢复点目标(RPO)——以时间衡量的最大可接受数据丢失量。
- Shift-left — Testing earlier in the development cycle to catch issues before they reach production.左移(Shift-left)——在开发周期的早期进行测试,以在代码进入生产前捕获问题。
- Circuit breaker — An automated mechanism that detects service failures and helps prevent cascading outages by temporarily blocking requests to failing services.断路器——一种自动化机制,能够检测服务故障并通过临时阻止对故障服务的请求,防止连锁故障。
- Canary deployment — A technique where changes roll out to a small percentage of users first to validate functionality before full deployment.金丝雀部署——一种技术,先将更改推送给少量用户,以验证功能后再全面部署。
Who this is for适用对象
This post targets cloud architects, DevOps engineers, and Site Reliability Engineering (SRE) teams responsible for system reliability. You should understand AWS services, distributed systems architecture, and CI/CD pipelines. While chaos engineering experience helps, it’s not required. This framework reduces the expertise barrier that traditionally prevented adoption.本文面向负责系统可靠性的云架构师、DevOps 工程师和站点可靠性工程(SRE)团队。 您应了解 AWS 服务、分布式系统架构和 CI/CD 流水线。 虽然拥有混沌工程经验有帮助,但并非必需。 该框架降低了传统上阻碍采用的专业门槛。
Resilience testing challenges弹性测试挑战
When infrastructure changes happen quickly, documentation tends to lag behind. A routing change to the payment service might not get reflected in the architecture diagrams, leaving single points of failure (like a single-Availability Zone authentication dependency) undocumented and untested. The payment service calls a legacy authentication API that only runs in one Availability Zone, a critical single point of failure. But documentation doesn’t reflect this because someone made a “quick fix” three weeks ago and forgot to update the diagrams.当基础设施快速变化时,文档往往滞后。 对支付服务的路由更改可能未在架构图中体现,导致单点故障(例如单可用区的身份验证依赖)未被记录和测试。 支付服务调用仅在单个可用区运行的遗留身份验证 API,这是关键的单点故障。 但文档未反映这一点,因为三周前有人做了“快速修复”,却忘记更新图纸。
Distributed systems contain hundreds of interconnected components. Tracking every dependency manually becomes impractical when you deploy changes continuously. Documentation created last month already misses dozens of new dependencies.分布式系统包含数百个相互连接的组件。 当您持续部署更改时,手动跟踪每个依赖变得不切实际。 上个月创建的文档已经遗漏了数十个新依赖。
Resilience testing can be challenging without dedicated specialists to design meaningful experiments. Resilience testing is most effective when it’s tailored to your actual architecture and runs continuously. Generic fault injection and one-off test runs can leave gaps, especially as your system changes over time.如果没有专职专家设计有意义的实验,弹性测试会变得困难。 当弹性测试针对实际架构并持续运行时,效果最佳。 通用的故障注入和一次性测试往往留下空白,尤其是系统随时间变化时。
The expertise barrier stops many organizations entirely. Effective chaos engineering demands understanding distributed systems architecture, failure mode analysis, experiment scope management, and safe experiment design. Without this specialized knowledge, you either avoid resilience testing or run superficial tests that miss critical vulnerabilities.专业门槛使许多组织完全放弃。 有效的混沌工程需要了解分布式系统架构、故障模式分析、实验范围管理以及安全实验设计。 没有这些专业知识,您要么回避弹性测试,要么进行表面化的测试,错过关键漏洞。
According to the 2024 IBM Security Services Benchmark Report, organizations with mature response capabilities reduce their MTTR by approximately 50% and achieve cost savings of up to 58% per event when compared to organizations with less mature capabilities. Yet Gartner research shows that 68% of organizations cite increasing system complexity as their reason for adopting chaos engineering, while 50% admit they weren’t prepared when failures occurred.根据 2024 年 IBM 安全服务基准报告,具备成熟响应能力的组织将 MTTR 降低约 50%,并在每次事件上实现最高 58% 的成本节约。 然而 Gartner 研究显示,68% 的组织因系统复杂度上升而采用混沌工程,50% 的组织承认在故障发生时未做好准备。
This framework automates discovery. Automated discovery reduces infrastructure mapping from weeks to hours, typically completing initial assessment in 2–4 hours for single-account environments with thousands of resources. Subsequent runs process only the changes tracked by AWS Config, so your architecture map stays current without manual effort. Agents hosted on AgentCore Runtime analyze your AWS CloudFormation templates, code repositories, and runtime behavior to identify every connection, including hidden dependencies that manual audits miss. Experiment templates analyze your specific architecture and produce targeted tests that validate your actual failure modes, removing the need for specialized chaos engineering expertise. Continuous integration into your CI/CD pipelines catches regressions before they reach production, shifting resilience from a one-time project into an ongoing practice embedded in your development workflow.该框架实现自动发现。 自动发现将基础设施映射时间从数周缩短到数小时,通常在拥有数千资源的单账户环境中完成初始评估需 2–4 小时。 随后运行仅处理 AWS Config 追踪的变更,使您的架构图保持最新,无需人工。 托管在 AgentCore Runtime 上的代理分析您的 AWS CloudFormation 模板、代码仓库和运行时行为,识别每个连接,包括手动审计遗漏的隐藏依赖。 实验模板分析您的特定架构并生成针对性测试,验证实际故障模式,消除对专业混沌工程专长的需求。 将其持续集成到 CI/CD 流水线,可在进入生产前捕获回归,将弹性从一次性项目转变为嵌入开发工作流的持续实践。
Solution overview解决方案概览
The framework addresses these gaps by automatically discovering infrastructure dependencies and continuously validating resilience as the system changes. AI agents, hosted on AgentCore Runtime for secure, scalable execution, discover system dependencies automatically, analyze architectural patterns, and create targeted experiments based on your actual risk profiles. Testing scales across your application portfolio while lowering the expertise barrier.该框架通过自动发现基础设施依赖并在系统变化时持续验证弹性,弥补了这些空白。 托管在 AgentCore Runtime 上的 AI 代理安全、可扩展地自动发现系统依赖、分析架构模式,并根据实际风险画像创建有针对性的实验。 测试可跨您的应用组合扩展,同时降低专业门槛。

Figure 1. Five-layer AI-powered resilience architecture图 1. 五层 AI 驱动弹性架构
Five-layer architecture diagram with AWS Resilience Hub — labeled “Next-Gen” with native dependency discovery and generative AI failure mode analysis — as the central orchestration hub. Layers 1–4 (Discovery, Test Generation, Experimentation, Gap Analysis) sit across the top, each connecting down to the hub. Layer 5 (Continuous Validation) sits below with CI/CD, drift detection, and dashboards. Three dashed feedback loops overlay the diagram: Gap Analysis feeds “Architecture updates” back to Discovery, Continuous Validation sends “Experiment learnings” up to Test Generation, and SSM Docs feeds “Validated recovery procedures” into FIS Templates within the Test Generation layer.五层架构图,中心为标注为“下一代”的 AWS Resilience Hub,具备原生依赖发现和生成式 AI 故障模式分析功能。 第 1–4 层(发现、测试生成、实验、差距分析)位于顶部,各自连接到中心枢纽。 第 5 层(持续验证)位于底部,包含 CI/CD、漂移检测和仪表盘。 三条虚线反馈环覆盖图示:差距分析将“架构更新”反馈至发现层,持续验证将“实验学习”上送至测试生成层,SSM 文档将“已验证的恢复流程”注入测试生成层的 FIS 模板中。
Each layer builds on the previous one, creating a comprehensive validation strategy. This architecture aligns with the AWS Well-Architected Reliability Pillar, specifically the Test reliability best practice area. Discovery maps your infrastructure. Test generation creates relevant experiments from that map. Experimentation executes those tests safely. Gap analysis identifies what needs fixing. Continuous validation helps verify that improvements persist as your systems evolve.每一层都在前一层之上构建,形成完整的验证策略。 该架构与 AWS Well‑Architected 可靠性支柱保持一致,特别是测试可靠性最佳实践。 发现层绘制您的基础设施图谱。 测试生成层基于该图谱创建相关实验。 实验层安全执行这些测试。 差距分析层识别需要修复的内容。 持续验证层确保改进随系统演进而持久。
Now that you understand the overall strategy, let’s examine each layer in detail, starting with how the discovery layer automatically maps your infrastructure.现在您已经了解整体策略,下面逐层详细说明,从发现层如何自动绘制您的基础设施开始。
Discovery layer发现层
The discovery layer forms the foundation by automatically identifying infrastructure components and their dependencies. The next generation of AWS Resilience Hub provides native dependency discovery that identifies AWS services, internal endpoints, and third-party endpoints your applications rely on. A custom agent deployed on Amazon Bedrock AgentCore (with read permissions to AWS APIs) extends this native discovery with code-level analysis, scanning your code repositories for hard-coded dependencies, connection strings, timeout configurations, and retry logic that infrastructure-level discovery alone cannot detect. A custom agent deployed on Amazon Bedrock AgentCore (with read permissions to AWS APIs) handles infrastructure discovery. Amazon Bedrock AgentCore Runtime provides dedicated MicroVM session isolation, supports long-running discovery sessions up to eight hours, and handles scaling and security without requiring you to manage infrastructure. The runtime’s built-in observability (traces, logs, and metrics) integrates natively with your existing Amazon CloudWatch dashboards without additional instrumentation.发现层是基础,通过自动识别基础设施组件及其依赖构建根基。 下一代 AWS Resilience Hub 提供原生依赖发现,能够识别您的应用所依赖的 AWS 服务、内部端点和第三方端点。 部署在 Amazon Bedrock AgentCore 上的自定义代理(具备对 AWS API 的读取权限)在此原生发现之上进行代码级分析,扫描代码仓库中的硬编码依赖、连接字符串、超时配置和重试逻辑,这些是仅靠基础设施层发现无法检测的。 部署在 Amazon Bedrock AgentCore 上的自定义代理负责基础设施发现。 AgentCore Runtime 提供专用 MicroVM 会话隔离,支持最长八小时的发现会话,并在无需您管理基础设施的情况下处理扩展和安全。 运行时内置的可观测性(追踪、日志、指标)可原生集成到您现有的 Amazon CloudWatch 仪表盘,无需额外仪表。
The AgentCore-hosted agent queries services including Amazon Elastic Compute Cloud (Amazon EC2), Amazon Relational Database Service (Amazon RDS), AWS Lambda, Amazon DynamoDB, and Amazon Simple Storage Service (Amazon S3) to build comprehensive inventory. It analyzes AWS CloudFormation templates and Terraform configurations to understand your intended architecture, and accesses code repositories to identify hard-coded dependencies, connection strings, and timeout configurations in your applications. AWS Config provides configuration data and tracks changes over time. This discovery completes in 2–4 hours for environments with thousands of resources.托管在 AgentCore 的代理查询包括 Amazon Elastic Compute Cloud(Amazon EC2)、Amazon Relational Database Service(Amazon RDS)、AWS Lambda、Amazon DynamoDB 和 Amazon Simple Storage Service(Amazon S3)等服务,以构建完整清单。 它分析 AWS CloudFormation 模板和 Terraform 配置,了解您期望的架构,并访问代码仓库识别硬编码依赖、连接字符串和超时配置。 AWS Config 提供配置数据并随时间追踪变更。 此发现过程在拥有数千资源的环境中完成时间为 2–4 小时。

Figure 2. AI-powered discovery workflow图 2. AI 驱动发现工作流
Workflow diagram organized in two swim lanes. Lane 1 (blue) shows AWS Resilience Hub native infrastructure-level discovery querying AWS service APIs (Amazon EC2, Amazon RDS, AWS Lambda, Amazon S3, Elastic Load Balancing, Amazon DynamoDB) for service topology, endpoints, and Multi-AZ configuration. Lane 2 (yellow) shows an agent on Amazon Bedrock AgentCore performing code-level discovery — scanning repositories (CodeCommit, GitHub, GitLab via IAM permissions) and analyzing CloudFormation/Terraform templates for connection strings, timeouts, and circuit breakers. Both lanes feed into a combined dependency map, which flows into the AWS Resilience Hub assessment engine. Outputs include single points of failure, dependency maps (infrastructure + code), baseline resilience scores, configuration drift, and resilience gaps. AWS Config monitors configurations in parallel. A dashed feedback arrow loops from the assessment engine back to the AgentCore agent labeled “Architecture updates & learnings.” Initial mapping completes in 2–4 hours for typical single-account environments; subsequent runs process only changes tracked by AWS Config.工作流图分为两条泳道。 泳道 1(蓝色)展示 AWS Resilience Hub 原生基础设施层发现,查询 AWS 服务 API(Amazon EC2、Amazon RDS、AWS Lambda、Amazon S3、Elastic Load Balancing、Amazon DynamoDB)获取服务拓扑、端点和多可用区配置。 泳道 2(黄色)展示在 Amazon Bedrock AgentCore 上的代理进行代码层发现——扫描仓库(CodeCommit、GitHub、GitLab 通过 IAM 权限)并分析 CloudFormation/Terraform 模板中的连接字符串、超时和断路器。 两条泳道的输出合并为依赖图,流入 AWS Resilience Hub 评估引擎。 输出包括单点故障、依赖图(基础设施+代码)、基线弹性分数、配置漂移和弹性差距。 AWS Config 并行监控配置。 虚线反馈箭头从评估引擎回到 AgentCore 代理,标记为“架构更新与学习”。 初始映射在典型单账户环境中完成时间为 2–4 小时;后续运行仅处理 AWS Config 追踪的变更。
Test generation layer测试生成层
While the next generation of Resilience Hub includes a generative AI-powered failure mode assessment that identifies potential weaknesses through static analysis, this test generation layer converts those recommendations into executable AWS Fault Injection Service experiment templates, complete with safety guardrails, progressive scope expansion, and business impact scoring tailored to your specific architecture.虽然下一代 Resilience Hub 包含生成式 AI 驱动的故障模式评估,可通过静态分析识别潜在弱点,但此测试生成层将这些建议转换为可执行的 AWS Fault Injection Service 实验模板,配备安全防护、渐进式范围扩展和业务影响评分,针对您的特定架构进行定制。
Building on the discovered infrastructure, the test generation layer creates targeted chaos experiments for your specific architecture. The agent hosted on AgentCore Runtime uses Amazon Bedrock foundation models to analyze your infrastructure context, combined with the RTO, RPO, and availability targets you define in AWS Resilience Hub, to identify single points of failure and produce hypothesis-driven test scenarios aligned with your business requirements. Each hypothesis is scored by potential business impact, prioritizing experiments for customer-facing systems and components where architectural patterns indicate high-availability intent. Each experiment includes business impact scoring based on your application tier definitions in AWS Resilience Hub, architectural patterns (such as internet-facing load balancers and Amazon API Gateway endpoints), dependency analysis, and AWS resource tags. This makes sure experiments prioritize your customer-facing systems and highest-impact components. Using the code repository analysis from the discovery layer, the system detects when your applications use Amazon RDS Multi-AZ but lack proper connection retry handling, and designs database failover tests that validate your actual recovery mechanisms rather than generic network disruption tests.基于已发现的基础设施,测试生成层为您的特定架构创建有针对性的混沌实验。 托管在 AgentCore Runtime 上的代理使用 Amazon Bedrock 基础模型分析您的基础设施上下文,并结合您在 AWS Resilience Hub 中定义的 RTO、RPO 和可用性目标,识别单点故障并生成与业务需求对齐的假设驱动测试场景。 每个假设根据潜在业务影响进行评分,优先考虑面向客户的系统和架构模式显示高可用性意图的组件。 每个实验包含基于您在 AWS Resilience Hub 中的应用层定义、架构模式(如面向互联网的负载均衡器和 Amazon API Gateway 端点)、依赖分析和资源标签的业务影响评分,确保实验优先覆盖客户面向系统和最高影响组件。 通过发现层的代码仓库分析,系统检测到您的应用使用 Amazon RDS 多可用区但缺乏适当的连接重试处理,并设计数据库故障转移测试,以验证实际恢复机制,而非通用网络中断测试。
For your production environments, implement a manual approval workflow where your infrastructure teams review experiment templates before execution. AWS Step Functions orchestrates approval gates. Step Functions is a workflow management service that coordinates multiple AWS services into serverless workflows.对于生产环境,实施手动审批工作流,由基础设施团队在执行前审查实验模板。 AWS Step Functions 编排审批门。 Step Functions 是一种工作流管理服务,可将多个 AWS 服务协调为无服务器工作流。
Experimentation layer实验层
After creating targeted experiments, the experimentation layer runs chaos tests with multi-layered safety guardrails on your infrastructure. AWS Fault Injection Service executes chaos tests with built-in safety mechanisms. Experiments start with minimal scope (affecting only 1% of your resources) and expand progressively based on your risk tolerance and validation results (for example, 1% → 5% → 10% → 25%). This follows progressive deployment strategies recommended in the AWS Well-Architected Reliability Pillar, similar to canary deployments where changes roll out incrementally to limit blast radius. Amazon CloudWatch alarms serve as stop conditions that halt experiments before they violate your Service Level Agreements (SLAs), which are contracts defining expected uptime and performance. Set alarm thresholds well below your SLA limits. If your SLA allows 1% error rate, configure stop conditions to trigger at 0.1%.在创建有针对性的实验后,实验层在您的基础设施上运行带有多层安全防护的混沌测试。 AWS Fault Injection Service 通过内置安全机制执行混沌测试。 实验从最小范围(仅影响 1% 资源)开始,并根据您的风险容忍度和验证结果逐步扩大(例如 1% → 5% → 10% → 25%)。 这遵循 AWS Well‑Architected 可靠性支柱推荐的渐进式部署策略,类似金丝雀部署,以限制冲击半径。 Amazon CloudWatch 警报充当停止条件,在实验违反服务等级协议(SLA)前终止实验。 将警报阈值设置在 SLA 限制以下。 若 SLA 允许 1% 错误率,则将停止条件配置为 0.1%。
Gap analysis layer差距分析层
After your experiments complete, the gap analysis layer processes results to identify weaknesses and prioritize remediation. AWS Resilience Hub correlates experiment outcomes with your resilience policies, categorizing gaps across architectural, operational, data protection, and testing dimensions. Each gap receives a priority score based on severity (how badly this violates your resilience policy), likelihood (how often this failure mode occurs), and business impact (the cost if this failure occurs in your environment).实验完成后,差距分析层处理结果,识别弱点并优先排序修复。 AWS Resilience Hub 将实验结果与您的弹性策略关联,按架构、运营、数据保护和测试维度对差距进行分类。 每个差距根据严重性(违反弹性策略的程度)、可能性(该故障模式发生的频率)和业务影响(若该故障在您环境中发生的成本)获得优先级分数。
Continuous validation layer持续验证层
The continuous validation layer integrates resilience testing into your development workflow. The right approach depends on your deployment velocity and testing goals.持续验证层将弹性测试集成到您的开发工作流中。 具体做法取决于您的部署速度和测试目标。
For most teams, a lightweight policy-as-code check (using tools like Open Policy Agent to validate Infrastructure as Code and Dockerfiles) runs in seconds and fits naturally in your CI/CD pipeline for every commit. This catches basic configuration issues, like missing health checks or single-AZ deployments, before code reaches staging.对大多数团队而言,使用 Open Policy Agent 等工具对基础设施即代码和 Dockerfile 进行的轻量级策略即代码检查,仅需几秒钟即可在每次提交的 CI/CD 流水线中自然运行。 这可在代码进入预发布前捕获基本配置问题,如缺少健康检查或单可用区部署。
Full resilience assessments are better suited as a pre-production gate, triggered on significant architectural changes rather than every commit. For routine deployments, lightweight resilience regression tests (validating a focused set of critical failure scenarios like database failover, Availability Zone loss, and circuit breaker activation) run automatically to catch unintended resilience degradation from code or configuration changes. This two-tiered approach gives you comprehensive safety validation for major changes and continuous regression coverage for everyday deployments, without slowing your pipeline. Your new code and infrastructure changes trigger automated resilience assessments that identify potential weaknesses during development rather than after deployment, embedding this shift-left strategy directly into your CI/CD workflow.完整的弹性评估更适合作为预发布门,在重大架构变更时触发,而非每次提交。 对于常规部署,轻量级弹性回归测试(验证数据库故障转移、可用区丢失、断路器激活等关键故障场景)可自动运行,以捕获代码或配置变更导致的意外弹性退化。 这种两层方法为重大变更提供全面安全验证,为日常部署提供持续回归覆盖,而不会拖慢流水线。 您的新代码和基础设施变更触发自动弹性评估,在开发阶段识别潜在弱点,而非部署后再发现,从而将左移策略直接嵌入 CI/CD 工作流。
The policy-as-code check adds seconds to each pipeline run. Full resilience assessments add approximately 2–3 minutes per experiment.策略即代码检查为每次流水线运行增加几秒钟。 完整的弹性评估每个实验约增加 2–3 分钟。
AWS Config drift detection identifies manual changes that bypass your deployment pipelines, helping keep your architecture aligned with tested configurations.AWS Config 漂移检测识别绕过部署流水线的手动更改,帮助保持架构与已测试配置一致。

Figure 3. Resilience testing in the CI/CD pipeline图 3. CI/CD 流水线中的弹性测试
Flowchart showing a CI/CD pipeline with two-tiered resilience testing. The top row shows the standard pipeline flow: Developer Commits Code → Build & Unit Tests → Deploy to Test Environment → Resilience Regression Tests (green hexagon gate, runs 3–5 critical scenarios in ~2–3 minutes on every deployment) → Integration Tests → Deploy to Staging. From staging, if an architectural change is detected, the flow drops to a second row: Full Resilience Assessment (orange hexagon gate, runs 15–20 experiments over ~15–45 minutes using AWS Resilience Hub, FIS, and Bedrock) → Manual Approval Gate → Deploy to Production → Continuous Monitoring. If no architectural change occurred, staging skips directly to the approval gate. Both resilience gates have failure paths (red) that block deployment and loop back to the developer. A legend and a side panel summarize the two tiers.流程图展示具有两层弹性测试的 CI/CD 流水线。 顶部行显示标准流水线流程:开发者提交代码 → 构建与单元测试 → 部署到测试环境 → 弹性回归测试(绿色六边形门,运行 3–5 个关键场景,约 2–3 分钟) → 集成测试 → 部署到预发布。 若在预发布检测到架构变更,流程转至第二行:完整弹性评估(橙色六边形门,运行 15–20 个实验,约 15–45 分钟,使用 AWS Resilience Hub、FIS 和 Bedrock) → 手动审批门 → 部署到生产 → 持续监控。 若未检测到架构变更,预发布直接跳至审批门。 两个弹性门均有失败路径(红色),阻止部署并回流至开发者。 图例和侧栏概述两层结构。
Continuous improvement through feedback通过反馈实现持续改进
Experiment results feed back into the discovery and test generation layers, creating a continuous improvement cycle. When experiments reveal undocumented dependencies, the discovery layer updates your architecture map. When remediation actions successfully resolve failure patterns, Systems Manager automation documents capture these procedures for future use. The Bedrock agent analyzes experiment outcomes to refine hypothesis generation, deprioritizing consistently passing scenarios and focusing on emerging risk areas as your architecture evolves.实验结果反馈至发现层和测试生成层,形成持续改进循环。 当实验揭示未记录的依赖时,发现层更新您的架构图。 当修复操作成功解决故障模式时,Systems Manager 自动化文档记录这些流程以供将来使用。 Bedrock 代理分析实验结果,优化假设生成,降低持续通过的场景优先级,聚焦于随架构演进出现的新风险领域。
Key benefits关键收益
Now that you’ve seen how each layer works together, let’s examine the concrete benefits this architecture brings to your organization.现在您已经看到各层如何协同工作,下面我们来看看该架构为组织带来的具体收益。
Faster infrastructure discovery: Manual infrastructure discovery requires significant effort across distributed teams, including cataloging resources, tracing dependencies, and validating configurations. Automated discovery reduces this from weeks to hours by programmatically querying cloud service APIs, analyzing infrastructure-as-code templates, and mapping dependencies. After implementation, the framework scales across your application portfolio without proportionally increasing staffing.更快的基础设施发现:手动发现需要跨分布式团队投入大量精力,包括资源清点、依赖追踪和配置验证。 自动化发现通过程序化查询云服务 API、分析基础设施即代码模板和映射依赖,将时间从数周缩短到数小时。 实施后,框架可在整个应用组合中扩展,而无需按比例增加人力。
Removes expertise barrier: Without chaos engineering specialists, you can implement resilience testing using automated scenarios. Start with the AWS Fault Injection Service Scenarios Library for common failure patterns, then expand with scenarios specific to your architecture and customize based on your application’s specific failure modes.消除专业门槛:即使没有混沌工程专家,您也可以使用自动化场景实现弹性测试。 从 AWS Fault Injection Service 场景库的常见故障模式开始,然后根据您的架构扩展并自定义特定的故障模式。
Proactive risk identification: Automated discovery reveals critical single points of failure that manual audits consistently miss, including hard-coded endpoints, missing circuit breakers, and absent health checks. The system identifies vulnerabilities across your infrastructure and prioritizes them by business impact, so your teams can focus remediation on the highest-risk items first.主动风险识别:自动发现揭示手动审计常常遗漏的关键单点故障,包括硬编码端点、缺失断路器和缺少健康检查。 系统跨基础设施识别漏洞并按业务影响排序,帮助团队优先处理最高风险项。
Faster recovery through automated remediation: Automated remediation reduces your mean time to resolution by removing manual intervention for common failure patterns in your environments. AWS Systems Manager automation documents codify recovery procedures discovered during chaos experiments. When Amazon CloudWatch alarms detect failure patterns in your systems, AWS Systems Manager automatically executes remediation actions, handling issues faster than manual response.通过自动化修复加速恢复:自动化修复通过消除对常见故障模式的人工干预,降低平均恢复时间。 AWS Systems Manager 自动化文档将混沌实验中发现的恢复流程编码。 当 Amazon CloudWatch 警报检测到系统故障模式时,AWS Systems Manager 自动执行修复操作,比人工响应更快。
Continuous resilience validation: Integrating resilience assessments into your CI/CD pipelines catches regressions before production deployment, maintaining resilience as your system evolves rather than treating it as a one-time validation.持续弹性验证:将弹性评估集成到 CI/CD 流水线,在生产部署前捕获回归,保持弹性随系统演进而持续,而不是一次性验证。
Framework components框架组件
AWS Resilience Hub serves as the central orchestration layer, defining your resilience policies, running assessments, and tracking improvements. Define RTO and RPO targets for each application tier based on your business impact analysis.AWS Resilience Hub 充当中心编排层,定义弹性策略、运行评估并跟踪改进。 根据业务影响分析,为每个应用层定义 RTO 和 RPO 目标。
Amazon Bedrock delivers the AI capabilities that power discovery and test creation. AgentCore Runtime provides the managed hosting layer, handling session isolation, scaling, identity management, and observability, so your agent runs securely in production without infrastructure overhead.Amazon Bedrock 提供驱动发现和测试创建的 AI 能力。 AgentCore Runtime 提供托管的运行时层,处理会话隔离、扩展、身份管理和可观测性,使您的代理在生产中安全运行且无需基础设施开销。
Deploy a custom agent on Amazon Bedrock AgentCore, a framework-agnostic managed runtime that supports agents built with Strands, LangChain, or custom Python. The agent uses Amazon Bedrock to analyze your infrastructure context against architectural patterns, AWS documentation, and best practices. AgentCore Runtime’s built-in tool gateway provides controlled, secure access to your AWS APIs during discovery.在 Amazon Bedrock AgentCore 上部署自定义代理,这是一种框架无关的托管运行时,支持使用 Strands、LangChain 或自定义 Python 构建的代理。 代理使用 Amazon Bedrock 分析您的基础设施上下文,结合架构模式、AWS 文档和最佳实践。 AgentCore Runtime 内置的工具网关在发现期间为您的 AWS API 提供受控、 安全的访问。
AWS Fault Injection Service runs controlled chaos experiments with built-in safety mechanisms on your infrastructure. Pre-built actions cover common failure scenarios: terminating Amazon EC2 instances, injecting network latency, throttling API calls, failing over Amazon RDS databases, and disrupting Availability Zone connectivity in your environments.AWS Fault Injection Service 在您的基础设施上运行受控混沌实验,具备内置安全机制。 预构建的操作覆盖常见故障场景:终止 Amazon EC2 实例、注入网络延迟、限制 API 调用、故障转移 Amazon RDS 数据库以及扰乱可用区连接。
AWS Systems Manager extends your resilience framework beyond the default AWS Fault Injection Service actions. You can create custom automation documents that codify recovery procedures and transform manual runbooks into automated self-healing responses. When you build custom actions, you take on responsibility for proper rollback procedures and service state restoration. Design these with the same rigor you’d apply to your production runbooks.AWS Systems Manager 将您的弹性框架扩展到默认的 AWS Fault Injection Service 操作之外。 您可以创建自定义自动化文档,将恢复流程编码为自动自愈响应,并将手动运行手册转化为自动化。 创建自定义操作时,您需负责适当的回滚流程和服务状态恢复。 请以与生产运行手册相同的严谨性设计这些操作。
AWS Config continuously monitors your resource configurations and tracks changes. AWS Config rules validate that your resources comply with resilience policies. For example, they verify your Amazon RDS instances use Multi-AZ deployment and confirm your Auto Scaling groups span multiple Availability Zones.AWS Config 持续监控资源配置并追踪变更。 AWS Config 规则验证资源是否符合弹性策略。例如,检查您的 Amazon RDS 实例是否使用多可用区部署,并确认 Auto Scaling 组跨多个可用区。
Prerequisites先决条件
To implement this framework, you’ll need:实现此框架,您需要:
- AWS account with administrative access.具备管理员访问权限的 AWS 账户。
- AWS Identity and Access Management (IAM) permissions for: AWS Resilience Hub, AWS Fault Injection Service, Amazon Bedrock AgentCore, AWS Systems Manager, and AWS Config. For each service, follow the principle of least privilege. Refer to the respective service documentation for minimum required permissions.AWS 身份与访问管理(IAM)权限,覆盖:AWS Resilience Hub、AWS Fault Injection Service、Amazon Bedrock AgentCore、AWS Systems Manager 和 AWS Config。 对每项服务遵循最小权限原则。 请参阅相应服务文档获取最低所需权限。
- The Amazon Bedrock AgentCore Starter Toolkitcreates broad dev/test permissions by default. Scope these down to least-privilege before production deployment.Amazon Bedrock AgentCore Starter Toolkit 默认创建广泛的开发/测试权限。 在生产部署前请将其收紧至最小权限。
- AWS Command Line Interface (AWS CLI) installed and configured.已安装并配置好的 AWS 命令行界面(AWS CLI)。
- Basic understanding of AWS CloudFormation or Terraform.对 AWS CloudFormation 或 Terraform 有基本了解。
- Non-critical application available for testing.可用于测试的非关键应用。
- Estimated time: 4–6 hours for pilot implementation (with a team of 2–3 engineers who have working knowledge of your AWS environment).预计时间:4–6 小时完成试点实现(2–3 名熟悉您 AWS 环境的工程师)。
- Cost awareness: This implementation creates billable AWS resources including AWS Resilience Hub, AWS Fault Injection Service, Amazon Bedrock AgentCore, AWS Systems Manager, AWS Config, and Amazon CloudWatch. Follow the cleanup procedures after testing to avoid ongoing charges.成本提示:此实现会创建计费的 AWS 资源,包括 AWS Resilience Hub、AWS Fault Injection Service、Amazon Bedrock AgentCore、AWS Systems Manager、AWS Config 和 Amazon CloudWatch。 测试结束后请遵循清理流程,以避免持续产生费用。
Getting started入门指南
A phased rollout builds confidence before expanding scope if you’re new to chaos engineering.分阶段 rollout 在您对混沌工程不熟悉时,可先建立信心再扩大范围。
Pilot phase (1–2 weeks, 2–3 engineers)试点阶段(1–2 周,2–3 名工程师)
- Select a non-critical application with well-understood architecture from your portfolio.从您的组合中选择一个架构清晰的非关键应用。
- Enable AWS Config across the regions where your application runs.在应用运行的所有区域启用 AWS Config。
- Package your discovery agent code using Strands, LangChain, or custom Python.使用 Strands、LangChain 或自定义 Python 打包您的发现代理代码。
- Deploy the packaged agent on Amazon Bedrock AgentCore using the Amazon Bedrock AgentCore Starter Toolkit. AgentCore Runtime handles the compute, session management, and security so you can focus on the agent logic and discovery scope. The runtime maintains stateful working context (including tool state and memory) across the multi-step infrastructure discovery workflow.使用 Amazon Bedrock AgentCore Starter Toolkit 将打包好的代理部署到 Amazon Bedrock AgentCore。 AgentCore Runtime 负责计算、会话管理和安全,让您专注于代理逻辑和发现范围。 运行时在多步骤基础设施发现工作流中保持有状态的工作上下文(包括工具状态和记忆)。
- Run a baseline resilience assessment in AWS Resilience Hub to identify initial architectural gaps. Resilience Hub evaluates your architecture against Well-Architected best practices and establishes your starting resilience posture. The Bedrock agent you deploy in the next step builds on this baseline, discovering undocumented dependencies and generating targeted experiments that go beyond standard recommendations.在 AWS Resilience Hub 中运行基线弹性评估,以识别初始架构差距。 Resilience Hub 根据 Well‑Architected 最佳实践评估您的架构并建立起始弹性姿态。 接下来部署的 Bedrock 代理将在此基线之上,发现未记录的依赖并生成超出标准建议的有针对性实验。
Verify: Check the AWS Resilience Hub console for a completed assessment report showing baseline resilience scores and identified gaps. You should see a resilience score for each disruption type (AZ, Region, Application) and a list of recommended actions.验证:在 AWS Resilience Hub 控制台检查已完成的评估报告,查看基线弹性分数和已识别的差距。 您应看到每种中断类型(可用区、区域、应用)的弹性分数以及推荐的操作列表。
Expansion phase (4–6 weeks, cross-functional team)扩展阶段(4–6 周,跨职能团队)
- Expand to 3–5 applications across different tiers after validating safety with your pilot.在验证试点安全后,扩展至 3–5 个不同层级的应用。
- Configure automated test creation to develop experiments specific to each of your application’s architectures.配置自动化测试创建,为每个应用架构生成特定实验。
- Run controlled chaos experiments starting with 1% scope during your low-traffic periods.在低流量时段,以 1% 范围启动受控混沌实验。
Verify: Review the AWS Fault Injection Service console for experiment status “Completed” and check Amazon CloudWatch metrics to confirm the 1% scope was applied without triggering stop conditions.验证:在 AWS Fault Injection Service 控制台检查实验状态为“已完成”,并在 Amazon CloudWatch 指标中确认 1% 范围已应用且未触发停止条件。
- Analyze results to identify common patterns across your applications.分析结果,识别跨应用的共性模式。
Enterprise scale (8–12 weeks, dedicated resilience team)企业规模(8–12 周,专职弹性团队)
- Expand resilience assessments in your CI/CD pipelines to comprehensive, multi-account validation.在 CI/CD 流水线中扩展弹性评估,实现跨账户的全面验证。
- Configure centralized reporting across organizational units.在组织单元之间配置集中式报告。
- Set up cross-account experiment coordination.设置跨账户实验协同。
- Distribute shared experiment templates across organizational units using AWS Organizations.使用 AWS Organizations 在组织单元之间分发共享实验模板。
- Deploy distributed worker pools for parallel testing across your multiple applications.为多个应用部署分布式工作池,实现并行测试。
- Implement executive dashboards using Amazon QuickSight to track resilience trends across your portfolio.使用 Amazon QuickSight 实现执行仪表盘,跟踪整个组合的弹性趋势。
When to move to enterprise patterns何时进入企业模式
Consider adopting the enterprise deployment patterns in the next section when you meet any of these criteria:当您满足以下任意条件时,请考虑采用下一节的企业部署模式:
- You manage dozens of applications across multiple AWS accounts.您管理数十个跨多个 AWS 账户的应用。
- Multiple business units require differentiated resilience policies based on varying risk tolerances, compliance requirements, or customer SLAs.多个业务单元需要基于不同风险容忍度、合规要求或客户 SLA 的差异化弹性策略。
- Compliance requirements demand centralized audit trails across accounts.合规要求需要跨账户的集中审计日志。
- Your testing cadence exceeds what a single-account setup can handle.您的测试频率超出单账户设置的承载能力。
Design considerations for enterprise deployment企业部署的设计考量
When you’re ready to scale beyond initial pilots, consider these enterprise deployment patterns that handle the unique challenges of managing resilience testing across large organizations.当您准备好超出初始试点规模时,请考虑以下企业部署模式,以应对大组织管理弹性测试的独特挑战。

Figure 4. Enterprise scalability patterns图 4. 企业可扩展性模式
Diagram showing multi-account structure with native AWS Resilience Hub and AWS Organizations integration. A central management account connects to Production and Non-Production organizational units with distributed application accounts. The framework extends this native integration with cross-account FIS coordination (coordinated experiments across OUs) and shared FIS experiment templates distributed via Organizations. Priority-based scheduling shows Tier 1 Mission-Critical (weekly assessments, 100+ applications), Tier 2 Business-Critical (monthly assessments, 500+ applications), and Tier 3 Non-Critical (quarterly assessments, 1000+ applications). Distributed worker pools operate across multiple AWS regions (us-east-1, us-west-2, eu-west-1, ap-southeast-1) with centralized monitoring via Amazon EventBridge, Amazon QuickSight dashboards, Amazon CloudWatch logs, and Amazon SNS notifications.图示显示多账户结构,集成原生 AWS Resilience Hub 与 AWS Organizations。 中央管理账户连接到生产和非生产组织单元,分布式应用账户。 框架在此原生集成基础上添加跨账户 FIS 协调(跨 OU 实验)和通过 Organizations 分发的共享 FIS 实验模板。 优先级调度显示 Tier 1 关键任务(每周评估,100+ 应用)、Tier 2 业务关键(每月评估,500+ 应用)和 Tier 3 非关键(每季评估,1000+ 应用)。 分布式工作池跨多个 AWS 区域(us-east-1、us-west-2、eu-west-1、ap-southeast-1)运行,集中监控通过 Amazon EventBridge、Amazon QuickSight 仪表盘、Amazon CloudWatch 日志和 Amazon SNS 通知实现。
Multi-account architecture多账户架构
The next generation of AWS Resilience Hub supports modular resilience policies that you can assign at the system, user journey, or service level. Choose this multi-account strategy if you manage more than 100 applications across different business units. When you have thousands of applications, assessing your workloads simultaneously from a single account becomes impractical. Implement a hub-and-spoke model (a centralized architecture pattern where a central “hub” account manages shared services while “spoke” accounts contain individual workloads) with centralized resilience testing infrastructure and distributed application ownership. Deploy AWS Resilience Hub and AWS Fault Injection Service in your central management account. Your production accounts contain application workloads and local AWS Config recorders. The next generation of AWS Resilience Hub natively integrates with AWS Organizations, enabling central teams to define resilience policies and monitor posture across all accounts and regions from a single dashboard. This framework extends native multi-account visibility with cross-account experiment coordination and shared experiment templates across organizational units. For guidance on structuring your multi-account environment, see Best practices for a multi-account environment and the Organizing Your AWS Environment Using Multiple Accounts whitepaper.下一代 AWS Resilience Hub 支持模块化弹性策略,可在系统、用户旅程或服务层面分配。 若您管理超过 100 个跨业务单元的应用,请选择此多账户策略。 当您拥有数千个应用时,从单一账户同时评估所有工作负载变得不切实际。 实施中心-辐射(hub‑and‑spoke)模型——中心“枢纽”账户管理共享服务,辐射“辐射”账户承载各自工作负载——并在中心部署弹性测试基础设施,分布式拥有应用所有权。 在中心管理账户部署 AWS Resilience Hub 和 AWS Fault Injection Service。 生产账户包含应用工作负载和本地 AWS Config 记录器。 下一代 AWS Resilience Hub 原生集成 AWS Organizations,使中心团队能够在单一仪表盘上定义弹性策略并监控所有账户和区域的姿态。 本框架在原生多账户可视化之上,添加跨账户实验协同和组织单元共享实验模板。 有关多账户环境结构的指南,请参阅《多账户环境最佳实践》和《使用多个账户组织您的 AWS 环境》白皮书。
Tiered resilience policies分层弹性策略
Not every workload justifies the same resilience investment. Implement tiered resilience based on your business impact analysis. For example, mission-critical applications might target stricter recovery objectives (such as RTO < 15 minutes, RPO < 5 minutes, 99.99% availability) with comprehensive quarterly chaos experiments, while business-critical applications might set moderate targets (such as RTO < 1 hour, RPO < 15 minutes, 99.9% availability) with monthly validation, and non-critical applications might accept longer recovery windows with quarterly assessments. This tiered strategy optimizes costs by focusing resilience investments on your highest-impact workloads.并非所有工作负载都需要相同的弹性投资。应根据业务影响分析实施分层弹性。例如,关键任务应用可以设定更严格的恢复目标(如 RTO < 15 分钟,RPO < 5 分钟,99.99% 可用性),并进行全面的季度混沌实验;业务关键应用可以设定适度的目标(如 RTO < 1 小时,RPO < 15 分钟,99.9% 可用性),并进行每月验证;非关键应用可以接受更长的恢复窗口,并进行季度评估。这种分层策略通过将弹性投资集中在影响最大的工作负载上来优化成本。
Security and compliance安全与合规
Under the AWS Shared Responsibility Model, AWS is responsible for security of the cloud (infrastructure), while you are responsible for security in the cloud (your configurations, data, and access management). The controls described below are your responsibility to configure and maintain. Encrypt your data at rest using AWS Key Management Service (AWS KMS) with customer-managed keys. Implement separate encryption keys per environment to limit the scope of unauthorized access in your infrastructure. Enforce Transport Layer Security 1.3 (TLS 1.3), a cryptographic protocol that secures data transmission, for data in transit. AWS CloudTrail delivers complete audit trails of your resilience operations. Automated compliance monitoring through AWS Security Hub (which continuously evaluates resources against standards including CIS AWS Foundations Benchmark, PCI DSS, and AWS Foundational Security Best Practices), combined with AWS Config conformance packs, streamlines evidence collection for Service Organization Control 2 Type II (SOC 2 Type II), International Organization for Standardization 27001 (ISO 27001), Payment Card Industry Data Security Standard (PCI DSS), and Digital Operational Resilience Act (DORA), an EU regulation requiring financial institutions to test operational resilience.在 AWS 共享责任模型下,AWS 负责云的安全(基础设施),而您负责云内的安全(您的配置、数据和访问管理)。以下控制措施由您负责配置和维护。使用 AWS 密钥管理服务(AWS KMS)和客户管理的密钥对静态数据进行加密。为每个环境实施单独的加密密钥,以限制基础设施中未授权访问的范围。对传输中的数据强制使用传输层安全协议 1.3(TLS 1.3),这是一种保护数据传输的加密协议。AWS CloudTrail 提供完整的弹性操作审计日志。通过 AWS Security Hub(持续评估资源是否符合包括 CIS AWS Foundations Benchmark、PCI DSS 和 AWS 基础安全最佳实践在内的标准)以及 AWS Config 合规包实现自动化合规监控,简化了对服务组织控制 2 类型 II(SOC 2 Type II)、国际标准化组织 27001(ISO 27001)、支付卡行业数据安全标准(PCI DSS)和数字运营弹性法案(DORA——欧盟要求金融机构测试运营弹性)的证据收集。
AI agent security: The Amazon Bedrock AgentCore-hosted agent operates with scoped IAM roles following least-privilege principles. AgentCore Runtime’s MicroVM session isolation makes sure that each discovery session runs in a dedicated, ephemeral environment with no cross-session data leakage. The agent’s infrastructure access is read-only during discovery and cannot modify resources. Amazon Bedrock interactions occur within your AWS account boundary, and no customer data is used for training purposes. For additional guardrails, you can configure Amazon Bedrock Guardrails to filter agent outputs and enforce responsible AI policies.AI 代理安全:Amazon Bedrock AgentCore 托管的代理使用遵循最小特权原则的受限 IAM 角色。AgentCore Runtime 的 MicroVM 会话隔离确保每个发现会话在专用的短暂环境中运行,防止会话之间的数据泄漏。代理在发现期间对基础设施的访问为只读,无法修改资源。Amazon Bedrock 的交互发生在您的 AWS 账户边界内,且不会使用客户数据进行训练。若需额外防护,您可以配置 Amazon Bedrock Guardrails 来过滤代理输出并强制执行负责任的 AI 政策。
Note: This framework supports your compliance efforts but does not guarantee compliance with any regulatory framework. Compliance is a shared responsibility. Consult your legal and compliance teams and qualified auditors to validate that your implementation meets your specific regulatory obligations.注意:此框架支持您的合规工作,但不保证符合任何监管框架。合规是共享责任。请咨询您的法律和合规团队以及合格审计员,以验证您的实现是否满足特定的监管义务。
Addressing common concerns解决常见关注点
Progressive scope expansion and automated stop conditions help you verify experiments reveal weaknesses without causing outages in your environments. Starting with 1% of your resources limits potential impact to statistically insignificant traffic. Organizations have validated this strategy using progressive scope expansion and automated stop conditions.渐进式范围扩展和自动停止条件帮助您验证实验能够揭示弱点,而不会导致环境中出现故障。首先仅使用 1% 的资源,将潜在影响限制在统计上不显著的流量上。各组织已使用渐进式范围扩展和自动停止条件验证了此策略。
Automated scenarios remove the expertise barrier. The AI-powered analysis examines your specific architecture to develop targeted experiments rather than demanding you design tests manually. AWS CloudTrail provides comprehensive audit trails of your chaos experiments, which can help support due diligence documentation for resilience testing. This evidence can contribute to your compliance documentation for SOC 2, ISO 27001, and other frameworks relevant to your organization. For financial services, the framework supports DORA scenario testing requirements.自动化场景消除了专业壁垒。AI 驱动的分析会检查您的特定架构,制定有针对性的实验,而不是要求您手动设计测试。AWS CloudTrail 为您的混沌实验提供完整的审计日志,可帮助支持弹性测试的尽职调查文档。这些证据可用于 SOC 2、ISO 27001 等组织相关框架的合规文档。对于金融服务行业,该框架支持 DORA 场景测试要求。
Clean up清理
To avoid ongoing charges, delete the resources you created during implementation:为避免持续产生费用,请删除您在实施过程中创建的资源:
- Delete your AWS Fault Injection Service experiment templates. Warning: This permanently removes experiment history and results. Consider exporting experiment data before deletion if you need to retain this information for compliance or analysis purposes.删除您的 AWS Fault Injection Service 实验模板。警告:这将永久删除实验历史和结果。如果需要保留这些信息以满足合规或分析需求,请在删除前导出实验数据。
- Remove your Amazon Bedrock AgentCore agent deployment, runtime endpoints, and associated configurations.删除您的 Amazon Bedrock AgentCore 代理部署、运行时端点以及相关配置。
- Delete your AWS Systems Manager automation documents. Warning: This removes your automation runbooks permanently. Back up any custom runbooks you may want to reuse in future implementations.删除您的 AWS Systems Manager 自动化文档。警告:这将永久删除您的自动化运行手册。请备份任何希望在未来实现中重复使用的自定义运行手册。
- Remove your Amazon CloudWatch alarms created for stop conditions.删除为停止条件创建的 Amazon CloudWatch 告警。
- Delete any AWS Step Functions state machines created for approval workflows.删除为审批工作流创建的任何 AWS Step Functions 状态机。
- Remove any Open Policy Agent configurations deployed for IaC validation.删除为 IaC 验证部署的任何 Open Policy Agent 配置。
- Delete Amazon QuickSight dashboards created for resilience tracking (if applicable). Warning: This removes resilience trend data and operational insights. Export dashboard data or save analysis snapshots before deletion.删除为弹性跟踪创建的 Amazon QuickSight 仪表板(如适用)。警告:这将删除弹性趋势数据和运营洞察。请在删除前导出仪表板数据或保存分析快照。
- Remove Amazon EventBridge rules and Amazon SNS topics created for notifications (if applicable).删除为通知创建的 Amazon EventBridge 规则和 Amazon SNS 主题(如适用)。
Your AWS Resilience Hub and AWS Config continue incurring minimal costs. Consider retaining them for ongoing resilience validation.您的 AWS Resilience Hub 和 AWS Config 仍会产生少量费用。可考虑保留它们以进行持续的弹性验证。
Conclusion结论
In this post, I showed you how to build a five-layer AI-powered resilience framework that automatically discovers dependencies, generates targeted experiments, and integrates with your CI/CD pipelines. Building on the next generation of AWS Resilience Hub’s native dependency discovery and generative AI-powered failure mode analysis, this framework adds automated experiment generation through Amazon Bedrock AgentCore, controlled execution via AWS Fault Injection Service, and continuous CI/CD validation through AWS Systems Manager, creating an end-to-end resilience pipeline that goes from discovery to prevention.在本文中,我向您展示了如何构建一个五层 AI 驱动的弹性框架,该框架可自动发现依赖关系、生成有针对性的实验,并与您的 CI/CD 流水线集成。基于下一代 AWS Resilience Hub 原生依赖发现和生成式 AI 驱动的失效模式分析,此框架通过 Amazon Bedrock AgentCore 实现自动实验生成,通过 AWS Fault Injection Service 实现受控执行,并通过 AWS Systems Manager 实现持续的 CI/CD 验证,形成从发现到预防的端到端弹性流水线。
The next frontier is shifting even earlier, scanning your Infrastructure as Code and application code for resilience anti-patterns before a single resource is deployed. When your CI/CD pipeline can flag a missing circuit breaker or a single-AZ dependency at the pull request stage, prevention becomes truly proactive.下一步的前沿是更早介入,在任何资源部署之前扫描您的基础设施即代码和应用代码,查找弹性反模式。当您的 CI/CD 流水线能够在拉取请求阶段标记缺失的断路器或单可用区依赖时,预防就真正变得主动。
The progressive strategy (starting with your single application pilot, expanding to multiple applications, then scaling organization-wide) builds confidence while demonstrating value at each phase. Organizations often realize positive return on investment through prevented events and reduced MTTR.渐进式策略(先从单个应用试点开始,扩展到多个应用,最终在全组织范围内推广)在每个阶段都建立信心并展示价值。组织通常通过防止事件和降低 MTTR 实现正向投资回报。
The framework makes resilience testing accessible to you, removing the expertise barrier that traditionally prevented adoption. Start with the pilot phase outlined earlier and expand to your mission-critical systems as confidence builds.该框架让弹性测试变得易于使用,消除了传统上阻碍采用的专业壁垒。先从前文概述的试点阶段开始,随着信心的提升再扩展到关键业务系统。
Ready to get started? Pick a non-critical application from your portfolio, deploy the discovery agent, and run your first assessment this week. Then share what you found. We’d love to hear about the hidden dependencies your team uncovered.准备好开始了吗?从您的产品组合中挑选一个非关键应用,部署发现代理,并在本周运行首次评估。随后分享您的发现。我们很想了解您团队发现的隐藏依赖。
Have you implemented chaos engineering in your organization? What challenges did you face? Share your experience in the comments below.您在组织中实施过混沌工程吗?遇到了哪些挑战?请在下方评论中分享您的经验。
Next steps后续步骤
For the latest Resilience Hub capabilities including native dependency discovery and generative AI-powered failure mode analysis, see Introducing the next generation of AWS Resilience Hub and the next generation documentation.欲了解包括原生依赖发现和生成式 AI 驱动失效模式分析在内的最新 Resilience Hub 功能,请参阅《Introducing the next generation of AWS Resilience Hub》和下一代文档。
Start with the resources most relevant to where you are in your resilience journey:从最符合您当前弹性旅程阶段的资源开始:
If you’re just getting started:如果您刚刚起步:
- For more information about resilience policies and assessment capabilities, see AWS Resilience Hub documentation.有关弹性策略和评估能力的更多信息,请参阅 AWS Resilience Hub 文档。
- For hands-on experience with chaos engineering, see AWS Fault Injection Service Workshop.想要动手体验混沌工程,请参阅 AWS Fault Injection Service Workshop。
If you’re ready to build:如果您已准备好构建:
- For information about creating fault injection experiments using natural language through Amazon Bedrock, see Chaos engineering made clear: Generate AWS FIS experiments using natural language through Amazon Bedrock.有关通过 Amazon Bedrock 使用自然语言创建故障注入实验的信息,请参阅《Chaos engineering made clear: Generate AWS FIS experiments using natural language through Amazon Bedrock》。
- For information about assessing application resilience with AWS Resilience Hub and AWS CodePipeline, see Continually assessing application resilience with AWS Resilience Hub and AWS CodePipeline.有关使用 AWS Resilience Hub 和 AWS CodePipeline 评估应用弹性的信息,请参阅《Continually assessing application resilience with AWS Resilience Hub and AWS CodePipeline》。
- For additional fault injection experiment templates, see AWS Fault Injection Service Template Library.更多故障注入实验模板,请参阅 AWS Fault Injection Service Template Library。
If you’re scaling to production: 6. For information about hosting production AI agents at scale, see Amazon Bedrock AgentCore 7. For sample agent code and deployment templates, see the Amazon Bedrock AgentCore Starter Toolkit.如果您正在向生产环境扩展:6. 有关大规模托管生产 AI 代理的信息,请参阅 Amazon Bedrock AgentCore。7. 有关示例代理代码和部署模板,请参阅 Amazon Bedrock AgentCore Starter Toolkit。
Systems will face failures. Discover and fix weaknesses before your customers experience them by shifting your resilience testing from reactive response to proactive prevention.系统终将出现故障。通过将弹性测试从被动响应转向主动预防,提前发现并修复弱点,避免客户遭遇问题。
If you have questions or need guidance implementing the framework, contact AWS or reach out to your AWS Solutions Architect.如果您有任何问题或需要实施框架的指导,请联系 AWS 或您的 AWS 解决方案架构师。