Stripe Benchmark Shows AI Agents Build Integrations but Struggle with Validation Stripe 基准测试显示：AI 智能体能构建集成，但在验证环节仍面临挑战

Stripe has introduced a benchmark suite to evaluate whether AI agents can build real Stripe integrations end-to-end across backend services, frontend applications, and browser-based checkout flows. The benchmark goal is to measure how far AI systems can move beyond code generation into full software engineering workflows that require execution, testing, and validation in realistic environments. The focus is on production-style integration scenarios in financial systems where correctness is critical and partial success is not sufficient.Stripe 推出了一套基准测试套件，旨在评估 AI 智能体是否能够端到端地构建真实的 Stripe 集成，涵盖后端服务、前端应用及基于浏览器的结账流程。该基准测试的目标是衡量 AI 系统在多大程度上能超越代码生成，进入需要执行、测试和验证的真实软件工程工作流。其重点在于金融系统中生产环境级别的集成场景，在这些场景中，准确性至关重要，部分成功是远远不够的。

The benchmark is built around 11 reproducible environments simulating Stripe integration projects such as Checkout migration and Billing API modeling. Each environment includes full application codebases, databases, scripts, and test Stripe API keys. Agents are evaluated on backend-only tasks, full-stack workflows involving browser-based checkout flows, and product-specific exercises such as subscriptions and Checkout integrations.该基准测试围绕 11 个可复现的环境构建，模拟了如结账迁移（Checkout migration）和计费 API 建模等 Stripe 集成项目。每个环境都包含完整的应用程序代码库、数据库、脚本和测试用的 Stripe API 密钥。智能体将接受后端任务、涉及基于浏览器结账流程的全栈工作流，以及针对订阅和结账集成等特定产品练习的评估。

Agents operate through a consistent harness based on Goose and Model Context Protocol (MCP), with terminal access, browser automation, and documentation retrieval tools. Tasks require not only code generation but also running services, interacting with APIs, and validating end-to-end behavior using automated tests or simulated user flows. Deterministic graders validate results through API calls, UI automation, and inspection of Stripe objects such as Checkout Sessions. While Stripe does not publish a single aggregate success rate, results vary significantly by task type, with stronger performance on backend integrations and weaker outcomes when cross-system validation and state tracking are required.智能体通过基于 Goose 和模型上下文协议（MCP）的一致性工具集进行操作，具备终端访问、浏览器自动化和文档检索功能。任务不仅要求生成代码，还要求运行服务、与 API 交互，并使用自动化测试或模拟用户流程来验证端到端行为。确定性评分器通过 API 调用、UI 自动化以及对结账会话（Checkout Sessions）等 Stripe 对象的检查来验证结果。虽然 Stripe 没有公布单一的综合成功率，但结果因任务类型而异：在后端集成方面表现较强，而在需要跨系统验证和状态跟踪时表现较弱。

In evaluation breakdowns, Claude Opus 4.5 achieved 92 percent average scores on full-stack API integration tasks across four scenarios, while GPT 5.2 reached 73 percent on structured gym-style tasks across two scenarios. Best-performing runs sustained an average of 63 interaction turns, indicating improved long-horizon execution, although correctness degradation still appeared in extended workflows.在评估细分中，Claude Opus 4.5 在四个场景的全栈 API 集成任务中获得了 92% 的平均分，而 GPT 5.2 在两个场景的结构化“健身房式”任务中达到了 73%。表现最好的运行平均持续了 63 个交互轮次，表明长周期执行能力有所提升，尽管在扩展工作流中仍会出现准确性下降的情况。

Full Stack Benchmarking result (Source: Stripe Blog Post)全栈基准测试结果（来源：Stripe 博客文章）

Carol L, Software Engineer at Stripe, noted in a LinkedIn post that the core limitation is validation rather than code generation.Stripe 软件工程师 Carol L 在 LinkedIn 上指出，核心限制在于验证，而非代码生成。

AI agents are not going to replace software engineers yet, at least not when building Stripe integrations, financial systems require strict correctness and that current agents lack a stable validation layer for integration workflows.AI 智能体目前还无法取代软件工程师，至少在构建 Stripe 集成时是这样。金融系统要求极高的准确性，而当前的智能体在集成工作流中缺乏稳定的验证层。

Two recurring failure modes are highlighted. In SDK upgrade scenarios, agents sometimes misinterpret validation signals. When given invalid Stripe inputs, they observe expected HTTP 400 responses and incorrectly conclude the integration is successful. In more robust runs, agents generate synthetic test data and use it to validate behavior correctly.文中重点提到了两种反复出现的故障模式。在 SDK 升级场景中，智能体有时会错误解读验证信号。当输入无效的 Stripe 数据时，它们观察到预期的 HTTP 400 响应，却错误地得出集成成功的结论。在更稳健的运行中，智能体会生成合成测试数据，并利用这些数据正确验证行为。

A second failure mode appears in browser-based checkout flows. Agents are required to complete a full payment process through a web interface, including entering address and card details and producing a Checkout Session ID. Tool interactions can disrupt browser state, such as shifting focus away from input fields. Although recovery is possible through refresh or refocus actions, agents often fail to recover and terminate tasks prematurely.第二种故障模式出现在基于浏览器的结账流程中。智能体需要通过 Web 界面完成完整的支付过程，包括输入地址和银行卡详细信息，并生成结账会话 ID。工具交互可能会破坏浏览器状态，例如将焦点从输入字段移开。尽管可以通过刷新或重新聚焦操作进行恢复，但智能体往往无法恢复并会过早终止任务。

Practitioners observing the benchmark note that观察该基准测试的从业者指出：

Many agent evaluations still miss production concerns such as idempotency, retries, and authorization scope errors, which frequently cause real integration failures. The benchmark therefore highlights limitations less in code generation and more in validation reasoning, state management, and recovery in multi-step execution.许多智能体评估仍然忽略了生产环境中的关注点，如幂等性、重试和授权范围错误，这些往往是导致实际集成失败的原因。因此，该基准测试强调的局限性更多在于验证推理、状态管理和多步执行中的恢复能力，而非代码生成能力。

Stripe positions the benchmark as an evolving framework for studying agentic software engineering in realistic environments. It has been open-sourced as part of its AI toolkit, enabling further experimentation. Future iterations are expected to improve handling of ambiguous validation signals, browser state continuity, and end-to-end integration correctness in production-like systems.Stripe 将该基准测试定位为一个不断发展的框架，用于研究真实环境下的智能体软件工程。它已作为其 AI 工具包的一部分开源，以支持进一步的实验。预计未来的迭代将改善对模糊验证信号的处理、浏览器状态的连续性，以及生产级系统中端到端集成的准确性。

About the Author

Leela Kumili

Leela is a Lead Software Engineer at Starbucks with deep expertise in building scalable, cloud-native systems and distributed platforms. She drives architecture, delivery, and operational excellence across the Rewards Platform, leading efforts to modernize systems, improve scalability, and enhance reliability. In addition to her technical leadership, Leela serves as an AI Champion for the organization, identifying opportunities to improve developer productivity and workflows using LLM-based tools and establishing best practices for AI adoption. She is passionate about building production-ready systems, enhancing developer experience, and mentoring engineers to grow in both technical and strategic impact. Her interests include platform engineering, distributed systems, developer productivity, and bridging technical solutions with business and product goals.

Show moreShow less