3 production patterns for AI agents and how to evaluate each oneAI Agent 的三种生产模式及其评估之道

You have probably built an agent, but shipping one is a different job.想必你已亲手造过 Agent，但要将其真正推向生产环境，却是另一番光景。

A local coding agent, an in-app customer assistant, and an AI SRE triaging production logs may all use the same model class. But they don’t have the same harness, eval plan, rollout risk, or failure modes. The useful distinction is not “agent vs. no agent”—it’s which kind of production agent you are building.无论是本地编程助手、应用内客服，还是负责排查生产日志的 AI SRE，它们或许用着同一个模型，但其架构底座、评估方案、发布风险与故障模式却大相径庭。区分 Agent 的关键，不在于“是否用了 Agent”，而在于你究竟在打造哪一类生产级 Agent。

That framing comes from a talk by Sam Bhagwat, CEO and founder of Mastra, at Arize Observe 2026. Drawing on thousands of teams moving from prototype to production, Sam’s point was simple: everybody’s talking about agents—the real question is what kind. This post walks through Mastra’s view of a production agent stack, the three patterns that most often make it into production, and the practical risks each one tends to hit first.此番洞见源自 Mastra 创始人兼 CEO Sam Bhagwat 在 Arize Observe 2026 大会上的演讲。Sam 见证了成千上万支团队从原型走向生产，他指出：人人都在谈论 Agent，但真正的核心在于明确其类别。本文将带你领略 Mastra 眼中的生产级 Agent 架构，剖析三种最常见的生产模式，并指明每种模式首当其冲的实战风险。

Mastra’s view of a production agent stackMastra 眼中的生产级 Agent 架构

Before we jump into the pattern table, it helps to name what Mastra thinks production work actually is.在步入模式详解前，先厘清何谓“生产级工作”至关重要。

According to Sam, when you build an agent, you are solving two problems at once. One is the primitives—the agent loop, workflows, harness, sandboxes. The other is the features you need to ship to users. Teams that reinvent the primitives on every project move slower. Mastra’s view is to give you those primitives out of the box so you can spend the cycle on product:Sam 认为，构建 Agent 犹如双管齐下：其一是打磨底层基建——Agent 循环、工作流、执行框架与沙箱；其二则是交付用户所需的功能。若团队在每个项目中都重造这些基建，进度必受阻滞。Mastra 的理念是提供现成的基建，让你能将精力倾注于产品本身：

Pick your production pattern择定你的生产模式

Before you write another prompt, decide which bucket you’re in. Your harness, eval plan, and rollout strategy all depend on it.在写下下一行 Prompt 之前，先定好位。你的执行框架、评估方案与发布策略，皆系于此。

Tip: Not sure which category you fall under? You can ask these questions:小贴士：若拿不准归属，不妨自问以下几点：

• Are real users (not employees) the customer? Customer-facing客户是否为真实用户（而非内部员工）？——面向客户型
• Are you automating internal workflows at org scale? Internal enterprise是否在组织规模下自动化内部工作流？——内部企业型
• Are you building the stack other developers will ship agents on? Developer platform是否在构建供其他开发者部署 Agent 的平台？——开发者平台型

If you are building a customer-facing agent若是面向客户的 Agent

From Mastra’s side of the room, this pattern is context engineering more than model shopping. Your users want an in-app agent that understands their account, their data, their workflow. Frontier models are increasingly good at general reasoning, but they do not automatically know your product state, customer permissions, account history, or workflow constraints. Sam’s line from Observe: consumer models go off the rails not because they’re bad at reasoning, but because they don’t have access to the right context. So your job isn’t picking a smarter model—it’s feeding the model everything it needs to know about your users.在 Mastra 看来，此模式重在“上下文工程”而非“模型选型”。用户需要的是能理解其账户、数据与工作流的助手。前沿模型虽擅长通用推理，却无法凭空知晓你的产品状态、权限设置或业务约束。正如 Sam 在会上所言：消费级模型失控，非因推理能力不足，实因缺乏正确的上下文。故你的职责不在于寻找更强的模型，而在于为模型喂入它所需的全部用户背景。

Design for context, not model swaps以上下文为本，莫纠结于模型更迭

Factorial (14,000 HR customers) watched users export CSV dumps and paste them into Claude or ChatGPT for salary analysis. Indeed built a career counselor agent on Mastra the other way: users bring resumes, the platform brings jobs and salary data. In both cases, the value is model plus data. If you swap in a newer model and quality flatlines, you likely have a context engineering problem.Factorial 曾见用户导出 CSV 再粘入 Claude 或 ChatGPT 进行薪酬分析。Indeed 则在 Mastra 上构建了职业咨询 Agent：用户提交简历，平台提供岗位与薪资数据。两者皆是“模型+数据”的价值体现。若你更换了新模型却发现质量不升反降，那多半是上下文工程出了岔子。

Before launch, make sure the agent can reach the account-scoped data users expect it to know, the policy docs and guardrails generic chatbots lack, and retrieval paths you can actually inspect in traces.上线前，务必确保 Agent 能触达用户预期的账户数据、通用聊天机器人所欠缺的策略文档与护栏，以及在追踪链路中可供审视的检索路径。

Plan for two production failures谨防两类生产故障

Mastra sees customer-facing teams hit the same two walls after internal testing:Mastra 观察到，面向客户的团队在内部测试后，常会撞上两堵墙：

Cost. Staging hides usage intensity. A few power users can burn hundreds—or in some cases thousands—of dollars in inference before finance notices. Watch token burn in production observability dashboards from day one.成本。预演环境掩盖了真实流量强度。几位重度用户在财务察觉前，便可能消耗掉数以千计美元的推理费用。务必从第一天起，就在生产观测面板中紧盯 Token 消耗。

Accuracy. Yes, you write evals before launch. But as Sam put it, before you’ve shown the agent to any users, you don’t know the full breadth of questions they’ll ask—so your eval suite is not complete. How can it be? Evaluation fundamentals matter before launch. Evals on traces matter more after.准确性。诚然，上线前你已编写了评估用例。但如 Sam 所言，在用户真正使用前，你无法穷尽他们会提出的所有问题，故评估集难言完备。上线前的评估是基础，而上线后的链路评估才见真章。

Rollout checklist发布检查清单

• Ship to 1% or 5% of users, or run invite-only early access, before general availability在全面开放前，先向 1% 或 5% 的用户小范围发布，或采取邀请制内测。
• Route simpler query classes to cheaper models once you see real traffic patterns洞悉真实流量后，将简单查询请求分流至成本更低的模型。
• Put someone who has shipped an agent before on the critical path if you can若条件允许，让有 Agent 交付经验的资深人士把关关键路径。
• Expand eval coverage from production failures, not from guesswork基于生产故障而非臆测，不断扩充评估覆盖面。

If you are building an internal enterprise agent若是内部企业级 Agent

This pattern shows up in healthcare, finance, banks, and other orgs north of 1,000 employees. Internal data volume rivals customer scale. Your agent needs tool access across fragmented systems, not a single clean API—which is why Mastra’s stack leans on workflows, memory across connectors, and a harness security will approve.此模式常见于医疗、金融、银行等千人以上规模的组织。内部数据体量堪比客户规模。Agent 需跨越碎片化的系统进行工具调用，而非仅靠单一接口——这正是 Mastra 架构倚重工作流、跨连接器记忆及安全合规框架的原因。

Start with one high-volume workflow从一个高频工作流切入

• Enterprise search across wikis, drives, and ticketing systems跨 Wiki、网盘与工单系统的企业级搜索。
• Process automation for paperwork, approvals, and compliance针对文书、审批与合规流程的自动化处理。

Brex built an internal agent framework first, then consolidated on Mastra after learning what they actually needed. Prototype on one painful workflow. Standardize once the harness is real.Brex 先构建了内部 Agent 框架，在摸清需求后转向了 Mastra。先从一个痛点工作流进行原型验证，待框架稳固后再行标准化。

Expect org friction, too. Leadership may push agent projects without a crisp use case, you may spot high-impact workflows that aren’t on the roadmap, and the domain experts and builders who should be talking early often aren’t in the same room.亦要预见组织内部的摩擦。领导层可能在缺乏明确用例的情况下强推项目；你或许会发现高价值的工作流并未列入路线图；而本该尽早沟通的领域专家与开发者，往往未能同席而坐。

If the official project is stuck, find one partner team and one workflow you can instrument end to end. Pair domain experts with engineers who understand what an agent harness and evaluation harness need before anyone promises ROI upstairs.若官方项目受阻，不妨寻找一个合作团队，针对单一工作流进行端到端部署。让领域专家与深谙 Agent 框架及评估机制的工程师结对，在向高层承诺 ROI 之前，先稳扎稳打。

If you are building a developer platform agent若是开发者平台型 Agent

You are building agents for other engineers, or agents that multiply how fast your org ships agents. In Mastra’s framing, this is where the primitives matter most: a blessed way to build agents so every team isn’t reinventing auth, tracing, and deployment.你是在为工程师造工具，或是为了加速组织内 Agent 的交付。在 Mastra 的框架下，基建最为关键：提供一套“官方认可”的构建方式，避免各团队在鉴权、追踪与部署上重复造轮子。

Pick a shape选准方向

• AI SRE: ingest logs and telemetry at scale (some teams run agents over 10 TB of logs for triage and anomaly detection)AI SRE：大规模摄取日志与遥测数据（部分团队甚至处理超过 10TB 的日志用于故障排查与异常检测）。
• Internal agent platform: a blessed stack with approved connectors, like MongoDB’s Sage on Mastra, so hundreds of developers can spin up domain agents without reinventing auth, tracing, and deployment内部 Agent 平台：一套集成受控连接器的标准栈（如 MongoDB 在 Mastra 上构建的 Sage），使数百名开发者无需重造鉴权与追踪，即可快速启动领域 Agent。

Your advantage, from Sam’s talk: you are often your own customer. You can look at your own workload and say you reduced MTTR because you built an agent to triage the page. Pick a metric you own (MTTR, triage time, deploy frequency) and iterate against it.正如 Sam 所言，你的优势在于：你往往就是自己的客户。你可以审视自身负载，通过构建 Agent 优化故障响应时间（MTTR）来证明价值。选定一个你掌控的指标（如 MTTR、排查时间、部署频率），并以此为基准不断迭代。

Run the improvement loop from day one从第一天起运行改进循环

1. Trace the run追踪运行过程
2. Evaluate the failure评估故障成因
3. Change the harness优化执行框架
4. Rerun再次运行

That is the improvement loop in practice. Platform teams that skip it rebuild the same agent three times with different names.这便是实战中的改进循环。若平台团队跳过此步，往往会陷入不断更名、重复构建同类 Agent 的泥潭。

Production checklist (all three patterns)生产检查清单（适用于三种模式）

Local agents lie to you. A demo on a curated set does not prove your eval suite is complete or your cost model works at user scale. Before you call an agent “production,” check:本地 Agent 往往会欺骗你。在精选数据集上的演示，无法证明你的评估集完备，也无法证明成本模型在用户规模下依然稳健。在称其为“生产级”之前，请核查：

• Observability: Can you see tool calls, latency, token use, and failures on real sessions?可观测性：在真实会话中，能否清晰看到工具调用、延迟、Token 消耗与故障？
• Evals: Do you score behavior on traces, not just final answers in a notebook?评估：是否针对运行轨迹（Trace）而非仅是笔记中的最终答案进行评分？
• Harness: When evals fail, do you know whether to change prompts, tools, context, or rubrics?框架：当评估失败时，你是否明确该调整 Prompt、工具、上下文还是评分规则？
• Failure modes: Have you read common agent failures against your own traces? Tool misuse and missing context beat weak base models most days.故障模式：是否已对照自身追踪记录研读过常见的 Agent 故障？多数情况下，工具误用与上下文缺失，比模型底座羸弱更具杀伤力。

If you need a reference implementation for the loop, start with improving the agent harness with traces and evals.若需改进循环的参考实现，不妨从利用追踪与评估来优化 Agent 框架开始。

How to evaluate production agents如何评估生产级 Agent

Mastra’s stack names evals and observability as separate layers for a reason. Observability tells you what happened on a run. Evaluation tells you whether that run was good enough—and where to change the harness when it was not.Mastra 将评估与可观测性设为独立层级，自有深意。可观测性告诉你运行中发生了什么；评估则告诉你这次运行是否合格，以及若不合格，该从何处改进框架。

An agent eval is a repeatable check on a trace: the prompt, tool calls, workflow steps, and final output. Agent evals are not unit tests. The correct path is often under-specified—the agent may call three tools or seven and still succeed—so you encode outcomes and constraints (answer grounded in retrieved docs, approval ticket created before status flips to complete) rather than a single expected call sequence.Agent 评估是对运行轨迹（Prompt、工具调用、工作流步骤、最终输出）的一项可重复检查。它绝非简单的单元测试。正确的路径往往难以穷尽——Agent 调用三个工具或七个工具皆可能成功——因此，你应编码约束条件（如：答案必须基于检索到的文档、状态流转前必须创建审批工单），而非死板地校验特定的调用序列。

Where to attach evals:评估的挂载点：

Once you know which level you’re scoring at, the next choice is how you score it.明确了评分层级后，下一步便是选择评分方式。

Evaluator types teams use most:团队最常使用的评估器类型：

• Code evaluators: deterministic checks on tool arguments, JSON shape, or workflow state代码评估器：针对工具参数、JSON 结构或工作流状态进行确定性校验。
• Binary evaluators: pass/fail on one failure mode; see binary vs. score evals二元评估器：针对特定故障模式进行通过/失败判定。
• LLM-as-a-judge: rubric scoring on open-ended output; calibrate against human labels first (production judge guide)LLM 作为裁判：对开放式输出进行规则评分；务必先用人工标注进行校准。
• Harness-as-a-Judge: when a fixed template is too rigid, an agent scores inside full trace context (docs)框架作为裁判：当固定模板过于僵化时，让 Agent 在完整的追踪上下文中进行评分。

The loop that actually sticks (evaluation fundamentals):真正行之有效的循环（评估基本功）：

1. Instrument. Traces flowing from Mastra into Arize AX or Phoenix. The @mastra/arize package exports OpenTelemetry spans; see Mastra tracing docs埋点：通过 Mastra 将追踪数据接入 Arize AX 或 Phoenix。@mastra/arize 包可导出 OpenTelemetry Span。
2. Error analysis. Cluster failed or flagged traces; name the failure mode in plain language错误分析：将失败或被标记的追踪记录聚类，并用通俗语言命名故障模式。
3. Label. Domain experts or engineers annotate a sample in a labeling UI, not raw JSON in a spreadsheet标注：由领域专家或工程师在标注界面进行样本标注，而非在电子表格中处理原始 JSON。
4. Build evaluators. One evaluator per failure mode; prefer binary over uncalibrated 1–100 scales构建评估器：每种故障模式对应一个评估器；优先使用二元判定，而非未经校准的 1-100 分制。
5. Golden set. Hold out a test partition; do not tune rubrics on the same traces you judge against黄金集：保留测试分区；切勿在用于评分的同一批轨迹上调整规则。
6. Online evals. Run evals on production traces after staged rollout; expand the suite from new failures在线评估：在分阶段发布后对生产轨迹运行评估；根据新发现的故障不断扩充用例库。

What to focus on for your next sprint下个冲刺阶段的重点

The demos were never the hard part. Naming which pattern you are in, and building the evals and traces to know when you have shipped, is the real work.演示从来不是难点。明确你所处的模式，并建立相应的评估与追踪机制以确认交付，这才是真正的苦功。

Watch Mastra’s full talk from Arize Observe 2026: What Production AI Agent Teams Are Building Today.观看 Mastra 在 Arize Observe 2026 的演讲全集：《当今生产级 AI Agent 团队正在构建什么》。