The Platform Engineering Playbook for Production LLMs 生产级 LLM 的平台工程手册

By the end of our first month running an LLM-driven inventory recommendation system in production, roughly fifteen percent of agent responses were hallucinations, confident-sounding outputs grounded in nothing real.在我们运行基于 LLM 的库存推荐系统的第一个月结束时，大约 15% 的 Agent 响应是幻觉——即听起来自信但毫无现实依据的输出。

Six months later the rate was 1.5 percent and the lever that moved the number wasn't a better foundation model. It was the decision to stop treating the LLM stack as an application concern and start treating it as platform infrastructure.六个月后，这一比例降至 1.5%，而起到关键作用的并非更好的基础模型，而是我们决定停止将 LLM 栈视为应用程序问题，转而将其视为平台基础设施。

The project behind this article is an inventory accuracy platform at a large retail organization. In retail supply chains, system inventory records drift from what is physically on shelves through misplacement, damage, theft, and miscounts; that drift degrades ordering, replenishment and product availability.本文背后的项目是一个大型零售组织的库存准确性平台。在零售供应链中，系统库存记录会因错放、损坏、盗窃和盘点错误而与货架上的实际情况产生偏差；这种偏差会削弱订货、补货和产品可用性。

Our multi-agent LLM system analyzes discrepancy signals across millions of SKUs and billions of historical inventory records and generates corrective recommendations for our consumers who aim to maintain inventory accuracy at all times.我们的多 Agent LLM 系统分析了数百万个 SKU 和数十亿条历史库存记录中的差异信号，并为旨在时刻保持库存准确性的消费者生成纠正建议。

The platform layer described in this article was built in collaboration with a team of business, product, software engineers, data scientists, and data engineering experts, and now serves dozens of application teams across use cases like inventory audits, replenishment review, discrepancy triage, and more.本文描述的平台层是与业务、产品、软件工程师、数据科学家和数据工程专家团队协作构建的，目前服务于数十个应用团队，涵盖库存审计、补货审查、差异分类等用例。

This article documents that shift, routing through foundation models such as Gemini and GPT under enterprise traffic. The patterns below are the ones I wish had been in place on day one.本文记录了这一转变，涵盖了在企业流量下通过 Gemini 和 GPT 等基础模型的路由过程。以下模式是我希望在第一天就部署到位的。

Every primitive described here is implemented in a companion GitHub repository, built deliberately with no hidden abstractions. Where a framework would hide the mechanics behind a decorator or a built-in agent type, the repo keeps the moving parts visible so the patterns stay legible.此处描述的每个原语都在配套的 GitHub 仓库中实现，特意不使用隐藏的抽象。如果框架通过装饰器或内置 Agent 类型隐藏了机制，该仓库则保持各部件可见，以确保模式易于理解。

The code snippets in this article come directly from that repo. One deliberate difference from production exists. The repo runs a local model (llama3.1 via Ollama), connected through LiteLLM instead of a hosted foundation model, so the entire stack is runnable end-to-end at zero API cost.本文中的代码片段直接来自该仓库。与生产环境有一个刻意的区别：该仓库运行本地模型（通过 Ollama 运行 llama3.1），并连接到 LiteLLM 而非托管的基础模型，因此整个栈可以在零 API 成本的情况下端到端运行。

Why Individual LLM Apps Break at Scale为什么单个 LLM 应用会在规模化时崩溃

The failure pattern is consistent enough across teams to be predictable. The prototype works. The first production deployment looks fine for a week. Then the issues that don't show up in evaluations start showing up in dashboards. 这种故障模式在各团队间高度一致，因而具有可预测性。原型运行正常。第一次生产部署看起来一周内没问题。然后，那些在评估中未显现的问题开始在仪表板上浮现。

In our case, three failures surfaced in the beta testing, each traced back to something the prototype never exercised. The first was LLM API throttling. Under real request volume, we started being rate-limited by the model provider. The second was context. When the system recommended an item that didn't fit, the root cause was almost always the data we fed the model, not the model itself, which made robust data-engineering pipelines central to the fix. The third was the big one, hallucinations, which stayed individually rare but became a steady rate once we were serving production traffic. We had engineered the platform so these failures could be tracked. That visibility is what turned each of them from a mystery into something we could fix. 在我们的案例中，Beta 测试中出现了三次故障，每一次都可以追溯到原型从未测试过的情况。首先是 LLM API 限流。在真实请求量下，我们开始受到模型提供商的速率限制。其次是上下文。当系统推荐了一个不合适的商品时，根本原因几乎总是我们喂给模型的数据，而不是模型本身，这使得稳健的数据工程流水线成为修复的核心。第三个也是最大的问题是幻觉，虽然单个幻觉很少见，但一旦我们开始处理生产流量，幻觉就变成了一个稳定的比率。我们设计了该平台，以便能够跟踪这些故障。正是这种可见性将每一个问题从一个谜团变成了我们可以修复的对象。

Tracking them also made clear why they belonged in a platform layer rather than any one application. Three properties stood out:跟踪这些故障也清楚地表明了为什么它们应该属于平台层而不是任何单一应用。三个特性尤为突出：

• Hallucinations as output, not as bugs.
  A hallucinated response isn't a stack trace. It passes every traditional health check. The downstream service consumes it, takes action on it, and only much later does someone notice the suggested output doesn't exist.幻觉作为输出，而非 Bug。 幻觉响应不是堆栈跟踪。它可以通过所有的传统健康检查。下游服务消耗它、采取行动，直到很久之后才有人注意到建议的输出并不存在。
• Uncontrolled token spend.
  Without per-request attribution, monthly bills arrive as a single line item from the model vendor. There's no way to tell which application, team or use case is driving cost, so optimization is guesswork.不受控制的 Token 支出。 如果没有按请求的归因分析，每月的账单只是模型供应商提供的一个单一明细项。无法判断是哪个应用、团队或用例导致了成本，因此优化只能靠猜测。
• No behavioral observability.
  Latency dashboards show p99 looks fine. The error rate is near zero. Meanwhile the model is silently drifting into producing lower-quality outputs that look syntactically valid.缺乏行为可观测性。 延迟仪表板显示 p99 看起来没问题。错误率接近于零。与此同时，模型正在悄悄地漂移，产生语法有效但质量较低的输出。

None of these are application bugs. Every team that ships an LLM feature hits the same set of issues independently and the first instinct is to fix it inside the application that produces N copies of the same scaffolding code across N teams. That's the part of work that should have been pushed down via a platform layer.这些都不是应用 Bug。每个发布 LLM 功能的团队都会独立遇到同样的问题，而第一直觉是在各自的应用内部修复它，导致 N 个团队产生了 N 份相同的脚手架代码。这部分工作本应通过平台层下沉。

The Case for a Shared LLM Platform构建共享 LLM 平台的理由

The argument for centralizing LLM concerns isn't theoretical. It maps almost exactly onto the case for centralizing authentication, logging, or service mesh in any large-scale system: once more than one application has the same cross-cutting requirement, the cost of duplication starts to dominate the cost of building shared infrastructure.集中化 LLM 需求的论点并非理论上的。它几乎完全对应于任何大型系统中集中化认证、日志记录或服务网格的案例：一旦多个应用有相同的横切需求，重复构建的成本就会开始超过构建共享基础设施的成本。

For LLMs the shared concerns are:对于 LLM，共享需求包括：

• Retry logic and failure classification重试逻辑和故障分类
• Prompt registry and versioningPrompt 注册表和版本控制
• Schema enforcement模式强制执行
• Token cost attribution by request按请求的 Token 成本归因
• Behavioral observability行为可观测性
• Tool and context routing工具和上下文路由
• Authentication and Authorization across tool servers跨工具服务器的认证和授权

There is a second-order effect that matters more than the duplication. A shared platform forces a contract. Applications declare what they need (i.e., a model, a prompt version, a schema, a tool set, and an eval engine) and the platform handles how to deliver it. That contract is what lets a new application reach production in days instead of months, because the hard parts are already solved.存在一种比重复构建更重要的二阶效应。共享平台强制执行契约。应用声明它们需要什么（例如模型、Prompt 版本、模式、工具集和评估引擎），而平台负责如何交付。正是这种契约让新应用能在几天而非几个月内进入生产环境，因为困难的部分已经解决了。

The decision point for building this platform isn't the first LLM application. It's the second. By the time you have ten teams each shipping LLM features, retrofitting a platform underneath them is long and difficult migration work.构建此平台的决策点不是第一个 LLM 应用，而是第二个。等到你有十个团队各自发布 LLM 功能时，在它们之下补救一个平台将是一项漫长而困难的迁移工作。

The Platform Architecture平台架构

Before walking through the primitives individually, Figure 1, below, shows how they compose. The repo implements the full request path and it mirrors the production environment.在逐一介绍原语之前，下图 1 展示了它们是如何组合的。该仓库实现了完整的请求路径，并镜像了生产环境。

Figure 1. LLM platform architecture: gateway, coordinator, specialist agents, and per-team MCP servers. (Image created by Author)图 1. LLM 平台架构：网关、协调器、专用 Agent 和各团队的 MCP 服务器。（图片由作者创作）

A single gateway is the only ingress. It authenticates the caller, issues a role-carrying JWT, exposes the agent-execution endpoint, serves per-user and per-team metrics, and hosts the admin-only prompt-rollback routes. Behind it, a root coordinator agent, built with Google’s Agent Development Kit (ADK), classifies intent on every turn and delegates to one or more of the specialist agents, each bound to its own registry-resolved prompt and its own structured-output schema. The specialists own the tool connections, which are multiple independent MCP servers, partitioned by team or usage, each of which verifies the caller's token and enforces the authorization policy itself before any tool runs.单一网关是唯一的入口。它对调用者进行认证，签发携带角色的 JWT，暴露 Agent 执行端点，提供用户和团队维度的指标，并托管仅限管理员访问的 Prompt 回滚路由。在其后，一个由 Google Agent 开发工具包（ADK）构建的根协调器 Agent 会在每一步对意图进行分类，并委派给一个或多个专用 Agent，每个 Agent 都绑定到其注册表解析的 Prompt 和结构化输出模式。这些专用 Agent 拥有工具连接，即由团队或用途划分的多个独立的 MCP 服务器，每个服务器在运行任何工具之前都会验证调用者的 Token 并强制执行授权策略。

A note on the open-source tools underneath, because the layering is deliberate.关于底层开源工具的说明，因为分层是刻意的。

Orchestration is handled by Google's Agent Development Kit (ADK): It provides the agent and sub-agent topology, the delegation call the coordinator uses to hand off to a specialist, and the callback hooks where the platform records cost and latency.编排由 Google 的 Agent 开发工具包（ADK）处理：它提供了 Agent 和子 Agent 的拓扑结构、协调器用于移交给专用 Agent 的委派调用，以及平台记录成本和延迟的回调钩子。

Each agent reaches its model through LiteLLM, which is the seam that makes the model provider swappable; the same agent code runs against a local model when running the application on a laptop and against hosted foundation models such as Gemini or GPT in production, changing only a model identifier. 每个 Agent 通过 LiteLLM 访问其模型，这是使模型提供商可替换的接缝；相同的 Agent 代码在笔记本电脑上运行时针对本地模型，在生产环境中针对 Gemini 或 GPT 等托管基础模型，只需更改模型标识符。

In the companion repo that local model is served by Ollama, which is what lets the entire stack, gateway, agents, and tool servers run end to end on a laptop at zero API cost. 在配套仓库中，该本地模型由 Ollama 提供服务，这使得整个栈（网关、Agent 和工具服务器）可以在笔记本电脑上以零 API 成本端到端运行。

The tools themselves live behind the Model Context Protocol. ADK's MCP client connects the agents to the team servers, which are built with FastMCP on the other side.工具本身位于模型上下文协议（MCP）之后。ADK 的 MCP 客户端将 Agent 连接到团队服务器，这些服务器在另一端使用 FastMCP 构建。

The value of this separation is that no single framework owns the whole stack. The model layer, the tool layer, and the orchestration layer each sit behind a stable interface, so any one of them can be replaced by a stronger model, a different tool server, or even a different orchestrator without rewriting the platform primitives around them. That is the same "no hidden abstractions" discipline applied at the dependency level: Each tool does one job and the path between them stays visible.这种分离的价值在于没有单一框架拥有整个栈。模型层、工具层和编排层各自位于稳定的接口之后，因此其中任何一层都可以被更强的模型、不同的工具服务器甚至不同的编排器替换，而无需重写其周围的平台原语。这与依赖层面上应用的“无隐藏抽象”准则相同：每个工具只做一件事，它们之间的路径保持可见。

Structured Outputs: Enforcing Schema at the Platform Layer and Retrying on Failures结构化输出：在平台层强制执行模式并在失败时重试

The single most cost-effective change we made was enforcing structured outputs at the platform boundary rather than at the application boundary.我们采取的最具成本效益的改变是在平台边界而非应用边界强制执行结构化输出。

The pattern is unremarkable in concept. Every LLM call declares a target schema (we use Pydantic models in Python) and the platform validates the response against that schema before returning it to the caller. If validation fails, the platform re-prompts with the validation error embedded as additional context, up to a configurable retry limit.这种模式在概念上并不稀奇。每个 LLM 调用都声明一个目标模式（我们在 Python 中使用 Pydantic 模型），平台在将响应返回给调用者之前会根据该模式验证响应。如果验证失败，平台会嵌入验证错误作为额外上下文进行重新提示，直到达到可配置的重试限制。

What changes when this pattern lives at the platform layer rather than in each application:当这种模式存在于平台层而非每个应用中时，会有什么变化：

• One implementation handles all schema mismatches, so the strategy improves once for everyone.单一实现处理所有模式不匹配问题，因此该策略一次性改进了所有人。
• Downstream consumers can rely on the contract and if the call returns, the response shape is guaranteed.下游消费者可以依赖该契约，如果调用返回，则响应形状是有保证的。
• Schema violations become a metric the platform tracks, not a runtime exception buried in application logs.模式违规成为平台跟踪的指标，而不是埋在应用日志中的运行时异常。

Our output schema doesn't just format data; it serves as explicit instructions for the model, complete with dedicated tracking fields designed specifically to isolate and measure hallucinations. Figure 2, below, shows what a document specialist's output schema looks like in code using Pydantic.我们的输出模式不仅格式化数据；它还作为模型的明确指令，配有专门设计的跟踪字段，专门用于隔离和测量幻觉。下图 2 展示了文档专用 Agent 的输出模式在代码中如何使用 Pydantic 实现。

from pydantic import BaseModel, Field

class DocsOutput(BaseModel):
    """Result of a grounded documentation lookup."""

    answer: str = Field(
        description="The answer, grounded entirely in the fetched documentation."
    )
    sources: list[DocSource] = Field(
        default_factory=list,
        description="Documentation sources cited; empty only when nothing was fetched.",
    )
    grounded: bool = Field(
        description="True if every claim is backed by the fetched docs; "
        "False if the docs did not cover the question.",
    )
    out_of_scope: bool = Field(
        default=False,
        description="True if the request fell outside documentation lookup.",
    )

Figure 2. The DocsOutput schema (app/model.py).图 2. DocsOutput 模式 (app/model.py)。

Figure 3, below, shows what the model actually returns when it satisfies this schema. The answer is populated, sources lists the documentation the answer was drawn from and grounded is true precisely because those sources were fetched and used. Had the question fallen outside the available docs, the same structure would come back with grounded set to false and an empty sources list, a response the platform records as a hallucination signal rather than passing off as a confident answer.下图 3 展示了模型在满足此模式时实际返回的内容。答案已填充，sources 列出了从中提取答案的文档，grounded 为 true 是因为这些来源确实被获取并使用了。如果问题超出了可用文档的范围，同样的结构会返回 grounded 设置为 false 且 sources 列表为空，平台会将此响应记录为幻觉信号，而不是将其作为自信的答案传递出去。

{
   "answer": "FastAPI's APIRouter lets you group related path operations and mount them on the main app with app.include_router().",
   "library_name": "FastAPI",
   "version": "0.115",
   "sources": [
       {
           "library_id": "/tiangolo/fastapi",
           "topic": "routing"
       }
   ],
   "grounded": true,
   "out_of_scope": false
}

Figure 3. Example DocsOutput response.图 3. DocsOutput 响应示例。

The grounded and out_of_scope flags cost nothing to generate but force the model to commit, in machine-readable form, to whether its answer is backed by retrieved context. A response that fails validation, has the wrong shape, missing fields, or unparseable output is the highest-precision hallucination signal a platform has.grounded 和 out_of_scope 标志生成成本为零，但强制模型以机器可读的形式承诺其答案是否由检索到的上下文支持。验证失败、形状错误、字段缺失或无法解析输出的响应，是平台所能拥有的最高精度的幻觉信号。

The grounded flag only means something because of what happens before the model answers. Every specialist is connected to the platform's MCP servers, while tool results are the only evidence the model is allowed to use. The root agent's instruction states it plainly. Tool results are the source of truth for the answer.grounded 标志的意义仅在于模型回答之前发生的事情。每个专用 Agent 都连接到平台的 MCP 服务器，而工具结果是模型被允许使用的唯一证据。根 Agent 的指令明确说明了这一点。工具结果是答案的事实来源。

The docs specialist is the clearest example. Its prompt gives it two documentation tools, used in sequence. One resolves a library name to an ID, while the other fetches the current official docs for that library and topic. The rules around those tools are strict. The agent must fetch docs before answering any question about an API or a parameter, never from training data, which may be outdated. Every claim, code example, and parameter name must come from the fetched documentation. If the docs do not cover the question, the agent must say so instead of guessing.文档专用 Agent 是最清晰的例子。其 Prompt 为其提供了两个按顺序使用的文档工具。一个将库名称解析为 ID，另一个获取该库和主题的当前官方文档。围绕这些工具的规则很严格。Agent 在回答关于 API 或参数的任何问题之前必须获取文档，绝不能使用可能过时的训练数据。每一个声明、代码示例和参数名称都必须来自获取的文档。如果文档没有涵盖该问题，Agent 必须说明这一点，而不是猜测。

The schema then makes the model show its work. The "sources" field records one entry per lookup and "grounded" may be true only if every claim is backed by what the tools returned. The same pattern holds across specialists. When the execution agent reports a deployment, that claim comes from the MCP tool's response, not the model's memory. Grounding here is not a prompt trick, it is a pipeline. Tools fetch the evidence, the prompt restricts the model to it, the schema forces a declaration and validation turns a broken declaration into a counted, retryable failure.模式随后让模型展示其工作过程。“sources”字段记录每次查找的一个条目，只有当每个声明都被工具返回的内容所支持时，“grounded”才能为 true。同样的模式适用于所有专用 Agent。当执行 Agent 报告部署时，该声明来自 MCP 工具的响应，而不是模型的记忆。这里的“根据依据（Grounding）”不是 Prompt 技巧，而是一个流水线。工具获取证据，Prompt 将模型限制在证据内，模式强制进行声明，而验证则将错误的声明转化为可计数、可重试的故障。

Enforcing strict output schemas turns the vaguest AI failures into clear, typed data events giving you the exact signal needed to drive your retries, metrics, and evals using a single Pydantic model.强制执行严格的输出模式将最模糊的 AI 故障转化为清晰的、类型化的数据事件，为你提供使用单个 Pydantic 模型驱动重试、指标和评估所需的精确信号。

Retry Orchestration with Validation带验证的重试编排

This is the layer where most of the fifteen percent to 1.5 percent hallucination reduction came from, so it warrants a more detailed discussion.这是幻觉率从 15% 降至 1.5% 的主要来源层，因此值得更详细地讨论。

Blindly rerunning failed LLM calls is actively harmful. Repeating a prompt that hallucinated just yields a different hallucination, while naive retries during an outage create traffic storms that crush your infrastructure.盲目地重新运行失败的 LLM 调用是有害的。重复一个产生幻觉的 Prompt 只会产生另一种幻觉，而在中断期间进行天真的重试会造成流量风暴，从而压垮你的基础设施。

The pattern that worked for us:对我们有效的模式是：

Classify the Failure分类故障

Is it a schema violation, a hallucination signal (low confidence, factual ungroundedness, or off-intent generation) or an infrastructure error (timeout, rate limit, or model unavailable)? The three classes receive three different responses; collapsing them into one generic retry is the mistake.是模式违规、幻觉信号（低置信度、事实无依据或偏离意图的生成）还是基础设施错误（超时、速率限制或模型不可用）？这三类故障会收到三种不同的响应；将它们合并为一种通用的重试是错误的。

Route by Class按类别路由

Schema violations re-prompt with the error appended, so the model sees exactly what broke (for example, "grounded" returned as the string "yes" instead of a boolean, or a missing required "sources" field). Hallucination signals re-prompt with stronger grounding context (for example, the docs agent returning "grounded: false", or citing a source for a library it was never given). Infrastructure errors back off exponentially (for example, a 429 rate-limit or a 503 from the provider, the throttling we hit in our first release).模式违规会附加错误信息进行重新提示，因此模型能确切看到哪里出错了（例如，“grounded”作为字符串“yes”返回而不是布尔值，或者缺少必需的“sources”字段）。幻觉信号会以更强的基础上下文进行重新提示（例如，文档 Agent 返回“grounded: false”，或引用了从未给出的库的来源）。基础设施错误会进行指数级退避（例如 429 速率限制或来自提供商的 503，即我们在首次发布时遇到的限流问题）。

Cap the Retry Budget per Request限制每个请求的重试预算

If retries have no limit, one bad request can retry forever and run up a large bill, so every request needs a hard cap. There are a few ways to set that cap. The simplest is a fixed number of tries, three in our case, because a structured-output failure that hasn't fixed itself by the third try almost never fixes itself on the fourth. Another way is to limit the total tokens a request may spend, which controls the actual cost rather than the number of tries, since one retry on a long input can cost more than three on a short one. A third way is a time limit, which is useful when a fast answer matters more than a complete one. Any of these approaches work. What matters is that the limit exists and is enforced, so no single request can keep retrying without end.如果重试没有限制，一个糟糕的请求可能会无限重试并产生巨额账单，因此每个请求都需要硬性上限。设置上限的方法有几种。最简单的是固定重试次数，在我们的案例中是三次，因为如果结构化输出故障在第三次尝试后没有自行修复，那么在第四次尝试时几乎也不会修复。另一种方法是限制请求可能消耗的总 Token 数，这可以控制实际成本而不是次数，因为对长输入进行一次重试的成本可能高于对短输入进行三次重试。第三种方法是时间限制，当快速答案比完整答案更重要时很有用。任何这些方法都有效。重要的是存在上限并被强制执行，这样单个请求就不会无休止地重试。

Instead of swallowing malformed data and handing back broken objects, our platform wraps the agent's execution loop to catch validation errors, log them as hallucinations and trigger smart retries.我们的平台没有吞下格式错误的数据并返回损坏的对象，而是封装了 Agent 的执行循环来捕获验证错误，将其记录为幻觉并触发智能重试。

The agent wraps its own run loop, catching the "ValidationError" a badly structured output raises, recording each failed attempt as a hallucination, and retrying up to a configurable cap before re-raising, as shown in Figure 4.Agent 封装了自己的运行循环，捕获结构不良的输出引发的“ValidationError”，将每次失败的尝试记录为幻觉，并在重新抛出异常之前重试达到可配置的上限，如图 4 所示。

from google.adk.agents import LlmAgent

class CustomLlmAgent(LlmAgent):

    def __init__(self, *args, max_validation_retries: int = 3, **kwargs):
        super().__init__(*args, **kwargs)
        self._max_validation_retries = max_validation_retries

    @property
    def max_validation_retries(self) -> int:
        """How many times structured-output validation is retried before failing."""
        return self._max_validation_retries


    async def _run_async_impl(self, ctx):
        """Run the agent with retry logic for validation errors.

        Both structured-output paths raise pydantic.ValidationError from
        inside ADK's _run_async_impl. This wrapper retries on validation
        errors up to max_validation_retries times, logging each failure
        before retrying.
        """
        for attempt in range(self.max_validation_retries):
           try:
               async for event in super()._run_async_impl(ctx):
                   yield event
               return  # Success
           except ValidationError as exc:
               is_last_attempt = attempt == self.max_validation_retries - 1
               logger.warning(
                   "[%s] VALIDATION ERROR (attempt %d/%d): output failed %s validation "
                   "(%d error(s)): %s%s",
                   self.name,
                   attempt + 1,
                   self.max_validation_retries,
                   getattr(self.output_schema, "__name__", "?"),
                   exc.error_count(),
                   exc.errors(include_url=False),
                   " -- retrying..." if not is_last_attempt else " -- max retries exceeded",
               )
               metrics.record_hallucination(get_current_user())
              
               if is_last_attempt:
                   raise  # Re-raise on last attempt

Figure 4. The CustomLlmAgent retry loop (app/agent.py).图 4. CustomLlmAgent 重试循环 (app/agent.py)。

We log every single failed attempt to keep our hallucination metrics completely transparent and we throw a hard exception on the final failure so callers never receive a silently broken payload.
Tool-call failures use an entirely different mechanic. When an MCP tool errors, the raw failure is fed directly back into the model's context window so it can self-correct and reattempt the call. The repo delegates this loop to ADK's built-in plugin, registered at the application level as shown in Figure 5, below.我们记录每一次失败的尝试以保持幻觉指标完全透明，并在最终失败时抛出硬异常，这样调用者永远不会收到悄无声息地损坏的有效负载。工具调用失败使用完全不同的机制。当 MCP 工具出错时，原始故障会直接反馈到模型的上下文窗口中，以便它能够自我纠正并重试调用。该仓库将此循环委托给 ADK 的内置插件，在应用层注册，如图 5 所示。

from google.adk.plugins import ReflectAndRetryToolPlugin

app = App(
    root_agent=root_agent,
    name="app",
    plugins=[
        ReflectAndRetryToolPlugin(max_retries=3),
    ],
)

Figure 5. ReflectAndRetryToolPlugin registration (app/agent.py).图 5. ReflectAndRetryToolPlugin 注册 (app/agent.py)。

We offload tool-error recovery to the framework because tool failures are already structured for the model, but we strictly manage schema validation ourselves, using Pydantic, because hiding it would destroy our core hallucination telemetry.我们将工具错误恢复卸载给框架，因为工具故障对于模型来说已经是结构化的，但我们严格自行管理模式验证（使用 Pydantic），因为隐藏它会破坏我们的核心幻觉遥测。

We track two key metrics at this layer: recovery rate to ensure our retry loop actually works and amplification factor to guarantee it stays affordable.我们在这一层跟踪两个关键指标：恢复率（以确保重试循环确实有效）和放大因子（以确保它保持在可负担范围内）。

Prompt Versioning and Change ManagementPrompt 版本控制和变更管理

Treat prompt changes like code changes. A prompt is executable text. Modifying it changes the runtime behavior of the system in ways that are not visible in a diff. Two prompts that differ by a single sentence can have meaningfully different hallucination rates, latency profiles, and token costs. There is no compiler that warns you the new version is worse.像对待代码一样对待 Prompt 变更。Prompt 是可执行文本。修改它会以在 Diff 中不可见的方式改变系统的运行时行为。仅相差一个句子的两个 Prompt 可能具有完全不同的幻觉率、延迟曲线和 Token 成本。没有编译器会警告你新版本更差。

The minimum platform primitive: every prompt has an ID, a version, and is deployable independently of the application code that uses it. Applications reference prompts by ID, not by string literal. The platform stores prompt versions, tracks which version is live and exposes a rollback path that does not require redeploying the application.最低限度的平台原语：每个 Prompt 都有一个 ID、一个版本，并且可以独立于使用它的应用代码进行部署。应用通过 ID 而非字符串字面量引用 Prompt。平台存储 Prompt 版本，跟踪哪个版本处于活动状态，并提供无需重新部署应用即可回滚的路径。

Each prompt is registered under a name with an explicit version, its content, a creation timestamp, and tags. Registering a new version does not overwrite the old one; it adds to the history and moves the "live" pointer forward. Figure 6 shows the "docs_agent" prompt registered at two versions, the original v1.0.0 and the current v2.0.0 that supersedes it.每个 Prompt 都在一个名称下注册，包含显式版本、内容、创建时间戳和标签。注册新版本不会覆盖旧版本；它会添加到历史记录并将“活动”指针向前移动。图 6 显示了“docs_agent”Prompt 注册了两个版本：原始的 v1.0.0 和取代它的当前 v2.0.0。

# Docs Agent Prompt -- v1.0.0 (registered first so v2.0.0 below stays active;
   # kept available as a rollback target).
   register_prompt(
       name="docs_agent",
       content="""You are a documentation lookup specialist. Your role is to:
               - Answer questions about software libraries and frameworks
               - Look up APIs, parameters, and usage patterns from their docs
               - Ground every answer in the documentation, not from memory
               - Say so explicitly when the docs do not cover a question
               - Keep answers concise and cite the library/version used""",
       version="1.0.0",
       tags=["documentation", "lookup", "specialist"]
   )

   # Docs Agent Prompt -- v2.0.0 (active)
   register_prompt(
       name="docs_agent",
       content="""You are a documentation lookup specialist. Your job is to answer questions about software libraries and frameworks using their current, official documentation -- never from memory.

           You have two tools, used in sequence:
           1. resolve-library-id -- pass the library or framework name (e.g. "next.js", "fastapi") to get its Context7-compatible library ID. If the name is ambiguous and several libraries match, choose the one that best fits the user's context and state which you picked.
           2. get-library-docs -- pass the resolved library ID, plus a topic when the question is about a specific area (e.g. "routing", "middleware", "authentication"), to retrieve the documentation.

           Rules:
           - ALWAYS resolve and fetch docs before answering any question about a specific API, parameter, version behavior, or usage pattern. Do not rely on training data for library details, which may be outdated.
           - Ground every claim in the fetched documentation. When you give a code example or name a parameter, it must come from the docs you retrieved, not from memory.
           - If the fetched docs do not cover the question, say so explicitly rather than guessing. Do not fill gaps with plausible-sounding but unverified API details.
           - If you cannot resolve the library at all, tell the user the library wasn't found and ask them to confirm the exact name.
           - Keep answers concise and cite which library/version the docs came from.

           You do not author new documentation, run code, or search the open web. If a request falls outside documentation lookup, say it's out of scope for this agent.

           Output format -- return a single JSON object matching the DocsOutput schema:
           - answer: your grounded answer, drawn only from the fetched docs.
           - library_name: the library the answer is about (e.g. "FastAPI").
           - version: the library version the docs reflect, or null if unknown.
           - sources: one entry per docs lookup, each with the resolved library_id and the topic (or null).
           - grounded: true only if every claim is backed by the fetched docs; false if the docs did not cover the question.
           - out_of_scope: true if the request was not a documentation lookup.
           Do not include any text outside the JSON object.""",
       version="2.0.0",
       tags=["documentation", "lookup", "context7", "grounded"]
   )

Figure 6. The docs_agent prompt at versions 1.0.0 and 2.0.0 (prompt_registry/prompts.py).图 6. 版本 1.0.0 和 2.0.0 的 docs_agent Prompt (prompt_registry/prompts.py)。

Rollback is not deletion. It repoints which version is active and keeps the full history, so rolling forward is as fast as rolling back, as shown in Figure 7.回滚不是删除。它重新指向哪个版本处于活动状态并保留完整历史记录，因此向前回滚与向后回滚一样快，如图 7 所示。

def rollback(self, name: str, version: str) -> str:
    """Repoint a prompt's active version to an existing earlier version.

    History is never deleted -- rollback only changes which version
    get_prompt(name) returns. Returns the version now active. Raises
    KeyError if the prompt or target version is unknown.
    """
    if name not in self._prompts:
        raise KeyError(f"Prompt '{name}' not found in registry")
    if version not in self._prompts[name]:
        raise KeyError(f"Version '{version}' not found for prompt '{name}'")
    self._latest_versions[name] = version
    return version

Figure 7. The rollback() method (prompt_registry/registry.py).图 7. rollback() 方法 (prompt_registry/registry.py)。

The detail that makes rollback take effect without a redeploy is on the agent side. Each agent resolves its instruction from the registry at run time rather than capturing a fixed string when it's constructed, shown in Figure 8, below.使回滚无需重新部署即可生效的细节在于 Agent 端。每个 Agent 在运行时从注册表中解析其指令，而不是在构建时捕获固定字符串，如图 8 所示。

def _prompt_provider(name: str):
    # Resolve at run time (ADK InstructionProvider) rather than capturing a
    # fixed string at construction, so a prompt rollback via the registry
    # takes effect on the next request without rebuilding the agents.
    return lambda _ctx: get_prompt(name)

Figure 8. The _prompt_provider runtime instruction provider (app/agent.py).图 8. _prompt_provider 运行时指令提供程序 (app/agent.py)。

Because of that lookup, an operator hitting the rollback endpoint changes the behavior of the next request, without deploying, restarting, or rebuilding. Rollback is an admin-only gateway route, so the same RBAC layer that protects tools also controls who can change a live prompt.由于这种查找机制，操作员点击回滚端点即可改变下一次请求的行为，而无需部署、重启或重建。回滚是仅限管理员的网关路由，因此保护工具的 RBAC 层同样控制着谁可以更改实时 Prompt。

One honest limitation, where I would extend the repo next is to address the issue of the registry keeping prompts in memory. That is fine for demonstration and local development, where you want to read the code and run it without external dependencies, but it is not suitable for production, where prompts must survive restarts and stay consistent across instances. The production step is to put a database behind the same interface, with a separate version binding per environment (i.e., dev, staging, and prod). The calling code does not change; only the storage layer behind it does.一个诚实的局限性，也是我接下来会扩展仓库的地方，是解决注册表将 Prompt 保留在内存中的问题。这对于演示和本地开发来说没问题，因为你想要阅读代码并在没有外部依赖的情况下运行它，但这不适合生产环境，因为在生产环境中 Prompt 必须在重启后存活并在实例间保持一致。生产环境的步骤是在同一接口背后放置一个数据库，并为每个环境（如开发、测试和生产）绑定单独的版本。调用代码无需更改，只有背后的存储层会更改。

Intent Validation as a Routing Gate意图验证作为路由门控

This was the most novel primitive in our setup. Intent validation answers one question before the expensive foundation model call. Does this request match something the platform knows how to serve and if so, which specialist should handle it? In production we ran a small, trained classifier in front of the foundation model on every request, the kind you might build by fine-tuning a transformer through Hugging Face or training a lightweight model with scikit-learn. But the architecture does not depend on the classifier being trained. What matters is a separate, cheap, inspectable gate that runs before routing and produces a structured decision. The repo implements the gate deterministically on purpose, so you can see the mechanics and, critically, it refuses to guess when nothing matches. Figure 9 shows the gate in the repo. It scores each agent against the YAML keyword rules and, when nothing clears the bar, returns "unclassified" rather than picking the highest-scoring agent by default.这是我们设置中最具创新性的原语。意图验证在昂贵的基础模型调用之前回答了一个问题：此请求是否匹配平台知道如何服务的意图？如果是，哪个专用 Agent 应该处理它？在生产环境中，我们在每次请求的基础模型之前运行了一个小型、经过训练的分类器，你可以通过 Hugging Face 微调 Transformer 或使用 scikit-learn 训练轻量级模型来构建这种分类器。但该架构并不依赖于分类器是否经过训练。重要的是一个独立的、廉价的、可检查的门控机制，它在路由之前运行并产生结构化的决策。该仓库特意确定性地实现了该门控，以便你可以看到其机制，且关键在于，当没有任何匹配时，它拒绝猜测。图 9 显示了仓库中的门控。它根据 YAML 关键字规则对每个 Agent 进行评分，当没有任何匹配时，返回“未分类”，而不是默认选择得分最高的 Agent。

def classify_intent(user_input: str) -> Dict:
    """Classify user intent based on YAML keyword rules.

    Returns a structured decision the calling agent can act on:
      - matched:    {"status": "classified", "agent": <name>, "score": <float>}
      - no match:   {"status": "unclassified", "agent": None, ...}
                    The request is left unclassified rather than defaulting to a
                    fixed agent; the caller decides or asks the user.
    """
    user_input_lower = user_input.lower()
    agent_scores = {}
    for agent_name, agent_rules in INTENT_RULES.get("agents", {}).items():
        keywords = agent_rules.get("keywords", [])
        weight = agent_rules.get("weight", 1.0)
        score = sum(1 for kw in keywords if kw in user_input_lower) * weight
        agent_scores[agent_name] = score

    # No confident match: do NOT default to a specific intent. Hand control
    # back so the platform can ask the user or fall through to a grounded path.
    if not agent_scores or all(score == 0 for score in agent_scores.values()):
        return {
            "status": "unclassified",
            "agent": None,
            "reason": "No intent rule matched the request.",
            "available_agents": list(INTENT_RULES.get("agents", {}).keys()),
            "next_action": "ask_user_about_web_search",
        }

    best_agent = max(agent_scores, key=agent_scores.get)
    return {"status": "classified", "agent": best_agent, "score": agent_scores[best_agent]}

Figure 9. The classify_intent gate (intent_classifier/classifier.py).图 9. classify_intent 门控 (intent_classifier/classifier.py)。

A basic router would force the request to the highest-scoring agent even on a weak match. That is exactly how an agent ends up handling a question it wasn't built for, leading to hallucinations. By explicitly returning "unclassified" instead, the platform chooses the safest next step, either rejecting the request, asking the user to clarify, or falling back to a strictly constrained search.基本的路由器即使在匹配度很低的情况下也会强制将请求发送给得分最高的 Agent。这正是 Agent 最终处理其未构建的问题并导致幻觉的原因。通过明确返回“未分类”，平台选择了最安全的下一步，即拒绝请求、要求用户澄清或退回到严格限制的搜索。

The payoff comes twice. The gate catches requests the model would hallucinate on. In addition, by picking the right tools and prompt up front, it shrinks the space the model has to search, which itself lowers the hallucination rate as well as keeping the context minimal and concrete.回报是双重的。门控机制捕获了模型会产生幻觉的请求。此外，通过预先选择正确的工具和 Prompt，它缩小了模型必须搜索的空间，这不仅降低了幻觉率，还保持了上下文的最小化和具体化。

We keep the router as a separate, swappable component so you can easily choose between a free keyword gate or a more accurate, trained classifier in front of the foundation model, for example a fine-tuned transformer via Hugging Face or a lightweight model built with scikit-learn. 我们将路由器作为一个独立的、可替换的组件，这样你可以轻松地在免费的关键字门控或基础模型前更准确的训练分类器之间进行选择，例如通过 Hugging Face 微调的 Transformer 或使用 scikit-learn 构建的轻量级模型。

Authentication and Authorization: Securing the MCP Layer认证与授权：保护 MCP 层

Most write-ups treat agent security as a prompt-injection problem. In a multi-team deployment, the more important problem is simpler. When several teams' tools live behind MCP servers and several roles share the platform, which caller may invoke which tool? That is RBAC. It belongs at the platform layer, enforced at the resource server instead of enforcing the access control in each application.大多数文章将 Agent 安全视为 Prompt 注入问题。在多团队部署中，更重要的问题更简单。当多个团队的工具位于 MCP 服务器之后，且多个角色共享平台时，谁可以调用哪个工具？这就是 RBAC。它属于平台层，在资源服务器上强制执行，而不是在每个应用中强制执行访问控制。

The platform issues a stateless JWT at login, signed with PyJWT. The token carries the caller's role as a claim, so any service holding the shared secret can verify it and learn the role without calling back to the auth server. Figure 10 shows how the platform mints that token. It signs the caller's identity and role into a short-lived JWT that any downstream service can verify with the shared secret.平台在登录时签发无状态 JWT，并使用 PyJWT 签名。Token 将调用者的角色作为声明携带，因此任何持有共享密钥的服务都可以验证它并了解角色，而无需回调认证服务器。图 10 展示了平台如何铸造该 Token。它将调用者的身份和角色签名到一个短期的 JWT 中，任何下游服务都可以使用共享密钥进行验证。

def issue_token(username: str, role: str, ttl_seconds: int = TOKEN_TTL_SECONDS):
    """Mint a signed token for an authenticated user."""
    now = datetime.now(timezone.utc)
    payload = {
        "sub": username,
        "role": role,
        "iat": int(now.timestamp()),
        "exp": int((now + timedelta(seconds=ttl_seconds)).timestamp()),
    }
    token = jwt.encode(payload, JWT_SECRET, algorithm=JWT_ALGORITHM)
    return {"token": token, "token_type": "Bearer", "role": role, ...}

Figure 10. The issue_token function (gateway/authz.py).图 10. issue_token 函数 (gateway/authz.py)。

Our permissions live in a YAML config file rather than hardcoded rules, using a default-deny posture so security changes only require a configuration edit, not a full deployment. Below is an example of the policies config shown in Figure 11.我们的权限存在于 YAML 配置文件中而不是硬编码规则中，采用默认拒绝的姿态，因此安全变更只需要编辑配置，而无需完整部署。以下是图 11 中显示的策略配置示例。

developer:
  description: Development-only access (no production)
  permissions:
    tools:
      - read_file
      - list_directory
      - query_database
    resources:
      - "file://src/*"
      - "db://dev/*"
      - "db://staging/*"
    operations: [SELECT, GET, LIST]
  # Explicitly blocked: production databases, deployment, admin operations

Figure 11. The authorization policy (gateway/policies.yaml).图 11. 授权策略 (gateway/policies.yaml)。

The decision I would defend hardest is where this policy is enforced. It would be easy to check permissions once at the gateway and trust everything downstream. We don't. Each MCP server, built with FastMCP, is an independent resource server that verifies the token and checks the policy itself, on every tool call, before the tool runs, as shown in Figure 12.我最坚持的决策是该策略在何处执行。在网关处检查一次权限并信任下游的一切很容易。我们不这样做。每个使用 FastMCP 构建的 MCP 服务器都是一个独立的资源服务器，它会在每次工具调用时、在工具运行前自行验证 Token 并检查策略，如图 12 所示。

from fastmcp.server.middleware import Middleware, MiddlewareContext

class AuthzMiddleware(Middleware):
    """Deny tool calls whose caller role isn't permitted by the policy."""

    async def on_call_tool(self, context: MiddlewareContext, call_next):
        tool_name = context.message.name
        auth_header = get_http_headers(include={"authorization"}).get("authorization")
        try:
            claims = decode_token(token_from_header(auth_header))
        except AuthError as exc:
            raise ToolError(f"Unauthorized: {exc}")

        role = claims.get("role", "")
        if not policy_engine.can_access_tool(role, tool_name):
            logger.warning("MCP authz DENY role=%r tool=%r", role, tool_name)
            raise ToolError(
                f"Forbidden: role '{role}' is not authorized to use tool '{tool_name}'"
            )
        return await call_next(context)

Figure 12. The AuthzMiddleware enforcement point (mcp_server/authz_middleware.py).图 12. AuthzMiddleware 执行点 (mcp_server/authz_middleware.py)。

Observability for LLMsLLM 的可观测性

Standard APM was the most expensive thing we got wrong. Latency, error rate, and throughput tell you whether the model is reachable; they tell you nothing about whether it's behaving correctly or not.标准 APM 是我们做错的最昂贵的事情。延迟、错误率和吞吐量告诉你模型是否可达；它们无法告诉你模型行为是否正确。

Here is what a production LLM system actually needs:以下是生产级 LLM 系统真正需要的东西：

• The hallucination rate is the percentage of AI responses that either break our formatting rules (schema failures) or invent facts missing from our source documents (grounding failures), tracked per app and prompt version.幻觉率是 AI 响应中违反格式规则（模式失败）或编造源文档中缺失事实（基础依据失败）的百分比，按应用和 Prompt 版本进行跟踪。
• Prompt drift is the rate of behavioral change across prompt versions. A new version that raises the hallucination rate by two percent must be visible before it reaches every user.Prompt 漂移是跨 Prompt 版本的行为变化率。必须在幻觉率提高 2% 的新版本触达每个用户之前就发现它。
• To drive effective cost optimization, we break down token spend by team, application, and specific use case to automatically highlight our highest consumers.为了推动有效的成本优化，我们按团队、应用和特定用例细分 Token 支出，以自动突出显示我们的最高消费者。
• Tool invocation patterns, such as which tools agents call, in what order, and with what failure and denial rates. This invocation is where multi-agent pipeline issues show up.工具调用模式，例如 Agent 调用了哪些工具、顺序如何、失败率和拒绝率如何。这种调用是多 Agent 流水线问题显现的地方。
• Behavioral correctness sampling, which is offline evaluation of sampled production traffic against a golden set, covered in the "Evaluating Agent Behavior" section.行为正确性采样，即对抽样生产流量进行离线评估，对照黄金标准，这在“评估 Agent 行为”一节中介绍。
• By capturing request-level success rates and latency per user, we tie semantic AI quality directly to traditional operational metrics.通过捕获每个用户的请求级成功率和延迟，我们将语义 AI 质量直接与传统运营指标联系起来。

The detail that makes "per team" work is that the team dimension must exist in the data from the first call. It cannot be rebuilt later from logs. The collector derives the team from the authenticated role and carries it on a context variable set at request ingress. Every record is attributable and no function needs a team ID threaded through it. Figure 13 shows how that attribution is structured. The role maps to a team, while each user's request, token, cost, and hallucination counts accumulate on a single per-user record.实现“按团队”工作的细节在于，团队维度必须从第一次调用开始就存在于数据中。它不能事后从日志中重建。收集器从认证角色中派生团队，并将其携带在请求接入点设置的上下文变量上。每条记录都是可归因的，且不需要在函数间传递团队 ID。图 13 展示了这种归因是如何结构的。角色映射到团队，而每个用户的请求、Token、成本和幻觉计数累积在单个用户记录上。

ROLE_TEAM: dict[str, str] = {
   "viewer": "shared",
   "analyst": "analytics",
   "developer": "developer",
   "deployer": "devops",
   "admin": "platform",
}

@dataclass
class UserMetrics:
    user_id: str
    team: str = "unknown"
    requests: int = 0
    successes: int = 0
    errors: int = 0
    hallucinations: int = 0
    input_tokens: int = 0
    output_tokens: int = 0
    total_cost_usd: float = 0.0
    total_latency_s: float = 0.0

Figure 13. The ROLE_TEAM map and UserMetrics record (observability/metrics/collector.py).图 13. ROLE_TEAM 映射和 UserMetrics 记录 (observability/metrics/collector.py)。

Token cost is recorded at every model call site, not just at the final response. The recording lives in the agent's after_model_callback, so a request that fans out to four specialists attributes all four calls to the same user and team. The hallucination counter on the same record is incremented by the schema-validation check. Latency and success/error counts come from a context manager wrapping the gateway's execution endpoint. One snapshot endpoint exposes everything per user or team. Figure 14 shows a real snapshot from the running system for a single devops user, with tokens, cost, latency, and success rate on one attributable record.Token 成本记录在每个模型调用点，而不仅仅是最终响应。记录存在于 Agent 的 after_model_callback 中，因此一个扇出到四个专用 Agent 的请求会将所有四次调用归因于同一个用户和团队。同一记录上的幻觉计数器由模式验证检查增加。延迟和成功/错误计数来自封装网关执行端点的上下文管理器。一个快照端点暴露每个用户或团队的所有内容。图 14 显示了运行系统中单个 devops 用户的真实快照，包含 Token、成本、延迟和成功率，记录在一条可归因的记录上。

{
       "user_id": "dale",
       "team": "devops",
       "requests": 1,
       "successes": 1,
       "errors": 0,
       "hallucinations": 0,
       "input_tokens": 191,
       "output_tokens": 79,
       "total_cost_usd": 0.0,
       "total_latency_s": 11.145345083001303
}

Figure 14. Example metrics snapshot for a user (observability/metrics/collector.py).图 14. 用户指标快照示例 (observability/metrics/collector.py)。

We track local token estimates separately from the actual API response data so our internal counts never mix with the true billing data.我们将本地 Token 估算与实际 API 响应数据分开跟踪，因此我们的内部计数永远不会与真实的计费数据混淆。

We use OpenTelemetry to ship our metrics out, but we wrap them in custom methods so we can swap backends like Prometheus, instantly without touching our application code.我们使用 OpenTelemetry 发送指标，但将其封装在自定义方法中，这样我们就可以在不触动应用代码的情况下即时切换后端，如 Prometheus。

Evaluating Agent Behavior评估 Agent 行为

Unit and contract tests pin down the deterministic path, but they cannot answer the question that matters most for an LLM system: Is the model still behaving? Answering that question is what the evaluation layer is for and it is the part most worth building carefully, because it is the closest thing an agent platform has to a regression suite.单元测试和契约测试可以固定确定性路径，但它们无法回答 LLM 系统最重要的问题：模型行为是否仍然正常？回答这个问题正是评估层的目的，也是最值得仔细构建的部分，因为它是 Agent 平台最接近回归测试套件的东西。

The process has four parts: a golden set of representative cases, a runner that drives the real agent through them, a set of scored metrics, and a threshold that turns those scores into a pass/fail gate.该过程分为四个部分：一组具有代表性的案例（黄金标准）、一个驱动真实 Agent 完成这些案例的运行器、一组评分指标，以及一个将这些分数转化为通过/失败门控的阈值。

The golden set is data, not code. In the repo it is a JSON file, with each case fixing one expected outcome, consisting of a user request, the tool trajectory the agent should follow, the response it should return, and the role the request runs under. The seed cases cover the routing contract. A documentation request should classify intent and then hand off to the docs specialist. A code-review request to the codebase specialist. A research request to the research specialist. In production this set is the artifact that grows fastest. Every incident that reaches a user becomes a new case, so the same failure cannot ship twice.黄金标准是数据，而非代码。在仓库中它是一个 JSON 文件，每个案例修复一个预期的结果，包括用户请求、Agent 应该遵循的工具轨迹、它应该返回的响应，以及请求运行的角色。种子案例涵盖了路由契约。文档请求应该分类意图，然后交给文档专用 Agent。代码审查请求交给代码库专用 Agent。研究请求交给研究专用 Agent。在生产环境中，这组数据是增长最快的工件。每一个触达用户的事故都成为一个新案例，这样同样的故障就不会发生两次。

The runner is ADK's AgentEvaluator. It executes the real agent against every case, several times per case and averages the scores, repetition matters because the model is stochastic and a single run tells you little. The metrics ADK exposes cover the two things worth checking separately: whether the agent took the right path (the tool trajectory, did it classify and then route to the correct specialist?) and whether it produced the right answer (response match against a reference and a model-graded quality score). A safety metric rounds out the set.运行器是 ADK 的 AgentEvaluator。它针对每个案例多次运行实际智能体，并计算平均得分。重复运行非常重要，因为模型具有随机性，单次运行提供的信息有限。ADK 暴露的指标涵盖了需要单独检查的两个方面：智能体是否选择了正确的路径（工具轨迹，即它是否进行了分类并路由到了正确的专家？），以及它是否产生了正确的答案（响应与参考答案的匹配度以及模型评分的质量得分）。一套安全指标完善了该评估体系。

The threshold is what makes it a gate rather than a dashboard. The repo requires an average tool-trajectory score of at least 0.8 across runs. Below that score, the evaluation fails. That number is deliberately visible in a three-line config file, not buried in code, so tightening the bar is a reviewable change.阈值设定使其成为一道“关卡”而非简单的仪表盘。该代码库要求在多次运行中，平均工具轨迹得分至少达到 0.8。低于该分数，评估即告失败。这个数字被特意放在一个三行的配置文件中，而不是隐藏在代码里，因此收紧标准是一个可以进行评审的变更。

Two design choices make this effort worthwhile. First, the split is enforced in continuous integration. The fast, hermetic tests run on every push and gate merge. The model-backed evals need a live model and the tool servers, so they run as a separate, explicitly triggered job that gates prompt promotions. You do not pay for a live model on every commit, but no prompt reaches production without clearing the bar. Second, the golden set itself has a test. A lightweight check runs on every push, with no model that validates every case still encodes a valid contract and every metric named is one the runner understands so the evaluation suite adheres to the rules on each run.两项设计选择使得这项工作物有所值。首先，这种分离在持续集成中得到强制执行。快速、封闭的测试在每次推送时运行并作为合并的门槛。而基于模型的评估需要实时模型和工具服务器，因此它们作为一项独立的、显式触发的任务运行，用于控制提示词（prompt）的发布。你无需为每次提交都支付实时模型费用，但没有提示词能在不通过该门槛的情况下进入生产环境。其次，黄金集（golden set）本身也包含测试。每次推送时都会运行一项轻量级检查（无需模型），验证每个案例是否仍符合有效的契约，并确保所命名的每个指标都是运行器所能理解的，从而保证评估套件在每次运行时都遵循规则。

Paired with the prompt registry, this design is what closes the loop on prompt changes. A candidate prompt version is evaluated against the golden set before it is promoted. If its trajectory or grounding scores fall below the live version, it never ships. That is how "prompt drift" moves from a post-incident finding to a predeployment gate and it is the same measurement discipline behind the hallucination numbers in this article, applied to sampled production traffic against the same references. Figure 15 shows one case from the golden set. The user request, the exact tool trajectory the agent should follow, and the role the request runs under.结合提示词注册表，这种设计闭环了提示词变更的管理。候选提示词版本在发布前会针对黄金集进行评估。如果其轨迹得分或接地（grounding）得分低于当前线上版本，它将永远不会上线。这就是“提示词漂移”如何从事故后的发现转变为部署前的门槛。这也是本文中幻觉数字背后的测量准则，应用于针对相同参考答案的采样生产流量。图 15 展示了黄金集中的一个案例。包括用户请求、智能体应遵循的精确工具轨迹，以及请求运行的角色。

# tests/evals/data/root_agent.evalset.json -- one golden-set case (abridged).
# Each case fixes the expected outcome: the user request, the tool
# trajectory the agent must follow, and the role the request runs under.

{
  "eval_id": "docs_lookup_routes_to_docs_agent",
  "conversation": [{
    "user_content":  {"parts": [{"text": "Write documentation explaining the FastAPI routing API."}]},
    "final_response": {"parts": [{"text": "Routed to the documentation specialist, which grounds its answer in the FastAPI docs."}]},
    "intermediate_data": {
      "tool_uses": [
        {"name": "classify_intent",   "args": {"user_input": "Write documentation explaining the FastAPI routing API."}},
        {"name": "transfer_to_agent", "args": {"agent_name": "docs_agent"}}
      ]
    }
  }],
  "session_input": {"app_name": "app", "user_id": "eval_user", "state": {"user_role": "viewer"}}
}

Figure 15. A golden-set evaluation case (tests/evals/data/root_agent.evalset.json).图 15。黄金集评估案例 (tests/evals/data/root_agent.evalset.json)。

As shown in Figure 16, the pass/fail bar lives in a three-line config file. The average tool-trajectory score across runs must clear 0.8 or the evaluation fails.如图 16 所示，通过/失败的标准位于一个三行的配置文件中。多次运行的平均工具轨迹得分必须超过 0.8，否则评估失败。

// tests/evals/data/test_config.json -- the pass/fail bar.
{ "criteria": { "tool_trajectory_avg_score": 0.8 } }

Figure 16. The evaluation threshold (tests/evals/data/test_config.json).图 16。评估阈值 (tests/evals/data/test_config.json)。

Architectural Patterns for Multi-Team Deployments多团队部署的架构模式

When more than one team builds on the same platform, the question shifts from "how do we serve this model" to "how do we serve N teams sharing this infrastructure". Three patterns mattered most:当多个团队在同一平台上进行构建时，问题就从“我们如何服务这个模型”转变为“我们如何服务共享此基础设施的 N 个团队”。以下三种模式最为重要：

• The gateway pattern
  One ingress that handles authentication, per-tenant rate limiting, routing to the right model and prompt, and observability emission. It is the API-gateway pattern from microservices, applied to LLM traffic.网关模式 一个处理身份验证、租户级速率限制、路由到正确模型和提示词以及可观测性输出的入口。这是微服务中的 API 网关模式在 LLM 流量中的应用。
• The agent coordinator pattern
  To prevent cascading system failures, a multi-agent pipeline must have a single orchestrator that owns the entire topology. Without a central owner to manage circuit breakers, enforce timeout budgets, and validate data between transitions, individual agents will independently try to recover from errors. When an upstream issue occurs, agents blindly retry the components above them, turning a single isolated failure into a system-wide crash.智能体协调器模式 为了防止系统级联故障，多智能体流水线必须拥有一个掌控整个拓扑结构的单一协调器。如果没有一个中心化的所有者来管理断路器、执行超时预算并在转换过程中验证数据，各个智能体将独立尝试从错误中恢复。当上游出现问题时，智能体会盲目重试其上方的组件，从而将单一的孤立故障变成全系统的崩溃。
• The cost attribution model
  Every request enters the platform tagged with team, application, and use case. The tags propagate through every downstream model call. This approach is what makes per-team cost dashboards possible and it has to be designed in from the start.成本归因模型 每个请求进入平台时都带有团队、应用程序和用例标签。这些标签会传播到每一个下游模型调用中。这种方法使得按团队划分的成本仪表盘成为可能，且必须从一开始就进行设计。

Summary总结

To summarize the article, the six concrete decisions, in priority order are:总结全文，按优先级排序的六项具体决策如下：

1. Build the observability layer before the first production deployment, not after the first incident. Skipping it for v1 is the most expensive shortcut available. Without behavioral metrics, you cannot tell the system is degrading until users complain. By then, you have lost trust.在首次生产部署前构建可观测性层，而不是在第一次事故后。在 v1 版本中跳过它是代价最昂贵的捷径。没有行为指标，你无法在用户投诉前发现系统正在退化。到那时，你已经失去了信任。
2. Adopt platform thinking before the second application, not the tenth. Every shared concern built into application code must be refactored out later, under deadline pressure.在第二个应用程序之前采用平台化思维，而不是第十个。在应用程序代码中构建的每一个共享关注点，以后都必须在截止日期的压力下进行重构。
3. Employ version prompts from day one. A registry with ID-based references is small to build. Building it after your first silent regression costs far more, and that incident is a near certainty.从第一天起就对提示词进行版本控制。构建一个基于 ID 引用的注册表成本很低。在第一次静默回归（silent regression）后才去构建它，成本要高得多，而且这种事故几乎是必然发生的。
4. Enforce schemas at the platform boundary on the first call, not after the first downstream parse failure. Anything less means defensive parsing in every consumer.在第一次调用时就在平台边界强制执行模式（schema），而不是在第一次下游解析失败后。否则，意味着每个消费者都必须进行防御性解析。
5. Tag every request for cost attribution from the start. Without per-team tags, the first time leadership asks what this approach costs, the answer will be a multi-week investigation. With them, the answer is a dashboard.从一开始就为每个请求标记成本归因。如果没有按团队的标签，当领导第一次询问这种方法成本如何时，答案将是一项耗时数周的调查。有了标签，答案就是一个仪表盘。
6. Stand up a golden-set evaluation per specialist before the first prompt change, not after it. The registry lets you roll back fast. The golden set tells you a version needed rolling back before your users do.在第一次提示词变更前，为每个专家建立黄金集评估，而不是之后。注册表让你能快速回滚。黄金集则能在用户发现之前告诉你某个版本需要回滚。

The unifying thread among these decisions is that LLM systems let you defer infrastructure decisions in ways traditional systems do not. They do not crash. They do not return 500 errors. They quietly produce slightly worse output. That gap between a wrong decision and its visible consequence is why platform thinking pays off up front, and why retrofitting it under fire costs so much.这些决策的共同主线是：LLM 系统允许你以传统系统无法实现的方式推迟基础设施决策。它们不会崩溃，不会返回 500 错误，只是悄悄地产生稍差的输出。错误决策与其可见后果之间的这种差距，正是平台化思维在前期就能产生回报的原因，也是在压力下进行补救成本如此之高的原因。

The shift from "we built an LLM application" to "we built on an LLM platform" is the shift the rest of the field will make over the next two years. The only question is whether a team makes it on its own timeline or under pressure from an incident.从“我们构建了一个 LLM 应用程序”到“我们基于 LLM 平台进行构建”的转变，是该领域其余部分在未来两年内将要完成的转变。唯一的问题是，团队是按照自己的时间表完成这一转变，还是在事故压力下被迫完成。

The complete, runnable implementation of every primitive in this article is here.本文中每个原语的完整、可运行实现都在这里。