Agent Evaluation Metric for multi-turn conversations用于多轮对话的智能体评估指标

Multi-turn agents fail in ways that single-turn evaluation misses: one early mistake quietly corrupts every later turn. This post introduces the Agent Evaluation Metric (AEM), a decomposable, turn-level way to measure agent quality. We apply it to its first dimension, correctness. We show how AEM pinpoints the one turn that caused a failure and separates it from the turns that merely inherited the problem.多轮智能体在单轮评估无法察觉的地方可能会失败：一个早期的错误会悄无声息地破坏后续的每一轮对话。本文介绍了智能体评估指标（Agent Evaluation Metric，简称 AEM），这是一种可分解的、基于轮次的智能体质量衡量方法。我们将其应用于第一个维度——正确性。我们展示了 AEM 如何精确定位导致失败的那一轮对话，并将其与仅仅继承了该问题的后续轮次区分开来。

The correctness challenge in multi-turn agentic conversations多轮智能体对话中的正确性挑战

Evaluating agent correctness in a multi-turn conversation is hard because the property itself is fragile, and holistic scores hide where it breaks. This section shows why cascading errors defeat outcome-level evaluation and motivates a decomposable, turn-level metric.评估多轮对话中智能体的正确性非常困难，因为这一属性本身就很脆弱，而整体评分往往掩盖了问题出现的位置。本节将说明级联错误为何会导致基于结果的评估失效，并阐述引入可分解的、基于轮次的指标的必要性。

Why correctness matters为何正确性至关重要

A single wrong tool call cascades into downstream failures across turns. Consider a five-turn conversation with an enterprise assistant. The user asks to create a sales report, then refine it. In turn 2, the agent selects the right action but passes “profit” instead of “revenue.” That single error then silently propagates through every subsequent turn.一次错误的工具调用会引发跨轮次的级联失败。以企业助手进行的五轮对话为例。用户要求生成一份销售报告，然后对其进行细化。在第 2 轮中，智能体选择了正确的操作，但传递的是“利润”而非“收入”。这一单一错误随后悄无声息地传播到了后续的每一轮对话中。

The following diagram traces that failure. It shows one early error in turn 2 cascading through turns 3–5, and how turn-level evaluation isolates the single root cause from its downstream effects.下图追踪了这一失败过程。它展示了第 2 轮中的一个早期错误如何级联影响到第 3 至第 5 轮，以及基于轮次的评估如何将单一根本原因与其下游影响隔离开来。

Figure 1: One early error cascades through later turns, and turn-level evaluation isolates the root cause图 1：一个早期错误会级联影响到后续轮次，而基于轮次的评估可隔离出根本原因

A task-level evaluation checks only the final outcome. It marks the entire interaction as failed without revealing that only one turn needs fixing. This is the core problem: in multi-turn agentic conversations, errors cascade, and outcome-level evaluation can’t tell root causes from their downstream effects.任务级评估仅检查最终结果。它会将整个交互标记为失败，却无法揭示其实只有一轮对话需要修正。这就是核心问题所在：在多轮智能体对话中，错误会级联发生，而基于结果的评估无法将根本原因与其下游影响区分开来。

Limitations of existing metrics现有指标的局限性

Most agent evaluation tools score quality holistically, at the task or response level. The current generation provides goal-completion scoring and large language model (LLM)-as-judge quality assessment. Some also add trace-level root cause analysis. These are valuable, but they share a common framing: agent quality is treated as a single signal, not broken down into parts that can be tracked independently.大多数智能体评估工具在任务或响应层面进行整体质量评分。当前一代评估工具提供目标完成度评分以及基于大语言模型（LLM）作为评判者的质量评估。有些工具还增加了追踪层面的根本原因分析。这些方法很有价值，但它们有着共同的局限性：将智能体质量视为一个单一信号，而不是将其拆解为可以独立追踪的组成部分。

What they don’t provide is a way to decompose quality into named, separately measurable sub-metrics tracked turn by turn. Knowing an agent “scored 70 percent on goal completion” doesn’t tell you whether the failures were factual errors, missing information, or wrong tool choices. A single score also can’t extend cleanly to new dimensions as requirements grow. Three gaps follow:它们没有提供一种将质量分解为可命名、可单独测量的子指标并逐轮追踪的方法。知道智能体“目标完成度得分为 70%”并不能告诉你失败的原因是事实错误、信息缺失还是工具选择错误。单一分数也无法在需求增长时清晰地扩展到新的维度。具体存在以下三个缺口：

• Task-level metrics (goal success rate) tell you whether the agent completed the task but not which quality dimension broke down.任务级指标（目标成功率）只能告诉你智能体是否完成了任务，但无法说明是哪个质量维度出现了问题。
• Single-turn metrics (helpfulness, faithfulness) evaluate responses in isolation without considering how errors propagate across turns.单轮指标（有用性、忠实度）仅在孤立状态下评估响应，而未考虑错误如何在轮次间传播。
• Holistic scores cannot distinguish a factual error from a missing required field, and they offer no clear path to add new dimensions (safety, instruction retention) without re-architecting the evaluation.整体评分无法区分事实错误与必需字段缺失，且在不重构评估架构的情况下，无法提供添加新维度（如安全性、指令保留）的清晰路径。

A decomposable, turn-level evaluation pattern closes these gaps. It breaks quality into named sub-metrics, evaluates each turn within the full trajectory, and composes them into a single indicator. The same approach extends to new dimensions without changing the mechanism.一种可分解的、基于轮次的评估模式填补了这些空白。它将质量分解为具名的子指标，在整个轨迹中评估每一轮对话，并将它们组合成一个单一指标。同样的方法无需更改机制即可扩展到新的维度。

Correctness as a composite metric作为复合指标的正确性

We define agent quality as a single composite indicator built from named, separately measurable sub-metrics, computed by AEM. In this first post, AEM measures correctness through two sub-metrics:我们将智能体质量定义为由 AEM 计算出的、由具名的且可单独测量的子指标构建而成的单一复合指标。在第一篇文章中，AEM 通过两个子指标来衡量正确性：

• Truthfulness: Are the values produced by the agent factually consistent with what was expected? This applies to both parameter values in tool calls and statements in natural language responses.真实性（Truthfulness）：智能体产生的值在事实层面是否与预期一致？这既适用于工具调用中的参数值，也适用于自然语言响应中的陈述。
• Completeness: Are all required elements present? No missing parameters, and no partial responses that omit requested information.完整性（Completeness）：所有必需的要素是否都已呈现？没有缺失参数，也没有遗漏所请求信息的局部响应。

The following diagram shows this decomposition. It illustrates how a top-level score breaks into named sub-metrics that are measured independently and then recombined, and how the same pattern extends to future dimensions.下图展示了这种分解。它说明了一个顶层分数如何分解为可独立测量并重新组合的具名子指标，以及同样的模式如何扩展到未来的维度。

Figure 2: Correctness decomposes into named sub-metrics, extensible to new dimensions图 2：正确性分解为具名子指标，并可扩展至新维度

The headline contribution is the decomposition itself. AEM isn’t a single opaque score but a composite of sub-metrics, each measurable per turn and composable across the trajectory. Tool and action selection form the structural foundation beneath them. The same decompose-evaluate-compose pattern extends to new dimensions such as safety, instruction retention, and reasoning depth. Correctness is the first dimension we instantiate.其核心贡献在于分解本身。AEM 不是一个单一的不透明分数，而是子指标的组合，每个子指标都可以在每一轮中测量，并在整个轨迹中进行组合。工具和操作选择构成了其下的结构基础。同样的“分解-评估-组合”模式可以扩展到安全性、指令保留和推理深度等新维度。正确性是我们实例化的第一个维度。

The Agent Evaluation Metric framework智能体评估指标框架

AEM turns the decomposition idea into a concrete, per-turn metric. This section defines the turn-level hierarchy, the two sub-metrics and how they compose, and the failure taxonomy that makes a score actionable.AEM 将分解思想转化为具体的、逐轮的指标。本节定义了轮次层级、两个子指标及其组合方式，以及使评分具有可操作性的失败分类法。

The turn-level hierarchy轮次层级

Correctness is computed per turn. A turn is either a response turn, where the agent replies to the user, or an action turn, where the agent invokes a tool. We lead with the response turn, since that is what the user ultimately sees, and the same metric applies to action turns too. Both fall under one hierarchy.正确性是按轮次计算的。轮次分为响应轮次（智能体回复用户）或操作轮次（智能体调用工具）。我们首先关注响应轮次，因为这是用户最终看到的内容，同样的指标也适用于操作轮次。两者都属于同一个层级。

The following diagram shows that shared hierarchy. It shows the same two sub-metrics evaluating a tool call (parameter keys and values) on one side and a natural-language response (coverage and grounding) on the other.下图展示了该共享层级。它显示了相同的两个子指标如何在一侧评估工具调用（参数键和值），并在另一侧评估自然语言响应（覆盖范围和基础性）。

Figure 3: The same two sub-metrics evaluate a tool call and a natural-language response图 3：相同的两个子指标用于评估工具调用和自然语言响应

For a response turn, the two sub-metrics apply to free text. Completeness asks whether the reply covers the full query, and truthfulness asks whether it’s factually consistent. The same composite applies to an action turn. There, completeness checks the parameter keys, confirming that all required parameters are present. Truthfulness checks the parameter values, confirming they are semantically correct. Both sit on top of a structural check that the right tool and action were selected.对于响应轮次，这两个子指标适用于自由文本。完整性检查回复是否涵盖了完整查询，真实性检查其是否符合事实。同样的组合也适用于操作轮次。在操作轮次中，完整性检查参数键，确认所有必需参数是否存在。真实性检查参数值，确认其在语义上是否正确。两者都建立在确认选择了正确工具和操作的结构检查之上。

For this post, a turn’s correctness is treated as binary: pass or fail, with a specific failure reason naming the sub-metric and the field. The same decomposition also supports finer-grained grading, scoring individual claims or fields on a continuous scale.在本文中，轮次的正确性被视为二元判定：通过或失败，并带有指明子指标和字段的具体失败原因。同样的分解也支持更细粒度的分级，以连续尺度对单个声明或字段进行评分。

Composed across a dialog, the AEM score is the proportion of turns that pass. Because it’s decomposed by sub-metric, a decline shows which dimension, truthfulness or completeness, drove the change, not just that correctness dropped.在整个对话中组合时，AEM 分数是成功轮次的比例。由于它是按子指标分解的，分数的下降可以显示是哪个维度（真实性或完整性）导致了变化，而不仅仅是正确性下降。

Decomposing and formalizing AEM分解与形式化 AEM

Both sub-metrics rely on semantic comparison rather than exact matching. For responses and parameter values, exact string matching is too brittle. “New York City” and “NYC” are semantically equivalent, and “Q3 2024 revenue” and “third quarter revenue figures for 2024” convey the same information.两个子指标都依赖于语义比较而非精确匹配。对于响应和参数值，精确字符串匹配过于僵化。“New York City”和“NYC”在语义上是等价的，“Q3 2024 revenue”和“third quarter revenue figures for 2024”传达的信息相同。

Semantic similarity scoring determines whether two values are semantically equivalent. Conceptually, the check looks like this:语义相似度评分用于确定两个值在语义上是否等价。在概念上，检查过程如下：

def evaluate_truthfulness(gold_value, predicted_value, scorer, threshold=0.5):
    """Score semantic equivalence of a predicted value against gold."""
    if gold_value == predicted_value:
        return True, 1.0  # Exact match (fast path)

    score = scorer.score(gold_value, predicted_value)
    return score >= threshold, score

The scorer can be an embedding-based similarity check (fast, cheap) or an LLM-as-judge call (more nuanced). The embedding check is a transparent encoder-plus-similarity-function, and the judge is a more opaque decoder-based model whose scoring isn’t directly inspectable. The threshold controls strictness: a higher threshold catches real errors but risks flagging semantic equivalents, while a lower one is more permissive. The 0.5 here is a neutral default to start from, not a tuned value. The right value depends on your domain’s tolerance for false positives versus false negatives (see Lessons learned).评分器可以是基于嵌入的相似度检查（快速、廉价），也可以是 LLM 作为评判者的调用（更细致）。嵌入检查是一个透明的编码器加相似度函数，而评判者是一个更不透明的基于解码器的模型，其评分无法直接检查。阈值控制严格程度：较高的阈值可以捕获真实错误，但有误报语义等价项的风险；较低的阈值则更宽松。此处的 0.5 是一个中性默认值，而非经过调优的值。正确的值取决于您所在领域对误报与漏报的容忍度（参见“经验教训”）。

The similarity score is continuous. The threshold is what collapses it to a binary turn verdict for this post, and a finer-grained setup can retain the per-claim scores instead.相似度分数是连续的。阈值是本文将其坍缩为二元轮次判定的关键，而更细粒度的设置可以保留每项声明的得分。

Completeness is checked semantically for response turns (does the response address the full question?) and structurally for action turns (are all required parameter keys present?):完整性在响应轮次中进行语义检查（响应是否解决了完整问题？），在操作轮次中进行结构检查（所有必需的参数键是否存在？）：

def evaluate_completeness(gold_args, predicted_args):
    """Check all required parameters are present, with no unexpected extras."""
    missing = set(gold_args.keys()) - set(predicted_args.keys())
    extra = set(predicted_args.keys()) - set(gold_args.keys())
    return len(missing) == 0 and len(extra) == 0, missing, extra

The returned missing and extra sets feed directly into the failure taxonomy. A non-empty missing set produces a missing_parameters failure, and a non-empty extra set produces an extra_parameters failure, pinpointing exactly which parameters were wrong.返回的缺失和额外集合直接输入到失败分类法中。非空的缺失集合产生 missing_parameters 失败，非空的额外集合产生 extra_parameters 失败，从而精确指出哪些参数有误。

Composing the score. Decomposition produces per-sub-metric, per-turn verdicts. Composition turns them into one number. The composition rule is itself a choice, in keeping with the framework’s composable principle. The default in this post is an unweighted mean of passing turns.组合分数。分解产生每个子指标、每轮次的判定。组合将它们转化为一个数字。组合规则本身是一种选择，符合框架的可组合原则。本文的默认规则是成功轮次的未加权平均值。

Other rules are equally valid. Weighted means give more weight to turns that carry higher cost when wrong. Gating rules let a single critical-turn failure cap the score. Per-sub-metric thresholds set a separate bar for each dimension. The framework treats this composition function as pluggable.其他规则同样有效。加权平均值对错误代价更高的轮次给予更高权重。门控规则允许单一关键轮次的失败限制总分。每个子指标的阈值则为每个维度设定了单独的标准。该框架将此组合函数视为可插拔的。

Failure taxonomy and action chains失败分类法与操作链

When a turn fails, a specific failure reason captures exactly what went wrong. The taxonomy spans both turn types: response-turn failures sit at the same level as action-turn failures, and the structural checks (tool and action selection) apply only where a turn invokes a tool.当一轮对话失败时，具体的失败原因会捕捉到到底哪里出了问题。该分类法涵盖了两种轮次类型：响应轮次失败与操作轮次失败处于同一层级，而结构检查（工具和操作选择）仅在轮次调用工具时适用。

The two sub-metrics, truthfulness and completeness, are the constant across both turn types. Only the structural checks are tool-specific. The prior_action_failed label is what makes the metric actionable in a multi-turn setting. It separates root causes from cascading effects, and it can attach to a response turn as readily as to an action turn. The evaluator assigns the label by dependency. A turn is a root cause when its failure originates in that turn. It gets prior_action_failed when it fails only because it consumed an already-failed turn’s output. In the opening example, only turn 2 is a root cause, and turns 3–5 inherit the label. The metric also tracks action chain length (single call, two-step, and complex three-step-plus sequences), since longer chains concentrate most degradation.两个子指标（真实性和完整性）在两种轮次类型中保持不变。只有结构检查是工具特定的。prior_action_failed 标签使得该指标在多轮设置中具有可操作性。它将根本原因与级联效应分离开来，并且可以像附加到操作轮次一样轻松地附加到响应轮次上。评估器根据依赖关系分配该标签。当失败起源于某一轮次时，该轮次即为根本原因。当某一轮次仅因使用了已失败轮次的输出而失败时，它会被标记为 prior_action_failed。在开篇示例中，只有第 2 轮是根本原因，第 3 至第 5 轮继承了该标签。该指标还追踪操作链长度（单次调用、两步及复杂的三步以上序列），因为更长的链条往往会集中大部分性能退化。

Evaluation pipeline for agentic workflows智能体工作流的评估管道

The metric described earlier runs inside a repeatable pipeline. Annotated dialogs go in, and a single decomposed AEM score comes out, attributed per turn and consumed across the agent lifecycle.前面描述的指标运行在一个可重复的管道中。输入标注好的对话，输出一个单一的分解 AEM 分数，该分数归因于每一轮次，并在智能体生命周期中被使用。

The following diagram shows that end-to-end flow, from annotated dialogs through per-turn scoring and attribution to a single score consumed in development and production.下图展示了从标注对话到逐轮评分和归因，再到开发和生产中使用的单一分数的端到端流程。

Figure 4: From annotated dialogs to a decomposed, per-turn correctness score图 4：从标注对话到分解的、逐轮正确性分数

For the five-turn sales-report example, the pipeline returns a compact, decomposed result (illustrative):对于五轮销售报告示例，管道返回一个紧凑的、分解的结果（示意）：

{"success_rate": 0.2, "test_pass": false,
 "first_failure_turn": 2, "root_cause": "inconsistent_parameter_values",
 "root_cause_count": 1, "cascading_count": 3}

Golden dataset design黄金数据集设计

Evaluation begins with ground truth: conversations where both the correct responses and the correct tool calls are annotated. This gold reference is typically human-annotated (or human-reviewed when bootstrapped from a stronger model), since it defines what correct means for each turn. Each dialog is a sequence of turns, and each turn pairs the gold (expected) output with the predicted (actual) output. A turn carries a turn role identifying who produced it. A response turn and an action turn look like this:评估始于事实真相（Ground Truth）：即标注了正确响应和正确工具调用的对话。这种黄金参考通常由人工标注（或在从更强模型引导时由人工审核），因为它定义了每一轮对话中“正确”的含义。每个对话都是轮次的序列，每一轮都将黄金（预期）输出与预测（实际）输出配对。轮次携带一个标识谁产生该轮次的轮次角色。响应轮次和操作轮次如下所示：

[
  {
    "turn_no": 1,
    "turn": "Agent",
    "gold_turn":    {"response": "Which region should the report cover?"},
    "predict_turn": {"response": "Sure, which region would you like the report for?"}
  },
  {
    "turn_no": 2,
    "turn": "Tool",
    "gold_turn":    {"tool_id": "reports", "action": "FilterData",
                     "args": {"metric": "revenue", "region": "EU"}},
    "predict_turn": {"tool_id": "reports", "action": "FilterData",
                     "args": {"metric": "profit",  "region": "EU"}},
    "tags": ["OrderInvariant_filter"]
  }
]

Comparing gold and predicted yields the per-turn correctness verdict. The response turn passes: the wording differs from gold but is semantically equivalent, which is exactly what semantic similarity scoring is for. The action turn fails: a truthfulness error surfaces on the metric value (profit where revenue was expected). The tags field supports order-invariant evaluation. When multiple tool calls are valid in any order, such as checking a calendar and searching flights, the evaluator checks against valid orderings. It does not penalize correct but differently sequenced behavior.比较黄金输出和预测输出得出逐轮正确性判定。响应轮次通过：措辞与黄金标准不同，但在语义上是等价的，这正是语义相似度评分的作用。操作轮次失败：真实性错误在指标值上显现（预期为收入，实际为利润）。标签字段支持顺序无关的评估。当多个工具调用以任何顺序都是有效时（例如检查日历和搜索航班），评估器会根据有效的排序进行检查。它不会惩罚正确但顺序不同的行为。

Error attribution错误归因

After every turn is evaluated, error attribution separates root causes from cascading effects. It operates on turn verdicts regardless of whether a turn was a response or an action:在每一轮评估后，错误归因将根本原因与级联效应分离开来。它对轮次判定进行操作，无论该轮次是响应还是操作：

def attribute_errors(turn_results):
    """Separate root cause failures from cascading failures."""
    root_causes = []
    cascading = []

    for result in turn_results:
        if not result.success:
            if result.failure_reason == "prior_action_failed":
                cascading.append(result)
            else:
                root_causes.append(result)

    return {
        "first_failure_turn": root_causes[0].turn_no if root_causes else None,
        "root_cause": root_causes[0].failure_reason if root_causes else None,
        "total_failures": len(root_causes) + len(cascading),
        "root_cause_count": len(root_causes),
        "cascading_count": len(cascading),
    }

# Example output:
# first_failure_turn: 2, root_cause: "inconsistent_parameter_values"
# root_cause_count: 1, cascading_count: 3
# Fix the parameter in turn 2; turns 3-5 likely resolve automatically.

This changes how teams prioritize fixes. Instead of investigating every failure independently, they focus on root causes, since cascading failures often resolve after the root is fixed. A single root cause in a multi-step chain can otherwise appear as several distinct failures.这改变了团队优先处理修复的方式。他们不再独立调查每一个失败，而是专注于根本原因，因为级联失败通常在修复根本原因后就会解决。否则，多步链中的单一根本原因可能会表现为几个不同的失败。

Production monitoring生产监控

In production, the overall correctness score is tracked continuously:在生产环境中，整体正确性分数会被持续追踪：

• Overall correctness by release (regression detection): Did the latest model update degrade turn-level success rate?按版本划分的整体正确性（回归检测）：最新的模型更新是否降低了轮次级成功率？
• Correctness by chain length: Do complex multi-step chains degrade over time?按链长度划分的正确性：复杂的多步链是否会随时间退化？
• Failure reason distribution (root cause trends): Is tool_mismatch increasing after a model swap?失败原因分布（根本原因趋势）：模型更换后 tool_mismatch 是否在增加？
• Latency correlation: Do conversations with higher accumulated latency show lower correctness?延迟相关性：累积延迟较高的对话是否显示出较低的正确性？

The framework outputs structured JSON that feeds into monitoring dashboards. Some errors are costly, such as a financial calculation or a compliance-related response. For those, the decomposed sub-metric scores can also be correlated against human or gold labels. A per-sub-metric correlation (for example, Pearson or Spearman) shows whether the automated score tracks human judgment and where to bring a reviewer into the loop.该框架输出结构化 JSON，输入到监控仪表板中。有些错误代价高昂，例如财务计算或合规相关响应。对于这些错误，分解的子指标分数也可以与人工或黄金标签进行相关性分析。子指标相关性（例如 Pearson 或 Spearman）显示自动化分数是否追踪人工判断，以及在何处需要引入审核员。

Integrating with the Strands Agents evaluation SDK与 Strands Agents 评估 SDK 集成

The methodology is framework-agnostic, but many teams run evaluations through an existing harness. The turn-level correctness signal integrates with the Strands Agents evaluation SDK as a custom evaluator. From there it plugs into the same pipeline teams already use for goal-completion and LLM-as-judge scoring. Those built-in evaluators report quality as a holistic, per-trajectory signal. AEM is complementary, contributing a decomposed, per-turn correctness score that attributes a failure to a specific sub-metric and turn. The wrapper reuses the per-turn checks built earlier in this post (evaluate_truthfulness, evaluate_completeness, and the attribution logic):该方法论与框架无关，但许多团队通过现有的工具链运行评估。轮次级正确性信号作为自定义评估器集成到 Strands Agents 评估 SDK 中。由此，它可以插入到团队已用于目标完成度和 LLM 作为评判者评分的同一管道中。这些内置评估器将质量报告为整体的、每轨迹的信号。AEM 是互补的，贡献了一个分解的、逐轮的正确性分数，将失败归因于特定的子指标和轮次。包装器重用了本文前面构建的逐轮检查（evaluate_truthfulness、evaluate_completeness 和归因逻辑）：

from strands_evals.evaluators import Evaluator
from strands_evals.types import EvaluationData, EvaluationOutput


class CorrectnessCustomEvaluator(Evaluator):
    """Wraps the turn-level correctness checks as a Strands Agents custom evaluator."""

    def __init__(self, threshold: float = 0.5, name: str = "correctness"):
        super().__init__(name=name)
        self._threshold = threshold

    def evaluate(self, evaluation_case: EvaluationData) -> list[EvaluationOutput]:
        # Run the per-turn truthfulness + completeness checks over the dialog
        turn_results = evaluate_dialog(evaluation_case, threshold=self._threshold)

        success_rate = sum(r.success for r in turn_results) / len(turn_results)
        failures = [r.failure_reason for r in turn_results if not r.success]
        return [
            EvaluationOutput(
                score=success_rate,
                test_pass=all(r.success for r in turn_results),
                reason=f"failing turns: {failures}" if failures else "all turns pass",
                label="correctness",
            )
        ]

Here evaluate_dialog applies the same per-turn truthfulness and completeness checks shown earlier and returns a result per turn. This pattern keeps the evaluation logic (the decomposition, the failure taxonomy, the turn-level composition) as portable custom code, while Strands Agents provides the runner, trace collection, and reporting. Concretely, each run yields per-turn traces (spans for tool calls and model invocations) and a structured report. You can display or export that report as JSON to your own dashboards and alerting. AEM adds the per-turn correctness score. The wrapper runs the correctness signal alongside other evaluators in a single run, including the safety evaluator we introduce in the next post.此处 evaluate_dialog 应用了前面展示的相同的逐轮真实性和完整性检查，并返回每轮的结果。这种模式使评估逻辑（分解、失败分类法、轮次级组合）保持为可移植的自定义代码，而 Strands Agents 提供运行器、追踪收集和报告。具体而言，每次运行都会产生逐轮追踪（工具调用和模型调用的跨度）和结构化报告。您可以将该报告显示或导出为 JSON 到您自己的仪表板和警报系统中。AEM 添加了逐轮正确性分数。包装器在单次运行中将正确性信号与其他评估器一起运行，包括我们在下一篇文章中介绍的安全性评估器。

Applying the framework to Amazon Quick Suite将框架应用于 Amazon Quick Suite

Amazon Quick Suite is an enterprise assistant that runs the kind of multi-turn, multi-tool conversations this metric targets. Rather than report internal production numbers, this section walks through how a team reads AEM output, using the sales-report conversation from the opening example.Amazon Quick Suite 是一款企业助手，运行着该指标所针对的多轮、多工具对话。本节不报告内部生产数据，而是通过开篇示例中的销售报告对话，演示团队如何解读 AEM 输出。

Reading an AEM score解读 AEM 分数

What the user ultimately judges is the response they receive at each turn, so that response is the unit we evaluate, turn by turn across the dialog. Behind a single response, the agent often chains several tool calls under interactive latency constraints, and AEM scores the correctness that results.用户最终评判的是他们在每一轮收到的响应，因此该响应是我们逐轮评估对话的单位。在单一响应背后，智能体通常会在交互式延迟约束下链接多个工具调用，而 AEM 对由此产生的正确性进行评分。

Running the evaluation pipeline across a dialog produces the single AEM score for the conversation and its decomposition. The success_rate is the leading indicator, alongside the per-sub-metric breakdown and the test_pass flag, which is true only when every turn passes. That same call returns the per-turn failure reasons that drive error attribution.在整个对话中运行评估管道会产生该对话的单一 AEM 分数及其分解。success_rate 是领先指标，连同子指标细分和 test_pass 标志（仅在每一轮都通过时为 true）。同样的调用会返回驱动错误归因的逐轮失败原因。

A worked attribution example归因示例

Return to the five-turn sales-report conversation. Turn 2 passes profit where revenue was expected, so it fails on truthfulness with inconsistent_parameter_values. Turns 3–5 build on that result and fail too, but as cascades: each carries prior_action_failed. A raw failure count reports four broken turns. Attribution reports one root cause at turn 2 and three downstream effects, which is the number that matters.回到五轮销售报告对话。第 2 轮在预期为收入的地方传递了利润，因此因 inconsistent_parameter_values 在真实性上失败。第 3 至第 5 轮基于该结果构建，也失败了，但属于级联失败：每一轮都带有 prior_action_failed。原始失败计数报告了四个损坏的轮次。归因报告了第 2 轮的一个根本原因和三个下游影响，这才是重要的数字。

The practical rule is to attribute first, then investigate. The following recurring patterns make this concrete, and the table summarizes where to look in each case.实践规则是先归因，再调查。以下循环模式使其具体化，表格总结了每种情况下需要查看的内容。

The takeaway is that longer chains push a larger share of failures back to earlier turns rather than to independent errors. Fixing a small number of root causes can resolve many of the observed failures, so attribution turns a noisy failure list into a short, ordered fix list.结论是，更长的链条将更大比例的失败推回到早期轮次，而不是独立的错误。修复少量的根本原因可以解决许多观察到的失败，因此归因将嘈杂的失败列表转化为简短、有序的修复列表。

Lessons learned经验教训

A few lessons emerged from applying AEM in practice. First, attribute before you investigate: the prior_action_failed label separates root causes from cascades, so one early error in a chain does not read as many independent failures.在实践中应用 AEM 得到了一些经验。首先，在调查前先归因：prior_action_failed 标签将根本原因与级联分离开来，因此链条中的一个早期错误不会被解读为多个独立失败。

Second, tune the similarity threshold to your domain. Set it by your tolerance for false positives versus false negatives. Too strict a threshold flags semantic equivalents such as “NYC” versus “New York City,” while too lenient a threshold misses real errors.其次，根据您的领域调整相似度阈值。根据您对误报与漏报的容忍度进行设置。过于严格的阈值会标记语义等价项（如“NYC”与“New York City”），而过于宽松的阈值则会漏掉真实错误。

Third, tag order-invariant steps. When several tool calls are valid in any order, marking them lets the evaluator credit valid alternate orderings instead of failing correct behavior.第三，标记顺序无关的步骤。当多个工具调用以任何顺序都是有效时，标记它们可以让评估器确认有效的替代排序，而不是判定正确的行为失败。

Two broader lessons concern trust and growth. For costly errors, validate against human labels by correlating the decomposed sub-metrics with human or gold judgments. Bring a reviewer into the loop where an undetected error carries real consequences. And extend with new sub-metrics rather than new pipelines: define per-turn criteria and a failure taxonomy, then compose with the same score. Planning quality, instruction retention, and safety all follow this approach.两个更广泛的经验涉及信任和增长。对于代价高昂的错误，通过将分解的子指标与人工或黄金判断相关联来进行验证。在未检测到的错误会产生实际后果的地方，引入审核员。并使用新的子指标而不是新的管道进行扩展：定义逐轮标准和失败分类法，然后用相同的分数进行组合。规划质量、指令保留和安全性都遵循这种方法。

Conclusion and next steps结论与后续步骤

This post presented the Agent Evaluation Metric (AEM) as a single composite indicator for multi-turn agentic conversations, applied to its first dimension, correctness. AEM decomposes correctness into named sub-metrics (truthfulness and completeness) and evaluates them at the turn level with precise error attribution. It goes beyond detecting failures: it identifies which dimension broke down, which turn caused it, and whether later failures are root causes or cascading effects.本文介绍了智能体评估指标（AEM），将其作为多轮智能体对话的单一复合指标，并应用于其第一个维度——正确性。AEM 将正确性分解为具名子指标（真实性和完整性），并在轮次层面进行精确的错误归因评估。它不仅能检测失败，还能识别出是哪个维度出了问题、哪一轮导致了问题，以及后续的失败是根本原因还是级联效应。

The methodology fits into an evaluation workflow a team already runs. Ground truth comes first, then AEM scores each turn through truthfulness and completeness. Error attribution then separates root causes from cascading failures, so the count reflects independent problems rather than downstream noise. Tracking the score across model versions turns the metric into a regression signal, run inside an existing Strands Agents pipeline through the custom-evaluator wrapper. A decline points to the sub-metric responsible, and the failure taxonomy identifies the precise turn to investigate.该方法论适合团队现有的评估工作流。首先是事实真相，然后 AEM 通过真实性和完整性对每一轮进行评分。错误归因随后将根本原因与级联失败分离开来，因此计数反映的是独立问题而非下游噪音。跨模型版本追踪分数将该指标转化为回归信号，通过自定义评估器包装器在现有的 Strands Agents 管道内运行。下降指向负责的子指标，失败分类法识别出需要调查的精确轮次。

Correctness is the first dimension AEM instantiates through a deliberately extensible approach: decompose a quality concept into named sub-metrics, evaluate each turn within the full trajectory, attribute failures, and compose the results into a single indicator. The next post in this series applies the same method to safety, and later posts extend it to multilingual and multimodal evaluation.正确性是 AEM 通过一种刻意可扩展的方法实例化的第一个维度：将质量概念分解为具名子指标，在完整轨迹中评估每一轮，归因失败，并将结果组合成单一指标。本系列的下一篇文章将相同的方法应用于安全性，后续文章将其扩展到多语言和多模态评估。

To get started, explore the working examples in the Strands Agents samples repository and the Strands Agents evaluation documentation, then adapt the custom evaluator shown earlier to your own dialogs. To learn more about the enterprise assistant used here, see Amazon Quick Suite. Teams adopting correctness today can add safety and other dimensions as their requirements grow.要开始使用，请探索 Strands Agents 示例存储库和 Strands Agents 评估文档中的工作示例，然后将前面展示的自定义评估器调整为您自己的对话。要了解有关此处使用的企业助手的更多信息，请参阅 Amazon Quick Suite。目前采用正确性指标的团队可以在需求增长时添加安全性和其他维度。

About the authors关于作者