How Similarweb Evaluates Long-Form Agent Research Reports with LangSmith Similarweb 如何利用 LangSmith 评估长篇智能体研究报告

Two Ways to Score an Output: Determinist Checks and LLM-as-a-Judge Scoring评估输出的两种方式：确定性检查与“LLM 即法官”评分

We run two kinds of checks on agent outputs.我们对智能体输出运行两种检查。

The first kind is deterministic. Did the agent call the tools it was required to? Did it avoid tools it should not have touched? Did it return valid structured output? These are mechanical pass/fail comparisons, with no model involved.第一种是确定性检查。智能体是否调用了所需的工具？是否避开了不该触碰的工具？是否返回了有效的结构化输出？这些是机械化的通过/失败对比，不涉及模型。

The second kind is an LLM-as-a-judge. Whenever the thing we care about is meaning or quality, deterministic rules fall short, so we hand the decision to a model. The judge prompt gives that model three things: 1.) the user question, 2.) the agent's output, and 3.) a standard to score against. It then returns a score and a comment explaining that score. The standard can be a golden answer ("does this say the same thing?") or a rubric with explicit scoring anchors ("how well does this meet each quality bar?").第二种是“LLM 即法官”。每当我们关注的是含义或质量时，确定性规则往往力不从心，因此我们将决策权交给模型。法官提示词会为该模型提供三样东西：1. 用户问题，2. 智能体输出，3. 评分标准。随后，模型返回一个分数和一段解释该分数的评论。标准可以是标准答案（“这是否表达了相同含义？”）或带有明确评分锚点的准则（“这在多大程度上达到了每个质量门槛？”）。

We treat the judge as a scalable reviewer. It  produces inspectable signals—a score, the reasoning behind it, and a link to the trace—so we can see what the agent actually did.我们将法官视为可扩展的审查员。它产生可审查的信号——评分、背后的推理以及指向追踪记录的链接——这样我们就能看到智能体实际做了什么。

Both kinds of checks run in the same evaluation loop and land side by side as feedback in LangSmith. The harder question for us was, what should the judge score against? 两种检查都在同一个评估循环中运行，并作为反馈并排呈现在 LangSmith 中。对我们而言，更难的问题是：法官应该以什么为准进行评分？

The Easy Case: Regular Chat简单案例：常规聊天

The simplest case to evaluate in an agentic system is regular chat.在智能体系统中，最简单的评估案例是常规聊天。

Caption: Regular chat: a focused question with a more expected answer shape.图注：常规聊天：针对性强的问题，答案形式相对可预期。

Each benchmark example has a fixed prompt, a golden answer, expected tool checks, and an agent output. We can compare the output to the golden answer, verify the agent called the required tools, and track scores over time.每个基准测试示例都有固定的提示词、标准答案、预期的工具检查和智能体输出。我们可以将输出与标准答案进行对比，验证智能体是否调用了所需工具，并随时间追踪评分。

experiment_results = await aevaluate(
    target,
    data=dataset_id,
    evaluators=evaluators,
    experiment_prefix=experiment_prefix,
    num_repetitions=num_repetitions,
    max_concurrency=max_concurrency,
)

Both kinds of checks show up here. The tool checks are deterministic. Comparing the output to the golden answer, though, is a job for an LLM-as-judge, because a correct answer can be phrased many ways and exact text matching is too brittle. That judge is the semantic evaluator, anchored to the golden answer. Its only question is whether the output means the same thing.这里会同时使用两种检查。工具检查是确定性的。但将输出与标准答案进行对比则需要“LLM 即法官”，因为正确的答案可以用多种方式表达，而精确的文本匹配过于僵化。该法官充当语义评估员，以标准答案为锚点。它唯一的问题是：输出的含义是否相同？

Each evaluator returns one piece of feedback, a key, a score, and a comment. For the traffic-by-channel question above, the semantic judge returns something like this:每位评估员都会返回一条反馈、一个键值、一个分数和一条评论。对于上述关于渠道流量的问题，语义法官返回的结果如下：

{
    "key": "semantic",
    "score": 0.7,
    "comment": "Matches the reference on the top channels (direct and organic search) and gets the direct share right, but omits the referral traffic the golden answer calls out, so it is close but incomplete."
}

LangSmith turns that feedback into experiment columns. Instead of collecting scores in a spreadsheet, we can compare runs, click into a score, and inspect the trace behind it.LangSmith 将这些反馈转化为实验列。我们无需在电子表格中收集分数，而是可以直接对比运行结果，点击分数并查看其背后的追踪记录。

For regular chat, this is enough. There is an expected answer, so the deterministic checks and a semantic judge anchored to it cover most of what matters.对于常规聊天，这已经足够了。由于存在预期答案，确定性检查和以其为锚点的语义法官涵盖了大部分关键点。

The Hard Case: Deep Research复杂案例：深度研究

Deep Research outputs are long-form reports with sources, interpretation, recommendations, caveats, and narrative structure. There may be many good reports for the same question.深度研究的输出是长篇报告，包含来源、解读、建议、注意事项和叙述结构。对于同一个问题，可能会有许多份优秀的报告。

Caption: Deep Research: an open-ended question where many good reports are possible.图注：深度研究：开放式问题，可能存在多种优秀的报告形式。

One strong report might focus on traffic acquisition. Another might focus on competitive positioning. Another might focus on monetization risk. All can be valid if the claims are grounded and the reasoning is sound.一份出色的报告可能侧重于流量获取，另一份可能侧重于竞争定位，还有一份可能侧重于变现风险。只要观点有据可依且推理合理，它们都可以是有效的。

A golden answer doesn’t work well for this case. There is no single reference to compare against, and matching a reference report would only reward similarity, not quality. So instead of a reference we give the judge a rubric: each quality dimension gets its own rubric with explicit scoring anchors, so the judge returns a score per dimension rather than one overall verdict. For example, the source_integration criterion asks whether the report uses external sources beyond raw platform data:标准答案在这种情况下效果不佳。没有单一参考基准可供对比，且匹配参考报告只会奖励相似度而非质量。因此，我们不提供参考，而是给法官一套评分标准：每个质量维度都有自己的准则和明确的评分锚点，这样法官会针对每个维度返回分数，而不是给出一个总评。例如，source_integration（来源整合）标准会询问报告是否使用了平台原始数据之外的外部来源：

source_integration:
Does the report effectively integrate external sources beyond raw platform data?
Evaluate both diversity AND quality of source integration.

Score 0.0: Relies purely on a single data API with no external context.
Score 0.3: Mentions external sources vaguely, e.g. "according to industry reports".
Score 0.8: Cites multiple named sources with dates, figures, and context.
Score 1.0: Uses extensive, attributed sources that are well-integrated into the narrative.

Return:
- source_integration: score from 0.0 to 1.0
- source_integration_gap: short summary of the gap
- source_integration_detail: one sentence explaining the score

Each criterion returns a score plus a short explanation, so a low number always comes with a reason we can inspect. A thinly-sourced report earns exactly that:每个准则都会返回一个分数加上简短解释，因此低分总是伴随着我们可以审查的原因。一份来源匮乏的报告会得到相应的评价：

{
    "source_integration": 0.3,
    "source_integration_gap": "External context gestured at, never attributed",
    "source_integration_detail": "Leans almost entirely on Similarweb traffic data and twice waves at 'industry reports' without naming a single source, date, or figure. The outside context is decorative, not evidential."
}

Alongside the quality rubrics, we run faithfulness checks to verify, does each claim actually follow from the retrieved data, or did the agent overstate what the sources support? This is where confident but ungrounded statements show up, which matters most when the output is long and persuasive.除了质量准则外，我们还会运行忠实度检查，以验证每个观点是否确实源自检索到的数据，还是智能体夸大了来源支持的内容？这是那些自信但缺乏依据的陈述出现的地方，当输出篇幅较长且具有说服力时，这一点尤为重要。

Rubrics and faithfulness checks score a report on its own terms. To decide whether a new version is actually better, we also run A/B comparisons against a baseline: a saved, accepted previous run we treat as a reference point, not as ground truth. The judge sees the new report beside the baseline and says which is stronger, instead of scoring in isolation.评分准则和忠实度检查是根据报告自身表现进行评分的。为了判断新版本是否真的更好，我们还会进行 A/B 对比，将其与基准进行比较：即一个已保存且被认可的先前运行结果，我们将其视为参考点，而非绝对真理。法官会同时查看新报告和基准报告，并判断哪一个更强，而不是孤立地进行评分。

LangSmith Connected Evaluator Scores to Traces and Baseline Comparisons LangSmith 将评估员评分与追踪记录及基准对比关联起来

Writing an evaluator prompt is only part of the workflow. The harder part is  running the loop repeatedly and deciding whether a change is real.编写评估员提示词只是工作流的一部分。更难的部分是反复运行循环并判断变更是否真正有效。

LangSmith made that possible because nothing was siloed. aevaluate ran the loop over a fixed dataset, with repetitions and concurrency; every evaluator score landed as a column, every score carried its comment, and every run linked straight to the trace and to a baseline for comparison. The differentiator was the connection between the  number, the evaluator reasoning, and the agent behavior behind it.LangSmith 使之成为可能，因为没有任何环节是孤立的。aevaluate 在固定数据集上运行循环，支持重复和并发；每个评估员评分都作为一列呈现，每个评分都带有评论，且每次运行都直接链接到追踪记录和对比基准。其差异化优势在于将分数、评估员推理和背后的智能体行为连接了起来。

Instead of asking only, "Did the average score move?", we could ask:我们不再仅仅询问“平均分是否变动？”，而是可以询问：

Which cases moved, which criteria moved, what did the evaluator say, and what happened in the trace?哪些案例变了、哪些准则变了、评估员说了什么、追踪记录中发生了什么？

That is the difference between evaluation as a scoreboard and evaluation as an engineering workflow for building agentic systems.这就是将评估视为计分板与将其视为构建智能体系统的工程工作流之间的区别。

Miscalibrated Rubrics Can Make Good Updates Look Like Regressions 校准错误的评分标准会让优秀的更新看起来像是性能倒退

The first version of the Deep Research evaluation slowed us down instead of speeding us up as intended. 第一版深度研究评估不但没能按预期加快我们的进度，反而拖慢了我们。

We made a small prompt change, re-ran the benchmark, and the overall score dropped. So we treated it as a regression: revert, tweak, re-run. Then again. The reports looked good to us every time, but the number kept disagreeing, and we trusted the number over our own reading. 我们对提示词做了一个小改动，重新运行基准测试，总分却下降了。于是我们将其视为性能倒退：回滚、调整、重跑。再次重复。在我们看来报告每次都很好，但分数一直不一致，而我们却盲目信任分数，忽略了自己的阅读判断。

We lost the better part of a week fighting our own evaluation before anyone opened the per-criterion comments. When we finally did, the answer was almost embarrassing…the new reports cited more sources, which lifted source breadth, but those sources were vague, which sank attribution. Two criteria were pulling against each other, and the aggregate score hid the conflict. The change had been fine the whole time. We had spent a week arguing with a miscalibrated ruler.我们在与自己的评估系统斗争中浪费了近一周时间，才有人打开查看每项准则的评论。当我们最终查看时，答案简直令人尴尬……新报告引用了更多来源，这提升了来源广度，但这些来源含糊不清，导致归属感评分下降。两个标准相互抵触，而总分掩盖了这种冲突。其实这次变更一直都是好的。我们花了一周时间与一把校准错误的尺子争论。

One issue was conflicting criteria, the exact one that cost us the week. From a product perspective, we learned that we did not want more sources for their own sake. We wanted named, relevant sources tied to specific claims.其中一个问题是准则冲突，这正是导致我们浪费一周的原因。从产品角度看，我们意识到我们并不想要为了引用而引用来源。我们想要的是与特定观点相关联的、具名的、相关的来源。

You could watch the miscalibration in the comments. Before we fixed the rubric, a report stuffed with vague references still scored well just for reaching outside the platform:你可以在评论中观察到这种校准错误。在我们修复评分标准之前，一份充斥着模糊引用的报告仅仅因为引用了平台外的内容就能获得高分：

{
    "source_integration": 0.7,
    "source_integration_detail": "Broad outside context, nine references, but most are unnamed ('industry reports', 'analysts expect'), so the attribution is thin."
}

The score rewarded breadth; the comment already knew the attribution was thin. Once we rewrote the anchor to prize named, verifiable sources over raw count, that same report settled at 0.3, and the number finally matched what the reasoning had been saying all along.分数奖励了广度；而评论其实已经指出归属感很弱。当我们重写锚点以奖励具名、可验证的来源而非原始数量后，同一份报告的分数稳定在 0.3，最终结果终于与推理逻辑保持了一致。

Another issue was wrong incentives. If conciseness was rewarded more clearly than completeness, reports became shorter even when the user asked for strategic research that needed caveats, methodology, and broader context.另一个问题是错误的激励机制。如果简洁性比完整性被奖励得更明确，报告就会变得简短，即使当用户要求进行需要注意事项、方法论和更广阔背景的战略研究时也是如此。

In both cases, the moving score was not the useful part. The comments and traces behind each score were what let us see which criterion was miscalibrated and why. We could debug the rubric instead of guessing from the aggregate score. 在这两种情况下，变动的分数本身并非重点。每个分数背后的评论和追踪记录才让我们能够看出哪个准则校准错误以及原因所在。我们可以调试评分标准，而不是从总分中盲目猜测。

A miscalibrated evaluation is worse than no evaluation, because it hands you false confidence. Before trusting an evaluation result, we had to ask whether the evaluator was measuring the behavior we actually cared about. 校准错误的评估比没有评估更糟糕，因为它会给你虚假的信心。在信任评估结果之前，我们必须自问：评估员是否在衡量我们真正关心的行为？

The Workflow That Worked行之有效的工作流

Once the criteria were calibrated, the loop we settled into was:准则校准完成后，我们确定的工作循环如下：

1. Start with a hypothesis.从一个假设开始。
2. Run a small evaluation for a signal.运行小规模评估以获取信号。
3. Inspect feedback comments and traces.检查反馈评论和追踪记录。
4. Run the full benchmark with repetitions.进行带重复测试的完整基准测试。
5. Compare against a baseline or A/B output.与基准或 A/B 输出进行对比。
6. Decide whether to merge, iterate, or recalibrate.决定是合并、迭代还是重新校准。

Final Takeaways: Golden Answers, Rubrics, Traces, and Baselines Serve Different Parts of Agent Evaluation最终要点：标准答案、准则、追踪记录和基准服务于智能体评估的不同部分

Golden answers are enough when there is an expected answer. Long-form agent outputs are not, so they need rubric prompts, faithfulness checks, and A/B judgment instead.当存在预期答案时，标准答案就足够了。长篇智能体输出则不然，因此需要评分标准提示词、忠实度检查和 A/B 判断。

Evaluation and observability should live together. The score tells you where to look. The trace tells you what happened. The feedback tells you why the evaluator scored it that way.评估与可观测性应紧密结合。分数告诉你该看哪里，追踪记录告诉你发生了什么，反馈告诉你评估员为何给出那样的评分。

That changed how we build agentic systems. We no longer treat evaluation as a release checklist item. We use it to shape product decisions such as which behavior is worth optimizing, which regressions are real, which prompts are miscalibrated, and where does the agent need better tool use or better synthesis.这改变了我们构建智能体系统的方式。我们不再将评估视为发布检查清单中的一项，而是用它来塑造产品决策，例如哪些行为值得优化、哪些性能倒退是真实的、哪些提示词校准错误，以及智能体在何处需要更好的工具使用或综合能力。

Evaluation does not remove human judgment. It makes judgment inspectable, repeatable, and connected to the agent’s actual behavior, so every update is based on evidence instead of one good-looking output. 评估并不能取代人类判断。它使判断变得可审查、可重复，并与智能体的实际行为挂钩，从而确保每次更新都是基于证据，而非仅仅因为一个看起来不错的输出。