Towards Automating Eval Engineering迈向评估工程自动化

Building the Environment & Task构建环境与任务

The skill first reads the repository and maps the agent surface including prompts, models, tools, skills, hooks, etc. It also identifies the data and services that back those behaviors such such as API calls.该技能首先会读取代码库并映射智能体的表面结构，包括提示词、模型、工具、技能、钩子（hooks）等。它还会识别支撑这些行为的数据和服务，例如 API 调用。

Users can also point the agent to traces which can be retrieved using tools like the langsmith-cli.  Traces show how tools behave in practice such as their arguments, results, and errors. These observed contracts help the skill reproduce relevant production behavior in a controlled environment.用户还可以让智能体指向使用 langsmith-cli 等工具检索到的追踪记录。追踪记录展示了工具在实际运行中的表现，例如参数、结果和错误。这些观察到的契约有助于该技能在受控环境中重现相关的生产环境行为。

Crawling the repo and traces gives the agent knowledge of a which abilities are important for the agent as it proposes eval tasks. We found that interviewing the user, leads to much better eval acceptance than one-shot generation.  The user chooses from the proposed eval directions, and gives guidance on questions such as which tools & dependencies should run live or need to be simulated.  For example, tool calls that incur costs or require writes to production can be simulated instead of being run on every eval invocation.通过爬取代码库和追踪记录，智能体能够了解哪些能力对其至关重要，从而提出相应的评估任务。我们发现，与“一次性生成”相比，通过与用户交互来确定评估方案能获得更高的认可度。用户可以从建议的评估方向中进行选择，并就哪些工具和依赖项应实时运行或需要模拟等问题提供指导。例如，产生费用或需要写入生产环境的工具调用可以被模拟，而无需在每次评估执行时真实触发。

We tested this flow on our documentation Q&A agent, chat-langchain.  For this agent, the environment required a data corpus exposed through agent search tools modeled on the production agent. The tasks included realistic documentation question pulled from real traces and a verifier that checked the answer using a golden answer string and cited documents.我们在文档问答智能体 chat-langchain 上测试了这一流程。对于该智能体，环境需要一个通过智能体搜索工具公开的数据语料库，并以生产环境中的智能体为模型。任务包括从真实追踪记录中提取的实际文档问题，以及一个使用标准答案字符串和引用文档来核验答案的验证器。

Eval Design is iterative评估设计是一个迭代过程

We found that while agents are sometimes able to one-shot evals, the best evals came from users providing feedback and specifying which capabilities were worth measuring in agents. Coding agents & skills provide a natural interface where domain knowledge on how to build a good eval are encoded and users can iterate over them over time.我们发现，虽然智能体有时能够一次性完成评估构建，但最佳的评估方案往往源于用户的反馈，以及用户明确指出哪些能力值得衡量。编码智能体和相关技能提供了一个自然的交互界面，能够将构建优秀评估的领域知识编码其中，并允许用户随时间推移不断迭代。

For example, we found that when building verifiers, the first verifier was rarely the final one. A useful way to improve it was to run the eval and inspect both sides of the result:例如，我们发现构建验证器时，最初的版本往往不是最终版本。改进它的一个有效方法是运行评估并检查结果的两个方面：

• the agent trajectory, including its messages, tool calls, and actions.智能体的运行轨迹，包括其消息、工具调用和动作。
• the verifier trajectory, evidence, reasoning, and final score.验证器的运行轨迹、证据、推理过程和最终得分。

This helped reveal if the task or verifier design was measuring what we cared about or if it could be reward hacked where agents could take shortcuts.  These shortcuts could include overciting irrelevant sources to receive full credit on the eval, claim an action it never took, exploit exposed answer material, or satisfy a proxy without completing the task. Observing the traces for how agents solve problems often reveal the source of these failures. The task, environment, and verifier can then be revised and run again.这有助于发现任务或验证器的设计是否在衡量我们关心的指标，或者是否存在智能体可以通过捷径获利的“奖励黑客”（reward hacking）行为。这些捷径可能包括过度引用无关来源以获得满分、声称执行了从未采取过的行动、利用暴露的答案材料，或在未完成任务的情况下满足代理指标。观察智能体解决问题的追踪记录，往往能揭示这些失败的根源。随后，可以对任务、环境和验证器进行修订并重新运行。

Evals are in Harbor format评估采用 Harbor 格式

The skill builds evals as Harbor tasks:该技能将评估构建为 Harbor 任务：

1. An Instruction: the message given to the agent at start describing the task指令（Instruction）：在开始时发送给智能体，描述任务的消息。
2. An environment: given as a Dockerfile containing the setup for the task such as what tools to install or what data to populate in the filesystem环境（Environment）：以 Dockerfile 形式提供，包含任务的设置，例如需要安装的工具或需要在文件系统中填充的数据。
3. A verifier that scores whether the agent completed the task correctly.验证器（Verifier）：对智能体是否正确完成任务进行评分。

The skill builds these components together as a Harbor task:该技能将这些组件整合为一个 Harbor 任务：

evals/<task-id>/
├── task.toml
├── instruction.md
├── environment/
└── tests/

Harbor runs the agent in the environment and records its trajectory, artifacts, reward, and errors. The same eval can then run against different models, prompts, tools, and agent versions.Harbor 在环境中运行智能体，并记录其轨迹、产出物、奖励和错误。同一评估可以在不同的模型、提示词、工具和智能体版本上运行。

Why this matters为什么这很重要

Continual learning can be thought of as a continuous data mining problem where production data is used to build evals that improve agents over time. Teams mine traces to find recurring user requests, errors, failed tool calls, and incorrect state changes. which become evals so the same behavior can be measured and prevented in the future.持续学习可以被视为一个连续的数据挖掘问题，即利用生产数据构建评估，从而随时间推移不断改进智能体。团队通过挖掘追踪记录来发现重复出现的用户请求、错误、失败的工具调用和不正确的状态变更，并将这些转化为评估任务，以便在未来衡量并预防相同的行为。

Evals are training data for agents. Teams can fit agent behavior to them through harness engineering such as changing prompts & tools or fine-tuning. The eval provides a fixed target for deciding whether those changes improved the intended capability.评估是智能体的训练数据。团队可以通过“工具工程”（harness engineering）——例如更改提示词、工具或进行微调——来调整智能体行为以适配这些评估。评估提供了一个固定的目标，用于判定这些更改是否真正提升了预期的能力。

Containerized evals make this process faster. The task and environment remain stable while the agent configuration changes, so builders can swap models, tools, prompts, or complete agent versions and compare results directly. Multiple configurations can run in parallel.容器化的评估使这一过程更加快捷。当任务和环境保持稳定时，构建者可以更改智能体配置（如更换模型、工具、提示词或整个智能体版本），并直接比较结果。多个配置可以并行运行。

Reproducible environments are critical to that signal. When an eval mirrors the relevant tools, data, permissions, state, and failure modes from production, builders get a stable testbed that is still representative of how the agent operates. They can experiment quickly without relying on changing production systems or writing to production state.可复现的环境对于获取有效信号至关重要。当评估能够反映生产环境中的相关工具、数据、权限、状态和故障模式时，构建者就能获得一个既稳定又具有代表性的测试床。他们可以快速进行实验，而无需依赖不断变化的生产系统或写入生产状态。

The resulting loop is:由此形成的循环是：

mine traces -> identify a failure -> build an eval -> improve the agent -> rerun 挖掘追踪记录 -> 识别故障 -> 构建评估 -> 改进智能体 -> 重新运行

Try it today立即尝试

The Eval Engineering Skill is available in the langchain-ai/langchain-skills repository.“评估工程技能”可在 langchain-ai/langchain-skills 代码库中获取。

Install the skill in Codex or Claude Code, open the repository containing the agent you want to evaluate, point to agent to a set of traces if available, and start with a simple prompt:‍在 Codex 或 Claude Code 中安装该技能，打开包含您想要评估的智能体的代码库，在有追踪记录的情况下将其指向追踪记录，并从一个简单的提示词开始：

Use the eval-engineering skill to create an eval with me. Inspect the agent first, propose a few abilities worth testing, recommend one, and wait for me to choose.
使用 eval-engineering 技能与我一起创建评估。先检查智能体，提出几个值得测试的能力点，推荐其中一个，并等待我做出选择。

The result is a Harbor task under evals/, an actual target run, and a review of whether the verifier measured the intended behavior correctly.结果将是在 evals/ 目录下生成一个 Harbor 任务、一次实际的目标运行，以及对验证器是否正确衡量了预期行为的评审。

We’re looking forward to expanding this skill and building tooling to make it easier to automatically build evals and fit agents to them autonomously.我们期待扩展这项技能，并开发相关工具，以更轻松地实现评估的自动化构建，并让智能体能够自主地进行自我适配。