Here’s how most people build an eval. They open a file, write an LLM judge prompt that says something like “rate this response from 1 to 10 on helpfulness”, run it over a few hundred traces, get a pile of numbers back, and then have absolutely no idea whether those numbers mean anything.大多数人构建评估的方式是这样的：他们打开一个文件，写一个 LLM 判官提示，内容类似于“对这个回复的有用性进行 1 到 10 的评分”，在几百个轨迹上运行它，得到一堆数字，然后完全不知道这些数字是否有意义。
I get it. It feels like progress. You’ve got a dashboard, you’ve got a metric trending, you can put it on a slide. But you’ve skipped the part that makes an eval worth running, which is knowing it agrees with a human. A number you can’t trust is worse than no number, because at least no number doesn’t lull you into thinking everything’s fine.我明白，这感觉像是进步。你有了仪表盘，有了指标趋势，可以放到幻灯片上。但你跳过了让评估值得运行的关键部分——即它是否与人类一致。一个不可信的数字比没有数字更糟，因为至少没有数字不会让你误以为一切都很好。
So here’s the longer way round. It’s five steps, and it borrows a trick from qualitative researchers - the people who code interview transcripts for a living and have been doing the “turn messy human judgement into something measurable” thing for decades.所以这里是更长的做法。它分为五步，并借鉴了定性研究者的技巧——这些人以编码访谈记录为生，已经做了数十年的“把混乱的人类判断转化为可测量的东西”。
Want it to hand? Download the infographic as a PDF.想要手动获取吗？将信息图下载为 PDF。
The whole flow goes left to right, from gather and observe to measure and validate. The first half is about understanding what actually goes wrong. The second half is about building checks for it and proving they work. Skip the first half and you’re just guessing in a more expensive way.整个流程从左到右进行，从收集与观察到测量与验证。前半部分是了解实际出错的地方。后半部分是为这些错误构建检查并证明它们有效。跳过前半部分，你只是在更昂贵的方式下猜测。
Step 1: Gather a golden dataset步骤 1：收集黄金数据集
You can’t measure anything without a set of examples to measure against. So the first job is to collect a representative set of real interactions your app has to handle - the dull ones, the weird ones, the ones that fall over.没有一套可供衡量的示例，就无法测量任何东西。因此第一件事是收集一套能代表你的应用必须处理的真实交互——包括乏味的、奇怪的、以及会出错的交互。
The word that matters here is representative. Everything downstream gets measured against this set, so if it doesn’t look like real usage, your evals are tuned for a world that doesn’t exist. Pull from production traffic if you’ve got it.这里关键的词是“代表性”。下游所有的测量都基于这套数据，如果它不像真实使用情况，你的评估就会针对一个不存在的世界进行调校。如果有生产流量，尽量从中抽取。
If you don’t have enough real data yet - and early on, you won’t - synthesise some. Generate examples that look like the traffic you expect. It’s not as good as the real thing, but a synthetic golden set you’ve thought about beats no golden set at all. Just be honest with yourself about which bits are made up.如果你还没有足够的真实数据——在早期阶段几乎不可能——可以合成一些。生成看起来像你预期流量的示例。虽然不如真实数据好，但一个你思考过的合成黄金集总比没有黄金集强。只要对自己诚实，说明哪些部分是捏造的。
Step 2: Human open coding步骤 2：人工开放编码
Now the unglamorous bit. A human - probably you - sits down and reads each example in the golden set, and writes free-form notes on what went wrong. Not against a checklist. Not “score this 3 out of 5”. Just plain notes: “made up a refund policy”, “ignored half the question”, “weirdly passive-aggressive tone”.现在是乏味的部分。一个人——可能是你——坐下来阅读黄金集中的每个示例，并对出错的地方写自由形式的笔记。不是对照检查表，也不是“给 3 分（满分 5 分）”。只是普通的笔记：“编造了退款政策”“忽略了问题的一半”“语气奇怪地被动攻击”。
This is called open coding, and the whole point is that you have no fixed categories yet. You’re not confirming a theory about what your app gets wrong. You’re letting the failure modes show up on their own.这叫开放编码，核心在于你还没有固定的类别。你并不是在验证关于你的应用出错的某个理论，而是让失败模式自行显现。
This matters more than it sounds. If you start with a list of things to check, you’ll only ever find the things on your list. Open coding surfaces the failures from the ground up - including the ones you’d never have thought to look for, which are usually the ones that bite.这比听起来更重要。如果你从一份检查清单开始，你只能发现清单上的问题。开放编码从底层挖掘出失败——包括那些你从未想到去寻找的，往往也是最致命的。
Step 3: Axial coding步骤 3：轴向编码
Open coding leaves you with a big messy pile of notes. Step three is turning that pile into a tight set of categories. This is axial coding, and it’s where you let an LLM earn its keep.开放编码会留下大量凌乱的笔记。第三步是把这些笔记整理成紧凑的类别。这就是轴向编码，也是让 LLM 发挥作用的地方。
You and the model go through the notes together and group them into recurring failure categories. Forty scattered observations collapse into something like five real buckets: hallucinated policy, incomplete answer, tone problems, ignored context, broke format. The LLM is genuinely good at this - spotting that “made up a refund policy” and “invented a returns window” are the same underlying failure - and you’re there to sanity-check that the buckets actually make sense.你和模型一起浏览笔记，并将其归类为重复出现的失败类别。四十条分散的观察可以归结为大约五个真实的桶：捏造政策、答案不完整、语气问题、忽略上下文、格式破坏。LLM 在这方面真的很擅长——它能识别出“编造了退款政策”和“虚构了退货窗口”是同一种根本性错误——而你则负责检查这些桶是否真的有意义。
What you get out of this is a taxonomy. And that taxonomy is the thing your evals will measure. You’ve gone from “I think our bot is a bit off sometimes” to “here are the five specific ways it fails”, which is a much better place to be standing.你得到的就是一个分类法。这个分类法就是你的评估将要衡量的对象。你已经从“我觉得我们的机器人有时有点跑偏”转变为“这里是它出错的五种具体方式”，这是一种更好的状态。
While you’re here, tag each example in the golden set with the categories it actually shows - including the ones that are simply fine. That labelled set is your ground truth, and it’s what you’ll check your evals against in a minute.在此过程中，为黄金集中的每个示例打上实际出现的类别标签——包括那些完全正常的示例。这个带标签的集合就是你的真实标签集，稍后你将用它来检验评估。
Step 4: Design the evals步骤 4：设计评估
Now, and only now, do you build evals. One per failure category. You know exactly what you’re checking for, because steps two and three told you.现在，只有现在，你才开始构建评估。每个失败类别对应一个评估。因为第二步和第三步已经告诉你要检查什么，你完全清楚自己的目标。
For each category, pick the cheapest method that’ll do the job. Broke format is a code check - free, fast, deterministic. Passive-aggressive tone needs an LLM judge, because no regex is going to catch attitude. Some categories want a bit of both. I went through the whole spectrum of how to run these - plain code checks, LLM judges, code-and-LLM hybrids, and full agentic harnesses - in a separate post on the four ways to run an eval, so I won’t repeat it here.对于每个类别，选择最便宜且能完成任务的方法。格式破坏可以用代码检查——免费、快速、确定性。被动攻击性语气需要 LLM 判官，因为正则表达式抓不住态度。某些类别可能需要两者结合。我在另一篇关于四种评估实现方式的帖子中已经详细讨论了各种实现——纯代码检查、LLM 判官、代码+LLM 混合以及完整的代理式实现——这里不再赘述。
The point is that by this stage, designing the eval is the easy part. You’re not guessing what to measure. You’re building a known check for a known failure.关键是到这一步时，设计评估已经是最容易的部分。你不再猜测要衡量什么，而是为已知的失败构建已知的检查。
Step 5: Test against the golden set步骤 5：在黄金集上测试
This is the step everyone skips, and it’s the one that makes the whole thing trustworthy.这是所有人都会跳过的步骤，却是让整个过程可信的关键。
You’ve got your golden set. You’ve got human labels on it from steps two and three - you know what actually went wrong in each example. So run your shiny new evals back over that same golden set and check whether they agree with the human verdicts.你已经拥有黄金集，并且在第二步和第三步为其打上了人工标签——你知道每个示例到底出了什么问题。于是把新建的评估再次运行在同一黄金集上，检查它们是否与人工判决一致。
If your tone judge flags the same responses you flagged, brilliant - that’s real evidence it’ll hold up on traffic it’s never seen. If it disagrees, you’ve caught it now, on a controlled set, instead of six weeks into production when it’s been quietly grading the wrong thing the whole time. Tweak the eval, run it again, repeat until it lines up with the humans.如果你的语气判官标记了与你标记相同的回复，太棒了——这是真实的证据，说明它在未见过的流量上也能可靠工作。如果出现分歧，你就在受控集合中发现了问题，而不是在生产环境运行了六周后才发现它一直在错误地打分。调整评估、重新运行、重复，直到它与人工一致。
That’s the bit that turns a number into a number you believe. Your eval has been checked against ground truth before it ever touches live data.这就是把一个数字变成你可信的数字的关键。你的评估在接触真实数据之前已经通过了真实标签的检验。
Why bother with all this为什么要这么做
Because the alternative is a metric that looks like signal and is actually noise, and those are dangerous - they’re confidently wrong, and they’re on a dashboard, so everyone trusts them.因为另一种选择是得到看似有信号实则噪声的指标，而这非常危险——它们自信地错误，却出现在仪表盘上，导致所有人都信任它们。
The open-coding-then-axial-coding move is the bit worth stealing even if you ignore everything else here. Look before you measure. Let the failures tell you what the categories are, instead of deciding the categories first and forcing reality to fit. Researchers have been doing it for decades because it works, and it carries over nicely from a stack of interview transcripts to a pile of agent traces.即使你忽略本文的其他内容，开放编码再到轴向编码的做法也值得借鉴。先观察再测量。让失败告诉你类别是什么，而不是先决定类别再强行让现实适应。研究人员已经这样做了数十年，因为它有效，而且可以轻松从一堆访谈记录迁移到一堆代理轨迹。
Build the golden set, read it properly, find the real failure modes, build a check for each one, and prove those checks agree with you. Then you’ve got an eval you can actually trust - and once you trust it, running it is the easy part.构建黄金集，认真阅读，找出真实的失败模式，为每个模式构建检查，并证明这些检查与你一致。这样你就拥有了一个真正可信的评估——一旦你信任它，运行评估就变得轻而易举。