In the previous post, you saw how agent debug logs reveal what an AI agent actually did versus what you expected. One key takeaway was that AI agents are non-deterministic – the same prompt can produce different execution paths on different runs. In this post, you’ll look at one practical consequence of that non-determinism: asking the model to produce a numeric score.在上一篇文章中，你已经看到智能体调试日志如何揭示 AI 智能体的实际操作与你的预期有何不同。一个关键的结论是：AI 智能体具有非确定性——相同的提示词在不同运行中可能会产生不同的执行路径。在本文中，你将看到这种非确定性的一个实际后果：要求模型生成数值评分。
The pattern that looks reasonable看似合理的模式
You’re building a triage workflow. Issues come in, and you want to rank them by priority so your team knows what to tackle first. You write a prompt like this:假设你正在构建一个分类工作流。当问题提交进来时，你希望按优先级对它们进行排序，以便团队知道首先处理什么。你编写了如下提示词：
1Score this issue from 1-10 based on the following criteria:
2- Impact: How many users are affected? (1=single user, 10=all users)
3- Urgency: How time-sensitive is the fix? (1=cosmetic, 10=data loss)
4- Risk: What's the blast radius if we don't fix it? (1=contained, 10=cascading)
5
6Final score = (Impact × 0.4) + (Urgency × 0.35) + (Risk × 0.25)You hand it an issue description. The model thinks carefully, assigns sub-scores, applies the formula, and returns 6.8. Looks great.你将问题描述交给它。模型经过仔细思考，分配子分数，应用公式，然后返回 6.8。看起来很棒。
You run it again on the exact same issue. This time: 5.2. The next run: 7.5. The criteria didn’t change. The input didn’t change. But the score swings by 40% between runs.你在同一个问题上再次运行它。这次结果是 5.2。下一次运行：7.5。准则没有变，输入也没有变，但分数在不同运行之间波动了 40%。
Why the numbers don’t stabilize为何数值无法稳定
Three things conspire to make model-generated scores unreliable.有三个因素共同导致了模型生成的评分不可靠。
First, the reasoning path changes every run. As discussed in the context of multiple subagents, models sample from probability distributions. That means the same prompt can produce different internal reasoning chains. One run might emphasize the word “all” in the issue description and rate impact highly. The next run might focus on “workaround available” and rate urgency low. Different reasoning chains produce different sub-scores, which then produce different final numbers.首先，推理路径每次运行都会改变。正如在讨论多个子智能体时所提到的，模型是从概率分布中进行采样的。这意味着相同的提示词可能产生不同的内部推理链。一次运行可能会强调问题描述中的“所有”一词，从而给出很高的影响评分；下一次运行可能会关注“有变通方法”而给出很低的紧迫性评分。不同的推理链产生不同的子分数，进而产生不同的最终数值。
Second, models are trained on language, not arithmetic. When you write (Impact × 0.4) + (Urgency × 0.35) + (Risk × 0.25), the model does not reliably evaluate that expression like a calculator. It predicts likely tokens based on its reasoning. Sometimes that prediction matches the right math. Sometimes it does not. The errors are usually small and inconsistent – off by 0.3 here, rounded differently there – but they compound with the first problem. While newer models with tool-use improve arithmetic, results still vary when subjective criteria and math are combined.其次，模型是基于语言而非算术训练的。当你写下 (影响 × 0.4) + (紧迫性 × 0.35) + (风险 × 0.25) 时，模型并不会像计算器那样可靠地评估该表达式。它根据推理预测可能的词元（token）。有时该预测符合正确的数学逻辑，有时则不然。这些错误通常很小且不一致——这里偏差 0.3，那里四舍五入不同——但它们会与第一个问题叠加。虽然支持工具调用的较新模型改善了算术能力，但在主观标准与数学运算结合时，结果仍然会产生差异。
Third, the model may not use the same strategy each run. On one pass, it might reason through each criterion step by step. On another, it might decide the criteria are subjective and pick a “gut feel” number. On a third, it might write a Python script, execute it, and return that result. You can sometimes see these strategy changes in agent debug logs. They can all happen for the same prompt.第三，模型可能每次运行使用的策略都不一样。在一次运行中，它可能会逐步推理每个标准；在另一次运行中，它可能会认为标准是主观的，并凭“直觉”给出一个数字；在第三次运行中，它可能会编写一段 Python 脚本并执行，然后返回结果。你有时可以在智能体调试日志中看到这些策略的变化。对于同一个提示词，这些情况都可能发生。
The result: scores that are not comparable across runs. You cannot rank Issue A (scored 6.8 on Tuesday) against Issue B (scored 5.2 on Wednesday) because the scoring mechanism was different each time. You cannot debug a single bad score because you cannot reproduce the reasoning that produced it.结果就是：分数在不同运行之间无法比较。你无法将问题 A（周二评分为 6.8）与问题 B（周三评分为 5.2）进行排名，因为每次的评分机制都不同。你无法调试单一的错误评分，因为你无法重现产生该分数的推理过程。
It can’t happen to me这不可能发生在我身上
I’ve had a lot of people push back on this point. They’ll say “I don’t see any problem” or “the scores seem fairly consistent to me.” Granted, sometimes you will get good results – especially with smaller scoring ranges or less ambiguous criteria. Newer frontier models appear to use tools that improve accuracy when reasoning about some kinds of data, including math and dates. They are getting better at these tasks over time.很多人曾就这一点反驳我。他们会说“我没发现任何问题”或“在我看来分数相当一致”。诚然，有时你会得到不错的结果——尤其是在评分范围较小或标准不那么模棱两可的情况下。较新的前沿模型似乎使用了能提高准确性的工具，在处理某些类型的数据（包括数学和日期）时表现更好。随着时间的推移，它们在这些任务上确实在进步。
However, this does not always work. For example, I asked a model to compare some numbers from two different time periods. I prompted it to compare the data for the current week (starting on Monday, ending today) to the same days the previous week. It then reasoned the following:然而，这并不总是有效。例如，我要求模型比较两个不同时间段的一些数字。我提示它比较当前周（从周一开始，到今天结束）与上一周相同天数的数据。随后它进行了如下推理：
See the problem? Don’t feel bad if you don’t – it’s easy to miss.看出问题了吗？如果没看出来也不必介意——这很容易被忽略。
The model needed to know “today”. With a model, that information can only come from one of two places – a tool that provides the details or the model itself (a hallucination). The model correctly identified the current date as July 2, 2026. However, it reasoned (hallucinated) that date was a Wednesday, when it was actually a Thursday – a subtle error that shows how reasoning paths can diverge. That led it to reason that it was comparing three days instead of four. This example illustrates a broader risk: small reasoning variations can compound into different conclusions.模型需要知道“今天”是哪一天。对于模型而言，该信息只能来自两个地方——提供详细信息的工具，或者是模型本身（幻觉）。模型正确识别出当前日期为 2026 年 7 月 2 日。然而，它推理（幻觉）出那天是周三，而实际上那天是周四——这是一个细微的错误，显示了推理路径是如何分叉的。这导致它认为自己是在比较三天而不是四天。这个例子说明了一个更广泛的风险：微小的推理偏差可能会叠加成完全不同的结论。
Next, it reasoned it needed the same three days from the previous week. It picked the correct three dates from the previous week. However, it misidentified them as Monday, Tuesday, and Wednesday instead of Tuesday, Wednesday, and Thursday.接下来，它推理出需要上一周相同的三天。它选对了上一周正确的三个日期。然而，它错误地将其识别为周一、周二和周三，而不是周二、周三和周四。
If you aren’t paying attention to that reasoning, you might not realize that the model is comparing the wrong days and mislabeling the days of the week. In my case, I had criteria that were simple enough to make identifying the right days easy. Now consider the scoring request I described earlier, where the model is asked to reason about impact, urgency, and risk.如果你不关注该推理过程，可能不会意识到模型比较的是错误的天数，并且搞错了星期几。就我而言，我的标准足够简单，很容易识别出正确的天数。现在考虑我之前描述的评分请求，即要求模型推理影响、紧迫性和风险。
The criteria are very subjective. There is nothing specific about how it should define those numbers. It’s not likely to have better results than I had with the date comparison. In fact, the model will likely produce different reasoning chains each time, and those chains will produce different scores.这些标准非常主观。关于如何定义这些数字，并没有具体的规定。它不太可能比我在日期比较中做得更好。事实上，模型很可能会每次产生不同的推理链，而这些链条会产生不同的分数。
What you actually need你真正需要的是什么
When scoring is part of a workflow – triage, prioritization, risk assessment, compliance checks – you need three properties that models cannot provide on their own:当评分成为工作流的一部分（如分类、优先级排序、风险评估、合规性检查）时，你需要模型无法单独提供的三个属性：
- Reproducibility – same inputs must produce the same output可重现性——相同的输入必须产生相同的输出
- Comparability – scores from different runs must use the same calculation可比性——不同运行中的分数必须使用相同的计算方法
- Debuggability – when a score is wrong, you need to trace which rule produced the error可调试性——当分数错误时，你需要追踪是哪条规则导致了错误
These are the properties of deterministic code, not probabilistic content generation.这些是确定性代码的属性，而非概率性内容生成的属性。
The fix: let AI design the logic, then run it as code解决方案：让 AI 设计逻辑，然后作为代码运行
Instead of asking the model to produce a score, ask it to write a scoring function. The model is excellent at translating human criteria into structured logic. It’s also good at helping you to examine edge cases to make that logic more reliable. It is only unreliable when it is asked to perform the logic itself.不要要求模型生成分数，而是要求它编写一个评分函数。模型非常擅长将人类标准转化为结构化逻辑。它也擅长帮助你检查边缘情况，从而使逻辑更可靠。只有在要求它自己执行逻辑时，它才变得不可靠。
Here’s the workflow:工作流如下：
- Describe your scoring criteria to the model in natural language – the same rubric you’d put in a prompt.用自然语言向模型描述你的评分标准——就像你在提示词中写的那样。
- Ask it to generate a script (Python, TypeScript, whatever fits your stack) that takes structured inputs and returns a score. I recommend also providing a
--helpoption that prints the expected inputs and details about what will be returned. If the inputs are not provided, invoke this option automatically.要求它生成一个脚本（Python、TypeScript 或任何适合你技术栈的语言），该脚本接收结构化输入并返回分数。我建议同时提供一个 --help 选项，用于打印预期的输入和返回内容的详细信息。如果未提供输入，则自动调用此选项。 - Review the script. Test it against known inputs. Consider creating unit tests – it’s code!审查脚本。使用已知输入进行测试。考虑创建单元测试——毕竟它是代码！
- Add an instruction or skill that explains how to execute the script and what values to pass.添加一个说明或技能，解释如何执行脚本以及应传递哪些值。
- Run the script from your agent workflow using a hook (deterministic inclusion) or a tool call (model reasoning).使用钩子（确定性包含）或工具调用（模型推理）从你的智能体工作流中运行脚本。
A hook means your workflow always runs the script at a specific point without asking the model to choose. A tool call means the model decides when to run the script based on your instructions.钩子意味着你的工作流总是在特定点运行脚本，而无需模型进行选择。工具调用意味着模型根据你的指令决定何时运行脚本。
The script might look something like this:脚本看起来可能像这样：
1def calculate_impact(issues: list[str]) -> int:
2 """Map an array of issue descriptions to an impact score (1-10)."""
3 # This score calculates the impact by using the number of issues
4 # as a proxy for how many users are affected. More issues = higher impact.
5 # Return 1 for each issue, up to a maximum of 10.
6 return min(len(issues), 10)
7
8# : More functions ...
9
10def score_issue(impact: int, urgency: int, risk: int) -> float:
11 """Score an issue on a 1-10 scale based on weighted criteria."""
12 return round((impact * 0.4) + (urgency * 0.35) + (risk * 0.25), 1)Now every run produces the same output for the same inputs. If the score is wrong, you fix the formula in the script – not your prompt. If the weights need adjusting, you change the code and every future run benefits. The model can still help you refine your criteria. For example, “1=single user; 10=multiple users” is ambiguous. How do you determine impact for 2 users versus 1,000 users? Instead of guessing – and getting different guesses across runs – let the model help define a precise mapping from language to structured inputs. In this example, the model suggested using the number of issues as a proxy for number of users. Alternatively, you might want the model to use an MCP tool to query a ticketing system and count affected customer accounts and licenses. That is why clear criteria matter.现在，每次运行对于相同的输入都会产生相同的输出。如果分数错了，你只需修复脚本中的公式，而不是修改提示词。如果权重需要调整，你只需更改代码，未来的所有运行都将受益。模型仍然可以帮助你完善标准。例如，“1=单个用户；10=多个用户”是模棱两可的。你如何确定 2 个用户与 1,000 个用户的影响？与其猜测——并在不同运行中得到不同的猜测——不如让模型帮助定义从语言到结构化输入的精确映射。在这个例子中，模型建议使用问题数量作为用户数量的代理。或者，你可能希望模型使用 MCP 工具查询工单系统，并统计受影响的客户账户和许可证数量。这就是清晰标准的重要性所在。
Where AI still earns its keepAI 仍然有用的地方
Removing the model from the arithmetic doesn’t mean removing it from the workflow. AI excels at the parts of scoring that are genuinely language-shaped:将模型从算术中移除并不意味着将其从工作流中移除。AI 在那些真正具有语言属性的评分环节表现出色：
- Normalizing free-form input into structured fields. An issue description says “this affects everyone on the enterprise plan.” The model can map that to
impact: 8based on context and tool output about your user base. Models are good at turning natural language into structured data requests.将自由格式的输入规范化为结构化字段。问题描述写着“这影响了企业计划中的所有人”。模型可以根据上下文和关于你用户群的工具输出，将其映射为影响值：8。模型非常擅长将自然语言转化为结构化数据请求。 - Explaining a score after the fact. Given the structured inputs and the formula’s output, the model can write a human-readable summary: “This scored 7.2 primarily because of high urgency (data loss risk) despite limited blast radius. The loss risks were….”事后解释分数。给定结构化输入和公式输出，模型可以编写人类可读的摘要：“此项得分为 7.2，主要是因为尽管影响范围有限，但紧迫性很高（存在数据丢失风险）。丢失风险在于……”。
- Identifying edge cases. The model can review an issue and say “this doesn’t fit cleanly into the existing criteria – the risk isn’t about blast radius, it’s about regulatory exposure.” That’s a signal to extend your scoring function, not to let the model improvise. You can instruct the model to always notify you when it sees a case that doesn’t fit the rubric, so it can then assist you to update the scoring function and keep it deterministic.识别边缘情况。模型可以审查问题并指出“这不符合现有的标准——风险不在于影响范围，而在于监管风险”。这是一个信号，提示你应该扩展评分函数，而不是让模型即兴发挥。你可以指示模型在发现不符合准则的情况时始终通知你，这样它就可以协助你更新评分函数并保持其确定性。
- Validating completeness. Before the script runs, the model can check whether all required fields have been populated and flag anything ambiguous.验证完整性。在脚本运行之前，模型可以检查所有必填字段是否已填充，并标记任何模棱两可的内容。
The key distinction: the model handles reasoning tasks (interpretation, explanation, classification), and deterministic code handles math (formulas, weights, thresholds). Each does what it’s good at.关键区别在于：模型处理推理任务（解释、说明、分类），而确定性代码处理数学（公式、权重、阈值）。各司其职。
Building the pipeline构建流水线
In practice, a reliable scoring workflow has two phases:在实践中，可靠的评分工作流包含两个阶段：
- Extraction (AI-powered): The model reads raw input (issue description, PR body, support ticket) and maps it to the structured fields your scoring function expects. This step is non-deterministic – the model might interpret “critical” differently across runs. But the variance is bounded because the output is constrained to a small set of integers, not a floating-point calculation. You can mix in deterministic logic too, such as tools or scripts that count key words, query a database, or load additional content. The goal is a small, structured set of inputs for the scoring function.提取（由 AI 驱动）：模型读取原始输入（问题描述、PR 正文、支持工单）并将其映射到评分函数预期的结构化字段。这一步是非确定性的——模型在不同运行中对“关键”一词的理解可能不同。但偏差是有限的，因为输出被限制在有限的一组整数中，而不是浮点计算。你也可以混合使用确定性逻辑，例如使用统计关键词、查询数据库或加载额外内容的工具或脚本。目标是为评分函数提供一组小而结构化的输入。
If extraction variance concerns you, add a third step: run extraction multiple times and take the median. You can also have the model explain its reasoning or the data it gathered for each sub-score so a human can sanity-check the inputs before the formula runs.如果你担心提取偏差，可以增加第三步：多次运行提取并取中位数。你也可以让模型解释其推理过程或为每个子分数收集的数据，以便人类在公式运行前对输入进行合理性检查。
- Calculation (deterministic): Your committed script takes those structured fields, applies the formula, and returns the score. Same inputs, same output, every time.计算（确定性）：你提交的脚本接收这些结构化字段，应用公式，并返回分数。相同的输入，每次都产生相同的输出。
This two-phase approach allows each component to do what it does best. The model handles the messy, language-shaped part of the problem and uses natural language to find and organize evidence. The code handles the precise, numeric part using the evidence.这种两阶段方法使每个组件都能发挥其所长。模型处理问题中混乱的、语言化的部分，并利用自然语言查找和组织证据；代码则利用这些证据处理精确的数字部分。
Improving the approach改进方法
If you want to make it even more robust, you can add a third phase: a review of the structured inputs before the formula runs. Checking the inputs helps catch where the model may have made an error or misunderstood data. If you want to ensure the model properly used the function and didn’t try to bypass it, add a hook that verifies a tool called the scoring script. If you want more traceability, add logging to capture the structured inputs and final score for auditing. These additional deterministic steps help keep the scoring process reproducible, comparable, and debuggable.如果你想让它更稳健，可以增加第三阶段：在公式运行前对结构化输入进行审查。检查输入有助于发现模型可能出现的错误或对数据的误解。如果你想确保模型正确使用了函数且没有试图绕过它，可以添加一个钩子来验证工具是否调用了评分脚本。如果你想要更好的可追溯性，可以添加日志记录来捕获结构化输入和最终分数以供审计。这些额外的确定性步骤有助于保持评分过程的可重现性、可比性和可调试性。
You cannot guarantee exactly what AI does in every run. That’s its strength and its challenge. You can, however, guarantee results from what you execute in code. Scoring is the case where this principle is most visible – the numbers make the inconsistency obvious. But the same logic applies anywhere you need reproducible, debuggable outcomes.你无法保证 AI 在每次运行中确切地做什么。这是它的优势，也是它的挑战。然而，你可以保证代码执行的结果。评分是这一原则最明显的案例——数字让不一致性变得显而易见。但同样的逻辑适用于任何你需要可重现、可调试结果的地方。