"System one" decision models are models that infer and respond with calibrated probabilities or every allowed answer.“系统一”决策模型是指那些能够推断并以校准概率或所有允许的答案进行响应的模型。

Consider your everyday language model, to get typed output from it (JSON), you may use Structured Output to constrain the output to guaranteed valid JSON. While model prefills the input in one pass, it still has to go perform a pass for every token in order to generate a valid response.以你日常使用的语言模型为例，要从它那里获得键入的输出（JSON），你可以使用结构化输出（Structured Output）将输出限制为保证有效的 JSON。虽然模型在一次传递中预填充了输入，但为了生成有效的响应，它仍然必须对每个标记（token）执行一次传递。

In this example, 11 passes are required to generate the final output. (We're not accounting for speculative decoding and other inference optimization techniques.)在此示例中，需要 11 次传递才能生成最终输出。（我们未考虑投机解码和其他推理优化技术。）

Decision models such as Jev, make the assumption that there are fixed options we can select from and we can do so quickly by making a single pass. In this example we constrain the set of possible outputs to the options A, B, C, D, E. By masking other items in the vocabulary, the model can only emit those tokens. By selecting the highest probability output, we get our answer.像 Jev 这样的决策模型假设存在我们可以选择的固定选项，并且我们可以通过单次传递快速做出选择。在此示例中，我们将可能的输出集限制为选项 A、B、C、D、E。通过掩盖词汇表中的其他项目，模型只能输出这些标记。通过选择概率最高的输出，我们得到了答案。

Since the outputs are constrained to only a fixed set of options, the model can't select anything outside of those. This however does not guarantee that the output will be correct. It's also common to treat the output token probabilities a confidence scores in this context, but without additional training, those scores likely reflect its confidence in what the next token will be rather than the true probability of the response being the correct answer.由于输出仅限于固定的选项集，模型无法选择这些选项之外的任何内容。然而，这并不能保证输出是正确的。在这种情况下，将输出标记概率视为置信度得分也很常见，但如果没有额外的训练，这些得分很可能反映了模型对下一个标记将是什么的信心，而不是响应是正确答案的真实概率。

Build your own构建你自己的

We can emulate this behavior by constraining output tokens using an LLM. Here I'm using Qwen/Qwen3-1.7B我们可以通过使用大语言模型（LLM）限制输出标记来模拟这种行为。在这里，我使用的是 Qwen/Qwen3-1.7B

import argparse
import json

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "Qwen/Qwen3-1.7B"
options = ["A", "B", "C", "D", "E"]

parser = argparse.ArgumentParser()
parser.add_argument("--input", default="question.json")
args = parser.parse_args()

# load the tokenizer and the model
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)

# the token the model would emit for each option as the first assistant token
option_token_ids = [tokenizer.encode(opt, add_special_tokens=False)[0] for opt in options]


def format_prompt(item):
    prompt = item["question"] + "\n"
    for opt in options:
        prompt += f"{opt}. {item[opt]}\n"
    prompt += "Answer:"
    messages = [
        {"role": "user", "content": prompt}
    ]
    return tokenizer.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True,
        enable_thinking=False
    )


with open(args.input) as f:
    item = json.load(f)

model_inputs = tokenizer(format_prompt(item), return_tensors="pt").to(model.device)
with torch.no_grad():
    logits = model(**model_inputs).logits[0, -1]
# constrained decoding: only the option tokens are allowed
probs = torch.softmax(logits[option_token_ids].float(), dim=-1)

print(f"prediction: {options[probs.argmax().item()]}")
for opt, prob in zip(options, probs.tolist()):
    print(f"{opt}: {prob:.4f}  {item[opt]}")

Running it with an simple question to test it yeilds the following output用一个简单的问题来测试它，得出了以下输出

// input

{
    "question": "What color is the sky?",
    "A": "Red",
    "B": "Blue",
    "C": "Green",
    "D": "Purple",
    "E": "I don't know"
}

// output

prediction: B
A: 0.0000  Red
B: 0.9988  Blue
C: 0.0000  Green
D: 0.0000  Purple
E: 0.0012  I don't know

The model was able to make sense of our input and make a prediction that reasonably corresponds to the correct answer.该模型能够理解我们的输入，并做出与正确答案合理对应的预测。

We can test the accuracy of the model by running it against public datasets. I ran this against a random sample holdout of CommonsenseQA我们可以通过在公共数据集上运行模型来测试其准确性。我将其针对 CommonsenseQA 的随机样本留出集进行了测试

    precision    recall        f1   support
A    0.5733    0.7197    0.6382       239
B    0.5506    0.7686    0.6416       255
C    0.5372    0.6598    0.5922       241
D    0.7206    0.3904    0.5065       251
E    0.7519    0.4255    0.5435       235
accuracy: 725/1221 = 0.5938
macro f1: 0.5844

Not bad for a 1.7B model, Running a quick finetune on the dataset gives us slightly better performance对于一个 1.7B 的模型来说还不错，在数据集上进行快速微调可以获得稍好的性能

    precision    recall        f1   support
A    0.6475    0.6611    0.6542       239
B    0.6113    0.6784    0.6431       255
C    0.6234    0.5975    0.6102       241
D    0.6700    0.5418    0.5991       251
E    0.5808    0.6426    0.6101       235
accuracy: 762/1221 = 0.6241
macro f1: 0.6234

Calibrating your model校准你的模型

Testing the model against a very ambiguous problem demonstrates an interesting problem.针对一个非常模棱两可的问题测试该模型，揭示了一个有趣的问题。

// input
{
    "question": "Where would you most likely find a bat?",
    "A": "Cave",
    "B": "Baseball game",
    "C": "Attic",
    "D": "Zoo",
    "E": "Sporting goods store"
}

// output

prediction: A
A: 0.9978  Cave
B: 0.0004  Baseball game
C: 0.0017  Attic
D: 0.0000  Zoo
E: 0.0001  Sporting goods store

There should be no clear answer here, but treating the output probabilities as a pseudo "confidence" score, shows that the model is extremely overconfident in this answer.这里应该没有明确的答案，但将输出概率视为伪“置信度”得分，显示该模型对这个答案极其过度自信。

If we bin the confidence score ranges in the eval I ran earlier, we can see that the model's confidence does not match its accuracy. This means that the model is not calibrated.如果我们对我之前运行的评估中的置信度得分范围进行分箱，我们可以看到模型的置信度与其准确性不匹配。这意味着该模型没有经过校准。

         bin   count  confidence  accuracy
(0.00, 0.10]       0      0.0000    0.0000
(0.10, 0.20]       0      0.0000    0.0000
(0.20, 0.30]       3      0.2834    0.0000
(0.30, 0.40]      26      0.3761    0.2692
(0.40, 0.50]      41      0.4538    0.2683
(0.50, 0.60]      70      0.5490    0.3286
(0.60, 0.70]      74      0.6476    0.3649
(0.70, 0.80]      77      0.7495    0.4286
(0.80, 0.90]     121      0.8555    0.4711
(0.90, 1.00]     809      0.9855    0.7009

We can notice that the model tends to be extremely overconfident in the 0.9 - 1.0 bin but it's only correct 70% of the time. When it makes a prediction with 0.8 - 0.9 confidence it's only accurate ~40% of the time. This means that the model is generally overconfident in its predictions.我们可以注意到，模型在 0.9 - 1.0 的区间内往往极其过度自信，但它只有 70% 的时间是正确的。当它以 0.8 - 0.9 的置信度进行预测时，其准确率仅为约 40%。这意味着该模型在预测时通常过于自信。

Since our goal is to have the model output scores that is reflective of its accuracy, one method we can use to callibrate it is through temperature scaling. By modifying the temperature value, we can flatten its output probability distribution curve and scale it to approximate its accuracy.由于我们的目标是让模型输出反映其准确性的得分，我们可以使用的一种校准方法是温度缩放（temperature scaling）。通过修改温度值，我们可以展平其输出概率分布曲线，并对其进行缩放以近似其准确性。

Curve fitting fit the temperature parameter to the model's accuracy, I found 3.797280788421631 as a temp value.曲线拟合将温度参数拟合到模型的准确性上，我发现 3.797280788421631 是一个合适的温度值。

         bin   count  confidence  accuracy
(0.00, 0.10]       0      0.0000    0.0000
(0.10, 0.20]       0      0.0000    0.0000
(0.20, 0.30]      82      0.2712    0.2317
(0.30, 0.40]     217      0.3507    0.3917
(0.40, 0.50]     199      0.4472    0.5126
(0.50, 0.60]     166      0.5475    0.5482
(0.60, 0.70]     139      0.6562    0.5827
(0.70, 0.80]     140      0.7492    0.7714
(0.80, 0.90]     169      0.8507    0.7988
(0.90, 1.00]     109      0.9333    0.9541

This gets us a much better calibration. If you want to play around with this, I made a GitHub repo with scripts that walk you through building a dataset, evaluating, finetuning and calibrating your own model. I encourage pulling it and trying it on other bigger models.这使我们获得了更好的校准。如果你想尝试一下，我创建了一个 GitHub 仓库，其中包含引导你构建数据集、评估、微调和校准你自己的模型的脚本。我鼓励你拉取它并在其他更大的模型上进行尝试。