Small Decisions: Engineering a Leading Model微小决策：设计一个领先模型

Or trying to, at least.

My day job has been primarily in AI for three years now, but I’d be the first to admit that’s been almost entirely in one corner of AI: infrastructure, safety, and tools for AI agents. That work has brought me in contact with a lot of the AI science (and I’d dabbled there over the previous decade), but I’m super far from the day-to-day of work like model building. I wanted to catch up a little (after all you have to know what you’re talking about), and the last couple weeks provided a perfect opportunity.我的日常工作主要集中在人工智能领域已有三年了，但我得承认，这几乎完全局限于人工智能的一个角落：基础设施、安全以及人工智能代理的工具。这份工作让我接触到了许多人工智能科学（过去十年我也曾涉猎），但我距离模型构建等工作的日常实践还非常遥远。我想稍微补补课（毕竟你得对自己在谈论的内容有所了解），而过去几周恰好提供了一个绝佳的机会。

On the 15th of this month, the TypeSafe AI folks announced Jev, a kind of general purpose calibrated classifier. You can see this as something exciting or not, but it sure has captured the world’s attention. And mine. I was particularly interested in the calibration, combined with low latency and the ability to answer questions in parallel, it’s a great building block for the more workflowy end of the spectrum of agents.本月 15 日，TypeSafe AI 的团队发布了 Jev，这是一种通用的校准分类器。无论你觉得这是否令人兴奋，它确实吸引了全世界的目光。也包括我的。我对它的校准能力特别感兴趣，再加上低延迟和并行回答问题的能力，它成为了代理领域中更偏向工作流方向的绝佳构建模块。

In an effort to understand these things well, it was time to build my own model: Hobson. A small one, because I wanted to use the GPU I have at home, and because I wanted to see if I could push the bounds on accuracy and calibration at very low latency. I decided to limit myself to about 2 billion parameters.为了透彻理解这些东西，是时候构建我自己的模型了：Hobson。这是一个小模型，因为我想使用家里现有的 GPU，也因为我想看看是否能在极低延迟下突破准确性和校准的极限。我决定将参数量限制在 20 亿左右。

How have I done so far?到目前为止我的表现如何？

Fairly well, I think. You can read that as a trajectory of how versions of my model have performed on accuracy (on x) and calibration (on y) as I’ve made improvements. The pareto optimal is the bottom right.我觉得还不错。你可以将其看作是我在进行改进时，模型版本在准确性（x 轴）和校准（y 轴）方面的表现轨迹。帕累托最优位于右下角。

On the jevbench public set I’m at the top in my size range¹. There’s a new version of decider-2b which beats me, but hasn’t been added to the leaderboard yet. I handily beat Qwen3.5-2B on both calibration and accuracy. I’m measuring calibration with the multi-class Brier score, roughly the mean-square prediction error in a range between 0 (perfect) and 2 (confidently wrong). Perhaps most usefully, Brier on the JevBench easy set is only 0.009 and accuracy is 100%, so we’re very well calibrated for easy tasks.在 JevBench 公共测试集上，我在同尺寸量级中名列前茅。虽然有一个新版本的 decider-2b 超过了我，但它尚未被加入排行榜。在校准和准确性方面，我轻松击败了 Qwen3.5-2B。我使用多类 Brier 分数来衡量校准，这大致相当于 0（完美）到 2（自信但错误）范围内的均方预测误差。或许最有意义的是，在 JevBench 的简易测试集上，Brier 分数仅为 0.009，准确率为 100%，因此我们在处理简单任务时校准得非常好。

How does it work?它是如何工作的？

The core idea is that we take a pre-trained LLM torso (in this case Qwen3.5-2B), and rip off the LM head, and so remove its ability to generate text. The LM head is replaced with a pointer head which scores the answers offered by the torso for each option. It does this by scoring the hidden state at each option position against the hidden state at the <answer> position. This head is pretty small, just over a million total parameters. The torso is fine-tuned with a rank-16 LoRA adapter.核心思想是我们采用一个预训练的 LLM 主干（本例中为 Qwen3.5-2B），拆掉它的语言模型（LM）头，从而移除其生成文本的能力。LM 头被替换为一个指针头，用于对主干为每个选项提供的答案进行评分。它通过将每个选项位置的隐藏状态与 <answer> 位置的隐藏状态进行对比来评分。这个头非常小，总参数仅一百多万。主干则使用 rank-16 的 LoRA 适配器进行微调。

This approach appears to give better calibration than the simple approach of reading the logits at the output of the equivalent size LLM. It’s fairly similar to the approach Kev takes.这种方法似乎比直接读取同等规模 LLM 输出端的 Logits 的简单方法能提供更好的校准效果。它与 Kev 采用的方法相当类似。

My first attempt (inspired by a conversation with a colleague at work, so not original to me) was a slot head which only read the hidden states for each <answer> and passed it through a single linear layer with 24 fixed output slots. This was smaller (53k total parameters), but had some real disadvantages: a limit of 24 options, positional bias (it would learn things like ‘the first option is often right’), and ignored the extra information in each option’s hidden states. I initially experimented with a more complex slot head (2.1M params with a hidden layer), but that approach seemed like a dead end.我的第一次尝试（受同事谈话启发，并非原创）是一个槽位头（slot head），它只读取每个 <answer> 的隐藏状态，并通过一个具有 24 个固定输出槽位的单线性层。这更小（总共 53k 参数），但有一些明显的缺点：限制为 24 个选项、存在位置偏差（它会学到诸如“第一个选项通常是正确的”之类的东西），并且忽略了每个选项隐藏状态中的额外信息。我最初尝试过一个更复杂的槽位头（2.1M 参数，带有一个隐藏层），但那种方法似乎走进了死胡同。

Training训练

The training approach is a fairly standard LoRA fine-tuning, with some self-distillation. The self-distillation was introduced to limit forgetting: the training process tends to make the fine-tuned model forget how to do tasks that it’s already good at. The basic recipe is to use KL between the trainee and a frozen version of the torso (basic distillation), and KL to a previous version of the model on some tasks where I was seeing regressions. The rest is pretty standard: one epoch, cross-entropy to the gold options in the training set, and options shuffled on every example to stop learning positional lessons.训练方法是相当标准的 LoRA 微调，并辅以一些自蒸馏。引入自蒸馏是为了限制遗忘：训练过程往往会使微调后的模型忘记它原本擅长的任务。基本配方是在学员模型和冻结的主干模型之间使用 KL 散度（基础蒸馏），并在我观察到性能倒退的任务上，与模型的前一个版本使用 KL 散度。其余部分都很标准：一个 epoch，对训练集中的正确选项进行交叉熵计算，并且在每个示例中打乱选项顺序，以防止模型学习到位置规律。

The data set is 115,000 rows, about 113k from public datasets, and 2k from synthetic ‘hard’ questions. None of the jevbench set is trained on, and the synthesis process doesn’t know about it either (the model used for synthesis is about six months old). On the other hand, I have seen the jevbench examples, and I designed the synthesis process, so it all comes down to how sub subconsciously intellectually honest I am. Science is hard.数据集包含 115,000 行，其中约 113k 来自公共数据集，2k 来自合成的“困难”问题。JevBench 测试集中的内容均未参与训练，合成过程也不知道这些内容（用于合成的模型大约是六个月前的）。另一方面，我确实看过 JevBench 的示例，而且我设计了合成过程，所以这最终取决于我在智力上有多诚实。科学研究很难。

I expected synthesis to be a big needle mover, by creating hard examples with the right structure. It helped, but wasn’t huge. I suspect there’s a good amount of juice left in that approach.我曾预期合成能通过构建具有正确结构的难题来起到关键作用。它确实有帮助，但作用没那么大。我怀疑这种方法还有很大的潜力可挖。

Again, I’m not an expert, but the training loss graph looks pretty normal. Validation accuracy (not pictured) keeps climbing towards the end, even though loss stalls out. So no huge surprises.再次强调，我不是专家，但训练损失图看起来很正常。验证准确率（未展示）在末尾持续攀升，尽管损失趋于平缓。所以没有太大的意外。

After training, part of the held out data (so data I didn’t train on) is used to calibrate scores. One temperature per question type (binary/noul, choice, score) is calculated that minimizes log loss on that question type (and so ideally improves Brier, but isn’t guaranteed to). At inference time, this temperature is used to scale the confidence scores (by dividing the raw logits by the temperatures).训练后，部分留存数据（即我没有训练过的数据）被用于校准分数。针对每种问题类型（二选一/是非、选择题、评分题）计算出一个温度系数，该系数能最小化该问题类型上的对数损失（从而理想情况下能改善 Brier 分数，但不保证一定能改善）。在推理时，使用此温度系数来缩放置信度分数（通过将原始 Logits 除以温度系数）。

Evaluation评估

The rest of the held-out set is used for evaluation, including some examples from tasks in the training set (i.e. the kind of work is in the training set, but not the individual example), and some tasks entirely held out.剩余的留存集用于评估，其中包括训练集中任务的一些示例（即工作类型在训练集中，但不是具体的示例），以及一些完全未见过的任务。

One of the most important things we learn at this stage is how well the model generalizes. Can it do tasks that it hasn’t seen before? After all, that’s what makes this kind of model interesting versus a custom classifier. The answer is that even at this small size it generalizes usefully, but isn’t great. As I’ve evolved the model, in-task accuracy has been much easier to move than generalization. I suspect this would be much easier with a bigger torso, but the rules of the game don’t allow that approach.我们在这一阶段学到的最重要的事情之一是模型泛化能力的好坏。它能处理之前没见过的任务吗？毕竟，这就是这种模型与定制分类器相比的有趣之处。答案是，即使在如此小的规模下，它也具有有用的泛化能力，但并不出色。随着我不断改进模型，任务内准确率比泛化能力更容易提升。我怀疑如果使用更大的主干会容易得多，但游戏规则不允许那种方法。

The lack of in-task progress here has more to do with my choice of test set than actual performance ceiling. I need to spend more time being more thoughtful about how I test progress.这里任务内进展不足，更多是与我选择的测试集有关，而非实际的性能上限。我需要花更多时间去更深入地思考如何测试进展。

Inference推理

One thing that’s attractive about small models, and about this class of decision models, is low latency and low cost. Inference in my model is either one or two forward passes: one when there’s only one question, and two for any number of questions beyond that (so still O(1), not O(questions)). Quantitative latency scales very well with the number of questions, thanks to the ability to cache the forward pass over the state.小模型以及这类决策模型的一个吸引人之处在于低延迟和低成本。我的模型推理只需一次或两次前向传递：当只有一个问题时为一次，超过该数量的任何问题数均为两次（因此仍然是 O(1)，而不是 O(问题数)）。由于能够缓存状态的前向传递，量化延迟随问题数量的增加而表现良好。

On the jevbench public set, on my 3090, p50 latency is just over 100ms, and p95 latency is less than 300ms. Most of the latency effect is driven by prompt length. Scaling looks super linear, as one might expect given Qwen3.2-2B’s six full attention layers with their quadratic term, but what’s really happening in this range is floor-then-linear and it doesn’t seem like the quadratic term has kicked in yet.在 JevBench 公共测试集上，在我的 3090 上，p50 延迟仅略高于 100ms，p95 延迟低于 300ms。大部分延迟效应是由提示词长度驱动的。考虑到 Qwen3.2-2B 具有六个完整的注意力层及其二次项，扩展看起来是超线性的，但在这个范围内实际发生的是“先平底后线性”，二次项似乎还没有发挥作用。

I suspect there’s a ton of scope to improve latency, mostly the floor. I haven’t worked on it yet, but roughly it seems like the entitlement is closer to 10ms on this hardware, which would bring p50 down substantially. On more modern hardware there’s also likely a significant gain available on the slope, but haven’t benchmarked that either (don’t tempt me to buy a 5090).我怀疑在改善延迟方面还有很大空间，主要是底限延迟。我还没着手处理，但粗略估计，在此硬件上其潜力更接近 10ms，这将使 p50 大幅下降。在更现代的硬件上，斜率方面也可能获得显著增益，但我还没有进行基准测试（别诱惑我去买 5090）。

What’s Next?接下来做什么？

There are a few things I want to try. Starting with more data synthesis, especially of harder problems. I think we’re not yet close to tapped out on capabilities with this number of parameters. The other big one is some form of reinforcement learning, mostly seeing if that can help calibration and generalization, especially on end-to-end decision utility (e.g. with a workflow that ‘does the thing if confidence >0.9’, which can’t be differentiated). Smaller ones include trying a few architectural tweaks, evaluating a second epoch or partial epoch, evaluating some different training schedules, larger LoRA ranks, and experimenting with other torsos (I tried instruct variants early on with negative results, but I’m not sold on that yet).有几件事我想尝试。首先是更多的数据合成，特别是针对更难的问题。我认为在目前的参数量下，我们远未挖掘出其全部能力。另一个重点是某种形式的强化学习，主要是观察它能否有助于校准和泛化，特别是在端到端的决策效用方面（例如，使用“如果置信度 >0.9 则执行操作”的工作流，这无法求导）。较小的尝试包括进行一些架构调整、评估第二个 epoch 或部分 epoch、评估一些不同的训练计划、更大的 LoRA rank，以及尝试其他主干（我早期尝试过指令微调变体，结果不佳，但我对此尚未定论）。

Some of the Development Steps部分开发步骤

• v2 scaled the torso from Qwen3-1.7B to Qwen3-4B (before I set my 2B goal). Actually made performance worse, because the hold out set didn’t have the right kind of tasks to see if it was better. Who among us hasn’t come to the wrong conclusion from a bad benchmark?v2 将主干从 Qwen3-1.7B 扩展到 Qwen3-4B（在我设定 2B 目标之前）。实际上性能反而变差了，因为留存集没有正确类型的任务来验证它是否更好。我们之中谁没有因为糟糕的基准测试而得出错误的结论呢？
• v6 was where I introduced the distillation technique, moving accuracy slightly.v6 版本是我引入蒸馏技术的地方，准确率略有提升。
• v7 started to feel like I was going somewhere. I switched here from the slot head to the pointer head, and that was our single biggest win so far.v7 版本让我感觉有了一些进展。我在这里将槽位头切换为指针头，这是我们迄今为止最大的单项胜利。
• v8 through v11 was a series of failed experiments: switching to Qwen3-1.7B-instruct, closed-form programmatic synthesis, playing with option order.v8 到 v11 是一系列失败的实验：切换到 Qwen3-1.7B-instruct、闭式程序化合成、尝试改变选项顺序。
• v12 Switching to Qwen3-8B improved performance a lot (and matched SemIf’s performance at this size), but I decided not to follow that path.v12 切换到 Qwen3-8B 后性能提升了很多（并达到了 SemIf 在此规模下的性能），但我决定不走那条路。
• v13 brought us back to the gold path, by switching to Qwen3.5-2B. I’d resisted that because it meant I needed to throw out some earlier work on optimizing inference, but it moved the needle significantly.v13 通过切换到 Qwen3.5-2B 将我们带回了正轨。我之前一直抗拒这一点，因为这意味着我必须丢弃一些早期关于优化推理的工作，但它确实带来了显著的改进。
• v14, v16, and v18 expanded the training corpus substantially, with public data sets (ContractNLI, BoardgameQA, MuSiQue), and LLM-generated examples (Qwen3.5-27B) validated by an even bigger model (Qwen3.5-397B). This is fundamentally a data game, and these were also big wins.v14、v16 和 v18 大幅扩展了训练语料库，使用了公共数据集（ContractNLI, BoardgameQA, MuSiQue）以及由更大模型（Qwen3.5-397B）验证的 LLM 生成示例（Qwen3.5-27B）。这从根本上是一场数据游戏，这些也是巨大的胜利。
• v17 was another change to training: to stop going backwards on performance on multi-step tasks, use v14 as the teacher for some examples rather than Qwen3.5-4B. decider-2B does something similar (replay toward a parent rather than the base teacher). I don’t love it, but it seems to work a little bit.v17 是训练的又一次改变：为了停止在多步任务上的性能倒退，在某些示例中使用 v14 作为教师模型，而不是 Qwen3.5-4B。decider-2B 也做了类似的事情（向父模型而非基础教师模型回放）。我不太喜欢这样，但它似乎确实有一点效果。

What worked: the head rearchitecture, a more modern and slightly bigger torso, document data with a teacher or verifier. What didn’t: templated data synthesis, distillation on small/easy problems, some training data additions (notably ShARC and ConditionalQA). I think this is compatible with what Zhang et al found about fine tuning: more base params, more types of problems (different skills). More of the same data doesn’t help a lot.有效的方法：头部的重构、更现代且稍大的主干、带有教师或验证器的文档数据。无效的方法：模板化数据合成、在简单/容易问题上的蒸馏、一些训练数据的添加（特别是 ShARC 和 ConditionalQA）。我认为这与 Zhang 等人关于微调的发现是一致的：更多的基础参数、更多类型的问题（不同的技能）。更多重复的数据并没有太大帮助。

Lessons经验教训

Maybe the biggest lesson here is how much easier it is to learn this stuff now than a year or so ago. Being able to ask Kiro or Claude to step me through concepts and then quiz me on my understanding was exceptionally helpful - it’s like having a custom textbook about exactly this problem at just the right level. Every line of code was written by an agent, but at each step I tried to make sure the core ideas and insights were mine, or at least I understood them. I might not set such a bar for a project at work, but for this project the outcome was mostly about me learning.也许这里最大的教训是，与一年前左右相比，现在学习这些东西容易多了。能够让 Kiro 或 Claude 为我梳理概念，然后测试我对知识的理解，这非常有帮助——就像拥有一本完全针对这个问题、且难度恰到好处的定制教科书。每一行代码都是由代理编写的，但在每一步中，我都尽量确保核心思想和见解是我自己的，或者至少是我理解的。对于工作中的项目，我可能不会设定这样的标准，但对于这个项目，结果主要在于我的学习。

I think I’ll need to do this a few times before all the new concepts stick. I’m not yet at the point I could stand at a white board and walk through each decision (especially at the algebra level), but I’m way further along that path than a week ago.我想我需要多做几次，才能完全掌握这些新概念。我还没有达到可以站在白板前讲解每一个决策（尤其是代数层面的）的地步，但我比一周前已经在该道路上走得远多了。

Even at 2B and below, we can build useful models of this class. That’s obvious from the JevBench website too, but getting hands-on has really helped calibrate my thinking about this problem.即使在 2B 及以下规模，我们也能构建这类有用的模型。从 JevBench 网站上也显而易见，但亲自动手确实帮助我校准了对这个问题的思考。

Finally, while this was fun, it showed how easy it is to get obsessed with this number go up model building game. People who had a bit of a, ah, problem with World of Warcraft or Diablo II should probably find another way to spend their time.最后，虽然这很有趣，但也显示出沉迷于这种“数值提升”的模型构建游戏是多么容易。那些在《魔兽世界》或《暗黑破坏神 II》上遇到过，呃，问题的人，可能应该找点别的事情做。

Footnotes脚注

1. joint first of 30 at 2B or below on the v1.4.2 board, level with decider-2b’s v10 entry. There are two slightly larger models, around 2.5B, that do beat my model. I think I was legitimately in the lead for models around 2B for a while, maybe 24h. Interesting times.在 v1.4.2 排行榜上，2B 或以下规模的 30 个模型中并列第一，与 decider-2b 的 v10 条目持平。有两个稍大的模型（约 2.5B）确实超过了我的模型。我想我确实在 2B 左右的模型中领先了一段时间，也许是 24 小时。有趣的时代。

Similar Posts

• 29 Jul 2026 » Lorenz and Little: How Much Does Your Tail Cost?
• 19 Jul 2026 » Aurora DSQL: Scalable, Multi-Region OLTP
• 19 Jun 2026 » Meet Alice. Alice is impatient.