Jev and System One Models: Calibration Beats AccuracyJev 与“系统一”模型：校准优于准确度

Last week TypeSafe AI released Jev, which it calls the first “System One model”: a model that does not chat, does not write, and does not reason step by step. It answers structured questions about an input, in a single forward pass, with a probability attached to every answer. Most of the coverage has focused on speed. I think the more interesting claim is the one about calibration, because calibration is the thing that has quietly limited every production classifier I have shipped, including the one in my COMPSAC paper.上周，TypeSafe AI 发布了 Jev，并将其称为首个“系统一”模型：它不会聊天、不会写作，也不会进行分步推理。它通过单次前向传播，针对输入回答结构化问题，并为每个答案附上概率。大多数报道都聚焦于其速度。我认为关于“校准”的说法更为有趣，因为校准一直是我所部署的每个生产环境分类器中默默受限的瓶颈，包括我在 COMPSAC 论文中所使用的那个。

This post is my attempt to work out what Jev actually changes, where it fits in a real ML stack, and how I intend to test the claim rather than take it on faith.本文旨在探讨 Jev 究竟带来了哪些改变、它在实际机器学习技术栈中的位置，以及我打算如何验证其声明，而非盲目相信。

What Jev is, without the marketing#Link to section: What Jev is, without the marketing剥离营销包装后的 Jev 是什么

Jev is built around three ideas, per TypeSafe’s launch post:根据 TypeSafe 的发布文章，Jev 基于三个核心理念：

1. Non-autoregressive output. A normal LLM produces its answer one token at a time, and each token depends on the last. Jev emits the entire structured answer at once. That is where the speed comes from: TypeSafe quotes 70–500 ms end to end and “40x–200x faster” than frontier LLMs on equivalent tasks.¹非自回归输出。普通的 LLM（大语言模型）逐个 token 生成答案，且每个 token 都依赖于前一个。Jev 则一次性输出完整的结构化答案。这就是其速度的来源：TypeSafe 称其端到端延迟为 70–500 毫秒，在同类任务上比前沿 LLM “快 40 到 200 倍”。
2. Typed questions, not prompts. You send a state (text, structured data, or a message history) and a set of questions. Each question is one of three types: choice (pick from a set, get a probability per option), score (rate against ordered levels, get a continuous score and distribution), or noul (a yes/no, returned as the probability the statement is true).² Every question in a request is evaluated in parallel, so adding questions barely changes latency.类型化问题，而非提示词。你发送一个状态（文本、结构化数据或消息历史）和一组问题。每个问题属于以下三种类型之一：选择题（从集合中选择，获取每个选项的概率）、评分题（根据有序级别进行评级，获取连续的分数和分布），或 noul（是非题，返回陈述为真的概率）。请求中的每个问题都是并行评估的，因此增加问题数量几乎不会改变延迟。
3. Training for calibration. The model is trained with what TypeSafe calls reinforcement learning for calibrated decisions (RLCD). The stated goal is “epistemically honest probabilities” rather than the human-preference or verifiable-reward objectives that chat models are tuned on.¹针对校准进行训练。该模型采用了 TypeSafe 所谓的“用于校准决策的强化学习”（RLCD）进行训练。其既定目标是“认知上的诚实概率”，而非聊天模型所采用的人类偏好或可验证奖励目标。

The constraints are just as important as the features. Jev cannot generate free text. A choice question supports at most 255 options. There is no image input yet. Pricing is $0.042 per million input tokens with output tokens free, and access is currently by waitlist.¹其约束条件与功能同样重要。Jev 无法生成自由文本。选择题最多支持 255 个选项。目前尚不支持图像输入。定价为每百万输入 token 0.042 美元，输出 token 免费，目前通过候补名单开放访问。

So it is not a smaller GPT. It is closer to a very fast, very general tabular classifier that reads unstructured input and returns a typed decision with a confidence you are meant to be able to trust.因此，它不是一个更小的 GPT。它更像是一个非常快速、非常通用的表格分类器，能够读取非结构化输入并返回一个类型化的决策，且该决策的置信度是值得信赖的。

Why calibration, not accuracy, is the real bottleneck#Link to section: Why calibration, not accuracy, is the real bottleneck为什么校准而非准确度才是真正的瓶颈

Here is the part of my own paper I keep coming back to. We predicted whether a pull request would be merged, using only signals available at submission time. Random Forest hit an F1 of 0.958. The majority-class baseline, which says “merged” to everything, hit 0.957. The number that actually separated a useful model from a useless one was ROC-AUC: 0.676 for the forest versus 0.500 for the baseline. And even at that, we wrote plainly that the models “should not be treated as perfectly calibrated probability models” and were fit for triage, not for automated accept/reject decisions.³这里是我论文中不断回顾的部分。我们仅利用提交时可用的信号，预测拉取请求（PR）是否会被合并。随机森林（Random Forest）的 F1 分数达到了 0.958。而将所有内容都标记为“已合并”的多数类基准模型达到了 0.957。真正区分有用模型与无用模型的指标是 ROC-AUC：随机森林为 0.676，而基准模型为 0.500。即便如此，我们也明确指出，这些模型“不应被视为完全校准的概率模型”，它们仅适用于分类筛选，而非自动化的接受/拒绝决策。

That is not a quirk of one dataset. It is the normal shape of a production classifier:这并非某个数据集的特例，而是生产环境分类器的常态：

• Accuracy saturates early. On imbalanced problems, most of the available accuracy is free. The hard part is the ranking and the confidence.准确度很快就会达到饱和。在不平衡问题上，大部分可获得的准确度是轻而易举的。难点在于排序和置信度。
• Downstream logic needs probabilities, not labels. “Route this order to manual review if the model is less than 80% sure” only works if 80% means 80%. If the model says 0.95 on things that are right 70% of the time, every threshold you set is a lie.下游逻辑需要的是概率，而非标签。“如果模型置信度低于 80%，则将此订单转入人工审核”这一规则，只有在 80% 确实代表 80% 的情况下才有效。如果模型对那些只有 70% 正确率的事物给出了 0.95 的置信度，那么你设置的每一个阈值都是谎言。
• Miscalibration is invisible in the usual metrics. F1, accuracy, even AUC are all threshold or rank metrics. A model can have a fine AUC and terrible calibration, and you will not know until the business rule built on top of it starts misfiring.模型校准偏差在常规指标中是不可见的。F1、准确度，甚至 AUC 都是基于阈值或排序的指标。一个模型可能拥有不错的 AUC 但校准极差，而在基于它构建的业务规则开始失效之前，你根本不会察觉。

The standard fixes are post-hoc: Platt scaling, isotonic regression, temperature scaling. They work, but they are another fitted component that drifts when the data does. What Jev is claiming, if I read it correctly, is that the probabilities come out of the model already honest, because honesty was the training objective. If that holds on tasks outside TypeSafe’s own benchmarks, it removes a whole layer of glue from production ML systems.标准的修复方法是事后处理：Platt 缩放、保序回归、温度缩放。它们确实有效，但属于额外的拟合组件，当数据发生漂移时，这些组件也会失效。如果我没理解错的话，Jev 声称其概率输出本身就是诚实的，因为诚实是其训练目标。如果这在 TypeSafe 自身的基准测试之外的任务中依然成立，那么它将从生产环境机器学习系统中消除整整一层“胶水”代码。

That “if” is the entire question, and it is testable.这个“如果”是问题的关键，而且它是可测试的。

Where a System One model fits in a real stack#Link to section: Where a System One model fits in a real stack“系统一”模型在实际技术栈中的位置

I work on ML inside a wholesale distribution business. Almost none of it is chat. Most of it is small, repeated decisions that sit between two systems:我在一家批发分销企业从事机器学习工作。其中几乎没有聊天任务。大部分工作是介于两个系统之间、小而重复的决策：

The pattern is clear. Anywhere I have an LLM doing a job that is really classification wearing a chat costume, a System One model is a plausible replacement with two orders of magnitude less latency and cost. Anywhere I have hand-written rules that keep breaking because the input is free text, it is a plausible replacement for the rules. Anywhere the job is generation or optimization, it is the wrong tool and TypeSafe says so themselves.模式很清晰。任何我使用 LLM 处理实际上是分类任务（却披着聊天外衣）的地方，系统一模型都是一个可行的替代方案，其延迟和成本可以降低两个数量级。任何我因输入是自由文本而导致手工编写的规则不断失效的地方，它都是这些规则的可行替代品。而对于生成或优化类任务，它显然不是正确的工具，TypeSafe 自己也承认了这一点。

The ERP integration story is also attractive. A model that returns {"is_exception": 0.93} in 100 ms can sit inside a request path. An LLM that returns a paragraph in four seconds has to sit beside it in a queue. That difference decides whether ML is a feature or a batch job.ERP 集成场景也很有吸引力。一个能在 100 毫秒内返回 {"is_exception": 0.93} 的模型可以嵌入请求路径中。而一个需要四秒钟才能返回一段文字的 LLM，则必须在队列中排队等待。这种差异决定了机器学习究竟是一个功能特性，还是一个批处理作业。

The claims I am not ready to accept yet#Link to section: The claims I am not ready to accept yet我还无法接受的声明

A few things in the launch material deserve a skeptical reading.发布材料中的几点内容值得持怀疑态度。

“Zero hallucination.” What TypeSafe can guarantee is that the output type is always valid: you asked for one of five categories, you get one of five categories, with probabilities that sum to one. That is real and useful, and LLM structured-output modes only approximate it. But it says nothing about whether the chosen category is right. A confidently wrong answer in a valid schema is still a wrong answer. The honest framing is “zero schema errors,” and calibration is what has to cover the rest.“零幻觉”。TypeSafe 能保证的是输出类型始终有效：你要求五个类别中的一个，你就会得到其中一个，且概率之和为一。这是真实且有用的，LLM 的结构化输出模式只能近似实现这一点。但这并不能说明所选类别是否正确。在有效的架构中给出一个自信的错误答案，依然是错误的答案。诚实的表述应该是“零架构错误”，而校准正是为了解决其余问题。

Calibration on whose distribution? A model can be well calibrated on its training and benchmark distribution and drift badly on yours. Calibration is a property of a model and a dataset. The only number I will trust is one measured on my data.在谁的数据分布上进行校准？一个模型可以在其训练和基准测试分布上校准良好，但在你的数据上出现严重漂移。校准是模型与数据集共同的属性。我唯一信任的数字，是在我的数据上测量出来的数字。

The comparison baseline. “200x faster than an LLM on classification” is true and also a bit unfair, because the right baseline for many of these tasks is not an LLM. It is a gradient-boosted tree on engineered features, which is also sub-millisecond and free. The interesting comparison is three-way: classical tabular model, LLM-as-classifier, and Jev, on the same task, on accuracy, ranking, calibration, latency and cost.比较基准。“比分类任务中的 LLM 快 200 倍”是事实，但也有些不公平，因为对于许多此类任务，正确的基准不是 LLM。而是基于工程特征的梯度提升树（Gradient-boosted tree），它同样是亚毫秒级且免费的。有趣的比较应该是三者对比：经典表格模型、作为分类器的 LLM 以及 Jev，在同一任务上，对比准确度、排序、校准、延迟和成本。

The experiment I want to run#Link to section: The experiment I want to run我想进行的实验

I have exactly the right testbed already built: the PR acceptance pipeline from my paper. It is leakage-aware, it has fixed 5-fold splits, and it has a published tree-model baseline with a known calibration weakness. Here is the design.我已经构建好了完美的测试平台：我论文中的 PR 接受流水线。它具有防止信息泄露的机制，拥有固定的 5 折交叉验证划分，并发布了具有已知校准缺陷的树模型基准。设计如下。

Task. Same as RQ1 in the paper: given a PR at submission time, predict merged vs. closed without merge. The Jev state will be the PR title, body, and the same submission-time metadata and diff statistics the trees see, serialized as text. Nothing that appears after submission (comments, CI, later commits) goes into the state. The leakage rules do not relax because the model is new.任务。与论文中的 RQ1 相同：给定一个提交时的 PR，预测其是被合并还是被关闭。Jev 的状态将是 PR 标题、正文，以及树模型所看到的相同的提交时元数据和 diff 统计信息，并序列化为文本。任何提交之后出现的内容（评论、CI、后续提交）都不会进入状态。即便模型是新的，防止泄露的规则也不会放宽。

Questions. One noul: “This pull request will be merged.” Optionally one choice over the task-intent tags (fix, feature, refactor, docs) to see whether Jev’s own reading of intent agrees with our keyword rules.问题。一个 noul 问题：“此拉取请求将被合并。”可选地对任务意图标签（修复、功能、重构、文档）进行选择题测试，以观察 Jev 对意图的理解是否与我们的关键词规则一致。

Baselines. The paper’s Random Forest (400 trees), the same forest with isotonic calibration fitted in-fold, and a frontier LLM asked the same question with structured output.基准。论文中的随机森林（400 棵树）、在折内拟合了保序回归的相同森林，以及被要求以结构化输出回答相同问题的前沿 LLM。

Metrics. Ranking and calibration, not just F1:指标。排序和校准，而不仅仅是 F1：

• ROC-AUC, so the result is comparable to the paper.ROC-AUC，以便结果可与论文进行比较。
• Brier score, the mean squared error of the probability against the outcome:Brier 分数，即概率与结果之间的均方误差：

$$\text{Brier} = \frac{1}{N}\sum_{i=1}^{N}\left(\hat{p}_i - y_i\right)^2$$Brier=1N∑i=1N(p^i−yi)2

• Expected calibration error, binning predictions by confidence and measuring how far each bin’s accuracy is from its stated confidence:预期校准误差（ECE），通过将预测按置信度分桶，并测量每个分桶的准确度与其所述置信度之间的距离：

$$\text{ECE} = \sum_{b=1}^{B} \frac{|S_b|}{N}\,\Big|\,\text{acc}(S_b) - \text{conf}(S_b)\,\Big|$$ECE=∑b=1B∣Sb∣N ∣ acc(Sb)−conf(Sb) ∣

• A reliability diagram per model, because a single ECE number hides where a model is over- or under-confident.每个模型的可靠性图表，因为单一的 ECE 数字会掩盖模型在哪些地方过度自信或不足自信。
• Median and p95 latency, and cost per 1,000 PRs.中位数和 p95 延迟，以及每 1000 个 PR 的成本。

What would change my mind. If Jev matches the forest’s AUC and beats the calibrated forest on Brier and ECE, without any post-hoc fitting, then the calibration claim is real on a distribution TypeSafe never saw, and I would start moving classification-shaped LLM calls at work onto it. If it beats the uncalibrated forest but not the calibrated one, then it is a convenience, not a capability. If its AUC is materially lower, the speed does not matter.什么会改变我的看法。如果 Jev 的 AUC 与森林模型相当，且在 Brier 和 ECE 指标上击败了经过校准的森林模型，且无需任何事后拟合，那么其校准声明在 TypeSafe 从未见过的数据分布上就是真实的，我将开始在工作中将分类任务的 LLM 调用迁移到它上面。如果它击败了未校准的森林模型但没能击败校准后的模型，那么它只是一种便利，而非能力提升。如果其 AUC 明显更低，那么速度就无关紧要了。

I will publish the numbers either way, and I will link them from here.无论结果如何，我都会公布数据，并在此处提供链接。

What I would tell a team today#Link to section: What I would tell a team today我现在会给团队的建议

If you are deciding whether to care about Jev right now, my advice is:如果你正在决定现在是否要关注 Jev，我的建议是：

1. Inventory your LLM calls. Tag each one as generate or decide. The decide ones are candidates. In my experience that is most of them.盘点你的 LLM 调用。将每一个标记为“生成”或“决策”。决策类任务就是候选对象。根据我的经验，大部分调用都属于此类。
2. Measure calibration on what you already have. Compute Brier and ECE for your current classifiers. If they are bad, you have a problem Jev might solve. If they are fine, you mostly have a latency and cost question.测量你现有系统的校准情况。为你当前的分类器计算 Brier 和 ECE。如果表现很差，那么你遇到了一个 Jev 可能解决的问题。如果表现良好，那你主要面临的就是延迟和成本问题。
3. Do not skip the classical baseline. A gradient-boosted tree on decent features is the bar. Any new model has to beat it on your data, with your leakage rules, or it is not an upgrade.不要跳过经典基准测试。基于良好特征的梯度提升树是衡量标准。任何新模型都必须在你的数据、你的泄露规则下击败它，否则它就不是升级。
4. Treat “calibrated” as a hypothesis. Test it on your distribution before a business rule depends on it.将“已校准”视为一个假设。在业务规则依赖它之前，先在你的数据分布上进行测试。

The idea behind System One models is sound: most of the decisions software needs from ML are small, structured, and latency-sensitive, and a chat model is a strange tool for them. Whether Jev delivers on the calibration promise is an empirical question. I have the dataset to answer it, and I intend to.系统一模型的理念是合理的：软件从机器学习中获取的大多数决策都是小规模、结构化且对延迟敏感的，而聊天模型对于它们来说是一个奇怪的工具。Jev 是否能兑现其校准承诺是一个经验性问题。我有数据集来回答这个问题，我也打算这样做。

Further reading#Link to section: Further reading延伸阅读

• Introducing System One Models & Jev, TypeSafe AIIntroducing System One Models & Jev, TypeSafe AI
• Building a harness with Jev, LangChain, for the request/response shapeBuilding a harness with Jev, LangChain, 关于请求/响应格式
• Predicting Pull Request Acceptance the Moment It’s Opened, my write-up of the COMPSAC paper this experiment builds onPredicting Pull Request Acceptance the Moment It’s Opened，我关于本实验所基于的 COMPSAC 论文的撰写
• Guo et al., On Calibration of Modern Neural Networks, ICML 2017, the standard reference for ECE and temperature scalingGuo et al., On Calibration of Modern Neural Networks, ICML 2017，关于 ECE 和温度缩放的标准参考资料

If you have Jev access and a labeled classification dataset with a known calibration problem, I would like to compare notes. My contact details are on the homepage.如果你有 Jev 的访问权限，并且拥有一个存在已知校准问题的标记分类数据集，我很乐意交流心得。我的联系方式在主页上。

Footnotes#Link to section: Footnotes脚注

1. TypeSafe AI, Introducing System One Models & Jev, September 2026. Latency, speedup, pricing, cardinality and modality limits are quoted from that post and are the vendor’s claims. ↩ ↩² ↩³TypeSafe AI, Introducing System One Models & Jev, 2026 年 9 月。延迟、加速比、定价、基数和模态限制均引用自该文章，为供应商声明。

2. LangChain, Building a harness with Jev, September 2026, which documents the state/questions request shape and the choice, score and noul question types. ↩LangChain, Building a harness with Jev, 2026 年 9 月，记录了状态/问题请求格式以及选择、评分和 noul 问题类型。

3. K. Pansuriya, E. Ghorbani, D. Singh, E. A. AlOmar. Predicting Acceptance and Review Effort in Human and Agent Pull Requests. IEEE COMPSAC 2026. arXiv:2607.12057. Table II reports RF F1 0.958 / AUC 0.676 and the majority baseline F1 0.957 / AUC 0.500. ↩K. Pansuriya, E. Ghorbani, D. Singh, E. A. AlOmar. Predicting Acceptance and Review Effort in Human and Agent Pull Requests. IEEE COMPSAC 2026. arXiv:2607.12057. 表 II 报告了 RF F1 0.958 / AUC 0.676 以及多数类基准 F1 0.957 / AUC 0.500。