Knowledge cutoff is a poor proxy for model capability

A model’s knowledge cutoff marks the latest date its training data could cover. But its actual knowledge of a product might end years earlier or be absent entirely. I learned this while trying to answer what seemed like a straightforward question: how much does a model know about a product? If the model’s knowledge ended around a particular date, I should have been able to translate that date to a product version.模型的知识截止日期标志着其训练数据所能涵盖的最晚日期。但它对某款产品的实际了解可能早在几年前就已停止，甚至完全不了解。我在试图回答一个看似简单的问题时发现了这一点：模型对一款产品到底了解多少？如果模型的知识在某个特定日期左右截止，我应该能够将该日期转换为产品版本。

The model knows the product up to roughly version X. Anything introduced after that needs additional context.该模型对产品的了解大致截止于 X 版本。在此之后引入的任何内容都需要额外的上下文。

On paper, that would be useful when building agent extensions. You could establish what the model already knows and supply the missing documentation, saving the context window for the work. In practice, the results showed no consistent boundary.理论上，在构建智能体扩展时，这会很有用。你可以确定模型已经掌握了哪些知识，并提供缺失的文档，从而为实际工作节省上下文窗口。但在实践中，结果显示并没有一致的界限。

Looking for the line Copy link寻找界限 复制链接

To test it, I used GPT-5.6 Luna with the GitHub Copilot SDK against two developer products: Dev Proxy and SharePoint Framework (SPFx). I fed the changelogs and release notes for both products to GPT-5.6 Sol and asked it to extract product changes, excluding things like dependency updates and CI infrastructure work. I then asked a model to classify whether each change was suitable for evaluation, based on whether it introduced enough difference from the previous behavior to test.为了进行测试，我使用 GPT-5.6 Luna 结合 GitHub Copilot SDK 对两款开发者产品进行了测试：Dev Proxy 和 SharePoint Framework (SPFx)。我将这两款产品的更新日志和发布说明提供给 GPT-5.6 Sol，并要求它提取产品变更，排除依赖项更新和 CI 基础设施工作等内容。然后，我要求模型根据每次变更是否与之前的行为有足够的差异以供测试，来评估该变更是否适合进行评估。

For each suitable change, a model generated a task prompt and an evaluation rubric. I ran the tasks in Vally with GPT-5.6 Luna as the model under test. GPT-5.6 Terra judged each result against its rubric.对于每一个合适的变更，模型都会生成一个任务提示词和评估准则。我在 Vally 中运行这些任务，并将 GPT-5.6 Luna 作为被测模型。GPT-5.6 Terra 根据准则对每个结果进行了评判。

Measuring internal model knowledge requires a strict information boundary. As I covered in Your AI coding agent evaluation is only as good as its sandbox, even a local product installation can give the agent access to the answer and invalidate the measurement. For this experiment, I removed access to external information, including documentation and web search.衡量模型的内部知识需要严格的信息边界。正如我在《你的 AI 编码智能体评估取决于其沙盒的质量》一文中所述，即使是本地产品安装也可能让智能体获取到答案，从而使测量结果失效。在此次实验中，我移除了对外部信息的访问权限，包括文档和网络搜索。

OpenAI lists GPT-5.6 Luna’s knowledge cutoff as February 16, 2026. Dev Proxy 2.1.0, released on January 28, was the latest stable release available by then. Yet when I asked the model where its product knowledge ended, it placed the boundary around Dev Proxy 0.29.0 and SPFx 1.21.1. Dev Proxy 0.29.0 had shipped on June 25, 2025, almost eight months before the model’s official cutoff. This was already evidence that a date cutoff and the model’s knowledge of a specific product are different boundaries.OpenAI 列出的 GPT-5.6 Luna 的知识截止日期为 2026 年 2 月 16 日。1 月 28 日发布的 Dev Proxy 2.1.0 是当时最新的稳定版本。然而，当我询问模型其产品知识的截止点在哪里时，它将界限设定在 Dev Proxy 0.29.0 和 SPFx 1.21.1 左右。Dev Proxy 0.29.0 发布于 2025 年 6 月 25 日，比该模型的官方截止日期早了近八个月。这已经证明了日期截止和模型对特定产品的了解是两个不同的界限。

If the model’s product estimates gave us a useful boundary, the results should still have had a recognizable shape. Older releases would perform relatively well and performance would deteriorate as the tasks approached those versions, then newer releases would consistently fail.如果模型的产品估算能为我们提供一个有用的界限，那么结果应该呈现出可识别的规律。较旧的版本表现应该相对较好，随着任务接近这些版本，性能会下降，然后较新的版本应该会持续失败。

Instead, I got noise. Across 336 Dev Proxy tasks spanning 53 versions, the model passed 61, or 18%. Dev Proxy 0.3.0 passed four of five tested tasks, while 0.4.0 passed none. Much later, versions 2.3.4, 3.0.0 and 3.1.0 each passed one of two, followed again by releases with no passing tasks.然而，我得到的是杂乱无章的数据。在跨越 53 个版本的 336 个 Dev Proxy 任务中，模型通过了 61 个，占比 18%。Dev Proxy 0.3.0 在 5 个测试任务中通过了 4 个，而 0.4.0 一个都没通过。很久之后，2.3.4、3.0.0 和 3.1.0 版本各通过了两个任务中的一个，随后又是没有任何通过任务的版本。

Meanwhile, SPFx looked remarkably similar. Across 413 tasks covering 40 versions, the model passed 61, or 15%. Correct answers and failures appeared throughout the product history, with no clean transition at any version.与此同时，SPFx 的情况也非常相似。在涵盖 40 个版本的 413 个任务中，模型通过了 61 个，占比 15%。正确答案和失败案例贯穿于整个产品历史，在任何版本都没有出现清晰的过渡。

A model can fail before its cutoff Copy link模型可能在截止日期之前就已失效 复制链接

A knowledge cutoff sets a ceiling. It tells us the latest knowledge the model could have encountered during training, without telling us how well any particular product is represented. Its knowledge of that product might stop two years earlier or be absent entirely. Even when the knowledge is present, the model might struggle to recall and apply it when needed.知识截止日期设定了一个上限。它告诉我们模型在训练过程中可能接触到的最新知识，但并没有告诉我们特定产品被呈现得有多好。它对该产品的知识可能在两年前就已停止，或者根本不存在。即使知识存在，模型在需要时也可能难以回忆和应用这些知识。

The SPFx results showed this clearly. One task for SPFx 1.11.0 asked about using a third-party module loader in the local workbench. The model produced plausible advice about SPFx’s module-loading model, AMD, webpack and iframe isolation. But it missed the fundamental fact: SystemJS support had been removed from the local workbench in that release. The model knew enough about the surrounding technology to construct a convincing answer while missing or failing to recover the one historical fact needed to solve the task.SPFx 的结果清楚地证明了这一点。SPFx 1.11.0 的一个任务询问关于在本地工作台中如何使用第三方模块加载器的问题。模型给出了关于 SPFx 模块加载模型、AMD、webpack 和 iframe 隔离的合理建议。但它忽略了一个基本事实：该版本已从本地工作台中移除了对 SystemJS 的支持。模型对周围的技术了解得足够多，从而构建了一个令人信服的答案，却忽略或未能检索到解决该任务所需的唯一历史事实。

Dev Proxy showed the same pattern. For a Dev Proxy 0.5.0 task, I asked which executable should be invoked. The model confidently answered devproxy. The required answer for that version was however mgdp, when Dev Proxy was still named Microsoft Graph Developer Proxy.Dev Proxy 也呈现出同样的模式。对于一个 Dev Proxy 0.5.0 的任务，我询问应该调用哪个可执行文件。模型自信地回答是 devproxy。然而，该版本要求的正确答案是 mgdp，当时 Dev Proxy 还被称为 Microsoft Graph Developer Proxy。

So product knowledge varies by task. A model can know a product’s architecture and common patterns while missing one particular fact. Features that predate the cutoff can still sit beyond the model’s capability.因此，产品知识因任务而异。模型可以了解产品的架构和通用模式，但却可能错过某个特定的事实。截止日期之前的功能仍可能超出模型的能力范围。

A model can succeed beyond the expected boundary Copy link模型可能在预期界限之外取得成功 复制链接

At the same time, the model solved tasks for releases far beyond where I expected its product knowledge to be useful. At first, those passes looked like a problem with the experiment. If I was looking for the last version the model knew, why did it occasionally answer questions about much newer versions correctly? Sometimes it could infer how a feature worked.与此同时，模型解决了远超我预期其产品知识范围的发布版本的任务。起初，这些通过的任务看起来像是实验出了问题。如果我是在寻找模型所知的最后一个版本，为什么它偶尔能正确回答关于更新版本的问题？有时它能够推断出某个功能是如何工作的。

In When the model has never seen your code, I described how models map unfamiliar technology to the closest patterns they already know. That behavior can produce plausible mistakes, but it can also help. A model might recognize older Dev Proxy concepts in a newer feature, while its general knowledge of proxies and software fills the remaining gap.在《当模型从未见过你的代码时》一文中，我描述了模型如何将不熟悉的技术映射到它们已知的最接近的模式上。这种行为可能会产生看似合理但错误的答案，但也可能有所帮助。模型可能会在较新的功能中识别出旧的 Dev Proxy 概念，而它对代理和软件的通用知识则填补了其余的空白。

Dev Proxy 2.3.4 shipped on April 23, 2026, followed by 3.0.0 on May 28 and 3.1.0 on July 1. All three releases came after GPT-5.6 Luna’s February 16 knowledge cutoff, yet the model passed one of the two tasks I tested from each release.Dev Proxy 2.3.4 发布于 2026 年 4 月 23 日，随后是 5 月 28 日的 3.0.0 和 7 月 1 日的 3.1.0。这三个版本都在 GPT-5.6 Luna 2 月 16 日的知识截止日期之后发布，但模型在我测试的每个版本的两个任务中都通过了一个。

Those passes could not come from direct exposure to the released versions in training data bounded by the communicated cutoff. They do not prove that every underlying idea first became public on its release date. The model may have inferred the answers from familiar patterns or guessed correctly. Either way, it solved tasks tied to product versions that did not exist at its cutoff.这些通过的任务不可能来自训练数据（受限于所传达的截止日期）中对已发布版本的直接接触。它们并不能证明每个潜在的想法都是在发布之日才首次公开的。模型可能是从熟悉的模式中推断出了答案，或者是猜对了。无论哪种方式，它都解决了与截止日期时尚不存在的产品版本相关的任务。

Capability has several sources Copy link能力有多种来源 复制链接

When a model produces a correct answer, we can’t reliably determine whether the product fact was represented in its weights. It might have extrapolated from an earlier version or transferred a pattern from another product. General programming knowledge may have been enough, or the model may simply have guessed correctly.当模型给出正确答案时，我们无法可靠地确定该产品事实是否已在其权重中得到体现。它可能是从早期版本中推断出来的，或者是从其他产品中迁移了模式。通用的编程知识可能已经足够，或者模型可能只是碰巧猜对了。

From the outside, each route produces the same observable result: the model solved the task. The knowledge cutoff is therefore a poor proxy for what I was trying to measure. I thought I wanted to know:从外部来看，每种途径都会产生相同的可观察结果：模型解决了任务。因此，知识截止日期并不是我试图衡量的内容的良好指标。我原以为我想知道的是：

What’s the latest version of this product the model knows?模型所知的该产品的最新版本是什么？

But what I actually wanted to know was:但我实际上想知道的是：

How capable is this model of working with this product without additional information?在没有额外信息的情况下，该模型处理此产品的能力如何？

The first question asks where the model’s knowledge came from. The second asks whether it can do the work, and it’s actually more useful when working with an AI agent.第一个问题询问的是模型的知识来源。第二个问题询问的是它是否能完成工作，而在使用 AI 智能体时，这实际上更有用。

Stop looking for the version Copy link停止寻找版本号 复制链接

Suppose you’re building applications that use Microsoft Graph and want your agent to use Dev Proxy. One option is to determine the newest Dev Proxy version represented in the model’s training data, then feed the agent documentation for everything released afterward.假设你正在构建使用 Microsoft Graph 的应用程序，并希望你的智能体使用 Dev Proxy。一种选择是确定模型训练数据中包含的最新 Dev Proxy 版本，然后为智能体提供之后发布的所有内容的文档。

The model may miss functionality from before the cutoff and infer functionality from after it. Supplying every newer feature wastes context on knowledge the model might already be able to apply. Meanwhile, older gaps remain. So instead, start with the work you expect the agent to perform. Create representative tasks from that work, then test the model without additional product knowledge. If you don’t have a recurring domain, use the specific problems you need to solve as the evaluation set.模型可能会错过截止日期之前的功能，并推断出截止日期之后的功能。提供每一个较新的功能会浪费上下文窗口，而这些知识模型可能已经能够应用了。与此同时，旧的知识缺口依然存在。因此，请从你期望智能体执行的工作开始。从这些工作中创建具有代表性的任务，然后在没有额外产品知识的情况下测试模型。如果你没有固定的领域，可以使用你需要解决的具体问题作为评估集。

Measure a workload because capability depends on the tasks and context in which you use the model. In What AI benchmarks are not telling you, I explain how to build that workload-based comparison and why public scores cannot replace it. Cutoff dates have the same limitation: they are broad signals that cannot tell you whether the model can do the work that matters to you.衡量工作负载，因为能力取决于你使用模型所执行的任务和环境。在《AI 基准测试没有告诉你的事》一文中，我解释了如何构建这种基于工作负载的比较，以及为什么公开评分无法取代它。截止日期也有同样的局限性：它们只是广泛的信号，无法告诉你模型是否能完成对你而言重要的工作。

Once you have that baseline, use the controlled comparison to measure the effect of adding documentation or agent extensions. Now you know where additional knowledge helps.一旦有了基准，就可以使用受控比较来衡量添加文档或智能体扩展的效果。现在你知道在哪里添加知识会有所帮助了。

Trust but verify Copy link信任但要核实 复制链接

Still, knowledge cutoffs tell us something useful about a model’s training. When a feature was released after the cutoff, the model may never have encountered it directly. The trouble starts when we use that date to predict what the model can do.尽管如此，知识截止日期确实告诉了我们关于模型训练的一些有用的信息。当某个功能在截止日期之后发布时，模型可能从未直接接触过它。当我们使用该日期来预测模型能做什么时，麻烦就开始了。

There was no clean point in my results where knowledge disappeared. Correct answers sat among failures, and failures among correct answers. The model understood the shape of a technology while missing individual facts. Sometimes it reasoned its way into functionality that a cutoff suggested it shouldn’t know.在我的结果中，没有一个清晰的知识消失点。正确答案夹杂在失败案例中，失败案例也夹杂在正确答案中。模型理解了技术的形态，却忽略了单个事实。有时它通过推理得出了截止日期暗示它不应该知道的功能。

To assess a model with your technology, give it the work, measure what it can do and supply knowledge where gaps appear. The cutoff gives you a date but it’s the eval that tells you whether the model can do the work.要评估模型对你所用技术的掌握程度，请给它布置工作，衡量它能做什么，并在出现缺口的地方提供知识。截止日期给你一个日期，但评估结果才能告诉你模型是否能完成工作。