Substack recently launched its AI detector feature in the UI, which is super interesting.Substack 最近在其 UI 中上线了 AI 检测功能，这非常有趣。
Separately, lots of people asked me about interesting local do-it-yourself LLM projects as demos to show what small language models (SLMs) are capable of.另外，很多人问我有没有什么有趣的本地 DIY 大语言模型项目，可以作为演示来展示小型语言模型（SLM）的能力。
Putting one and one together, I thought it would be interesting to show how an AI detector can be implemented. I will also use it as a verifier to train a small language model to produce text that avoids detection. This is a small educational project for studying the limitations of AI detectors and exploring a verifier-based LLM application beyond regular reasoning models trained on math and code.将两者结合起来，我觉得展示如何实现一个 AI 检测器会很有意思。我还会将其用作验证器，训练一个小语言模型来生成能够规避检测的文本。这是一个小型的教育项目，旨在研究 AI 检测器的局限性，并探索除数学和代码推理模型之外，基于验证器的大语言模型应用。
So, as mentioned above, the intended goal of this tutorial is to explain how AI detectors work by building (a simple) one.因此，如上所述，本教程的目标是通过构建一个（简单的）AI 检测器来解释它们的工作原理。
In practice, such a detector can be used to filter out spammy content, but also to potentially improve your personal writing without turning it into AI-generated text. For example, if you wrote a lengthy article and want to improve spelling and grammar, it is tempting (and actually useful) to use a grammar checker to polish it and improve readability. There are different services for that, including general-purpose LLMs like ChatGPT. However, this also runs the risk that these tools turn your writing, even though it’s still your own writing, into something that is then overpolished and now sounds like AI and gets flagged as spammy content.在实践中，这种检测器不仅可以用于过滤垃圾内容，还可以帮助你在不改变个人写作风格的前提下提升文笔。例如，如果你写了一篇长文，想要润色拼写和语法以提高可读性，使用 ChatGPT 等通用大语言模型进行检查确实很有用。然而，这也存在风险：即便内容仍出自你手，这些工具也可能将其过度润色，使其听起来像 AI 生成的，进而被标记为垃圾内容。
For example, with an AI checker, one could say, “Fix my grammar while ensuring that my text still scores 0% AI-generated.”例如，有了 AI 检测器，你可以要求：“修改我的语法，同时确保文本的 AI 生成得分保持在 0%。”
Anyway, while we are building a fully functional checker here, the goal is to explain 1) how AI checkers (can) work and 2) use this as a case study for a more general topic on how to build a scorer or verifier that can be used with LLMs.总之，虽然我们在这里构建的是一个功能完备的检测器，但其目的是解释：1）AI 检测器（如何）工作；2）将其作为一个案例研究，探讨如何构建可用于大语言模型的评分器或验证器这一更广泛的主题。
Disclaimer: AI checkers are essentially a cat-and-mouse game. AI checkers may learn to detect a certain pattern that is indicative of AI-generated content. Then, the next LLM may incidentally or deliberately not exhibit that pattern and avoid detection. The AI checker then has to be updated to detect said LLM, and so forth. Plus, it’s also likely to encounter false positives (human written text flagged as AI-generated), but more on that later.免责声明：AI 检测本质上是一场猫鼠游戏。AI 检测器可能会学会识别某种 AI 生成内容的特定模式，而随后的大语言模型可能会无意或刻意避开这种模式以逃避检测。AI 检测器则必须不断更新以识别这些新模型，以此类推。此外，它还可能出现误报（将人类撰写的文本标记为 AI 生成），稍后我会详细说明。
Project goals项目目标
There are several goals of this project. The overarching goal is, of course, to illustrate how AI detectors work and show an applied end-to-end LLM project including evaluation, training, and local deployment for real-world use.本项目有几个目标。当然，最宏观的目标是阐明 AI 检测器的工作原理，并展示一个包含评估、训练和本地部署的端到端大语言模型应用项目，以供实际使用。
The outcome of this is an AI-detector API that can be used by humans and agents, and a user-friendly UI.最终成果将是一个可供人类和智能体使用的 AI 检测器 API，以及一个用户友好的 UI。
Method overview方法概述
Here, we are going to develop a method similar to Pangram models, which, as far as I know, are behind Substack AI detection feature.我们将开发一种类似于 Pangram 模型的方法，据我所知，Substack 的 AI 检测功能背后就是这种技术。
I wrote a short article about AI-text detection a while back in 2023: What Are the Different Approaches for Detecting Content Generated by LLMs Such As ChatGPT? And How Do They Work and Differ?我在 2023 年写过一篇关于 AI 文本检测的短文：《检测 ChatGPT 等大语言模型生成内容的各种方法是什么？它们的工作原理和区别是什么？》
In essence, there are different ways to detect AI-written text, from supervised classifiers and perturbation-based probability tests to perplexity measures and watermarking.本质上，检测 AI 生成文本的方法多种多样，从监督分类器和基于扰动的概率测试，到困惑度度量和水印技术，不一而足。
In this tutorial, we will build a model that returns a 0-100 score. It’s essentially a classifier with an estimated probability score. The probability score will denote how likely a text is AI-generated according to the classifier. (Or, to be precise the score is the classifier’s estimated probability for the AI-generated class based on its training distribution. However, we shouldn’t interpreted it as a general probability that the text was written by AI.)在本教程中，我们将构建一个返回 0-100 分数的模型。它本质上是一个带有估计概率得分的分类器。该概率得分将表示根据分类器判断，文本是 AI 生成的可能性。（更准确地说，该分数是分类器基于其训练分布对“AI 生成”这一类别给出的估计概率。然而，我们不应将其简单理解为文本由 AI 撰写的通用概率。）
For this, we are going to fine-tune a DistilBERT classifier (similar to what I described in one of my early Substack articles, Finetuning Large Language Models), but more details on that later when we get to that stage.为此，我们将微调一个 DistilBERT 分类器（类似于我在早期 Substack 文章《微调大语言模型》中描述的方法），具体的细节稍后会详细介绍。