The inside story on why OpenAI agents hacked Hugging FaceOpenAI智能体为何入侵Hugging Face的内幕

“It’s not something you can solve overnight,” says Kai Chen, who runs OpenAI’s alignment research team. “There are challenges we’ve been tracking for a very long time, and we’re now seeing them with much greater precision.”“这并非一夜之间就能解决的事，”负责OpenAI对齐研究团队的Kai Chen说。“我们追踪一些挑战已经很久了，现在能以更高的精度来观察它们。”

The Hugging Face hack was a product of months of misbehavior from OpenAI agents, first as they were being trained and then as their abilities were being evaluated. This May, agents in training figured out how to use OpenAI’s infrastructure to communicate with one another and get support with difficult training tasks, including some that were impossible to solve without hacking or otherwise misbehaving. That “message board” was shut down.Hugging Face入侵事件是OpenAI智能体数月不当行为的产物，这些行为首先出现在训练期间，随后又出现在能力评估期间。今年5月，训练中的智能体想出了如何利用OpenAI的基础设施相互通信，并在困难的训练任务中寻求支持，其中一些任务如果不进行黑客攻击或其他不当行为就无法解决。那个“留言板”被关闭了。

Then in July, while being evaluated for their cybersecurity abilities, some models created a new message board. They were supposed to be isolated from the internet, but by working together they managed to get online, hack Hugging Face, and obtain solutions for the cybersecurity problems that had stumped them.随后在7月，一些模型在评估其网络安全能力时创建了新的留言板。它们本应与互联网隔离，但通过协作，它们设法上网、入侵了Hugging Face，并获得了难倒它们的网络安全问题的解决方案。

Based on their investigation, OpenAI researchers believe that events during the training phase led directly to the hack. “For almost every behavior that was worrisome at evaluation time, [we were able to] find some sort of associated behavior at training time that actually we think might have contributed to it,” says Eric Wallace, a member of OpenAI’s alignment research team. 根据调查，OpenAI研究人员认为，训练阶段的事件直接导致了这次入侵。“在评估时几乎每一个令人担忧的行为，我们都能够在训练时找到某种我们认为可能促成了该行为的相关行为，”OpenAI对齐研究团队成员Eric Wallace表示。

When models correctly solve problems during training, the behaviors that led them to that solution are reinforced, and they become more likely to engage in them in the future. So if a model completed a task in May after using the original message board, it became more likely to participate in a new message board later on. This phenomenon, where AI agents misbehave in ways that are reinforced during the training process, is known as reward hacking.当模型在训练中正确解决问题时，引导它们找到解决方案的行为会被强化，它们未来更有可能重复这些行为。因此，如果一个模型在5月使用原始留言板完成了一项任务，它后来就更有可能参与一个新的留言板。这种AI智能体以在训练过程中被强化的方式做出不当行为的现象，被称为奖励黑客（reward hacking）。

Reward hacking also helps to explain why the models worked so hard to make their way onto the internet. During its investigation of the incident, the OpenAI team found that, over the course of training, the models became more and more likely to probe their digital environment for weaknesses and use the tools at their disposal in unexpected ways—a sign that these behaviors were being gradually reinforced. By the time the models were facing tricky cybersecurity problems, they had learned that hacking was an effective way to achieve their goals.奖励黑客（reward hacking）也有助于解释为什么这些模型如此努力地想接入互联网。在调查这一事件的过程中，OpenAI团队发现，在训练过程中，模型越来越倾向于探查其数字环境的弱点，并以意想不到的方式使用手头的工具——这表明这些行为正在被逐步强化。当模型面临棘手的网络安全问题时，它们已经知道黑客攻击是实现目标的有效方式。

These results suggest that the Hugging Face hack could have been avoided if the models weren’t rewarded for misbehaving during training. While researchers don’t yet know how to prevent reward hacking entirely, OpenAI is taking some steps toward mitigating its effects. The company will now look for signs of cheating in all frontier models during training by keeping an eye on their chains of thought—internal notepads where they sketch out their answers and plan their actions. 这些结果表明，如果在训练过程中模型没有因不当行为而获得奖励，Hugging Face入侵事件本可以避免。虽然研究人员还不知道如何完全防止奖励黑客，但OpenAI正在采取一些措施来减轻其影响。该公司现在将在所有前沿模型的训练过程中寻找作弊迹象，方法是关注它们的思维链——即它们草拟答案和规划行动的内部记事本。

This solution isn’t as much of a slam dunk as it might seem: In earlier research, OpenAI showed that punishing models that mention cheating in their chains of thought teaches them to keep their intentions hidden from researchers. But monitoring its models’ thinking does give OpenAI the chance to halt the training process and reassess its approach if models do start learning to reward hack.这个解决方案未必像看起来那么稳操胜券：在早先的研究中，OpenAI表明，惩罚那些在思维链中提到作弊的模型，会教会它们向研究人员隐藏自己的意图。但监控模型的思维确实给了OpenAI一个机会，如果模型确实开始学习奖励黑客，就可以暂停训练过程并重新评估其方法。

If OpenAI stops reinforcing reward hacking in its models—and that’s a huge “if”—that would be a huge step forward. But it wouldn’t solve the alignment problem. The first time a model communicated with other agents or hacked its infrastructure during training, those behaviors had never been reinforced, so agent misbehavior can’t only be attributed to that reinforcement.如果OpenAI停止在其模型中强化奖励黑客行为——这是一个很大的“如果”——那将是一个巨大的进步。但这并不能解决对齐问题。当一个模型在训练中第一次与其他智能体通信或攻击其基础设施时，这些行为从未被强化过，因此智能体的不当行为不能仅仅归因于这种强化。

Jeffrey Ladish, director of the AI safety nonprofit Palisade Research, compares the agents to a human who commits their first financial crime. “It’s not like they had to do fraud before to figure out that fraud is an effective strategy, and you have the same problem with models,” Ladish says. “Alignment science needs to be understanding how model motivations get shaped, such that we can actually figure out how to get models to care about the consequences of their actions.”人工智能安全非营利组织Palisade Research的主任Jeffrey Ladish将这些智能体比作第一次实施金融犯罪的人。“他们并不是必须先实施欺诈才能发现欺诈是一种有效策略，模型也面临同样的问题，”Ladish说。“对齐科学需要理解模型的动机是如何形成的，这样我们才能真正找出如何让模型关心其行为的后果。”

OpenAI’s researchers do have a hypothesis for where some of the misbehavior originated. Before the models formed their first secret message board, they had been trained to communicate and coordinate with subagents—less powerful agents to whom a main agent can delegate tasks. OpenAI的研究人员确实对部分不当行为的来源提出了假设。在模型建立第一个秘密留言板之前，它们已经接受了与子智能体通信和协调的训练——子智能体是能力较弱的智能体，主智能体可以将任务委托给它们。

That learned communication behavior could have transferred to this new setting. The METR report, which investigates the messages that the models sent to one another in detail, supports this hypothesis: One agent on the message board took charge and assigned tasks to the other agents, effectively treating them as subagents. OpenAI could try to prevent agents from secretly communicating with one another by not training this subagent behavior in the future, but that would make the models less useful. 这种习得的通信行为可能迁移到了这种新环境中。METR的报告详细调查了模型之间发送的消息，支持了这一假设：留言板上有一个智能体负责指挥并向其他智能体分配任务，实际上是将它们视为子智能体。OpenAI可以尝试在未来不训练这种子智能体行为，以防止智能体之间秘密通信，但这会使模型的实用性降低。

This tension between capability and safety is at the heart of what went wrong with the Hugging Face incident. OpenAI researchers also identified the models’ persistence as a key factor in the hack. 能力与安全之间的这种张力是Hugging Face事件出错的核心。OpenAI研究人员还认为模型的坚持不懈是导致入侵的一个关键因素。

When they were accidentally given unsolvable problems, the models didn’t give up; instead, they strove to find solutions by any means necessary. But persistence is also a virtue, of course, especially if we want agents that can undertake large amounts of difficult work independently.当它们被意外赋予无法解决的问题时，模型并没有放弃；相反，它们竭尽全力寻找解决方案，不惜一切手段。当然，坚持不懈也是一种美德，尤其是当我们希望智能体能够独立承担大量困难工作时。

OpenAI is working on giving models ways to alert humans if they are given impossible tasks. The problem of teaching models when they should deploy their abilities and when they should hold back, however, won’t be settled in a single postmortem. The training strategies that create superhuman coders—rewarding them when they successfully solve problems—might not work to teach models to use their skills judiciously and respect human desires and values.OpenAI正在努力为模型提供在遇到无法完成的任务时提醒人类的方法。然而，教给模型何时应该施展能力、何时应该克制的这个问题，不会通过一次事后分析就能解决。培养超人类编码者的训练策略——在成功解决问题时给予奖励——可能并不适用于教模型明智地运用技能并尊重人类的愿望和价值观。

“I think there’s a bunch of alignment science that still needs to be done where we can move past just using proxies for task completion,” says Ladish. “That will work to make models very capable, but I don’t think it will work to make them aligned.” “我认为还有大量对齐科学工作需要完成，这样我们才能超越仅仅使用任务完成的代理指标，”Ladish说。“这可以让模型变得非常强大，但我认为它无法让模型实现对齐。”