The concept of recursive self-improvement (RSI) dates back to I. J. Good (1965), where he defined an “ultraintelligent machine” as a system that can surpass humans in all intellectual activities and design better machines to improve itself. Yudkowsky (2008) used the phrase “recursive self-improvement” for a specific feedback loop: an AI uses its current intelligence to improve the cognitive machinery that produces its intelligence.递归自我提升（RSI）的概念最早可追溯至 I. J. Good（1965年）。他将“超智能机器”定义为一种能够在所有智力活动中超越人类，并能设计出更优机器以完善自身的系统。Yudkowsky（2008年）将“递归自我提升”这一短语用于描述特定的反馈循环：AI 利用其当前的智能水平，去改进那些产生自身智能的认知机制。

This feedback loop in modern AI may indicate the model rewriting its own weights directly, or more broadly the model improves the training pipeline and the deployment system, which in turn enables a better successor model with improved performance across economically valuable tasks. The speed of research development in AI has been shown to drastically accelerated in frontier labs (Anthropic; OpenAI).在现代 AI 中，这种反馈循环可能表现为模型直接重写自身的权重，或者更广泛地表现为模型改进了训练流水线和部署系统，进而促成了在经济价值任务上表现更佳的后续模型。事实证明，前沿实验室（如 Anthropic 和 OpenAI）的 AI 研究开发速度已大幅加快。

I explicitly mention “deployment system” because the layer between the raw model and the real-world context seems to be as important as the model’s raw intelligence (i.e. the evals right after pretraining). Harnesses are important components of AI deployment, as shown by successful coding agent products such as Claude Code and Codex. A harness is the system surrounding a base model that orchestrates execution and decides how the model thinks and plans, calls tools and acts, perceives and manages context, stores artifacts, and evaluates results.我明确提到“部署系统”是因为，原始模型与现实世界环境之间的那层架构，其重要性似乎不亚于模型本身的原始智能（即预训练后的评估结果）。正如 Claude Code 和 Codex 等成功的编码智能体产品所展示的那样，Harness（套件/支撑框架）是 AI 部署的重要组成部分。Harness 是围绕基础模型构建的系统，负责编排执行过程，并决定模型如何思考与规划、调用工具与行动、感知与管理上下文、存储工件以及评估结果。

This one post will focus on research around harness engineering and how it contributes to RSI. Much recent work on auto-research, self-improving agents, and evolutionary program search can be organized around this question. Other work on model self-play, synthetic data, test-time training and a broader theme of continual learning also matches the RSI vision (e.g. Yuan et al. 2024, Chen et al. 2024), Zhao et al. 2025, Choi et al. 2026)) but they will not be the focus of this post.本文将聚焦于 Harness 工程相关的研究，以及它如何促进 RSI。近期关于自动研究、自我提升智能体和进化程序搜索的大量工作，都可以围绕这一问题进行梳理。其他关于模型自我博弈、合成数据、测试时训练以及更广泛的持续学习主题的研究（例如 Yuan 等人 2024年，Chen 等人 2024年，Zhao 等人 2025年，Choi 等人 2026年），也与 RSI 的愿景相契合，但它们不是本文的重点。

Harness Design PatternsHarness 设计模式#

Compared with early agent frameworks, “agent = LLM + memory + tools + planning + action”, harnesses engineering additionally include workflow design (e.g. loop engineering), evaluation, permission controls, and persistent state management. It is no longer only prompt templates, but closer to runtime and software system design: how the model observes, acts, memorizes, checks itself, and improves.与早期的智能体框架（“智能体 = 大语言模型 + 记忆 + 工具 + 规划 + 行动”）相比，Harness 工程额外包含了工作流设计（如循环工程）、评估、权限控制和持久化状态管理。它不再仅仅是提示词模板，而更接近于运行时和软件系统设计：即模型如何观察、行动、记忆、自我检查以及自我改进。

The design should be deliberately simple and generic to enable generalization, likely with reference to existing software engineering practices to benefit from prertaining knowlege. There is also a strong analogy between operating systems and harnesses. Similar to an OS, a harness should encapsulate complicated logic while keeping the interface simple. Meanwhile, configs, tool interfaces and other protocols may gradually become standardized across the industry.设计应力求简洁通用以实现泛化，并很可能参考现有的软件工程实践，从而受益于预训练知识。操作系统与 Harness 之间存在强烈的类比关系。类似于操作系统，Harness 应在保持接口简洁的同时封装复杂的逻辑。与此同时，配置、工具接口及其他协议可能会在整个行业中逐渐趋于标准化。

Pattern 1: Workflow Automation模式 1：工作流自动化#

Defining a workflow in which the model can operate, test, and iterate is a key design for automation. Karpathy’s autoresearch repo (https://github.com/karpathy/autoresearch) is a clean example of how such a workflow can be constructed. A common workflow follows a goal-oriented loop of plan, execute, observe/test, improve, and execute again until the goal is achieved. The process may trigger proactive requests to users for clarity in task specification or execution preference.定义一个模型可以进行操作、测试和迭代的工作流，是实现自动化的关键设计。Karpathy 的 autoresearch 仓库 (https://github.com/karpathy/autoresearch) 是构建此类工作流的一个清晰示例。一种常见的工作流遵循以目标为导向的循环：规划、执行、观察/测试、改进，然后再次执行，直到达成目标。该过程可能会触发主动请求，向用户询问任务规范或执行偏好，以求明确。

The workflow graph also emphasizes the model analyzing its own trajectories and failure cases and then iterating on its progress through an “agent runtime” rather than a static prompt template.工作流图还强调了模型分析自身的轨迹和失败案例，并通过“智能体运行时”而非静态提示词模板来迭代其进展。

Pattern 2: File System as Persistent Memory模式 2：作为持久化内存的文件系统#

A recurring pattern in long-horizon agent systems is simple control over rich states and artifacts. A harness should not carry the entire workflow and all logs in context; instead, it should keep durable state in files. In long-horizon agentic rollout, artifacts such as experiment logs, code diffs, paper summaries, error traces, and past rollout trajectories often grow much longer than the context window that the model has trained for.长程智能体系统中一个反复出现的模式是：对丰富的状态和工件进行简单的控制。Harness 不应将整个工作流和所有日志都放在上下文中；相反，它应该将持久化状态保存在文件中。在长程智能体执行过程中，实验日志、代码差异、论文摘要、错误追踪和过往执行轨迹等工件，往往会变得远超模型训练时的上下文窗口长度。

Learning how to read, write, and edit the file system (commonly via bash commands) is a foundation skill for LLMs, and thus managing persistent memory in the simple form of files naturally benefits from improvements in core model capability.学习如何读取、写入和编辑文件系统（通常通过 bash 命令）是大语言模型的一项基础技能，因此，以文件这种简单形式来管理持久化内存，自然会受益于核心模型能力的提升。

Pattern 3: Sub-agent and Backend Jobs模式 3：子智能体与后端作业#

A harness can spawn multiple subagents to execute in parallel and monitor backend jobs. This is useful when the main agent needs to search multiple hypotheses, run experiments concurrently, or delegate isolated subtasks without polluting the main context. The parent agent then needs a small process manager: launch jobs, inspect logs, cancel failed runs, and merge results back into the main agent thread.Harness 可以生成多个子智能体并行执行并监控后端作业。当主智能体需要搜索多个假设、同时运行实验，或在不污染主上下文的情况下委派独立的子任务时，这非常有用。此时，父智能体需要一个小型进程管理器：启动作业、检查日志、取消失败的运行，并将结果合并回主智能体线程。

The key design choice is to make parallelism explicit and inspectable. If subagent outputs only live in a transient chat context, they quickly become obselete and hidden. If they are stored as files, logs, and status records, the model can recover after interruptions and reason over its own execution history.关键的设计选择是使并行性变得明确且可检查。如果子智能体的输出仅存在于瞬时的聊天上下文中，它们很快就会过时并被隐藏。如果它们被存储为文件、日志和状态记录，模型就可以在中断后恢复，并对其自身的执行历史进行推理。

Case study: Coding Agent Harness案例研究：编码智能体 Harness#

The core interface of mainstream coding agents has become stabilized across Claude Code, Codex, OpenCode, and Cursor-style agents. They commonly use a loop like:主流编码智能体的核心接口已在 Claude Code、Codex、OpenCode 和 Cursor 风格的智能体中趋于稳定。它们通常使用如下循环：

With access to a set of tools, the coding agent is able to develop and debug issues in a given repository, similar to how human developers are equipped with IDEs.通过访问一系列工具，编码智能体能够像配备了 IDE 的人类开发者一样，在给定的仓库中开发和调试问题。

(Not a comprenhensive list; shown for demonstration. Read this if interested.)（并非详尽列表；仅供演示。如有兴趣请阅读原文。）

Group	Tool definitions
File system	- File discovery: glob, grep, ls - File read: read, read_many - File modification: write (a whole new file); edit (string exact-match replacement); multi_edit; apply_patch (applies a structured patch/diff)
Shell execution	Run commands: bash, PowerShell
IO	lsp, git tools like git_status, git_diff, git_commit
External context	MCP tools, Skills
Web search	web_search, web_fetch, browser tools
Artifacts	Read docs, images; generate HTML, images
Backend processes	Such as: CronCreate, CronDelete, CronList
Agent delegation	Such as: spawn_agent, resume_agent, wait_agent, list_agents, close_agent, interrupt_agent, etc.

Harness Layer vs Core Intelligence?Harness 层与核心智能的区别#

It is hard to forecast how much the future of RSI will rely on harness engineering, but the near-term path of RSI is unlikely to start as a model directly rewriting its weights. My prediction of a practical near-term path is:很难预测 RSI 的未来在多大程度上依赖于 Harness 工程，但 RSI 的近期路径不太可能直接从模型重写自身权重开始。我对近期实用路径的预测是：

1. Harness engineering will evolve in the direction of meta-methodology (i.e. improving the machinery for getting better answers, not just improving the answer itself). The harness system itself becomes an optimization target, with fewer heuristic rules and more general mechanisms.Harness 工程将朝着元方法论的方向演进（即改进获取更好答案的机制，而不仅仅是改进答案本身）。Harness 系统本身成为优化目标，减少启发式规则，增加通用机制。
2. In turn, mature harnesses enable auto-research for model self-improvement loop and smarter models prevents harnesses from overengineering and keep the system sustainable.反过来，成熟的 Harness 能够实现用于模型自我提升循环的自动研究，而更聪明的模型则能防止 Harness 过度工程化，并保持系统的可持续性。

Eventually it is possible that many harness improvements will be internalized into core model behavior, but the interface with external context and tools should remain. We have seen a softer version of this pattern with prompt engineering: manual prompt tricks became less central as instruction tuning and model reasoning improved, but the need to specify goals, constraints, context, and evaluation did not disappear.最终，许多 Harness 的改进很有可能被内化到核心模型行为中，但与外部上下文和工具的接口应当保留。我们在提示词工程中已经看到了这种模式的较温和版本：随着指令微调和模型推理能力的提升，手动提示词技巧变得不再那么核心，但指定目标、约束、上下文和评估的需求并未消失。

Harness OptimizationHarness 优化#

The progression in the object being optimized in the harness system is roughly: instruction prompts → structured context → workflow → harness code → optimizer code. As the model becomes more intelligent and powerful, we move toward more complex targets and generic methods.Harness 系统中被优化对象的演进大致为：指令提示词 → 结构化上下文 → 工作流 → Harness 代码 → 优化器代码。随着模型变得越来越智能和强大，我们正朝着更复杂的目标和通用方法迈进。

Context Engineering上下文工程#

Simply appending all the tool responses and model generations into the context can quickly grow out of control as the agentic job horizon increases significantly. Context management is a layer to construct a more structed and concise context for LLM and manage persistant states. There is no doubt that long-context research will keep on making progress but at the moment long-context intelligence and context engineering sometime intertwines.随着智能体作业视野的显著增加，简单地将所有工具响应和模型生成内容添加到上下文中，很快就会变得难以控制。上下文管理是构建更结构化、更简洁的 LLM 上下文并管理持久化状态的一层。毫无疑问，长上下文研究将持续取得进展，但目前长上下文智能与上下文工程有时是交织在一起的。

Agentic Context Engineering (ACE; Zhang et al. 2025) treats context as an evolving playbook rather than an increasingly lengthening prompt. It has three components to maintain one context playbook of bullet points, each with an identifier and a description.智能体上下文工程（ACE; Zhang 等人 2025年）将上下文视为一个不断演进的剧本，而不是一个日益增长的提示词。它有三个组件来维护一个包含要点、每个要点都有标识符和描述的上下文剧本。

1. Generator: produces task trajectories, with reference to bullet points.生成器：参考要点，生成任务轨迹。
2. Reflector: distills insights from successful and failed trajectories.反射器：从成功和失败的轨迹中提炼洞察。
3. Curator: updates the structured context with incremental, itemized entries.策展人：通过增量、逐项的条目更新结构化上下文。

To prevent context collapse and brevity bias during iterative rewrites, one key design choice in ACE is that the curator does not rewrite a full prompt blob. It instead outputs a collection of structured, itemized bullets in the form of (identifier, description), and these bullets are merged into a structured context logbook with deterministic logic. The context items are refined and deduplicated periodically.为了防止在迭代重写过程中出现上下文崩溃和简洁性偏差，ACE 中的一个关键设计选择是：策展人不会重写完整的提示词块。它转而输出一系列结构化、逐项的要点（标识符，描述），这些要点通过确定性逻辑合并到一个结构化的上下文日志中。上下文条目会被定期细化和去重。

The fact that ACE learns insights from rollouts helps us move toward self-managed memory, but the update rules and the overall workflow are still handcrafted. To move toward a more self-improving loop, Meta Context Engineering (MCE; Ye et al. 2026) separates the mechanism (how to manage context) from the artifact content (what is in context), running skill evolution at the meta-optimization level and context optimization at the base level.ACE 从执行结果中学习洞察的事实有助于我们迈向自我管理内存，但更新规则和整体工作流仍然是手工制作的。为了迈向更具自我提升性的循环，元上下文工程（MCE; Ye 等人 2026年）将机制（如何管理上下文）与工件内容（上下文中的内容）分离开来，在元优化层面运行技能演化，在基础层面运行上下文优化。

An MCE skill $s\in \mathcal{S}$ defines a context function $c_s=({\rho }_s,F_s)$ and maps an input $x$ to context $c=F_s(x;{\rho }_s)$, where:MCE 技能 s∈S 定义了一个上下文函数 cs=(ρs,Fs)，并将输入 x 映射到上下文 c=Fs(x;ρs)，其中：

• ${\rho }_s=\{{\rho }_1,\dots ,{\rho }_m\}$ are static components (prompts, knowledge bases, code libraries).ρs={ρ1,…,ρm} 是静态组件（提示词、知识库、代码库）。
• $F_s=\{F_1,\dots ,F_k\}$ are dynamic operators (search, selection, filtering, formatting).Fs={F1,…,Fk} 是动态操作符（搜索、选择、过滤、格式化）。

The bi-level optimization is to find the best context $c_s^{\ast }$ given skill $s$ on the training data, while the outer loop finds the optimal skill that provides the best performance on the validation set:双层优化旨在给定训练数据上的技能 s 时找到最佳上下文 cs∗，而外层循环则找到在验证集上提供最佳性能的最优技能：

The skill database tracks the history of previous skills, context functions and eval metrics ${\mathcal{H}}_{k-1}=\{(s_i,c_i,J_i^{\text{train}},J_i^{\text{val}}){\}}_{i=1}^{k-1}$. A meta-level agent performs agentic crossover over prior skills to create a new skill given a task $\tau$: $s_k=\text{crossover}(\tau ,{\mathcal{H}}_{k-1})$.技能数据库跟踪先前技能、上下文函数和评估指标的历史 Hk−1={(si,ci,Jitrain,Jival)}i=1k−1。元级智能体对先前的技能进行智能体交叉，以在给定任务 τ 时创建新技能：sk=crossover(τ,Hk−1)。

Then a base-level context engineer executes the skill $s_k$ and learns the context function from rollout feedback ${\mathcal{R}}_k$, guided by the current skill: $c_k=\text{engineer}(\tau ,s_k;c_{k-1}^{\ast },{\mathcal{R}}_k)$.然后，基础级上下文工程师执行技能 sk，并在当前技能的引导下从执行反馈 Rk 中学习上下文函数：ck=engineer(τ,sk;ck−1∗,Rk)。

MCE does not enforce a heuristic rule for how to structure context as ACE does. It uses free-form skills to store the most important knowledge for a task, and evolves the skill and the skill-conditioned context iteratively together. Implementation-wise, a context function $c$ is instantiated as a collection of files in a dedicated directory, including both static (skill.md) and dynamic (context and data rollouts) components. Both meta-level and base-level optimization are executed in agentic coding envs with a standard tool set,MCE 不像 ACE 那样强制执行启发式规则来构建上下文。它使用自由形式的技能来存储任务最重要的知识，并迭代地共同演进技能和技能条件下的上下文。在实现方面，上下文函数 c 被实例化为专用目录中的一组文件，包括静态（skill.md）和动态（上下文和数据执行轨迹）组件。元级和基础级优化均在具有标准工具集的智能体编码环境中执行。

Meta-Harness (Lee et al. 2026) moves another level deeper: the optimized object is the code that determines and optimizes what information should be stored, retrieved, and presented to the model. “Meta-” in its name means it is a harness for optimizing harnesses.元 Harness（Lee 等人 2026年）更进了一步：优化的对象是决定和优化应存储、检索和呈现给模型哪些信息的代码。“元”在名称中意味着它是用于优化 Harness 的 Harness。

The proposer for creating a new harness is itself a coding agent and the final output is a collection of harness candidates on the Pareto frontier.用于创建新 Harness 的提议者本身就是一个编码智能体，最终输出是帕累托前沿上的一组 Harness 候选者。

• The entire execution history is accessible via a file system, and thus the coding agent uses commands like grep or cat to read through it instead of shoveling everything into a single prompt context.整个执行历史可通过文件系统访问，因此编码智能体使用 grep 或 cat 等命令来阅读它，而不是将所有内容塞进单个提示词上下文中。
• The proposed harness is a dictionary in the file system containing its own source code, scores, rollout trajectories, and state updates.所提议的 Harness 是文件系统中的一个字典，包含其自身的源代码、分数、执行轨迹和状态更新。
• The mete-harness loop iteratively creates new harnesses, and only qualified ones are kept.元 Harness 循环迭代地创建新的 Harness，只有合格的才会被保留。

Still, the important lesson is clear: once harness design becomes an executable search space, a strong coding agent can exploit the same design space human engineers use.尽管如此，重要的经验很明确：一旦 Harness 设计成为一个可执行的搜索空间，强大的编码智能体就可以利用人类工程师使用的相同设计空间。

Workflow Design工作流设计#

Workflow design in harness engineering can be handcrafted by domain experts. Taking auto-research as an example, various frameworks have been proposed and tested. The AI Scientist system (Lu et al. 2026) builds a pipeline to propose research ideas, write code, run experiments, analyze results, write a manuscript, and perform peer review. Meng et al. (2026) make verifiability the central design constraint in ScientistOne, where every claim (citation, numerical, methodological, conclusion) must trace to an evidence source and is audited by Chain-of-Evidence checks.Harness 工程中的工作流设计可以由领域专家手工制作。以自动研究为例，各种框架已被提出并测试。AI Scientist 系统（Lu 等人 2026年）构建了一个流水线来提出研究想法、编写代码、运行实验、分析结果、撰写手稿并进行同行评审。Meng 等人（2026年）将可验证性作为 ScientistOne 的核心设计约束，其中每一项主张（引用、数值、方法论、结论）都必须追溯到证据来源，并由证据链检查进行审计。

The Autodata agent (Kulikov et al. 2026) is designed to work as a data scientist for generating training and evaluation data. The main agent manages a challenger that proposes problems, a weak solver, a strong solver, and a verifier/judge, aiming to synthesize data at the “just right” level of difficulty, meaning that the strong solver succeeds but the weak solver fails.Autodata 智能体（Kulikov 等人 2026年）被设计为数据科学家，用于生成训练和评估数据。主智能体管理着一个提出问题的挑战者、一个弱求解器、一个强求解器和一个验证者/裁判，旨在合成难度“恰到好处”的数据，即强求解器成功而弱求解器失败。

In Autodata, the challenger prompt is updated iteratively according to feedback from the solvers and verifier. The limitation here is that synthesized tasks are used to fine-tune weak solvers but not strong solvers; if the loop cannot iteratively improve the strong model, it is more like indirect distillation over a generated prompt distribution, with less RSI flavor.在 Autodata 中，挑战者提示词根据求解器和验证者的反馈进行迭代更新。这里的局限性在于，合成任务用于微调弱求解器而非强求解器；如果循环无法迭代改进强模型，它更像是对生成提示词分布的间接蒸馏，RSI 的味道较淡。

The design space for workflow is enormous, and naturally we can think of workflow design as a search problem, and therefore we should be able to find good solutions by algorithms rather than only manually craft them. Following this direction, Automated Design of Agentic Systems (ADAS; Hu et al. 2025) formulates agent design itself as an optimization problem, “meta-agent search” where a meta-agent proposes new designs of agentic workflows.工作流的设计空间巨大，自然地，我们可以将工作流设计视为一个搜索问题，因此我们应该能够通过算法找到好的解决方案，而不是仅仅手工制作它们。遵循这一方向，智能体系统自动化设计（ADAS; Hu 等人 2025年）将智能体设计本身公式化为一个优化问题，即“元智能体搜索”，其中元智能体提出智能体工作流的新设计。

1. Initialize an archive of agentic workflows with simple agents such as CoT and self-refine.用简单的智能体（如 CoT 和自我修正）初始化智能体工作流档案。
2. Ask a meta-agent to program new agents, all in code, inspired by existing solutions in the archive.
   • The meta-agent first generates a high-level description of the new workflow, and then implements it in code.元智能体首先生成新工作流的高级描述，然后用代码实现它。
   • The draft program then goes through two self-refine steps (i.e. ask the model to provide feedback and then ask the same model to refine the previously generated outputs based on the feedback; Madaan et al. 2023) by the meta-agent to check its novelty.草稿程序随后由元智能体经历两个自我修正步骤（即要求模型提供反馈，然后要求同一个模型根据反馈修正之前生成的输出；Madaan 等人 2023年）以检查其新颖性。
3. Evaluate each new candidate and add successful ones back to the archive.评估每个新候选者，并将成功的候选者加回档案。
4. Repeat steps 2-3 until the maximum iteration count is reached.重复步骤 2-3，直到达到最大迭代次数。

AFlow (Zhang et al. 2025) represents an agentic workflow as a graph, where nodes represent LLM-invoking actions and edges implement logical operations in code. The workflow optimization relies on MCTS (Monte Carlo Tree Search):AFlow（Zhang 等人 2025年）将智能体工作流表示为图，其中节点代表调用 LLM 的动作，边通过代码实现逻辑运算。工作流优化依赖于 MCTS（蒙特卡洛树搜索）：

1. Initialize the starting workflow $W_0$ in the tree with a template.用模板初始化树中的起始工作流 W0。
2. Select a workflow node using a soft mixture of score and uniform exploration.使用分数和均匀探索的软混合来选择工作流节点。
3. Expand it by asking an LLM to produce a modified workflow conditioned on its evaluation performance.通过要求 LLM 根据其评估性能生成修改后的工作流来扩展它。
4. Execute and evaluate the new workflow.执行并评估新工作流。
5. Add it back to the tree if the new workflow shows improvement within a budget of $N$ rounds.如果新工作流在 N 轮预算内显示出改进，则将其加回树中。
6. Repeat steps 2-5 and stop when the top-$k$ average score plateaus or hit the budget.重复步骤 2-5，当 top-k 平均分数趋于平稳或达到预算时停止。

Experiments of AFlow in QA, code, and math tasks showed decent improvement of AFlow over manually designed workflows and ADAS.AFlow 在 QA、代码和数学任务中的实验显示，AFlow 相比手工设计的工作流和 ADAS 有不错的改进。

Self-Improving Harness自我提升 Harness#

Either context engineering or workflow design is only one part of a harness. We need to search through the entire design space and optimize context-management logic, workflow, permissions, and many other harness components together. As we have seen in work like Meta-Harness, ADAS, and AFlow, ✨code✨ is a universal language for defining programs and systems. In simple words, a harness is code that programs how prompts, tool calls, subagents, control flow, memory, and workflow logic work together. If an LLM can optimize the code that executes agents, it can access a much larger design space than hand-written prompts.无论是上下文工程还是工作流设计，都只是 Harness 的一部分。我们需要搜索整个设计空间，并共同优化上下文管理逻辑、工作流、权限以及许多其他 Harness 组件。正如我们在 Meta-Harness、ADAS 和 AFlow 等工作中看到的那样，✨代码✨是定义程序和系统的通用语言。简单来说，Harness 是编程提示词、工具调用、子智能体、控制流、内存和工作流逻辑如何协同工作的代码。如果 LLM 可以优化执行智能体的代码，它就能访问比手写提示词大得多的设计空间。

Self-Taught Optimizer (STOP; Zelikman et al. 2023) is one of the early examples of recursive scaffolding improvement. A seed improver $I_0$ at step $t=0$ takes an initial solution $s$, a utility function $u$, and a black-box language model $M$, and returns an improved solution $s^{\prime }$, that is, $s^{\prime }=I(u,s;M)$. The goal of STOP is not directly to improve $s$ but to improve the improver $I$ itself.自学优化器（STOP; Zelikman 等人 2023年）是递归脚手架改进的早期示例之一。在 t=0 时的一个种子改进器 I0 接收初始解决方案 s、效用函数 u 和黑盒语言模型 M，并返回一个改进的解决方案 s′，即 s′=I(u,s;M)。STOP 的目标不是直接改进 s，而是改进改进器 I 本身。

First, let’s define the meta-utility as the average utility of a given improver function $I$ over a collection of downstream tasks $\mathcal{D}$:首先，我们将元效用定义为给定改进器函数 I 在一系列下游任务 D 上的平均效用：

Because improving the improver function is an optimization problem itself, we can recursively get a new version of $I_t$ based on $I_{t-1}$’s performance measured by meta-utility via a self-improvement update:由于改进改进器函数本身就是一个优化问题，我们可以通过自我提升更新，根据 It−1 在元效用下测得的性能，递归地获得 It 的新版本：

In Zelikman et al. (2023)’s experiments, the improved improver discovered various strategies, such as genetic algorithms, decomposing and improving parts, multi-armed prompt bandits, simulated annealing, varying temperature, and beam/tree search. This is analogous to how a harness workflow can be represented as an object for optimization.在 Zelikman 等人（2023年）的实验中，改进后的改进器发现了各种策略，例如遗传算法、分解与改进部分、多臂提示词老虎机、模拟退火、调整温度以及束/树搜索。这类似于 Harness 工作流如何被表示为优化对象。

A cautionary result in their findings is that STOP improved mean downstream performance across iterations with GPT-4 but degraded with weaker models like GPT-3.5 and Mixtral. Recursive structure alone is not enough. The base model must be capable enough to improve the mechanism. This implies that harness improvement enables better deployment of the model but intelligence is still the core.他们发现的一个警示性结果是，STOP 在使用 GPT-4 进行迭代时提高了平均下游性能，但在使用 GPT-3.5 和 Mixtral 等较弱模型时性能下降。仅有递归结构是不够的。基础模型必须足够强大才能改进机制。这意味着 Harness 改进能够实现模型更好的部署，但智能仍然是核心。

A more recent work, Self-Harness (Zhang et al. 2026), relies on LLM agents to improve their own harness via a propose-evaluate-accept loop.最近的一项工作 Self-Harness（Zhang 等人 2026年）依赖于 LLM 智能体通过提议-评估-接受循环来改进其自身的 Harness。

The loop in Self-Harness has three stages:Self-Harness 中的循环有三个阶段：

1. Weakness mining: cluster failures into verifier-grounded failure patterns.
   • The current harness $h_t$ is used to evaluate on tasks and execution traces are collected for analysis.当前的 Harness ht 被用于在任务上进行评估，并收集执行轨迹以供分析。
   • Note that two runs can share the same verifier outcome in the error logs on the surface, such as timeout or missing artifact, while having different causal mechanisms. Therefore we need a failure record of rich information, containing the terminal verifier-level cause, the causal status of the relevant agent behavior, and the abstract agent mechanism exposed by the trace, to uncover the root causes.请注意，两次运行在错误日志的表面上可能共享相同的验证器结果（如超时或缺少工件），但具有不同的因果机制。因此，我们需要包含丰富信息的失败记录，其中包含终端验证器级原因、相关智能体行为的因果状态，以及轨迹暴露出的抽象智能体机制，以揭示根本原因。
2. Harness proposal: propose bounded harness edits based on mined failure patterns.
   • The same model is invoked under $h_t$ as a proposer.同一个模型在 ht 下被调用作为提议者。
   • The model is provided with a bounded proposal context: (1) the editable surfaces of the current harness, (2) the verifier-grounded failure patterns from the evaluation system, (3) records of passing behaviors that should be preserved, and (4) summaries of previously attempted edits.模型被提供了一个有界的提议上下文：（1）当前 Harness 的可编辑表面，（2）来自评估系统的验证器基础失败模式，（3）应保留的通过行为记录，以及（4）先前尝试编辑的摘要。
   • Harness edits should prefer recurrent error patterns that are addressable (e.g. not task-specific difficulty) and can be resolved by narrow changes.Harness 编辑应优先考虑可解决的反复出现的错误模式（例如非任务特定的难度），并可以通过细微的更改来解决。
   • Harness edit candidates should be distinct and diverse.Harness 编辑候选者应具有独特性和多样性。
3. Proposal validation: validate and merge qualified edits to create a new harness $h_{t+1}$.
   • Candidate edits are evaluated by regression tests on held-in $D_{\text{in}}$ (for testing whether the weakness is resolved) and held-out $D_{\text{out}}$ (for checking whether other unknown issues were introduced) splits.候选编辑通过在保留内 Din（用于测试弱点是否已解决）和保留外 Dout（用于检查是否引入了其他未知问题）拆分上的回归测试进行评估。
   • Candidates are accepted only if they have no regression on both held-in and held-out data.仅当候选者在保留内和保留外数据上都没有回归时，才会被接受。
   • Accepted candidates are merged to update the harness to $h_{t+1}$, while rejected candidates are logged without changing the active harness.被接受的候选者被合并以将 Harness 更新为 ht+1，而被拒绝的候选者被记录下来，而不更改活动的 Harness。

When running MiniMax M2.5, Qwen3.5-35B-A3B, and GLM-5 on Terminal-Bench-2, Self-Harness was shown to learn model-specific harness instructions that target at different weaknesses of different base models and improve held-out pass rates.在 Terminal-Bench-2 上运行 MiniMax M2.5、Qwen3.5-35B-A3B 和 GLM-5 时，Self-Harness 被证明可以学习针对不同基础模型不同弱点的模型特定 Harness 指令，并提高保留外通过率。

Self-harness type of work does raise my concerns that if a program is allowed to edit the OS system, abstraction boundaries are broken. The editable surface needs to be properly designed and the permission control and security layers need to live outside this loop. All the challenges around reward hacking still remain.Self-harness 这类工作确实引起了我的担忧：如果允许程序编辑操作系统，抽象边界就会被打破。可编辑表面需要经过精心设计，权限控制和安全层需要存在于此循环之外。围绕奖励黑客攻击的所有挑战仍然存在。

Evolutionary Search进化搜索#

Evolutionary search is an optimization method inspired by natural selection (see my old post on evolutionary algorithm). It evolves a population of solutions by mutating them and only keeping those with high “fitness” in the crowd. Evolutionary search comes in handy when (1) the search space is extensive or weirdly shaped; and (2) it is hard to optimize directly with gradients but easy to evaluate solutions. Harness search seems to be a good fit here.进化搜索是一种受自然选择启发的优化方法（参见我关于进化算法的旧文）。它通过变异来演化一组解决方案，并仅保留群体中具有高“适应度”的解决方案。当（1）搜索空间广泛或形状怪异；以及（2）难以直接通过梯度优化但容易评估解决方案时，进化搜索就派上用场了。Harness 搜索似乎非常适合这里。

Evolutionary search has been used in prompt engineering in the past studies. Promptbreeder (Fernando et al. 2023) optimizes task-specific prompts through a rich set of mutation operations, and interestingly the mutation prompts (i.e. instructions to an LLM to mutate a task prompt) are themselves also improved through evolution. GEPA (Agrawal et al. 2025) combines reflection-based prompting with evolutionary search and uses natural language reflection over trajectories of trial and error to propose prompt updates.进化搜索在过去的研究中已被用于提示词工程。Promptbreeder（Fernando 等人 2023年）通过丰富的变异操作优化任务特定提示词，有趣的是，变异提示词（即指示 LLM 变异任务提示词的指令）本身也通过进化得到改进。GEPA（Agrawal 等人 2025年）将基于反射的提示词与进化搜索相结合，并利用对试错轨迹的自然语言反射来提出提示词更新。

Novikov et al. (2025) introduced AlphaEvolve as a coding-agent evolutionary search system, which stores a pool of candidate programs and prompts frozen LLMs to generate diffs for improvement. As the system repeatedly evaluates child programs and keeps successful ones, it discovers better solutions in time.Novikov 等人（2025年）引入了 AlphaEvolve 作为一种编码智能体进化搜索系统，它存储了一池候选程序，并提示冻结的 LLM 生成差异以进行改进。随着系统反复评估子程序并保留成功的程序，它会及时发现更好的解决方案。

A few details matter in the design of AlphaEvolve:AlphaEvolve 的设计中有一些细节很重要：

• The prompt includes parent programs, results, instructions, and sometimes meta information.提示词包括父程序、结果、指令，有时还包括元信息。
• The coding agent has access to the full repo, but code regions for improvement are explicitly marked with # EVOLVE-BLOCK-START and # EVOLVE-BLOCK-END.编码智能体可以访问整个仓库，但需要改进的代码区域被明确标记为 # EVOLVE-BLOCK-START 和 # EVOLVE-BLOCK-END。
• Meta-prompt co-evolves with instructions and context as suggested by LLM, in a similar way as how we evolve solution programs.元提示词与 LLM 建议的指令和上下文共同演进，方式与我们演化解决方案程序的方式类似。

Ablations show the evolution procedure, context in prompts, meta-prompts, full-file evolution and the use of stronger LLMs.消融实验显示了进化过程、提示词中的上下文、元提示词、全文件进化以及使用更强 LLM 的效果。

Recent variants such as ThetaEvolve (Wang et al. 2025) combines evolutionary search with RL and in-context learning. ShinkaEvolve (Lange et al. 2025), on the other hand, introduced three new components to improve LLM sampling efficiency:最近的变体如 ThetaEvolve（Wang 等人 2025年）将进化搜索与 RL 和上下文学习相结合。另一方面，ShinkaEvolve（Lange 等人 2025年）引入了三个新组件来提高 LLM 采样效率：

• More sample-efficient exploration by designing parent sampling to balance performance rank and offspring count.通过设计父采样来平衡性能排名和后代数量，从而实现更具样本效率的探索。
• Code-novelty rejection sampling by discarding candidates that are too similar to the existing population based on embedding-based cosine similarity.代码新颖性拒绝采样，通过基于嵌入的余弦相似度丢弃与现有群体过于相似的候选者。
• Identifying good patterns in successful solutions in a meta-scratchpad to guide future mutation.在元暂存板中识别成功解决方案中的良好模式，以指导未来的变异。

Unlike the methods above, which focus on solution improvement, Darwin Gödel Machine (DGM; Zhang et al. 2025) explicitly targets the evolution of an editable harness-code repository with an LLM-based coding agent. Precisely, this agent is allowed to modify its own harness. A follow-up work on Hyperagents (Zhang et al. 2026) introduced a meta-agent to control how to modify existing task agents to create new ones.与上述专注于解决方案改进的方法不同，达尔文哥德尔机（DGM; Zhang 等人 2025年）明确针对具有基于 LLM 的编码智能体的可编辑 Harness 代码库的演进。准确地说，该智能体被允许修改其自身的 Harness。关于 Hyperagents（Zhang 等人 2026年）的一项后续工作引入了一个元智能体来控制如何修改现有任务智能体以创建新的智能体。

1. Start with one coding agent in the pool.从池中的一个编码智能体开始。
2. In each iteration, pick one parent with a probability proportional to its performance and inversely to the number of children it has, to modify and branch off to produce new agents.在每次迭代中，选择一个概率与其性能成正比、与其拥有的子代数量成反比的父代，进行修改并分支以产生新的智能体。
3. The selected parent agent examines its own benchmark evaluation log and then proposes improvements to its own harness codebase to generate a new version of the coding agent. Code editing is implemented with two basic tools: (1) bash (args: <bash_command>) and (2) editor (args: view/create/edit <file_path>).选定的父智能体检查其自身的基准评估日志，然后提出对其自身 Harness 代码库的改进，以生成编码智能体的新版本。代码编辑通过两个基本工具实现：（1）bash（参数：<bash_command>）和（2）编辑器（参数：view/create/edit <file_path>）。
4. New coding agents are evaluated, and only those with sufficiently high performance are added back into the pool.新的编码智能体被评估，只有性能足够高的才会被加回池中。
5. Repeat steps 2-4 until some stop criteria hit.重复步骤 2-4，直到达到某些停止标准。

DGM is harness evolution under a fixed model. In experiments with Claude 3.5 Sonnet as the base LLM and simple initial harness configs, the DGM-discovered agents are comparable to or outperform handcrafted agents on SWE-bench Verified (20% to 50%) and Polyglot (14.2% to 30.7%).DGM 是固定模型下的 Harness 演进。在使用 Claude 3.5 Sonnet 作为基础 LLM 和简单的初始 Harness 配置的实验中，DGM 发现的智能体在 SWE-bench Verified（20% 到 50%）和 Polyglot（14.2% 到 30.7%）上与手工制作的智能体相当或表现更优。

This family of methods works well when candidate solutions are automatically evaluable and candidate fitness is easy to quantify, such as matrix multiplication, GPU kernel optimization, algorithm contests, datacenter scheduling. It struggles with domains where evaluation is slow, ambiguous, or mostly heuristic-based. The compute efficiency and effectiveness of evolution are also concerns.当候选解决方案可自动评估且候选适应度易于量化时，这一系列方法效果很好，例如矩阵乘法、GPU 内核优化、算法竞赛、数据中心调度。它在评估缓慢、模棱两可或主要基于启发式方法的领域中表现吃力。进化的计算效率和有效性也是令人担忧的问题。

Joint Optimization with Model Weights与模型权重的联合优化#

Harness evolution changes the non-parametric system around the model. To enable full self-improvement, the model can totally be allowed to update its own weights at the same time. The weight update can be implemented via improvements in the model training pipeline or continual learning at test time. The topic of continual learning is worthy of its own post in the future.Harness 演进改变了模型周围的非参数系统。为了实现完全的自我提升，模型完全可以在同时更新其自身的权重。权重更新可以通过模型训练流水线的改进或测试时的持续学习来实现。持续学习这一主题值得在未来单独撰写一篇文章。

SIA (Hebbar et al. 2026) is an early attempt to combine harness improvement and model-parameter updates in the same optimization loop, with three components in the design:SIA（Hebbar 等人 2026年）是早期尝试在同一个优化循环中结合 Harness 改进和模型参数更新的尝试，设计中有三个组件：

• Meta-Agent: proposes the initial harness.元智能体：提出初始 Harness。
• Task-Specific Agent: executes the task.任务特定智能体：执行任务。
• Feedback-Agent: chooses whether to update the harness or the model weights based on recent trajectories.反馈智能体：根据最近的轨迹选择更新 Harness 还是模型权重。

There are a few confounding choices in SIA’s experiments that make the results hard to interpret. For example, the task-specific agent is much weaker than the models used for the Meta-Agent and Feedback-Agent (gpt-oss-120b vs Claude Sonnet 4.6), and the baselines are too weak to cross-reference cleanly against related methods. I would consider the direction interesting, but the evidence provisional. Yet many challenges, such as training stability and Goodhart effect, still remain open.SIA 的实验中有一些混杂的选择，使得结果难以解释。例如，任务特定智能体比用于元智能体和反馈智能体的模型（gpt-oss-120b vs Claude Sonnet 4.6）弱得多，且基线太弱，无法与相关方法进行清晰的交叉参考。我认为这个方向很有趣，但证据是暂时的。然而，训练稳定性和古德哈特定律等许多挑战仍然悬而未决。

Future Challenges未来挑战#

The AI Scientist line of work is a strong demonstration that an expert-designed harness can coordinate a large portion of auto-research loop, experimented in the form of writing research papers. But paper production is not identical to scientific discovery. A system can write a plausible manuscript while still having fabricated citations, implementation drift, or weak experimental results.AI Scientist 系列工作有力地证明了专家设计的 Harness 可以协调自动研究循环的很大一部分，并以撰写研究论文的形式进行了实验。但论文生产并不等同于科学发现。一个系统可以在拥有虚假引用、实现漂移或实验结果薄弱的同时写出一份貌似合理的手稿。

Trehan & Chopra (2026) tested whether LLMs can go from a research idea to a paper with minimal scaffolding and basic tools (i.e., read_file, write_file, llm_search, list_files). Each idea had a dedicated workspace where agents could generate and read documents as part of context. They experimented in three domains (world models, multi-agent RL, AI safety & alignment), with each domain containing 45-50 high-quality seed documents to inspire new ideas. Only four ideas were selected by human experts to run through the full pipeline, and only one was fully executed into a paper. They observed six recurring failure modes in the experiments:Trehan & Chopra（2026年）测试了 LLM 是否可以在最少的脚手架和基本工具（即 read_file, write_file, llm_search, list_files）下从研究想法发展到论文。每个想法都有一个专用工作区，智能体可以在其中生成和阅读文档作为上下文的一部分。他们在三个领域（世界模型、多智能体 RL、AI 安全与对齐）进行了实验，每个领域包含 45-50 份高质量种子文档以激发新想法。只有四个想法被人类专家选中进行完整流水线运行，只有一个最终被完全执行成论文。他们在实验中观察到了六种反复出现的失败模式：

• Bias toward training-data defaults: use old libraries, stale commands, standard formats, or assumptions not grounded in the actual repository or dataset.偏向训练数据默认值：使用旧库、陈旧命令、标准格式或未扎根于实际仓库或数据集的假设。
• Implementation drift under execution pressure: when implementation becomes technically complex, the model may move toward a common simpler solution rather than the proposed method.执行压力下的实现漂移：当实现变得技术复杂时，模型可能会转向通用的更简单解决方案，而不是所提议的方法。
• Memory and context degradation: long-horizon projects lose critical details unless logs are written as persistent artifacts.内存和上下文退化：除非日志被写为持久化工件，否则长程项目会丢失关键细节。
• Over-optimism: the model declares success despite noisy or failed experiments, similarly observed as “p-hacking and eureka-ing” pattern by Bubeck et al. (2025) where models can introduce “numerical duct tape” and declare victory when signals are still noise.过度乐观：模型尽管实验嘈杂或失败仍宣布成功，类似于 Bubeck 等人（2025年）观察到的“p-hacking 和 eureka-ing”模式，其中模型可能会引入“数字胶带”并在信号仍是噪声时宣布胜利。
• Insufficient domain intelligence: the model lacks tacit craft knowledge, e.g. predicting implementation complexity, judging whether an experimental result is plausible, or knowing which baselines matter.领域智能不足：模型缺乏默会工艺知识，例如预测实现复杂性、判断实验结果是否合理，或知道哪些基线重要。
• Weak scientific taste: experiments may be executable but fail to answer the right question.科学品味薄弱：实验可能是可执行的，但未能回答正确的问题。

Toward full RSI, researchers have made real progress, but several bottlenecks remain.迈向完全 RSI，研究人员已经取得了真正的进展，但仍存在几个瓶颈。

1. Weak and fuzzy evaluators. Many research claims do not have a fast and precise verifier, and the same is true for many real-world tasks. Current self-improvement loops work best for tasks when evaluation metrics are measurable and objective, similar as how RL works.1. 弱且模糊的评估器。许多研究主张没有快速且精确的验证器，许多现实世界任务也是如此。当前的自我提升循环在评估指标可测量且客观的任务上效果最好，类似于 RL 的工作方式。

Research taste, novelty, and long-term scientific value are much harder to measure. For example, research taste often mixes problem framing, experimental design, and judgment about which surprising results are worth pursuing and which failure cases are worth retries.研究品味、新颖性和长期科学价值是极难衡量的。例如，研究品味往往混合了问题框架、实验设计，以及对哪些令人惊讶的结果值得追求、哪些失败案例值得重试的判断。

2. Context and memory lifecycle. Memory grows as AI agents become more autonomous and independent. A useful harness needs to manage context and memory to complement existing limitation in long-context generation while still maximizing the success of long-horizon tasks. Since humans are able to maintain memory through our life time, I see an anoloy here that context engineering will and should become a core part of intelligence, rather than staying in the software system layer.2. 上下文与记忆生命周期。随着 AI 智能体变得越来越自主和独立，记忆的需求也在增长。一个有用的工具框架（harness）需要管理上下文和记忆，以弥补现有长上下文生成技术的局限性，同时最大限度地确保长程任务的成功。由于人类能够在整个生命周期中保持记忆，我从中看到了一个类比：上下文工程将成为智能的核心部分，并理应如此，而不是仅仅停留在软件系统层。

3. Negative results. Researchers are incentivized to publish successful results and thus literature is biased toward successes. LLMs trained on a vast amount of data (mostly human created, at least for now, lol) may be bad at deciding when to abandon a hypothesis, report a negative result, or even acknowledge a failure due to the imablance of success vs failure cases in data. A research harness should make failed attempts easy to preserve, as learning from failure is the best way to trim down the task search space.3. 负面结果。研究人员倾向于发表成功的结果，因此文献对成功存在偏见。在海量数据（目前大部分由人类创造，哈哈）上训练的 LLM 可能不擅长决定何时放弃假设、报告负面结果，甚至承认失败，这是因为数据中成功案例与失败案例的比例失衡。研究工具框架应该让失败的尝试易于保存，因为从失败中学习是缩小任务搜索空间最好的方法。

4. Diversity collapse. Evolutionary and RL loops tend to exploit known high-reward patterns. We need mechanisms to prevent the population from collapsing into variants of the same solution. This is especially critical for open-ended research, where the best path may initially look worse under the current evaluator.4. 多样性崩溃。进化和强化学习循环倾向于利用已知的、高回报的模式。我们需要机制来防止群体崩溃为同一解决方案的变体。这对于开放式研究尤为关键，因为在当前的评估器下，最佳路径最初看起来可能更差。

5. Reward hacking. A self-improvement loop optimizes whatever signal it is given. If the reward comes from unit tests, the agent may overfit to tests; if it comes from a judge model, it may learn reward hacking tricks specific to this judge; if it comes from benchmark scores, it may exploit benchmark artifacts.5. 奖励作弊（Reward hacking）。自我改进循环会优化它被赋予的任何信号。如果奖励来自单元测试，智能体可能会对测试过拟合；如果来自评判模型，它可能会学习针对该评判模型的奖励作弊技巧；如果来自基准测试分数，它可能会利用基准测试的漏洞。

The evaluator and permission control should likely sit outside the loop that evolves harness, with held-out tests, trace audits, and human review at decision points that matter—how much oversight can be scaled up and automated remains an open research area.评估器和权限控制可能应该置于进化工具框架的循环之外，通过预留测试集、追踪审计，以及在关键决策点进行人工审查——如何扩展和自动化这种监督仍然是一个开放的研究领域。

6. Long-term success. An extrinsic loop of optimization works on rewards outside of individual rollouts that we can simulate in training sandbox.6. 长期成功。外在的优化循环作用于我们在训练沙箱中可以模拟的单个执行之外的奖励。

Take coding agent as an example. Coding agents have already increased daily productivity in software engineering, but many optimization goals are still too short-term. It can often complete the task at hand, but less obvious how it should protect the long-term health of a repo collectively maintained by hundreds or thousands of engineers. Standard sandbox-based RLVR-style training rarely captures maintainability, ownership boundaries, migration cost, backwards compatibility, or future debugging burden.以编码智能体为例。编码智能体已经提高了软件工程的日常生产力，但许多优化目标仍然过于短期。它通常可以完成手头的任务，但如何保护由成百上千名工程师共同维护的代码库的长期健康，尚不明确。标准的基于沙箱的 RLVR 式训练很少能捕捉到可维护性、所有权边界、迁移成本、向后兼容性或未来的调试负担。

7. The role of humans. Humans should move up the stack, not be removed from the loop, meaning that human should provide oversight at the right time, at the right abstraction level and our system design should consider when and how to set up such touch points.7. 人类的角色。人类应该向上移动到更高的层级，而不是被排除在循环之外。这意味着人类应该在合适的时机、合适的抽象层级提供监督，而我们的系统设计应该考虑何时以及如何设置这些交互点。

Many challenges listed above need human’s feedback and steering. After all, we are building the technology for better future of humanity, not other way around.上面列出的许多挑战都需要人类的反馈和引导。毕竟，我们构建这项技术是为了人类更美好的未来，而不是相反。

Citation引用#

Please cite this work as:请按如下方式引用本工作：

Weng, Lilian. “Harness Engineering for Self-Improvement”. Lil’Log (Jul 2026). https://lilianweng.github.io/posts/2026-07-04-harness/Weng, Lilian. “Harness Engineering for Self-Improvement”. Lil’Log (Jul 2026). https://lilianweng.github.io/posts/2026-07-04-harness/

Or use the BibTeX citation:或者使用 BibTeX 引用：

@article{weng2026harness,
  title = {Harness Engineering for Self-Improvement},
  author = {Weng, Lilian},
  journal = {lilianweng.github.io},
  year = {2026},
  month = {July},
  url = "https://lilianweng.github.io/posts/2026-07-04-harness/"
}
copy

Appendix: Some useful benchmarks附录：一些有用的基准测试#

• PaperBench: replicate 20 ICML 2024 Spotlight and Oral papers from scratch, including understanding paper contributions, developing a codebase, and successfully executing experiments.
  • Each replication task is decomposed into smaller, individually gradable tasks.每个复现任务都被分解为更小、可单独评分的任务。
  • 8,316 rubrics in total, co-developed with the paper authors.共有 8,316 个评分标准，与论文作者共同开发。
  • The best model at the time (Claude 3.5 Sonnet, ~21%) does not outperform ML PhDs.当时表现最好的模型（Claude 3.5 Sonnet，约 21%）并未超过机器学习博士。
  • Includes PaperBench, PaperBench Code-Dev (a lighter version), and JudgeEval.包括 PaperBench、PaperBench Code-Dev（轻量版）和 JudgeEval。
• CORE-Bench: evaluate computational reproducibility of published research.
  • 270 tasks based on 90 scientific papers across computer science, social science, and medicine.基于计算机科学、社会科学和医学领域 90 篇科学论文的 270 个任务。
  • Tasks involve reproducing results from provided code and data.任务涉及复现所提供代码和数据中的结果。
  • Includes multiple difficulty levels and both language-only and vision-language tasks.包括多个难度级别，以及纯语言任务和视觉语言任务。
  • The best reported agent at the time (GPT-4o and GPT-4o-mini) achieved only 21% accuracy on the hardest task.当时报道表现最好的智能体（GPT-4o 和 GPT-4o-mini）在最难的任务上仅达到 21% 的准确率。
• ScienceAgentBench: evaluate LLM agents for data-driven scientific discovery.
  • Extracts 102 tasks from 44 peer-reviewed publications in four disciplines (math, chemistry, biology, geography).从四个学科（数学、化学、生物学、地理学）的 44 篇同行评审出版物中提取了 102 个任务。
  • Covers basic data-science tasks in these domains: data processing, model development, data analysis, and information visualization.涵盖了这些领域中基础的数据科学任务：数据处理、模型开发、数据分析和信息可视化。
• RE-Bench: evaluate frontier AI agents on realistic ML research-engineering envs against human experts.
  • 7 challenging, open-ended ML research-engineering environments.7 个具有挑战性的开放式机器学习研究工程环境。
  • Each environment = (scoring function, starting solution, reference solution); each can be run with 8 or fewer H100 GPUs.每个环境 = (评分函数, 起始解决方案, 参考解决方案)；每个环境均可在 8 个或更少的 H100 GPU 上运行。
  • Examples: optimize a kernel, run a scaling-law experiment, fix an embedding, fine-tune GPT-2 for QA, etc.示例：优化内核、运行缩放定律实验、修复嵌入、微调用于问答的 GPT-2 等。
  • Includes data from 71 eight-hour attempts by 61 distinct human experts.包含来自 61 位不同人类专家进行的 71 次八小时尝试的数据。
  • Human experts achieved non-zero score in 82% of 8-hour attempts; 24% matched or exceeded strong reference solutions.人类专家在 82% 的 8 小时尝试中获得了非零分数；24% 的尝试匹配或超过了强大的参考解决方案。
  • Best AI agents scored 4× higher than humans at a 2-hour budget, but humans had better returns to longer budgets and exceeded agents at 8-hour and 32-hour settings.最好的 AI 智能体在 2 小时预算下得分比人类高出 4 倍，但人类在更长的预算下有更好的回报，并在 8 小时和 32 小时的设置中超过了智能体。
• MLE-bench: evaluate ML engineering agents on offline Kaggle competitions.
  • Contains 75 ML-engineering competitions curated from Kaggle.包含 75 个从 Kaggle 精选的机器学习工程竞赛。
  • Tests training models, preparing datasets, running experiments, and submitting predictions to grading scripts.测试训练模型、准备数据集、运行实验以及提交预测结果给评分脚本。
  • Uses Kaggle public leaderboards as human baselines.使用 Kaggle 公开排行榜作为人类基准。
  • Best setup in the paper, o1-preview with AIDE scaffolding, reached at least Kaggle bronze-medal level in 16.9% of competitions.论文中表现最好的配置（o1-preview 配合 AIDE 脚手架）在 16.9% 的竞赛中达到了至少 Kaggle 铜牌水平。
  • Includes resource-scaling and contamination analyses.包括资源扩展和污染分析。
• KernelBench: evaluate correctness and speed for generated GPU kernels.
  • 250 PyTorch tasks to evaluate whether LLM can write fast and correct kernels.250 个 PyTorch 任务，用于评估 LLM 是否能编写快速且正确的内核。
  • The evaluation metric fast_p = the percentage of generated kernels that are correct and faster than baseline.评估指标 fast_p = 正确且比基准更快的生成内核的百分比。

References参考文献#

[1] Good, I. J. “Speculations Concerning the First Ultraintelligent Machine.” Advances in Computers, 6:31–88, 1965.[1] Good, I. J. “Speculations Concerning the First Ultraintelligent Machine.” Advances in Computers, 6:31–88, 1965.

[2] Yudkowsky, Eliezer. “Recursive Self-Improvement.” LessWrong, 2008.[2] Yudkowsky, Eliezer. “Recursive Self-Improvement.” LessWrong, 2008.

[3] Choi, et al. “Anchored Self-Play for Code Repair.” ICML 2026.[3] Choi, et al. “Anchored Self-Play for Code Repair.” ICML 2026.

[4] Zhao, et al. “Absolute Zero: Reinforced Self-play Reasoning with Zero Data.” arXiv preprint arXiv:2505.03335, 2025.[4] Zhao, et al. “Absolute Zero: Reinforced Self-play Reasoning with Zero Data.” arXiv preprint arXiv:2505.03335, 2025.

[5] Yuan, et al. “Self-Rewarding Language Models.” arXiv preprint arXiv:2401.10020, 2024.[5] Yuan, et al. “Self-Rewarding Language Models.” arXiv preprint arXiv:2401.10020, 2024.

[6] Chen, et al. “Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models.” ICML 2024.[6] Chen, et al. “Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models.” ICML 2024.

[7] Zhang, et al. “Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models.” ICLR 2026.[7] Zhang, et al. “Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models.” ICLR 2026.

[8] Ye, et al. “Meta Context Engineering via Agentic Skill Evolution.” arXiv preprint arXiv:2601.21557, 2026.[8] Ye, et al. “Meta Context Engineering via Agentic Skill Evolution.” arXiv preprint arXiv:2601.21557, 2026.

[9] Lee, et al. “Meta-Harness: End-to-End Optimization of Model Harnesses.” arXiv preprint arXiv:2603.28052, 2026.[9] Lee, et al. “Meta-Harness: End-to-End Optimization of Model Harnesses.” arXiv preprint arXiv:2603.28052, 2026.

[10] Lu, et al. “Towards end-to-end automation of AI research.” Nature, 651:914–919, 2026.[10] Lu, et al. “Towards end-to-end automation of AI research.” Nature, 651:914–919, 2026.

[11] Meng, et al. “ScientistOne: Towards Human-Level Autonomous Research via Chain-of-Evidence.” arXiv preprint arXiv:2605.26340, 2026.[11] Meng, et al. “ScientistOne: Towards Human-Level Autonomous Research via Chain-of-Evidence.” arXiv preprint arXiv:2605.26340, 2026.

[12] Kulikov, et al. “Autodata: An agentic data scientist to create high quality synthetic data.” arXiv preprint arXiv:2606.25996, 2026.[12] Kulikov, et al. “Autodata: An agentic data scientist to create high quality synthetic data.” arXiv preprint arXiv:2606.25996, 2026.

[13] Hu, Lu, and Clune. “Automated Design of Agentic Systems.” ICLR 2025.[13] Hu, Lu, and Clune. “Automated Design of Agentic Systems.” ICLR 2025.

[14] Madaan, et al. “Self-Refine: Iterative Refinement with Self-Feedback.” NeurIPS 2023.[14] Madaan, et al. “Self-Refine: Iterative Refinement with Self-Feedback.” NeurIPS 2023.

[15] Zhang, et al. “AFlow: Automating Agentic Workflow Generation.” ICLR 2025.[15] Zhang, et al. “AFlow: Automating Agentic Workflow Generation.” ICLR 2025.

[16] Zelikman, et al. “Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generation.” COLM 2024.[16] Zelikman, et al. “Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generation.” COLM 2024.

[17] Zhang, et al. “Self-Harness: Harnesses That Improve Themselves.” arXiv preprint arXiv:2606.09498, 2026.[17] Zhang, et al. “Self-Harness: Harnesses That Improve Themselves.” arXiv preprint arXiv:2606.09498, 2026.

[18] Fernando, et al. “Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution.” arXiv preprint arXiv:2309.16797, 2023.[18] Fernando, et al. “Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution.” arXiv preprint arXiv:2309.16797, 2023.

[19] Agrawal, A. et al. “GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning.” arXiv preprint arXiv:2507.19457, 2025.[19] Agrawal, A. et al. “GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning.” arXiv preprint arXiv:2507.19457, 2025.

[20] Novikov, et al. “AlphaEvolve: A coding agent for scientific and algorithmic discovery.” arXiv preprint arXiv:2506.13131, 2025.[20] Novikov, et al. “AlphaEvolve: A coding agent for scientific and algorithmic discovery.” arXiv preprint arXiv:2506.13131, 2025.

[21] Lange, Imajuku, and Cetin. “ShinkaEvolve: Towards Open-Ended And Sample-Efficient Program Evolution.” arXiv preprint arXiv:2509.19349, 2025.[21] Lange, Imajuku, and Cetin. “ShinkaEvolve: Towards Open-Ended And Sample-Efficient Program Evolution.” arXiv preprint arXiv:2509.19349, 2025.

[22] Wang, et al. “ThetaEvolve: Test-time Learning on Open Problems.” arXiv preprint arXiv:2511.23473, 2025.[22] Wang, et al. “ThetaEvolve: Test-time Learning on Open Problems.” arXiv preprint arXiv:2511.23473, 2025.

[23] Zhang, et al. “Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents.” arXiv preprint arXiv:2505.22954, 2025.[23] Zhang, et al. “Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents.” arXiv preprint arXiv:2505.22954, 2025.

[24] Zhang, et al. “Hyperagents.” arXiv preprint arXiv:2603.19461, 2026.[24] Zhang, et al. “Hyperagents.” arXiv preprint arXiv:2603.19461, 2026.

[25] Yuksekgonul, et al. “Learning to Discover at Test Time.” arXiv preprint arXiv:2601.16175, 2026.[25] Yuksekgonul, et al. “Learning to Discover at Test Time.” arXiv preprint arXiv:2601.16175, 2026.

[26] Riaz, et al. “Epistemic Uncertainty for Test-Time Discovery.” arXiv preprint arXiv:2605.11328, 2026.[26] Riaz, et al. “Epistemic Uncertainty for Test-Time Discovery.” arXiv preprint arXiv:2605.11328, 2026.

[27] Hebbar, et al. “SIA: Self Improving AI with Harness & Weight Updates.” arXiv preprint arXiv:2605.27276, 2026.[27] Hebbar, et al. “SIA: Self Improving AI with Harness & Weight Updates.” arXiv preprint arXiv:2605.27276, 2026.

[28] Trehan and Chopra. “Why LLMs Aren’t Scientists Yet: Lessons from Four Autonomous Research Attempts.” arXiv preprint arXiv:2601.03315, 2026.[28] Trehan and Chopra. “Why LLMs Aren’t Scientists Yet: Lessons from Four Autonomous Research Attempts.” arXiv preprint arXiv:2601.03315, 2026.

[29] Bubeck, et al. “Early science acceleration experiments with GPT-5.” arXiv preprint arXiv:2511.16072, 2025.[29] Bubeck, et al. “Early science acceleration experiments with GPT-5.” arXiv preprint arXiv:2511.16072, 2025.

[30] Starace, et al. “PaperBench: Evaluating AI’s Ability to Replicate AI Research.” ICML 2025.[30] Starace, et al. “PaperBench: Evaluating AI’s Ability to Replicate AI Research.” ICML 2025.

[31] Wijk, et al. “RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts.” ICML 2025.[31] Wijk, et al. “RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts.” ICML 2025.

[32] Chan, et al. “MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering.” arXiv preprint arXiv:2410.07095, 2024.[32] Chan, et al. “MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering.” arXiv preprint arXiv:2410.07095, 2024.

[33] Chen, et al. “ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery.” ICLR 2025.[33] Chen, et al. “ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery.” ICLR 2025.

[34] Siegel, et al. “CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark.” TMLR 2024.[34] Siegel, et al. “CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark.” TMLR 2024.

[35] Ouyang, et al. “KernelBench: Can LLMs Write Efficient GPU Kernels?” arXiv preprint arXiv:2502.10517, 2025.[35] Ouyang, et al. “KernelBench: Can LLMs Write Efficient GPU Kernels?” arXiv preprint arXiv:2502.10517, 2025.