Building an Internal Coding Agent at Zup:
Lessons and Open Questions在Zup构建内部编码代理：
经验教训与开放问题
Abstract摘要
Enterprise teams building internal coding agents face a gap between prototype performance and production readiness. The root cause is that technical model quality alone is insufficient—tool design, safety enforcement, state management, and human trust calibration are equally decisive, yet underreported in the literature. We present CodeGen, an internal coding agent at Zup, and show that targeted tool design (e.g., string-replacement edits over full-file rewrites) and layered safety guardrails improved agent reliability more than prompt engineering, while progressive human oversight modes drove organic adoption without mandating trust. These findings suggest that the engineering decisions surrounding the model—not the model itself—determine whether a coding agent delivers real value in practice.企业团队在构建内部编码代理时，面临着从原型性能到生产就绪之间的差距。根本原因在于，仅凭技术模型质量是不够的——工具设计、安全执行、状态管理和人类信任校准同样具有决定性作用，然而这些方面在文献中报道不足。我们介绍了Zup的内部编码代理CodeGen，并表明针对性的工具设计（例如，字符串替换编辑而非全文件重写）和分层安全护栏比提示工程更能提高代理的可靠性，而渐进式人工监督模式在不强制要求信任的情况下推动了有机采用。这些发现表明，围绕模型的工程决策——而非模型本身——决定了编码代理在实践中能否带来真正的价值。
1 Introduction1 引言
Large language models (LLMs) are increasingly being integrated into software development workflows through coding agents—systems that go beyond token-level code completion to autonomously read files, edit source code, execute shell commands, and iterate on their own outputs [11, 2]. Tools such as GitHub Copilot [6], Cursor, and Claude Code have demonstrated that LLM-based agents can accelerate routine development tasks, and a growing number of organizations are now exploring how to build or customize such agents for internal use [9].大型语言模型（LLM）正越来越多地通过编码代理——即超越令牌级代码补全，能够自主读取文件、编辑源代码、执行 shell 命令并迭代自身输出的系统——被集成到软件开发工作流中 [11, 2]。诸如 GitHub Copilot [6]、Cursor 和 Claude Code 等工具已证明，基于 LLM 的代理可以加速日常开发任务，越来越多的组织正在探索如何构建或定制此类代理以供内部使用 [9]。
However, building a coding agent that performs well on benchmarks or in isolated demonstrations is fundamentally different from deploying one that developers actually use in their daily work. In enterprise settings, agents must operate on real codebases with complex build systems, interact with internal tooling and CI/CD pipelines, and earn the trust of developers who are accountable for the code the agent produces. For example, an agent that rewrites entire files risks introducing subtle truncation errors in large codebases, while an agent with unrestricted shell access can execute destructive commands such as rm -rf or unauthorized git push --force on a developer’s machine. These are not hypothetical risks—they are failure modes that emerged repeatedly during the development of the system described in this paper111https://github.com/openai/codex/issues/3934.然而，构建一个在基准测试或孤立演示中表现良好的编码代理，与部署一个开发者实际在日常工作中使用的代理有着根本区别。在企业环境中，代理必须在具有复杂构建系统的真实代码库上运行，与内部工具和 CI/CD 流水线交互，并赢得对代理生成的代码负责的开发者的信任。例如，一个重写整个文件的代理可能会在大型代码库中引入微妙的截断错误，而一个具有无限制 shell 访问权限的代理可能会在开发者的机器上执行破坏性命令，如 rm -rf 或未经授权的 git push --force。这些并非假设性风险——它们是本文所述系统开发过程中反复出现的故障模式111https://github.com/openai/codex/issues/3934。
When these engineering challenges are not addressed, there are consequences such as: 1) agents produce unreliable edits that developers must manually verify and correct, 2) safety incidents erode trust and stall adoption, and 3) teams invest months building prototypes that never transition to production use. In a landscape where multiple organizations are simultaneously building internal coding agents, the absence of shared knowledge about these practical challenges means that engineering teams are likely rediscovering the same failure modes independently—wasting effort that could be directed toward advancing the state of the practice.当这些工程挑战未被解决时，会产生以下后果：1) 智能体产生不可靠的编辑，开发者必须手动验证和纠正；2) 安全事故侵蚀信任并阻碍采用；3) 团队投入数月构建原型，却从未投入生产使用。在多个组织同时构建内部编码智能体的背景下，缺乏关于这些实际挑战的共享知识意味着工程团队很可能独立地重新发现相同的失败模式——浪费了本可用于推进实践发展的精力。
Existing literature on LLM-based coding agents has focused primarily on model-level performance—benchmark evaluations [3, 12], prompt engineering techniques, and reasoning strategies [15, 14]. Little has been reported about the engineering decisions that determine whether a coding agent succeeds in production: how tools should be designed and specified for LLM consumption [10, 8], how safety policies should be enforced across tools with overlapping capabilities, how session state should be managed for resilience, and how human oversight should be structured to enable adoption without mandating blind trust [5]. These dimensions are not secondary concerns; in our experience, they proved more consequential than model selection or prompt optimization for the system’s real-world effectiveness.现有关于基于LLM的编码智能体的文献主要关注模型层面的性能——基准评估[3, 12]、提示工程技术以及推理策略[15, 14]。很少有关于决定编码智能体能否在生产中成功的工程决策的报道：如何为LLM消费设计和指定工具[10, 8]，如何在具有重叠功能的工具之间强制执行安全策略，如何管理会话状态以实现弹性，以及如何构建人类监督以在无需强制盲目信任的情况下实现采用[5]。这些维度并非次要问题；根据我们的经验，它们对系统在现实世界中的有效性比模型选择或提示优化更为关键。
This paper addresses this gap by presenting the key architectural and engineering decisions behind CodeGen, an internal coding agent developed and deployed at Zup Innovation222https://www.zup.com.br, for use by enterprise software development teams. We focus on four core dimensions that shaped the system’s evolution: model strategy (how reasoning is delegated to the LLM), framework strategy (when to build manually vs. adopt orchestration frameworks), tooling strategy (how tools are designed, specified, and constrained), and human oversight strategy (how trust is calibrated during adoption).本文通过介绍CodeGen背后的关键架构和工程决策来填补这一空白，CodeGen是Zup Innovation（https://www.zup.com.br）内部开发并部署的编码智能体，供企业软件开发团队使用。我们聚焦于塑造系统演进的四个核心维度：模型策略（如何将推理委托给LLM）、框架策略（何时手动构建与采用编排框架）、工具策略（如何设计、指定和约束工具）以及人类监督策略（如何在采用过程中校准信任）。
Among the key findings: (1) refining tool descriptions, parameter schemas, and error contracts produced more consistent improvements in agent reliability than prompt engineering alone; (2) safety guardrails must be enforced holistically across the entire tool manifest, since restricting one tool is ineffective when equivalent capabilities are accessible through another; (3) a progressive human oversight model—where developers begin in approval mode and organically migrate to autonomous mode—proved critical for enterprise adoption; and (4) implementing the agentic loop manually before adopting frameworks gave the team the understanding necessary to evaluate later abstractions on their merits, a strategy validated when modern frameworks converged on the same design.主要发现包括：(1) 优化工具描述、参数模式和错误契约比单纯的提示工程更能持续提升智能体可靠性；(2) 安全护栏必须在整个工具清单中整体实施，因为当等效功能可通过其他工具访问时，限制单一工具是无效的；(3) 渐进式人类监督模型——开发者从审批模式开始，有机地迁移到自主模式——对企业采用至关重要；(4) 在采用框架之前手动实现智能体循环，使团队能够理解后续抽象的价值，这一策略在现代框架收敛于相同设计时得到了验证。
The contributions of this paper are:本文的贡献如下：
-
•
A description of CodeGen’s architecture—a three-tier system comprising a CLI executor, a FastAPI backend, and a centralized orchestration engine (Maestro)—including its state management, resilience mechanisms, and deployment pipeline.CodeGen架构的描述——一个三层系统，包括CLI执行器、FastAPI后端和集中式编排引擎（Maestro），涵盖其状态管理、弹性机制和部署流水线。
-
•
A catalog of 13 concrete design decisions across framework selection, tool design, safety enforcement, and human oversight, each accompanied by the trade-off that motivated it and the lesson derived from it.包含13个具体设计决策的目录，涉及框架选择、工具设计、安全执行和人工监督，每个决策都附带了其动机权衡和从中获得的经验教训。
-
•
An analysis of how tool specification quality, cross-tool safety policy consistency, and progressive trust calibration affect agent reliability and adoption in enterprise environments.分析工具规范质量、跨工具安全策略一致性以及渐进式信任校准如何影响企业环境中代理的可靠性和采用率。
-
•
Six open questions on tool manifest design, reasoning boundaries, cross-tool safety enforcement, adaptive trust models, long-term agent memory, and quality assurance for agent-generated code.关于工具清单设计、推理边界、跨工具安全执行、自适应信任模型、长期代理记忆以及代理生成代码质量保证的六个开放问题。
2 Origins of CodeGen2 CodeGen的起源
In early 2024, enterprise AI coding tools were shifting from passive assistants to autonomous agents. Products such as Devin and Cursor attracted significant attention for their ability to not only suggest code but also execute multi-step tasks autonomously—reading files, running commands, and iterating on their own output. For Zup, this shift had immediate strategic implications. The company’s flagship developer platform, StackSpot AI [7], offered IDE extensions that operated as conversational assistants: developers could ask questions and receive code suggestions, but the extensions could not take actions in the environment.2024年初，企业AI编码工具正从被动助手转向自主代理。Devin和Cursor等产品因其不仅能建议代码，还能自主执行多步骤任务——读取文件、运行命令并迭代自身输出——而备受关注。对于Zup来说，这一转变具有直接的战略意义。该公司旗舰开发者平台StackSpot AI [7]提供的IDE扩展作为对话助手运行：开发者可以提问并获得代码建议，但扩展无法在环境中执行操作。
An initial assessment revealed that adding agentic capabilities—file editing, command execution, iterative tool-calling loops—to StackSpot AI’s enterprise codebase would require substantial changes across the IDE extensions, the backend, and the model integration layer. The platform’s roadmap, oriented toward a broader agent infrastructure rather than code generation specifically, could not absorb this scope. Two members of the development team decided to build a proof of concept independently. The goal was to construct a minimal but functional agentic system—a CLI-based coding agent backed by a lightweight orchestration backend—and demonstrate that the agentic pattern could work within Zup’s infrastructure. The POC was deliberately scoped outside the StackSpot AI codebase to avoid dependency on its release cycle. This independence freed agentic from the platform’s backlog constraints, the team could iterate rapidly, make opinionated design choices, and validate ideas with real developer usage in days rather than quarters. What began as a side project gradually evolved into CodeGen, a production ready agentic-system used every day by real developers.初步评估显示，为StackSpot AI的企业代码库添加代理能力——文件编辑、命令执行、迭代工具调用循环——需要对IDE扩展、后端和模型集成层进行重大更改。该平台路线图侧重于更广泛的代理基础设施而非专门的代码生成，无法吸收这一范围。开发团队的两名成员决定独立构建一个概念验证。目标是构建一个最小但功能完整的代理系统——一个基于CLI的编码代理，由轻量级编排后端支持——并证明代理模式可以在Zup的基础设施中工作。该POC特意限定在StackSpot AI代码库之外，以避免依赖其发布周期。这种独立性使代理系统摆脱了平台积压的限制，团队可以快速迭代，做出有主见的设计选择，并在数天而非数季度内通过真实开发者使用来验证想法。最初作为副项目的系统逐渐演变为CodeGen，一个每天被真实开发者使用的生产级代理系统。
3 CodeGen Internals3 CodeGen内部机制
CodeGen is an internal coding agent developed at Zup that assists software developers by autonomously performing code-related tasks such as editing source files and executing shell commands. Unlike token-level code-completion tools [1], CodeGen operates as a fully agentic system grounded in the ReAct (Reasoning + Acting) paradigm [15]: given a natural-language prompt, it reasons about the task, selects and invokes tools, observes their outputs, and iterates until the task is complete. The system delegates internal reasoning to a LLM while providing the structural scaffold—context assembly, iterative tool dispatch, and output re-injection—that enables the model to act over successive observations. A configurable thinking effort parameter controls reasoning depth, and a dedicated planning mode was introduced to allow developers to review a proposed action sequence before execution begins.CodeGen 是 Zup 内部开发的一个编码代理，通过自主执行代码相关任务（如编辑源文件和执行 shell 命令）来协助软件开发人员。与基于 token 的代码补全工具 [1] 不同，CodeGen 作为一个完全自主的系统，基于 ReAct（推理+行动）范式 [15] 运行：给定自然语言提示，它推理任务、选择并调用工具、观察输出，并迭代直到任务完成。系统将内部推理委托给 LLM，同时提供结构框架——上下文组装、迭代工具调度和输出重新注入——使模型能够在连续观察中行动。可配置的思考努力参数控制推理深度，并引入了专用规划模式，允许开发人员在执行前审查提议的操作序列。
3.1 Architecture3.1 架构
CodeGen’s architecture comprises three components, organized as follow: a CLI interface, a backend API, and a maestro component.CodeGen 的架构由三个组件组成，组织如下：CLI 接口、后端 API 和 maestro 组件。
The first component is the CLI (Node.js client), responsible for user interaction and local tool execution. CodeGen was designed as a CLI primarily for portability: since all major IDEs—VSCode, IntelliJ, Visual Studio, among others—already include built-in terminal emulators, a single CLI serves as a unified interface across the diverse set of development environments used within the enterprise. Building native IDE extensions would have required developing and maintaining a separate plugin for each IDE, multiplying engineering effort without proportional benefit. A CLI also enabled tighter integration with the enterprise’s internal tools and workflows, offering a level of customization and execution control that off-the-shelf, general-purpose coding agents could not match.第一个组件是 CLI（Node.js 客户端），负责用户交互和本地工具执行。CodeGen 设计为 CLI 主要是出于可移植性考虑：由于所有主流 IDE——VSCode、IntelliJ、Visual Studio 等——都已包含内置终端模拟器，单个 CLI 可作为企业内使用的多样化开发环境的统一接口。构建原生 IDE 扩展需要为每个 IDE 开发和维护单独的插件，这会成倍增加工程工作量而收益不成比例。CLI 还实现了与企业内部工具和工作流的更紧密集成，提供了现成的通用编码代理无法匹敌的定制化和执行控制水平。
A CLI also enabled tighter integration with the enterprise’s internal tools and workflows, offering a level of customization and execution control that off-the-shelf, general-purpose coding agents could not match. Although the CLI is the primary interface, the architecture supports multiple client types—including IDE plugins and VM-based executors—through the same communication protocol.CLI 还实现了与企业内部工具和工作流的更紧密集成，提供了现成的通用编码代理无法匹敌的定制化和执行控制水平。尽管 CLI 是主要接口，但架构通过相同的通信协议支持多种客户端类型——包括 IDE 插件和基于 VM 的执行器。
The second component is the Backend API, built with FastAPI and responsible for maintaining connectivity with clients, handling authentication, and routing requests. The backend exposes a REST API for task lifecycle management (task creation, log retrieval, user-initiated actions) and supports two real-time communication channels: WebSocket connections for executor clients (CLI, IDE plugins) that require bidirectional tool dispatch, and Server-Sent Events (SSE) for web portal interfaces that consume task updates as a read-only stream. This dual-channel design allows different client types to interact with the system through the protocol best suited to their needs.第二个组件是后端 API，使用 FastAPI 构建，负责维护与客户端的连接、处理身份验证和路由请求。后端公开 REST API 用于任务生命周期管理（任务创建、日志检索、用户发起的操作），并支持两个实时通信通道：用于需要双向工具调度的执行器客户端（CLI、IDE 插件）的 WebSocket 连接，以及用于消费任务更新作为只读流的 Web 门户界面的服务器发送事件（SSE）。这种双通道设计允许不同类型的客户端通过最适合其需求的协议与系统交互。
The third component is the Maestro, the orchestration engine that controls the agentic loop. Upon initialization, a bootstrap phase collects environment metadata—operating system, working directory, recent git history, project structure, etc. When the user issues a prompt, the backend sends the LLM a payload containing the system prompt, conversation history, bootstrap metadata, and a tool manifest describing all available tools and their parameter schemas. The LLM either responds with text or requests a tool call; in the latter case, the Maestro relays the invocation to the CLI via WebSocket, collects the result, and feeds it back to the model. This cycle repeats until the model emits a final textual response instead of a tool call, at which point the loop terminates.第三个组件是 Maestro，即控制代理循环的编排引擎。初始化时，引导阶段收集环境元数据——操作系统、工作目录、最近的 git 历史、项目结构等。当用户发出提示时，后端向 LLM 发送包含系统提示、对话历史、引导元数据和描述所有可用工具及其参数模式的工具清单的有效负载。LLM 要么以文本响应，要么请求工具调用；在后一种情况下，Maestro 通过 WebSocket 将调用转发给 CLI，收集结果，并将其反馈给模型。此循环重复，直到模型发出最终文本响应而非工具调用，此时循环终止。
3.2 State Management and Resilience3.2 状态管理与弹性
Session state is persisted in PostgreSQL (via asyncpg for asynchronous access), with Redis serving as a caching layer and messaging backbone. Redis Streams are used for event distribution, enabling horizontal scaling of the backend in multi-tenant deployments. Session memory uses Redis with a 24-hour TTL, with PostgreSQL as a fallback when Redis entries expire. WebSocket sessions are dropped after 20 minutes of inactivity to conserve server resources. Importantly, the system supports task reconnection: if a client disconnects due to network instability or inactivity, it can resume an in-progress task without losing prior context—a feature that proved essential for long-running coding tasks in enterprise environments where network interruptions are common.会话状态持久化在PostgreSQL中（通过asyncpg实现异步访问），Redis作为缓存层和消息传递骨干。Redis Streams用于事件分发，支持多租户部署中后端的水平扩展。会话内存使用Redis，TTL为24小时，当Redis条目过期时，PostgreSQL作为后备。WebSocket会话在20分钟无活动后断开，以节省服务器资源。重要的是，系统支持任务重连：如果客户端因网络不稳定或无活动而断开连接，它可以恢复正在进行的任务而不会丢失之前的上下文——这一功能在企业环境中对长时间运行的编码任务至关重要，因为网络中断很常见。
The backend maintains a durable event timeline for every task execution, recording each tool invocation, model response, and state transition. This audit trail serves four purposes: debugging failed agent runs, providing analytics on agent behavior and tool usage patterns, satisfying compliance requirements in regulated enterprise environments, and enabling iterative self-improvement of the system. By analyzing execution logs, the team can identify recurring tool invocation errors that suggest unclear parameter schemas, detect inefficient workflows where the agent makes redundant tool calls, and observe behavioral patterns that inform refinements to tool descriptions and system prompts–a feedback loop that has proven more effective for improving agent reliability than prompt tuning in isolation.后端为每个任务执行维护一个持久的事件时间线，记录每次工具调用、模型响应和状态转换。这个审计追踪有四个目的：调试失败的代理运行、提供代理行为和工具使用模式的分析、满足受监管企业环境中的合规要求，以及实现系统的迭代自我改进。通过分析执行日志，团队可以识别出表明参数模式不清晰的重复工具调用错误，检测代理进行冗余工具调用的低效工作流，并观察行为模式，从而为工具描述和系统提示的改进提供信息——这一反馈循环已被证明比单独进行提示调优更能有效提高代理可靠性。
3.3 Tool Manifest3.3 工具清单
A central design decision is that tool design quality is a first-order determinant of agent effectiveness, often more impactful than prompt engineering alone. CodeGen exposes its set of tools to the LLM through a structured tool manifest—a declarative specification of each tool’s name, description, parameter schema, and expected behavior. Core tools include:一个核心设计决策是，工具设计质量是代理有效性的首要决定因素，通常比单独的提示工程影响更大。CodeGen通过结构化的工具清单向LLM公开其工具集——这是一个声明性规范，包含每个工具的名称、描述、参数模式和预期行为。核心工具包括：
-
•
read: retrieves file contents and enforces a read-before-edit policy to prevent stale-context errors.read：检索文件内容，并强制执行先读后改策略，以防止过时上下文错误。
-
•
edit: performs targeted string replacement rather than full-file rewriting, to mitigate LLM truncation failures when modifying large files.edit：执行目标字符串替换，而不是全文件重写，以减轻修改大文件时LLM截断失败的问题。
-
•
shell: executes terminal commands subject to multiple guardrail layers (command-level blocking, human approval mode, full audit logging).shell：执行终端命令，受多层防护（命令级阻止、人工审批模式、完整审计日志）。
All tool invocations are executed client-side on the developer’s machine, ensuring that the agent operates on the actual project state rather than a server-side copy. Two execution modes–approval mode and autonomous mode–allow teams to calibrate agent autonomy as trust develops. In approval mode, only potentially destructive tools (edit and shell) require explicit human confirmation before execution; read-only operations such as read file proceed without approval, since they carry no risk of modifying the environment and requiring confirmation for every file read would introduce prohibitive friction in practice.所有工具调用都在开发者的机器上客户端执行，确保代理操作的是实际项目状态，而不是服务器端副本。 两种执行模式——审批模式和自主模式——允许团队随着信任的发展校准代理自主性。在审批模式下，只有潜在破坏性工具（edit和shell）需要执行前的人工确认；只读操作（如读取文件）无需审批即可进行，因为它们没有修改环境的风险，如果每次文件读取都需要确认，在实践中会带来过高的摩擦。
4 Design Decisions4 设计决策
We group our design decisions in terms of (1) Architecture and Framework Decisions (Section 4.1), (2) Tool Design and Safety (Section 4.2), and (3) Human Oversight and Adoption (Section 4.3).我们将设计决策分为三类：(1) 架构与框架决策（第4.1节），(2) 工具设计与安全性（第4.2节），以及(3) 人工监督与采用（第4.3节）。
4.1 Architecture and Framework Decisions4.1 架构与框架决策
We start by describing seven architectural decisions.我们首先描述七项架构决策。
LangChain’s unidirectional chain model was inadequate for the cyclical, stateful interaction pattern required by an agentic loop.LangChain的单向链式模型无法满足代理循环所需的循环、有状态交互模式。
At project inception, the team experimented with LangChain333https://www.langchain.com/ as an orchestration framework. However, LangChain’s early abstractions were designed around linear chains of components—a unidirectional pipeline where each step feeds into the next. The agentic coding assistant required a cyclical interaction pattern in which the model requests a tool, the client executes it, and the result is fed back into the model repeatedly until the task is complete. This fundamental mismatch made it difficult to express the iterative loop within LangChain’s original API, forcing the team to work around the framework rather than with it. Framework adoption must be evaluated against the specific interaction pattern of the target system, not just its general popularity or feature set.项目初期，团队尝试使用LangChain333https://www.langchain.com/作为编排框架。然而，LangChain早期的抽象设计围绕线性组件链——一种单向管道，每个步骤依次输入下一步。代理编码助手需要一种循环交互模式：模型请求工具，客户端执行工具，结果反复反馈给模型直至任务完成。这种根本性的不匹配使得在LangChain原始API中表达迭代循环变得困难，迫使团队绕开框架工作而非利用框架。框架的采用必须根据目标系统的具体交互模式来评估，而不仅仅是其普遍流行度或功能集。
Manual implementation of the agentic loop provided more operational control and faster iteration than early framework adoption.手动实现代理循环比早期采用框架提供了更多的操作控制和更快的迭代速度。
After abandoning LangChain, the team implemented the agentic loop directly in project code—the Maestro component described in Section 4.1. This decision, while more labor-intensive upfront, gave the team explicit control over stop criteria, tool dispatch, WebSocket communication, and error propagation. Debugging was easier because there were no framework abstractions obscuring the execution flow. For novel or poorly understood interaction patterns, manual implementation can accelerate learning and provide clearer ownership of execution semantics—advantages that outweigh the convenience of a framework during early project phases.放弃LangChain后，团队直接在项目代码中实现了代理循环——即第4.1节描述的Maestro组件。这一决策虽然前期更耗费人力，但使团队能够明确控制停止条件、工具调度、WebSocket通信和错误传播。调试也更容易，因为没有框架抽象掩盖执行流程。对于新颖或理解不足的交互模式，手动实现可以加速学习并提供更清晰的执行语义所有权——这些优势在项目早期阶段超过了框架的便利性。
The team is now transitioning toward modern orchestration abstractions because the framework ecosystem has converged on what was already built manually.团队现在正转向现代编排抽象，因为框架生态系统已经收敛到手动构建的内容上。
As the agentic pattern matured and became widespread, frameworks evolved to support it natively. For isntance, LangChain introduced LangGraph444https://www.langchain.com/langgraph, offering first-class support for cyclical tool-calling loops. When the team evaluated these newer abstractions, they found that the design closely resembled what had already been built by hand. This convergence validated the original architectural choices and made the transition cost low. Building manually first and adopting frameworks later—once they mature to match actual requirements—can be a pragmatic strategy that avoids both premature abstraction and long-term maintenance burden.随着代理模式成熟并变得普遍，框架也进化以原生支持它。例如，LangChain引入了LangGraph444https://www.langchain.com/langgraph，为循环工具调用循环提供了一流支持。当团队评估这些较新的抽象时，发现其设计与手动构建的非常相似。这种趋同验证了最初的架构选择，并降低了迁移成本。先手动构建，待框架成熟以匹配实际需求后再采用——这是一种务实的策略，既避免了过早抽象，也避免了长期维护负担。
FastAPI was chosen for the backend due to its native async support.后端选择FastAPI是因为其原生异步支持。
The backend must (1) sustain concurrent WebSocket connections with multiple executor clients, (2) serve SSE streams to web portals, and (3) perform asynchronous database operations—all within a single service. FastAPI’s async-first design, combined with non-blocking PostgreSQL access (through asyncpg) and native WebSocket support, provided the concurrency model needed without introducing the complexity of a multi-process architecture. The choice of Redis Streams as the messaging backbone further supports horizontal scaling: as the number of concurrent agent sessions grows, additional backend instances can consume from the same stream group, enabling multi-tenant deployments without architectural changes.后端必须（1）维持与多个执行器客户端的并发WebSocket连接，（2）向Web门户提供SSE流，以及（3）在单个服务内执行异步数据库操作。FastAPI的异步优先设计，结合非阻塞的PostgreSQL访问（通过asyncpg）和原生WebSocket支持，提供了所需的并发模型，而无需引入多进程架构的复杂性。选择Redis Streams作为消息传递骨干进一步支持了水平扩展：随着并发代理会话数量的增长，额外的后端实例可以从同一个流组消费，从而实现多租户部署而无需架构变更。
Session-scoped memory already delivers substantial UX value even without long-term cross-session memory.即使没有长期的跨会话记忆，会话范围内的记忆已经提供了显著的用户体验价值。
Although cross-session long-term memory remains an area of ongoing development, the session-scoped approach—Redis with a 24-hour TTL, PostgreSQL fallback, and task reconnection support—already provides meaningful conversational continuity within a working session. The agent can recall prior context, tool results, and user instructions throughout an iterative task, and developers can resume interrupted sessions without starting over. A bounded, operationally simple memory layer can deliver significant user experience improvements without requiring the full complexity of persistent, long-term memory systems.尽管跨会话的长期记忆仍是一个持续发展的领域，但会话范围的方法——使用24小时TTL的Redis、PostgreSQL回退以及任务重连支持——已经在一个工作会话内提供了有意义的对话连续性。代理可以在迭代任务中回忆先前的上下文、工具结果和用户指令，开发者可以恢复中断的会话而无需从头开始。一个有边界、操作简单的记忆层可以在不需要持久化长期记忆系统的全部复杂性的情况下，带来显著的用户体验改进。
Internal reasoning is delegated to the LLM rather than being hand-coded in the orchestrator.内部推理被委托给LLM，而不是在编排器中手工编码。
Rather than implementing a custom reasoning engine within the Maestro, the team chose to rely on the LLM’s native reasoning capabilities, controlled through a thinking effort parameter (low, medium, high) exposed by LLM APIs. The orchestrator provides the structural scaffold—context assembly, tool dispatch, output re-injection—while the model performs the cognitive work of deciding what to do next. This approach simplifies the orchestration code and accelerates product iteration, but offers less deterministic control over how the model reasons internally. With sufficiently capable models, investing in orchestration infrastructure around the model can be more productive than attempting to replicate or constrain its reasoning process.团队没有在Maestro中实现自定义推理引擎，而是选择依赖LLM的原生推理能力，通过LLM API暴露的思考努力参数（低、中、高）进行控制。编排器提供结构框架——上下文组装、工具分发、输出重新注入——而模型执行决定下一步做什么的认知工作。这种方法简化了编排代码并加速了产品迭代，但对模型内部推理方式的确定性控制较少。使用足够强大的模型时，投资于围绕模型的编排基础设施可能比尝试复制或约束其推理过程更有效。
Strong model capabilities reduce implementation burden but do not eliminate the need for orchestration to enforce safety boundaries.强大的模型能力减少了实现负担，但并不能消除编排以强制执行安全边界的必要性。
While delegating reasoning to the LLM simplified much of the implementation details, model capability alone does not guarantee safe or predictable behavior. The orchestrator remains essential for enforcing stop criteria, managing tool execution order, handling errors, and applying guardrails. A powerful model can reduce the amount of hand-coded logic needed for task decomposition and planning, but it cannot replace the infrastructure that governs when and how actions are taken. Model and orchestration are complementary dimensions: improvements in one do not substitute for investment in the other.虽然将推理委托给LLM简化了许多实现细节，但仅凭模型能力并不能保证安全或可预测的行为。编排器对于执行停止条件、管理工具执行顺序、处理错误和应用防护措施仍然至关重要。一个强大的模型可以减少任务分解和规划所需的手工编码逻辑量，但它不能取代管理何时以及如何采取行动的基础设施。模型和编排是互补的维度：一方面的改进不能替代另一方面的投入。
4.2 Tool Design and Safety4.2 工具设计与安全
Next we provide five decisions regarding tool design and usage.接下来，我们提供关于工具设计和使用的五项决策。
Tool design quality proved more impactful on agent reliability than prompt-only tuning.工具设计质量对智能体可靠性的影响比仅调整提示词更大。
Over the course of development, the team observed that refining how tools are described and parameterized produced more consistent improvements in agent behavior than adjusting prompts alone. The semantic quality of a tool’s description determines how well the LLM understands when and why to use it. The parameter schema determines whether the model can invoke the tool correctly. And the error signaling behavior determines whether the model can recover from failed invocations. Together, these variables define the interface contract between the model and the external environment [10, 8]. In agentic systems, tool specification is a first-class engineering concern—not an afterthought—and should receive the same design rigor as API design.在开发过程中，团队观察到，优化工具的描述和参数化方式比单独调整提示词能更持续地改善智能体行为。工具描述的语义质量决定了LLM理解何时以及为何使用该工具的程度。参数模式决定了模型能否正确调用工具。而错误信号行为决定了模型能否从失败的调用中恢复。这些变量共同定义了模型与外部环境之间的接口契约[10, 8]。在智能体系统中，工具规范是首要的工程问题——而非事后考虑——应受到与API设计同等的设计严谨性。
The edit tool was designed for targeted string replacement because LLMs tend to truncate or omit content in long file rewrites.编辑工具被设计用于目标字符串替换，因为LLM在长文件重写时倾向于截断或遗漏内容。
When tasked with rewriting an entire file, LLMs frequently produce outputs that are incomplete, truncated, or subtly different from the intended result [12]. To mitigate this, the edit tool was designed around a targeted replacement model: the LLM specifies the file name, the old string to be found, and the new string to replace it. This constrains the model to making small, localized changes rather than regenerating entire files, significantly reducing the surface area for errors. Tool design can actively compensate for known LLM weaknesses by narrowing the scope of each individual action.当被要求重写整个文件时，LLM经常产生不完整、截断或与预期结果略有不同的输出[12]。为缓解这一问题，编辑工具围绕目标替换模型设计：LLM指定文件名、要查找的旧字符串和替换的新字符串。这限制了模型进行小的、局部化的更改，而不是重新生成整个文件，从而显著减少了错误范围。工具设计可以通过缩小每个单独操作的范围来主动补偿已知的LLM弱点。
A read-before-edit policy prevents hallucinated edits on files the model has not recently inspected.先读后编辑策略可防止模型对近期未检查的文件进行幻觉编辑。
Without explicit guidance, the LLM may attempt to edit files based on stale or imagined content—producing edits that reference code that no longer exists (or never existed). To address this, the system prompt instructs the model to always invoke the read file tool before calling edit, ensuring that the model operates on the current state of the file. This policy is enforced at the prompt level rather than at the tool level, meaning it relies on model compliance rather than hard constraints. Prompt-level policies can effectively mitigate hallucination risks in tool use, though they introduce a dependency on the model’s instruction-following reliability.在没有明确指导的情况下，LLM可能会基于过时或想象的内容尝试编辑文件——产生引用不再存在（或从未存在）代码的编辑。为解决此问题，系统提示指示模型在调用编辑之前始终调用读取文件工具，确保模型基于文件的当前状态进行操作。该策略在提示级别而非工具级别执行，意味着它依赖于模型的合规性而非硬性约束。提示级别的策略可以有效缓解工具使用中的幻觉风险，尽管它们引入了对模型指令遵循可靠性的依赖。
The shell tool is simultaneously the most useful and the most dangerous tool, requiring multiple guardrail layers.Shell工具同时是最有用和最危险的工具，需要多层防护。
The shell tool enables the agent to run tests, invoke build tools, execute git commands, and perform a wide range of environment operations—making it indispensable for realistic coding workflows. However, it also grants the agent the ability to execute arbitrary commands on the developer’s machine, including destructive ones such as rm -rf or unauthorized git push. To manage this risk, the team implemented a layered guardrail system: the tool can be blocked entirely, specific commands can be blocked granularly via a configuration file, and an approval mode can require human confirmation for every shell invocation. Additionally, all shell executions are recorded in the durable event timeline, providing a complete audit trail for post-hoc review. High-utility tools with broad execution capabilities require proportionally sophisticated safety mechanisms; a single-layer restriction is insufficient.Shell工具使智能体能够运行测试、调用构建工具、执行git命令以及执行各种环境操作——使其在现实编码工作流中不可或缺。然而，它也赋予智能体在开发者机器上执行任意命令的能力，包括破坏性命令如rm -rf或未经授权的git push。为管理此风险，团队实施了分层防护系统：工具可以被完全阻止，特定命令可以通过配置文件进行精细阻止，并且审批模式可以要求每次shell调用都需人工确认。此外，所有shell执行都记录在持久事件时间线中，为事后审查提供完整的审计追踪。具有广泛执行能力的高效用工具需要相应复杂的安全机制；单层限制是不够的。
Policy consistency across tools is required because blocking one tool category is insufficient if equivalent capabilities remain accessible through another.工具间的策略一致性是必需的，因为如果等效能力可通过其他工具访问，则阻止一个工具类别是不够的。
During development, the team discovered that restricting a specific tool (e.g., blocking direct file deletion) could be bypassed if the shell tool remained unrestricted, since a shell command can achieve the same effect. Safety policies had to be designed holistically across the entire tool manifest, not on a per-tool basis. Any tool with overlapping capabilities must be subject to consistent restrictions, or the guardrails become effectively meaningless. Safety in agentic systems is a system-level property: it cannot be achieved by hardening individual tools in isolation, but requires coherent policy enforcement across all execution channels.在开发过程中，团队发现如果shell工具不受限制，限制特定工具（例如阻止直接文件删除）可能会被绕过，因为shell命令可以实现相同的效果。安全策略必须从整个工具清单的整体角度设计，而不是基于单个工具。任何具有重叠功能的工具都必须受到一致的约束，否则防护措施实际上将变得毫无意义。代理系统中的安全性是一个系统级属性：它不能通过孤立地强化单个工具来实现，而是需要在所有执行通道中实施一致的策略执行。
4.3 Human Oversight and Adoption4.3 人工监督与采用
Finally, we provide four human oversights.最后，我们提供四种人工监督方式。
Human approval mode serves as an effective trust-calibration mechanism during onboarding.人工审批模式在入职阶段是一种有效的信任校准机制。
CodeGen offers two execution modes: approval mode, where every file edit and shell command requires explicit human confirmation, and autonomous mode, where the agent operates with minimal interruption. The team observed a consistent adoption pattern: developers begin in approval mode to understand the agent’s behavior and verify its decisions, then progressively switch to autonomous mode as their confidence grows [4]. This transition is organic and self-paced, not enforced by the agentic system. We noted that providing a low-risk entry point is critical for adoption in enterprise environments—human oversight mechanisms function not as permanent constraints but as transitional scaffolds that developers shed at their own pace.CodeGen提供两种执行模式：审批模式，其中每次文件编辑和shell命令都需要明确的人工确认；以及自主模式，代理以最小中断运行。团队观察到一致的采用模式：开发人员从审批模式开始，以了解代理的行为并验证其决策，然后随着信心的增长逐步切换到自主模式[4]。这种转变是自然的、自定进度的，而非由代理系统强制。我们注意到，提供低风险入口对于企业环境中的采用至关重要——人工监督机制并非作为永久约束，而是作为开发人员按自身节奏摆脱的过渡性支架。
Separating planning from execution addresses a recurrent limitation of single-pass execution.将规划与执行分离解决了单次执行的一个反复出现的局限性。
In the default execution flow, the LLM begins acting immediately upon receiving a prompt—reading files, editing code, running commands—without first presenting a plan for the user to approve. This single-pass behavior proved problematic for complex or high-stakes tasks, where developers wanted to verify the agent’s strategy before any modifications were made [5]. In response, the team developed a dedicated planning mode that generates an explicit action plan as a separate step, allowing the user to review, modify, or reject it before execution begins. Decoupling reasoning from action introduces a natural checkpoint for human oversight, improving both controllability and user confidence in the agent’s behavior.在默认执行流程中，LLM在收到提示后立即开始行动——读取文件、编辑代码、运行命令——而无需先向用户提交计划供审批。这种单次执行行为对于复杂或高风险任务来说存在问题，因为开发人员希望在做出任何修改之前验证代理的策略[5]。为此，团队开发了一种专门的规划模式，该模式生成一个明确的行动计划作为单独的步骤，允许用户在开始执行之前审查、修改或拒绝该计划。将推理与行动分离引入了人工监督的自然检查点，提高了可控性和用户对代理行为的信心。
Progressive deployment across environments mirrors the trust-building pattern observed in individual adoption.跨环境逐步部署反映了个人采用中观察到的信任建立模式。
Just as individual developers transition from approval mode to autonomous mode, the deployment pipeline enforces a progressive validation path: changes flow from development to staging to production through automated CI/CD gates. This organizational-level trust calibration ensures that updates to the agent’s behavior—new tools, modified system prompts, updated guardrail configurations—are validated in controlled environments before reaching the full user base. The parallel between individual trust-building (approval to autonomous mode) and organizational trust-building (dev to staging to production) suggests that progressive exposure is a general principle for agent adoption, applicable at multiple levels of scale.正如个人开发人员从审批模式过渡到自主模式一样，部署管道强制执行逐步验证路径：变更通过自动化CI/CD门控从开发环境流向预发布环境再到生产环境。这种组织级别的信任校准确保对代理行为的更新——新工具、修改后的系统提示、更新的防护配置——在受控环境中得到验证，然后才到达完整用户群。个人信任建立（从审批到自主模式）与组织信任建立（从开发到预发布再到生产）之间的相似性表明，逐步暴露是代理采用的一般原则，适用于多个规模级别。
Most design decisions involved balancing competing concerns rather than optimizing a single metric.大多数设计决策涉及平衡相互竞争的问题，而不是优化单一指标。
Throughout CodeGen’s development, the team repeatedly faced decisions where improving one dimension came at the cost of another. Increasing safety through approval modes added latency and friction. Delegating reasoning to the LLM simplified the codebase but reduced deterministic control. Session memory improved user experience but introduced infrastructure complexity. Choosing FastAPI and asyncpg optimized for concurrency but added complexity compared to synchronous alternatives. No single metric could serve as the optimization target; instead, each decision required explicit trade-off analysis in context. Practical agent engineering in enterprise settings is fundamentally an exercise in trade-off management, and the most consequential design skill is the ability to identify which trade-off matters most in each specific situation.在CodeGen的整个开发过程中，团队反复面临这样的决策：改进一个维度往往以牺牲另一个维度为代价。通过审批模式提高安全性增加了延迟和摩擦。将推理委托给LLM简化了代码库，但减少了对确定性的控制。会话记忆改善了用户体验，但引入了基础设施的复杂性。选择FastAPI和asyncpg优化了并发性，但与同步方案相比增加了复杂性。没有单一的指标可以作为优化目标；相反，每个决策都需要在具体情境中进行明确的权衡分析。企业环境中的实用智能体工程本质上是一项权衡管理的工作，而最重要的设计技能是识别在每种特定情况下哪种权衡最为关键的能力。
5 Open Questions5个开放问题
For researchers and tools builders, our learnings lead us to a few questions.对于研究人员和工具构建者来说，我们的经验引出了几个问题。
1. How should tool manifests be designed to minimize model misuse and maximize correct invocation? Tool descriptions, parameter schemas, and error contracts proved more impactful than prompt tuning on agent reliability. Yet there is no established methodology for designing tool specifications—teams rely on trial and error. What principles, metrics, or evaluation frameworks can guide the systematic design of tool interfaces for LLM-based agents?1. 如何设计工具清单以最小化模型误用并最大化正确调用？工具描述、参数模式和错误合约对智能体可靠性的影响比提示调优更大。然而，目前还没有建立设计工具规范的方法论——团队依赖试错。哪些原则、指标或评估框架可以指导基于LLM的智能体工具接口的系统化设计？
2. What is the optimal boundary between model-delegated reasoning and orchestrator-enforced control? CodeGen delegates nearly all reasoning to the LLM while the orchestrator handles safety and control flow. But this boundary was drawn pragmatically, not systematically. Under what conditions should reasoning responsibility shift from the model to the orchestrator (or vice versa), and how does this boundary affect reliability, latency, and safety as model capabilities evolve?2. 模型委托推理与编排器强制控制之间的最佳边界是什么？CodeGen将几乎所有推理委托给LLM，而编排器负责安全和控制流。但这个边界是实用主义地划定的，而非系统化的。在什么条件下，推理责任应从模型转移到编排器（或反之），随着模型能力的演进，这个边界如何影响可靠性、延迟和安全性？
3. How can safety policies be specified and enforced consistently across tools with overlapping capabilities? The experience with the shell tool showed that per-tool guardrails are insufficient when multiple tools can achieve the same effect [13]. What formalisms or policy languages can express cross-tool safety constraints in a way that is complete, verifiable, and maintainable as the tool manifest grows?3. 如何跨具有重叠功能的工具一致地指定和执行安全策略？shell工具的经验表明，当多个工具可以实现相同效果时，每个工具的防护栏是不够的[13]。哪些形式化方法或策略语言可以表达跨工具的安全约束，使其在工具清单增长时保持完整、可验证和可维护？
4. What mechanisms best support the transition from human-supervised to autonomous agent operation? Developers naturally migrated from approval mode to autonomous mode, but this transition was unstructured and self-paced. Are there adaptive trust models—based on task complexity, historical success rate, or action reversibility [4, 5]—that can dynamically calibrate the level of human oversight required, rather than relying on a binary mode switch?4. 哪些机制最能支持从人工监督到自主智能体操作的过渡？开发者自然地从审批模式迁移到自主模式，但这种过渡是非结构化和自定进度的。是否存在基于任务复杂性、历史成功率或操作可逆性[4, 5]的自适应信任模型，可以动态校准所需的人工监督水平，而不是依赖二元模式切换？
5. How should agent memory be structured to support long-term learning without compromising session reliability? Session-scoped memory delivered immediate UX value, but cross-session memory remains unsolved. What memory architectures balance persistence, retrieval accuracy, and staleness management across sessions—and how should agents decide what to retain, forget, or update over time without explicit user instruction?5. 如何构建智能体记忆以支持长期学习而不损害会话可靠性？会话范围的记忆带来了直接的用户体验价值，但跨会话记忆仍未解决。哪些记忆架构能在会话之间平衡持久性、检索准确性和陈旧性管理——智能体应如何决定保留、遗忘或随时间更新哪些内容，而无需用户明确指示？
6. How should agent-generated code be integrated into existing quality assurance pipelines? CodeGen’s experience shows that pre-commit hooks, static analysis, and test coverage requirements serve as effective guardrails for agent output. But as agents take on more complex tasks—refactoring across multiple files, modifying CI configurations, or generating migration scripts [16]—existing quality gates may be insufficient. What additional verification mechanisms are needed, and how should they differ from those applied to human-authored code?6. 如何将智能体生成的代码集成到现有的质量保证流程中？CodeGen的经验表明，预提交钩子、静态分析和测试覆盖率要求是智能体输出的有效防护栏。但随着智能体承担更复杂的任务——跨多个文件重构、修改CI配置或生成迁移脚本[16]——现有的质量门控可能不足。需要哪些额外的验证机制，它们应如何与应用于人类编写代码的机制不同？
6 Conclusion6 结论
CodeGen evolved through a pragmatic, operations-first strategy rather than a framework-first strategy. The team’s most consequential decisions were not limited to model selection; they centered on how reasoning is operationalized, how tools are specified and constrained, and how state is managed across interactive sessions.CodeGen 的演进遵循了一条务实的、以运维为先的策略，而非以框架为先的策略。团队最重要的决策不仅限于模型选择，更集中在如何将推理操作化、如何指定和约束工具、以及如何在交互会话中管理状态。
The architecture—a CLI executor communicating via WebSocket with a FastAPI backend and a centralized Maestro orchestrator, backed by PostgreSQL and Redis—was shaped by the need to keep secrets and orchestration logic server-side while enabling client-side tool execution on the developer’s actual environment. The choice to implement the agentic loop manually before adopting frameworks gave the team the understanding necessary to evaluate later abstractions on their merits rather than their marketing. The investment in tool design—particularly the targeted edit model, read-before-edit policy, and layered shell guardrails—produced more reliable agent behavior than prompt engineering alone. And the combination of approval mode, planning mode, and progressive deployment created an adoption path that allowed developers and the organization to build trust incrementally.该架构——一个通过 WebSocket 与 FastAPI 后端和集中式 Maestro 编排器通信的 CLI 执行器，并由 PostgreSQL 和 Redis 支持——其设计源于将机密和编排逻辑保留在服务端，同时允许在开发者的实际环境中进行客户端工具执行的需求。在采用框架之前手动实现代理循环的选择，使团队能够根据其实际价值而非营销宣传来评估后续的抽象层。在工具设计上的投入——特别是定向编辑模型、先读后编辑策略以及分层 shell 防护——产生了比单纯提示工程更可靠的代理行为。而审批模式、规划模式和渐进式部署的结合，创造了一条让开发者和组织逐步建立信任的采用路径。
Several open questions remain, particularly around systematic tool manifest design, cross-tool safety policy enforcement, adaptive trust calibration, and long-term agent memory. These challenges are not unique to CodeGen; they reflect broader gaps in the emerging practice of enterprise agent engineering. We hope that the concrete decisions and trade-offs reported here contribute to a shared understanding of what it takes to move coding agents from prototypes to production.仍存在几个开放性问题，特别是关于系统化工具清单设计、跨工具安全策略执行、自适应信任校准以及长期代理记忆。这些挑战并非 CodeGen 独有，它们反映了企业代理工程新兴实践中的更广泛差距。我们希望这里报告的具体决策和权衡有助于形成共识，了解将编码代理从原型推进到生产所需的条件。
References
- [1] (2021) Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. External Links: Document Cited by: §3.
- [2] (2023) Large language models for software engineering: survey and open problems. In Proceedings of the International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE), External Links: Document Cited by: §1.
- [3] (2024) SWE-bench: can language models resolve real-world GitHub issues?. In Proceedings of the International Conference on Learning Representations (ICLR), External Links: Document Cited by: §1.
- [4] (2004) Trust in automation: designing for appropriate reliance. Human Factors 46 (1), pp. 50–80. External Links: Document Cited by: §4.3, §5.
- [5] (2000) A model for types and levels of human interaction with automation. IEEE Transactions on Systems, Man, and Cybernetics—Part A: Systems and Humans 30 (3), pp. 286–297. External Links: Document Cited by: §1, §4.3, §5.
- [6] (2023) The impact of AI on developer productivity: evidence from GitHub Copilot. arXiv preprint arXiv:2302.06590. External Links: Document Cited by: §1.
- [7] (2024) Lessons from building stackspot AI: A contextualized AI coding assistant. In Proceedings of the 46th International Conference on Software Engineering: Software Engineering in Practice, ICSE-SEIP 2024, Lisbon, Portugal, April 14-20, 2024, pp. 408–417. External Links: Link, Document Cited by: §2.
- [8] (2024) ToolLLM: facilitating large language models to master 16000+ real-world APIs. In Proceedings of the International Conference on Learning Representations (ICLR), External Links: Document Cited by: §1, §4.2.
- [9] (2024) CodePori: large-scale system for autonomous software development using multi-agent technology. SSRN Electronic Journal. External Links: Document Cited by: §1.
- [10] (2023) Toolformer: language models can teach themselves to use tools. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Document Cited by: §1, §4.2.
- [11] (2024) A survey on large language model based autonomous agents. Frontiers of Computer Science 18 (6), pp. 186345. External Links: Document Cited by: §1.
- [12] (2022) A systematic evaluation of large language models of code. In Proceedings of the International Symposium on Machine Programming (MAPS), External Links: Document Cited by: §1, §4.2.
- [13] (2024) SWE-agent: agent-computer interfaces enable automated software engineering. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Document Cited by: §5.
- [14] (2023) Tree of thoughts: deliberate problem solving with large language models. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Document Cited by: §1.
- [15] (2023) ReAct: synergizing reasoning and acting in language models. In Proceedings of the International Conference on Learning Representations (ICLR), External Links: Document Cited by: §1, §3.
- [16] (2024) AutoCodeRover: autonomous program improvement. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA), External Links: Document Cited by: §5.