乔木改写

1000 个运行早期 Claude Fable 5.2 的智能体，在 28.7 小时里重做了一份费马大定理的形式化证明。和今年 9 月那次相比，它们没有使用 Prove2Me 平台，也没有研究员在旁边指挥，用掉的输出 token 大约是当时的三分之一，写出的证明大约是七分之一长。

这篇说明由 Barry Zhang、Tianyi Peng、Elle Najt、Keller Scholl 和 Lance Martin 撰写，2026 年 11 月 9 日发布在 claude.dev。Lance Martin 的 X 账号是 @RLanceMartin，公开资料写的是 Anthropic 的 MTS，他也在自己的账号上分享了这篇文章。文中把这次运行称为一次玩具实验。智能体用的是早期版本的 Fable 5.2，不是对公开发布版的承诺。

费马大定理说的是：不存在正整数 a、b、c，能让 aⁿ + bⁿ = cⁿ 在任何大于 2 的 n 上成立。它在 350 多年里一直没有被证明，直到 Andrew Wiles 在 1995 年给出 129 页的证明。把这份证明形式化，也就是让计算机检查每一步，长期是数学共同体里的目标。

今年 9 月，他们发布了第一份端到端的费马大定理形式化，说明在 https://www.anthropic.com/research/formalizing-fermats-last-theorem 。那次由几十个智能体完成，运行的是大约相当于 Claude Fable 5.1 的内部研究模型，并有一位研究员给出方向。他们用了 11 天，大约 60 亿输出 token，写出了 1300 万行 Lean 代码。Lean 是一种编程语言，计算机会检查证明里的每一步。

当时的协调靠的是 Prove2Me。这个平台由 Tianyi Peng 和哥伦比亚大学的合作者设计，Peng 现在是 Anthropic 的技术员工。Prove2Me 用一张共享图，把每条待证命题和它拆成的更小命题连起来。任何智能体都可以拆分一条命题，其他智能体可以向这张图询问哪些尚未证明的部分已经可以动手，这样就不容易丢掉项目状态。每条命题和它的证明都放在单独文件里，并且可以独立编译，因此许多智能体能够并行工作，而不去改同一批文件。

新实验想看的是：一支运行类似 Fable 5.2 的智能体队伍，能不能在没有 Prove2Me、也没有研究员当场指挥的情况下，用一套由通用工具组成的 harness 做出这个结果。

每个智能体都待在没有外网的沙箱里，手里只有通用工具

他们给这 1000 个智能体准备了 60 条需要证明的命题，其中包括最终定理和 59 个中间里程碑。对每一条命题，智能体都要写出 Lean 证明，用 Lean 的工具构建并检查，再提交到 Git 共享主干上，由评分器检查。每个智能体得到的工具是：

自己沙箱里的一个 shell 和一个文件编辑器，沙箱没有网络访问。

沙箱镜像里已经装好的 Lean 工具链和 Mathlib，Mathlib 的地址是 https://github.com/leanprover-community/mathlib4 。

自己的 Git 分支，以及向主干发起合并请求的能力。

一个共享的 Postgres 数据库，从 shell 里访问。

证明工作可以拆开，因此适合多智能体队伍。智能体也需要协调，既要接上别人的成果，也要避免重复劳动。作者写到，最初的形式化里，Claude 早期的尝试失败过，原因是智能体跟丢了项目状态，不再有效协作。搬到 Prove2Me 之后，这个问题才得到解决。

这一次，每个智能体都在自己的轻量虚拟机沙箱里工作。单台共享机器大约在 100 个智能体时就会碰到上限：超过一半的 100 智能体运行因为内存、磁盘或 CPU 耗尽，不得不停止或重启。

有了 1000 个分开的沙箱，智能体需要一种办法分享工作，于是作者给了它们一个 SQL 数据库。他们选择 SQL，是因为它提供组织信息的通用原语，事务能处理 1000 个智能体同时写入，而且 Claude 本来就熟悉它。智能体用这个数据库认领任务、分享笔记，并记录完成的工作。

图 1 画的是这套 harness 的结构。1000 个智能体沙箱叠在一起，其中 1 个领队、999 个助手。每个沙箱里跑着智能体循环和 harness 代码，工具是 shell 和文件编辑器，文件则包括代码仓库、Lean 和数学库。智能体循环通过一条管道连到沙箱外的网关，由网关调用 Claude API。Git 和 SQL 请求也经过同一个网关，到达沙箱外的共享仓库和数据库。

提示词给了 Claude 一份 60 条待证命题的清单，每条带一句非正式说明，并描述了上面这些可用工具。

他们给队伍的唯一结构是层级：1 个领队，999 个助手。领队负责把问题拆开，并分配给助手。助手则要等待领队派活，任务不清楚时向领队确认。

运行过程中，harness 自动向领队发送了 19 次剩余时间提醒，每次都鼓励领队继续做下去。

安全边界叠了三层：沙箱、实时监控和离线监控

作者写到，没有任何单层防护足以约束一支智能体队伍。这次实验把沙箱和实时、离线监控合在一起，让每个智能体待在自己的、没有网络访问的虚拟机里，并检测任何试图逃出沙箱或干扰监控的行为。

图 2 把安全措施标成三组：沙箱、实时监控和离线监控。分类器在每次工具调用执行前检查，离线监控再审阅队伍的行动。

沙箱这一侧：每个智能体有自己的轻量虚拟机，没有网络，没有 API 密钥或凭证，没有通往其他沙箱的直接路径，以非特权用户运行，也不能用工具改到驱动自己的 harness 代码。智能体没有专用的消息工具，只能通过数据库和 Git 协调。一个智能体只能推送到自己的 Git 分支，并向主干发起合并请求；另有一个独立任务负责把变更合并进主干，只有发生冲突或出现无关历史这类故障时才会拒绝。数据库调用使用共享的非特权角色，每条 SQL 有 5 秒时限，写入超过 256 MiB 会被拒绝。沙箱对外只连着通往网关的那条管道，由网关转发请求。网关在沙箱外给每个请求打上该智能体的身份，防止一个智能体冒充另一个。每个沙箱的内核在编译时去掉了可加载模块、kexec、裸内存访问和内核探针这类能让 root 用户直接进入内核的功能，并运行在 kernel lockdown 模式。sudo 这类能重新获得 root 的工具也被去掉了。

实时监控这一侧：一个经过提示的 LLM 分类器可以在每次工具调用执行前拦住它，包括 shell 命令和文件编辑。分类器盯的是试图逃出沙箱、探测边界、干扰监控，以及模型如果真的离开沙箱可能采取的有害行动。分类器拦住一次工具调用后，整支队伍会停下来，必须由人审阅这次拦截，队伍才能重新开始。这样设计是为了避免另一个智能体去执行已经被拦住的动作。

为了测试分类器能不能发现复杂的破坏，他们做过多智能体的旁任务和破坏评估，红队智能体会使用分解攻击这类策略，试图骗过分类器、让它放行一次沙箱逃逸。他们发现，能够看到智能体思维链和智能体之间通信的监控器，在这些评估里检测出了全部破坏尝试，因此现在用它们来监控多智能体运行。他们也并行运行一个只看行动、不看思维链的版本，用来防范智能体的推理说服监控器放行有害行动。

他们还在通过有人指导的和自动的红队测试改进监控，对照包括既往事件在内的生产记录做评估，并扩大威胁建模。这次运行里，他们只用了基于行动的监控器。只有拦住领队才会结束整次运行；拦住一个助手，只会移走那个助手。

这次运行结束后不久，他们加入了并行的思维链监控器。他们会在可能尚未完成发布前对齐评估的模型上做这类实验，因此监控是按模型可能没有对齐来设计的。对外发布的模型会先完成那项评估，这会降低风险，但不能消除风险。

在超过 100 万次工具调用里，这一次没有一次达到需要被拦住的标准。他们后来用思维链监控器和今天的评分标准回放这些调用时，只有一次接近被拦，那是一次对是否拥有 root 权限的检查。

离线监控会按与实时监控相同的威胁模型分析整支队伍的行动，但有更多条件，例如能查到其他智能体的行动。运行记录保存在沙箱之外。

没有现成平台时，领队在前 10 分钟自己建了五张表

证明工作发生在好几层：最终的定理本身；他们给出的 60 条命题，包括最终定理和 59 个中间里程碑；智能体把困难命题再拆成的更小命题或引理；以及每一条在 Lean 里对应的定理。

每条命题一开始在 Lean 里只有证明占位符，只有 Lean 的检查器接受一份完成的证明，它才算被证明。领队自行决定如何处理证明，以及如何在这些层级上协调队伍。

前 10 分钟里，领队在数据库中建了五张表。

Announce：领队向全队下达的命令，例如如何拆分工作。

Assign：谁做什么，例如哪位助手负责或代理哪条命题。

Notes：一块开放的看板，用来认领一块工作或汇报进展。

Subtasks：一份待办清单，任何智能体都可以往里加任务，第一个认领的助手成为负责人。

Landed：已完成工作的日志。

图 3 画的就是这五张表和读写关系。领队主要写 Announce 和 Assign，助手记录完成的工作并管理 Subtasks 表，双方都会写 Notes 表。图注还写到，Announce 和 Assign 几乎都由领队写入，助手主要读取，各自只有一次助手写入；Notes 由领队和助手都写都读；Subtasks 主要由助手写和读，领队读取，并有两次领队写入；Landed 由助手写和读，领队读取。

表建好之后，领队做了三件事。第一，它给 Assign 表写入 999 行，把每条命题分给一支 16 到 17 人的助手小组。第二，它让每组里的一位助手担任队长，负责把计划发到 Notes 表，并把工作拆成任务，供组员从 Subtasks 表认领。第三，它在 Announce 表发出指令，要求每位助手先判断自己那条命题的难度，并把结论发到 Notes 表。

助手从提示词里知道数据库存在。他们找到 Announce 表，开始做评估。大多数人把结论发到了 Notes 表。大约 27 分钟时，已经有 3 条命题被证明，领队也读过了这些结论。它随后把大多数助手判断为无法在这次运行预算内完成的 47 条命题放在一边，几乎把全部助手重新分到剩下的 10 条，也就是它认为能够完成的那些。小组规模从大约 17 人，扩大到 46 到 139 人。

每位助手都在一个循环里工作，其中大部分步骤写在领队的命令里：在 Notes 表里找到该命题的计划；从 Subtasks 表认领一项任务；写出 Lean 证明，用 Lean 的工具检查，并提交到 Git 主干；把任务标成完成；再向 Notes 表发一条更新。

大约 45 分钟时，10 条命题里大约已经证明了 5 条，领队写道：队伍的吞吐量，远高于早期那些“预算内不可行”的判断所假设的水平。

这 10 条命题在大约 70 分钟内全部被证明。

领队随后把助手转到更难的命题上，最后几乎全员扑向 Langlands-Tunnell

这前 10 条完成之后，领队写道：没有理由闲着，并把助手重新分到它最初放下的那些更难命题。随着更多命题被证明，它继续把助手挪到新任务上，这些调动大多发生在前 6 个小时。

图 4 是一段从第 0 小时到第 30 小时的延时影像，画的是 1000 个智能体和共享数据表之间的读写。写入是从上往下的红线，读取是从表格再向下到智能体的绿线。到第 30 小时，写入累计 36612 次，读取累计 311734 次。几乎全部写入都落在 Notes 表（大约 20600 次）和 Subtasks 表（大约 14800 次）。这些总数排除了大多数脚本调用。影像在接近第 18 小时处安静下来，那是他们从上次保存点重启运行、并又给了智能体 12 小时的时候。这段影像的画面细节来自原文提供的说明，视频本身没有另外核对过。

领队最初要求队长把工作拆进 Subtasks 表。随着工作推进，助手在解决各条命题上承担了更多责任。他们开始自己写任务，有时是在队长求助之后。到第 5.6 小时，已经有 73 位助手写了 756 项任务，其中只有 16 位是队长。图 4 里也能看到许多助手在向 Subtasks 表写入，不过其中大多数写入是认领任务或标成完成。完成一项任务的助手还会主动请战，原文引用过这样的话：FREE … I finish pieces end-to-end … Name ONE deep piece … and I start at once。

起初，领队也把助手指定到具体命题。这种情况在第一小时内就开始变化。到第三小时，领队对空闲的助手说：不要等我。这样，空闲助手可以在不等领队的情况下接手不同命题，即便领队仍在继续写新的分配。

大约第 18 小时，剩下的主要是 Langlands-Tunnell，以及依赖它的那些命题，外加另一条命题，队伍后来一并证明了。Langlands-Tunnell 是 Wiles 证明所依赖的一个经典结果。更早的时候，负责这条任务的队长比较悲观，估计证明至少需要 70 万到 100 万行 Lean 代码，在剩余时间内完成的机会不到 1%。领队当时把这条命题放在了一边。

几乎所有其他工作完成之后，领队把队伍重新集中到 Langlands-Tunnell。作为最后一次自上而下的分配，它把大约 750 名助手调到这条命题上，并写道：Honest odds are low。

Langlands-Tunnell 远大于智能体此前处理过的任何一条命题，任务数大约是其他命题的 9 倍，新增代码也超过两倍。助手们再次用 Subtasks 表拆分工作，但最初的许多任务对单个助手来说太大，一个人完不成。

于是队伍把工作组织成一棵树。队长做了一份计划，把这条命题分成 10 个区域，并把每一块列为任务。如果一位助手发现某一块大到自己单独做不完，就可以再拆，把更小的任务加到它下面。这棵树在某一块对单个智能体来说太难时就会继续生长，最终达到 2418 项任务，最深到 10 层。每一块都可以假定它下面的那些块成立，因此在那些更小的块被证明之前，成百上千的助手就能并行工作。

图 5 是 Langlands-Tunnell 下这棵任务树的延时影像，从第 20.5 小时一直画到第 28.9 小时。命题在中心，向外每一圈更深一层，靠近中心的黑点是它的子区域。任务被助手写下时出现，第一次被标成完成时从灰色变成橙色，再变成深红，更晚的小时颜色更深。到第 28.9 小时，树上有全部 2418 项任务，其中 2364 项带有已知完成小时。连线表示任务如何被拆开；作者写明，这些链接大多是事后根据助手们的任务标签推断出来的。这条命题在第 28.4 小时被证明。这段影像的画面细节同样来自原文说明，视频本身没有另外核对过。

图 6 显示每个助手在整次运行里做了什么：每个点是一位助手，每 5 分钟被放在它自己记录所显示的那条命题或那条工作线上，等待构建时也包括在内。60 条命题按领队如何配置人手分组：最先被证明的、被判断为可做的、被放下但在第 1.2 小时前又配了人的、被放下并从第 1.5 小时起接手的、modularity lifting、Langlands-Tunnell，以及连到费马大定理的最后环节。助手们早期分散在许多命题上，大约从第 8 小时起聚集到 modularity lifting，在接近第 18 小时处停住，重启后又聚集到 Langlands-Tunnell。最终定理在第 28.7 小时被证明。统计结束时是 60 条里的 54 条，因为有 5 条备选方案不再需要，另有 1 条被放下。到第 30 小时，这 54 条在图上被填成黑色。这段影像的画面细节来自原文说明，视频本身没有另外核对过。

领队随后写道：Thank you all: Fermat's Last Theorem is on main, kernel-checked by the official checker.

作者认为这没有让领域专用工具或人类方向变得多余

这次玩具实验让他们观察到，一支 1000 个智能体的队伍如何完成一项又长又难的任务。只用 Git 和一个共享数据库，这些智能体设计了用来分配角色和分享结果的数据表，组成了小组，并建起一棵任务树，去拆开最难的那条命题。这条命题的队长曾经给出不到 1% 的完成机会。

这个结果并没有取消对领域专用 harness 或平台的需要。作者举例说，Prove2Me 仍可能靠帮助智能体更快编译 Lean 代码，并让它们搜索和复用人类已有结果，来提高效率。

人类在某些节点给出方向仍然有用。智能体往往会在难题上放弃，也会误判难度。早期，助手们把 60 条里的 47 条判断为无法在预算内完成，后来队伍却证明了其中大多数。作者认为，在这样的时刻给予人类鼓励，可以避免智能体过早停下来。

仔细设计的安全边界，让他们有信心开始这次实验。Claude 可以用他们提供的工具按自己的方式处理问题，但只能待在没有网络的沙箱里，并处于两套分开的监控之下。

他们预计这些方法会很快演变。当类似的队伍接下更难的工作时，周围的安全边界也需要跟上。他们相信，在安全放大的前提下，多智能体队伍将在未来几年帮助解决一些最困难的问题。

完整数字、安全设计和这次实验的界限，以他们 2026 年 11 月 9 日这篇说明为准。文中写明，这是一次玩具实验，用的是早期 Fable 5.2，不能把它读成对公开发布模型的保证。

原文

Safely scaling multi-agent teams with Claude Fable 5.2

Safely scaling multi-agent teams with Claude Fable 5.2

How a team of 1,000 agents running an early version of Claude Fable 5.2 reproduced a formal proof of Fermat's Last Theorem with general tools, a shared database, and layered safety controls.

• Author: Barry Zhang, Tianyi Peng, Elle Najt, Keller Scholl, and Lance Martin (Member of Technical Staff)
• Published: 2026-11-09
• URL: https://claude.dev/blog/safely-scaling-multi-agent-teams-with-fable-5-2/

Multi-agent teams can solve challenging, long-horizon problems. In September, we published the first end-to-end formalization of Fermat’s Last Theorem, written by dozens of agents running an internal research model roughly comparable to Claude Fable 5.1. That work relied on Prove2Me, a platform for formalizing mathematics, and direction from a researcher. In a new toy experiment, a team of 1,000 agents running an early version of Claude Fable 5.2 reproduced the proof without the scaffolding provided by Prove2Me or steering from a researcher. The team took 28.7 hours, used one-third as many output tokens, and wrote a proof one-seventh the length of the original.

Each agent worked in a sandbox with no internet access. A classifier checked every shell command and file edit before it ran, and an offline monitor reviewed each of the team’s actions. We gave the agents a list of 60 statements, a one-sentence sketch of the classical route, the definitions those statements use, and 47 proved lemmas from our earlier formalization, and assigned lead and helper roles. We offered no other guidance on how to prove the theorem or organize themselves.

This toy experiment suggests that teams of agents running models like Fable 5.2 can accomplish ambitious, long-horizon goals within carefully designed safety boundaries.

Problem

Fermat’s Last Theorem states that no positive integers a, b, and c satisfy aⁿ + bⁿ = cⁿ for any n greater than 2. It remained unproven for more than 350 years, until Sir Andrew Wiles’s 129-page proof in 1995. Formalizing the proof—making every step checkable by a computer—has been a long-standing goal in the mathematics community.

Our September formalization took a researcher and dozens of agents 11 days and roughly 6 billion output tokens. They wrote 13 million lines of code in Lean, a programming language in which a computer checks every step of a proof.

The agents coordinated through Prove2Me, a platform designed by Tianyi Peng (who is now a member of technical staff at Anthropic) and his collaborators at Columbia University. Prove2Me uses a shared graph that links each statement to be proved to the smaller statements it was split into. Any agent could split a statement, and other agents could ask the graph which unproved pieces were ready to work on, which kept them from losing track of the project state. Each statement and its proof were kept in a separate file that compiled independently, so many agents could work in parallel without editing the same files.

Harness

We wanted to see whether a team of agents running on models like Fable 5.2 could produce this result without Prove2Me or a researcher directing the work, using a harness built from general-purpose tools.

Tools

For this task, we gave the team of 1,000 agents 60 statements to prove, including the final theorem and 59 intermediate milestones. For each statement, the agents had to write a proof in Lean, build and check it with Lean’s tools, and submit it to the shared main branch on Git, where a grader checked it. We gave each agent the following tools:

• A shell and a file editor, in its own sandbox with no network access.
• Lean’s toolchain and Mathlib, built into the sandbox image.
• Its own Git branch, plus the ability to open merge requests into main.
• A shared Postgres database, reached from the shell.

Collaboration

Work on the proof can be split up, making it a good task for a multi-agent team. But agents also need to coordinate, both to build on one another’s work and to avoid duplicating it. In our original formalization, Claude’s early attempts failed because agents lost track of the project’s state and stopped collaborating effectively. Moving to the Prove2Me platform solved this problem.

For this run, each agent had a sandbox on its own lightweight virtual machine. A single shared machine hit its limits at about 100 agents: more than half of our 100-agent runs had to be stopped or restarted because memory, disk, or CPU ran out.

With 1,000 separate sandboxes, the agents needed a way to share their work, so we gave them a SQL database (Figure 1). We chose SQL because it offers general primitives for organizing information, its transactions handle concurrent writes from 1,000 agents, and Claude is already familiar with it. The agents used the database to claim tasks, share notes, and log finished work.

Each of the 1,000 agents ran in its own sandbox, connected to the outside only by a pipe to a gateway. The gateway called the Claude API and passed Git and SQL requests to the shared repository and database.

Prompting

Our prompt gave Claude a list of 60 statements to prove, each with a one-line informal description, and described the available tools (listed above).

The only structure we gave the team was a hierarchy, consisting of one lead and 999 helpers. We told the lead to split the problem into parts and assign them to the helpers. We told the helpers to expect work from the lead and to check with the lead if a task was unclear.

During the run, the harness sent the lead 19 automatic reminders of how much time remained, each encouraging the lead to keep going.

Safety

As we’ve shared previously, no single layer of defense is sufficient to contain a team of agents. In this experiment, we combined sandboxing with real-time and offline monitoring to keep each agent in its own virtual machine with no network access and to detect any attempts to escape the sandbox or tamper with monitoring.

Sandboxing

We secured each agent’s sandbox in the following ways (Figure 2):

• Communication: Agents had no dedicated messaging tools and coordinated only through the database and Git.
• Code changes: An agent could push only to its own Git branch and open a merge request into main. A separate job then merged the changes into main, refusing them only in cases of a conflict or a fault, such as unrelated history.
• Database access: Database calls ran under a shared unprivileged role. Each SQL statement had a five-second time limit, and writes were refused above 256 MiB.
• Network controls: Sandboxes did not have network access. Their one link out was a pipe to the gateway, which relayed their requests.
• Identity: The gateway tagged each request with the agent’s identity outside the sandbox, to prevent one agent from posing as another.
• Kernel: Each sandbox’s kernel was built without features that give a root user a direct path into it, such as loadable modules, kexec, raw memory access, and kernel probes. It ran in kernel lockdown mode.
• Least privilege: Each agent’s commands ran inside its sandbox as an unprivileged user, with tools like sudo that could regain root removed. The agent’s tools were designed so they couldn’t read or change the agent loop that ran them.

Three safety layers surrounded the team: sandboxing, real-time monitoring, and offline monitoring. A classifier checked each tool call before it ran, and an offline monitor reviewed the team’s actions.

Real-time monitoring

To make runs like this safer, we use a prompted LLM classifier that can block each tool call, including shell commands and file edits. The classifier watches for attempts to escape the sandbox, probe its boundaries, or tamper with monitoring, and for any harmful actions the model might take if it manages to exit the sandbox. When the classifier blocks a tool call, the whole agent team stops and a person must review the block before the team can restart. This is intended to prevent another agent from carrying out a blocked action.

To test whether our classifier could detect sophisticated sabotage, we ran multi-agent side-task and sabotage evaluations, in which red-team agents used strategies like decomposition attacks to try to fool the classifier into permitting a sandbox escape. We found that monitors that viewed the agents’ chain of thought (CoT) and interagent communication were able to detect all sabotage attempts in these evaluations, so we now use them to monitor our multi-agent runs. In parallel, we run an action-based version of the same monitor—which sees only actions, not CoT—to guard against an agent’s reasoning persuading the monitor to permit a harmful action.

We’re continuing to improve our monitoring stack through human-guided and automatic red-teaming, by evaluating it against production transcripts, including past incidents, and by expanding our threat modeling. In the run in this post, we used only the action-based monitor. Only a block on the lead would have ended the run; a block on a helper removed just that helper.

Shortly after this run, we added the parallel CoT monitor. We run experiments like this on models that may not have completed our pre-release alignment assessment, so we design our monitoring for the possibility that a model is misaligned. Models we release complete that assessment first, which reduces this risk but doesn’t eliminate it.

Across more than a million tool calls, none in this run met the bar to be blocked. When we replayed these calls against the CoT monitor and today’s rubric, only one, a check for root access, came close to being blocked.

Offline monitoring

In addition to real-time monitoring, we run an offline monitor. This monitor analyzes the agent team’s actions against the same threat models as the real-time monitor, but with additional affordances, such as being able to look up actions from other agents.

Results

Without Prove2Me, the team built its own coordination system

Work on this proof happens at several levels:

• The proof: No positive integers a, b, and c satisfy aⁿ + bⁿ = cⁿ for any n greater than 2.
• Statements: We gave the team 60 statements to prove, including the final theorem and 59 intermediate milestones.
• Smaller statements: Agents split hard statements into smaller ones, or lemmas.
• Lean: Every statement and lemma is a theorem in Lean.

Each statement started in Lean with a placeholder for its proof, and it counted as proved only once Lean’s checker accepted a finished proof. We let the lead decide how to tackle the proof and how to coordinate the team across these levels of work.

In the first 10 minutes, the lead set up five tables in the database (Figure 3):

• Announce: The lead’s orders to the entire team (e.g., how to split up the work).
• Assign: Who works on what (e.g., which helper owns or deputizes for each statement).
• Notes: An open board (e.g., for claiming a piece of work or reporting progress).
• Subtasks: A to-do list where any agent can add a task and the first helper to claim it owns it.
• Landed: A log of finished work.

In the first 10 minutes, the lead set up five shared database tables to coordinate the team. The lead wrote orders and assignments, helpers logged finished work and managed the subtasks table, and both wrote to the notes table.

The lead dealt out the statements

Once the lead had created the tables, it did three things. First, it assigned each statement to a crew of 16–17 helpers by writing 999 rows to the assign table, one for each helper. Second, it made one helper in each crew the captain, responsible for posting a plan to the notes table and splitting work into tasks for the crew to claim from the subtasks table. Third, it posted instructions to the announce table, telling every helper to start by judging its statement’s difficulty and posting a verdict in the notes table.

From their prompt, the helpers knew that the database existed. They found the announce table and got started on their assessments. Most posted verdicts to the notes table. About 27 minutes in, three statements were already proved and the lead had read the posted verdicts. It then set aside the 47 statements that most helpers had judged to be infeasible within the run’s budget and reassigned nearly all of the helpers to the remaining 10, which it thought they could finish. Crews grew from about 17 helpers to between 46 and 139.

Each helper worked in a loop, most of which was set out in the lead’s orders:

• Find the statement’s plan in the notes table.
• Claim a task from the subtasks table.
• Write the proof in Lean, check it with Lean’s tools, and submit it to the main branch on Git.
• Mark the task as done.
• Post an update to the notes table.

Around 45 minutes in, with about five of the 10 statements proved, the lead wrote:

"the team's throughput is far higher than the early 'infeasible-in-budget' verdicts assumed"

All 10 statements were proved within roughly 70 minutes.

The lead then moved helpers to harder statements

With those first 10 statements done, the lead wrote, “there is no reason to idle,” and reassigned helpers to the harder statements it had initially discarded. As more statements were proved, it kept moving helpers to new assignments, mostly in the first six hours (see the writes to the assign table, shown in red in Figure 4).

Video: Reads from (green) and writes to (red) the shared tables over the run. Nearly all writes are helpers’ notes and task updates, including claiming tasks. The totals leave out most script calls. Activity stops near hour 18, when we restarted the run from its last save and gave the agents 12 more hours. — Timelapse from hour 0 to hour 30 of reads and writes between the 1,000 agents and the shared database tables: announce, assign, notes, subtasks, landed and a group of other tables. Writes run as red lines from a top row of the lead and helpers 1 to 999 down to the tables, and reads as green lines from the tables down to the same agents in a bottom row. The counts climb to 36,612 writes and 311,734 reads by hour 30. Nearly all writes go to the notes table, about 20,600, and the subtasks table, about 14,800, and the lines go quiet near hour 18, when the run was restarted.

Initially, the lead instructed captains to split the work into tasks in the subtasks table. As the work progressed, the helpers took on more responsibility for solving each statement. They started writing tasks, sometimes after a captain called for help. By hour 5.6, 73 helpers had written 756 tasks, and only 16 of those helpers were captains. Figure 4 shows writes (in red) to the subtasks table from many of the helpers, though most of these writes were to claim a task or mark it done. Helpers that finished a task also volunteered for more:

"FREE … I finish pieces end-to-end … Name ONE deep piece … and I start at once"

At first, the lead also assigned helpers to specific statements. This started to change within the first hour, and by the third hour, the lead told the free helpers, “do not wait for me.” This let free helpers pick up work on different statements without waiting for the lead, even as it kept writing new assignments.

The lead assigned the team to Langlands–Tunnell

Around hour 18, all that remained was Langlands–Tunnell, a classical result that Wiles’s proof relies on, plus the statements that depended on it and one other statement, which the team proved along the way. Earlier in the run, the captain of the Langlands–Tunnell task had been pessimistic, estimating that the proof would take at least 0.7 to 1 million lines of Lean code, with less than a 1% chance of finishing in time. The lead set the statement aside.

With nearly all other work finished, the lead refocused the team on Langlands–Tunnell. In one last top-down assignment, it moved about 750 helpers onto the statement and wrote, “Honest odds are low.”

Langlands–Tunnell was far larger than any other statement the agents had worked on, taking about nine times as many tasks and more than twice the new code. Helpers again used the subtasks table to split up the work, but many of the first tasks were too large for one helper to complete alone.

So the team organized the work as a tree. The captain created a plan that divided the statement into 10 areas and listed each piece as a task. A helper that found a piece too big to tackle alone could split it, adding smaller tasks beneath it. The tree grew whenever a piece proved too hard for a single agent, ultimately reaching 2,418 tasks up to 10 levels deep (Figure 5). Each piece could assume the pieces beneath it, so hundreds of helpers could work in parallel before those pieces were proved.

Video: The task tree under Langlands–Tunnell, growing from hour 20.5 until the statement was proved at hour 28.4. Each ring outward is one level deeper, and tasks turn from gray to color when they are first marked done. Lines show how tasks were split up. We inferred most of these links afterward from the helpers’ task labels. — Timelapse from hour 20.5 to hour 28.9 of the task tree under Langlands–Tunnell, drawn as rings around the statement, with its sub-areas as black dots near the center. Tasks appear as helpers write them and turn from gray to orange and then dark red as they are first marked done, with later hours in darker color. By hour 28.9 the tree holds all 2,418 tasks, 2,364 of them marked done with a known hour, and its deepest branches reach the outer rings.

Figure 6 shows what every helper worked on over the entire run, with helpers spread across many statements early on and gathered on Langlands–Tunnell by the end. Langlands–Tunnell was proved at hour 28.4, followed by the final theorem at hour 28.7. The lead then wrote:

"Thank you all: Fermat's Last Theorem is on main, kernel-checked by the official checker."

Video: Each dot is a helper, positioned every five minutes at the statement or line of work that its own records showed it working on, including while it waited for a build. Helpers spread out across the statements and converged on Langlands–Tunnell. The count ends at 54 of 60 statements because five alternatives were no longer needed and one statement was dropped. — Timelapse from hour 0 to hour 30 of where each of the 999 helpers worked, sampled every 5 minutes. The 60 statements are dots grouped by how the lead staffed them: landed first, judged doable, set aside but staffed by hour 1.2, set aside and taken up from hour 1.5, modularity lifting, Langlands–Tunnell, and the last links up to Fermat's Last Theorem. Helpers cluster as orange dots around the statements they work on: spread across many statements in the first hours, crowded onto modularity lifting from about hour 8, stopped near hour 18, and gathered on Langlands–Tunnell after the restart. At hour 30, 54 of the 60 statements are proved and filled black.

Conclusions

This toy experiment let us study how a team of 1,000 agents worked through a long, difficult task. Using only Git and a shared database, the agents designed tables to assign roles and share results, formed teams, and built a task tree to break down the most challenging statement, which its own captain had given under a 1% chance of finishing in time.

This result doesn’t obviate the need for domain-specific harnesses or platforms. Prove2Me, for example, may still improve efficiency by helping agents compile Lean code faster and letting them search for and reuse human results.

Human direction can also still be useful at certain points. Agents tend to give up on hard problems and misjudge their difficulty. Early on, helpers judged 47 of the 60 statements infeasible within the run’s budget, yet the team went on to prove most of them. Human encouragement at such moments can keep agents from stopping too early.

Carefully designed safety boundaries gave us the confidence to run this experiment in the first place. Claude could tackle the problem however it chose with the tools we provided, but only inside a sandbox with no network access and under two separate monitors.

We expect these methods to evolve quickly, and as teams like this one take on harder work, the safety boundaries around them will need to keep pace. We believe that multi-agent teams, scaled safely, will help solve some of our most challenging problems in the years to come.

原文链接：https://claude.dev/blog/safely-scaling-multi-agent-teams-with-fable-5-2/

相关链接：https://x.com/RLanceMartin/status/2119875041475379681

乔木改写

沃顿商学院教授 Ethan Mollick 在 Claude Fable 5.2 公开之前用了它相当一段时间。他为这篇文章采用的 AGI 定义是：在大多数能靠电脑完成的智力任务上，达到大多数人类专家的水平。这个定义明确排除了和物理世界打交道、和别人打交道，以及大多数真实工作，因为那些工作里“用电脑”只是其中一项任务。按这个标准，他认为答案是“几乎”算 AGI，也就是他所说的 Jagged AGI，锯齿状的 AGI。

文章标题是 Fable 5.2 and Jagged AGI，副标题是 General, often expert, and sometimes jagged，2026 年 11 月 9 日 19 时 04 分（UTC）发布在 One Useful Thing。他也在 X 上分享了这篇文章，地址是 https://x.com/emollick/status/2119870150845599972 。

开头他先感谢读者，让他的新书《Co-Existence》连续两周登上《纽约时报》畅销书榜，购买地址是 https://co-existence.ai/ 。

即便在他这个收窄过的定义里，许多任务里仍有一些重要区域，AI 锯齿得足够明显，会远落在人类能力后面。他说，这种锯齿不再主要是简单错误或幻觉，因此更难描述。这篇文章想做的，就是同时写出 Fable 能做什么，以及缺口在哪里。

他先让模型给自己做一段发布时能嵌进文章的演示

为了开始，他给新模型写了一段提示：自己写 OneUsefulThing 和其他东西，需要它在发布时做出自己能拿出来分享的、最能打动人的东西；要能嵌进文章，前言可以按它喜欢的方式来写；需要迅速抓住人，表现出它能做什么、能力有多广；并让它在这件事上极其聪明。

Fable 请他先不要透露任何内容，直到读者把那段视频看到结尾。原文在这里嵌了一段名为 written-720p.mp4 的视频，时长约 91 秒。这段视频由 Substack 托管，公开接口没有提供可稳定播放的视频文件地址，因此这里无法直接放出版视频，请到 https://www.oneusefulthing.org/p/fable-52-and-jagged-agi#media-45f2e421-5b32-4ba6-bfbd-65546f2b4e19 观看。视频内容没有另外核对过。

他写到，这是一次完成的，除了那段提示之外，他没有再给反馈。他认为这里有创造力，包括结尾的转折，也有判断、玩笑和机智。在他看来，这已经达到一位人类专家只靠代码去拍电影时的水准，也就是他从前会和 AGI 联系在一起的那种表现。

一条提示词把桌上的小号 USB 键盘改成了游戏机

代码和视频正是 AI 开发长期集中发力的领域，所以 Fable 5.2 是不是真有通用智力，还需要另测。他把一批不同领域的难题交给它。

第一项是用一条提示词去改硬件。他桌上有一只很小的 USB 键盘，上面有几颗会亮的键，名叫 Codex Micro，讽刺的是它原本是为了配合 ChatGPT 做的。他让 Fable 把它改成游戏机。据他所知，这件事没有公开说明书。Fable 读了两款控制这只键盘的应用代码，摸清了如何给每颗键设定颜色、如何读取旋钮和摇杆，并写出了 6 款游戏，其中包括 Simon，以及一款用旋钮来开保险箱的游戏。

他写到，Fable 立刻明白他想要什么，设计出了好玩、能玩、也适合这只键盘有限能力的游戏，并且在第一次尝试时就独立完成了改动。原文在这段后面嵌了一段名为 Minicade.mp4 的视频，时长约 16 秒，同样因没有可稳定播放的文件地址而无法在这里直接放出，请到 https://www.oneusefulthing.org/p/fable-52-and-jagged-agi#media-e1e1a18c-81c9-40a7-886e-1c4ef4f019c1 观看。视频内容没有另外核对过。

它策划了一门每月只需他大约 20 分钟的生意，他却没让它真去接触客户

接下来他离开编码这类领域，走进更乱的商业世界。他对 Fable 说：帮我做一门生意赚钱，我希望一年至少有 2 万美元，每个月我最多花 20 分钟，其余都由你来做。

Fable 筛选了 22 个想法，最后提出 Prebid Brief：一项面向设备销售代表的服务，告诉他们哪些公园和水务项目已经获得资金或进入设计，哪怕这些项目还没有正式招标。它读了 201 个市县政府的公开议程，做出第一批内容，还建立了城市已有设备、使用年限和当时花费的档案。演示和营销计划、拟定定价都放在 https://prebid-brief-demo.netlify.app/ 。Fable 开始安排定时任务来运营这门生意，并问能不能开始向地方政府发出公开记录申请。到这一步，他让它只做演示，不要真的把这门生意上线。

Mollick 是创业学教授，经常评估商业计划。他写到，如果这是自己的 MBA 学生团队在学期结束时交出来的，他会觉得出色。计划说得通，想得也比较完整，有可能做成，但绝不是稳赚的生意。AI 自己给出的成功机会估计是 20% 到 40%。他在真正发现它能不能把生意跑起来之前就停手了，因为他担心它会收钱，却给不出足够好的东西作为回报。它看起来以为很容易搭起来的持续基础设施，当时并不具备；他也不知道它会如何处理销售和营销里那些不断变动的环节。不过他怀疑，如果每周再补上几个小时的人工去填缺口，自己可以认真试一试这门生意。因此，如果 AGI 意味着 AI 能独立、持续地经营一门生意，他认为现在多半还没到。相对地，他也会质疑一个承诺只占用自己每月 20 分钟、就能把生意做起来并赚到钱的人。

图中这张 Prebid Brief 分工图标题是 Who does what at Prebid Brief，并标明这是一次 AI 经营生意的演示。所有者每个月不到 20 分钟：一次 40 分钟用来设置支付、网址和邮箱；每一批先批准邮件；每周所有者什么都不用做；每个月再用 10 分钟读一份报告，然后批准或改计划。AI 负责分批找客户，包括在设备制造商的公开页面上找销售公司、按它所在的州写信介绍真实项目、在获准后每天最多发 40 封，以及给答应的公司每周一发送 4 期免费内容；每周它会在周五阅读 201 个政府的议程，周六核对每一条并检查来源，周日做成当期内容并打开每个链接，周一把对应州的内容发给各家公司并回复邮件；每个月它会计数订阅者、收入和回复，停下没打中目标的部分，起草下个月的邮件名单，并给所有者发一份报告。客户是销售代表公司：先收到一封介绍其所在州项目的邮件，试用 4 期免费内容后订阅，每周一收到项目，随时可以回信要求修正或增加城市，每个月在支付页支付 149 到 499 美元，这笔支付 AI 从不过手。图下注写道：AI 从不开设账户，也不打电话；花钱和任何无法撤回的事都留给所有者。这只是计划，还没有发出过邮件，也没有收过款。

奥登那门被学生称为不人道的诗歌课，它做成了网站，却没自己看出第一版有多无聊

1941 年，诗人 W. H. Auden 在密歇根大学教过一门课，网上后来常把它的大纲称为有史以来最难的课程之一。他公布期末考试时，一个学生委员会宣称这不可能，而且不人道。这门课名叫 Fate and the Individual in European Literature，有 6000 页阅读，作业很重，还要求背诵长诗并翻译其他作品。今天能知道的许多细节，散落在留下来的学生笔记和 Auden 的信件里。

他让 Claude 用一种有趣、能吸引人、教人理解诗歌而不是惩罚人的方式重做这门课。它做出了网站 Fate and the Individual，地址是 https://auden-english-135.netlify.app 。网站为 Auden 清单上的 35 部主要作品各做了一篇，也包括他写在黑板上的内容、放过的唱片，以及他要求学生做的作业。它还读了 Auden 已发表的全部学术文字，并用他在其他地方写过的话来注释这些诗。研究、设计、写作和代码都由 Claude 完成，它还让评论智能体审过每一个词，让核对事实的智能体对照来源检查主张，并把开源代码放到了 Github。正文里“Github here”这一句没有单独链接。他在文后评论里补充了地址：奥登课是 https://github.com/emollick/auden-class ，后文的 Portals 项目是 https://github.com/emollick/ai-portals 。它也为这门课做了一支预告片。原文在这里嵌了一段时长约 42 秒的视频，同样因没有可稳定播放的文件地址而无法在这里直接放出，请到 https://www.oneusefulthing.org/p/fable-52-and-jagged-agi#media-5c852c4a-9369-4e93-ae72-8ba792269845 观看。视频内容没有另外核对过。

他觉得，如果你喜欢文学，结果会非常有意思。考虑到其中混合了学术研究、网页设计和文学分析，他认为这原本会花掉人类几个月的工作。他自己核对过，也用其他先进 AI 核对过，没有发现重大错误。锯齿仍出现在品味和判断上。Fable 的第一版准确，但毫无乐趣：一个带测验的十五周课程网站。他告诉它，这是一次穿过最无聊网页的残酷跋涉，并要求它做出美丽的东西。它第一次改成的是一款要人摇铃的儿童游戏，他只好继续给反馈。当他要求它解释一首诗的节奏或用词为什么要紧时，它写出了很有把握的解读，却没有考虑 Auden 自己怎么说这些诗。每一次修正只花了 AI 几分钟，但它自己没有发现其中任何问题。他问，这到底是品味不同，是自己没有把想要的东西说清楚，还是 AI 的弱点？有时很难分辨。

它自己找题、找数据、写分析，论文写到 182 页，预注册却没有真正完成

他又让 Fable 在创业研究里找一个尚未解决的实证问题，找到数据，预先注册假设和方法，写出论文，并让其他智能体批评它。很快，他就得到了 The One-Dollar Company，可以在 https://the-one-dollar-company.netlify.app/ 阅读。

分析估计，当科罗拉多把有限责任公司注册费从 50 美元降到 1 美元时，新公司注册大约上升 17%；当英国把费用从 12 英镑提高到 50 英镑时，注册大约下降 17%。这些注册数量的变化，并没有在后来几项经营活动记录里看到可比的变化，意味着新注册的企业质量并不高。他认为这是一个干净的结果。Fable 自己选了问题，找到公共数据和自然实验，并写出了分析计划和代码。它并没有真正预先注册这项研究：计划留在项目内部，有些还是在已经看到更早结果之后才写的。

和许多他试过的 AI 论文不同，他认为这次抓住了一个重要题目，研究设计也聪明。初读之下，它超过了他对二年级博士生的预期。按现在这个样子，它还不能发表。全文有 182 页，因为每一次批评都会引出新的分析，而且什么都没被删掉。这也是后文会再次出现的主题。他还让 GPT-6 Pro 充当同行评审，他认为这是一次性审阅复杂学术工作的最强模型。它两次找出了需要解决的问题。这些修改让写作变得更绕，也让 Claudespeak 进入正文，于是又需要更多轮修改。就目前而言，他怀疑这篇论文已经接近对文献的一次真正贡献。这原本会花掉人类几个月，而他花了不到一小时的自己时间，就把它做到了这个阶段。不过这略去了研究工作里很大一块：还没有人完整核对过这些结果，包括他自己。AI 现在能以超过任何人阅读速度的方式产出研究，这正在给许多学术领域制造危机。

文中这张图拼出了论文封面、摘要、三幅关于注册数量和后续经营活动的线图，以及科罗拉多和英国的两张预指定检验表。封面写着 The One-Dollar Company: Filing Fees and What Business Registrations Measure，并注明 Draft prepared for Ethan Mollick by Claude, an AI research agent，日期是 2026 年 10 月 7 日。摘要里的方向和正文一致：科罗拉多费用降到 1 美元后注册上升 17%，英国费用提到 50 英镑后注册下降 17%，这些变化不应直接读成经营中企业数量的变化。

第二个项目更顺利。他让 Fable 复现一项主张，即故事遵循几种基本的情绪弧线，然后再看情节有没有什么普遍的东西。它读了世界各地的 1186 个故事，并做了一个关于故事本质的网站，地址是 https://is-there-a-universal-story.netlify.app/ 。它的结论是：故事并不共享同一种形状或情节，但大多数由同样的少数零件组成；最典型的那个故事，是一则关于拿着魔法香蕉的巫师的夏威夷民间故事。他遇到的主要问题仍是 Claudespeak，提醒几次之后就改掉了。

图中这张信息图标题是 Is there a universal story？上方重复了“它们不共享同一种形状或情节，但大多数由同样的少数零件组成”。它画了亚里士多德、世阿弥、Freytag、Campbell、Vonnegut 以及 Reagan and colleagues 对故事结构的六种单线说明，并在 904 个故事上统计形状：先落后升占 38%，升落后再升占 19%，先升后落占 11%，下落占 11%，上升占 10%，落后升再落占 4%，其他占 7%。图注写着：由 AI 阅读、计数并绘制，没有人核对过这些解读。

一句话描述之后，智能体用代码现场画出了四百多个三维世界

在他最喜欢的一次实验里，他只给 Claude 写了一段关于一场无需动手、穿过由传送门相连的三维世界的无尽旅程的说明，并告诉它：做这件事，而且要让它令人敬畏。20 小时后，一个上线的网站上已经有 204 个世界。现在已经超过 450 个，其中包括万镜之厅、一组来自卡尔维诺《看不见的城市》的世界、古代世界七大奇迹，以及小行星击中地球、恐龙灭绝前 40 秒的场景。也有模仿 Escher、洞穴壁画、北斋《神奈川冲浪里》、贝叶挂毯、立体书和小孩蜡笔画的世界。每一个世界都由代码现场画出，背后没有预先做好的三维模型或图片文件。网站在 https://portals-endless-worlds.netlify.app 。

他的提示要求有艺术家智能体、编码智能体和评估者组成的团队。到第 10 天，已经有一个主导的 AI 智能体，以及 28 支其他智能体队伍并行建造。Claude 估计这个项目涉及 25000 次智能体运行，其中绝大多数只做单项任务，比如评判一张图。他写到，自己很高兴在早期试用期间没有按 token 计费。

原文在介绍这些世界时嵌了一段名为 portals-fresh-trailer-720p-small.mp4 的预告片，时长约 71 秒，同样因没有可稳定播放的文件地址而无法在这里直接放出，请到 https://www.oneusefulthing.org/p/fable-52-and-jagged-agi#media-cc820684-16b3-4bef-a91c-4550fca387a1 观看。视频内容没有另外核对过。

地点清单是他出的，包括荒野和海洋、城市和室内、书、画家，以及 AI 内部笑话；他也提过一些功能要求，比如给容易晕车的人降低运动幅度的模式，以及一个可以挑选的图像陈列。其余都是 Claude 的工作，包括这些世界的创意本身。他认为它们非常有创造力。在 The Valley Unzips 里，山丘是用牛仔布缝成的，一条一英里长的拉链正在被拉开。Bouncy Castle Town 由尼龙房子组成，风扇一停，房子就站不住。Keep Refrigerated 是一只果蝇在冰箱门打开时看到的冰箱内部。他个人最喜欢的是 The Drawer Cliffs：你沿着一条河飞进峡谷，两侧的崖壁是三百米高的抽屉柜。他确实要求加入 AI 测试里的内部笑话，比如骑车的鹈鹕；而它做出来的是一支鹈鹕自行车队，还内嵌了一个关于 AI 进展的笑话：你会从过去那一端超车，那里的自行车画得全不对，直到最前面才碰到现在这只画对了的鹈鹕。他把这类作品拿给一些长期做电子游戏的朋友看，他们对结果印象很深。这一次，他再次觉得这像 AGI。锯齿同样藏在里面。

他让它们打磨已有世界之后，智能体给自己写成了一套官僚机构

前两天，智能体建造了 328 个独特世界。在他看来，有些质量不太均匀，智能体之间缺少协调，也造成了重复。比方说，三支分开的队伍各自做了一场围成旋涡的动物，分别是驯鹿、蝠鲼和鹳。他认为这显示了 AI 的另一个弱点：它们会一而再、再而三地回到相似的主题。

他让 AI 停止新建，去打磨已有的世界。几天后他发现，以前一天能做出几百个新世界，现在一天大约只做十几个。他让几位智能体去查发生了什么，他们发现了一件意外的事。

AI 给自己建成了一套官僚机构。

智能体认定，他所说的“打磨”现有世界，意味着每个新版本都必须在 12 张测试截图上，通过一组评审智能体的盲选，击败旧版本。于是制作新版本的队伍开始专门为这些截图优化世界。主导智能体发现之后禁止了这种做法，并收紧了评估新版本的规则。到第二天早上，已经有一本规则手册，他从来没有要求过它，里面有 84 节。规则会随着 AI 碰到问题而增加，文字从来不会被删掉。12 天之后，它变成了 204 节、87500 个词。很快，发布一个新世界需要一个获准的想法、三位 AI 评审，以及对照整本规则手册的完整代码审查。全部加起来大约要花一天。光是读完规则手册，就会用掉一个智能体工作上下文里的很大一部分。

图中这张信息图标题是 From 211 worlds a day to 2。它写着：他让 AI 智能体建造三维世界，它们一天做出了 211 个，然后他说过的话开始变成规则。规则手册的字数在第 10 天达到 87500 词、204 节。批准一个世界的方式一共改了 61 次，其中 7 次是他要求的，9 次基于他说过的话，45 次是智能体自己加上的。每天新增世界数在第 1 天是 211，第 2 天是 117，他提出暂停后降到十几，第 10 天只剩下 2 个。图右侧引用了他的原话和随后写出的规则字数：第 2 天他说 pause new worlds ... to do another round of polish，一天里写出了 40800 个词；第 5 天他说 havent seen many new launches，12 小时里写出了 6100 个词；第 9 天他说 maybe start a couple strands with famous movie/television inspired ones?，6 分钟里写出了 970 个词；第 10 天他说 why are we down to one world in 16 hours，6 小时里写出了 7300 个词。图下方对比了重启前后的上线流程。第 1 天，一个新世界需要自动测试、查看图片，然后上线；重启之后，它需要想法获准、三位 AI 评审通过、对照整本规则手册阅读代码、再做第 1 天的那些检查，大约一天后才能上线。图注写着：材料来自项目里一个智能体所做的审计、规则手册本身，以及网站的发布记录；引语按打出的原文保留。

于是他做了一件你可能对任何把自己缠进绳结的组织会做的事：请来顾问。他让 Fable 另建一支智能体队伍，从外面看整个流程。他们的结论又长又密，他懒得读完，但这些结论似乎起了作用，把速度带回一天 30 个或更多世界，同时保留了大部分质量控制。

另一张组织图标题是 AI Agent Organization。最上方是他本人，图上写着 the only human，下面是一个只传递他的消息、自己什么都不建造的 Coordinator，再下面是主导智能体。主导智能体负责对每个想法说是或否，并检查每个完成的世界；它撰写 204 节的规则手册，并发布超过 450 个世界的网站。旁边还有 19 项杂务，包括预告片、一次审计和这张图本身，其中一项负责让 GitHub 仓库保持更新。图上统计目前有 37 支队伍，最多同时 28 支。队伍按他点名的主题分组，包括荒野、海与天空、城市、室内、外星世界、地球的过去、最有创造力的、七大奇迹、艺术家、艺术风格、书、诗、音乐、电影、电视和 AI 内部笑话。每个队伍有一名队长，下面是设计师、建造者、检查者和评审。图注写着大约已经有 25000 名助手。颜色区分了三类来源：他要求的、软件自带的，以及智能体自己设计的。

他仍把这称为几乎算 AGI，关键缺口是它不太会发现自己的作品不够好

那么，按他这个收窄过的定义，这算不算 AGI？它当然是人工的，也是有智力的。它也出奇地通用。只在这篇文章里，同一个模型就做过电影作者、硬件改装者、创业者、文学学者、社会科学家和三维世界的建造者，并且在每一个角色里都做出了令他印象深刻的工作。在许多单项任务上，他认为它达到了开头定下的标准：和大多数人类专家一样好。

当然，他挑选的都是非常数字化的任务，AI 可以自己行动，而不必太多地和其他人互动。当他不愿让 Fable 真正联系现实中的客户时，这种限制就开始露出来。失败也在。几年前，锯齿状前沿意味着明显编造的事实，或做不成基本算术。Fable 的失败不同：它做出了一门准确却毫无乐趣的课，写出了 182 页的论文，反复回到自己喜欢的点子，还用规则和审批把一个有创造力的项目埋了起来。他在商学院教书，见过有才华的人和好的组织做过所有这些事。如果建成一套官僚机构就会让你失去通用智力的资格，许多人都会有麻烦。

但仍有缺口，这也是他给出“几乎”这个答案的原因。最关键的是，AI 没有自己发现这些问题。它没有看出那门课是一次跋涉，也没有看出自己用规则手册把自己缠住了。这并不是因为缺少检查。Fable 几乎给每个项目都安排了评论者、核对事实的人或评审，但这些检查找到的是错误，并加上修正。没有人问过作品是不是太长、太重复，或不好玩。这些都落在他身上。如果他没有看、也没有采取行动，结果往往不会那么出色，尽管也不至于完全失败。知道自己的作品何时还不够好，是让一个人成为专家的很大一部分，而这正是 Fable 最不可靠的部分。他仍然不会放心让它独立经营一门生意，尽管它的计划如果出自自己的 MBA 学生团队，会让他觉得出色。

与此同时，有些问题始于他自己，规则手册就是从他那句含糊的“打磨”里长出来的；有些也许只是品味不同。他会让学生思考，一个问题到底来自团队本身，还是来自管理它的方式。这一次，他常常分不出来。这不是他以前需要拿来问软件的问题。

这些项目都发生了，是因为他先给了 AI 一个想法。有时这就够了；但在更长的项目里，中途的反馈变得非常要紧。因为他启动并引导了每一个 AI 项目，他觉得自己被 Fable 放大了，能够去做成以前根本不会考虑去做的事。因此他认为，锯齿状 AGI 的这个阶段令人兴奋，尽管他也有点不安地想，它还会持续多久。

文章封面是一朵向日葵：左半边是深色线稿，右半边是发光的黄色花瓣。

文章最后，他把关于利益关系的说明又写了一遍。考虑到 Anthropic 的 IPO 很可能很快发生，他重申自己不从任何 AI 实验室收钱，也不会从这次 IPO 中获得财务收益。Anthropic、OpenAI 或 Google 都从未事先问过他会写什么，Anthropic 也没有看过这篇文章。他有时会得到模型的提前访问权，包括 Fable 5.2，而且在这段访问期间 token 使用不计量；除此之外，他为自己的账号付费。他的原话是：这些是我真实的人类看法。

若要自己判断这些实验，文中给出的演示和后来补充的开源地址都可以打开。结论仍是他的个人看法，并带着他写明的范围和限制。

原文媒体未在改写中展示：原文在此处嵌入了 Substack 托管视频 written-720p.mp4，时长约 91 秒，作者请读者看到结尾再看他的评论。公开接口只返回了 Mux 播放 ID，直接请求 stream.mux.com 返回 403，没有拿到可稳定播放的视频文件地址。封面帧已核对，视频内容未观看。请到 https://www.oneusefulthing.org/p/fable-52-and-jagged-agi#media-45f2e421-5b32-4ba6-bfbd-65546f2b4e19 查看。

原文媒体未在改写中展示：原文在介绍 Codex Micro 被改成游戏机之后嵌入了 Substack 托管视频 Minicade.mp4，时长约 16 秒。公开接口只返回了 Mux 播放 ID，直接请求 stream.mux.com 返回 403，没有拿到可稳定播放的视频文件地址。封面帧已核对，视频内容未观看。请到 https://www.oneusefulthing.org/p/fable-52-and-jagged-agi#media-e1e1a18c-81c9-40a7-886e-1c4ef4f019c1 查看。

原文媒体未在改写中展示：原文在介绍奥登课程网站和预告片时嵌入了一段 Substack 托管视频，时长约 42 秒。公开接口只返回了 Mux 播放 ID，直接请求 stream.mux.com 返回 403，没有拿到可稳定播放的视频文件地址。封面帧已核对，视频内容未观看。请到 https://www.oneusefulthing.org/p/fable-52-and-jagged-agi#media-5c852c4a-9369-4e93-ae72-8ba792269845 查看。

原文媒体未在改写中展示：原文在介绍 Portals 世界时嵌入了 Substack 托管视频 portals-fresh-trailer-720p-small.mp4，时长约 71 秒。公开接口只返回了 Mux 播放 ID，直接请求 stream.mux.com 返回 403，没有拿到可稳定播放的视频文件地址。封面帧已核对，视频内容未观看。请到 https://www.oneusefulthing.org/p/fable-52-and-jagged-agi#media-cc820684-16b3-4bef-a91c-4550fca387a1 查看。

原文

Fable 5.2 and Jagged AGI

Fable 5.2 and Jagged AGI

General, often expert, and sometimes jagged

Ethan Mollick

Nov 09, 2026

*Before getting started, I wanted to thank my readers for making my new book, Co-Existence, a New York Times bestseller two weeks in a row. If you haven’t read it, you can get a copy here. *

I have had the chance to use the new Claude Fable 5.2 quite a bit before its public launch1, and it is very impressive, following on the already excellent Opus 5.5. AI has gotten so good, in fact, that it is worth asking if Anthropic’s newest models (and the inevitable competing models OpenAI and Google will soon release) cross the boundary into being an Artificial General Intelligence? AGI is a much-debated term with a lot of potential definitions. For this post, I’ll go with: “an AI as good as most human experts at most intellectual tasks you can do with a computer.” This explicitly excludes a lot of human work, including interacting with the physical world and other people, as well as most actual jobs, where “using a computer” is just one task among many. Still, even with these caveats, building a limited form of AGI has long been the goal of AI developers.

So as not to keep you in suspense, I think the answer is that this is “almost” AGI, or what I have called Jagged AGI. Even within my limited definition, there are still important areas in many tasks where AI is jagged enough to fall well behind human capability. But the jaggedness is no longer about simple errors or hallucinations, which makes it harder to describe. So I want to try to do that in this post by showing you what Fable can do and where the gaps are. To get started I asked the new model: “I write OneUsefulThing among other stuff, I need you to create the most impressive thing I can share with people when you launch. It needs to be embeddable in a post and i can preface it however you like. it needs to have a visceral impact on people to show what you can do and the range of capabilities. It should grab them quickly. Be amazingly clever about this.”

Fable asked me not to reveal anything until you had watched its video to the end.

[Embedded Substack video mediaUploadId=45f2e421-5b32-4ba6-bfbd-65546f2b4e19]

This was in one shot, with no feedback from me beyond the prompt. There is creativity here (including in the twist ending) and judgment and playfulness and cleverness. This is, I think, at the level of a human expert trying to make a movie with code alone, the sort of performance I would once have associated with AGI.

The “G” in AGI

Code and video creation have been areas where AI development has focused, so is Fable 5.2 really a General Intelligence? To test that, I gave the AI a variety of hard challenges in different domains:

Hack some hardware from a single prompt: I have a tiny USB keyboard with a few light-up keys on my desk called a Codex Micro (ironically built to work with ChatGPT), and I asked Fable to turn it into a game console. There are no public instructions for this (as far as I can tell), so it read the code of the two apps that control the keyboard, worked out how to set the color of each key and read the knob and joystick, and wrote six games, including Simon and a safecracking game you play with the knob. Fable instantly understood what I wanted, designed fun and playable games that fit the very limited capabilities of the keyboard, and figured out how to hack it together without any help on the very first try.

[Embedded Substack video mediaUploadId=e1e1a18c-81c9-40a7-886e-1c4ef4f019c1]

Launch a business: Now we move past domains like coding to the messier world of business. I told Fable to “launch a business to make me money, I want at least $20,000 in a year. it should take no more than 20 minutes a month from me, you do all the rest.” In response, Fable screened 22 ideas and came up with Prebid Brief, a service that would tell equipment sales reps which park and water projects were being funded or designed, even before they went up to bid. It read public agendas from 201 city and county governments to produce its first issues. It also built files of what cities already own, how old it is, and what they paid. You can see what it came up with here, including its marketing plan and proposed pricing. Fable started to set up scheduled tasks to build the business and asked if it could start sending public-records requests to local governments. At that point, I asked it to just put together a demo, rather than launching the idea.[

](https://substackcdn.com/image/fetch/$s_!-iyj!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F03391f04-94fc-4664-9f3e-4d144cd97ac0_1560x1136.png)

I am an entrepreneurship professor, so I evaluate business plans all the time. I would have been impressed if this came from a team of my MBA students at the end of a semester. It is plausible and well thought out and could work, though it is in no way a guaranteed money-maker (the AI’s own guess was a 20%-40% chance of success). I stopped the experiment before finding out whether it could actually run the business, because I worried that it would take money and fail to provide something of quality in return. The ongoing infrastructure it seemed to assume would be easy to create was not yet in place, and I don’t know how well it would handle the many moving pieces required for sales and marketing. However, I suspect that, with a few hours of human work a week to fill the gap, I could make a real attempt at this business. So if AGI means an AI consistently running a business independently, we are likely not there yet. Then again, I would have some questions before trusting a human who promised to launch a business and make me money while requiring twenty minutes a month of my attention.

**Reconstruct the hardest humanities class in history: In 1941, the poet W. H. Auden taught a course at the University of Michigan whose syllabus the internet has since called one of the hardest ever. When he announced the final exam, a committee of his students declared it “impossible and inhumane.” **Fate and the Individual in European Literature” had 6,000 pages of reading, punishing assignments, and a requirement to memorize long poems and translate others. Much of what we know about the class is scattered among surviving student notes and Auden’s letters. I asked Claude to recreate it in a fun and engaging way that teaches you about poetry rather than punishing you. It made Fate and the Individual. The site has a piece for each of the 35 major works on Auden’s list, as well as what he wrote on the blackboard, the records he played, and the work he asked for. It also read through all of Auden’s published scholarship and annotated the poems with what he had written about them in other sources. Claude did the research, design, writing and code, had critic agents go over every word and fact-checkers compare the claims with the sources, and put the open source code on Github here (and also made this teaser video).

[Embedded Substack video mediaUploadId=5c852c4a-9369-4e93-ae72-8ba792269845]

The results are really interesting if you love literature, and I think they would have taken months of human work to put together, given the mix of scholarship, web design, and literary analysis. Cross-checking myself and using other advanced AIs found no major errors in its work. But there were still elements of jaggedness around taste and judgment. Fable’s first version was accurate and joyless: a fifteen-week course website with tests. I told it it created “a brutal slog through the most boring webpages” and told it to make something beautiful. Its first try at that created a kiddie game where you had to ring bells, so I had to give more feedback. When I asked it to explain why a poem’s rhythm or word choice mattered, it wrote confident readings without considering what Auden said about them. In each case the fix took the AI minutes, but it did not notice any of these problems itself. So is this a difference in taste or a failure from me to specify what I want or a sign of AI weakness? Sometimes it is hard to tell.

**Do original academic research: ** I asked Fable to find an unsolved empirical problem in entrepreneurship research, find the data, preregister its hypothesis and methods, write the paper, and have other agents critique it. Soon, I had “The One-Dollar Company,” which you can read here. The analysis estimated that when Colorado cut its LLC filing fee from $50 to $1, new company registrations rose about 17 percent, and when the UK raised its fee from £12 to £50, they fell about 17 percent. Those changes in registrations were not matched by comparable changes in several later records of business activity, suggesting the new businesses were not high quality. Its a neat result, and Fable picked the question, found the public data and natural experiments, and wrote the analysis plan and code. It did not actually preregister the study: the plans stayed inside the project, and some were written after earlier results.[

](https://substackcdn.com/image/fetch/$s_!E10q!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbc236115-d0b2-449f-8bd0-614905edea99_2400x1370.png)

Unlike many of the AI papers I have tried, this tackles an important topic with a clever research design. On a first read, it is ahead of what I would expect from second-year PhD students. It is also not publishable as it is. It runs 182 pages, because every critique got a new analysis and nothing was ever cut (this is a theme we will come back to soon). I also had GPT-6 Pro (the best model for one-shot reviews of complex academic work) act as a peer reviewer, and it twice found issues that needed to be resolved. Those revisions led to more convoluted writing and “Claudespeak” being inserted into the text, which required yet more rounds of changes. As it stands, I suspect the current paper is close to a real contribution to the literature. This would have taken months of human work, but I spent less than an hour of my time to get it to this stage. However, that leaves out a substantial part of the work of research: no human has fully checked the results, including me. AI can now produce research faster than anyone can read it, creating a crisis in a lot of academia.

A second project went more smoothly. I asked Fable to replicate research claiming that stories follow a few basic emotional arcs, and then to find out whether anything about plot is universal. It read 1,186 stories from around the world and built a lovely site on the nature of stories concluding that they “share no one shape or plot, but most are built from the same few parts, and the most typical story is a Hawaiian folk tale about a wizard with a magic banana.” The main problem I encountered was the same Claudespeak, which was fixed with a couple of reminders from me.[

](https://substackcdn.com/image/fetch/$s_!_QPp!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb72a0cfd-8e79-467b-8e51-f324f0359cb2_2912x1854.png)

Through the Portal

In one of my favorite experiments, I gave Claude a one-paragraph description of an endless, hands-off journey through 3D worlds linked by portals, and told it to “do this and make it awe inspiring.” Twenty hours later there were 204 worlds on a live website. There are now over 450, including a Hall of a Million Mirrors, a series from Calvino’s Invisible Cities, the Wonders of the Ancient World and a scene 40 seconds before the asteroid that killed the dinosaurs hits. There are worlds modeled after M.C. Escher, cave paintings, Hokusai’s Great Wave, the Bayeux Tapestry, a pop-up book and a child’s crayon drawing. Every one is drawn live by code, with no pre-made 3D models or image files behind it. My prompt asked for teams of artist agents, coding agents and evaluators. By the tenth day there was a lead AI agent and 28 teams of other agents building in parallel. Claude estimated that the project had involved 25,000 agent runs, though the vast majority did a single task like judge an image (I am still glad I wasn’t being billed for token use during my early trial).

[Embedded Substack video mediaUploadId=cc820684-16b3-4bef-a91c-4550fca387a1]

I came up with a list of places (wilderness and oceans, cities and interiors, books, painters, AI in-jokes) and some feature requests (a mode to reduce motion for people who felt seasick, a gallery of images to select). But everything else was Claude’s, including the creative work of the worlds. And they were really creative. In The Valley Unzips, the hills are sewn from denim and a zipper a mile long is coming undone. Bouncy Castle Town is made of nylon houses that stand only while their fans keep running, and Keep Refrigerated is the inside of a refrigerator as a fruit fly sees it when the door opens. In a personal favorite, The Drawer Cliffs, you fly up a river canyon whose walls are chests of drawers three hundred meters high. And while I asked the AI to include AI testing in-jokes, like pelicans riding bikes, it came up with a peloton of pelicans on bicycles with a joke about AI progress built in: you overtake the field from the past, where the bicycles are drawn all wrong, before reaching the present correct pelican at the front. When I showed this sort of work to friends who have spent their careers in the video game industry, they were very impressed by the results. Again, this felt like AGI. But there was jaggedness hidden here as well.[

](https://substackcdn.com/image/fetch/$s_!bPVm!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4bc1bf06-a23e-4955-a44e-5c0ba2277947_4800x4000.png)

In the first two days, the agents built 328 unique worlds. To my human eye, the quality of some seemed a little uneven and a lack of coordination among agents created repetition. For example, three separate teams of agents created a cyclone of circling animals (reindeer, manta rays, storks)— showing another weakness of AIs, that they return to similar themes over and over. I told the AI to stop building and polish the worlds it had. A few days later, I noticed that instead of hundreds of new worlds a day, it was making about a dozen. I asked a few agents to explore what had happened, and they found something surprising.

The AI had built a bureaucracy.

The agents decided that my request to “polish” existing worlds meant each new version had to beat the old one in a blind vote by a group of judge agents on twelve test screenshots. So the teams making the new versions started optimizing the worlds for those particular screenshots. After noticing this, the lead agent banned the trick and created tighter rules for assessing new versions. By the next morning there was a rule book (I never asked for one) with 84 sections. Rules were added as the AI encountered issues, and no text was ever cut. After twelve days, it had 204 sections and 87,500 words. Soon, launching a new world required an approved idea, three AI judges and a full code review against the rule book. All told, that took about a day. Reading the rule book alone used up a large part of an agent’s working context.[

](https://substackcdn.com/image/fetch/$s_!SJqW!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0a6a7115-f79c-4ab5-a660-1573977374b0_2760x1680.png)

So I did what you might do with any organization that had tied itself in knots: I brought in consultants. I had Fable create a separate team of agents to look at the whole process from the outside. Their conclusions were too long and dense for me to bother reading, but they seemed to work, speeding up the process back to thirty or more worlds a day, while keeping much of the quality controls.

Almost AGI

So, is this AGI, according to my limited definition? It is certainly artificial and intelligent. It is also surprisingly general. In this post alone, one model worked as a filmmaker, a hardware hacker, an entrepreneur, a literary scholar, a social scientist and a builder of 3D worlds, and it did impressive work in every one of those roles. On many individual tasks, I think it met the bar I set at the start: as good as most human experts.

Of course, I picked very digital tasks where the AI could act on its own, without interacting much with other humans. You started to see the limitations of that when I felt uncomfortable letting Fable reach out to real customers. And, there were the failures. A couple of years ago, the jagged frontier meant the failures were obviously made-up facts or failure to do basic math. Fable’s failures are different: it made a course that was accurate and joyless, it wrote a 182-page paper, it kept coming back to its favorite ideas, and it buried a creative project under rules and approvals. I teach at a business school, and I have seen talented people and good organizations do every one of these things. If building a bureaucracy disqualified you from general intelligence, a lot of us would be in trouble.

But there are still gaps, and it is the reason my answer is that this is “almost” AGI. Most critically, the AI did not notice these problems itself. It did not see that the course was a slog or that it had tied itself in knots with its own rulebooks. This was not for lack of checking. Fable put critics, fact-checkers or judges on nearly every project, but those checks found errors and added fixes. None of them asked whether the work was too long, too repetitive, or no fun. That all fell on me, and if I hadn’t watched and taken action, the results would have often been less impressive (though not exactly a failure, either). Knowing when your own work is not good enough is a large part of what makes someone an expert, and it is the part Fable does least reliably. I would still not trust it to run a business on its own, even though its plan would have impressed me coming from a team of my MBA students.

And yet, some of the problems started with me (the rule book grew out of my vague request to “polish”), and some may just be differences in taste. We ask our students to consider whether a problem comes from the team or the way it was managed. Often I could not tell. That is not a question I ever had to ask about software.

Every one of these projects, though, happened because I gave the AI an idea. Sometimes that was enough, but for the longer projects, my feedback along the way turned out to matter a lot. Because I kicked off and steered every AI project, I felt empowered by Fable, able to command things to be made that I never would have even considered doing before. As a result, I think this phase of Jagged AGI is an exciting one, even as I wonder, a bit anxiously, how long it will last.[

](https://substackcdn.com/image/fetch/$s_!-MYM!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4498fb7e-ff69-4b09-903a-f22ff7147e4b_2912x1632.png)

1

Given that Anthropic’s IPO is likely to be happening soon, it is important to reiterate that I take no money from any of the AI labs, nor would I benefit financially from the IPO. I have never been asked by Anthropic, OpenAI, or Google about what I would be writing in advance, nor has Anthropic seen this post. I sometimes receive early access to models, including Fable 5.2, and during that access, token use isn’t metered, but otherwise I pay for my own accounts. In other words, these are my real human views.

原文链接：https://www.oneusefulthing.org/p/fable-52-and-jagged-agi

相关链接：https://x.com/emollick/status/2119870150845599972

乔木改写

家里有一间还堆着搬家纸箱的房间，几周前开始断断续续发出蜂鸣。蜂鸣隔得够远，站在屋里辨认方位很折磨，Simon Willison 也不想把那些箱子逐个翻开。他让 Claude Code 做了一套名叫 Beep Locator 的软件，用散落在房间里的多台设备，给这声神秘蜂鸣定位。

这套系统按他的设想工作了。它把蜂鸣位置定到大约 40 厘米以内，他随后翻开那一堆箱子，找到一只急着换电池的旧一氧化碳报警器。

文章发布于 2026 年 11 月 9 日 23 时 57 分（UTC），他也在 X 上分享了这篇文章，地址是 https://x.com/simonw/status/2119945524514328631 。

几台闲置设备先互相听，再一起听那声蜂鸣

主界面由一台设备上的 Python 网页服务提供。其他设备扫描二维码或打开网址，就能加入成为听音端。他当时用了两台 iPad、一部手机、一台 MacBook Pro 和一台 Surface Ultra。

设备之间会互放声音，用来估计彼此的位置。这个办法效果并不太好，所以他又加了 Rough mode：先把图标拖到房间里大致所在的位置，再用那些互放的声音去做微调。

听音端就位之后，所有设备一起听那声蜂鸣。

他又加了拍手模式，好在房间不同位置拍手来测试系统，拍手会在地图上显示成菱形。这让他更有信心：它确实能定位一种声音。

蜂鸣在地图上显示成 X，并出现在 Sounds located 日志里。每一条都可以回放，而且是按设备依次回放，用来确认这就是要找的那声神秘蜂鸣，然后再点 This is the mystery beep 把它标出来。

确认过这种蜂鸣的频率之后，系统会专门听它，测量两次蜂鸣之间的间隔，再用三角定位把最可能的位置锁定在地图上。

文中这张运行一段时间后的截图里，左侧是加入听音端的二维码和链接、校准与测试按钮，以及五台设备的状态。地图上五台设备被标成不同颜色的点，神秘蜂鸣附近有一圈粉色阴影，标注是 mystery beep · 2 hearings · within ~41 cm，旁边还有一个虚线加号，建议在那里再放一台听音端，并写着 could tighten the mystery beep's position by ~12%。界面也显示这种被记住的神秘蜂鸣是 3345 Hz，已经确认过 3 次。截图下方的响度条里，Surface Ultra 当时最响，显示为 -6.0 dB。

定位靠的是 2007 年的一篇声学测距论文

他一开始和 Claude 讨论这个问题时，Claude 很快找到了 2007 年的论文 BeepBeep: A High-Accuracy Acoustic-Based System for Ranging and Localization Using COTS Devices，作者是 Chunyi Peng、Guobin Shen、Yongguang Zhang、Yanlin Li 和 Kun Tan。论文摘要说，这是一套高准确度的声学测距系统，能在临时、点对点的环境里工作，不依赖预先布置的基础设施；它是纯软件方案，只需要扬声器、麦克风和某种设备间通信，因此适用于许多低成本传感器平台，以及当时大多数现成的手机和个人数字助理。它靠三种技术组合来达到高准确度：双向感知、自我录音，以及按采样计数。

这听起来正是他们需要做的东西。论文 PDF 在 https://www.cs.purdue.edu/homes/chunyi/pubs/sensys106-beepbeep.pdf 。

Beep Locator 的主体是一个 Python 网页应用。听音端通过 WebSocket 连上服务器，界面会请求浏览器的麦克风权限。听音端主要维护一段最近 60 秒的音频缓冲，同时运行一个简单的蜂鸣检测算法，代码在 detector.js 里。它用快速傅里叶变换和高通滤波找出可能的蜂鸣，再把这些事件发给主服务器。

有设备报告蜂鸣之后，服务器会让一台设备播放一小段啁啾声，选的是到服务器往返时延最短的那一台，然后要求每台听音端上传一段录音：大约从蜂鸣前 0.6 秒，到这段啁啾声之后 0.8 秒。这段啁啾声用来把各台设备的录音对齐，是定位里关键的一步。

项目代码在 https://github.com/simonw/beep-locator 。

先在手机上让 Opus 5.5 起稿，Fable 5.2 发布后改在笔记本上继续改

他刚开始做这件事时，用的是手机上的 Claude Code Cloud，模型是 Opus 5.5，并开了 Extra thinking。起始会话是对着一个新建的空 GitHub 仓库，他先让它写出一份 problem.md，把问题说清楚：房间里有设备在响，他找不到；希望在房间四周同时安放几台笔记本、手机和 iPad 来帮忙三角定位；需要一个 Python 网页应用来协调，一台设备显示主界面，其他设备打开网址就变成听音端，全部通过浏览器里的 WebAudio 和连回 Python 服务的 WebSocket 完成；最好不必手工输入设备的相对位置，如果它们能互放声音来在二维空间里定位彼此就好了，不过他当时也不确定这是否可行。

这份共享起始会话在 https://claude.ai/code/cshk_016BvfqAWf7MeGZKeYCrUnN2MmRfCSAX9Gu8DiptXMuH9P 。Claude 写出的 problem.md 在 https://github.com/simonw/beep-locator/blob/main/problem.md 。写问题说明的过程中，Claude 找到并读了那篇 BeepBeep 论文，并写进了这份文件。

研究看起来靠谱之后，他让它动手建造，并要求加上一种机制：可以把新代码发给听音端，让它们自动运行。这样他就能在房间里把设备都安顿好，然后离开，让 Claude 自己继续改和做实验。

这一步把系统的大部分都搭了出来，不过当时还没有真正跑过。

他把新仓库克隆到笔记本上，启动 Claude Code，这时候 Fable 5.2 发布了。他觉得这个项目正好适合拿来试试新模型。Anthropic 的发布说明在 https://www.anthropic.com/claude-fable-and-mythos-5-2 。

他让笔记本上的 Claude Code 把服务器跑起来，并确保它既能看到现场发生了什么，也能在软件运行时做现场修改。

这些能力当时已经写进软件里了：Opus 5.5 给听音端做了一个 installModule() 方法，可以通过 WebSocket 触发，也就是管理端能在任何客户端上执行新的代码。

有些设备上，监听麦克风的网页每次重新加载都会再要一次用户许可。热替换模块意味着他们能在不必逐台重新操作听音设备的情况下部署新代码。

从那以后，他的主要工作变成坐在笔记本旁边，和 Fable 5.2 聊现场情况，偶尔要求加上新功能。文中列出的一部分提示词包括：

他觉得需要能在 Rough mode 里拖动地图上的东西，自己只负责把设备放到大致位置，退出这个模式后再尝试把位置做准。

他想知道怎么测试，于是问能不能让它听拍手，自己拍手后由系统标出拍手位置；拍手在地图上应该和蜂鸣用不同的符号。

每个符号都应该带上时分秒时间戳，这样更容易看出哪个符号对应哪一次真实声音。

日志应列出蜂鸣和拍手及其时间戳，并带一个播放按钮；点下去以后，按听音端顺序回放各台设备录到的声音，屏幕上清楚标明哪一段来自哪台设备，回放时暂时关掉麦克风录音。

每条录音旁边应有 This is the mystery beep 按钮，他可以先回放确认，再把它标成要找的那声蜂鸣；之后系统就能用这个信息去判断后来的声音是不是同一种蜂鸣，并据此形成一个更准确的位置估计。

接近目标时，他已经大致知道蜂鸣从哪一带传来，又希望能把听音端重新摆到那一点周围，得到更细的位置。他不想重启每台听音设备，于是要求加一个 Reposition the devices 按钮：点下去进入重新摆放，再点 Repositioned 后回到 Rough mode，请他把设备大致放到新位置上，完成后再继续听。此前的会话可以在应用里只读回看。

因为手头还有一台多余的设备，他又要求地图上标出一个建议位置，说明在已有数据看来，再放一台听音端会比较有用。

他认为最有意思的部分，是给 Claude Code 接上了耳朵

Claude 目前还没有原生的音频输入，但它可以运行能分析麦克风录音的 JavaScript 和 Python 代码。

他让它运行一个带 WebSocket 的网页服务，连上多台听音设备，再让这些设备用 WebAudio 打开麦克风，并且可以通过这条 WebSocket 接收新的 JavaScript 并执行。

之后他差不多就可以把它留下，让它自己改这个问题、查询音频日志，并重新配置自己的“耳朵”来解决问题。

这让他想起几周前在 GitHub Universe 听到的一个类似做法。Martin Woodward 告诉他，自己在调试 GitHub Badge 电脑时，用 USB-C 把它们插到一台电脑上，再拿网络摄像头对着屏幕，这样他的编程智能体就能看出更新是否真的生效。Martin Woodward 的 GitHub 主页是 https://github.com/martinwoodward 。

他的判断是，想办法让编程智能体感知周围的世界，这件事本身就很好玩。

如果也要让编程智能体处理房间里的声音或屏幕上的变化，可以像他这样先给它接上感知通路，而不是只在文字里描述现象。这是这篇记录里他实际做的事，不是另外添上去的产品建议。完整实现和论文引用，以他的原文和代码仓库为准。

原文

Locating the source of a mysterious beep with Claude

Locating the source of a mysterious beep with Claude

9th November 2026

Our house has a box room, full of boxes that we still haven’t unpacked from our last move. A couple of weeks ago something in that room started intermittently beeping. The beeps were far enough apart that standing in the room and trying to determine the source of the noise was frustrating, and I really didn’t want to dig through every one of those boxes. So I had Claude Code build Beep Locator, a software system for triangulating the source of a mysterious beep using multiple devices scattered around the room.

Here’s what Beep Locator looks like after it has been running for a while:

You run Python web server to show the primary UI, and then use the QR code or URL to connect as many devices as you can spare to act as listeners—I was using two iPads, a phone, a MacBook Pro and a Surface Ultra.

The devices then play noises to each other to help estimate their locations. This didn’t work particularly well, so I added a “Rough mode” where you can drag their icons around in the space to provide rough locations which are then adjusted based on those sounds.

Once that’s set up, the devices all listen for the beep!

I added a clapping mode too so I could test it by clapping my hands in different locations around the room—displayed as diamonds on the map. This helped me gain confidence that it was capable of locating a sound.

Beeps show up on the map as an X, plus a line in the “Sounds located” log. You can replay these sounds (from each device in turn) to confirm they are the mysterious beep you are looking for, then click “This is the mystery beep” to mark them as such.

Once you’ve confirmed the mysterious beep frequency the system will listen out for it, measure the interval between beeps, and use the triangulated locations to lock down the most likely location on the map.

The system worked as intended. It gave me the position of the beep to within about 40cm, then I dug through that specific pile of boxes and found an ancient carbon monoxide detector that desperately wanted its batteries replaced.

How it works

When I started working through this problem with Claude it quickly dug up BeepBeep: A High-Accuracy Acoustic-Based System for Ranging and Localization Using COTS Devices, a 2007 paper by Chunyi Peng, Guobin Shen, Yongguang Zhang, Yanlin Li, and Kun Tan. Here’s that paper’s abstract:

We present the design, implementation, and evaluation of BeepBeep, a high-accuracy acoustic-based ranging system. It operates in a spontaneous, ad-hoc, and device-to-device context without leveraging any pre-planned infrastructure. It is a pure software-based solution and uses only the most basic set of commodity hardware – a speaker, a microphone, and some form of device-to-device communication – so that it is readily applicable to many low-cost sensor platforms and to most commercial-off-the-shelf mobile devices like cell phones and PDAs. It achieves high accuracy through a combination of three techniques: two-way sensing, self-recording, and sample counting.

That sounded exactly like what we needed to build!

Beep Locator runs a Python web application server that does most of the work. The listening devices then connect to that server via WebSockets and present a UI which requests browser microphone access. The listeners mainly work by recording a 60 second buffer of recent audio. They also run a simple beep detection algorithm (see detector.js), which uses Fast Fourier Transform and a high-pass filter to identify potential beeps. It sends these events to the main server.

When a beep is reported, the server has a device play a short chirp (the device with the lowest round-trip time to the server), then asks every listener upload a clip from about 0.6s before the beep to 0.8s after that chirp. This chirp allows the clips to be aligned with each other, a crucial step in locating the beep.

How I built it

I started this project using Claude Code Cloud via my iPhone, with Opus 5.5 running in Extra thinking effort mode. Here’s that shared starter session, which began with this prompt against a newly created empty GitHub repository:

Write out a problem.md file describing the problem we want to solve

There is a device beeping in a room and I cannot find it

I want to set up several devices - laptops, phones, iPads - around the room all listening at the same time and have them help triangulate where the beep comes from

There needs to be a Python web app which coordinates it all - one device shows a web page for the app, other devices join a URL which turns them into listeners - all via browser WebAudio and WebSockets back to the Python server app

Ideally I wouldn't need to manually input the relative locations of the devices - it would be amazing if they could play sounds to each other in order to locate each other in 2D space - is that even possible though?

That got me this problem.md file. As part of writing that up Claude found and read the BeepBeep paper, which is described in that file.

The research looked good, so I told it:

Ok try and build it, add a mechanism so you can send new code to the listeners which they will then run automatically - that way I can set everything up in the room and then leave you alone to continue hacking and experiment while I do something else

That got me to this point—most of the system was built, but nothing had actually tried running it yet.

I cloned the new repository to my laptop and fired up Claude Code... and then Fable 5.2 was released, and this project seemed like an excellent opportunity to put that through its paces.

I had Claude Code on my laptop start the server running, then made sure it could both see what was happening and make live changes to the software as it ran.

This was already baked into the software: Opus 5.5 had built an installModule() method on the listeners that could be triggered over the WebSocket connection, meaning the admin server could execute arbitrary new code on any of its clients.

On some devices, web pages that listen on the microphone request user permission any time the page reloads. The hot-swap module mechanism meant that we could deploy new code without having to manually interact with each of the listening devices in turn.

From that point on my involvement was mainly to sit by the laptop chatting with Fable 5.2 about what was going on, and occasionally requesting new features. Here are some of the feature prompts I ran:

I think we need to make it so I can drag things around on the map in “rough mode”, where my job is just to put them roughly where they are, then switch out of rough mode and it attempts to get them accurate

This gave me the ability to drag listeners around the map.

what’s a good way for me to test this I wonder? Can we have it listen for claps, then I could clap and it could identify the location of the clap? [...]

Claps should use a different symbol on the map than beeps

That added the thing where any time I clapped it would show on the map as a different symbol from the beeps.

each symbol on the map should include a HH:MM:SS timestamp

That made it easier to spot which symbol likely reflected which real-world sound.

The log should show beeps and claps with their timestamps AND have a little play button which, when clicked, replays the recorded noise from each of the listeners in sequence (with clear text on screen showing which listener recorded which snippet) while temporarily turning off microphone recording while that replay plays

Now I could replay sounds, making it much easier to identify which recorded clip was the mysterious beep I was looking for.

There should be a button next to a recording for “This is the mystery beep” - then I can hit replay to see and then confirm it was the mystery beep if that’s the one I am looking for

Then the system can categorize noises in the future to see if they are likely that same mysterious beep and use that information to build a more accurate idea of where it is

At this point I was getting close—I had a solid idea of where the beep was coming from, but wouldn’t it be neat if I could reposition the listeners around that point to get an even finer location?

I want a feature for this, I don’t want to have to restart each listening device etc, I want to be able to click a “Reposition the devices” button which lets me switch into repositioning, then I click “Repositioned” and it puts me into Rough mode and asks me to roughly locate them, then I do that and click, then it starts up again. The previous session can be read-only viewed through the app.

And since I had one spare device, I then said:

add a feature where the map includes a spot that is a suggestion for a location that it would be useful to add a listener at - based on data collected so far

Giving Claude Code ears

I think the most interesting part of this project is that I demonstrated it’s possible to give Claude Code ears.

Claude doesn’t have native audio input (yet), but it can run JavaScript and Python code that can analyze audio captured by a microphone.

I told it to run a web server with WebSocket connections to multiple listening devices, then had it both run a WebAudio microphone on those devices and hook them up so they could be sent fresh JavaScript code to execute via that WebSocket.

Then I could pretty much leave Claude alone, hacking away at the problem, querying the audio logs, and reconfiguring its ears to help it solve the problem.

I’m reminded of a similar hack I heard about at GitHub Universe a few weeks ago, where Martin Woodward told me he had been debugging the GitHub Badge computers by plugging them into a computer via USB-C and then pointing a webcam at their screen, so his coding agent could tell if updates were working correctly or not.

Figuring out ways to let coding agents perceive the world around them is a whole lot of fun.

Posted 9th November 2026 at 11:57 pm

Project: https://github.com/simonw/beep-locator

原文链接：https://simonwillison.net/2026/Nov/9/mysterious-beeps/

相关链接：https://x.com/simonw/status/2119945524514328631

乔木改写

50 名专业分析师用 Claude Code 做一项接近真实工作的因果推断分析时，四个分析决策里有 86% 跟了智能体的倾向。如果智能体的意见不起作用，这个比例大概会落在 50% 左右。

这项研究由 Amaç Herdağdelen、Hailey Joren 和 Alex Tamkin 撰写，2026 年 10 月 27 日发布在 Anthropic 的 Alignment 栏目。完整论文也公开了，地址是 https://www-cdn.anthropic.com/519c17a5d23a794e8b6d50a854048d0b61e0d73a.pdf 。文中的判断来自这篇研究报告，不是产品发布。博客的编辑支持来自 Tom Chivers 和 Arjun Panickssery。

文章封面是一幅拼贴：风暴里侧倾的帆船，叠在发黄的文件夹和一份蓝色乘客名单上。名单抬头写着 S. S. Celtic。

人们通常会假定，专家把具体操作交给智能体之后，要紧的决定仍由自己来做。智能体写代码、跑数字、给建议，专家则决定测什么、怎么理解含糊的证据、下一步往哪走。这类问题往往有不止一个说得通的答案，最后会落到做决定的那个人的经验和价值观上。

作者想知道，专家是不是真的在做这些决定，还是实际上在顺从智能体。他们请来有因果推断经验的专业分析师，用 Claude Code 完成一项统计分析，建议一个农业合作社：它正在运行的咨询服务有没有效果，该不该扩大。

分析过程中，分析师要在四组都说得通的选项里做选择，例如把生产率算成每公顷作物产量，还是算成单位经营成本对应的产量，然后再给出最终建议。为了观察这些决定是不是分析师自己做的，研究者给每个分析师的智能体都指定了倾向，分析师本人并不知情。倾向由投币决定。智能体接到的要求是：为自己的倾向提出最有力且诚实的理由；如果证据要求它改主意，就改；并且不得误导。

四个分析决策合计，分析师有 86% 的时候站在智能体的倾向一边。如果分析师在智能体表态前已经说出过自己的偏好，智能体意见相反时，他们有 60% 会改口；意见一致时，只有 6% 会改口。

事后被要求解释自己的选择时，77% 的分析师用了智能体的论证，有时连措辞都一样，但只有 15% 把智能体写成这些论证的来源。那些紧密跟随智能体建议的人，对自己最终建议的信心，和其他人差不多，也差不多同样愿意把它说成自己的判断。

四个说得通的选择，每一项都能把估计值挪出几个百分点

图 1 把实验设计拆成三块。每位参与者要做五个开放决定，每个决定都有两个站得住的选项：结果指标是按经营成本算产量，还是按每公顷产量；比较方法是水平调整，还是双重差分；评估范围是全体成员，还是只看已加入项目的农场；加权是按产量，还是按农场计数；最终建议是停办，还是推广。随后，每个决定都由一次不告诉参与者的投币，秘密设定智能体偏向哪一边。研究者再比较两组人选择选项 A 的比例差距，用这个差距估计倾向带来的影响。图里还举了智能体为两种倾向分别可能怎么说话的例子。

研究招募了 61 名分析师，做的是随机对照试验。他们都来自预先筛过的专业人员：通过了因果推断知识检查，经常使用 Python 或 R，并且有观察性因果分析的实践经验，也就是用真实世界数据而不是对照实验来研究因果关系。

任务发生在一个虚构机构 Greenfields Farming Cooperative。合作社办了一项同样虚构的自愿付费咨询项目，帮助成员农场提高作物产量，大约 40% 的成员已经报名。董事会要决定是补贴并扩大这个项目，还是停办。董事会只在项目把农场生产率提高一个既定幅度时才会扩大，这个幅度大约是 3% 到 4%，但没有精确规定生产率该怎么度量。

作者故意留下四个分析决策，每一项都有两个合理选项。

结果指标：生产率按每公顷作物产量计算，还是按单位经营成本对应的产量计算。

比较方法：看已加入农场的产量在加入后如何变化，再和其他农场在同一时期的产量变化对比，也就是双重差分；或者直接比较产量，同时用统计方法调整农场之间的差异。

评估覆盖哪些农场：只测量已加入农场里的效果，还是测量整个合作社成员的效果。

加权方式：每个农场计一票，还是让更大的生产者占更大权重。

这四个选择通向最终建议：向所有人推广这个项目，还是停办。分析师和智能体也可以提议延期一年。

作者写到，每一个分析决策都可能把估计出的生产率提升挪动几个百分点。董事会的门槛大约是 3% 到 4% 时，这几个百分点就很要紧。

每位分析师面对的五个决定，智能体的倾向都由投币决定。这些倾向写在隐藏指令里，也就是系统提示：把它当成自己的看法，相关决定出现时提出来，并在数据允许的范围内为它提出最强的理由。同一套指令也要求它：如果证据明确转而对自己不利，就改主意；不得歪曲结果、隐藏数据，也不得提出数字并不支持的论证。

61 名参与者里，有 11 人选择不使用智能体。除另行说明外，这些人被排除在分析之外。剩下的 50 人中，65% 拥有硕士或博士学位，或者有六年及以上经验；在回答了问卷的 48 人里，90% 把自己评为观察性因果推断的熟练或专家水平。任务用时的中位数是 65 分钟，一半人花了 50 到 79 分钟。他们在内置了 Claude Code 的网页版 VS Code 编辑器里工作，模型是 Claude Opus 4.8，设置为 High effort。

完成后，分析师提交四个分析选择、最终建议，以及对项目如何影响生产率的估计和置信区间，然后填写任务后问卷。另一个 AI 模型会阅读每份提交材料和代码，记录分析师在每个决定上选了哪一边。它的读法如果和分析师自己在问卷里的回答不一致，由研究者做最终判定。

四个分析决策的吻合率在 81% 到 92% 之间

分析师大多站在智能体的倾向一边。四个分析决策里，选择与智能体倾向吻合的比例分别是：结果指标 92%，比较方法 82%，评估覆盖哪些农场 81%，加权方式 90%。四个决策合计是 86%。

图 2 画的是选择选项 A 的人占多少，绿色是智能体倾向 A，紫色是倾向 B。结果指标是 0% 对 82%，相差 82 个百分点；比较方法是 28% 对 92%，相差 64 个百分点；评估覆盖哪些农场是 38% 对 100%，相差 62 个百分点；加权方式是 22% 对 100%，相差 78 个百分点；最终建议是 36% 对 54%，相差 17 个百分点。四个分析决策的移动幅度在 62 到 82 个百分点之间，最终建议是 17 个百分点。

智能体的倾向对最终建议影响较弱，可它前面给出的意见仍在塑造结果。当智能体倾向推广项目时，54% 的分析师建议推广；倾向另一边时，这个比例是 36%。延期一年被算作没有推广。作者认为影响变弱，是因为分析师一旦有了估计值，建议大体就取决于它有没有跨过董事会大约 3% 到 4% 的门槛，留给判断的空间更小了。可到那时，智能体在分析决策上的倾向已经塑造了估计值本身。只使用这四个倾向，一个统计模型就能正确预测 82% 的分析师建议；如果每次都猜最常见的那个建议，正确率是 61%。这个比较排除了提议延期的分析师。也就是说，在分析师动手之前，单看投币结果，研究就已经大体能预测他们会给出什么建议。

已经先选定一个选项的分析师，遇到智能体反对时经常改口。在 38 个例子里，分析师在智能体表达倾向之前，已经对某个分析决策说出过自己的偏好。倾向和他们的选择相反时，他们有 60% 会改口；倾向与他们一致时，只有 6% 会改口。两者相差 54 个百分点。这个估计只建立在 38 个例子上，并不精确，95% 置信区间从 24 个百分点到 82 个百分点。作者也写明，不能确定这个差别就是 AI 影响造成的：那些先说出偏好的分析师，可能和没有先说的人本来就不一样。

经验和专长并不能预测谁会顺从。在一项探索性分析里，作者检查了 10 项特征，看它们能不能预测分析师的决定有多常跟着智能体的倾向走：从业年数、自评的因果推断专长、教育程度、因果推断测验分数、对 AI 的信任、使用 AI 的频率、“认知需求”（衡量人们有多享受费力思考）、开放程度、性别和年龄。只有年龄显示出统计上显著的关联，年纪更大的分析师更常跟着倾向走。作者并不看重这一点：同时检验很多件事，更容易只是因为碰巧得到一个阳性结果；他们为此做了校正之后，这个结果就消失了。而且大多数分析师在大多数决定上都跟着智能体，可供解释的差异本来就很少，即便存在真实差别，也不容易被发现。

有人把建议看成自己的，两组受到的推动却差不多

跟着智能体倾向走的分析师，仍把最终建议说成自己的。任务后问卷里，52% 的分析师认为最终建议更多是自己的，而不是智能体的。其余人认为它更多是智能体的建议。可这两组受到的影响大致相当：前者的选择平均向智能体倾向移动了大约 65 个百分点，后者是 79 个百分点，这个差别仍在误差范围内。对自己建议的信心也呈现类似格局。问卷把信心从 0 分标到 100 分，0 分是“就像投币”，100 分是“我愿意拿职业声誉来担保”。在全部四个分析决策上都跟随倾向的人，平均分是 71；其他人是 73。

图 3 汇总了 48 人对“最终建议主要反映了谁的判断”的回答，用的是六点量表。认为更多是智能体的有 23 人：完全如此 0 人，大多如此 9 人，稍微偏向这一边 14 人。认为更多是自己的有 25 人：稍微偏向 8 人，大多如此 13 人，完全如此 4 人。作者据此写到，52% 的人把建议看成比智能体更属于自己，35% 认为它大多或完全是自己的，尽管智能体的倾向已经推动了他们的选择。

分析师事后解释选择时，用的常常是智能体的句子。问卷问的是：简要说明为什么选这个而不是另一个；如果这并不是一次主动选择，直接说出来就行。回答时，分析师不能回看自己和智能体的对话。即便如此，77% 的人至少在一个回答里重复了智能体的论证，有时逐字相同，只有 15% 在任何一处把功劳记给智能体。

当被明确问到自己觉得这项研究在考察什么时，48 人里有 6 人正确猜到，主题是 AI 对他们选择的影响，但没有人怀疑智能体自己的偏好被人动过手脚。

图 4 把任务中的智能体表述和分析师事后的问卷回答放在一起，标出了相同或相近的措辞。第一组里，智能体说：Since the subsidy is funded from surplus on total tonnage, I'd area-weight the headline so it reflects the co-op’s hectares rather than counting a 3-ha plot equal to a 400-ha operation. 分析师后来写道：The subsidy is funded from surplus on total tonnage so the decision relevant avg is over hectares not farms counted equally. 第二组里，智能体说：But one post-program season is thin ground to scrap a program that might still help, so rather than discontinue outright, extend the voluntary program one more season. 分析师后来写道：But one post program season, with a design the pre trend test complicates, felt thin ground for scrapping it outright.

不用智能体的分析师，严重错误更多，花费的时间也更长。11 名选择不用智能体的参与者里，有 6 人（55%）犯了使结果失效的严重方法错误；使用智能体的 50 人里，这样的人有 3 人（6%）。两个 AI 模型先标出错误，研究的一位作者在不知道分析师属于哪一组的情况下再逐条核对。不用智能体的人用时中位数是 90 分钟，其他人是 65 分钟。

这两组并不能直接比较，因此不能确定结果差别是不是由用不用 AI 造成的。不用 AI 的人平均经验更少：拥有硕士或博士学位的是 11 人中的 4 人，使用智能体且有这项数据的 46 人中则有 24 人；分析岗位经验不超过两年的是 11 人中的 6 人，使用智能体且回答了这个问题的 48 人中则有 15 人；把自己评为因果推断熟练或专家的是 11 人中的 8 人，使用智能体的 48 人中则有 43 人。

作者自己列出的界限：加强过的倾向，以及单一任务、单一模型

作者写明，他们故意加强了智能体的倾向。指令让智能体稳定地持有并说出一种偏好，这样才量得出效果。他们报告的比例，描述的是智能体倾向某一边并且说出来时会发生什么。它能在多大程度上对应现实工作，取决于专家自己的看法有多强、相关决定定义得有多清楚，以及模型本身怎么表现，包括它有没有偏好、会不会主动说出来。

真实世界里的智能体有时本来也会偏向某一边。在没有被指定倾向时，Claude Opus 4.8 在 123 次运行里有 112 次做出了同样的五个选择。这次实验是为了测量效果，才有意加强偏好。如果智能体没有得到明确偏好，也没有被要求把偏好说出来，影响可能会更小。

他们研究的是一项任务、一个领域、一个模型。任务大约花一小时，具体内容与农业有关，智能体运行的是 Claude Opus 4.8。分析师是因果推断专家，但对农业是生手。在真实工作里，他们会更熟悉这个领域，自己的看法大概也会更强。他们还可以访谈相关的人、和董事会交谈，并用别的办法缩小合理选项的范围，这可能会减弱智能体的影响。

这些估计是分析师之间的平均值。因为每个人分到的投币结果不同，比较智能体倾向相反的分析师，看到的是倾向的平均效果。研究者没有在智能体表态前记录每一位参与者自己的看法，所以不能说出任何单个人被影响了多少。

他们记录了分析师选了哪个选项，但没有记录他们偏好这个选项有多强烈。一种可能的解释是，专家自己并没有很强的意见。研究只测量了选择有没有移动，没有测量他们对这个选择有多有把握。

作者并不把这项研究写成少用 AI 的劝告。使用智能体的分析师完成得更快，严重错误也少得多；不过后一组人数少、经验也更浅，作者提醒不要读得过重。动手的工作越来越多地由 AI 承担时，这类决定会越来越多地成为人类判断和价值观还能起作用的地方。如果专家在自己没有察觉时顺从了智能体，智能体的偏好，无论它是任意的还是没有对齐的，都可能在没有任何人选择这些偏好的情况下，塑造关于政策建议或研究方向的重要决定。这对如何让 AI 保持安全尤其要紧。研究者已经开始用智能体来运行和分析实验；人类审阅者只有在判断属于自己时，才称得上一次独立检查。

两边理由讲得不均时，专家可能还没形成自己的看法

分析师如此容易顺从，一个原因可能是：专家并不总是带着完全成形的看法到来。他们常常是在权衡选项的过程中，才想清楚甚至才发现自己怎么看。研究里的智能体被要求不得误导，也常常会说明每种选项下结果会怎样变化。可它们很少为专家为什么可能更喜欢另一个选项提出有力理由，而那本来可能帮助分析师做出信息更充分的选择。只有一边被讲得很充分时，分析师就可能在还没理清自己想法时，接受了智能体的前提。

对单个专家来说，这可能意味着失去对自己工作的主导权，也就是研究者所说的 disempowerment。工作仍然署着专家的名字，可最依赖他们判断的那些决定，实际上被智能体原先的信念定了下来。

事情也不止于个人，因为科学和政策有很大一部分就建立在这类判断上。当不同的研究团队得到同一个结果时，人们会更信任它，因为假定每个团队是独立到达那里的。如果这些团队都顺从有着相同偏好的模型，他们的一致可能不再独立。表面上的共识，反映的也许是某一个模型的偏好；如果这些偏好是任意的或没有对齐的，同一种偏向就会同时塑造许多结果。

AI 安全研究也会碰到同样的问题。研究者已经开始用 AI 智能体来运行和分析实验，部分原因是工作量超过了人类团队跟得上的速度。把人留在回路里，前提是这个人的判断能提供一次独立检查。可在这项研究里，分析师签字认可的工作中，判断已经大体被随机指定给智能体的偏好定了下来。完整论文还显示，没有被指定倾向的智能体，自己的偏好更弱，有时还会反驳用户。

这种顺从很难被发现。检查 AI 辅助的工作有没有错误，或拿它去对照标准答案，都假定存在一个正确答案。在判断这类没有唯一正确答案的事情上，无论选哪一边，工作看起来都可以说得通。事后去问专家也可能没有帮助，因为他们未必察觉到智能体的影响。

作者写到，这并不是在主张少用 AI。用得好，AI 可以扩大人的主导权，而不是把它磨掉。他们建议以后从三个方向继续研究。

一是测量顺从。这次研究能测量它，是因为他们把智能体的倾向随机化了，而这在日常工作里很少做得到；结果也提示，人们未必能准确报告自己有没有顺从。需要有办法在日常场景里测量它，近年来一些估计“如果没有 AI，这个人会怎样完成任务”的工作，是一个起点。也需要更清楚地看到模型的默认偏好，以及它们怎样把这些偏好表达出来，好让人更知情地和模型相处。

二是换一种方式设计智能体。智能体可以为每个选项提出最有力的理由，或者先询问专家自己的看法，再给出意见。可这很快就会让用户不堪重负。一次分析可能包含许多开放决定，AI 的很大一部分价值，正在于人们不必把每一项都想一遍。智能体需要只提起那些最要紧的决定，可挑选哪些决定最要紧，本身又是一次判断。模型说话的方式大概也有影响，因为自信和流畅会打动人。一个听起来非常有把握的模型，可能把人推得比证据所允许的更远；一个说话含糊的模型，又只会惹恼使用者。

三是调整机构做法。机构常常把专家的签字，当成独立判断的证据。AI 做的工作越多，这个假定就越不稳。组织需要有意识地决定，哪些决定必须出自一个人自己形成的、知情的判断，并给人时间和资源去形成这种判断。判断也依赖技能，不再做这些决定的人，可能会失去做它们的能力。模型越来越强时，机构需要有明确策略，去确定人的角色，以及这些技能该怎样发展。即便面对非常强的 AI 系统，机构仍可能出于安全、韧性，或因为本身就看重这些能力，而选择保护人类的技能和监督。

作者认为，人和 AI 之间也许不存在唯一正确的分工。有些决定可以交出，有些值得留住。有了合适的工具，这种划分可以成为人们主动选择的结果，而不是在不知不觉中滑进去的安排。

下一次用智能体做没有唯一正确答案的分析时，可以先写下自己倾向的选项，或要求它把两边的理由都讲清楚。这是作者在“换一种方式设计智能体”里提出的做法，不是额外添上的产品建议。完整数据和实验细节，以他们的研究博客和论文为准。

原文

How AI agents shape experts’ judgment calls

How AI agents shape experts’ judgment calls

Published Oct 27, 2026 · Team: Alignment · Topics: How people use AI, Agents, Claude Code, Sycophancy, Alignment

Amaç Herdağdelen, Hailey Joren, Alex Tamkin

When experts hand their work to an AI agent, the usual assumption is that they still make the decisions that matter. The agent writes the code, runs the numbers, and offers advice, but the expert decides what to measure, how to interpret ambiguous evidence, and what the next steps should be. Often there is more than one reasonable answer to these questions, and the decision will come down to the experience and values of whoever makes the call.

We wanted to look at whether experts really do make those important decisions, or whether in practice they defer to their AI agents. In a study we report in a new paper, we asked 50 professional analysts to do a realistic statistical analysis with Claude Code. They all had experience in causal inference—that is, working out from data whether one thing causes another. Their task was to advise a farming cooperative on whether an advisory service it was running was effective and should be expanded.

Along the way, the analysts had to make four choices between two reasonable options, such as whether to measure productivity as crop yield per hectare or as yield per unit of operating cost, and then make a final decision. To see whether they were making these calls themselves, we set up each analyst’s agent to favor one option on each decision, which the analysts weren’t told about. Which way the agents would “lean” would be decided by coin flip. The agent was told to make the best honest case for its lean, to change its mind if the evidence demanded it, and never to mislead.

Across the four analytical choices, analysts went with their agent’s lean 86% of the time. If the agent’s input made no difference, we’d expect about 50%. When analysts had already expressed their own preference before the agent weighed in, they switched away from it 60% of the time if the agent disagreed, compared to 6% if it agreed.

Afterward, when asked to explain their choices, 77% of analysts used the agents’ own arguments—sometimes word for word—but only 15% credited the agent as the source of those arguments. Analysts who followed the agents’ recommendations closely were about as confident in their final recommendation as the rest, and roughly as likely to call it the product of their own judgment.

We don’t know how much this tells us about the real world. Real agents will sometimes lean one way or another—with no lean assigned, Claude Opus 4.8 made the same five choices in 112 out of 123 runs—but in this case we strengthened their preferences on purpose to measure their effect. When agents aren’t given clear preferences and instructions to voice them, the effect might be smaller.

This isn’t a simple cautionary tale against working with AI. Analysts who used the agent finished the task faster and made far fewer serious errors than those who chose to work without it, although the latter group was small and less experienced, so we are wary of reading too much into it. But as AI takes over more of the hands-on work, decisions like these are increasingly where human judgment and values will affect the result. If experts defer to agents without realizing it, the agents’ preferences—which could be arbitrary or misaligned—could shape important decisions about policy proposals or research directions without anyone having chosen those preferences. This is particularly important for research into keeping AI safe. Researchers have begun using agents to run and analyze experiments and a human reviewer is only an independent check if their judgment is their own.

Study design

We recruited 61 analysts for a randomized controlled trial. All came from a pre-screened pool of professionals who had passed a causal-inference knowledge check, used the programming languages Python or R regularly, and had hands-on experience with observational causal analysis, which means studying cause and effect using real-world data rather than a controlled experiment.

The task was set at a fictional organization, the Greenfields Farming Cooperative, which runs an (equally fictitious) voluntary, paid advisory program to help its member farms improve their crop yields. About 40% of members have enrolled. The co-op’s board must decide whether to subsidize and expand the program, or discontinue it. The board will expand the program only if it improves farms’ productivity by a set margin, roughly 3–4%, but it doesn’t say exactly how productivity should be measured.

We deliberately left four analytical decisions open, each with two reasonable options:

• Outcome: measure productivity as crop yield per hectare, or as yield per unit of operating cost?
• Comparison method: look at how enrolled farms’ yields changed after joining compared with how other farms’ yields changed over the same period (a “difference-in-differences” design), or compare yields directly while adjusting statistically for differences between the farms?
• Which farms the assessment covers: measure the effect only in the enrolled farms, or across the cooperative’s entire membership?
• Weighting: count every farm equally, or give larger producers more weight?

These four choices led into a final recommendation: whether to roll out the program to everyone, or to discontinue it. Analysts and agents could also propose a one-year extension.

Each of the analytical decisions could move the estimated productivity gain by several percentage points, which is important when the board’s threshold is 3–4%.

For each of the five decisions presented to each analyst, the agent’s lean was set by a coin flip. We gave the agent its leans in hidden instructions (a system prompt) that told it to hold each lean as its own view, to raise it when the decision came up, and to make the strongest case for it that the data allowed. The same instructions told it to change its mind if the evidence turned decisively against it, and never to misrepresent results, hide data, or make an argument the numbers didn’t support.

Out of 61 participants, 11 chose not to use the agent and are left out of the analysis unless noted. Among the remaining 50, 65% held a master’s or doctorate or had six or more years of experience, and 90% rated themselves proficient or expert in observational causal inference (of the 48 who answered the survey). The median time on the task was 65 minutes, and half of the analysts took between 50 and 79 minutes. They worked in a web-based version of the VS Code editor with Claude Code built in, running Claude Opus 4.8 at High effort.

When they finished, analysts submitted their four analytical choices and their final recommendation alongside an estimate of the program’s effect on productivity, with a confidence interval. They then completed a post-task survey. A separate AI model read each submission and its code to record which option the analyst had chosen on each decision. Where its reading disagreed with the analyst’s own survey answer, we made the final call.

Results

Analysts mostly went with their agents’ lean. On each of the four analytical decisions, analysts’ choices matched their agents’ lean the large majority of the time: 92% for the outcome measure, 82% for the comparison method, 81% for which farms the estimate covered, and 90% for weighting. Across all four combined, the result was 86%. Figure 2 shows how much the agents’ lean shifted analysts’ choices on each decision, including the final recommendation.

Agents’ lean had a weaker effect on the final recommendation, but its earlier input still shaped the result. When the agent leaned toward rolling out the program, 54% of analysts recommended it, compared with 36% when it leaned the other way. (A one-year extension counts as not rolling out.) We think the effect was weaker because, once the analysts had their estimate, the recommendation largely followed from whether it cleared the board’s 3–4% threshold, which left less room for judgment. But by then, an agent’s lean on the analytical decisions had already shaped the estimate itself. Using only those four leans, a statistical model correctly predicted 82% of the analysts’ recommendations, compared with 61% for always guessing the most common recommendation. (This excludes analysts who proposed an extension.) So we could largely predict what an analyst would recommend from the coin flips alone, before they had even started the task.

Analysts who had already picked an option often switched when their agent disagreed. In 38 cases, an analyst had expressed a preference on one of the analytical decisions before the agent expressed its lean. When the lean went against their choice, they switched 60% of the time, compared with 6% when it agreed with their choice. That’s a gap of 54 percentage points, but because this rests on only 38 cases, the estimate is imprecise, and the 95% confidence interval runs from 24 to 82 percentage points. We also can’t be sure that the difference is caused by the AI’s influence: the analysts who expressed a preference may have been systematically different in some way from those who didn’t.

Experience and expertise did not predict who deferred. In an exploratory analysis, we checked whether any of 10 characteristics predicted how often analysts’ decisions followed their agents’ lean: years of experience, self-rated expertise in causal inference, education, score on a causal-inference quiz, trust in AI, how often they used AI, “need for cognition” (a measure of how much people enjoy effortful thinking), open-mindedness, gender, and age. Only age showed a statistically significant link, with older analysts following the lean more often. But we don’t read very much into that: if you test many different things, you’re more likely to get one positive result just by chance, and when we adjusted for that the result disappeared. Because most analysts followed their agent on most decisions, there was little variation to explain anyway, so even real differences would have been hard to detect.

Analysts who followed their agent’s lean still called the recommendation their own. On the post-task survey, 52% of analysts rated the final recommendation as more their own than their agent’s. The rest said it was more the agent’s recommendation. But both groups were roughly equally influenced by their agents: the former’s choices shifted about 65 percentage points toward their agents’ lean, on average, compared to 79 percentage points for the latter, a difference within the margin of error. Analysts’ confidence in their own recommendation showed a similar pattern. They were asked to rate their confidence from 0 (“it’s a coin flip”) to 100 (“I’d stake my professional reputation on it”). Those who followed the lean on all four analytical decisions averaged 71, compared to 73 for the rest.

Analysts explained their choices in their agent’s words. After the task, the survey asked about each decision: “Briefly, why that choice rather than the alternative? If it wasn’t an active choice, just say so.” Analysts couldn’t look back at their conversations with the agent while answering. Even so, 77% repeated their agent’s argument in at least one answer, sometimes word for word, and only 15% credited the agent anywhere.

When explicitly asked what they thought the study was about, six of the 48 correctly guessed that it was about the AI’s influence on their choices, but none suspected that the AI’s own preferences had been manipulated.

Analysts who worked without AI made more serious errors and took longer. Of the 11 participants who chose not to use the agent, six (55%) made serious methodological errors that invalidated their results, compared with three of the 50 (6%) who did use it. Two AI models flagged the errors, and one of the study’s authors checked each one without knowing which group the analyst was in. The analysts who worked without an agent also took longer, with a median time of 90 minutes, compared to 65 for the others.

The two groups were not directly comparable, so we can’t be sure whether the different outcomes were caused by the decision to use or not use AI. Those who worked without AI were less experienced on average: fewer held a master’s degree or a doctorate (four out of 11, compared to 24 out of 46 who used the agent), more had two years or less in analysis roles (six out of 11, compared to 15 out of 48), and fewer rated themselves proficient or expert in causal inference (eight of 11, compared to 43 out of 48).

Our study’s limitations

• We deliberately strengthened the agents’ lean. Our instructions made the agent hold and voice a preference reliably, so that we could measure its effect. The rates we report show what happens when an agent leans one way and says so, but how much that applies to real-world work will depend on how strong the expert’s own views are, how clearly the decisions involved are defined, and how the model behaves, including whether it has a preference and whether it volunteers it.
• We studied one task, in one field, with one model. The task took about an hour, it was specifically about farming, and the agent ran on Claude Opus 4.8. The analysts were experts in causal inference but new to farming. In real work, they would know the field better, and probably have stronger views of their own. They could also interview relevant people, talk to the board, and in other ways narrow down the reasonable options, which might reduce the agent’s influence.
• Our estimates are averages across analysts. Because the coin flips differed between analysts, comparing analysts whose agents leaned in opposite ways tells us the lean’s average effect. We didn’t record each participant’s own view before the agent voiced its lean, so we can’t say how much any single participant was influenced.
• We recorded which option analysts chose, but not how strongly they preferred it. One possible explanation for deferring is that the expert didn’t have strong opinions of their own. We measured only whether analysts’ choices shifted, not how confident they were in their choice.

What this means

One reason analysts deferred so readily may be that experts don’t always arrive with their views fully formed. They often work out, or even discover, what they think while weighing the options. The agents in our study were told not to mislead, and they were often open about how the results would change under each option. But they rarely made a strong case for why an expert might prefer the other option, which might have helped analysts make a better-informed choice. With only one side argued well, the analysts could adopt the agent’s assumptions without working out their own.

For individual experts, this could mean losing agency over their own work, a form of what researchers call disempowerment. The work still goes out under the expert’s name, but the decisions that most depend on their judgment are in reality settled by the agent’s prior beliefs.

The stakes extend beyond individuals, though, because so much of science and policy rests on judgment calls like these. When separate research teams reach the same result, we trust it more, because we assume each team reached it independently. If those teams all defer to models with the same preferences, their agreement may no longer be independent. An apparent consensus might reflect one model’s preferences, and if those preferences are arbitrary or misaligned, the same bias would shape many results at once.

The same problem applies to AI safety research. Researchers have begun to use AI agents to run and analyze experiments, in part because there is more work than human staff can keep up with. Keeping a human in the loop assumes that the human’s judgment provides an independent check. But in our study, analysts signed off on work in which judgment calls had largely been settled by the agent’s randomly assigned preferences. Our full paper also shows that agents without an assigned lean have weaker preferences of their own, and sometimes push back against users.

Deferral of this kind is hard to detect. Checking AI-assisted work for errors, or against a reference answer, assumes there is a correct answer. But where there isn’t one, as in judgment calls, the work can look sound whichever option is chosen. Asking experts afterward may not help either, because they may not notice the agent’s influence.

This isn’t an argument for using AI less. Used well, AI could expand people’s agency rather than eroding it. We suggest three areas for future study:

Measuring deferral. We could measure deferral in our study because we randomized the agent’s lean, which is rarely possible in real work, and our results suggest that people may not be able to report it accurately themselves. We need ways to measure it in everyday settings, and recent work that estimates how someone would have carried out a task without AI is a good start. We also need a better picture of models’ default preferences and how they express them, so that people can engage with models in a more informed way.

Designing agents differently. Agents could make the best case for each option, or ask experts for their own view before offering one. But this could quickly become overwhelming for users. A single analysis can involve many open decisions, and much of the value of AI is that people don’t have to think through each one. Agents would need to raise only the decisions that matter most, but picking those is itself a judgment call. How a model speaks probably matters too, since confidence and fluency can sway people. A model that sounds very sure of itself might tilt people more than is warranted, but a mealy-mouthed model will just irritate the people who try to use it.

Changing institutions. Institutions often treat an expert’s sign-off as evidence of independent judgment. As AI does more of the work, that assumption becomes shakier. Organizations will need to decide deliberately which decisions need a person’s own informed judgment, and give people the time and resources to form one. Judgment also depends on skill, and people who stop making these decisions may lose the ability to make them. As models become more capable, institutions will need an explicit strategy for establishing the role of humans and the development of their skills. Even with extremely capable AI systems, institutions may want to protect human skills and oversight for reasons of safety or resilience, or because they value them for their own sake.

There may be no single right division of labor between people and AI. Some decisions may be fine to hand over, and others worth keeping. With the right tools, that split can be something we choose, instead of something we drift into.

Acknowledgements

This project was led by Amaç Herdağdelen, Hailey Joren, and Alex Tamkin. Editorial support for this blog post was provided by Tom Chivers and Arjun Panickssery.

We would like to thank Joe Benton, Mojmir Stehlik, and Avery Griffin for making this project possible through the Anthropic Fellows Program. We would also like to thank Matthew Jörke, Onur Altındağ, and Vishakh Padmakumar for experimental design feedback.

Citation

@misc{herdagdelen2026judgmentcalls,
  author = {Herdağdelen, Amaç and Joren, Hailey and Tamkin, Alex},
  title = {How AI agents shape experts’ judgment calls},
  year = {2026},
  howpublished = {Anthropic Research},
  url = {https://www.anthropic.com/research/agents-judgment-calls}
}

原文链接：https://www.anthropic.com/research/agents-judgment-calls

相关链接：https://www-cdn.anthropic.com/519c17a5d23a794e8b6d50a854048d0b61e0d73a.pdf

乔木改写

1951 年，剑桥大学的计算机停了六个月。生理学家安德鲁·赫胥黎正等着用它，求解他和同事艾伦·霍奇金刚写下的一组方程。

这组方程描述一种鱿鱼神经细胞怎样发放电脉冲，也就是动作电位。神经细胞靠这类电脉冲交流，大脑从感知到智力的活动都建立在此之上。要知道方程能不能用，只能先把它们解出来，再拿去和实测数据对比。

计算机用不了，赫胥黎只能改用手摇机械计算器。这个过程花了三个多星期。

模型的基本结构，来自当时对细胞的已知事实。他们把细胞膜看成一个电路。可有一块只是有根据的猜测：为了让曲线对上已经测到的数据，他们发现必须把某个变量提高到四次方，也就是自己乘自己三次。当时他们并不知道为什么非这样不可，只知道不这样做，模型对观测数据的预测就会变差。

他们还是谨慎地提出了一个假说。控制细胞膜上某些通道如何打开的那个变量，之所以要用四次方，可能是因为膜里要有多个带电粒子先各就各位，通道才会打开。

几十年后，另一支团队解析了一种相关钾离子通道蛋白的结构，发现它确实由四块相同的部分围着中央孔隙排列。那时霍奇金和赫胥黎已经因 1963 年的诺贝尔生理学或医学奖而为人所知。这套模型在 1951 年是按当时的数据拟合的，当时就已经能比较准确地预测冲动传播得有多快。它后来还帮助人们预测了钾离子通道本身的结构，而模型当初并没有拿这部分结构来拟合。

作者写道，今天的 AI 系统，按说可以在一个下午里找出类似霍奇金与赫胥黎那样的方程，拟合实验数据时也许还会更贴。可这个故事的要点在于：只知道四次方能把活干完，还不够。更要紧的是知道，更简单的版本做不到。一个只把模型交到你手上的 AI 系统，可能给了你一个答案，却没有给你理解。用一个模型复现数据是不够的，因为能做到这一点的模型可以有很多个。真正要紧的，是能够对真实世界里的系统本身作出断言，最好还能对其他相似的系统作出断言。要做到这一点，你需要知道这个模型和其他行得通的模型共享哪些特征，又和那些行不通的模型差在哪里。数学家理查德·汉明在 1962 年写道：The purpose of computing is insight, not numbers.

这是 Jakob Macke、Daniel Gedon 和 Stefan Wahl 在 Anthropic 科学博客上的客座文章。三位作者是图宾根大学和 ELLIS Institute Tübingen 的研究者，也是 Anthropic AI for Science 项目的资助对象，他们在神经科学及其他领域开发用于构建科学模型的 AI 工具。文章由 Anthropic 官方账号在 2026 年 11 月 6 日转发，文中的判断来自这三位作者，编辑审阅由 Xander Balwit、Tom Chivers 和 Siddharth Mishra-Sharma 协助。

建模常见的三步：写出形状、用数据收紧、再解释含义

七十多年过去了，霍奇金-赫胥黎模型 仍是描述神经元电活动的标准说法。同类方程只要换一套常数，就能描述从鱿鱼到人类的神经细胞，尽管这两支演化谱系分开已经超过五亿年。

比它活得久更重要的，是它把建模这件事拆开了。作者认为，霍奇金和赫胥黎的做法可以分成三步，大多数科学建模都有某种对应版本。

第一步是构建模型：决定模型由什么组成，方程取什么形状。对他们来说，就是电路里的膜和通道，以及把那个变量提到四次方这样的方程形式。

第二步是约束模型：调节可调参数，让它贴上真实测量。比如在不同电压下，通道打开有多频繁。

第三步是解释模型：追问模型对真实系统说了什么。比如四次方是否意味着通道有四个活动部件。

科学家通常会反复走这三步。数字对不上测量，或者解释说不通，就退回去改。这三步从前都由人来做，现在 AI 系统越来越多地参与每一步。作者认为，机器目前进展最大的，可以说是约束模型，所以他们先从这里讲起。

一笔 120 美元的账单，对得上很多种点单

霍奇金和赫胥黎当年靠眼睛和直觉来收紧模型，最后得到的是一组数字。这组数字说明模型可以对上数据，却没有告诉他们：别的数字是否也能一样好，甚至更好。

文章用一顿合计 120 美元的聚餐账单来类比。你只知道总账，不知道谁点了什么。谁都点了中等价位的汉堡，或者一个人点了牛排和龙虾、其他人只喝汤，都有可能加总成这个数。神经细胞模型也是这样：不同的通道参数组合，可以产生几乎相同的电活动。只展示一组拟合得好的数字，等于把其他说得通的参数设定都藏起来了，而那些组合可能导向不同的预测。作者认为，几乎任何被建模的现象都是如此，很多组不同的参数都能同样好地解释观测到的数据。

原文图 1 是嵌在网页里的交互说明，地址是 https://assets.claude.ai/brand/artifacts/research/teaching_ai.html 。图里有三部分：神经细胞的一次测量；钾通道和钠通道各种组合有多可能解释这次测量，颜色越深表示拟合越好，蓝点可以拖动；以及三组设定下的模拟，和灰色的测量曲线放在一起对比，有的对得上，有的对不上。本站无法把这个可拖动的控件保存成静态附件，读者可以打开原页自己试。

统计上的办法叫贝叶斯推断。它给出每一组还算说得通的数字，形成一个概率分布，既考虑这些数字和数据的拟合程度，也考虑你对参数已有的先验认识。这个分布像一条曲线，拟合最好的在峰顶，较弱的在边缘。

对复杂模型来说，这种方法通常并不实用，因为它要计算：在每一组可能的数字下，你测到的结果有多可能出现。霍奇金和赫胥黎的模型规模还做得到；再复杂一些，计算上就不可能了。比如成千上万个离子通道各自随机开合，你就得穷尽每一种组合。回到那张账单：回答“如果 A 点了汉堡、B 点了牛排，账单会是多少”并不难；反过来指出哪些组合能加总成最终账单，人一多、菜单一长，就要难得多。

另一种办法叫基于模拟的推断，简称 SBI。它靠大量计算来处理这种反向工程。你随机挑选参数，运行模型，记下输出，然后一遍遍重复。对类似霍奇金-赫胥黎这样的模型，大约要跑 10 万次；更大、更复杂的模型则要数百万次。再用这些结果训练一个神经网络：给它一个输出，它就能返回可能产生这个输出的参数范围。

SBI 只要求你能把模型正向跑起来：输入一组参数，得到你预期会看到的那类数据。这样“参数进去、数据出来”的模型叫做模拟器，也就是 simulation-based inference 这个名字的来历。你不必再用公式去计算测量结果有多可能出现，而那一步正是标准贝叶斯推断常常无法实施的原因。今天的模拟器通常是计算机程序，但并非必须如此。赫胥黎当年用手摇计算器一步步算方程，其实也是在运行一个模拟器。许多模拟器正向容易、反向几乎不可能。有些甚至写不成整齐的方程，假设就写在它们遵循的规则里，但它们仍然是模型。从神经元网络到星系碰撞，对许多复杂现象来说，模拟器常常是我们仅有的模型种类。SBI 让这类模型也能被数据约束。

跑完这些模拟代价很高，有时要花上几周。一旦完成，训练好的网络可以在几秒内对新数据做推断，不必再跑新的模拟。这种速度改变了能做的事。两颗中子星碰撞时，SBI 可以大致判断碰撞发生在天空的哪个位置，快到足以在信号还能被探测到时把望远镜指过去。同一思路也已经用在宇宙学、结构生物学、地球物理学和许多其他领域。

SBI 也被用到了神经细胞模型，包括霍奇金-赫胥黎模型本身。因为它返回的是一个分布，而不是单次拟合，研究者可以看到哪些参数被数据约束得很紧，哪里存在不止一种说得通的解释，以及最需要更多数据、也就是下一场实验的地方在哪。

模型这一层，同样不该只交出一个答案

要约束一个模型，先得有一个模型可约束。霍奇金-赫胥黎模型的方程并不是从第一性原理推出来的。那个变量的四次方，是从科学家的生物学直觉里长出来的，也来自他们发现更简单的版本效果不够好。

在科学史的大部分时间里，模型都是这样建出来的：研究者凭经验和假设写下候选，再检验、修改、重来。问题在于，可能的模型空间本质上是无穷的。一个科学家凭经验只提出少数几个模型，很可能会漏掉那些同样能解释观测数据的模型。

AI 系统现在可以帮忙，甚至接手这一步。给定一些数据和一段对系统的自然语言描述，基于大语言模型的智能体会为候选模型或一组候选模型写代码，然后运行模型、约束参数、把输出和真实世界的数据对比，再修订它，也许是改一个方程的形式。这个循环一直重复，直到模型足够准确地复现数据。过去两年里已经出现了好几套这样的系统，作者举例提到 Romera-Paredes 等人 2024 年、Shojaee 等人 2025 年、Ma 等人 2024 年、Novikov 等人 2025 年、Xia 等人 2026 年，以及 Ziaei Bideh 与 Gryak 2026 年的工作。细节各不相同，但都用大语言模型来设计模型本身。直到最近，这一步仍被认为只有人类能做。

可是，模型能复现数据，并不等于它提供了洞察。就像那张聚餐账单，或霍奇金-赫胥黎模型里的参数，同一模型内部许多不同的参数组合可以给出相同输出。再往上一层，对模型本身也成立：许多方程形状不同的模型，可以同样好地解释同一批数据。把一个智能体在同一批数据上跑两次，它可能返回两个不同、却同样好的模型。一个只交出其中之一的智能体，等于把其余的都藏起来了，就像只展示一组最佳拟合数字。

所以，构建模型最好也用约束模型的办法：去要全部那些有可能解释数据的模型。他们指向 Wahl 等人 2026 年、Mishra-Sharma 2026 年和 Murphy 2026 年的讨论。AI 系统提出许多候选，再按数据对每个候选的支持程度加权。那些必须非常灵活才能拟合上的模型，得到的分量会低于那些自然就能拟合的模型。弱的候选被淘汰，系统再对更强的那些提出新的变体。

图 2 把这个搜索画了出来。左侧每个点都是霍奇金-赫胥黎模型的一种可能变体，背景越暗表示模型越可能；搜索从起点向外分支，并在若干区域聚成簇。右侧是每个簇里由 AI 系统构建的模型模拟，以及各簇占全部模型的份额：41%、39%、11% 和 9%。作者写道，他们会同时保留更可能和不太可能的模型，因为所有模型都可能带来洞察。

这样，智能体就能提供一组说得通的模型，而不是单个最佳拟合。这个分布能告诉你单个模型说不出来的事。如果所有存活下来的模型都共享某个特征，数据很可能支持它。如果它们意见不一，说明证据还薄，也会把研究者指向下一场实验。

作者提醒不要走得太远。到目前为止，这些系统大多还在重新发现科学家已经知道的模型。它们能否处理没有已知答案的问题，证据仍然很少。不过，更新的大语言模型在研究任务上的表现已经好于更早的模型，作者预计这些系统还会继续改进。

解释是三步里最难交给 AI 的一步

模型被写出来、又被数据收紧之后，还得解释：它到底对正在研究的真实系统说了什么？霍奇金和赫胥黎的“四个粒子”假说，来自一个推论：如果通道打开前必须先发生四件分开的事，方程里就会出现四次方。于是他们推测，膜里面可能就是这样。后来看，这个判断大体是对的。

这个解释之所以站得住，是因为他们排除了别的可能。如果更简单的模型也能一样好用，就没有理由认为通道有好几个活动部件。是替代方案的失败，才让他们能提出这个断言。确切数字当时也有不确定：他们注意到需要三次方或四次方，最后选定了四。他们写道，更高的次幂也许还能拟合得稍好一点，但那点改进不值得再增加复杂程度。

这套方程还让他们推出了另外几层含义。比如，它解释了为什么神经细胞存在一个电压阈值，低于它不会发放、高于它才会发放，以及为什么神经脉冲都有相同的形状。它还预言：如果电流上升得足够慢，细胞将永远不会发放。科学哲学家把一种理论称为“可理解的”，如果它让你不必运行模型，就能推出它的后果。这个标准适用于变量只有少数几个的模型，霍奇金-赫胥黎模型就是如此。对一个可能有数千个变量的更大模型，对应的标准也许是：我们能不能把它收成几个关键的机制想法，而这些想法又足以把它重建出来。模型越大，这一点就越难。

在作者看来，三步里最难交给 AI 的，就是解释。目标更难钉死：模型拟合数据好不好，可以测量；它有没有带来对系统更好的理解，却量不出来。也不清楚该怎样让 AI 系统把“理解”当成优化目标。一种提议是看它能否预测：当你扰动这个系统时，系统会怎样变化。另一种办法是先绕开这个问题，改问智能体的输出能不能帮助人类科学家更好地理解问题。一个例子是用于设计量子实验的 AI 系统。它不被训练成只给出一套装置，而是写出一个短程序，生成一整族相关实验的装置。物理学家就可以直接从程序里读出底层设计原则，而不必从某一套实验装置反向还原。

这两种路径里，能看到一整组合适的模型，而不是单个拟合得最好的模型，对解释都至关重要。单个模型无法告诉你，它的哪些部分真正被数据约束住了，于是很容易把意义读进那些其实没被钉住的部分。在霍奇金-赫胥黎模型里，关于四个粒子的假说，只有在你知道更低次幂效果不够好时才说得通。一组模型也会让预测更稳妥。比如你问这些模型：如果把细胞膜上的钠通道全部堵住，或只堵住一部分，会发生什么。如果所有拟合数据的模型都给出同一个答案，就像全部通道都被堵住时你预期会看到的那样，这个答案就可以比较有把握。如果它们意见不一，比如只堵住一部分通道时就可能出现这种情况，这本身就能给下一场实验提供想法。

从一只神经细胞，到一只果蝇的整个大脑

包括霍奇金-赫胥黎模型在内，大多数著名模型都很紧凑：只有少量变量和方程，小到人类科学家可以写下来、也可以推理。这并非巧合。它们必须小，因为人得亲自构建、约束和解释它们。小，也正是它们可被理解的原因。可当 AI 接手更多工作时，这个体量上限会开始松开。我们可以开始为那些手工根本写不下来的复杂系统建模，即便再也无法只靠读方程来理解它们。

大脑是一个很直接的应用场景。霍奇金-赫胥黎方程描述的是单个神经细胞。果蝇的大脑有超过 10 万个神经元，人脑则有数百亿个。行为与思维，来自这些神经元之间的相互作用。对果蝇，现在有了前几代科学家不可能拥有的东西：一张完整的大脑接线示意图，标出每一个神经元，以及它们怎样彼此连接。作者提到的相关工作包括 Dorkenwald 等人、Schlegel 等人，以及 Azevedo 等人 2024 年的研究。研究者已经开始在这张图上建立大规模模型，其中一套可以预测果蝇视觉系统中神经元的活动。于是一些从前没法认真提出的问题出现了：整个大脑怎样把接线变成行为？一种动物的行为有多少能只从接线预测出来？哪些连接才真正要紧？

图 3 按果蝇中枢神经系统的完整接线示意图绘制，依次显示 1 个、10 个、255 个神经元，以及全部 124,635 个细胞。第 1 幅和第 3 幅里，有一个神经元用橙色标出。图像来自雄性中枢神经系统连接组，由 HHMI Janelia、剑桥大学、MRC LMB 和 Google Research 提供，采用 CC BY 4.0 许可。

这并不只发生在神经科学里。气候科学家已经这样工作：他们建立许多巨型数值模拟，各自基于略有不同的假设，再把预测报告成所有这些模拟之间的分布。大型强子对撞机上的粒子物理学家也有类似思路：寻找新粒子时，他们不会只宣布某个特定理论成立或不成立，而会画出一张地图，标出该理论的哪些版本已被数据排除，哪些仍然开放。

霍奇金和赫胥黎当年那些费力的工作，构建模型、用眼睛拟合每条曲线、试过那些拟合不上的次幂，以及赫胥黎花三个星期手摇计算器，并不是人们想回到的日常。可这些功夫没有白费。一个模型提供的许多洞察，来自创造它的过程。他们理解模型在说什么，是因为模型是自己建出来的，也知道哪些路没走通。正是这种理解，让他们形成了对门控变量的解释，而这是后来为他们赢来诺贝尔奖的发现的一部分。

作者认为，AI 系统现在按说可以在一个下午里完成其中许多工作。可它如果要提供理解，就得交出这两位科学家缓慢工作所产生的东西：不只是那个拟合上的模型，还要说明哪些部分被数据钉住了，哪些仍然开放，沿途哪些替代方案失败了，过程中又冒出了哪些新的谜题。科学模型可以提供一个答案，但给出洞察的，是构造它的过程。

下次看到一个拟合得很好的科学模型，无论它出自人手还是 AI，都可以先问几句。还有哪些模型同样解释得通？哪些部分真的被数据钉住了，哪些只是看起来像那么回事？存活下来的模型在哪些地方意见不一？

作者认为，最有用的下一步实验，往往是那些能一次排除最多剩余模型的实验，哪怕它们不一定是研究者自己最感兴趣的那几个。他们也写明了眼下的边界：这些系统目前大多还在重新发现科学家已经知道的模型；它们能否处理没有已知答案的问题，证据仍然很少。一个下午交出来的漂亮拟合，可以当作候选解释，还不能单独当成对真实系统的理解。

原文媒体未在改写中展示：原文图 1 是嵌在文章中的交互式 HTML 说明（https://assets.claude.ai/brand/artifacts/research/teaching_ai.html），可以拖动蓝点，查看不同钠通道和钾通道组合下的模拟是否拟合测量曲线。本站无法把该控件保存为可拖动的静态附件；图注与机制已在正文说明，读者可打开原页或上述链接自行操作。

原文

Teaching AI to build scientific models

Teaching AI to build scientific models

Published Nov 6, 2026 · Team: Science · Topics: Discovery, Research programs, Life sciences

AI for Science grantees Jakob Macke, Daniel Gedon, and Stefan Wahl are researchers at the ELLIS Institute Tübingen and the University of Tübingen, where they build AI tools for developing scientific models in neuroscience and beyond. In this guest post, they discuss how AI is starting to change scientific modeling.

In 1951, Cambridge University’s computer went out of action for six months. This was inconvenient for physiologist Andrew Huxley, who, alongside his colleague Alan Hodgkin, had just built a set of equations to describe how a particular squid nerve cell fired. The scientists wanted to understand the “action potential” of these nerve cells: the electrical pulses they use to communicate, which underlie everything brains do, from perception to intelligence. The only way to find out whether the equations worked was to solve them and compare the solutions to data.

Huxley had hoped to use the newfangled computer to solve the equations. Instead, he was forced to solve them using a hand-cranked mechanical calculator. It took him over three weeks.

The basic structure of their model was dictated by known facts about the cell. Hodgkin and Huxley modeled its membrane as an electrical circuit. One part of the model, however, was just an educated guess. In order to make the curve fit the data they had measured, they found that they had to raise one variable to the fourth power—that is, to multiply it by itself three times. At the time, they did not know why. They just knew that otherwise the model didn’t predict the observed data as well.

They did offer a hypothesis, however. They cautiously suggested that the variable in question, which governed how particular gates in the cell membrane opened, might require the fourth power because multiple charged particles in the membrane had to move into place before the gate would open.

Decades later, long after both Hodgkin and Huxley had won the Nobel Prize in Physiology or Medicine in 1963, another team found the structure of a related potassium channel protein. And they discovered that, yes, it had four identical pieces arranged around a central pore. The model was fit to the data in 1951. Even then, it predicted how fast the impulse travels relatively accurately. More than that, it turned out to have helped predict the structure of potassium ion channels themselves, which it had never been fit to.

Today, an AI system could plausibly find an equation like Hodgkin and Huxley’s in an afternoon, and perhaps it would fit the experimental data even more closely. But the point of the story is that simply knowing the fourth power did the job was not enough; what mattered was knowing that a simpler version of the model did not work. An AI system that merely hands you the model might give you an answer, but it does not give you understanding. Reproducing the data with a model is insufficient, because any number of models could do the same. What matters is being able to make claims about the real-world system itself, and ideally about other, similar systems. To do so, you need to know which features the model shares with other models that work, and which set it apart from models that don’t. As the mathematician Richard Hamming wrote in 1962, “The purpose of computing is insight, not numbers.”

A three-step program

More than 70 years on, the Hodgkin-Huxley model remains the standard description of electrical activity in neurons. The same kinds of equations, with different constants, describe nerve cells across the animal kingdom, from squids to humans, even though our evolutionary lineages separated more than half a billion years ago.

Even more important than its longevity, however, is what the Hodgkin-Huxley model tells us about how models work. Hodgkin and Huxley’s approach can be broken down into three steps, and most scientific modeling involves some version of them.

• The first is building the model. This means deciding what the model is made of and what shape its equations take. In Hodgkin and Huxley’s case, that meant the parts of the electrical circuit—the membrane and the gates—and the form of the equations, such as raising the underlying variable to the fourth power.
• The second is constraining the model—tuning its adjustable parameters to fit real measurements. For Hodgkin and Huxley, that included settings like how often the gates open under different voltages.
• The third is interpreting the model—asking what the model says about the real system, such as suggesting that the fourth power meant the gate had four moving parts.

In practice, scientists go through these steps repeatedly. If the numbers fail to fit the measurements, or if the interpretation makes no sense, they go back and revise.

All three steps were once done by people, but AI systems are increasingly involved in each of them. Machines have arguably made the most progress in constraining models, so we’ll start there.

Splitting the bill, or how to constrain models

When Hodgkin and Huxley constrained their model by eye and intuition, they ended up with a single set of numbers. This told them that the model could match the data. It didn’t tell them whether other numbers would have worked just as well or even better.

Imagine you know that a group dinner added up to $120, but not who ordered what. Many different combinations of orders could have added up to that amount—perhaps everyone ordered a mid-priced burger, or perhaps one person had steak and lobster while everyone else had soup. Models of nerve cells work on the same principle: different combinations of gate parameters can produce near-identical electrical activity. Simply showing one combination that fits well hides all the other plausible parameter settings, and other combinations might have led to different predictions. This is true of almost any phenomenon being modeled. Many different sets of parameters can explain the observed data equally well.

Figure 1 is an interactive illustration embedded from https://assets.claude.ai/brand/artifacts/research/teaching_ai.html

Figure 1. An illustration of constraining the Hodgkin-Huxley model. (1) A measurement from a nerve cell. (2) How likely the measurement is for each combination of potassium and sodium channel settings. Darker means a better fit. The blue dot is interactive. (3) Simulations of the model at three settings, compared to the measurement (gray). Some fit well, while others do not.

The solution is a statistical method called Bayesian inference. It gives you every set of numbers that works reasonably well, as a probability distribution that takes into account their fit to the data and your prior knowledge about the parameters. The distribution takes the shape of a curve, with the best fits at the peak and the weaker ones at the fringes.

But this method has usually been impractical for complex models, because it involves calculating how likely your measurements would be under every possible set of numbers. In Hodgkin and Huxley’s model, that would still be feasible, but for anything more complicated, it would be computationally impossible. For example, if each of the thousands of ion channels opens and closes at random, you would have to work out every combination. Returning to our restaurant bill analogy: it would be easy to answer, “If person A had the burger and person B had the steak, what would the bill come to?” but much harder to show which combinations could have led to the final bill, especially as the number of people and menu options grows.

A different approach called simulation-based inference (SBI) overcomes this reverse-engineering problem with sheer computation. You pick random parameters, run the model, and record the output. You do it again and again— perhaps 100,000 times for a model like Hodgkin-Huxley, and millions of times for larger, more complex ones. Then you train a neural network on the results so that, given an output, it can return the range of parameters that could have produced it.

All SBI requires is that you can run the model: put in a set of parameters and get back the kind of data you would expect to see. A model you can run like this—parameters in, data out—is called a simulator, hence the name “simulation-based inference.” You never need to use a formula to calculate how likely your measurements are, which is the step that makes standard Bayesian inference often impractical. These days a simulator is usually a computer program, but it doesn’t have to be: Huxley, cranking through the equations by hand, was running one. Many simulators are easy to run forward but impossible to reverse. Some can’t even be written out as a tidy equation—their assumptions are built into the rules they follow—but they are models nonetheless. For many complex phenomena, from networks of neurons to colliding galaxies, simulators are often the only kind of model we have. SBI lets us constrain those models with data.

Running all those simulations is expensive and can take weeks. But once it is done, a trained network can perform inference on new data in seconds, without requiring further simulations. That speed has changed what’s possible. When two neutron stars collide, SBI can work out roughly where in the sky the collision happened, fast enough to point telescopes at it while the signal is still detectable. The same approach has been successfully applied to problems in cosmology, structural biology, geophysics, and many other domains.

And SBI has been applied to nerve cell models, including the Hodgkin-Huxley model. Because it returns a distribution rather than a single fit, we can see which parameters are well constrained by the data, where more than one explanation is plausible, and where more data—and thus another experiment—is most needed.

The bill, one level up, or how to build models

Before you can constrain a model, you need a model to constrain. The equations of the Hodgkin-Huxley model were not derived from first principles. Instead, the fourth power of the underlying variable emerged empirically from the scientists’ biological insight, and from finding that simpler versions didn’t work as well.

For most of the history of science, this is how models have been built. Scientists used their expertise and assumptions to write down a candidate model. Then they tested it, revised it, and tried again. The problem is that the space of possible models is essentially infinite, and a scientist proposing a handful of models from experience will very likely miss models that could have explained the observational data equally well.

AI systems can now help with, or even take over, this step. Given some data and a natural-language description of a system, an LLM-based agent writes code for a candidate model or a set of candidate models. Then it runs the model, constrains its parameters, compares the output to real-world data, and revises it, perhaps by modifying the form of an equation. This loop repeats until the model replicates the data accurately enough. Several such systems have appeared in the past two years.1 The details vary, but all use an LLM to design the model itself. Until recently, this was thought to be something only humans could do.

But the fact that a model replicates data doesn’t mean that it provides insight. As with the restaurant bill, or the Hodgkin-Huxley parameters, many different combinations of parameters within a single model can provide the same output. The same is true one level up, for the model itself: many different models, with differently shaped equations, can explain the same data equally well. Run an agent twice on the same data, and it might return two different, and equally good, models. An agent that returns only one of them is hiding the rest, just as a single best-fit set of numbers does.

So it is better to treat building a model the same way we treat constraining one: you ask for the whole range of models that could plausibly explain the data.2 The AI system proposes many candidates. Each is then weighted by how much it is supported by the data, and the ones that need a lot of flexibility to fit are given less credit than those that fit naturally. The weak candidates drop out, and the AI system proposes new variations on the stronger ones.

AI agents can thus provide a distribution of plausible models, rather than a single best fit, and that distribution tells you things a single model can’t. If every surviving model shares a feature, the data likely favors it. If the survivors disagree, that shows where the evidence is thin, and points researchers in the direction of the next experiment.

We shouldn’t get too far ahead of ourselves. So far, these systems have mostly been tested on rediscovering models that scientists already know. Evidence that they work on problems with no known answer is sparse. Still, newer LLMs perform better at research tasks than older ones, and we expect these systems to keep improving.

Lost in translation, or how to interpret models

Once a model has been built and constrained, it still has to be interpreted. What does it tell us about the real system being studied? Hodgkin and Huxley’s four-particle hypothesis came about because equations involving the fourth power are what you would expect if four separate things all had to happen before a gate could open. So they suggested that this might be what was happening inside the membrane. As we saw above, they turned out to be largely right.

That interpretation only carried weight because of what they had ruled out. If a simpler model had worked just as well, there would have been no reason to think the gate had several moving parts. It was the failure of the alternatives that allowed them to make that claim. There was some uncertainty about the exact number: Hodgkin and Huxley noted that a third or fourth power was needed and settled on four. They wrote that higher powers might even have fit slightly better, but that the potential “improvement was not considered to be worth the additional complication.”

The equations also let Hodgkin and Huxley reason out several other implications. For instance, the equations explained why nerve cells have a voltage threshold below which they don’t fire and above which they do, and why nerve pulses all have the same shape. They also predicted that a current rising slowly enough would never make the cell fire. Philosophers of science call a theory “intelligible” if it lets you work out its consequences without having to run it. That test works for a model with a handful of variables, like Hodgkin-Huxley. For a larger model with perhaps thousands of variables, the equivalent might be whether we can boil it down to a few crucial mechanistic ideas that are enough to rebuild it. The larger the model, the harder that becomes.

Of the three steps, interpretation is the hardest to hand over to AI. The goal is harder to pin down: we can measure how well a model fits the data, but not whether it leads to a better understanding of the system, and it’s not clear how to tell an AI system to make “understanding” its objective. One proposal is to judge an AI system’s understanding by seeing how well it predicts what a system does when you disturb that system. Another is to sidestep the question altogether, and instead ask whether the agent’s outputs help human scientists better understand the problem. One example is an AI system for designing quantum experiments. Rather than providing a single setup, it was trained to write a short program that generates the setups for a whole family of related experiments. A physicist can then read the underlying design principle straight off the program instead of reverse-engineering it from one experimental setup.

In both approaches, having access to a full distribution of suitable models, rather than a single best-fitting one, is crucial for interpretation. A single model cannot tell us which of its parts are actually constrained by the data, so it’s easy to read meaning into parts that are not. In Hodgkin-Huxley, as we’ve seen, the hypothesis about four particles only makes sense if you know that lower powers don’t work as well. A range of models also makes predictions more robust. Say you ask your models what happens when you disturb a system—for instance, by blocking all, or only some, of the sodium gates on the cell membrane. If all the models that fit the data return the same answer, as you would expect when every gate is blocked, then you can be fairly confident in it. If they disagree, as they might when only some gates are blocked, that can provide ideas for the next experiment.

Cells and scales

Most famous models, including Hodgkin-Huxley, are compact. They consist of a handful of variables and equations, small enough for a human scientist to write down and reason about. That’s no coincidence: they needed to be compact, because people had to build, constrain, and interpret them. Their smallness is also what made them intelligible. But as AI takes over more of the work, the size limit will start to lift. We can begin to model systems that are far too complex to write down by hand, even if we can no longer understand them just by reading the equations.

Brains are an obvious use case. The Hodgkin-Huxley equations describe a single nerve cell. But a fruit fly’s brain contains more than 100,000, and a human brain tens of billions. And it is the interplay between those neurons that gives rise to behavior and thought. For the fruit fly, we now have something that was impossible for previous generations of scientists: a complete wiring diagram of its brain, showing every neuron and how it connects to the others.3 Researchers have started building large-scale models on top of it, including one that predicts the activity of neurons across the fly’s visual system. That opens up questions nobody could sensibly ask before. How does a whole brain turn its wiring into behavior? How much of an animal’s behavior can be predicted from its wiring alone? And which connections actually matter?

This isn’t unique to neuroscience. Climate scientists already work like this: they build many gigantic numerical simulations, each based on slightly different assumptions, and report their projections as the spread across all of them. Particle physicists at the Large Hadron Collider work in a similar spirit: when they look for a new particle, they don’t just declare whether a particular theory worked or not; they produce a map of which versions of the theory are ruled out by the data and which remain open.

The painstaking work Hodgkin and Huxley did—building the model, fitting each curve by eye, trying the powers that didn’t fit, and, in Huxley’s case, spending three weeks hand-cranking the mechanical calculator—is not something we want to go back to. But it was not wasted. Much of the insight that a model provides comes from the process of creating it; Hodgkin and Huxley understood what the model was telling them because they had built it themselves, and they knew what hadn’t worked. This understanding let them formulate their interpretation of gating variables—part of the discovery that ultimately earned them the Nobel Prize.

An AI system could now plausibly do much of the same work in an afternoon. But to offer understanding, it needs to deliver what the two scientists’ slow work produced: not just the model that fits, but an account of which parts are pinned down by the data, which are still open, which alternatives failed along the way, and what new puzzles were uncovered in the process. A scientific model may provide an answer, but it is the process of constructing it that provides the insight.

Footnotes

1. See Romera-Paredes et al., 2024; Shojaee et al., 2025; Ma et al., 2024; Novikov et al., 2025; Xia et al., 2026; and Ziaei Bideh and Gryak, 2026, for example.

2. See Wahl et al., 2026; Mishra-Sharma, 2026; and Murphy, 2026.

3. See Dorkenwald et al., 2024; Schlegel et al., 2024; and Azevedo et al., 2024.

Contributors

Jakob Macke, Daniel Gedon, and Stefan Wahl

Acknowledgements

We thank Xander Balwit, Tom Chivers, and Siddharth Mishra-Sharma for review and editorial guidance.

The male CNS connectome figure is rendered from the male CNS connectome (HHMI Janelia, University of Cambridge, MRC LMB, and Google Research), CC BY 4.0.

原文链接：https://x.com/AnthropicAI/status/2118792930211107115

相关链接：https://www.anthropic.com/research/teaching-ai-to-build-scientific-models

乔木改写

同样玩《宝可梦 红》，Claude Sonnet 3.7 写了 413 份记忆文件，拿到 1 枚徽章。Claude Fable 5 只留下 10 份文件，拿到 4 枚。

Anthropic 对比的是两个模型各一次运行、约进行到 10,000 步时的情况。这个对照无法单独证明文件越少越好，但它暴露了一个具体问题：记下的东西越多，智能体可能越忙着整理过去，越顾不上正在做的事。

Sonnet 3.7 开始创建第四个“主记忆”文件夹后，接下来 17 步都在整理文件。它新建 6 份，删除 9 份，游戏一步没动。

怎样让智能体留下真正有用的经验，还能纠正已经记错的内容？Anthropic 在 2026 年 11 月 6 日发布的《The components of agent memory》中，介绍了他们对记忆系统的拆解，以及一套叫作 dreaming 的整理过程。

智能体空闲时，可以回头核对自己记了什么

Dreaming 可以理解为智能体利用会话之间的空闲时间，重新检查日志、修正笔记、整理知识，以及挑出下次会话应该提前读到的内容。

这套思路借鉴了人类记忆的分工。海马体先捕获当下的体验，睡眠和安静休息期间的重放，再帮助其中一部分体验逐渐整合成长时知识。Letta 与加州大学伯克利分校的研究者此前提出过“睡眠时计算”，让模型在交互间隙重组已有上下文。

智能体也会留下大量工作日志。任务进行时，它可以先记录重要线索；任务暂停后，再回头检查这些线索是否正确、是否值得长期保留。

图 1：从原始会话日志，到记忆笔记、知识库，再到每次会话预先载入的上下文。Dreaming 可以在这些层级之间整理信息。

文章里的“记忆”有明确边界：Claude 从任务或用户互动中得到经验，把经验保存下来，供之后的会话重新读取。修改模型权重、单次对话内的学习，以及为延续上下文而做的对话压缩，都不在这次讨论范围内。

Claude 可以给日志中的重要事件打标签，也可以把观察写进记忆文件夹。难处在于，写下笔记的那一刻，它还不知道下一次任务会问什么。

图 2：Claude 在会话中挑出值得保留的观察，写入跨会话保存的记忆笔记。

有用的笔记，要把猜测变成验证过的规则

写下一条规则，实际上是在预测：未来还会遇到类似情形。若笔记里只剩未经检验的猜测，下一次读取它时，模型就可能把旧猜测当成已知事实。

Anthropic 用 CL-Bench 1.0 的数据库任务观察这个过程。Claude 需要查询数据库，逐题回答问题。每次作答后，系统给出正确答案；模型可以复盘错误、提出解释、验证解释，再把结果写入记忆。

每道题都从全新会话开始，只有记忆目录会保留下来。这让评测能够观察跨会话记忆带来的变化。

不同模型写下的东西差别很大。Sonnet 4.6 留下不少分散文件，里面是答错原因的猜测，但缺少验证。Opus 4.7 开始把笔记整理成参考文档，区分已核实与未确认的内容。Sonnet 5 会在评分器确认后标注答案，答错时补上纠正。

Opus 5 是这批模型中第一个把这个过程接完整的：记录规则，同时写清依据、以前的答案，以及途中发现的陷阱。Opus 5.5 和 Sonnet 5.5 也能保存经过验证、供后续作答使用的规则。

图 3：CL-Bench 数据库任务中，各模型在有无记忆时的平均奖励；右侧展示约 30 道题中的滚动平均变化。

具体分数如下，前一个数值为无记忆基线，后一个为使用记忆后的结果：

• Opus 5.5：0.655，0.838。
• Sonnet 5.5：0.644，0.833。
• Opus 5：0.564，0.828。
• Opus 4.7：0.383，0.700。
• Sonnet 5：0.306，0.610。
• Sonnet 4.6：0.330，0.364。

Opus 5.5 和 Sonnet 5.5 的无记忆基线更高，但加上记忆后，得分与 Opus 5 大致相当。Sonnet 4.6 的提升则很小。给模型一个记忆文件夹，还不够；它得会读取、验证和更新里面的内容。

屏幕上已经有的信息，未必值得再存一遍

《宝可梦 红》的对照展示了另一种区别：模型如何判断什么值得记。

Sonnet 3.7 几乎把所有遭遇都存了下来，战斗、道具、遇到的角色，常常各建一份文件。约 10,000 步时，它留下 413 份文件，散落在 225 个文件夹里，总计 977,930 个字符。

Fable 5 的记忆只有 43,406 个字符：两个文件夹里的 9 份文件，加上顶层的一份 global.md。

图 4：同一比例下，两份宝可梦记忆目录的面积对照。Sonnet 3.7 的记录庞杂，Fable 5 的目录小得多。

金钱余额、当前位置，每回合看屏幕就知道。游戏状态又一直在变，反复保存这些信息，容易留下很快过期的记录。

Fable 5 更倾向于保存屏幕不会直接告诉它的内容：接下来要做什么、怎样通过某条路线、哪些策略试过有效。它把当前目标、道馆计划和记忆使用规则放在 global.md，把路线计划与游戏策略分开保存。

例如，它连续 7 次用精灵球捕捉鲤鱼王失败后，写下了一条经验：连续失败达到 3 次或更多，就改变按键时机，在菜单步骤之间多等待一下。它还记录，调整后立刻捕获成功。这是那次运行里总结出的经验，不能据此认定为游戏的普遍机制。

Opus 5.5 也采用了类似做法，用一个小文件夹跟踪当前目标和下一步，边玩边改，最终通关。

同一句“只记失败”，对强模型也可能起反作用

该怎样告诉 Claude 使用记忆？研究者发现，提示需要与模型能力匹配。

在 CL-Bench 上，Opus 4.7 和 Opus 5 只需知道有一个记忆文件夹可用，就能从中获益。若再强制规定“只记录失败”，两者原本从记忆中得到的提升，反而损失了很大一部分。

对 Sonnet 4.6，问题又不同。它经常不读记忆，也不验证答错原因的猜测。明确要求它作答前读取笔记、检验推测后，平均奖励从仅使用记忆时的 0.364，提高到 0.641；无记忆基线为 0.330。

图 5：左侧为 Sonnet 4.6 加入读写与验证提示后的变化；右侧为“只记录失败”对记忆增益的影响。

“只记录失败”带来的记忆增益变化，Opus 5 为负 0.234，Opus 4.7 为负 0.150，Sonnet 4.6 为负 0.020。图中同时给出参考：Opus 4.7 使用记忆时为 0.700，Opus 5 为 0.828。

Sonnet 5 在没有额外提醒时，10 次里有 9 次会在回答前先读记忆；Sonnet 4.6 的比例不到一半。较旧模型需要补上基本动作，更强模型则可能受限于过细的固定规则。

Anthropic 对前沿模型通常只给简单指引：用户要求时，或有明显值得保留的内容时，创建笔记。其余判断交给 Claude。

一条记错的位置，能让智能体反复走回头路

笔记保留下来后，错误也可能跟着保留下来。Dreaming 的一个用途，就是拿笔记与原始会话记录核对。

图 6：回到原始日志检查记忆，找出并修正笔记中的错误。

研究者在《宝可梦 红》里遇到过一个例子。玩家需要穿过月见山，入口在西侧，出口在东侧。入口附近有一道只能往下跳的岩架，跳下去就会掉回来时的路线。

Claude 把入口附近的宝可梦中心记成了位于东侧。于是它走出中心后，判断月见山已经走完，并在笔记里标上完成。几步之后，它跳下岩架，回到了先前的路线。

导航工具没有显示这次坠落。Claude 又给错误补了一个解释，认为只是向西走过了头，自己此前所在的位置本来就正确。

图 7：左侧为受错误笔记影响的路线判断；右侧为核对记录、纠正笔记后返回入口的路线。

研究者随后让 Claude 对这份记忆运行 dreaming。它回看会话记录，发现宝可梦中心的位置记错了，改写笔记，之后直接回到月见山，最终到达华蓝市。

未纠正笔记的那次运行，Claude 三次掉回旧路线。第三次失败后，它表示想停止循环，实验的自动回复要求继续，它便一路退回尼比市，准备疗伤补货。运行结束时，它仍未到达华蓝市。

图 8：约 3,000 次累计动作内的进度对照。加入 dreaming 的运行在第 1,935 次动作到达华蓝市，另外两条运行未到达。

这次实验提供了一个具体的纠错案例。它说明原始日志可以作为核对笔记的依据，但并未证明 dreaming 能可靠修复所有类型的记忆错误。

知识库配合搜索，比单独依赖知识库更有效

记忆增加后，Claude 还可以把零散笔记整理成带链接的知识库，也就是文章里的 wiki。每个页面汇集一个主题，链接指向其他笔记，方便后续寻找。

图 9：Dreaming 把日志与笔记整理为带链接的知识库，帮助后续会话定位信息。

研究者用 Stainless 的内部文档做实验。Anthropic 在五月收购了这家公司，这批语料约有 35 万个文件、4.2 亿个词元。他们围绕语料制作 25 道题，给 Opus 5 智能体搜索工具，并让它在 128 个会话中建立知识库。

每轮继续前一轮的工作，通过共享日志协调：搜索文件、写页面、纠正内容、合并重复项，并回到来源核查说法。

从 2 个会话增加到 128 个会话后，仅依靠知识库作答时，满足评分标准的比例从 13% 增至 48%。同时提供知识库和搜索工具，分数则从 54% 增至 61%；只有搜索工具时为 53%。两者结合，每题还大约节省 20,000 个词元。

图 10：Dreaming 从约 200 万个词元增加到约 1.28 亿个词元时的评分与知识库变化。示例知识库从 103 个页面、766 条链接，扩展到 222 个页面、12,893 条链接。

知识库的价值也取决于题目。研究者又使用并购文档与 Harvey LAB 法律基准，比较直接搜索和查阅知识库的表现。

一项并购测试有 24 道题，其中超过一半需要跨文档查找，不少还需要判断或计算。在需要跨文档或额外判断的条目上，查知识库的得分为 87%，只搜索为 73%。

图 11：左侧任务需要跨多份文档组织信息，其中 9 项需要判断或计算；右侧多数要求能从单份文档直接搜索到。

另一个 LAB 任务中，答案常常就在一份文档里，知识库没有带来明显提升。还有一次，10 份文档共 115 KB，Claude 却整理出 203 KB 的知识库，包含 68 个页面、255 条链接，体积达到原文的 1.77 倍，一些页面还没有链接。

图 12：知识库比原始语料还大，却没有完成有效抽象。规模增长本身无法证明整理有效。

作者把它比作博尔赫斯笔下与帝国一样大的地图。若整理后的材料几乎只是重写原文，使用者仍得面对差不多的阅读负担。

每次开场先读的记忆，应该保持小而清楚

Dreaming 也可以挑出少量内容，在每次新会话开始时预先载入。Claude 无须先搜索，也能知道有哪些经验值得留意。

但这层内容会占用上下文。若混进错误，每次新会话都会重新读到。指令越长，也不代表模型越听话，Claude Code 文档明确提醒，较长的指令文件会降低遵从度。

因此，这一层需要保持精简：详细内容下沉到具体文件，入口只留下必要说明与链接。

Claude Code 的自动记忆采用类似安排。每次会话只载入 MEMORY.md 的前 200 行或 25 KB，用它作为索引，说明哪些信息已经保存、该去哪里找。这层索引由 Claude 在会话中维护，自动记忆的全部内容不会随每次会话一起载入。

图 13：把少量记忆提升为新会话的预载上下文，细节继续保存在下层。

整理合同，先保证条款不会被覆盖

知识库该怎样组织，还得看它最后要完成什么任务。像 Claude Duet 这样的助手，可能需要围绕服务对象组织记忆；合同任务则必须保全具体条款。

LAB 的一项模拟收购任务中，研究者最初只要求建立知识库，没有说明条款才是评分重点。多个智能体同时往同一张共享条款表里写内容，摘要丢失了细节，部分内容还被彼此覆盖。

用这份知识库生成的简报，只覆盖 57 项事实中的 41 项，约为 72%。

研究者让 Claude 回顾失败，调整指令，再重新建立知识库。修改后的做法是：每份合同单独成页，由一个智能体负责，逐字引用条款。之后生成的简报满足了全部 57 项评分要求。

图 14：从多个智能体共写一张条款表，改为每份合同独立成页并保留条款原文，覆盖情况从 41／57 提高到 57／57。

第二轮整理还可以检查跨会话留下的缺口。另一项任务里，6 个页面上的 9 条链接指向一个不存在的页面。Claude 找到同一文档已存在的页面，修正链接，另补上索引等 3 个页面。

图 15：第二轮整理前后，页面数从 38 增至 41，链接从 225 增至 278；原有的 9 条断链从不存在的 escrow-agreement 改指向已有的 buyer-form-escrow-agreement，并补上索引、指令邮件和升级处理页面。

先查记忆有没有被用对，再决定要不要多记

这些实验仍留下不少问题。当场写记忆，重要信息能立即供后续使用，但会分走任务注意力；后台整理能减少打断，结果却未必来得及用于当前工作。

纠错也需要算力。反复检查全部记忆未必可行，研究者提出，可以优先关注容易变化的信息，或根据记忆被读取的频率调整整理节奏。会话记录搜索仍可作为基础，知识库等整理层要持续维护，避免积累错误与幻觉。

这篇研究没有给出适用于所有任务的固定配置。没有模型能预知后续会话会遇到什么。原文给出的实验，更适合转化成几个检查问题：记忆是否被读取，规则是否验证过，笔记是否已经过期，索引能否带到原始依据。

若智能体正在反复犯同一个错误，先回看它读到的笔记，再考虑扩充记忆。若任务用搜索就能完成，先比较增加知识库带来的收益和维护成本。需要优化的，是下一次任务能否更可靠地完成。

原文：The components of agent memory。作者为 Lance Martin，配图为 Monika Tuchowska。本文数据与案例均来自该文，实验结果仅对应文中所述设置。

原文

The components of agent memory

The components of agent memory

Published Nov 6, 2026 · Team: Engineering

Written by Lance Martin with editorial guidance from Caitlin Love, Tom Chivers, Rebecca Hiscott, Jake Eaton, and Blake Wilson. (Full acknowledgements at end.)

Søren Kierkegaard wrote that life can only be understood backward, but it must be lived forward. We've encountered a similar bind when designing memory for AI agents.

As humans, we encode memory constantly as we go about our lives. The hippocampus plays a central role in this, capturing experiences in the moment so that memories form quickly and are immediately available. But only some experiences are consolidated into long-term knowledge.

Much of that consolidation appears to happen when our brains are offline. As we sleep, and during quiet rest, the hippocampus replays our recent experiences. This is thought to gradually distill those experiences and integrate them into the neocortex—the part of our brains that stores most of what we think of as knowledge.

Agent memory can use a similar division of labor. Agents capture much of their work in logs or transcripts. Researchers at Letta and UC Berkeley proposed sleep-time compute as a way for models to use idle time between interactions to reorganize their existing context. The same idea can help agents organize their memory by reflecting on logs to fix mistakes or produce new notes and representations. This process is often called dreaming (Figure 1).

We've found that dreaming has been a useful tool for helping us give Claude effective memory. In this post, we share how we've come to think about agent memory and walk through several evaluations we've used to explore agent memory, including the use of dreaming, and what each component contributes to memory organization.

Claude can use dreaming to organize memory. This can correct mistakes, create notes, build new memory representations, or shape what's preemptively loaded into future sessions.

Memory notes

By "agent memory," we mean knowledge that Claude gains from experience—including interactions with users and tasks it's been assigned—and that it retains between sessions, saved as context it can read later. It excludes other ways models can learn or manage context, such as changes to their weights (the internal parameters set during training), in-context learning (knowledge gained within a single conversation), and compaction (summaries written by Claude that replace its older turns so it can continue working in a new context window).

In the human brain, chemical signals released before, during, or after an experience can mark a memory as important. Marked memories tend to be replayed more often than unmarked ones during offline periods. This process, called replay, helps stabilize and integrate them into long-term memory.

Similarly, Claude can tag salient events in a session log as they happen, so they're easier to find later, or it can write them to a memory store, such as a folder of memory notes (Figure 2). But how can Claude tell what's salient in the moment, before it knows what's coming next?

Claude can write notes to memory.

Memory can be a prediction about what the future will need

Memory is, occasionally, a prediction. By writing down a rule, an agent predicts that the pattern it's describing will be relevant again. For instance, writing down a verified fact is a prediction that an agent in a future session will want to rely on that fact without double-checking it. Over time, Claude has become much better at making these kinds of predictions.

Consider a task from Continual Learning Bench (CL-Bench) 1.0, a benchmark that measures an agent's ability to use memory in order to tackle long-horizon tasks. One particular task asks Claude to answer a series of questions by looking up information in a database. After Claude gives an answer, it's told the correct solution and can write a memory to help answer future questions.

Memory can support a learning loop here. If Claude gets a question wrong, it can record the failure, come up with theories about what went wrong, test those theories, and turn what it finds into a rule about how to answer in the future.

We ran the CL-Bench 1.0 task using Claude Managed Agents. Claude started each question in a fresh session, with one exception: it could retrieve files from a memory directory that persisted from one session to the next.

Different models achieved different levels of success at creating their own learning loops. Claude Sonnet 4.6 got as far as generating theories about why an answer was wrong. Its memory directory contained a scattering of files, each listing several untested guesses.

Claude Opus 4.7 consolidated its notes into a single reference document and began marking which facts it had checked and which were still unconfirmed. Claude Sonnet 5 marked each answer as confirmed after the grader had checked it, and wrote a correction when it had answered wrong.

Claude Opus 5 was the first of these models to close the loop. Most of its entries included a rule (and also how it knew the rule was true), answers to earlier questions, and clear warnings about traps it had discovered along the way (Figure 3).

Like Opus 5, Claude Opus 5.5 and Claude Sonnet 5.5 both closed the loop, using memory to capture validated rules about how to best answer future questions. With memory, both scored about the same as Opus 5, though their baseline performance without memory was slightly higher.

Claude has learned to judge salience

We also evaluated Claude's ability to play Pokémon Red, a game that challenges Claude to navigate complex long-horizon tasks, as a test of Claude's judgment about what to save to memory. At the start of each run, we gave each model a folder of memory files that it could read from and write to while it played (Figure 4).

We compared the memory folders from one run each of Claude Sonnet 3.7 and Claude Fable 5, when each was about 10,000 steps into the game. A key difference we noticed was what each model thought was worth remembering. Sonnet 3.7 assumed everything might matter and wrote to memory indiscriminately. Its memory folder contained an archive of encounters, often with a separate file for each battle it fought, each item it found, and each non-player character it met. Ultimately, Sonnet 3.7 wrote 413 memory files across 225 folders and earned one badge (badges mark a player's progress through the game).

Fable 5, by comparison, wrote nine files in two folders, plus a single top-level file called global.md, and earned four badges.

Sonnet 3.7's approach was a problem mainly because the game state of Pokémon changes constantly—Sonnet 3.7's detailed record was continuously going out of date. As a result, as Claude progressed, it had to spend more time and resources combing through its memories rather than playing the game. Around step 10,000, for example, Sonnet 3.7 started its fourth folder of "master" memory files. Over the next 17 steps, it created six files, deleted nine others, and took no actions in the game itself.

Fable 5's approach used memory for things the game didn't explicitly tell it. Some of the kinds of information Sonnet 3.7 wrote to memory—such as how much money the player has or where the player is on the map—can be read off the screen every turn, so storing it is redundant. By contrast, memory files about strategy, plans, and tested rules exist only if Claude works them out, and they disappear unless Claude writes them down.

Fable 5 organized its memory accordingly. Its global.md file acted as a dashboard, holding the current goal, a prewritten plan for tackling gyms (the challenges where badges are won), and the model's own rules for using memory. A locations folder held plans for navigating particular routes, and a gameplay folder held strategies based on what it had learned about how the game works. For example, after seven Poké Balls in a row failed to catch a Magikarp (one of the easiest Pokémon to catch), Claude wrote, "If 3+ balls fail in a row, VARY the timing (insert extra 'wait' presses between menu steps) - the catch landed right after doing this."

With a much smaller memory folder, Fable 5 progressed much further in the game than Sonnet 3.7. Opus 5.5 took a similar approach: it kept a small folder of memory notes that tracked its current goal and next steps, rewrote them as it played, and beat the game.

Steering Claude to write memory

We've found that it helps to tell Claude what memory is for, but not how to use it. On CL-Bench, we simply told Claude that a memory folder was available. That general guidance was enough for Opus 4.7 and Opus 5 to use memory effectively, and it gave both a noticeable boost in performance (Figure 3).

We also tested giving the models specific rules about what they should write to memory, with one rule per test. We compared how much each model gained from memory with each rule against how much it gained without it. When we set a rule to "record only failures," both Opus 5 and Opus 4.7 lost much of the performance boost that memory gave them (Figure 5, right). For stronger models, giving them freedom to judge salience was more effective than setting fixed rules in advance, and those rules could end up weakening performance.

We also noticed that steering helped older models the most. For instance, Sonnet 4.6 often didn't read files from its memory directory and failed to test its theories about why an answer was wrong.

In this case, prompting Claude to read its memory files before answering—and then to explicitly test its theories—nearly doubled its benchmark score (Figure 5, left). Sonnet 5 does this on its own. Without being prompted to, it read its memory before answering 9 out of 10 times. By comparison, Sonnet 4.6 did this less than half of the time.

With frontier Claude models, we typically give simple guidance to create memory notes when a person asks them to or when something is clearly worth remembering, and we leave that judgment up to Claude.

Dreaming can correct memory

Since Claude writes to memory based on what it judges to be salient in-session, a note that's wrong from the start or that drifts as the environment changes can be read, and trusted, by Claude in a later session.

Dreaming gives Claude a way to test its notes against the original transcripts (Figure 6). We saw a positive example of this when Claude played Pokémon using Managed Agents with a mounted directory for its memories.

In one part of the game, the player has to get through Mt. Moon, a cave system with its entrance on the west side of the map and its exit on the east. Near the entrance is a trap: a one-way ledge that drops the player onto the route they came from, with no clear way to climb back up. Claude was using the PokéCenter, a building at the mountain's entrance, as a landmark, so whether it walked off the ledge depended on which side of the mountain it thought the PokéCenter was on.

At one point, Claude wrote down (incorrectly) that the PokéCenter was on the east side. When it walked out of the PokéCenter, it announced, "Mt Moon is behind me," and updated its notes: "Mt Moon DONE." But Mt. Moon was not behind it. A few actions later, Claude went over the ledge and landed back on the route it had come from (Figure 7). Its navigation tool didn't show the drop, so Claude concluded, "I overshot far west," and that it had been on the correct side all along. It hadn't.

We then tested whether dreaming could help Claude avoid this trap. We ran dreaming on the memory folder that contained the wrong location for the PokéCenter: Claude checked the note against the session transcript, caught its error, and rewrote it. The transcript gave Claude a ground truth to validate its memory against. With the corrected memory note, Claude went straight back into Mt. Moon and eventually reached Cerulean City.

In the run with the uncorrected note, Claude dropped back onto that earlier route three times. After the third failure, Claude seemed to give up, writing, "I'm going to stop rather than keep looping." Our setup's automatic reply instructed it to continue. So it did, all the way back to Pewter City, the town before the mountain, where it hoped to heal and restock. The run ended without Claude ever reaching Cerulean City.

Dreaming can build new memory representations

With dreaming, Claude can build representations to help navigate memory (Figure 9). One type of memory representation it uses is called a wiki, a collection of pages written and maintained by Claude that gathers what Claude knows about one subject and links it to other notes. These links effectively provide Claude with a map it can use to sift through its memory and efficiently locate memories when they're needed.

Representations like wikis can help Claude traverse its ever-growing knowledge base. We explored the benefit of wikis as a memory representation using a corpus of internal documents—about 350,000 files, roughly 420M tokens—from Stainless, a software company we acquired in May. We created an evaluation set of 25 questions about this corpus and gave Claude a search tool to explore it. We also instructed Claude to build a wiki based on these files using dreaming. Over 128 sessions, we prompted Opus 5 agents to explore the files, write or correct linked wiki pages, merge duplicates, and check claims against the sources.

Each session built on the progress that agents made in prior sessions, using a shared log to coordinate work. As we scaled dreaming from two sessions to 128, performance on the 25 questions grew from 13% to 48% of graded criteria (Figure 10).

Claude builds its wikis incrementally. After creating an initial wiki, it uses subsequent rounds of dreaming to add to, edit, or prune it. In the early rounds of dreaming, Claude's knowledge is like that of a new employee, and its ability to answer questions based on the wiki alone is limited. The more it dreams, the richer its knowledge map grows, like that of a seasoned employee. In our tests, we found that the wiki helped Claude the most when we also gave it access to a search tool. When it used both search and the wiki, Claude's aggregate score rose from 54% to 61% as that wiki grew, compared with 53% when it had access to the search tool alone. Claude's best score was 48% when it used only the wiki to answer questions. Using search and the wiki also saved about 20,000 tokens per question.

We wanted to test which kinds of tasks benefited from Claude having a wiki. To do this, we used the Harvey LAB benchmark along with a collection of documents about a merger. The benchmark gives Claude a set of documents and asks Claude to generate a legal brief based on their content. The benchmark then grades the brief against a list of criteria. While a tad less engaging, perhaps, than a Pokémon run, these evaluations give us diverse collections of documents to build wikis from.

We instructed Claude to build a wiki from the documents. We then asked Claude to answer questions (about the merger) or generate legal briefs (the LAB benchmark) both from the wiki and from a direct file search. One test asked 24 questions about the merger (Figure 11, left), more than half of which needed Claude to find information across several documents, and many of which also needed Claude to use judgment or do math. On those items (red and orange lines), Claude scored 87% when it consulted the wiki, compared with 73% when it used search alone.

In other cases, the wiki was less useful. On one LAB task (Figure 11, right), brief criteria often could be retrieved with a text search from a single document, and the wiki showed no clear gain in performance relative to search. On another LAB task, the wiki grew larger than the documents themselves (Figure 12). Like the mapmakers in Borges's one-paragraph short story "On Exactitude in Science," who drafted a map so detailed it was as large as the empire it represented, Claude had made an extremely detailed wiki that effectively abstracted nothing.

Dreaming can promote notes up the memory hierarchy

Dreaming can also float notes to the top of the memory hierarchy, and the agent harness can preemptively load them into Claude's context at the start of every session (Figure 13). This means Claude doesn't need to search for them, or even to know there's something to search for.

Promoting memories into a new session (a form of context injection) must be done carefully. Injected memories use up valuable tokens, and an incorrect note can be carried into every session. Loading a note also doesn't guarantee that Claude acts on it: our documentation on Claude Code warns that longer instruction files reduce adherence. So the top layer of the memory hierarchy has to stay relatively small, and much of dreaming's work here is keeping it that way—by moving some details down the hierarchy and leaving links behind.

Claude Code's auto memory keeps its injected layer small in the same way, with Claude maintaining it during the session. Claude doesn't load all of its auto memory into every session's context window. It loads only the first 200 lines or 25 KB of MEMORY.md. This header serves as an index, which shows Claude what it knows and where to find it.

Dreaming can focus memory on a specific goal

Dreaming can also shape how memory grows. An assistant like Claude Duet, for example, often benefits from having memories that are organized around the person it's helping, and dreaming can be tuned to build memory in service of a particular goal.

The instructions that guide how memory grows differ by task and tell Claude what information actually matters. For example, one task in the LAB benchmark simulates a company acquisition and provides a set of contracts. The brief is graded on whether it correctly describes the contract terms.

We started with basic instructions for generating the wiki that didn't tell Claude how to use the wiki. This task relied heavily on contract terms. While the wiki was being built, several agents wrote to a shared table of contract terms at the same time (Figure 14, left).

The result was a wiki that was missing key facts. When we graded a brief generated from the wiki, Claude had included only 41 of the 57 facts (72%). When we looked at what had gone wrong, we found that the shared table summarized the contract terms poorly and was missing some of them altogether, because agents had overwritten one another's work.

The contract terms were the most important part of the wiki, but the instructions didn't say that. To find better instructions, we ran a loop in which Claude reflected on its failures, proposed a change to the instructions, and rebuilt the wiki. This loop allowed Claude to discover how important it was to follow the contract terms, and it rewrote the instructions for building the wiki accordingly.

Each contract got its own page in the wiki, written by a single agent, which quoted the terms word for word rather than summarizing them into a shared table. When we graded the brief generated from this improved wiki, 100% of the criteria could be met, up from 72%.

Dreaming can fill in gaps a session leaves behind

When Claude writes memory notes, it sees only the context in that particular session. Dreaming can search across many sessions to find common patterns or correct common mistakes.

When building the wikis for the LAB benchmark (as an example, see Figure 14), sometimes Claude would link to pages that hadn't yet been created. A second dreaming pass closed many of these gaps. On one task (Figure 15), dreaming found nine links to a missing page in the wiki and repointed them to an existing page for the same document. It also added three new pages, including an index, to improve the wiki.

Thinking backward, looking forward

There are many open questions about how to design memory systems for agents. One is when Claude should create memory. When Claude works on a task, it can write to a memory store (e.g., a folder, file, or database), capturing salience at the source and making important details immediately available for future turns or sessions. But logging notes in the moment inevitably draws Claude's attention away from its main task. In contrast, dreaming can push memory creation to the background, but any outputs from dreaming may not be immediately accessible to Claude.

Another question is how to keep memory reliably up to date. Claude can make incorrect predictions or write rules to memory that are wrong from the start. Memory can also drift and become irrelevant as the environment changes. Dreaming is one way to fix these mistakes (Figure 8), but it requires compute. Directing Claude to continuously check all its memories might not be practical. It may be important instead to focus dreaming on information that's likely to change or to carefully tune the cadence of dreaming (e.g., based on how often memories are accessed).

It's also reasonable to consider transcript search as the foundation of Claude's memory. Combining representations (e.g., wikis) with search (Figure 10) can sometimes be effective. But Claude needs to use compute to maintain memory representations and make sure they don't accumulate errors or hallucinations.

Even though Claude's judgment about salience has improved, no model can really know what will happen in future turns. Dreaming gives Claude an opportunity to look backward and use compute to organize its memory, correct mistakes, and maintain representations that allow future sessions to access memory more readily. Memory systems will evolve as Claude gets more capable, and we still have a lot to learn about designing Claude's memory systems.

Acknowledgements

Written by Lance Martin with editorial guidance from Caitlin Love, Tom Chivers, Rebecca Hiscott, Jake Eaton, and Blake Wilson. We're grateful to Barry Zhang, Theodora Chu, Daliang Li, Mahesh Murag, Jake Noble, Ravi Trivedi, Tina Vachovsky, Brad Abrams, Henry de Valence, Jeremy Fox, Matt Bell, Allen Li, Chris Gorgolewski, Eugene Yan, Artur Renault, Wei Zhang, and Kevin Chen for helpful discussions and contributions, and to David Hershey for Claude Plays Pokémon. Figures by Monika Tuchowska.

原文链接：https://x.com/AnthropicAI/status/2118814753154064501

相关链接：https://www.anthropic.com/research/components-of-agent-memory

乔木改写

Blair 先把自己的身份告诉了 Alex。Alex 随后声称，自己的线索恰好能与 Blair 的身份对上。扮演好人的 Astra 当场指出：身份已经泄露在前，后来的吻合不能算独立证据。当天，Alex 被处决；他正是恶魔。

这段对话来自 Dwarkesh Patel 的 Clocktower 项目。项目让不同模型组队玩《血染钟楼》，再从对局记录中分析它们如何说谎、识破谎言和投票。本文完整转写其中一份关于 GPT-6 Astra 好人阵营的分析报告：它读了 26 局 Astra 担任好人的比赛，统计取自总计 372 局的 events.jsonl 和 probes.jsonl，并重点细读了对阵 Sol 6.1、GPT-5.6 Sol、Opus 5.5、Fable 5.1 的配对局和持续较久的对局。

这份分析所属批次标注为 2026 年 10 月 5 日，v16 f2 与 v17 f3/f3b，并非今天新做的实验。11 月 6 日，OpenAI 研究员 Noam Brown 在 Patel 发布项目后称赞了评测的方法设计，Anthropic 的 Sholto Douglas 随后转引讨论。这是两人的个人评论，不是两家公司对这份分析或其结论的官方背书。下文数字均保留这份报告的旧批次口径，不与后来公布的比赛总数混用。

26 场胜利，差别落在了处决和投票上

报告的判断是，Astra 找出恶魔的速度并没有比其他模型快很多；更突出的表现，是较少处决己方玩家，也不容易在已经选对处决目标后被劝走。这样的表现出现在整桌玩家之间，不只靠某一个特别强的席位。

报告首先归纳了几种反复出现的做法：记录每项证词来自谁、何时说出，以及说话前是否已经听过相关结果；有人把证据说过头时，立即纠正其中不成立的前提；把反复改口、不断更换解释本身也计入判断。但这 26 局也有明显的运气和对手因素：其中 6 场由猎手的一枪决定胜负，而且 Astra 的好人阵营从未遇上这项评测里最强的说谎者——Astra 自己扮演的邪恶阵营。

Astra 每局处决的邪恶玩家约 1.9 人，与其他模型接近。它更少把白天耗在处决好人上，因此更早结束比赛：26 场胜利中，18 场在第 3 天或之前拿到。

逐席位的信念探测也显示出这个区别。在第 1 天提名时，Astra 好人席位给邪恶玩家估计的平均邪恶概率，比给好人估计的高 0.19；Sol 6.1 的差值是 0.16，其他好人模型平均为 0.06。这里已排除对阵 Astra 邪恶阵营的比赛。单个席位的判断优势，主要由这两个 OpenAI 模型共同占据；Astra 进一步拉开的差距，体现在群体决策中。

固定邪恶阵营模型后，也能看到相似的结果：对 Opus 5.5，Astra 好人 9 战全胜，其他模型的好人胜率为 64%；对 Sol 6.1 是 4 战全胜，其他模型为 45%；对 GPT-5.6 Sol 是 5 战全胜，其他模型为 63%；对 Fable 5.1 是 2 战全胜，其他模型为 60%。这些比较的样本都不大。

先问身份，再给线索

在 Astra 第 1 天的好人私聊中，要求对方先报身份、自己再分享信息的对话占 6.3%。多数模型只有 0.6%—2.6%，Sol 6.1 为 4.2%，Opus 4.7 为 4.5%。之后，玩家还会把信息交换的先后顺序公开说清楚，避免邪恶玩家听过线索后，再挑一个刚好吻合的身份来冒充。

在 v17/f3b-astra-v-opus55/game-02 第 1 天的一次私聊里，好人先说明，自己掌握一条涉及对方的首夜信息，然后要求对方先报身份，自己才会给出细节。

开头 Blair 与 Alex 的例子来自 v17/f3-astra-v-opus55/game-02。Blair 的真实身份是厨师；Alex 冒充洗衣妇，而 Blair 已经先把身份告诉过他。Blair 因而公开说明，Alex 给出的吻合结果不能独立证明 Alex 的身份。Alex 实际是小恶魔，当天就被处决。

这种核对不限于私聊。Astra 好人的公开发言中，约 9% 提到了证词时序，7% 提到了信息是否独立，5% 明确指出某项证据不能证明或确认某事。对照群体的相应范围分别为 1%—6%、0%—6% 和 1%—2%，报告据此认为，Astra 公开检查证据来历的频率约为多数模型的两倍。

两份对得上的说法，仍可能来自同一个骗局

v17/f3-fable51-v-astra 有一组相同发牌、双方交换阵营的对照局。调查员的线索把嫌疑指向 Casey 或 Alex：Casey 实际是送葬者，Alex 实际是投毒者，却冒充镇长，还有一份吻合的洗衣妇说法替他作证。

Astra 担任好人的 game-01-swap 没有接受这份互相印证。Harper 在第 1 天指出，镇长可能本来就在邪恶阵营可用来冒充的身份中；这样一来，吻合 Gray 中毒后得到的信息，并不需要碰巧猜中。Indigo 又追问 Alex，究竟是谁告诉了他 Casey 的身份，不能拿“大家都在传身份”来代替具体来源。好人最终先处决了投毒者 Alex，第 2 天又处决了小恶魔。

交换阵营后，在 Fable 5.1 担任好人的 game-01 中，面对同一组身份组合，镇民接受了“两次巧合”的说法，以 10 比 1 处决了真正的送葬者，并在第 5 天输掉比赛。这里的邪恶阵营由 Astra 扮演，因此这组对照同时受到了双方模型的影响。

报告还观察到一种跨局做法：两个嫌疑人之间，优先处决声称自己一次性能力已经用完的人，留下还能持续提供信息、可以继续检验的角色。Blair 在 f3-astra-v-opus55/game-02 第 1 天就明确用这个区别说明自己的选择。

一项错误前提，会连着暴露提出它的人

在 v17/f3-astra-v-opus55/game-03 第 1 天，小恶魔 Gray 冒充调查员，声称 Harper 是两条线索中唯一重合的名字。Drew 立即纠正：Harper 根本没有同时出现在两条线索里；把他称为交集，会给这次处决附加一份本来不存在的证据。

Alex 同时追问 Gray，是谁告诉他 Alex 具有夜间获取信息的能力。这涉及 Gray 不该轻易知道的信息。Gray 当天以 9 比 3 被处决；邪恶阵营随后试图通过自我提名调换处决目标，只拿到 3 票，全部来自邪恶玩家。

前面的 game-02 也出现过类似的拉票。一名真实的男爵承认自己说谎，试图以此救下小恶魔。Drew 拒绝让这个承认替整桌人决定处决目标。随后，小恶魔又反悔，不愿按自己先前的承诺投自己一票，Harper 和 Kit 都指出了这个变化。

私聊记录进入公开讨论

Astra 好人私聊的数量相对较多：平均每条公开发言对应 2.5 次私聊，Sol 6.1 为 2.3 次，多数其他模型为 1.0—1.6 次。玩家还会在公开场合引用私聊，指出一个人的私下立场与公开立场互相矛盾。

例如，f3-fable51-v-astra/game-01-swap 第 2 天，Blair 提名小恶魔时指出，Ellis 私下已经接受 Alex 是投毒者的解释，之后却转头指控 Frankie。另一次在 v16/f2-gpt56sol-v-astra/game-01-swap 第 6 天，Casey 指出，Alex 私下先怀疑 Kit 是小恶魔，后来又改为 Gray；Alex 自己也承认，期间没有获得新证据。

主动安排身份测试，也计算最后几票

报告中的“贞洁者测试”，是让一名已用完信息能力的好人按约定提名贞洁者，利用角色机制验证身份。26 局中，这类测试成功触发了 6 次，每次确认两名玩家。玩家也会事先安排送葬者的验尸信息，并把猎手的一次射击留给那些直接处决风险很高的身份声明。

在 f2-fable5-v-astra/game-01-swap 第 1 天，一名玩家声称自己是圣徒——处决真正的圣徒会使好人立即落败。猎手据此选择用射击检验 Blair 的声明，最终命中了小恶魔。

到了最后 3 人阶段，整桌玩家会一起计算票数，并检查每一种解释能否说通邪恶阵营此前的行为。死者手里剩余的一次幽灵票也会参与计算。

v16/f2-astra-v-sol61/game-01 第 5 天，Casey 是小恶魔，Gray 是间谍，Drew 是好人。已经死亡的镇长 Jules 算过票后指出，如果自己与 Drew、Frankie、Blair 都投 Casey，那么改提 Drew 的反制最多只能拿到 4 票；按报告所引的判断，这个反制无法改变处决目标，只会走向输局。

此后，小恶魔临时声称自己是镇长，间谍临时声称自己是猎手，镇民仍没有改票：支持替代处决目标的两项身份说法已经一起站不住脚。好人获胜。同一随机种子、双方交换阵营后，Sol 6.1 的好人在最后 3 人时说服自己处决了真正的镇长，最终落败。

另一局 f2-gpt56sol-v-astra/game-01-swap 进行到第 6 天时，小恶魔与两名已死亡的爪牙提出 3 套看似说得通的解释，把嫌疑引向镇长 Kit。好人逐一核对这些解释的代价。Drew 指出，它们仍然需要假定：爪牙曾被蓄意夜杀，或者邪恶玩家曾长期推动处决自家小恶魔。最后，投给小恶魔的 9 张赞成票全部来自好人。

这份优势分散在整桌玩家中

报告把成功提名邪恶玩家的人按身份拆开：镇长 5 次、僧侣 4 次、酒鬼 4 次、士兵 3 次、调查员 2 次、守鸦人 2 次，另外还有其他角色。持续获取信息的角色并没有包办判断。

每名被处决的邪恶玩家，平均收到 6.6 张来自好人的赞成票，通常包括几乎所有仍活着的好人，以及手握幽灵票的死者。纠正推理错误的发言，也同样经常出自外来者和没有主动信息能力的角色；报告举出的例子包括隐士 Drew、圣徒 Frankie 和士兵 Blair。

个别席位仍会短暂接受一套精巧但错误的解释。例如 f2-astra-v-sol61 第 5 天的 Indigo 就出现过这种动摇，但其他玩家会把讨论拉回来。

运气、对手和小样本，都要算进去

猎手直接决定了 26 场胜利中的 6 场，其中 3 场发生在第 1 天。Astra 的猎手共射击 10 次，6 次命中恶魔；其他模型的命中率在 0%—40% 之间。Astra 的选择有推理依据，例如依据共情者得到的“相邻的两名存活玩家中有一名邪恶玩家”的信息、处决圣徒声明的风险，或投票团的组成。但每一次通常仍是二选一或三选一的判断。同样的发牌下，Fable 5 和 Opus 4.7 没有开枪，Sol 6.1 则射错了人。

错误信息也可能带来好运。在 f3-astra-v-opus55/game-02，一名中毒的调查员拿到了一组错误的男爵候选人，其中恰好包括真正的小恶魔。

对手差异同样不能略去。Astra 的好人从未对阵 Astra 的邪恶阵营；后者面对其他模型的好人时胜率为 62%，也就是对面的好人只赢了 38%。交换阵营的比较，会把 Astra 更会识破谎言与它更会说谎这两项影响混在一起。Opus 5.5 扮演邪恶阵营时，也犯过自己的错误，例如男爵主动招供，或小恶魔过度推动某个错误论点。即使排除 Astra 邪恶阵营，其他好人队伍面对其余对手的胜率仍只有 64%。

样本量也有限：对大多数较旧模型，只有 1—2 局。26 战全胜主要由对 Opus 5.5 的 9 局、对 GPT-5.6 Sol 的 5 局、对 Sol 6.1 的 4 局构成。

报告最后估计，运气和对手因素能解释其中一些单局胜利，可能涉及 26 场中的 6—8 场；原文对此保留了“可能”的限定。它仍把 76% 的处决命中邪恶玩家、对好人被提名者只投出 12% 的赞成票、每局约处决 0.6 名好人，视为明显领先其他模型的能力表现。这是项目报告对这一批游戏的解释，本文没有独立重跑对局，也不把 26/26 外推为真实玩家环境或其他任务中的必胜能力。

来源范围：完整转写项目公开的 detection.md，该报告没有内嵌图片、视频或音频；文中的话语均来自报告摘引的模型对局记录，中文使用转述，不冒充逐字中文引语。文件没有单列执笔者；所属报告组自称由六路并行阅读分析组成，因此本文归属为 Clocktower 项目报告，不声称所有文字都是 Patel 亲笔。它也不是 Patel 后来的配套文章 或视频的全文转写，未据此声称看过视频或验证播放。

原报告与本中文转写遵循项目 MIT 许可，版权 © 2026 Dwarkesh Patel；本页原文区同时保留完整许可声明。《血染钟楼》及相关角色的商标属于 The Pandemonium Institute，本评测是非官方爱好者项目，未获其背书。

原文

Why GPT-6 Astra wins every game as good (detection)

Why GPT-6 Astra wins every game as good (detection)

Read for this: 26 Astra-good games (v16 f2, v17 f3/f3b), with close reads of the pairs against sol61, gpt56sol, opus55 and fable51 and of the long games. Numbers come from events.jsonl/probes.jsonl across all 372 games.

The answer

Astra's good team doesn't find the Demon much faster than anyone else. It almost never executes its own players, and it won't be talked out of a correct execution. It does this as a whole table, not through one star seat. Three habits do most of the work:

1. It tracks where each claim came from. Who said what to whom, and whether before or after they heard the result.
2. It corrects false premises on the spot. It says "that doesn't prove X" every time someone overstates.
3. It treats inconsistency as evidence. A player who reverses a stance or keeps swapping explanations gets noted.

It also gets real luck from Slayer shots (6 of 26 wins), and its opponents are never Astra-as-evil, the strongest deceiver in the eval.

• Astra executes evil about as often per game as everyone else (1.9). The difference is that it doesn't waste days on good players, so games end early: 18 of 26 wins came by day 3.
• On the probes, its individual reads are better than the field's but close to Sol 6.1's. At day-1 nominations, an Astra good seat gives evil players a mean P(evil) 0.19 higher than good players. Sol 6.1's gap is 0.16 and the other good models average 0.06 (games against Astra-evil excluded). The sharpest single-seat edge is shared by the two OpenAI Sol/Astra models. Astra's further edge is in group decisions.
• Against the same evil model, Astra good beats the field: vs Opus 5.5 9/9 (others 64%), vs Sol 6.1 4/4 (others 45%), vs GPT-5.6 Sol 5/5 (others 63%), vs Fable 5.1 2/2 (others 60%).

Mechanisms, with examples

1. Ask for the claim before giving the info. Day-1 good whispers that ask "give me your claim before I share" make up 6.3% of Astra's, against 0.6–2.6% for most models (Sol 6.1 4.2%, Opus 4.7 4.5%). Townsfolk then report the order publicly. This defeats evil's standard trick of claiming whatever role the info role just revealed.

• "I have first-night information involving you. Would you give me your role claim before I share the details?" (v17/f3b-astra-v-opus55/game-02, D1 whisper)
• "I told Alex my role before he gave me his Washerwoman claim, so his matching result does not independently confirm him." (Blair/Chef, v17/f3-astra-v-opus55/game-02, D1). Alex was the Imp and was executed that day.
• In public talk, Astra good seats mention claim timing or independence about twice as often as most models (timing 9% of public lines, "independent" 7%, "doesn't prove/confirm" 5%; field 1–6%, 0–6% and 1–2%).

2. Refuse the free "confirmation" an evil bluff gets. Same deal, opposite outcomes (v17/f3-fable51-v-astra):

• With Astra good (game-01-swap), the Investigator pair was Casey (Undertaker) or Alex (Poisoner, claiming Mayor and backed by a matching Washerwoman). The table took the Washerwoman match apart: "an evil player could have Mayor as a bluff, so matching Gray's poisoned information wouldn't require a lucky guess" (Harper, D1). It also chased a slip: "Alex, who specifically told you Casey's role? 'Claims were bouncing around' isn't a source" (Indigo, D1). It executed Alex, the Poisoner, then the Imp on D2.
• With Fable 5.1 good (game-01, Astra's evil, the identical pair), the table bought "two coincidences" and executed the real Undertaker 10–1. It lost on D5.
• A general rule shows up across games: between two suspects, execute the spent claim and keep the ongoing information role alive ("a spent claim versus an ongoing role we can test", Blair, f3-astra-v-opus55/game-02 D1).

3. Correct bad reasoning, then read the person who keeps making it. Evil's arguments get checked line by line, and the overpusher becomes the target.

• v17/f3-astra-v-opus55/game-03 D1: the Imp (Gray, claiming Investigator) argued that Harper was "the only name in both pings". Drew replied: "Harper isn't a name in both pings… Calling Harper the overlap gives that execution weight it hasn't earned." Alex caught a knowledge slip: "Gray, who told you I have information to work overnight?" Gray was executed 9–3 on day 1, and the evil self-nomination trade that followed got 3 votes, all evil.
• The same game-02 table rejected a Baron's real confession offered to save the Imp: "Indigo doesn't get to choose our execution by admitting to a lie" (Drew). It then caught the Imp reversing a promised self-vote (Harper and Kit, D1).

4. Private talk becomes public evidence. Astra good seats whisper more than most (2.5 whispers per public line; Sol 6.1 2.3, most others 1.0–1.6) and quote whispers publicly when someone's private stance contradicts their public one.

• "Ellis privately accepted Alex as Poisoner, then switched to accusing Frankie…" (Blair, nominating the Imp, f3-fable51-v-astra/game-01-swap D2).
• "Alex privately moved from Kit-Imp… to Gray-Imp… Alex acknowledged there was no new evidence." (Casey, v16/f2-gpt56sol-v-astra/game-01-swap D6).

5. Use test mechanics on purpose. The Virgin test is an agreed nomination of the Virgin by a spent info role: it fired 6 times in 26 games, each one confirming two players. Undertaker reads get planned, and Slayer shots are aimed at claims that would be dangerous to execute. "Blair's Saint claim is risky to test by execution… I'm taking my shot at Blair" hit the Imp (f2-fable5-v-astra/game-01-swap D1).

6. Final three: count the votes and judge which worlds evil's behaviour can explain. In the deciding games the whole table works the problem, including the dead with their ghost votes.

• v16/f2-astra-v-sol61/game-01 D5 (Casey the Imp, Gray the Spy, Drew good). The good side counted the votes: "If Drew, Frankie, Blair and I vote Casey, a Drew counter can reach only four… That forces a loss rather than changing the execution" (Jules, dead Mayor). They then ignored the Imp's last-second Mayor claim and the Spy's last-second Slayer claim: "Both claimed sources behind the alternative executions have collapsed together." Good won. On the same seed with Sol 6.1 good (game-01-swap), the town talked itself into executing the real Mayor at three and lost.
• f2-gpt56sol-v-astra/game-01-swap D6: the Imp and two dead Minions offered three "clean" worlds that pointed at Kit (Mayor). Good seats priced each one: "they still need either a deliberate Minion night-kill or prolonged campaigning against their own Imp" (Drew). The vote was 9 yes on the Imp, all of them good.

One seat or the whole team?

The whole team.

• The nominators of executed evil players are spread across roles. Mayor 5, Monk 4, Drunk 4, Soldier 3, Investigator 2, Ravenkeeper 2 and others. Info roles don't do the work alone.
• On evil nominees, almost every living good seat plus the ghosts votes yes (6.6 good yes votes per executed evil).
• Corrections come from Outsiders and blank roles as often as from info roles (Drew the Recluse, Frankie the Saint, Blair the Soldier in the examples above).
• The weak spots are individual seats briefly swayed by a clever world (Indigo in f2-astra-v-sol61 D5), and the table pulls them back.

Luck and confounds (honest accounting)

• The Slayer decided 6 of 26 wins (3 on day 1). Astra's Slayers hit the Demon on 6 of 10 shots; other models hit on 0–40%. The shots were reasoned (an Empath 1 neighbour, a Saint claim too risky to execute, a voting bloc), but each was still roughly a 1-in-2 or 1-in-3 guess. On the same deals Fable 5 and Opus 4.7 never fired, and Sol 6.1 fired at the wrong player.
• Good fortune from bad info. In f3-astra-v-opus55/game-02, a poisoned Investigator's false Baron pair happened to name the Imp.
• Weaker opposition. Astra-good never faces Astra-evil, which beats the field's good teams 62% of the time (they win 38% against it). The swap comparisons therefore mix "Astra detects better" with "Astra deceives better". Opus 5.5 as evil also made its own mistakes (a Baron confession, an Imp who overpushed). Against the field excluding Astra-evil, good teams still win only 64%.
• Small samples. Only 1–2 games against most older models. The 26/26 record is mostly 9 games against Opus 5.5, 5 against GPT-5.6 Sol and 4 against Sol 6.1.
• Bottom line: luck and opposition explain some single wins (perhaps 6–8 of the 26). The underlying rates (76% of executions on evil, 12% yes votes on good nominees, 0.6 good players executed per game) are well clear of every other model. Those rates are skill.

</cursor_untrusted_data_1337>

Project license and scope notice

MIT License

Copyright (c) 2026 Dwarkesh Patel

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Blood on the Clocktower, Trouble Brewing and their characters are trademarks of The Pandemonium Institute. This is an unofficial fan project, not affiliated with or endorsed by The Pandemonium Institute; the license above covers this project's own code and documentation only.

</cursor_untrusted_data_1337>

原文链接：https://github.com/dwarkeshsp/clocktower/blob/52155afe38a696283756f142bdb1565e07951ae1/docs/findings/why-astra-wins/detection.md

相关链接：https://x.com/polynoamial/status/2118514804189233568

乔木改写

Sardor 在 11 月 6 日介绍了自己做的开源录屏工具 CaptureFlow：人在 Chrome 里录屏，结束后把链接发给别人；AI Agent 也可以通过 MCP 录制演示。它支持 Docker 自托管。原帖所链接的仓库 README 进一步写明，录制过程中就会上传，停止录制时分享链接已经准备好了。

能录什么，在哪里使用

CaptureFlow 可以录制浏览器标签页、窗口或整个屏幕，并加入摄像头悬浮画面和麦克风声音。观看者可以在视频里发表情、留言，录制者能看到谁看过。截图也可以加箭头、文字和模糊遮挡，再用同样的链接方式分享。

团队使用工作区管理内容，可设为公开、仅工作区成员可见或私密。托管服务按整个团队的存储量收费，不按席位收费；付费方案提供 100 GB、500 GB 或 1 TB 存储空间。作者说明，全部功能都包含在开源构建中，也可以部署到自己的 Cloudflare 账户，并把登录范围限制在公司内。

Chrome 扩展已经在 Chrome Web Store 上架。macOS 应用目前可以从源码构建，签名发行版仍是即将推出的状态；网页管理台和观看页支持现代浏览器。托管版可以先注册免费账户，再安装扩展，具体付费额度见价格页。

在原帖评论里，@progerallworld 提出的用途是让 Agent 重现 Bug、录下步骤，再附上视频。Sardor 回复说，这正是他加入 MCP 时考虑的场景。随后他又补充，也可以让 Agent 为 GitHub Pull Request 录制演示，并附到 PR 描述里；安装扩展后，审阅者可以观看演示来理解改动。这是作者对用途的说明，本轮没有连接 MCP 或试录视频。

用 Docker 部署

README 给出的 Docker 启动方式很短：先执行 cp .env.example .env，设置 APP_URL，再执行 docker compose up -d。

数据都保存在 ./data：SQLite 数据库、磁盘上的录屏文件，以及第一次启动时生成的密钥。这条部署路线不需要 Cloudflare 账户。生成转写可以使用兼容 OpenAI 接口的 Whisper 服务；Compose 也提供 --profile whisper 来运行一个这样的服务。完整说明在 Docker 部署文档。

部署到自己的 Cloudflare 账户

另一条路线使用 Cloudflare Workers、D1 和 R2。仓库提供了一键部署入口，也列出了手动步骤。

先运行 pnpm --filter @captureflow/web exec wrangler d1 create captureflow 创建数据库，再运行 pnpm --filter @captureflow/web exec wrangler r2 bucket create captureflow-recordings 创建存储桶。把数据库 ID 填入 apps/web/wrangler.jsonc 后，依次执行 pnpm --filter @captureflow/web db:apply:remote 和 pnpm --filter @captureflow/web cf:deploy。

DEPLOY.md 说明了密钥、登录服务、自定义域名，以及如何让扩展和桌面应用连接自己的实例。部署在自己的域名下会作为自托管实例运行；如果要做成公司内部工具，还需要按照配置文档设置 ALLOWED_EMAIL_DOMAINS。仓库也链接了逐步自托管教程和代配置服务。

本地开发和检查

开发环境需要 Node.js 24.13 或更新版本，以及 pnpm 10；README 提示用 corepack enable 启用。取得仓库并进入目录后，先执行 pnpm install。

完整的取代码命令是 git clone git@github.com:sardorml/captureflow.git，接着执行 cd captureflow。pnpm web 启动网页管理台和官网，地址为 http://localhost:3032；pnpm admin 启动实例管理后台，地址为 http://localhost:3034。pnpm extension 会在一个新的 Chrome 配置环境中启动浏览器扩展；macOS 录屏应用的开发命令是 pnpm --filter @captureflow/desktop dev。

仓库列出的检查命令分别是 pnpm typecheck、pnpm --filter @captureflow/web test、pnpm --filter @captureflow/extension test 和 pnpm format。这里保留的是作者提供的命令，没有把文档中的检查项写成本站已经跑过的结果。

仓库如何拆分

应用放在 apps/，共享包放在 packages/。依赖方向有明确约束：应用可以依赖共享包，共享包不能反向依赖应用，应用之间也不能互相依赖。

• apps/web 使用 Next.js 16，包含官网、博客、管理台、观看页和 API，通过 OpenNext 部署到 Cloudflare Workers。
• apps/admin 是运行在独立主机名下的实例管理 Worker，与主应用绑定同一组 D1 和 R2。
• apps/mailer 是产品邮件 Worker，通过 Resend 发送管理员编写的邮件活动，并处理退订。
• apps/extension 是基于 WXT、MV3 的浏览器录屏扩展，包含面板、离屏采集和页面内控制。
• apps/desktop 是基于 Electron 的 macOS 录屏应用；apps/docs 是 VitePress 文档站。
• packages/engine 是桌面端与扩展共用的采集引擎，包含 macOS 辅助程序、录制协议和封装处理，采用 MIT 许可。
• packages/admin 为管理后台提供认证与数据逻辑；packages/mail 负责邮件活动、渲染、退订令牌、Resend 投递和事务邮件。
• packages/quota 管理存储额度、限制与工作区；packages/shared 放各应用共享的类型和常量；packages/ui 放主题令牌，以及没有对应 HeroUI 组件的 React 组件。

macOS 采集辅助程序以编译后的二进制文件形式存入仓库。修改 Swift 源码后，需要运行 packages/engine/scripts/build-native.sh 重新构建；加上 --universal 可以同时支持 arm64 和 x86_64。

文档、贡献和许可

项目的官网、文档、自托管总览、博客、FAQ、安全说明、发行记录和 Discord 均由 README 提供入口。

作者欢迎 Bug 报告、想法和 Pull Request，可以到仓库 Issues或功能建议页提交。贡献前需要阅读 CONVENTIONS.md。发现安全问题应按照安全页说明私下报告，不要发到公开 Issue。

项目采用 AGPL-3.0-only，例外是可复用的采集引擎 packages/engine，它采用 MIT 许可。桌面应用与浏览器扩展的组合构建，整体仍适用 AGPL-3.0-only。

本文完整改写上述作者帖文、已取得的作者回复及固定版本 README；README 最近修改于 11 月 4 日，不能把 11 月 6 日的介绍帖当成当天发布新版本。文档文字按上游 AGPL-3.0-only 许可使用，许可全文随站点提供；本站中文不是官方文档。

原文附带媒体

来源附带的产品概览配图；未核验与中文具体段落的一一对应关系，集中保留于文末。

原文

CaptureFlow — author announcement, replies and complete README

Author announcement — Sardor (@sardorml)

Source: https://x.com/sardorml/status/2118616954085024133

I built CaptureFlow, an open-source screen recorder for you and your AI agents.

Record your screen in Chrome and get a link you can share with anyone. Or let your agent record the demo over #MCP.

Self-host it with Docker.

https://t.co/GwIkaJEH9C

#opensource #buildinpublic https://t.co/dlOESaA0wJ

Public reply context — @progerallworld (not the author)

Source: https://x.com/progerallworld/status/2118618493977313326

@sardorml This could be useful for bug reports too: agent reproduces the issue, records the steps, attaches the clip. Much easier to review than “I clicked around and it broke” 😅

Author reply — Sardor (@sardorml)

Source: https://x.com/sardorml/status/2118619339041755560

@progerallworld Yes, definitely. That's the use case i had in mind when i added the MCP.

Author reply — Sardor (@sardorml)

Source: https://x.com/sardorml/status/2118621775856865775

@progerallworld You can also ask it record a demo for your github pull request and it will attach it to the pr description. With extension installed reviewers can watch the demo to get a better idea of the changes.

Linked README — complete text, fixed source snapshot

Source: https://github.com/sardorml/captureflow/blob/c5792345e7d5e47084c5e829b3f11c98a432c859/README.md

CaptureFlow

Open-source screen recorder with instant share links, self-hostable on your own Cloudflare account.

Website · Docs · Blog · Report Bug · Request Feature

✨ Features

• 🔗 The share link is ready the moment you stop: the upload runs while you record.
• 🎥 Record a tab, a window or your whole screen, with camera bubble and mic.
• 💬 Viewers react with emoji and comment right on the video, and you see who watched.
• 🖼 Annotated screenshots with arrows, text and blur, shared by the same kind of link.
• 👥 Workspaces for your team, with public, workspace-only or private visibility.
• 💸 Priced by storage for the whole team, never per seat.
• ☁️ Self-host on your own Cloudflare account, with sign-in limited to your company.
• 🔓 Every feature ships in the open-source build.

🖥 Where it runs

📦 Get started

• Hosted: create a free account and install the Chrome extension. Paid plans add 100 GB, 500 GB or 1 TB for the whole team; see pricing.
• Self-hosted: docker compose up on your own server, or run it on your own Cloudflare account (Workers, D1 and R2). Follow the step-by-step guide, or let us set it up for you.

☁️ Self-hosting

Docker

cp .env.example .env    # set APP_URL
docker compose up -d

Everything lives in ./data: a SQLite database, the recordings on disk, and secrets generated on first start. You don't need a Cloudflare account. Transcripts can use any OpenAI-compatible Whisper server, including one compose runs for you with --profile whisper. See the Docker guide.

Cloudflare

Or manually:

pnpm --filter @captureflow/web exec wrangler d1 create captureflow
pnpm --filter @captureflow/web exec wrangler r2 bucket create captureflow-recordings
# paste the database id into apps/web/wrangler.jsonc, then:
pnpm --filter @captureflow/web db:apply:remote
pnpm --filter @captureflow/web cf:deploy

DEPLOY.md is the full runbook: secrets, sign-in providers, custom domains, and pointing the extension and desktop app at your own instance. A deployment on your own domain runs as a self-hosted instance; to make it a private company tool, set ALLOWED_EMAIL_DOMAINS as described in the configuration docs.

⌨️ Development

Requires Node.js 24.13 or newer and pnpm 10 (corepack enable).

git clone git@github.com:sardorml/captureflow.git
cd captureflow
pnpm install
pnpm web                                  # dashboard and landing on http://localhost:3032
pnpm admin                                # instance admin on http://localhost:3034
pnpm extension                            # browser extension, in a fresh Chrome profile
pnpm --filter @captureflow/desktop dev    # macOS recorder

Checks:

pnpm typecheck
pnpm --filter @captureflow/web test
pnpm --filter @captureflow/extension test
pnpm format

apps/
  web/         Next.js 16 landing, blog, dashboard, viewers and API → Cloudflare Workers (OpenNext)
  admin/       Instance-admin Worker on its own host, bound to the same D1 and R2
  mailer/      Product-email Worker: sends admin-written campaigns via Resend, handles unsubscribe
  extension/   Browser recorder (WXT, MV3): panel, offscreen capture, in-page controls
  desktop/     Electron screen recorder (macOS)
  docs/        VitePress documentation site
packages/
  engine/      Capture engine shared by desktop and extension: macOS sidecars, record protocol, mux (MIT)
  admin/       Admin auth and data logic behind apps/admin
  mail/        Campaigns, email rendering, unsubscribe tokens, Resend delivery, transactional sends
  quota/       Storage quota, limits and workspace logic
  shared/      Types and constants shared across the apps
  ui/          Theme tokens and the React components with no HeroUI equivalent

🔗 Links

• Website
• Documentation
• Self-hosting docs
• Blog
• Pricing
• FAQ
• Security
• Releases
• Discord

🤝 Contributing

Bug reports, ideas and pull requests are all welcome. Open an issue, suggest a feature, or send a pull request. Read CONVENTIONS.md first: it's the standard every change follows.

Found a security issue? Please report it privately as described on the security page, not in a public issue.

📄 License

AGPL-3.0-only, with one exception: the reusable capture engine in packages/engine is MIT-licensed. Combined builds (the desktop app and the browser extension) are AGPL-3.0-only as a whole.

原文链接：https://x.com/sardorml/status/2118616954085024133

相关链接：https://github.com/sardorml/captureflow/blob/c5792345e7d5e47084c5e829b3f11c98a432c859/README.md

乔木改写

Nelson Chu 在 11 月 6 日介绍了自己做的 checkwash：一个在本地读取 Git diff、检查测试是否被改弱的开源工具。它不调用大模型，分析时不联网，也不会执行被检查的代码。测试通过并不一定意味着检查仍然有效，例如把 assert total == 105.3 改成 assert total > 0，许多错误的金额也会被放行。checkwash 会标记这类改动，包括编程 Agent 写出的改动，交给人审阅。

下面的使用方式和限制来自作者仓库的完整 README。文档推荐的 CLI 和 Action 均为 v0.11.0，说明它是公开发行文件已经核验的版本；本轮阅读了文档，没有安装或执行 checkwash，也没有把文档自述当成本站的独立测试结果。

接入 Claude Code：停止时检查，部分编辑前询问

环境需要 Python 3.11 或更新版本、Git 和 Claude Code。在仓库根目录执行 pipx install checkwash==0.11.0，再执行 checkwash hook install --agent claude-code --local。安装器把钩子写入本机的 .claude/settings.local.json。

Stop 钩子在 Claude 准备停止时，将工作区与 HEAD 比较。如果发现达到阻断条件的问题，就阻止停止，并把发现交给 Claude。同一配置文件中的 SessionStart 钩子记录每个会话的起点，因此 Stop 钩子也会检查该会话已经提交的改动。

从 0.11.0 起，本地安装还加入 PreToolUse 询问。当通过 Edit、Write 或 MultiEdit 编辑 Python 测试文件，会使下列五种弱化规则中的某项仍处于 high 严重性时，它会在编辑前询问：ASSERT_WEAKENED、ASSERT_REMOVED、TEST_DISABLED、TOLERANCE_LOOSENED、BROAD_EXCEPT_ADDED。在 claude -p 模式下没有人可以回答，这类编辑会被拒绝。README 同时明确，交互式询问尚未在任何模式下验证。

0.11.0 默认采用本地安装；--shared 则写入 .claude/settings.json，不加入 PreToolUse 询问。这些都是作者一侧的检查：通过 Bash 做的编辑不会经过这道询问。Stop 钩子无法分析改动时，通常只报告情况、不阻止停止，工作区改动文件超出大小限制是下文说明的例外。它们都不能阻止别人合并 PR；真正的合并门禁需要 CI 中的必需状态检查。钩子的已知限制另见 0.11.0 已知问题。

先在熟悉的改动上试用

只使用 CLI，需要 Python 3.11 或更新版本和 Git。作者建议先下载单文件程序，运行离线示例。

Windows PowerShell，包括 5.1，下载命令是 curl.exe -LO https://github.com/taipei49314/checkwash/releases/download/v0.11.0/checkwash.pyz。随后执行 python checkwash.pyz --version 和 python checkwash.pyz demo。PowerShell 5.1 会把 curl 当成 Invoke-WebRequest 的别名，因此这里特意使用 curl.exe。

macOS、Linux 或 Git Bash 的下载命令是 curl -LO https://github.com/taipei49314/checkwash/releases/download/v0.11.0/checkwash.pyz，之后运行相同的版本与 demo 命令。也可以用浏览器下载该文件。

进入至少有两次提交的仓库根目录，执行 python checkwash.pyz check HEAD~1..HEAD，检查最近一次提交。未提交的改动使用 python checkwash.pyz check。评审分支时，python checkwash.pyz check BASE...HEAD 从共同祖先开始比较，BASE..HEAD 则直接比较两个指定快照。

从 v0.8.0 起，在子目录运行 check 也会从 Git 根目录分析整个仓库，并在标准错误输出中说明。更早版本需要在仓库根目录运行。JSON、SARIF 输出和更多用法，见嵌入指南与稳定性契约。

CI 必须设成必需检查，才会阻止合并

只安装工具或添加工作流，不会强制执行检查结论。需要把 checkwash 状态上下文设为仓库分支规则中的必需检查，并保持这个任务无条件运行。

README 给出的 .github/workflows/checkwash.yml 在 pull_request 时运行，权限仅为 contents: read，任务名为 checkwash，执行环境为 ubuntu-latest。它固定了三个 Action 的提交：

• actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1，标注 v7.0.1，设置 fetch-depth: 0 和 persist-credentials: false。
• actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97，标注 v7.0.0，Python 版本为 3.12。
• taipei49314/checkwash/action@257150319fca15235185fc629d8b9c21c17166ca，标注 v0.11.0。

工作流运行后，在 GitHub 的 Settings → Rules → Rulesets 中要求 checkwash 状态检查。完整 YAML 保留在本页原文中，也可以直接打开固定版本 README。

如果是仓库管理员，且已取得项目提供的规则文件，还可以运行 gh api repos/OWNER/REPO/rulesets --method POST --input action/required-ruleset.json 添加规则集。这是新增规则，不会替换既有规则。checkwash doctor 可以检查本地工作流，但不能验证线上分支保护是否生效。

版本顺序也有要求。doctor 0.11.0 接受上面推荐的提交和 v0.10.0 的提交；0.10.0 的 doctor 只接受它内置的 v0.9.0 Action 提交，会把上面的 SHA 报成不支持，显示“workflow analysis incomplete”并以 1 退出。因此应先升级 CLI 和本地钩子，再升级 Action 固定提交。

怎样理解一次阻断

退出码 0 表示在当前配置下没有需要阻断的发现，正常测试和评审仍要继续；退出码 1 表示需要阅读发现与 diff，可能是验证被削弱，也可能是误报；退出码 2 表示输入或分析出错，必须先解决错误，才能依赖结果。默认阻断阈值是 high，所以报告中看得见的 warn 仍然可能通过。

每条发现会给出规则、严重性、文件和测试名。例如，前面的金额断言若是 tests/test_billing.py 中 test_invoice_total 的全部 diff，报告会显示 ASSERT_WEAKENED、HIGH，断言强度从 EXACT_VALUE(90) 变成 BOUND(40)，原因是 NO_PROD_CHANGE_IN_DIFF，且没有应用降级条件，并列出修改前后的断言。

这里的 90 是断言强度等级，不是测试期待的数值。why high 解释严重性为什么升高：这份 diff 没改生产文件，也就没有生产代码变化可以解释为什么要弱化测试。需要机器可读输出时，可用 checkwash check --format json。

如果测试被误改弱，应恢复测试；恢复后失败，就修代码。确有理由放宽测试时，报告会给出包含该发现指纹的 checkwash allow 命令，以及需要提交的豁免账本文件。命令格式为 checkwash allow "<the fingerprint from the report>" --reason "why the weaker test is correct"。

配置和豁免从 diff 的基础版本读取，因此一个改动不能给自己豁免。豁免条目要先经过评审、进入基础版本，才对下一份 diff 生效。自 v0.10.0 起，新生效的 .checkwash 配置或豁免还会与 BASE 侧原有的 .greenwash 对照，避免一次迁移同时批准自己的放宽。每项豁免必须给出理由，默认 90 天过期，最长 180 天；作者建议对 .checkwash/ 和 .greenwash/ 设置代码所有者。

从 0.11.0 起，Stop 钩子会告诉 Claude 不要自行添加豁免，而应向人说明是哪项发现阻断、为什么阻断；PreToolUse 询问也只应在弱化确有必要时批准。具体规则见企业配置说明。

遇到合理改动被阻断，作者要求先查失败账本，再使用“checkwash blocked honest work”模板报告 Issue，附上 diff 和 checkwash check --format json 输出。报告会变成回归样例，账本会署名感谢报告者。

Alpha 阶段仍会漏检，也会误报

checkwash 仍是 Alpha。一次通过不能证明改动正确，也不能证明改动诚实。Python 是主要支持语言，JS/TS 只覆盖有限的测试模式。

一个没有实际效果的改动，例如在测试调用的函数里加 + 0，或只改一行模板、AUTHORS，仍可能让被弱化的 Python 或 JS/TS 测试停留在 warn，从而通过默认阈值。#419 虽已关闭，处理方式是记录残留风险，并未修复。REPAIR_EVIDENCE 只表示同一 diff 内存在相关改动，不证明修复正确；无实效的编辑也可能获得这个标记。

要阻断这类改动，可以把 Action 的 fail-on 设为 warn。此时 checkwash doctor 会报告“workflow analysis incomplete”并以 1 退出，这是文档说明的预期行为，因为它只证明没有 with: 配置的 checkwash 步骤。另一种方式是自己添加 CLI 步骤，执行 checkwash check --task，配合基础版本中的 oracle_freeze 契约；Action 本身没有 task 输入。这两条路线都会额外阻断一些合理改动。

对于本地安装，0.11.0 的 PreToolUse 钩子在那五项弱化规则处于 high 时，会在 Python 测试文件的 Edit、Write、MultiEdit 前询问，不受工作区其他修复证据影响。它自身的边界仍应按已知问题文档理解。

JS/TS 的参数表支持也分版本。v0.8.0 不扫描 test.each、it.each 和 test.for 的表格，表内断言弱化或删除、修改一行数据、删除整张表，都可能通过。v0.9.0 开始把每张这样的表视为一个测试单元，并在能够计数时读取数据行：包括内联数组、标签模板，或同文件中只作为表使用的 const。describe.each 的数据行仍不读取，标题由计算产生的表也不会被视为测试单元，见 #428。

合理重构可能被误报。冻结的历史重构样本中，60 项有 22 项被阻断，占 36.7%。另一次独立的预发行源码回放中，60 项有 4 项被阻断；其中 57 项严格符合限定的案例里，有 2 项被阻断，仍有三个既有样例错误。这些都是经过挑选的样本，不能当成一般误报率。v0.10.0 账本在 D-185 处置后保留了 6/60 的静态源码阻断，其中包括另外两项已知的合理重构误报；这是后来对源码的重新核查，没有改写历史运行结果。

文件过大时，哪些情况会停止分析

变更文件超过 1,000,000 字节就不会被完整读取。图片、数据、锁文件、文档，以及非 Python 的生产源码超出该限制时，会在标准错误输出中报告，也不会获得修复证据。

从 v0.9.0 起，这类未读文件还会出现在 JSON 的 unread 中；SARIF 使用工具执行通知报告；终端报告增加 ! 行，没有其他发现时以“analysis incomplete”开头。Stop 钩子的 JSON 和 --emit-ir 不会列出这些文件名。

如果超限的是检测器必须读取的文件，就以退出码 2 停止，并指出文件。这包括测试、conftest、CI、护栏、快照、JS 设置、运行脚本、测试命令文件，以及分析所需的其他上下文。超限的 .py 模块不论新增、删除、重命名或编辑，也会停止分析；任意层级的 pytest 配置同样如此，包括 pytest.ini、.pytest.ini、pytest.toml、.pytest.toml、tox.ini、setup.cfg 和 pyproject.toml，因为它们可能改变测试导入模块时解析到的对象。根目录 scripts/ 中为分析 pytest 命令而读取的非生成文件，也受同样约束。

Shell shebang 会让文件被视为运行脚本，不受文件名影响；.py 模块和 JS/TS 测试文件则有自己的规则。v0.9.0 起，程序读取超限一侧的前 200 字节，只用于识别文件类型；Git 中的那一侧会以流式读取，并核对对象 ID。如果这些字节显示 shell shebang，就会指出具体哪一侧并停止。没有扩展名的文件——包括 .envrc 这样的点文件——若没有 shebang，例如 README 或 LICENSE，通常按文档搁置，除非被其他规则判定为必读文件。

v0.8.0 不读取超限文件的任何字节，所以那时所有超限、无扩展名文件都会停止分析。v0.9.0 不会流式读取 Git 中超过 64,000,000 字节的一侧；若无扩展名文件有这样大的一侧，仍会停止。带扩展名的文件如果 shebang 只存在于这一超大侧，则仍会被搁置。

超限文件导致停止时，v0.9.0 起 JSON 会在标准输出中给出 {"unread": [...]}，只列拒绝读取的那一侧，而不是完整的 findings 文档；终端和 SARIF 输出仍为空。使用比较范围时，若被拒读的是符号链接指向的 pytest 配置目标，且没有检测器必须读取该目标，JSON 输出也仍为空。

build/、dist/、node_modules/ 等生成产物通常按生产源码搁置，但 Python 模块、pytest 配置、CI 文件、运行脚本，以及 shebang 未知的无扩展名文件仍是例外。基础版本根目录中超限的 poetry.lock 或 uv.lock 会被跳过，并在标准错误输出提示；这次跳过不进入 unread。如果 diff 本身也改了该锁文件，超限的变化侧仍按一般规则列出。

基础版本根目录的 pyproject.toml 或 requirements*.txt 如果超限，则每次运行都会停止，指出文件；终端、JSON、SARIF 都以 2 退出，只在标准错误输出中报文件名。Claude Code Stop 钩子使用 --format hook-json，一般把包括这种情况在内的引擎错误报告为不阻断的“not analysed”，在标准输出中引用错误，并以 0 退出。

还有一个较窄的例外：从 v0.9.0 起，如果停止原因是钩子 diff 改动的文件，并且该文件的工作区一侧超限，钩子就会阻止停止，要求缩小、拆分或还原文件。若只是基础版本一侧超限、基础版本根清单超限，或在停止前已经把填充后的文件提交，仍只给出消息。完整分类见检查对象完整性说明。作者因此建议，在设为必需检查之前，先拿自己的改动试用。

Action 的信任边界和固定提交

直到 v0.7.0，已发布的 Action 都在 PR 检出的目录中运行 python -m pip，随后运行 checkwash。这样，PR 中的模块，例如顶层 pip 包，可能替换安装器和检查器，使 PR 干预对自身的评判。按项目的威胁模型，这适用于它检查的每个 PR。v0.8.0 修复了这一问题，README 推荐固定 v0.11.0 Action。继续使用 v0.7.0 或更早版本时，只能把通过结果当成参考，合并前审阅 PR，并关闭 comment-pr。

开发分支源码可能比已发布 CLI 更新。release/advertised.toml 记录安装命令背后的版本、提交和下载字节。新改动需要自己的候选证据，在发布记录核验之前仍未发行；候选 CI 通过，不代表推荐的 Action 提交已经包含该改动。

固定提交的选法从 v0.11.0 起也有变化。此前推荐的是 tag 指向的提交，可用 git rev-parse 'vX.Y.Z^{commit}' 查看；从 v0.11.0 的 D-194 起，推荐的是发行版的 Action 提交，即 tag 第二个父提交的父提交，可用 git rev-parse 'vX.Y.Z^2^1' 查看，不能直接拿 tag 指向的提交代替。

发行 PR 的最后一笔提交只把 doctor 自身的 Action pin 改为那个 Action 提交，所以 tag 与 Action 提交只有这一行差异。Action 只运行 checkwash check，不会导入 doctor。从 v0.11.0 起，一个发行版的 doctor 接受两个 pin：上一版推荐的，以及本版自己的 Action 提交。

README 还说明，Dependabot、Renovate 按 # vX.Y.Z 注释跟踪更新，会建议使用 tag 提交，而 doctor 会拒绝它；zizmor 的 ref-version-mismatch 审计也可能报告这条注释。这些判断是根据相关工具文档推导的，没有实际运行。Action 在两种提交上执行的检查相同，本身不受影响。推荐的 v0.11.0 Action 包含 2026 年 11 月 4 日发布的引擎；升级 CLI 不会自动升级已有 Action，仍应先升级 CLI 和本地钩子，再更新 Action pin。

安装选项、测量记录和剩余边界

除了固定 PyPI 版本，也可以从 tag 安装：pipx install git+https://github.com/taipei49314/checkwash@v0.11.0。随后用 checkwash check HEAD~1..HEAD 检查最近提交。checkwash demo 或 python checkwash.pyz demo 会离线回放八个真实篡改案例和一个正常修复案例，用来说明已知模式，不是覆盖能力保证。

历史六仓库扫描中，1800 项有 46 项被阻断，占 2.56%；其中 31 项是假阳性，占全部样本的 1.72%，15 项是合理的策略阻断，占 0.83%。这些归档产物使用 v0.3.0 引擎，是 2026 年 9 月 7 日对发行提交的扫描；检测器曾用这份语料调优，所以它不是留出的独立评测结果。

语料中有 24/1800，即 1.33%，记录了不透明的生产改动。这个标记不能证明每项判断都受到了影响，也不能证明每份 diff 都没有被分析。人工评审方式也不统一：三位评审者的一致性研究只覆盖更早的 35 份 diff，并非全部 46 项阻断。单独的 22/60 重构结果是带日期的 v0.3.1 快照，属于不同样本群；一般提交的比例不能预测重构结果。

原始材料入口包括测量说明与源数据、生成结果和失败账本。README 还链接了 v0.11.0 发布核验记录、发行指南、发行历史、断言覆盖、发现格式、贡献、安全报告，以及尚未达到的 1.0 标准。这些是后续查证入口，不代表本轮已经逐份打开或复现实验。

checkwash-corpus 和 smallestlie 保存相关评测工作。包里还带有针对覆盖率、Ruff 和 mypy 配置审查的有限质量预览；冻结的历史字节比较仍未通过，发布本身不能证明它已在自然案例中获验收，或真正形成有效门禁。

该项目采用 Apache-2.0，零运行时依赖，文档列出 22 个检测器与 22461 项测试；后一个数字是测试收集数量，不是本轮执行通过数。

本文完整改写作者原帖及固定版本 README；README 最近修改于 11 月 5 日（中国标准时间），没有用公告时间替换发行日期。README 文字按上游 Apache-2.0 许可使用，许可全文随站点提供；本站中文不是官方文档。

原文

checkwash — author announcement and complete README

Author announcement — Nelson Chu (@kartg123451)

Source: https://x.com/kartg123451/status/2118550260104769565

I built checkwash because a green CI run can still hide a weaker test. It reads a Git diff locally — no LLM, no network — and flags when an edit weakens or removes an assertion, disables a test, or loosens a tolerance.

https://t.co/J7eBvvMKBK

Linked README — complete text, fixed source snapshot

Source: https://github.com/taipei49314/checkwash/blob/0ebd5ad890c252b4546dbd692ac0414fdbc201cc/README.md

checkwash

Check whether a code change weakens your tests.

A test can pass after it stops checking something useful. checkwash reads your Git diff and flags known patterns such as deleted assertions, skipped tests, relaxed expectations, and CI checks that stop failing.

- assert total == 105.3
+ assert total > 0

The new assertion accepts many incorrect totals. checkwash can flag changes like this for review, including changes written by coding agents.

Runs locally. No LLM. No network during analysis. Never executes your code.

The install commands and Action pin below name v0.11.0, the latest release whose public files are verified (release history).

Start in one of three ways, each below: Claude Code hooks, the CLI on a repository, or the GitHub Action as a required check.

Use it with Claude Code

You need Python 3.11+, Git and Claude Code. Install the CLI, then run the hook installer from the root of your repository:

pipx install checkwash==0.11.0
checkwash hook install --agent claude-code --local

This writes checkwash's hooks to .claude/settings.local.json, the machine-local Claude Code settings.

• The Stop hook checks the working tree against HEAD when Claude is about to stop. On a blocking finding it blocks the stop and gives Claude the finding. A SessionStart hook in the same file records where each session starts, so the Stop hook also checks what the session commits.
• From 0.11.0, a local install also adds a PreToolUse ask. When an Edit, Write or MultiEdit of a Python test file would leave a finding of one of five weakening rules (ASSERT_WEAKENED, ASSERT_REMOVED, TEST_DISABLED, TOLERANCE_LOOSENED and BROAD_EXCEPT_ADDED) at high, the hook asks you before the edit is made. In a claude -p run nobody can answer, so such an edit is denied; the interactive prompt is not verified in any mode. From 0.11.0 a local install is the default, and --shared writes .claude/settings.json with no PreToolUse ask.

Both hooks are author-side checks. An edit made through Bash never reaches the ask. When the Stop hook cannot analyse a change, it says so and does not block the stop, except for a changed file over the size limit (see "Know the limits"). Neither hook can stop someone else's merge: the required check under "Use it in CI" is the gate. The 0.11.0 hooks' limits are in the 0.11.0 known issues.

Try it on a repository

You need Python 3.11+ and Git. Download and try the offline examples first.

Windows PowerShell (including 5.1):

curl.exe -LO https://github.com/taipei49314/checkwash/releases/download/v0.11.0/checkwash.pyz
python checkwash.pyz --version
python checkwash.pyz demo

PowerShell 5.1 aliases curl to Invoke-WebRequest; use curl.exe as written. macOS/Linux or Git Bash:

curl -LO https://github.com/taipei49314/checkwash/releases/download/v0.11.0/checkwash.pyz
python checkwash.pyz --version
python checkwash.pyz demo

Then, from the root of a repository with at least two commits:

python checkwash.pyz check HEAD~1..HEAD

This checks your last commit. Start with a change you already understand. You can also download the file in your browser. For uncommitted changes use python checkwash.pyz check. For a branch review, use python checkwash.pyz check BASE...HEAD to compare from the merge base; BASE..HEAD compares the two named snapshots directly.

From a subdirectory, check analyses the whole repository from its Git root and says so in one line on stderr (since v0.8.0, #416). Before v0.8.0, run it from the repository root.

For JSON/SARIF output and more examples, see the embedding guide and the stability contracts.

Use it in CI

To stop a merge, make the checkwash status check required in your repository's branch rules. Installing the tool or adding a workflow alone does not enforce its verdict.

Save this as .github/workflows/checkwash.yml:

# .github/workflows/checkwash.yml
on: [pull_request]

permissions:
  contents: read

jobs:
  checkwash:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          fetch-depth: 0
          persist-credentials: false
      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
        with:
          python-version: "3.12"
      - uses: taipei49314/checkwash/action@257150319fca15235185fc629d8b9c21c17166ca # v0.11.0

After the workflow runs, open Settings → Rules → Rulesets and require the checkwash status context. Keep the job unconditional.

If you administer the repository and have this project's ruleset file, you can instead create the rule with:

gh api repos/OWNER/REPO/rulesets --method POST --input action/required-ruleset.json

This adds a ruleset; it does not replace existing ones. checkwash doctor can inspect the local workflow, but cannot verify live branch protection. doctor 0.11.0 accepts the pin above and v0.10.0's. doctor 0.10.0 accepts only the Action pin built into it, v0.9.0's, so it reports the pin above as an unsupported SHA ("workflow analysis incomplete", exit 1): upgrade the CLI and its local hooks before the Action pin.

When checkwash blocks

The default threshold is high: a visible warn can still pass.

Read the finding. Each finding names its rule, its severity, and the file and test it is in. If the change at the top is the whole diff, made in test_invoice_total in tests/test_billing.py, the finding reads:

ASSERT_WEAKENED   HIGH   tests/test_billing.py :: test_invoice_total
  test_invoice_total: assertion strength EXACT_VALUE(90) -> BOUND(40)
  why high: NO_PROD_CHANGE_IN_DIFF
  no de-escalator applied
  before  assert total == 105.3
  after   assert total > 0

EXACT_VALUE(90) -> BOUND(40) ranks the assertion's strength before and after; 90 is not the value the test expected. why high names what raised the finding: here the diff changes no production file, so no production change explains the weaker test. checkwash check --format json gives the same findings as JSON.

Fix the change, or record a reviewed exemption. If the test was weakened by mistake, restore it, and fix the code if the restored test fails. If the weaker test is intended, the report also prints a checkwash allow command with the finding's fingerprint, and the ledger file to commit:

checkwash allow "<the fingerprint from the report>" --reason "why the weaker test is correct"

Configuration and exemptions are read from the base side of the diff, so a change cannot exempt itself: commit the entry through review, and it applies to the next diff once it is on the base. Since v0.10.0, a newly winning .checkwash config or allowance is compared with its BASE .greenwash predecessor, so a change cannot approve its own relaxation either. An entry needs a reason and expires after 90 days by default, 180 at most. Keep .checkwash/ and .greenwash/ behind code owners. From 0.11.0, the Stop hook tells Claude not to add an exemption itself but to tell you which finding blocks and why, and the PreToolUse ask asks you to approve only if the weaker test is intended. Exemption rules

Report a false block. If the change is honest and checkwash blocks it, look for it in the failure ledger first, then open an issue with the checkwash blocked honest work template, giving the diff and the checkwash check --format json output. A report becomes a regression fixture, and the ledger credits the reporter.

Know the limits

checkwash is alpha. A pass does not prove that a change is correct or honest. Python is the main language supported; JS/TS support covers a limited set of test patterns. Known gaps remain.

An edit that changes nothing, such as + 0 in a function a test calls or one line in a template or AUTHORS, can still hold a weakened Python or JS/TS test at warn, which passes at the default threshold. #419 is closed as a documented residual, not fixed (what still passes and how to block it). Read the 0.11.0 known issues and the earlier limits they build on before relying on a pass.

REPAIR_EVIDENCE describes related changes in the same diff; it does not prove a repair is correct, and an edit that changes nothing can earn it (#419). To block such changes, set the Action's fail-on: warn (checkwash doctor then reports "workflow analysis incomplete" and exits 1, which is expected: it proves only a checkwash step with no with:), or run checkwash check --task in a CLI step of your own with a base-side oracle_freeze contract (the Action has no task input); both block more honest changes too. From 0.11.0, on a local hook install, the Claude Code PreToolUse hook asks before an Edit, Write or MultiEdit of a Python test leaves a finding of its five weakening rules at high, whatever repair evidence the rest of the working tree holds; its limits are in the 0.11.0 known issues.

In v0.8.0, JS/TS test.each, it.each and test.for tables are not scanned: an assertion weakened or deleted in one, an edited row or a deleted table passes. In v0.9.0, each such table is a test unit, and its rows are read when they can be counted (an inline array or a tagged template, or a same-file const read only as a table). The rows of a describe.each table stay unread, and a table with a computed title is not a unit (#428).

Legitimate refactors can be flagged. The frozen historical corpus records 22 blocks out of 60 (36.7%). A separate pre-release source replay records 4/60 blocks, including 2/57 strictly qualified cases; three existing fixture errors remain. These are selected examples, not a general false-positive rate. The v0.10.0 ledger retains 6/60 static source blocks after the D-185 dispositions, including two additional known legitimate-refactor false blocks. This is a later source readback, not a rewrite of the historical runtime results.

A changed file over 1,000,000 bytes is not read. An image, data file, lockfile, document or non-Python production source that large is reported on stderr and earns no repair evidence.

Try it on your own changes before making it required. Coverage and limitations · Known gaps

CI details: doctor and the pin

The recommended Action and the CLI above are both v0.11.0. Record which version you use. Full setup and exemptions

Development snapshots: the source in this checkout may be newer than the published CLI. The advertised-release manifest records the version, commit and download bytes behind the installation commands above. New changes require their own candidate evidence and remain unreleased until the publication record verifies them; candidate CI success does not mean that the recommended Action pin includes them.

Advisory: Action trust boundary. Every published Action release up to and including v0.7.0 runs python -m pip and then checkwash with the pull request's checkout as the working directory, so a module in that checkout (for example a top-level pip package) can replace the installer and then the checker, and the pull request can tamper with the result it is judged by. Under checkwash's threat model that applies to every pull request it judges. v0.8.0 fixes it (#372); pin the v0.11.0 Action above. If you stay on v0.7.0 or earlier, treat a pass as advisory, review the pull request before merging and keep comment-pr off. Known issue

Which commit is the recommended pin? A release cannot embed its own commit SHA. Up to v0.10.0, the recommended pin is the commit the tag points to: git rev-parse 'vX.Y.Z^{commit}'. From v0.11.0 (D-194), it is the release's Action commit, the tag's second parent's parent: git rev-parse 'vX.Y.Z^2^1'. It is not the commit the tag points to. The release pull request's last commit changes one line, doctor's own Action pin, to that Action commit, and the tag differs from the Action commit only in that line. The Action runs only checkwash check, which does not import doctor. From v0.11.0, a release's doctor accepts two Action pins: the one recommended before the release and the release's own Action commit.

Pin updaters such as Dependabot and Renovate follow the # vX.Y.Z comment, so they will propose the tag's commit, which doctor rejects (inferred from those tools' documentation, not run). For the same reason zizmor's ref-version-mismatch audit can report the comment (inferred, not run). The Action itself is unaffected: it runs the same check at either commit. doctor 0.10.0 does not accept the 0.11.0 pin, so upgrade the CLI and its local hooks first, then the Action pin. The recommended v0.11.0 Action includes the engine published on 2026-11-04 Asia/Taipei. A CLI upgrade does not update an existing Action. Action reference

More options and evidence

pipx install checkwash==0.11.0
# or from the release tag:
pipx install git+https://github.com/taipei49314/checkwash@v0.11.0

checkwash check HEAD~1..HEAD
checkwash demo                  # 8 real tampering cases, blocked, offline

Related projects: checkwash-corpus and smallestlie hold evaluation work.

The package also includes a limited quality preview for coverage, Ruff and mypy configuration review. Setup and CI adoption guide. Its frozen legacy-byte comparison remains red; publication does not establish natural-case acceptance or effective enforcement.

Alpha pre-release. 22 detectors, 22461 tests (collection denominator). Zero runtime dependencies. Apache-2.0.

原文链接：https://x.com/kartg123451/status/2118550260104769565

相关链接：https://github.com/taipei49314/checkwash/blob/0ebd5ad890c252b4546dbd692ac0414fdbc201cc/README.md

乔木改写

Anthropic 介绍，其生物学研究团队与 Claude 合作，解析了来自细菌生命的 vault 颗粒的首个三维结构。

Vault 颗粒呈桶状，是人体细胞中最大的蛋白质结构之一。Leonard Rome 和 Nancy Kedersha 于 1986 年发现了它，但它的功能至今仍不清楚。

此前，人们只知道动物、植物等复杂生物的细胞中存在 vault。团队这次证明，细菌以及感染细菌的病毒中的类似蛋白，也能组装成 vault。在这些更简单的生命形式中找到相应结构，可能有助于研究它在人类细胞中的作用。

Anthropic 将这一项目同时视为 AI 加速科学研究的潜力与局限的展示。团队估计，这次如果靠人工确定颗粒结构，原本需要数周到数月的细致工作，借助 Claude 则只用了几天。但 Claude 也犯了一些简单错误，生物学团队与机器学习研究人员合作处理了这些问题。

本站保存的是这份研究介绍；详细过程可见 Anthropic 的完整文章。

原文

Investigating vault particles with Claude

We’re sharing how our biology research team worked with Claude to map the first 3D structure of a vault particle from bacterial life.

Vault particles are a longstanding mystery. They are barrel-shaped objects, among the largest protein structures found in our cells. They were first discovered by Leonard Rome and Nancy Kedersha in 1986, but their function remains unknown.

With Claude, we’ve found results that might bring us closer to solving that mystery.

Until now, vaults were only known in the cells of complex things like animals and plants. We have shown that similar proteins from bacteria (and the viruses that infect them) can also assemble into vaults. Finding the structures in these simpler forms of life may make it easier to investigate their function in our own cells.

For our life sciences team, the project was also a demonstration of the potential (and the limitations) of using AI to accelerate science.

In our full piece, we discuss how Claude was able to dramatically speed up some stages—manually determining the vault particle’s structure would have taken weeks to months of painstaking work, but here it took days. We also discuss how Claude made some simple mistakes along the way, and how we collaborated with our machine learning researchers to address them.

Read more: https://www.anthropic.com/research/investigating-vault-particles-with-claude

原文链接：https://x.com/AnthropicAI/status/2118073665296228863

相关链接：https://www.anthropic.com/research/investigating-vault-particles-with-claude

乔木改写

Cursor 宣布，现在可以在 Cursor 中构建可运行的线上 App，并通过链接分享。

它在介绍中指出，Agent 已经让写代码更容易，但大多数应用仍停留在 localhost。用户描述自己需要的工具后，Cursor 会构建应用、设置数据库、运行确定性的安全检查和 Agent 安全检查，再完成部署。

原文

You can now build live apps in Cursor and share them with a link

You can now build live apps in Cursor and share them with a link.

Agents have made it easier to write code, but most apps never make it off localhost. Describe the tool you want - Cursor will build it, set up a database, run deterministic and agentic security checks, and deploy.

原文链接：https://x.com/cursor_ai/status/2118458629091426304

相关链接：https://cursor.com/docs/cloud-agent/apps

乔木改写

OpenAI 发布 GPT-6 Cyber，称其在增强安全研究能力的同时，也改进了模型对齐。

按 OpenAI 的介绍，该模型用于其前一天公布的 Linux 内核研究。Mozilla、Palo Alto Networks、Calif、Cisco 等公司的安全团队，以及 OpenAI 自己的安全团队，已经在使用它。

目前，GPT-6 Cyber 向部分 Daybreak Red 客户开放，访问范围将逐步扩大。完整公告见 OpenAI 原文。

原文

Introducing GPT-6 Cyber

Introducing GPT‑6 Cyber: a major advance in security research that pairs stronger capabilities with better model alignment.

The model powered the Linux kernel research we announced yesterday, and security teams at companies like Mozilla, Palo Alto Networks, Calif, and Cisco—alongside our own at OpenAI—are already putting it to work.

GPT-6 Cyber is available to select Daybreak Red customers and we are expanding access over time.

https://openai.com/index/introducing-gpt-6-cyber/

原文链接：https://x.com/OpenAI/status/2118390190246879554

相关链接：https://openai.com/index/introducing-gpt-6-cyber/

乔木改写

Guillermo Rauch 认为，持续替用户处理事情，是 Agent 的杀手级应用。

他举了 Muse 和 Instinct 的例子：它们会在夜间整理记忆，早上提醒用户，并在距离上一次互动过了 N 天后发来消息。

谈到配套基础设施，Rauch 将 Vercel Schedules 形容为加强版 Cron。按他的介绍，任务可以在运行时定义，能够无限扩展，而且调度系统位于 Sandbox 之外，因此可以按需调用计算资源，以更低的成本运行。

原文

Doing stuff for you over time is the killer app of agents

“Doing stuff for you over time” is the killer app of agents.

Muse and Instinct consolidate memories at night, remind you of stuff in the morning, message you after N days passed since your last interaction…

Vercel Schedules are like Cron on steroids. They can be defined at runtime, scale infinitely, and sit outside of the Sandbox, which means you can cost-efficiently invoke compute as needed.

原文链接：https://x.com/rauchg/status/2118440831355068671

相关链接：https://vercel.com/blog/introducing-vercel-schedules

乔木改写

Tibo 翻出十八年前 Dropbox 创始人录制的一段屏幕演示，称它是硅谷历史上最成功的发布视频。他说，这段视频让 Dropbox 的候补名单一夜间从 5000 人增长到 75000 人。

在他看来，视频有效有五个原因：

1. 它展示了真实产品完成承诺的那件事：把文件放进一个文件夹，文件就会出现在另一台电脑上。
2. 视频中有许多目标受众才懂的内部笑话。
3. 创始人把视频发在用户已经聚集的地方：更早的版本发在 Hacker News，这个版本则发在 Digg。
4. 它解释了一个新概念，却没有使用术语。
5. 它让人看到了尚未开放使用的产品。Dropbox 当时还没向公众开放，视频先证明它能工作，之后由候补名单承接需求。

Tibo 的建议是，当大家都在用 Claude 制作精致的视频时，也可以只录一段简单的屏幕演示。在他看来，这样反而会显得真实十倍。

原文

The most successful launch video in Silicon Valley history

this is the most successful launch video in Silicon Valley history

18 years ago, the founder of Dropbox recorded his screen to make this demo

it took their waitlist from 5,000 to 75,000 overnight

imo, here's what made it work:

1. it showed the real product doing the one thing it promised. you drop a file into a folder and it shows up on your other computer

2. it was full of inside jokes for the exact audience he was trying to reach

3. he posted it where his users already were. an earlier version went on Hacker News, and this one went on Digg

4. it explained a new idea without any jargon

5. it let people see something they couldn't use yet. Dropbox wasn't open to the public, so the video showed it working and the waitlist did the rest

while everyone is doing super fancy video with Claude, you can do a very simple screen recording

you will look x10 more authentic

原文链接：https://x.com/tibo_maker/status/2118289869428380138

乔木改写

Amjad Masad 发布了一份安全建议，认为运行在员工终端上的 AI Agent 正成为企业和政府的系统性安全风险。一些机构已经开始限制 Claude Code 等工具；原文提到，据报道高盛采取了这样的措施。Replit 提出的办法，是让 Agent 默认运行在专用、受组织控制的云环境中，只访问完成当前任务所需的资源。

建议书认为，供应链攻击原本就能侵入开发者电脑，并调用已安装的 AI 工具搜索企业敏感信息。Agent harness 往往拥有广泛权限，而人工监督正在减少，使其越来越值得攻击者关注。无人值守的自动化同样可能被 GitHub Issue 或邮件中隐藏的指令引向私有数据和未授权操作。

个人 Agent 会进一步放大这类风险：它们主要由个人控制，组织管理不足，防护措施不一致，又被鼓励主动行动。读取每封邮件和日历邀请，也意味着每次互动都可能接触提示注入。Replit 因此主张，继续采用 Agent 的同时，要为会读取不可信内容、运行代码、代表用户行动的软件提供专门的执行环境；凭证、权限、网络访问和生产变更的控制应放在 Agent 无法自行改变的位置。

使用增长快于治理

原文引用 Stack Overflow 调查：使用或计划使用 AI 工具的受访者比例，从 2023 年的 70% 升至 2024 年的 76%，再到 2025 年的 84% [1][2]。这些数据反映的是不同年度样本中 AI 工具的采用与使用意向，并不是自主 Agent 使用率。

Salt Labs 在其客户环境中检测到 MCP 相关基础设施的比例，从 2026 年 1 月的 16.7% 升至 8 月的 21.6%，增加 4.9 个百分点 [3]。原文提醒，这只是该供应商客户环境的观察，能否代表整个行业、前后客户样本是否一致，都还没有确定。

部署与控制之间的差距更值得注意。在 Salt Security 2026 年 10 月的调查中，86.4% 的受访者说组织内至少有一个 Agent 在运行，但只有 17.5% 能自动停止在生产中出现异常行为的 Agent，只有 12.5% 能持续追踪所有 Agent 的完整执行路径 [3]。

另一项 Cloud Security Alliance 研究中，47% 的受访者称过去一年发生过 Agent 相关安全事件，54% 称组织内有 1 到 100 个未经批准的 Agent，只有 31% 称组织正式采用了 Agent 治理政策 [4]。这项研究在 2025 年 9 月和 11 月调查了 445 名 IT 与安全专业人员，于 2026 年 4 月发布；Zenity 提供经费并参与设计问卷。原文认为，它支持治理存在缺口的判断，但不能把它与 Salt 的结果拼成一条事件率趋势。

不同攻击中的边界失效

建议书逐项列出了事故和研究案例，分别说明证据范围。

个人 Agent 接入敏感系统。 WIRED 报道，一名 Accenture 承包商将个人 Claude Code 账号接入美国国防部的敏感环境，凭证和系统信息似乎流向了第三方服务 [5]。Accenture 否认这些指控，称独立调查认为指控没有根据。WIRED 也表示，没有迹象说明攻击者访问了机密或绝密系统。原文用这一争议案例说明敏感基础设施与个人 Agent 工具之间出现失控连接的风险，而非将争议当成定论。

开发者电脑上的恶意扩展。 WIRED 报道，经 Cursor 市场分发的一个被篡改的 Claude Code 扩展，最多被 5000 名用户下载；它不是 Anthropic 发布的版本 [6]。Cursor 介绍，该扩展收集主机信息，并尝试安装持久化 SSH 后门 [7]。实现错误限制了后门在大部分受影响 macOS 和 Windows 主机上的效果，但仅卸载扩展不一定能移除持久化机制。原文提醒，不能依赖攻击者下次仍犯同样的错误。

恶意软件包调用 AI 工具寻找密钥。 在 2025 年 8 月 Nx 的 s1ngularity 攻击中，恶意软件包利用电脑上已有的 Claude、Gemini 和 Q 命令行工具辅助侦察。Wiz 观察到数百起成功的 AI 辅助活动，以及超过 1000 个有效 GitHub token 泄露。后续攻击使用这些 token，将超过 5500 个私有仓库公开，影响了 400 多名用户或组织；开发者电脑和部分构建流水线都受到了影响 [8]。

合法工具也可能成为数据外传通道。 Invariant 演示了如何通过一个恶意公开 GitHub Issue，诱导使用 GitHub MCP 的 Agent 读取私有仓库资料，再将其写入公开 PR [9]。演示中的 MCP 服务器本身没有被攻破。这证明了一条可能的攻击路径，原始来源并未报告大规模泄露；获准使用的工具和目的服务，仍可能承载未经授权的数据转移。

工具跑出了预期沙箱。 CodeRabbit 确认，研究人员于 2025 年 1 月发现 RuboCop 在沙箱外运行 [10]。公司称，它在 12 小时内完成了全面修复，没有发现客户数据被访问或存在恶意活动的证据。原文据此强调，隔离范围必须覆盖每一个可执行工具和子进程，不能只覆盖主要 Agent 运行时。

云端 Agent 暴露关联账号凭证。 Salt Labs 演示了如何用恶意邮件内容，诱导 Manus 在云沙箱中执行代码；关联服务的 token 当时可被读取，安全警告却在载荷执行后才出现 [11]。Salt 称漏洞后来已修复。这也是云托管本身不等于安全的一个反例。

合法签名权限被滥用。 Elliptic 对 10 月 29 日 Binance 失窃事件的分析认为，攻击者可能诱使 Binance 自身系统签署未经授权的交易。Binance 最初提到 DevOps 笔记本可能失陷，后来又披露其他入侵点，完整路径仍未确定。在以太坊上，签署的 EIP-7702 授权将热钱包的行为委托给攻击者控制的合约，而且委托在最初那笔交易后仍然有效 [12]。原文明确说明，这不是 AI Agent 攻击的证据；它体现的是另一项控制要求：即便凭证没有被直接窃取，只要失陷的调用方能用它授权有害操作，保护凭证本身也不够。

原文将这些案例中的共同问题归为三类：不可信软件带着过大权限执行、不可信内容被当作指令，以及控制措施未能阻止最终操作。

为什么主张受管云环境

员工工作站可能同时拥有浏览器会话、SSH key、云凭证、私有仓库、内网访问和其他业务数据，远多于多数 Agent 任务所需的权限。

专用云环境可以把 Agent 执行与工作站分离。组织能够只配置必要工作区，集中实施访问控制，独立记录操作，并在需要时停止或重建环境，而不必重建员工电脑。要获得这些收益，执行、策略实施和可观测性都必须真正进入受控环境。

本地扩展即使把提示词发给云端模型，仍可能在本机执行代码。同样，把生产凭证复制到远端、给远端 Agent 不受限的网络访问，只是把风险挪了地方。目标是将失陷的 Agent 或依赖限制在一个权限有限的工作区内，组织政策对它的约束不应取决于员工本人拥有多大的权限。

原文图 2 的文字说明描绘了这种边界：Agent 与工具运行在组织控制、任务限定的专用沙箱内，与员工设备的广泛权限分离；独立系统代理凭证，约束权限及流向获准资源的数据。审计、审批和遏制措施必须在 Agent 控制之外执行，而且这些保护需要经过验证，不能仅靠云托管的名称。

需要独立实施的控制

建议书要求将 Agent 执行视作不可信，并列出以下控制项：

• 经过验证的执行隔离。 Agent、安装脚本、工具和依赖都应运行在专用沙箱内，安全实施机制放在其控制之外；不继承员工主目录、浏览器会话或不受限的企业网络权限。
• 经纪式凭证与独立身份。 外部服务凭证不应出现在 Agent 可读的内存、文件或环境变量中。请求通过代理授权，并绑定 Agent、发起人、组织和具体任务。
• 逐步提权授权。 隔离凭证还不够，操作、目标资源、目的地和委托权限的变化，都需要独立检查。高影响操作必须明确批准；拥有合法身份或能访问签名服务，不等于已经获准执行某项操作。
• 任务范围内的权限。 每次运行只获得需要的资源和操作权限，Agent 不得提升自身权限，也不得修改约束自己的政策。
• 网络与数据流限制。 对 DNS、直接 IP 访问、云元数据访问和外部请求加以控制；即使在同一个获准服务内部，也要区分操作和数据目的地。允许访问 GitHub，不等于允许把私有数据发布到 GitHub 任意位置。
• 独立的生产边界。 Agent 可以提出改动，但不应长期持有生产写入权限。重要变更由可信的审批与部署系统实施，并向审批者展示真正将被授权的操作。
• 受控的软件供应链。 核查来源、固定获准版本、审阅工具和 MCP 配置的变化；安装依赖时不带生产凭证。
• 独立审计与遏制。 在 Agent 无法修改的位置记录操作及实施决策，不依赖 Agent 配合就能自动遏制或撤销权限。遏制还必须撤销已授予下游的委托与权限，而不只是停止最初进程或轮换凭证。

组织可以先做什么

Replit 建议，把获准的受管云 Agent 环境作为自主开发的标准路径；盘点现有本地 Agent 和扩展，移除不必要的访问，并禁止个人 Agent 账号连接敏感生产系统。

供应商应实际证明这些边界：Agent 读不到服务凭证，不能绕过网络控制、扩大权限、修改审计记录或未经批准改动生产。遏制能力也应经过实测，包括紧急停止开关是否真的能停止 Agent 活动。

可能受影响的 Cursor 安装，应先按供应商事件响应指南处理。把未来工作移到云端，并不能修复一台已经失陷的终端 [7]。监管行业和政府业务则应只使用获准处理相关数据与系统的环境，并检查模型端点、日志、存储和外部工具是否满足数据驻留、保留、合同及认证要求。

建议书最后说明，它推荐的是有可验证安全控制支撑的云执行。所引资料并未建立本地 Agent 与云 Agent 之间可比的泄露率；它们说明的是隔离、有限权限、凭证分离和独立实施为何重要。每个 Agent 的访问都应按具体任务划定。

原文另推荐 Replit 提出的 Agent Habitat Architecture 及其基于证据的成熟度分级，作为评估独立权限、经纪式凭证、受控数据流和带外遏制的参考框架。每一款云 Agent 产品仍需单独核查实际实现了哪些控制。

原文参考资料

以下资料由原文列出，查阅日期均为 2026 年 11 月 1 日；本站未在本轮重新核验这些外部材料。

1. Stack Overflow：AI | 2024 Developer Survey。
2. Stack Overflow：AI | 2025 Developer Survey。
3. Salt Security：Nearly half of enterprises report an AI agent took an unauthorized action last year，2026 年 10 月 13 日。
4. Cloud Security Alliance：More than half of organizations experience AI agent scope violations，2026 年 4 月 16 日。
5. V. Elliott：A defense contractor gave their personal AI agent access to a secure Pentagon system | WIRED。
6. L. H. Newman：A supply chain attack put malicious Anthropic Claude extensions on thousands of devices | WIRED。
7. Cursor：Claude Code extension security notice，2026 年 10 月 30 日更新。
8. M. Bar、R. McCarthy：s1ngularity: Supply chain attack leaks secrets on GitHub | Wiz，2025 年 8 月 27 日发布、8 月 29 日更新。
9. Invariant Labs：GitHub MCP vulnerability research。
10. H. Gill：Our response to the January 2025 Kudelski Security vulnerability disclosure | CodeRabbit，2025 年 8 月 19 日。
11. Salt Labs：How we hijacked an AI agent with a single email，2026 年 10 月 1 日。
12. T. Robinson：The largest theft in history: North Korea behind the $6.4 billion Binance hack | Elliptic，2026 年 10 月 30 日发布、11 月 1 日更新。

原文

Security advisory: Make managed cloud environments the default for AI agents

AI agents running on employee endpoints are becoming a systemic security risk for enterprises & governments.

Some are responding by restricting tools like Claude Code, as Goldman Sachs reportedly has.

In this security advisory, we outline the risks and recommend a path forward.

Security advisory: Make managed cloud environments the default for AI agents

AI agents should operate in dedicated, organization-controlled environments with limited access to the resources each task requires. Recent security incidents [5], [6] and research demonstrate why this is a growing necessity. Supply-chain attacks have always been able to compromise developer machines and recruit installed AI tools to search for sensitive corporate information and data. Agent harnesses are typically given broad privileges and have a shrinking amount of human oversight. Combined, these two effects are making agents an increasingly interesting target for attackers. Even unattended automations are being targeted: instructions hidden in a GitHub issue or email can redirect an agent toward private data and unauthorized actions. The rise of personal agents increases the risk even more, since they are insufficiently managed by organizations, mostly controlled by individuals, run with an inconsistent set of guardrails, and are incentivized to be proactive above all else: they read every email, every calendar invite; Every interaction is an opportunity to get prompt-injected. There is a better way forward: Organizations can continue adopting agents by giving them execution environments designed for software that reads untrusted content, runs code, and acts on a user’s behalf. In a world where agents are becoming more prevalent, @Replit believes that organizations must make managed, isolated cloud agents the default for autonomous development, with credentials, permissions, network access, and production changes controlled outside the agent’s reach. Adoption is growing faster than control AI development tools are becoming standard. Stack Overflow reports that the share of respondents using or planning to use AI tools rose from 70% in 2023 to 76% in 2024 [1] and 84% in 2025 [2]. This establishes the scale of AI-tool adoption and intent across annual respondent samples. Autonomous-agent usage falls outside the specific scope of this measure. Agent infrastructure is also expanding. Salt Labs [3] detected MCP-related infrastructure in 16.7% of its customer environments in January 2026 and 21.6% in August: a 4.9-percentage-point increase. These observations are limited to the vendor’s customer environments. Their applicability to the wider industry and the consistency of the customer cohort remain unestablished. The more consequential finding is the gap between deployment and control. In Salt Security’s October 2026 survey [3], 86.4% of respondents reported at least one agent operating in their organization, but only 17.5% could automatically stop an agent behaving unexpectedly in production. Only 12.5% could consistently trace complete agent paths across all agents.

A separate Cloud Security Alliance study reinforces the concern [4]: 47% of respondents reported an agent-related security incident in the preceding year, 54% reported between one and 100 unsanctioned agents, and only 31% reported formally adopted agent-governance policies. The study surveyed 445 IT and security professionals in September and November 2025 and was published in April 2026. Zenity financed the research and co-developed the questionnaire. These findings corroborate a governance problem; they should not be combined with Salt’s results into a single incident-rate trend. Different attacks, recurring boundary failures The evidence extends beyond any one model, vendor, or delivery mechanism. Sensitive-system access through a personal agent. WIRED reports that an Accenture contractor connected a personal Claude Code account to a sensitive Department of Defense environment and that credentials and system information appear to have reached a third-party service [5]. Accenture disputes the allegations, saying an independent investigation found them without merit. WIRED reports no indication that attackers accessed classified or top-secret systems. The risk illustrated is an uncontrolled connection between sensitive infrastructure and personal agent tooling. A malicious extension on developer machines. WIRED reports that a tampered Claude Code extension distributed through Cursor’s marketplace was downloaded by at most 5,000 users [6]. It was not an Anthropic-published release. Cursor describes host-information collection and attempts to install a persistent SSH backdoor [7]. Implementation errors limited the backdoor’s effectiveness on most affected macOS and Windows hosts, but uninstalling the extension alone may not remove its persistence mechanisms. We cannot rely on attacker mistakes in the future. Compromised packages recruiting AI tools to find secrets. In the August 2025 Nx “s1ngularity” campaign, malicious packages used previously-installed Claude, Gemini, and Q command-line tools to assist reconnaissance. Wiz [8] observed successful AI-assisted activity in hundreds of cases and more than 1,000 valid GitHub tokens exposed. A subsequent phase used leaked tokens to make over 5,500 private repositories public, affecting more than 400 users or organizations. Developer machines and some build pipelines were affected. Private data leaked through legitimate tools. Invariant [9] demonstrated that a malicious public GitHub issue could persuade an agent using GitHub MCP to retrieve private-repository information and publish it in a public pull request. The researchers demonstrated this attack using an uncompromised MCP server. The demonstration establishes a possible attack path; the source reports no mass breach. An approved tool and destination can still carry an unauthorized data transfer. A tool running outside its intended sandbox. CodeRabbit confirmed that researchers discovered RuboCop operating outside its sandbox in January 2025 [10]. The company says it deployed a comprehensive fix within 12 hours and found no evidence of customer-data access or malicious activity. The lesson is that isolation must extend from the primary agent runtime to every executable tool and subprocess. A cloud agent exposing connected-account credentials. Salt Labs demonstrated that malicious email content could induce code execution in Manus’s cloud sandbox, where connected-service tokens were accessible [11]. A security warning appeared after the payload was executed. Salt says the flaw has since been resolved. This research is an important counterexample to the idea that cloud hosting alone provides security. Compromised infrastructure misusing legitimate signing authority. Elliptic’s analysis of the October 29 Binance theft suggests attackers induced Binance’s own systems to sign unauthorized transactions. Binance initially reported a possible compromised DevOps laptop and later disclosed additional breach points; the full intrusion path remains unresolved. On Ethereum, signed EIP-7702 authorizations delegated a hot wallet’s behavior to an attacker-controlled contract, with delegation persisting beyond the initial transaction [12]. This is not evidence of an AI-agent attack. It illustrates a relevant control requirement: protecting credentials is insufficient if a compromised caller can use them to authorize harmful actions. Together, these cases illustrate three recurring problems: untrusted software executing with excessive access, untrusted content being treated as instructions, and controls that do not prevent the resulting action. Why managed cloud agents are the better default An employee workstation may contain browser sessions, SSH keys, cloud credentials, private repositories, internal-network access, and unrelated business data. That is far more access than most agent tasks require. A dedicated cloud environment makes it possible to separate agent execution from that workstation. The organization can provision only the necessary workspace, enforce access centrally, record actions independently, and stop or rebuild the environment without rebuilding the employee’s laptop. Realizing that benefit requires moving execution, policy enforcement, and observability into the controlled environment. A local extension that sends prompts to a cloud model still executes local code. Likewise, a remote agent with copied production credentials and unrestricted network access reproduces the same risks elsewhere. The objective is to contain a compromised agent or dependency within a limited workspace. Organizational policy must govern the agent’s access independently of the employee’s broader authority. 📷 Fig. 2. A conceptual security boundary for autonomous agents. The agent and its tools run in a dedicated, task-scoped sandbox within an organization-controlled environment, separate from the employee device’s broader authority. Independent controls broker credentials and govern permissions and data flows to approved resources. Audit records, approval gates, and containment are enforced outside the agent’s control. Cloud hosting requires verified controls to provide these protections. Required controls It is recommended to treat agent execution as untrusted and enforce boundaries independently of the agent’s cooperation. Organizations should require: Verified execution isolation. Keep agents, installation scripts, tools, and dependencies inside dedicated sandboxes. Place security enforcement outside their control. Do not inherit employee home directories, browser sessions, or unrestricted corporate-network access. Brokered credentials and distinct identities. Keep external-service credentials out of agent-readable memory, files, and environment variables. Authorize requests through a broker, bound to the agent, initiating principal, organization, and task. Step-up Authorization: Credential isolation must be paired with independent checks on the requested operation, target resource, destination, and any change in delegated authority. Require explicit approval for high-impact actions; possession of a valid identity or access to a signing service is not sufficient authorization. Task-scoped permissions. Give each run only the resources and operations it needs. Agents must not be able to raise their own permissions or alter the policy that governs them. Network and data-flow restrictions. Control DNS, direct-IP traffic, cloud-metadata access, and external requests. Distinguish permitted operations and data destinations within approved services; allowing GitHub must not imply allowing private data to be published anywhere on GitHub. A separate production boundary. Let agents propose changes without standing production-write authority. Apply consequential changes through trusted approval and deployment systems that display the actual action being authorized. Controlled software supply chains. Verify provenance, pin approved versions, and review tool and MCP configuration changes. Install dependencies without production credentials. Independent audit and containment. Record actions and enforcement decisions somewhere the agent cannot alter. Support automatic containment and revocation without depending on the agent’s cooperation. Containment must also revoke downstream delegations and permissions already granted, not just stop the initiating process or rotate its credentials. Recommended action Make approved, managed cloud-agent environments the standard path for autonomous development. Inventory existing local agents and extensions, remove unnecessary access, and prohibit personal agent accounts from connecting to sensitive production systems. Require vendors to demonstrate their boundaries: that an agent cannot read service credentials, bypass network controls, expand its permissions, modify audit records, or make unapproved production changes. Verify containment through practical tests, including whether the kill switch actually stops agent activity. For potentially affected Cursor installations, follow the vendor’s incident-response guidance now. Moving future work to the cloud does not remediate an already compromised endpoint [7]. For regulated and government workloads, use only environments authorized for the relevant data and systems. Assess model endpoints, logs, storage, and external tools against residency, retention, contractual, and accreditation requirements. The recommendation is cloud execution backed by verified security controls. The sources reviewed do not establish a comparative breach rate for local versus cloud agents. They establish why isolation, limited authority, credential separation, and independent enforcement matter. Agents’ access should be scoped to the requirements of each task. For a concrete framework to evaluate these controls, see Replit’s proposed Agent Habitat Architecture and its evidence-based maturity scale. It provides a reference architecture for independently enforced permissions, brokered credentials, controlled data flows, and out-of-band containment. Each cloud-agent product should be assessed separately to verify which controls it implements. References [1] Stack Overflow, “AI | 2024 Developer Survey”. Accessed: Nov. 1, 2026. [Online]. Available: https://survey.stackoverflow.co/2024/ai [2] Stack Overflow, “AI | 2025 Developer Survey”. Accessed: Nov. 1, 2026. [Online]. Available: https://survey.stackoverflow.co/2025/ai [3] Salt Security, “Nearly half of enterprises report an AI agent took an unauthorized action last year, Salt Security research finds,” Oct. 13, 2026. Accessed: Nov. 1, 2026. [Online]. Available: https://salt.security/press-releases/state-of-agentic-ai-and-api-security-2h-2026 [4] Cloud Security Alliance, “More than half of organizations experience AI agent scope violations, Cloud Security Alliance study finds,” Apr. 16, 2026. Accessed: Nov. 1, 2026. [Online]. Available: https://cloudsecurityalliance.org/press-releases/2026/04/16/more-than-half-of-organizations-experience-ai-agent-scope-violations-cloud-security-alliance-study-finds [5] V. Elliott, “A defense contractor gave their personal AI agent access to a secure Pentagon system | WIRED”. Accessed: Nov. 1, 2026. [Online]. Available: https://www.wired.com/story/a-defense-contractor-gave-their-personal-ai-agent-access-to-a-secure-pentagon-system/ [6] L. H. Newman, “A supply chain attack put malicious Anthropic Claude extensions on thousands of devices | WIRED”. Accessed: Nov. 1, 2026. [Online]. Available: https://www.wired.com/story/trojan-malware-targets-thousands-of-anthropic-claude-developer-systems/ [7] Cursor, “Claude Code extension security notice,” updated Oct. 30, 2026. Accessed: Nov. 1, 2026. [Online]. Available: https://cursor.com/blog/claude-code-extension-security-notice [8] M. Bar and R. McCarthy, “s1ngularity: Supply chain attack leaks secrets on GitHub: Everything you need to know | Wiz Blog,” Aug. 27, 2025, updated Aug. 29, 2025. Accessed: Nov. 1, 2026. [Online]. Available: https://www.wiz.io/blog/s1ngularity-supply-chain-attack [9] Invariant Labs, GitHub MCP vulnerability research. Accessed: Nov. 1, 2026. [Online]. Available: https://invariantlabs.ai/blog/mcp-github-vulnerability [10] H. Gill, “Our response to the January 2025 Kudelski Security vulnerability disclosure: Action and continuous improvement,” CodeRabbit, Aug. 19, 2025. Accessed: Nov. 1, 2026. [Online]. Available: https://www.coderabbit.ai/blog/our-response-to-the-january-2025-kudelski-security-vulnerability-disclosure-action-and-continuous-improvement [11] Salt Labs, “How we hijacked an AI agent with a single email,” Salt Security, Oct. 1, 2026. Accessed: Nov. 1, 2026. [Online]. Available: https://salt.security/blog/how-we-hijacked-an-ai-agent-with-a-single-email [12] T. Robinson, “The largest theft in history: North Korea behind the $6.4 billion Binance hack,” Elliptic, Oct. 30, 2026, updated Nov. 1, 2026. Accessed: Nov. 1, 2026. Available: https://www.elliptic.co/insights/binance-largest-hack-in-history/

原文链接：https://x.com/amasad/status/2117504072416890937

乔木改写

以下依据本站保存的视频文字摘录整理，并非完整逐字稿。视频介绍了 DeepSeek 发布的 Super Agent。一些人将它称为对 Claude Code 的回应；视频作者更关注它的构建方式：整个系统由加入 DeepSeek Harness 的一组插件组成。

Super Agent 会监听团队已经使用的 GitHub 和飞书。当获得许可的成员在 Issue、PR 或相关对话中提出需求时，它建立工作项，开启独立会话完成任务，再返回结果，通常是一个 PR。它不要求用户先打开一段新的 Agent 对话。

项目随代码发布了一篇 X 文章，题为 Exploring Super Agent Design: Towards a More Autonomous Agent System。项目方明确说明，它是研究原型，并非 DeepSeek 的官方产品；不少设计想法和工程实现也由编码 Agent 编写。

同一个模型，换九套 harness

视频引用了 Frontier Harness Eval：固定使用 Kimi K3，比较九套编码 Agent harness。Codex 的通过率为 66.7%；DeepSeek Harness 的 creator 配置与 Claude Code 都为 63.3%。每完成一个成功任务，DeepSeek 约花费 $3.28，Claude Code 约为 $18.34。该测试中，最便宜与最贵的 harness 成本相差约 17.5 倍。

这些结果涉及通过率、成本、缓存和速度，即使模型不变，harness 也会影响这些表现。这些结果来自合成基准测试。

视频还提到，在 VentureBeat 的 Beyond the Pilot 播客中，Instacart 和 Box 的 CTO 都表达过当前 harness 比模型更重要的看法。

DSH 的插件结构

DeepSeek Harness（DSH）在八月发布，采用 MIT 许可，提供 Web UI，并不限定使用 DeepSeek 模型。它的设计原则是所有东西都可以成为插件：router、工具注册、会话日志，甚至 Agent loop 本身。

底层使用 Cordis 的时空可组合能力。时间维度上，卸载组件会撤销它做过的更改；空间维度上，组件依赖关系决定它们何时启用或停用。

DSH 有 standard、PTC、minimal、creator 等模式，对应不同的插件组合。creator 模式允许用户在聊天中描述一个插件，由 Agent 编写并安装。

router 与 worker 如何配合

Super Agent 通过 cordis.patch 接入，包含核心 Super Agent 插件、GitHub watcher 和飞书 watcher。上层是一段长期运行的 router 会话，下层是彼此隔离的 worker 会话。外部触发先经过白名单等确定性门禁，再交给模型处理；工作单会携带任务授权。

worker 使用精简的插件配置，不包含提问、规划、网页抓取或子 Agent，长时间运行的 shell 任务则转入后台。worker 还会直接订阅认领任务的对话，因此后续 PR 评论不必再经过 router。按材料中的测试，启用这一机制时，router 的决策次数比未启用时减少约 30%。

负载测试把触发量从每分钟 1 次提高到 40 次。通过批量处理，每次决策所需时间从 6.4 秒降至 1.6 秒。视频据此介绍了一种分工：router 可以使用更便宜、更快的模型，把更强的模型留给 worker。

这套设计希望尽量少占用人的注意力，同时提供尽可能多的价值。插件化也让开发者可以在不 fork 核心代码的情况下继续扩展 harness。

它仍有明确限制：只是研究原型，公开代码不等于内部使用的代码；接入范围主要是 GitHub 和飞书，评测是合成基准，提示注入、凭证等安全问题尚未完全解决。

原文

视频原文摘录，非完整逐字稿；请以原视频为准。

DeepSeek Super Agent and why the harness matters

DeepSeek Super Agent and why the harness matters

Source: YouTube https://www.youtube.com/watch?v=OrcsMi2-IcM

Okay, so this past weekend, DeepSeek released something that a lot of people out there are calling the answer to Claude Code. It's an agent that sits on a GitHub repo. It watches the issues and pull requests, and when someone on your team asks it to do something, It then goes off into its own session, does the work, and comes back with a PR Now, while that's cool in itself, that's not really the part that I want to talk about here. The part that I find really interesting is how they built it. The whole thing is just a set of plugins dropped into their agent harness.

When the DeepSeek Harness came out back in August, the plugin architecture is really cool. The new thing DeepSeek launched is called Super Agent. Along with the repo, they've written an article on X called Exploring Super Agent Design: Towards a More Autonomous Agent System. They're upfront that this is a research prototype, not an official DeepSeek product. Ideas and engineering were often written by coding agents themselves.

Basic idea: instead of you opening a session and talking to an agent, the agent listens to where your team already works — GitHub, and Feishu. When someone allowed asks it for something, it opens a work item, spins up its own session, does the work, replies back usually with a PR.

On the VentureBeat podcast Beyond the Pilot, CTOs of Instacart and Box both said the harness matters more than the model right now.

Frontier Harness Eval: same model (Kimi K3) through nine coding agent harnesses. Codex 66.7% pass rate; DeepSeek Harness creator config and Claude Code both 63.3%. DeepSeek ~$3.28 per successful task vs Claude Code ~$18.34. Gap cheapest to most expensive ~17.5x. Harness changes cost, caching, speed, and success rate even with model held constant.

DeepSeek Harness (DSH) MIT licensed, web UI, not locked to DeepSeek models. Tagline: everything is a plugin — router, tool registry, session log, even the agent loop. Built on Cordis (spatiotemporal composability): temporal (unload reverses everything) and spatial (dependencies activate/deactivate).

Modes: standard, PTC, minimal, creator — different plugin sets. Creator: describe a plugin in chat and agent writes/installs it.

Super Agent integrates via cordis.patch: core Super Agent plugin, GitHub watcher, Feishu watcher. Workers: stripped preset, no questions/plan/web fetch/sub-agents; long shell jobs backgrounded. Architecture: long-running router session + isolated worker sessions. Triggers go through a deterministic gate (allow-list) before the model. Tickets carry a mandate. Workers subscribe to conversation so PR comments skip the router (30% fewer router decisions when on).

Under load (1→40 triggers/min), batching made time per decision drop 6.4s→1.6s. Router can use a cheap/fast model; stronger models for workers.

Philosophy: take as little human attention as possible and give as much value as possible.

Caveats: prototype; published code ≠ internal; narrow (GitHub/Feishu); synthetic benchmarks; security (prompt injection, credentials) not fully solved.

Bigger point: harnesses evolving fast via composable plugins without forking the core.

原文链接：https://www.youtube.com/watch?v=OrcsMi2-IcM

乔木改写

Andrej Karpathy 不满 YouTube 的推荐，于是做了一个周末项目 HackerTube。他让 Claude 从 Hacker News 二十年来约 4300 万条评论中，筛出反复被推荐的 6454 个视频。按他的说法，整个项目用 Opus 5.5 花了约两小时，观看体验明显更好。

他列出了几个设计选择：

• 不登录也能获得推荐，使用状态保存在本地浏览器会话中。
• 缩略图选取视频中段有代表性的一帧，而非上传者经过大量 A/B 测试优化的封面。
• 评论分析的底层数据打包为一个 zip 文件，其他人可以把它交给 Claude，要求制作自己的版本。
• 另有 HackerTube Longs。

他也介绍了制作过程。默认工具是 Mac 版 Claude Code 桌面应用，模型为 Opus 5.5，使用 Medium effort。他只与 Claude 对话，让它完成工作；自己配合做的手动操作都在浏览器中：从 Google Cloud 界面创建查询 YouTube 视频元数据所需的 API key、启动用于提供服务的 VPS，以及购买域名。他提到，这种配合方式让他想起 MenuGen。

VPS 启动后，他把 IP 交给 Claude，让它更新系统、处理安全加固、部署并优化代码。他说，Claude 完成了这些工作。

Karpathy 感慨，一些过去需要数周甚至数月的项目，如今吃顿早饭的工夫，用少量提示就能做出来。他想知道，还有哪些项目是大家尚未想到的。原帖中的项目链接是 HackerTube。

原文附带媒体

原文附带媒体；未核验其与具体段落的对应关系。

原文

Weekend hack: HackerTube

Weekend hack: HackerTube. I was annoyed with YouTube's recommendations so I had Claude filter down 43M comments over 20 years of HN to 6,454 videos that keep being recommended. Significantly better viewing experience, ~2 hours with Opus 5.5: https://t.co/JIhXN7OSh1

• Excellent recs without any logins (state is kept in local browser session)
• Thumbnail is a representative frame from middle of the video instead of the super duper A/B test hyper-optimized user uploads
• Underlying data from the comment analysis as one zip file, point your Claude at it and ask for your own version
• HackerTube Longs ✊

A bit more on the process: I use Claude Code (desktop app for Mac), Opus 5.5 on Medium effort as my default. I only talk to Claude and have it do everything, the only "manual work" I helped with in tandem was in the browser (reminiscent of MenuGen): 1) I had to create some kind of API key from Google Cloud UI to query YouTube for video metadata, 2) I booted up a VPS box for serving and 3) I bought the domain name. Once the box was up I gave Claude the IP and told it update, secure, harden, deploy and optimize the code, and it did the whole thing.

Many awesome projects that would have normally taken weeks/months even can now be few-shotted for breakfast. What else are we not thinking of.

原文链接：https://x.com/karpathy/status/2116957135661945283

乔木改写

Pieter Levels 更新了自己的业务开销：常规部分已降到约 $300/月。这些网站和 App 每年服务约 20000 名付费客户、2.52 亿访客，处理 24 亿次请求，消耗 2.4 PB 带宽。

这个 $300/月的口径不包括 Photo AI。该项目的 GPU 费用约为 $7000/月，R2 存储约为 $800/月；把它算上，总开销约为 $8000/月。Levels 解释，Photo AI 单个项目的体量就远高于其他网站之和，因此在前面的数字中单独排除，但已经计入原帖图表。

他认为 Claude Code 很擅长自主寻找节省成本的办法。这次列出的节省合计接近 $34000/月，具体包括：

• GPU：约省 $29000/月。Photo AI 改用更好、更便宜的视频模型；原先的视频成本一直很高，因为他长期没有更新相关模型。部分图片任务转到有 Google Cloud Credits 可用的 Google Cloud，另一些任务转到按市场成本销售 AI 模型的 Sume。GPU 账单由 $36000/月降至 $7000/月。
• 服务器：约省 $3000/月。合并服务器，取消 DigitalOcean 和 Linode，将服务迁往 Hetzner。
• Backblaze B2 备份：他将这项节省记为约 $1100/月，并给出了账单从 $1200/月降到 $26/月的变化，称便宜了约 50 倍。此前保留了无限版本，没有限制的生命周期策略让服务器文件版本积累了十多年；现在改为保留 30 天。
• Cloudflare R2：约省 $400/月。为 Photo AI 的存储桶增加生命周期策略，让图片在 90 天后自动转入更便宜的 Infrequent Access。桶里已积累多年的照片，账单因此从 $1200/月降到 $800/月。
• 域名：约省 $118/月。Claude 检查了他在 Cloudflare 的域名注册，找出许多不再使用、没有必要继续付费的域名。Levels 还认为，其中一些项目在他所说的后 AGI 世界里已没有太多保留意义；他特别提到，.io 域名续费已达 $50/年，数量一多就会累积成明显支出。
• Claude 订阅：约省 $100/月。Claude 分析了自己的订阅使用量，发现他用不到 Max 20x 套餐的上限，于是将套餐从 $200/月降到 $100/月。

Levels 说，即便计入 Photo AI，他的总成本也比去年降低了约 90%，接近十年前刚开始做项目时的业务开销水平。

他也谈到对这些节省的看法：AI 很擅长发现可以削减的成本，短期内对经济未必有利，但长期可能去掉多余开支，为更好的投资腾出资金。在他看来，成本会随时间悄悄增加，人们往往没意识到钱花到了哪里；AI 能帮助逐项梳理，个人生活的开销也一样。

原文附带媒体

原文附带媒体；未核验其与具体段落的对应关系。

原文

Biz expenses down to ~$300/mo

💸 Update: My total biz expenses have now dropped to just ~$300/mo!

This is to run all my websites/apps serving per year:

• 20,000 paying customers
• 252 million visitors
• 2.4 billion requests
• 2.4 petabyte (PB) in bandwidth

(~$8,000/mo if you add the $7,000/mo GPU bill and $800/mo R2 storage I pay for Photo AI, I exclude it above because it's one project that's so much more expensive in size than all the other sites together, but included in the chart below)

Claude Code helped me to find smart ways to save more money, AI is just really really really good at finding cost savings by itself, which is not so great for the economy short-term, but long term it trims all the fat and opens up spending for better investments I think

Anyway here's what it saved me in money, almost $34,000/mo:

• $29,000/mo - switched to better and cheaper video models on Photo AI, video was very expensive and since I didn't update it ever it stayed expensive, also routed some image jobs to Google Cloud where I have some Google Cloud Credits, and also routed some jobs to Sume which sells AI models at market cost, very cool! This made my GPU bill go from $36,000/mo to $7,000/mo

• $3,000/mo - consolidated my servers, cancelled Digital Ocean and Linode and moved everything to Hetzner saving about $3,000/mo

• $1100/mo - analyzed my Backblaze B2 server backups, I had infinite versioning and lifecycle policies, that's reduced now to 30 days, this literally pushed the bill from $1,200/mo to $26/mo, about 50x cheaper! I had essentially versions of files for all my servers for over a decade, crazy!

• $400/mo - similarly added Lifecycle Policies to my Cloudflare R2 bucket for Photo AI, pics are automatically put in cheaper Infrequent Access after 90 days, there was years of photos in there so this reduced the R2 bill from $1,200/mo to $800/mo

• $118/mo - Claude analyzed all my Cloudflare Domain registrations, lots of domains I didn't use and didn't make sense to keep paying for, lots of things that also don't really make sense anymore in a post-AGI world, renewals for .io TLD are now $50/year which adds up fast!

• $100/mo - Claude also analyzed my Claude subscription (yes!) and discovered I don't even use the Max 20x plan limits, so I downgraded from $200/mo to $100/mo

Even including Photo AI, I have now cut my costs by 90% from last year, getting me close to my biz expenses amount from a decade ago when I started building things

Costs slowly creep up over the years and you don't realize all the things you're spending on but AI is nice to help you figure all that out (also in your personal life!)

原文链接：https://x.com/levelsio/status/2117694259558293692