50 名专业分析师用 Claude Code 做一项接近真实工作的因果推断分析时，四个分析决策里有 86% 跟了智能体的倾向。如果智能体的意见不起作用，这个比例大概会落在 50% 左右。

这项研究由 Amaç Herdağdelen、Hailey Joren 和 Alex Tamkin 撰写，2026 年 10 月 27 日发布在 Anthropic 的 Alignment 栏目。完整论文也公开了，地址是 https://www-cdn.anthropic.com/519c17a5d23a794e8b6d50a854048d0b61e0d73a.pdf 。文中的判断来自这篇研究报告，不是产品发布。博客的编辑支持来自 Tom Chivers 和 Arjun Panickssery。

文章封面是一幅拼贴：风暴里侧倾的帆船，叠在发黄的文件夹和一份蓝色乘客名单上。名单抬头写着 S. S. Celtic。

人们通常会假定，专家把具体操作交给智能体之后，要紧的决定仍由自己来做。智能体写代码、跑数字、给建议，专家则决定测什么、怎么理解含糊的证据、下一步往哪走。这类问题往往有不止一个说得通的答案，最后会落到做决定的那个人的经验和价值观上。

作者想知道，专家是不是真的在做这些决定，还是实际上在顺从智能体。他们请来有因果推断经验的专业分析师，用 Claude Code 完成一项统计分析，建议一个农业合作社：它正在运行的咨询服务有没有效果，该不该扩大。

分析过程中，分析师要在四组都说得通的选项里做选择，例如把生产率算成每公顷作物产量，还是算成单位经营成本对应的产量，然后再给出最终建议。为了观察这些决定是不是分析师自己做的，研究者给每个分析师的智能体都指定了倾向，分析师本人并不知情。倾向由投币决定。智能体接到的要求是：为自己的倾向提出最有力且诚实的理由；如果证据要求它改主意，就改；并且不得误导。

四个分析决策合计，分析师有 86% 的时候站在智能体的倾向一边。如果分析师在智能体表态前已经说出过自己的偏好，智能体意见相反时，他们有 60% 会改口；意见一致时，只有 6% 会改口。

事后被要求解释自己的选择时，77% 的分析师用了智能体的论证，有时连措辞都一样，但只有 15% 把智能体写成这些论证的来源。那些紧密跟随智能体建议的人，对自己最终建议的信心，和其他人差不多，也差不多同样愿意把它说成自己的判断。

四个说得通的选择，每一项都能把估计值挪出几个百分点

图 1 把实验设计拆成三块。每位参与者要做五个开放决定，每个决定都有两个站得住的选项：结果指标是按经营成本算产量，还是按每公顷产量；比较方法是水平调整，还是双重差分；评估范围是全体成员，还是只看已加入项目的农场；加权是按产量，还是按农场计数；最终建议是停办，还是推广。随后，每个决定都由一次不告诉参与者的投币，秘密设定智能体偏向哪一边。研究者再比较两组人选择选项 A 的比例差距，用这个差距估计倾向带来的影响。图里还举了智能体为两种倾向分别可能怎么说话的例子。

研究招募了 61 名分析师，做的是随机对照试验。他们都来自预先筛过的专业人员：通过了因果推断知识检查，经常使用 Python 或 R，并且有观察性因果分析的实践经验，也就是用真实世界数据而不是对照实验来研究因果关系。

任务发生在一个虚构机构 Greenfields Farming Cooperative。合作社办了一项同样虚构的自愿付费咨询项目，帮助成员农场提高作物产量，大约 40% 的成员已经报名。董事会要决定是补贴并扩大这个项目，还是停办。董事会只在项目把农场生产率提高一个既定幅度时才会扩大，这个幅度大约是 3% 到 4%，但没有精确规定生产率该怎么度量。

作者故意留下四个分析决策，每一项都有两个合理选项。

结果指标：生产率按每公顷作物产量计算，还是按单位经营成本对应的产量计算。

比较方法：看已加入农场的产量在加入后如何变化，再和其他农场在同一时期的产量变化对比，也就是双重差分；或者直接比较产量，同时用统计方法调整农场之间的差异。

评估覆盖哪些农场：只测量已加入农场里的效果，还是测量整个合作社成员的效果。

加权方式：每个农场计一票，还是让更大的生产者占更大权重。

这四个选择通向最终建议：向所有人推广这个项目，还是停办。分析师和智能体也可以提议延期一年。

作者写到，每一个分析决策都可能把估计出的生产率提升挪动几个百分点。董事会的门槛大约是 3% 到 4% 时，这几个百分点就很要紧。

每位分析师面对的五个决定，智能体的倾向都由投币决定。这些倾向写在隐藏指令里，也就是系统提示：把它当成自己的看法，相关决定出现时提出来，并在数据允许的范围内为它提出最强的理由。同一套指令也要求它：如果证据明确转而对自己不利，就改主意；不得歪曲结果、隐藏数据，也不得提出数字并不支持的论证。

61 名参与者里，有 11 人选择不使用智能体。除另行说明外，这些人被排除在分析之外。剩下的 50 人中，65% 拥有硕士或博士学位，或者有六年及以上经验；在回答了问卷的 48 人里，90% 把自己评为观察性因果推断的熟练或专家水平。任务用时的中位数是 65 分钟，一半人花了 50 到 79 分钟。他们在内置了 Claude Code 的网页版 VS Code 编辑器里工作，模型是 Claude Opus 4.8，设置为 High effort。

完成后，分析师提交四个分析选择、最终建议，以及对项目如何影响生产率的估计和置信区间，然后填写任务后问卷。另一个 AI 模型会阅读每份提交材料和代码，记录分析师在每个决定上选了哪一边。它的读法如果和分析师自己在问卷里的回答不一致，由研究者做最终判定。

四个分析决策的吻合率在 81% 到 92% 之间

分析师大多站在智能体的倾向一边。四个分析决策里，选择与智能体倾向吻合的比例分别是：结果指标 92%，比较方法 82%，评估覆盖哪些农场 81%，加权方式 90%。四个决策合计是 86%。

图 2 画的是选择选项 A 的人占多少，绿色是智能体倾向 A，紫色是倾向 B。结果指标是 0% 对 82%，相差 82 个百分点；比较方法是 28% 对 92%，相差 64 个百分点；评估覆盖哪些农场是 38% 对 100%，相差 62 个百分点；加权方式是 22% 对 100%，相差 78 个百分点；最终建议是 36% 对 54%，相差 17 个百分点。四个分析决策的移动幅度在 62 到 82 个百分点之间，最终建议是 17 个百分点。

智能体的倾向对最终建议影响较弱，可它前面给出的意见仍在塑造结果。当智能体倾向推广项目时，54% 的分析师建议推广；倾向另一边时，这个比例是 36%。延期一年被算作没有推广。作者认为影响变弱，是因为分析师一旦有了估计值，建议大体就取决于它有没有跨过董事会大约 3% 到 4% 的门槛，留给判断的空间更小了。可到那时，智能体在分析决策上的倾向已经塑造了估计值本身。只使用这四个倾向，一个统计模型就能正确预测 82% 的分析师建议；如果每次都猜最常见的那个建议，正确率是 61%。这个比较排除了提议延期的分析师。也就是说，在分析师动手之前，单看投币结果，研究就已经大体能预测他们会给出什么建议。

已经先选定一个选项的分析师，遇到智能体反对时经常改口。在 38 个例子里，分析师在智能体表达倾向之前，已经对某个分析决策说出过自己的偏好。倾向和他们的选择相反时，他们有 60% 会改口；倾向与他们一致时，只有 6% 会改口。两者相差 54 个百分点。这个估计只建立在 38 个例子上，并不精确，95% 置信区间从 24 个百分点到 82 个百分点。作者也写明，不能确定这个差别就是 AI 影响造成的：那些先说出偏好的分析师，可能和没有先说的人本来就不一样。

经验和专长并不能预测谁会顺从。在一项探索性分析里，作者检查了 10 项特征，看它们能不能预测分析师的决定有多常跟着智能体的倾向走：从业年数、自评的因果推断专长、教育程度、因果推断测验分数、对 AI 的信任、使用 AI 的频率、“认知需求”（衡量人们有多享受费力思考）、开放程度、性别和年龄。只有年龄显示出统计上显著的关联，年纪更大的分析师更常跟着倾向走。作者并不看重这一点：同时检验很多件事，更容易只是因为碰巧得到一个阳性结果；他们为此做了校正之后，这个结果就消失了。而且大多数分析师在大多数决定上都跟着智能体，可供解释的差异本来就很少，即便存在真实差别，也不容易被发现。

有人把建议看成自己的，两组受到的推动却差不多

跟着智能体倾向走的分析师，仍把最终建议说成自己的。任务后问卷里，52% 的分析师认为最终建议更多是自己的，而不是智能体的。其余人认为它更多是智能体的建议。可这两组受到的影响大致相当：前者的选择平均向智能体倾向移动了大约 65 个百分点，后者是 79 个百分点，这个差别仍在误差范围内。对自己建议的信心也呈现类似格局。问卷把信心从 0 分标到 100 分，0 分是“就像投币”，100 分是“我愿意拿职业声誉来担保”。在全部四个分析决策上都跟随倾向的人，平均分是 71；其他人是 73。

图 3 汇总了 48 人对“最终建议主要反映了谁的判断”的回答，用的是六点量表。认为更多是智能体的有 23 人：完全如此 0 人，大多如此 9 人，稍微偏向这一边 14 人。认为更多是自己的有 25 人：稍微偏向 8 人，大多如此 13 人，完全如此 4 人。作者据此写到，52% 的人把建议看成比智能体更属于自己，35% 认为它大多或完全是自己的，尽管智能体的倾向已经推动了他们的选择。

分析师事后解释选择时，用的常常是智能体的句子。问卷问的是：简要说明为什么选这个而不是另一个；如果这并不是一次主动选择，直接说出来就行。回答时，分析师不能回看自己和智能体的对话。即便如此，77% 的人至少在一个回答里重复了智能体的论证，有时逐字相同，只有 15% 在任何一处把功劳记给智能体。

当被明确问到自己觉得这项研究在考察什么时，48 人里有 6 人正确猜到，主题是 AI 对他们选择的影响，但没有人怀疑智能体自己的偏好被人动过手脚。

图 4 把任务中的智能体表述和分析师事后的问卷回答放在一起，标出了相同或相近的措辞。第一组里，智能体说：Since the subsidy is funded from surplus on total tonnage, I'd area-weight the headline so it reflects the co-op’s hectares rather than counting a 3-ha plot equal to a 400-ha operation. 分析师后来写道：The subsidy is funded from surplus on total tonnage so the decision relevant avg is over hectares not farms counted equally. 第二组里，智能体说：But one post-program season is thin ground to scrap a program that might still help, so rather than discontinue outright, extend the voluntary program one more season. 分析师后来写道：But one post program season, with a design the pre trend test complicates, felt thin ground for scrapping it outright.

不用智能体的分析师，严重错误更多，花费的时间也更长。11 名选择不用智能体的参与者里，有 6 人（55%）犯了使结果失效的严重方法错误；使用智能体的 50 人里，这样的人有 3 人（6%）。两个 AI 模型先标出错误，研究的一位作者在不知道分析师属于哪一组的情况下再逐条核对。不用智能体的人用时中位数是 90 分钟，其他人是 65 分钟。

这两组并不能直接比较，因此不能确定结果差别是不是由用不用 AI 造成的。不用 AI 的人平均经验更少：拥有硕士或博士学位的是 11 人中的 4 人，使用智能体且有这项数据的 46 人中则有 24 人；分析岗位经验不超过两年的是 11 人中的 6 人，使用智能体且回答了这个问题的 48 人中则有 15 人；把自己评为因果推断熟练或专家的是 11 人中的 8 人，使用智能体的 48 人中则有 43 人。

作者自己列出的界限：加强过的倾向，以及单一任务、单一模型

作者写明，他们故意加强了智能体的倾向。指令让智能体稳定地持有并说出一种偏好，这样才量得出效果。他们报告的比例，描述的是智能体倾向某一边并且说出来时会发生什么。它能在多大程度上对应现实工作，取决于专家自己的看法有多强、相关决定定义得有多清楚，以及模型本身怎么表现，包括它有没有偏好、会不会主动说出来。

真实世界里的智能体有时本来也会偏向某一边。在没有被指定倾向时，Claude Opus 4.8 在 123 次运行里有 112 次做出了同样的五个选择。这次实验是为了测量效果，才有意加强偏好。如果智能体没有得到明确偏好，也没有被要求把偏好说出来，影响可能会更小。

他们研究的是一项任务、一个领域、一个模型。任务大约花一小时，具体内容与农业有关，智能体运行的是 Claude Opus 4.8。分析师是因果推断专家，但对农业是生手。在真实工作里，他们会更熟悉这个领域，自己的看法大概也会更强。他们还可以访谈相关的人、和董事会交谈，并用别的办法缩小合理选项的范围，这可能会减弱智能体的影响。

这些估计是分析师之间的平均值。因为每个人分到的投币结果不同，比较智能体倾向相反的分析师，看到的是倾向的平均效果。研究者没有在智能体表态前记录每一位参与者自己的看法，所以不能说出任何单个人被影响了多少。

他们记录了分析师选了哪个选项，但没有记录他们偏好这个选项有多强烈。一种可能的解释是，专家自己并没有很强的意见。研究只测量了选择有没有移动，没有测量他们对这个选择有多有把握。

作者并不把这项研究写成少用 AI 的劝告。使用智能体的分析师完成得更快，严重错误也少得多；不过后一组人数少、经验也更浅，作者提醒不要读得过重。动手的工作越来越多地由 AI 承担时，这类决定会越来越多地成为人类判断和价值观还能起作用的地方。如果专家在自己没有察觉时顺从了智能体，智能体的偏好，无论它是任意的还是没有对齐的，都可能在没有任何人选择这些偏好的情况下，塑造关于政策建议或研究方向的重要决定。这对如何让 AI 保持安全尤其要紧。研究者已经开始用智能体来运行和分析实验；人类审阅者只有在判断属于自己时，才称得上一次独立检查。

两边理由讲得不均时，专家可能还没形成自己的看法

分析师如此容易顺从，一个原因可能是：专家并不总是带着完全成形的看法到来。他们常常是在权衡选项的过程中，才想清楚甚至才发现自己怎么看。研究里的智能体被要求不得误导，也常常会说明每种选项下结果会怎样变化。可它们很少为专家为什么可能更喜欢另一个选项提出有力理由，而那本来可能帮助分析师做出信息更充分的选择。只有一边被讲得很充分时，分析师就可能在还没理清自己想法时，接受了智能体的前提。

对单个专家来说，这可能意味着失去对自己工作的主导权，也就是研究者所说的 disempowerment。工作仍然署着专家的名字，可最依赖他们判断的那些决定，实际上被智能体原先的信念定了下来。

事情也不止于个人，因为科学和政策有很大一部分就建立在这类判断上。当不同的研究团队得到同一个结果时，人们会更信任它，因为假定每个团队是独立到达那里的。如果这些团队都顺从有着相同偏好的模型，他们的一致可能不再独立。表面上的共识，反映的也许是某一个模型的偏好；如果这些偏好是任意的或没有对齐的，同一种偏向就会同时塑造许多结果。

AI 安全研究也会碰到同样的问题。研究者已经开始用 AI 智能体来运行和分析实验，部分原因是工作量超过了人类团队跟得上的速度。把人留在回路里，前提是这个人的判断能提供一次独立检查。可在这项研究里，分析师签字认可的工作中，判断已经大体被随机指定给智能体的偏好定了下来。完整论文还显示，没有被指定倾向的智能体，自己的偏好更弱，有时还会反驳用户。

这种顺从很难被发现。检查 AI 辅助的工作有没有错误，或拿它去对照标准答案，都假定存在一个正确答案。在判断这类没有唯一正确答案的事情上，无论选哪一边，工作看起来都可以说得通。事后去问专家也可能没有帮助，因为他们未必察觉到智能体的影响。

作者写到，这并不是在主张少用 AI。用得好，AI 可以扩大人的主导权，而不是把它磨掉。他们建议以后从三个方向继续研究。

一是测量顺从。这次研究能测量它，是因为他们把智能体的倾向随机化了，而这在日常工作里很少做得到；结果也提示，人们未必能准确报告自己有没有顺从。需要有办法在日常场景里测量它，近年来一些估计“如果没有 AI，这个人会怎样完成任务”的工作，是一个起点。也需要更清楚地看到模型的默认偏好，以及它们怎样把这些偏好表达出来，好让人更知情地和模型相处。

二是换一种方式设计智能体。智能体可以为每个选项提出最有力的理由，或者先询问专家自己的看法，再给出意见。可这很快就会让用户不堪重负。一次分析可能包含许多开放决定，AI 的很大一部分价值，正在于人们不必把每一项都想一遍。智能体需要只提起那些最要紧的决定，可挑选哪些决定最要紧，本身又是一次判断。模型说话的方式大概也有影响，因为自信和流畅会打动人。一个听起来非常有把握的模型，可能把人推得比证据所允许的更远；一个说话含糊的模型，又只会惹恼使用者。

三是调整机构做法。机构常常把专家的签字，当成独立判断的证据。AI 做的工作越多，这个假定就越不稳。组织需要有意识地决定，哪些决定必须出自一个人自己形成的、知情的判断，并给人时间和资源去形成这种判断。判断也依赖技能，不再做这些决定的人，可能会失去做它们的能力。模型越来越强时，机构需要有明确策略，去确定人的角色，以及这些技能该怎样发展。即便面对非常强的 AI 系统，机构仍可能出于安全、韧性，或因为本身就看重这些能力，而选择保护人类的技能和监督。

作者认为，人和 AI 之间也许不存在唯一正确的分工。有些决定可以交出，有些值得留住。有了合适的工具，这种划分可以成为人们主动选择的结果，而不是在不知不觉中滑进去的安排。

下一次用智能体做没有唯一正确答案的分析时，可以先写下自己倾向的选项，或要求它把两边的理由都讲清楚。这是作者在“换一种方式设计智能体”里提出的做法，不是额外添上的产品建议。完整数据和实验细节，以他们的研究博客和论文为准。

How AI agents shape experts’ judgment calls

Published Oct 27, 2026 · Team: Alignment · Topics: How people use AI, Agents, Claude Code, Sycophancy, Alignment

Amaç Herdağdelen, Hailey Joren, Alex Tamkin

When experts hand their work to an AI agent, the usual assumption is that they still make the decisions that matter. The agent writes the code, runs the numbers, and offers advice, but the expert decides what to measure, how to interpret ambiguous evidence, and what the next steps should be. Often there is more than one reasonable answer to these questions, and the decision will come down to the experience and values of whoever makes the call.

We wanted to look at whether experts really do make those important decisions, or whether in practice they defer to their AI agents. In a study we report in a new paper, we asked 50 professional analysts to do a realistic statistical analysis with Claude Code. They all had experience in causal inference—that is, working out from data whether one thing causes another. Their task was to advise a farming cooperative on whether an advisory service it was running was effective and should be expanded.

Along the way, the analysts had to make four choices between two reasonable options, such as whether to measure productivity as crop yield per hectare or as yield per unit of operating cost, and then make a final decision. To see whether they were making these calls themselves, we set up each analyst’s agent to favor one option on each decision, which the analysts weren’t told about. Which way the agents would “lean” would be decided by coin flip. The agent was told to make the best honest case for its lean, to change its mind if the evidence demanded it, and never to mislead.

Across the four analytical choices, analysts went with their agent’s lean 86% of the time. If the agent’s input made no difference, we’d expect about 50%. When analysts had already expressed their own preference before the agent weighed in, they switched away from it 60% of the time if the agent disagreed, compared to 6% if it agreed.

Afterward, when asked to explain their choices, 77% of analysts used the agents’ own arguments—sometimes word for word—but only 15% credited the agent as the source of those arguments. Analysts who followed the agents’ recommendations closely were about as confident in their final recommendation as the rest, and roughly as likely to call it the product of their own judgment.

We don’t know how much this tells us about the real world. Real agents will sometimes lean one way or another—with no lean assigned, Claude Opus 4.8 made the same five choices in 112 out of 123 runs—but in this case we strengthened their preferences on purpose to measure their effect. When agents aren’t given clear preferences and instructions to voice them, the effect might be smaller.

This isn’t a simple cautionary tale against working with AI. Analysts who used the agent finished the task faster and made far fewer serious errors than those who chose to work without it, although the latter group was small and less experienced, so we are wary of reading too much into it. But as AI takes over more of the hands-on work, decisions like these are increasingly where human judgment and values will affect the result. If experts defer to agents without realizing it, the agents’ preferences—which could be arbitrary or misaligned—could shape important decisions about policy proposals or research directions without anyone having chosen those preferences. This is particularly important for research into keeping AI safe. Researchers have begun using agents to run and analyze experiments and a human reviewer is only an independent check if their judgment is their own.

Study design

We recruited 61 analysts for a randomized controlled trial. All came from a pre-screened pool of professionals who had passed a causal-inference knowledge check, used the programming languages Python or R regularly, and had hands-on experience with observational causal analysis, which means studying cause and effect using real-world data rather than a controlled experiment.

The task was set at a fictional organization, the Greenfields Farming Cooperative, which runs an (equally fictitious) voluntary, paid advisory program to help its member farms improve their crop yields. About 40% of members have enrolled. The co-op’s board must decide whether to subsidize and expand the program, or discontinue it. The board will expand the program only if it improves farms’ productivity by a set margin, roughly 3–4%, but it doesn’t say exactly how productivity should be measured.

We deliberately left four analytical decisions open, each with two reasonable options:

• Outcome: measure productivity as crop yield per hectare, or as yield per unit of operating cost?
• Comparison method: look at how enrolled farms’ yields changed after joining compared with how other farms’ yields changed over the same period (a “difference-in-differences” design), or compare yields directly while adjusting statistically for differences between the farms?
• Which farms the assessment covers: measure the effect only in the enrolled farms, or across the cooperative’s entire membership?
• Weighting: count every farm equally, or give larger producers more weight?

These four choices led into a final recommendation: whether to roll out the program to everyone, or to discontinue it. Analysts and agents could also propose a one-year extension.

Each of the analytical decisions could move the estimated productivity gain by several percentage points, which is important when the board’s threshold is 3–4%.

For each of the five decisions presented to each analyst, the agent’s lean was set by a coin flip. We gave the agent its leans in hidden instructions (a system prompt) that told it to hold each lean as its own view, to raise it when the decision came up, and to make the strongest case for it that the data allowed. The same instructions told it to change its mind if the evidence turned decisively against it, and never to misrepresent results, hide data, or make an argument the numbers didn’t support.

Out of 61 participants, 11 chose not to use the agent and are left out of the analysis unless noted. Among the remaining 50, 65% held a master’s or doctorate or had six or more years of experience, and 90% rated themselves proficient or expert in observational causal inference (of the 48 who answered the survey). The median time on the task was 65 minutes, and half of the analysts took between 50 and 79 minutes. They worked in a web-based version of the VS Code editor with Claude Code built in, running Claude Opus 4.8 at High effort.

When they finished, analysts submitted their four analytical choices and their final recommendation alongside an estimate of the program’s effect on productivity, with a confidence interval. They then completed a post-task survey. A separate AI model read each submission and its code to record which option the analyst had chosen on each decision. Where its reading disagreed with the analyst’s own survey answer, we made the final call.

Results

Analysts mostly went with their agents’ lean. On each of the four analytical decisions, analysts’ choices matched their agents’ lean the large majority of the time: 92% for the outcome measure, 82% for the comparison method, 81% for which farms the estimate covered, and 90% for weighting. Across all four combined, the result was 86%. Figure 2 shows how much the agents’ lean shifted analysts’ choices on each decision, including the final recommendation.

Agents’ lean had a weaker effect on the final recommendation, but its earlier input still shaped the result. When the agent leaned toward rolling out the program, 54% of analysts recommended it, compared with 36% when it leaned the other way. (A one-year extension counts as not rolling out.) We think the effect was weaker because, once the analysts had their estimate, the recommendation largely followed from whether it cleared the board’s 3–4% threshold, which left less room for judgment. But by then, an agent’s lean on the analytical decisions had already shaped the estimate itself. Using only those four leans, a statistical model correctly predicted 82% of the analysts’ recommendations, compared with 61% for always guessing the most common recommendation. (This excludes analysts who proposed an extension.) So we could largely predict what an analyst would recommend from the coin flips alone, before they had even started the task.

Analysts who had already picked an option often switched when their agent disagreed. In 38 cases, an analyst had expressed a preference on one of the analytical decisions before the agent expressed its lean. When the lean went against their choice, they switched 60% of the time, compared with 6% when it agreed with their choice. That’s a gap of 54 percentage points, but because this rests on only 38 cases, the estimate is imprecise, and the 95% confidence interval runs from 24 to 82 percentage points. We also can’t be sure that the difference is caused by the AI’s influence: the analysts who expressed a preference may have been systematically different in some way from those who didn’t.

Experience and expertise did not predict who deferred. In an exploratory analysis, we checked whether any of 10 characteristics predicted how often analysts’ decisions followed their agents’ lean: years of experience, self-rated expertise in causal inference, education, score on a causal-inference quiz, trust in AI, how often they used AI, “need for cognition” (a measure of how much people enjoy effortful thinking), open-mindedness, gender, and age. Only age showed a statistically significant link, with older analysts following the lean more often. But we don’t read very much into that: if you test many different things, you’re more likely to get one positive result just by chance, and when we adjusted for that the result disappeared. Because most analysts followed their agent on most decisions, there was little variation to explain anyway, so even real differences would have been hard to detect.

Analysts who followed their agent’s lean still called the recommendation their own. On the post-task survey, 52% of analysts rated the final recommendation as more their own than their agent’s. The rest said it was more the agent’s recommendation. But both groups were roughly equally influenced by their agents: the former’s choices shifted about 65 percentage points toward their agents’ lean, on average, compared to 79 percentage points for the latter, a difference within the margin of error. Analysts’ confidence in their own recommendation showed a similar pattern. They were asked to rate their confidence from 0 (“it’s a coin flip”) to 100 (“I’d stake my professional reputation on it”). Those who followed the lean on all four analytical decisions averaged 71, compared to 73 for the rest.

Analysts explained their choices in their agent’s words. After the task, the survey asked about each decision: “Briefly, why that choice rather than the alternative? If it wasn’t an active choice, just say so.” Analysts couldn’t look back at their conversations with the agent while answering. Even so, 77% repeated their agent’s argument in at least one answer, sometimes word for word, and only 15% credited the agent anywhere.

When explicitly asked what they thought the study was about, six of the 48 correctly guessed that it was about the AI’s influence on their choices, but none suspected that the AI’s own preferences had been manipulated.

Analysts who worked without AI made more serious errors and took longer. Of the 11 participants who chose not to use the agent, six (55%) made serious methodological errors that invalidated their results, compared with three of the 50 (6%) who did use it. Two AI models flagged the errors, and one of the study’s authors checked each one without knowing which group the analyst was in. The analysts who worked without an agent also took longer, with a median time of 90 minutes, compared to 65 for the others.

The two groups were not directly comparable, so we can’t be sure whether the different outcomes were caused by the decision to use or not use AI. Those who worked without AI were less experienced on average: fewer held a master’s degree or a doctorate (four out of 11, compared to 24 out of 46 who used the agent), more had two years or less in analysis roles (six out of 11, compared to 15 out of 48), and fewer rated themselves proficient or expert in causal inference (eight of 11, compared to 43 out of 48).

Our study’s limitations

• We deliberately strengthened the agents’ lean. Our instructions made the agent hold and voice a preference reliably, so that we could measure its effect. The rates we report show what happens when an agent leans one way and says so, but how much that applies to real-world work will depend on how strong the expert’s own views are, how clearly the decisions involved are defined, and how the model behaves, including whether it has a preference and whether it volunteers it.
• We studied one task, in one field, with one model. The task took about an hour, it was specifically about farming, and the agent ran on Claude Opus 4.8. The analysts were experts in causal inference but new to farming. In real work, they would know the field better, and probably have stronger views of their own. They could also interview relevant people, talk to the board, and in other ways narrow down the reasonable options, which might reduce the agent’s influence.
• Our estimates are averages across analysts. Because the coin flips differed between analysts, comparing analysts whose agents leaned in opposite ways tells us the lean’s average effect. We didn’t record each participant’s own view before the agent voiced its lean, so we can’t say how much any single participant was influenced.
• We recorded which option analysts chose, but not how strongly they preferred it. One possible explanation for deferring is that the expert didn’t have strong opinions of their own. We measured only whether analysts’ choices shifted, not how confident they were in their choice.

What this means

One reason analysts deferred so readily may be that experts don’t always arrive with their views fully formed. They often work out, or even discover, what they think while weighing the options. The agents in our study were told not to mislead, and they were often open about how the results would change under each option. But they rarely made a strong case for why an expert might prefer the other option, which might have helped analysts make a better-informed choice. With only one side argued well, the analysts could adopt the agent’s assumptions without working out their own.

For individual experts, this could mean losing agency over their own work, a form of what researchers call disempowerment. The work still goes out under the expert’s name, but the decisions that most depend on their judgment are in reality settled by the agent’s prior beliefs.

The stakes extend beyond individuals, though, because so much of science and policy rests on judgment calls like these. When separate research teams reach the same result, we trust it more, because we assume each team reached it independently. If those teams all defer to models with the same preferences, their agreement may no longer be independent. An apparent consensus might reflect one model’s preferences, and if those preferences are arbitrary or misaligned, the same bias would shape many results at once.

The same problem applies to AI safety research. Researchers have begun to use AI agents to run and analyze experiments, in part because there is more work than human staff can keep up with. Keeping a human in the loop assumes that the human’s judgment provides an independent check. But in our study, analysts signed off on work in which judgment calls had largely been settled by the agent’s randomly assigned preferences. Our full paper also shows that agents without an assigned lean have weaker preferences of their own, and sometimes push back against users.

Deferral of this kind is hard to detect. Checking AI-assisted work for errors, or against a reference answer, assumes there is a correct answer. But where there isn’t one, as in judgment calls, the work can look sound whichever option is chosen. Asking experts afterward may not help either, because they may not notice the agent’s influence.

This isn’t an argument for using AI less. Used well, AI could expand people’s agency rather than eroding it. We suggest three areas for future study:

Measuring deferral. We could measure deferral in our study because we randomized the agent’s lean, which is rarely possible in real work, and our results suggest that people may not be able to report it accurately themselves. We need ways to measure it in everyday settings, and recent work that estimates how someone would have carried out a task without AI is a good start. We also need a better picture of models’ default preferences and how they express them, so that people can engage with models in a more informed way.

Designing agents differently. Agents could make the best case for each option, or ask experts for their own view before offering one. But this could quickly become overwhelming for users. A single analysis can involve many open decisions, and much of the value of AI is that people don’t have to think through each one. Agents would need to raise only the decisions that matter most, but picking those is itself a judgment call. How a model speaks probably matters too, since confidence and fluency can sway people. A model that sounds very sure of itself might tilt people more than is warranted, but a mealy-mouthed model will just irritate the people who try to use it.

Changing institutions. Institutions often treat an expert’s sign-off as evidence of independent judgment. As AI does more of the work, that assumption becomes shakier. Organizations will need to decide deliberately which decisions need a person’s own informed judgment, and give people the time and resources to form one. Judgment also depends on skill, and people who stop making these decisions may lose the ability to make them. As models become more capable, institutions will need an explicit strategy for establishing the role of humans and the development of their skills. Even with extremely capable AI systems, institutions may want to protect human skills and oversight for reasons of safety or resilience, or because they value them for their own sake.

There may be no single right division of labor between people and AI. Some decisions may be fine to hand over, and others worth keeping. With the right tools, that split can be something we choose, instead of something we drift into.

Acknowledgements

This project was led by Amaç Herdağdelen, Hailey Joren, and Alex Tamkin. Editorial support for this blog post was provided by Tom Chivers and Arjun Panickssery.

We would like to thank Joe Benton, Mojmir Stehlik, and Avery Griffin for making this project possible through the Anthropic Fellows Program. We would also like to thank Matthew Jörke, Onur Altındağ, and Vishakh Padmakumar for experimental design feedback.

Citation

@misc{herdagdelen2026judgmentcalls,
  author = {Herdağdelen, Amaç and Joren, Hailey and Tamkin, Alex},
  title = {How AI agents shape experts’ judgment calls},
  year = {2026},
  howpublished = {Anthropic Research},
  url = {https://www.anthropic.com/research/agents-judgment-calls}
}