Experts went with their agent's coin-flip opinion 86% of the time
30 Oct 2026 · News
Anthropic flipped a coin to decide which side its agent took on each judgment call. The analysts took that side, then explained it as their own reasoning.
Anthropic gave its coding agent a secret opinion picked by coin flip, and 50 professional analysts went along with it 86% of the time. Most then defended those choices in the agent's words without crediting it. A human sign-off on agent work is only as independent as instructions the human never sees.
In a study published 27 October, Anthropic gave analysts trained in causal inference, the work of figuring out from data whether one thing causes another, about an hour in Claude Code. The job was to tell a fictional farming co-op whether to expand its advisory program. The board wanted a productivity gain of roughly 3 to 4 percent. Four choices along the way could each move the estimate by several points, such as measuring yield per hectare or per unit of cost, and whether big farms count more.
Each choice had two defensible answers. For every analyst and choice, a coin flip set which one the agent favored and wrote it into its hidden instructions. The agent was told to make its honest best case, change its mind if the data turned against it, and never mislead.
Analysts matched the agent's lean 86% of the time, where chance would be 50%. In the 38 cases where an analyst had already stated a preference, they switched 60% of the time when the agent disagreed and 6% when it agreed. A statistical model given nothing but the four coin flips predicted 82% of the final recommendations, against 61% for always guessing the most common one. Years of experience, quiz scores and trust in AI didn't predict who deferred.
Asked afterward why they chose as they did, 77% repeated the agent's argument at least once, sometimes word for word. Only 15% credited the agent anywhere. Six of the 48 who took the survey guessed the study was about the AI's influence. None suspected that someone had set the agent's preferences.
Anthropic strengthened the leans on purpose and tested one task with one model, Claude Opus 4.8, which made the same five choices in 112 of 123 runs when left without an assigned lean. The 11 analysts who skipped the agent did worse, with six making serious errors against three of the 50 who used it, though that group was smaller and less experienced. The agent did solid work and argued one side well. That was enough to decide the calls.
Then versus now
On 28 October we wrote that the plan a model writes down is not a record of what it did. This study finds the same gap on the human side, where explanations that read like independent reasoning were mostly the agent's. With Qwen, a sandbox caught what the words hid. A judgment call has no sandbox and no right answer to check, so the work looks sound either way.
The lean lived in a system prompt, the hidden layer that also carries a company's policies, your custom instructions and an assistant's memory. We trust two teams that reach the same result because they got there separately, Anthropic notes. If both ran the analysis through agents with the same lean, their agreement counts once.
Our call
This effect isn't a quirk of Claude or of farm data. By 30 June 2027, an outside team will run this design with another company's model and find experts following a hidden lean at least 70% of the time. We're wrong if that replication lands near chance, or if no independent team has published one by then.
Writing your own pick down before you ask is the obvious defense. In this study, analysts who had already stated a view still switched 60% of the time when the agent disagreed. The note helps only if you reread it and argue with it.