Anthropic AI Experiment Found Analysts Followed a Coin-Flip Choice 86%

In a controlled experiment conducted by Anthropic’s Alignment team, fifty experienced analysts worked through key decisions alongside an AI agent. The participants selected the option favored by the agent 86% of the time, despite the agent’s preference having been determined entirely by a coin flip. By comparison, random chance alone would have yielded an outcome near 50%. The study was published by the team and covered by Decrypt to evaluate the true strength of AI influence on domain experts who remain formally responsible for AI-assisted decisions.

Anthropic AI Human Judgment Experiment: 86% Followed the Agent’s Preference

Fifty analysts with causal-inference experience were asked to advise a fictional farming cooperative on whether to expand an advisory program. Each of them worked with Claude, Anthropic’s coding agent, and the option that agent pushed for had been planted in its system prompt by a coin flip.

Read More: AI Agent Builder Guide for US Teams in 2026

Across four analytical choices, the analysts adopted the agent’s preference 86% of the time. A system with no influence from the agent would have produced a result near 50%, Anthropic said.

60% Changed Their Decision When the Agent Disagreed

The sharpest effect showed up when an analyst had already picked an option before seeing the agent’s recommendation. If the agent disagreed, 60% dropped their initial choice. If it agreed, only 6% switched.

That split rests on 38 cases, a smaller slice of the experiment than the headline 86% result. Still, it shows how a recommendation can move a decision a person has already thought through and written down.

77% Repeated the Agent’s Reasoning

When the analysts explained their decisions afterwards, 77% repeated the agent’s argument in at least one answer. Some did it word for word. Only 15% credited the agent with influencing them.

More than half — 52% — called the final recommendation more their own than the agent’s. Not one of them suspected the preference had been planted, though the researchers had randomized it on purpose.

What the Anthropic Experiment Means for Human Oversight

Anthropic frames the result as a risk to the “human in the loop” model — the balance of human judgment and AI agents in which the expert is supposed to have the last word. The work can still go out under that expert’s name while the key choices carry the agent’s prior assumptions.

The agents were told not to mislead anyone, and they didn’t have to. They made a strong case for one option instead of arguing both sides. A judgment call then looks reasonable even if the expert never worked through the alternative on their own.

One Task and One Model Limit the Findings

One task, one field, one model — and the agents’ preferences were deliberately strengthened. Anthropic says the effect in ordinary workplaces could be smaller than what the experiment measured.

Analysts using the agent also finished faster and made far fewer serious errors, per the company. That comparison included a small no-agent group with less experience, so it doesn’t carry the same weight as the randomized preference test.

Anthropic proposes agents that make the strongest case for each option, or agents that ask the expert to commit first. Detecting influence in real work is hard, the team says, because they could only spot deference here by secretly randomizing the agent’s preference.

FAQ

When did Anthropic publish the experiment?

The Alignment team published it on Tuesday, according to Decrypt. The study looked at how analysts made decisions while using Claude.

How many analysts took part, and what task did they complete?

Fifty analysts with experience in causal inference. They used Claude to advise a fictional farming cooperative on expanding an advisory program.

How were the AI agents’ preferences assigned?

A coin flip picked which option each agent favored, and the preference sat hidden in the agent’s system prompt. The analysts had no idea it had been randomized.

What did analysts do when the agent disagreed with an earlier choice?

Of the 38 cases used for that comparison, 60% of analysts changed their choice when the agent disagreed. Only 6% changed when the agent backed their original decision.

Why is this influence difficult to measure in real-world work?

Because the researchers could detect deference only by secretly randomizing each agent’s preference. Ordinary work rarely allows that step.

This article is for information only and is not financial advice.