Psychology has a greatest-hits album: the Ultimatum Game (opens in a new tab), the Milgram (opens in a new tab) obedience study, the wisdom of crowds. What happens if you play those tracks to a language model?
The Turing Experiments
Gati Aher (opens in a new tab), Rosa Arriaga (opens in a new tab) and Adam Tauman Kalai (opens in a new tab) proposed a neat test they call a Turing Experiment: instead of asking whether one AI can pass for one human, ask whether a simulated group reproduces a classic human study. In their ICML 2023 paper, Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies (opens in a new tab), the Ultimatum Game, garden-path sentences and the Milgram shock experiment were all replicated "using recent models."
The fourth study is the fun one. On wisdom-of-crowds questions, newer models were too good. The authors describe a "hyper-accuracy distortion": the simulated crowd knew the right answer far more often than any real crowd would. Real people are wonderfully wrong in ways a model has to learn to be.
Machine psychology grows up
Marcel Binz (opens in a new tab) and Eric Schulz (opens in a new tab) took the opposite route in PNAS (opens in a new tab): they gave GPT-3 the standard battery of cognitive psychology tests. It "outperforms humans in a multi-armed bandit task," yet "small perturbations to vignette-based tasks can lead GPT-3 vastly astray." Two years later, their team trained a model called Centaur on Psych-101, a dataset of "trial-by-trial data from more than 60,000 participants" making "in excess of 10,000,000 choices in 160 experiments." Published in Nature (opens in a new tab), Centaur predicted held-out human behavior better than existing cognitive models.
And the pushback, which is healthy
Not everyone was convinced. Jeffrey Bowers (opens in a new tab) at the University of Bristol (opens in a new tab) called the Centaur claims absurd (opens in a new tab), noting its "short-term memory and reaction times were considerably different to those of the human mind." Another team found Centaur kept performing well even when task instructions were removed, concluding it "does not truly understand instructions" (National Science Open (opens in a new tab)). And Peter Park (opens in a new tab) and colleagues showed a model calling itself conservative in 99.6% of runs, then liberal in 99.3% when the answer order flipped (arXiv (opens in a new tab)).
Zhicheng Lin (opens in a new tab)’s summary in Six Fallacies in Substituting LLMs for Human Participants (opens in a new tab) is where we land too: use these models as "supplementary rather than primary" tools, and validate against human data.
What we take from the greatest hits
Three practical rules for Cast. Randomize answer order, because order effects are real. Write questions that ask about lived experience, which is where simulated people are strongest. And never mistake confidence for truth, which is why a separate Critic reviews every line before it reaches your report.
Why replicating old studies matters at all
Psychology went through a painful replication crisis (opens in a new tab) in the 2010s, when large coordinated efforts found that many famous effects were smaller than reported, or did not come back at all. That history makes AI replications more interesting, not less. If a model reproduces an effect that also holds up in big human replications, it is probably capturing something real about how people reason. If it reproduces an effect that failed with humans, it may simply be repeating the textbook.
Textbook knowledge versus behavior
This is the trap. Language models have read the original papers, the lecture notes and the blog posts about them. When a model "behaves" like a Milgram participant, is it simulating a person under pressure, or recalling what the textbook said people did? Researchers try to separate the two by changing names, settings and numbers so the scenario no longer matches anything famous. Results that survive those changes are much more convincing.
The same logic applies to your research. A question that sounds like a well-known case study ("would you switch to a cheaper airline?") invites a textbook answer. A question grounded in a specific, unfamiliar situation ("your fleet contract renews in March and the new quote is 12% higher") invites reasoning.
Order and wording effects
People are affected by the order in which options appear, a pattern related to the serial-position effect (opens in a new tab). Models show order effects too, sometimes stronger ones. That is why Cast randomizes option order across AI users and treats a finding as solid only when it holds across different phrasings.
What this means for your study design
- Write scenarios that are specific to your market, not famous examples the model may recognize.
- Randomize the order of options and statements.
- Ask the same important question two ways, and trust only the patterns that agree.
- Use AI runs to debug your questionnaire before you spend money on a human panel.
A short glossary for AI replications
Turing Experiment. Aher, Arriaga and Kalai's name for testing whether a group of simulated participants reproduces a classic human result, rather than whether one AI can fool one judge.
Machine psychology. A research program, set out by Thilo Hagendorff, Marcel Binz, Eric Schulz and colleagues in Machine Psychology (opens in a new tab), that applies psychology experiments to language models. The authors also highlight "the important caveats of applying methodologies designed for understanding humans to machines."
Hyper-accuracy. When a model is more correct than real people would be, which makes it a poor stand-in for a real, fallible crowd.
Contamination. When a model has seen a study, or discussions of it, during training, so it can repeat the famous result instead of reasoning its way to it.
When models grew out of human mistakes
One of the clearest examples of hyper-accuracy comes from the Cognitive Reflection Test (opens in a new tab), first described by Shane Frederick in 2005. Its puzzles are built to trigger a quick, wrong gut answer. In Nature Computational Science (opens in a new tab), Thilo Hagendorff, Sarah Fabi and Michal Kosinski found that as earlier GPT models grew, they increasingly displayed "human-like intuitive system 1 thinking and associated cognitive errors." Then ChatGPT models arrived and tended to respond correctly, "avoiding the traps embedded in the tasks."
For a scientist, that is fascinating. For anyone doing customer research, it is a warning. Real customers make snap judgments, misread offers and skim pricing pages. A simulated customer who calmly works out the best deal is not being realistic, it is being too clever. That is one reason Cast builds AI users from concrete circumstances (a busy week, a tight budget, a habit they like) instead of asking a model what a rational buyer would do.
Which humans are being replicated?
Most classic psychology studies were run with university students in Western countries. Joseph Henrich, Steven Heine and Ara Norenzayan made this famous in The weirdest people in the world? (opens in a new tab), arguing that Western, educated, industrialized, rich and democratic samples are unusual, not typical. So when an AI "replicates" a classic study, it may be replicating a narrow slice of humanity very well. Cast tries to counter this by researching the market, city and circumstances of each AI user before the interview starts.
A worked example: disguise the classic
Imagine an edtech founder who wants to know whether showing her price as "per day" instead of "per month" changes how parents react. This is a cousin of the framing effect (opens in a new tab)), and a model has read plenty about it. Asking "does daily pricing feel cheaper?" invites a textbook answer.
A better hypothetical design runs two separate casts with the same backstories. One sees the offer framed daily, the other monthly, and neither is told a comparison exists. She asks the same question in both: "Would you sign up this week, and what would stop you?" If the reasons differ in a consistent way, she has a pattern worth testing on a real landing page. If they do not, she saved herself a design sprint.
Limitations, and what to do next
- A replicated classic is evidence about the model, not about your market. It tells you the tool behaves sensibly, not that your idea is good.
- Famous effects are the most likely to be memorized, so trust disguised versions more than textbook ones.
- If you plan to publish, preregister your hypothesis and confirm the result with human participants recruited through a panel provider such as Prolific (opens in a new tab).
- Keep the AI and human results side by side in your write-up, and say which is which.
Questions about AI replications
If the AI reproduced Milgram, is it obedient?
No. It reproduced the pattern of a group of simulated participants. AI users are designed to push back, and skeptics are cast on purpose.
Should I trust results that match famous studies more?
Trust results that hold up when the scenario is disguised and the wording changes. Matching a famous result is a good sign only if the model could not simply have remembered it.
What is a Turing Experiment in AI research?
It is a test of whether a group of simulated participants reproduces the pattern of a known human study, proposed by Aher, Arriaga and Kalai. It asks about groups and distributions, not whether a single AI seems human.
Can I use AI participants instead of humans in an academic study?
Most researchers treat them as a pilot or a supplement, not a replacement. Use them to sharpen your materials and predictions, then collect human data and report both clearly.
Read next: why the fastest way to fail is building first.
