Every researcher knows the feeling. You design an experiment, recruit hundreds of people, wait weeks, and the effect you were sure about turns out to be tiny. What if you could get a decent preview first?
That is the question behind Large language models can predict the results of social science experiments (opens in a new tab), published in Nature in July 2026 by Ashwini Ashokkumar (opens in a new tab), Luke Hewitt (opens in a new tab), Isaias Ghezae (opens in a new tab) and Robb Willer (opens in a new tab).
A very big test
The team gathered "70 preregistered, nationally representative survey experiments in the USA involving 469 experimental effects and 119,330 participants." Then they asked GPT-4 to predict how people would respond to each treatment, and compared those predictions with what actually happened. The predictions were strongly correlated with the real effects, and their accuracy was similar to forecasts pooled from human experts.
Here is the part that convinced us the model was not just remembering papers: the correlations held for studies that had not been published when the model was trained. As Hewitt told Stanford HAI (opens in a new tab): "we found the LLM could make fairly accurate predictions even for entirely novel experiments."
You can even try it yourself: the authors built a public demo at treatmenteffect.app (opens in a new tab) that predicts treatment effects on U.S. adults.
The caveats are in the abstract, which we love
The paper does not oversell. Its abstract says plainly: "Despite high correlations, predictions systematically overestimated effect sizes." On a second archive of 15 megastudies with 606 effects, "correlations were lower but comparable to those of pooled expert forecasters." The researchers also surveyed 460 social scientists about uses and risks.
One more honest note: if you read older coverage you will see slightly different numbers (476 effects, r = 0.85). Those come from the 2024 preprint, described on the lab's project page (opens in a new tab). The published Nature version is the one to cite.
What this means for founders
If AI can rank which of your ideas will move people, and roughly how much, you can afford to test ten ideas instead of one. You still validate the winner with real customers, but you walk into that study with a sharper hypothesis and a smaller budget.
There is a practical lesson in "systematically overestimated," too. Cast treats percentages from AI users as a ranking of options, not a forecast, and says so in every report. If eight of twelve AI users object to night charging, that objection matters. Whether it is exactly 67% in the market is a question for real customers.
Why this study is different from earlier ones
Earlier work showed that language models could reproduce famous findings. A fair criticism was that those findings are all over the internet, so the model might simply remember them. This study was designed to answer that criticism. It used a large archive of survey experiments run on nationally representative samples, including many that were never published, and checked whether predictions held up for work the model could not have seen.
It also compared the model with people whose job is to predict these things: social scientists. When the model's forecasts are about as good as pooled expert forecasts, and much faster, it becomes a practical tool for deciding what to test.
What "overestimated effect sizes" means for you
Suppose you test two headlines for a landing page. An AI prediction might say headline B is much more persuasive than A. The research suggests the ranking (B beats A) is likely to hold, but the size of the gap will probably be smaller in real life. In practice, that means you should use AI predictions to choose which variants deserve real traffic, and expect real-world differences to be more modest.
A simple workflow inspired by this research
- Write down three to five ideas you are genuinely unsure about, such as messages, prices or features.
- Run them through Cast with the same audience and compare the patterns and reasons, not just the percentages.
- Drop the ideas that clearly lose. Keep the top one or two.
- Test those with real customers. Expect smaller differences than the AI suggested, and plan your sample size accordingly.
Humans overestimate effects too
"Systematically overestimated effect sizes" sounds like a flaw peculiar to AI. It is not. Stefano DellaVigna and Eva Vivalt studied 100 projects posted on the Social Science Prediction Platform (opens in a new tab), a site that collects expert forecasts of research results before studies are run. In their NBER (opens in a new tab) working paper, Forecasting Social Science: Evidence from 100 Projects (opens in a new tab), they report that forecasters also tend to overestimate treatment effects, while the average forecast still tracks actual results well. They also found that higher confidence went with lower accuracy.
So the AI is making a very human mistake. That is reassuring in one way and a warning in another. Whether your forecast comes from a model, an adviser or your own gut, the effect you expect is probably bigger than the one you will get.
Why most ideas do less than you hope
If you want a vivid reminder, read the exercise megastudy (opens in a new tab) led by Katherine Milkman and published in Nature in 2021. Thirty scientists from 15 US universities designed 54 different four-week programs to encourage 61,293 members of an American gym chain to exercise. Only 45% of the interventions significantly increased weekly gym visits, and only 8% produced change that was significant and measurable after the four weeks ended.
That is a sobering hit rate from experts working on a well-studied problem. It is also the strongest argument for cheap pre-testing: if most ideas will fail, the winning move is to test more of them, earlier.
What a prediction can and cannot tell you
- Ranking: which of your variants is likely to do best. This is where the evidence is strongest.
- Direction: whether a change will help or hurt. Usually reliable for clear, simple manipulations.
- Size: how big the effect will be. Expect real effects to be smaller, and plan sample sizes as if they will be.
- Subgroups: whether different kinds of people react differently. The Nature paper reports accuracy for US demographic subgroups separately, but treat these as leads to investigate.
Why averaging helps
One detail in the paper's extended data deserves more attention: averaging across more prompts improved predictive accuracy. A single prompt, like a single interview, is noisy. Many differently worded prompts, like many different people, cancel some of that noise out.
This is the same reason Cast never relies on one AI user. Each study interviews a whole cast in parallel, Fuzziness keeps the AI users genuinely different from each other, and skeptics are always included. Findings come from the pattern across people, not from one persuasive voice.
A worked example: planning a real test
Imagine a founder of a language-learning app who wants to test three onboarding messages: one about speed, one about confidence at work and one about family abroad. A Cast study suggests the confidence message resonates most with working professionals, while the family message wins with older users.
Rather than treat that as the answer, she uses it to design a real test. She keeps the two strongest messages, drops the third, and plans her experiment assuming a modest difference, because both humans and models tend to overestimate. A quick look at statistical power (opens in a new tab) shows why this matters: smaller expected effects need larger samples to detect. Starting with two variants instead of three makes that sample easier to reach.
Glossary
Treatment effect: the difference in outcomes between people who saw a change and people who did not. Effect size: how large that difference is, often in standardized units, as explained here (opens in a new tab). Preregistration: publishing your hypotheses and analysis plan before collecting data. Megastudy: a large experiment comparing many interventions in the same population on the same outcome.
Questions about predicting experiments
Which experiments did the model predict worst?
The abstract reports that correlations were lower on a separate archive of large megastudies, though still comparable to expert forecasters. Complex, multi-condition designs are harder to predict.
Does this mean we no longer need experiments?
No. The authors frame AI as a way to augment research, help choose what to test and pilot designs, not as a replacement for running studies.
Is there a way to try the model's predictions myself?
Yes, the authors released a public demo at treatmenteffect.app, linked above.
Are AI forecasts better than asking experts?
In this research they were comparable to pooled expert forecasts, and much faster to obtain. Both tend to overestimate effect sizes, so neither replaces running the experiment.
How many variants should I test with real customers after a Cast study?
Usually two or three. Use Cast to drop the clear losers, then test the strongest options with real people, planning your sample size for a smaller effect than the AI suggested.
Read next: why two-hour interviews beat demographics.
