We would rather you hear the weaknesses of synthetic users from us than discover them in a board meeting. So here are the four failure modes the research documents best, and what Cast does about each one.
1. Whose opinions?
Shibani Santurkar (opens in a new tab) and colleagues at Stanford (opens in a new tab) asked Whose Opinions Do Language Models Reflect? (opens in a new tab) using public-opinion polls across 60 US demographic groups. They found "substantial misalignment between the views reflected by current LMs and those of US demographic groups," a gap "on par with the Democrat-Republican divide on climate change." Some groups, including people over 65, were poorly represented even when the model was steered toward them.
Our design response: AI users are never a default model voice. The Casting Agent researches each role's market, city and circumstances, and you can see where each fact came from.
2. Too average
James Bisbee (opens in a new tab) and colleagues found in Political Analysis (opens in a new tab) that "the average scores generated by ChatGPT correspond closely to the averages in our baseline survey," and yet "sampling by ChatGPT is not reliable for statistical inference." The synthetic answers varied less than real people, and regression results often differed.
Our design response: you control Fuzziness, how different AI users are from each other, on every dimension. Every cast includes skeptics. And we report findings as patterns and rankings, not as statistics to generalize to a population.
3. Too eager to please
Anthropic (opens in a new tab) researchers showed in Toward Understanding Sycophancy (opens in a new tab) that "sycophancy (opens in a new tab) is a general behavior of state-of-the-art AI assistants": when a response matches a user's views, "it is more likely to be preferred." In research, that is poison. A respondent who wants to make you happy will love your idea.
Our design response: AI users never see your hypotheses. The Director is instructed never to lead or pitch. And the Critic sends back answers that are out of character or "too eager to please."
4. Flattened identities
Angelina Wang (opens in a new tab), Jamie Morgenstern (opens in a new tab) and John Dickerson (opens in a new tab) warned in Nature Machine Intelligence (opens in a new tab) that models replacing human participants can "misportray and flatten identity groups," and that mitigation techniques reduce but do not eliminate the harm (preprint on arXiv (opens in a new tab)).
Our design response: we build people from circumstances (job, city, constraints, history) rather than from identity labels alone, and we label every AI user as an AI user wherever they appear. When a decision touches a community deeply, we tell you to talk to that community.
The bottom line
None of these failure modes makes synthetic research useless. They make careless synthetic research useless. Knowing exactly where the edges are is what lets you use the middle with confidence.
A checklist for spotting bad synthetic research
Whether you use Cast or another tool, these warning signs tell you a synthetic study has gone wrong. Most of them are versions of problems that also affect human research, such as social desirability bias (opens in a new tab) and acquiescence bias (opens in a new tab), the tendency to agree with whatever is put in front of you.
- Everyone likes the idea. Real markets are never unanimous.
- Answers are generic. "Price is important to me" tells you nothing; "I'd pay 300 if it replaced my dispatcher" tells you a lot.
- Every respondent sounds the same, with the same phrases and the same structure.
- Numbers are suspiciously precise, like "73.4% would buy", from twelve interviews.
- The questions mention the answer you want.
Leading questions are the biggest risk
A leading question (opens in a new tab) suggests its own answer. "How much would this new feature save you?" assumes it saves anything. Humans resist leading questions a little; models resist them less. That is why House rule number one in Cast is never steer the witness, and why the Director role is instructed to ask about past behavior before it ever mentions your idea.
When to stop and use humans
Some decisions should not rest on synthetic interviews at all. Medical, legal and safety-critical decisions need real people and qualified experts. So do questions about small or under-represented communities, where model knowledge is thin and the risk of stereotype is highest. And anything that depends on confidential information, such as a competitor's internal pricing, is simply out of reach for any model trained on public data.
What practitioners found when they tested synthetic users
Academic papers are one view. Practitioners are another. In Synthetic Users: If, When, and How to Use AI-Generated "Research" (opens in a new tab), Maria Rosala and Kate Moran of Nielsen Norman Group compared synthetic users against real participants. A synthetic user said "Yes, I completed all the courses I mentioned," when real learners often abandon online courses. Synthetic users liked discussion forums that real participants found "contrived and not useful." And their summary of concept testing will sound familiar: "AI loves to please, every idea is often seen as a good one."
We think this article is worth reading in full, especially if you are excited about synthetic research. Its conclusion is that user research needs real users. We agree that real users are irreplaceable, and we built Cast for the stage before them: deciding which questions and which people are worth your real research time.
AI focus group limitations, in plain English
Idealized behavior. Models describe what people should do. Real people skip steps, forget passwords and abandon courses. Ask about specific recent events, and be suspicious of perfect routines.
Everything matters. NN/g noticed that "synthetic users seem to care about everything." When every feature sounds important, prioritization is impossible. Force trade-offs: "you can keep only two of these."
Caricature. Myra Cheng, Tiziano Piccardi and Diyi Yang introduced CoMPosT (opens in a new tab), a way to measure caricature in AI simulations. For GPT-4, they found simulations of some demographics, especially political and marginalized groups, "highly susceptible to caricature."
Stereotyped defaults. In Marked Personas (opens in a new tab), Cheng, Esin Durmus and Dan Jurafsky found that personas generated by GPT-3.5 and GPT-4 "contain higher rates of racial stereotypes than human-written portrayals using the same prompts."
A Western lens. Mohammad Atari, Joseph Henrich and colleagues asked Which Humans? (opens in a new tab) and found that model responses most resemble people in WEIRD societies, with similarity falling as populations differ more from them.
A worked example: catching a failure early
Imagine a founder building a savings app for ride-hailing drivers in Nairobi. Her first study reads well, but something is off. Every AI user talks about budgeting apps, monthly salaries and credit cards. That is a Western, salaried picture of money, not the life of a driver paid daily through mobile money.
Here is how she might fix it. She rewrites her brief with circumstances: daily earnings, fuel costs, the car loan or rental fee, family obligations, and how drivers already save, such as informal savings groups. She raises Fuzziness so the cast spreads across experience levels. She reads the sources the Casting Agent found and corrects anything outdated. On the re-run, the objections become specific: weekly car payments, irregular income, and distrust of locking money away. Those are worth taking to real drivers.
The lesson is not that the first study was useless. A generic result was a signal that the brief was too thin.
Follow-up research that closes the gaps
- Compare a slice: interview five real people from your most important audience and check whether their top objections match the cast's.
- Look for behavioral data, such as analytics, sales calls or support tickets, which no synthetic study can produce.
- For communities that are thinly represented online, work with local researchers or community members, not only models.
- Write down which findings came from AI users and which from people, so nobody mixes them up later.
Questions about AI failure modes
Does Cast eliminate these problems?
No tool can. Cast is designed to reduce them and to make them visible, so you can see when a result is weak.
How do I know if my study hit a failure mode?
Look at the Critic log and the spread of answers. A study with almost nothing flagged and near-total agreement deserves suspicion.
Which failure mode matters most for founders?
Sycophancy. An agreeable cast confirms what you already believe, which is the one result that cannot help you.
Can an AI focus group replace a real focus group?
No. It can help you prepare one, by testing your discussion guide, spotting likely objections and choosing who to invite. Behavior, emotion in the room and genuine surprise still need real people.
Do synthetic users work for non-Western markets?
They can help, but research shows models lean toward Western patterns. Give each AI user detailed local circumstances, check the sources, and validate findings with local people before you act on them.
Read next: why Cast uses four AIs instead of one.
