Here is a humbling fact about people: if you ask someone the same survey question two weeks apart, they do not always give the same answer. We are moody, busy and inconsistent. So when researchers ask "how accurate is an AI copy of a person?", the fairest yardstick is not perfection. It is how consistent that person is with themselves.
That is the clever benchmark at the heart of a Stanford (opens in a new tab) study led by Joon Sung Park (opens in a new tab), first released as Generative Agent Simulations of 1,000 People (opens in a new tab) and now titled LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals.
What they did
The team recruited "a diverse national sample of 1,052 Americans" and interviewed each one for about two hours. The average transcript ran to 6,491 words. They then built an AI agent for each person from their interview and surveys, and asked the agents to answer questions from the General Social Survey (opens in a new tab) that the people had also answered.
The result, from the current version of the paper: "interview-only, survey-only, and combined agents achieved accuracies equal to 83%, 82%, and 86% of participants' own two-week test-retest (opens in a new tab) consistency benchmark, respectively, compared with 74% for demographics-only agents."
Why the gap between 74 and 86 matters
Read those numbers again. An agent built from age, gender, race and politics alone does well. An agent built from a real conversation does noticeably better. As Park put it to Stanford HAI (opens in a new tab): "The beauty of having interview data is that it includes people's idiosyncrasies."
Idiosyncrasies are the whole game in customer research. Two 38-year-old fleet managers in Riyadh with the same salary can make opposite decisions because one of them got burned by a supplier last year. Demographics cannot see that. A life story can.
“Researchers had 1,000 people participate in two-hour interviews with an AI.”
The careful reading
If you saw headlines saying "85% accuracy," that was the first version of the preprint; numbers shifted slightly in the revision. And the authors are candid: "we do not yet fully understand why grounding agents in self-reports is so predictive, and the limits of this prediction." They also note demographic-only prompting does worse for cultures far from the US. The agents are not public, either: researchers must apply for access with strict privacy assurances, which we think is exactly right.
How it shaped Cast
This paper is one of the reasons AI users are not demographic stubs. Our Casting Agent writes each person a backstory grounded in public research on their market: their city, their work, what things cost them, what went wrong last month. AI users are not copies of real individuals (we never simulate a real, identifiable person), but they are built the way this research suggests matters: from texture, not just categories.
How the interviews were run
Each participant spoke with an AI interviewer that followed a semi-structured script adapted from a well-known oral history protocol, covering their life story, values, work, relationships and views on current issues. The interviewer could ask follow-up questions, which is why transcripts were long and personal. Those transcripts became the memory of each agent.
After the interview, participants completed a battery of measures, including General Social Survey questions, personality tests and economic games. Two weeks later they took the same measures again. That second round provided the yardstick: how consistent is a person with themselves?
Why personality and games were harder
The headline results focus on survey questions, where agents did best. The early coverage also reported correlations for personality tests and for economic games, with games noticeably lower. That makes intuitive sense. Answers to a survey question are often stable opinions. Choices in an economic game depend on momentary judgment, risk appetite and context, which are harder for anyone, human or AI, to predict.
What this means for building AI users
- Give every AI user a life, not a label: where they live, what they do all day, what they pay and what has frustrated them recently.
- Ask about past behavior first, because stories about real decisions carry more signal than opinions about hypotheticals.
- Use follow-up questions to go deeper, the way the AI interviewer in this study did.
- Look for idiosyncrasies in the answers. They are often where your best insight hides.
The ablation results most people miss
The headline numbers get the attention, but some of the most practical findings sit in the exploratory analyses. The researchers randomly removed 80% of each interview transcript, equivalent to cutting 96 minutes from a 120-minute conversation. The agents still reached a normalized accuracy of 0.79 on the General Social Survey questions. That is lower than the full interview, but still above the demographics-only agents.
They also turned each transcript into a bullet-point summary of facts ("only child", "married twice") with the person's own words removed. Those summary agents reached 0.81. In other words, much of the value seems to come from the information in an interview, not just the way people talk. For anyone building synthetic users, that is encouraging: well-chosen facts about a person's life carry a lot of signal.
Diminishing returns, and fairer results
The current version of the paper adds two findings worth knowing. First, combining interviews with surveys gave the best accuracy, but "gains over either source alone were modest," which the authors read as a sign that benefits "begin to asymptote once the model has observed sufficient evidence." More data helps, then levels off.
Second, the interview and survey agents reduced "accuracy disparities across racial and ideological groups relative to demographics-only agents." That fits with other work. Angelina Wang (opens in a new tab) and colleagues showed in Nature Machine Intelligence (opens in a new tab) that identity-based prompts tend to misportray and flatten groups. Grounding agents in what people actually said about their lives appears to be one way to push back on that.
Where the interview script came from
The interviews followed the schedule of the American Voices Project (opens in a new tab), a study led by researchers at Princeton (opens in a new tab), Stanford and Georgetown to build in-depth portraits of American life, covering daily routines, relationships, health and finances. The paper also gave each agent "expert reflections": short notes written by the model from the perspective of a psychologist, a behavioral economist, a political scientist and a demographer, drawing out implications the person had not stated directly.
We find that a lovely idea. It is essentially asking several specialists to read the same transcript and tell you what they notice.
What persona research says about labels
A separate study by Tiancheng Hu and Nigel Collier, Quantifying the Persona Effect in LLM Simulations (opens in a new tab), found that demographic, social and behavioral persona variables accounted for less than 10% of the variance in human annotations across the subjective datasets they studied. Adding those variables to prompts gave "modest but statistically significant improvements." The lesson is consistent with the Stanford work: labels help a little, and real circumstances help more.
A worked example
Imagine a founder building a payroll tool for small construction firms. Version one of her synthetic audience is "site manager, male, 45, Riyadh." Version two adds circumstances: he manages 30 workers on rotating contracts, pays partly in cash, was fined last year for a late filing, and checks everything on his phone between site visits.
The second version will produce answers about cash handling, mobile access and fear of penalties. The first will mostly produce generic enthusiasm. In Cast, the Casting Agent researches the market to build that second kind of person, and every AI user is labeled as an AI user so nobody mistakes the result for a real interview.
Glossary
Test-retest consistency: how similarly a person answers the same questions on two occasions, here two weeks apart. Normalized accuracy: an agent's accuracy divided by that human consistency, so 1.0 would mean the agent predicts you as well as you predict yourself. Ablation: removing part of a system to see how much it mattered. Big Five (opens in a new tab): a widely used model of personality covering openness, conscientiousness, extraversion, agreeableness and neuroticism.
Questions about interview-grounded agents
Does Cast simulate real individuals the way this study did?
No. The Stanford agents were built from interviews with consenting real people, under strict access controls. Cast builds fictional people from public research about their market and never simulates a real, identifiable person.
Why not just use demographics? 74% sounds good.
It is good, and it is a useful baseline. But the gap to 83 to 86% is exactly the texture that explains why two similar people make different decisions, which is what product teams need to understand.
Where can I read the paper?
The full paper is on arXiv, linked above, and Stanford HAI published an accessible summary.
Do I need a two-hour interview to build a good synthetic user?
No. In the Stanford study, agents built from only 20% of the interview, or from a bullet-point summary of facts, still beat demographics-only agents. What matters is relevant information about a person's life, not length for its own sake.
Did the agents predict experiment results as well as survey answers?
In five replication experiments, effect sizes from the agents were highly correlated with those from the real participants, though they were larger. The authors note five studies are too few to compare effect sizes properly.
Read next: the AI town that threw a Valentine’s Day party.
