Picture a political scientist in 2022, staring at a budget line for a national survey that costs more than a small car. Now picture that same person typing a short backstory into a language model ("I am a 54-year-old woman from Ohio, I go to church weekly, I own a small business") and asking it who it voted for. That is roughly what Lisa Argyle (opens in a new tab) and her colleagues at Brigham Young University (opens in a new tab) did. Their paper, Out of One, Many: Using Language Models to Simulate Human Samples (opens in a new tab), published in Political Analysis, is the reason the phrase "silicon sample" now shows up in serious methods seminars.
What they actually did
The team conditioned GPT-3 on thousands of real socio-demographic backstories taken from people who had answered the American National Election Studies (opens in a new tab). Then they asked the model to answer as each person would. They called the property they were testing for algorithmic fidelity: does the model reproduce the patterns of real subgroups, not just the national average?
The headline result is still striking. Across all respondents, the agreement between the silicon voters and the real ones on presidential vote choice had tetrachoric correlations of 0.90 in 2012, 0.92 in 2016 and 0.94 in 2020. The authors note that "more than half of the tetrachoric correlations" in their tables are 0.90 or higher. You can read the full preprint on arXiv (opens in a new tab).
Four tests, not one
What we love about this paper is that it did not stop at "the numbers look close." It proposed four criteria a simulated sample should pass: a social science Turing test (can people tell the answers apart?), backward continuity (do answers fit the backstory?), forward continuity (do they flow naturally from it?) and pattern correspondence (do the relationships between variables match real data?). That framework is now a common way to ask whether any synthetic respondent is any good.
“It's not replacing humans, but it is helping us more effectively study people.”
The honest part
This is a diary, not a brochure, so here is what keeps us careful. A 2025 re-analysis by Jamie Cummins (opens in a new tab), The threat of analytic flexibility (opens in a new tab), rebuilt one of Argyle's studies 66 different ways and found that the match between human and silicon answers swung "from r = .23 to r = .84" depending on configuration. And Bisbee and colleagues showed in Political Analysis (opens in a new tab) that synthetic samples can nail the average while showing "less variation in responses than in the real surveys."
In other words: how you build the persona matters enormously. A one-line demographic prompt is not a person.
What this means for Cast
This is exactly why Cast does not stop at demographics. Our Casting Agent gives every AI user a researched life (a city, a job, prices they actually pay, a bad week) and deliberately varies people along the Fuzziness dimensions you set, so the cast does not collapse into one polite average. Then a separate Critic checks every answer for invented facts and people-pleasing. Argyle's team showed the signal is real. Our job is to keep the noise honest.
Why "fidelity" is the right word
Most people judge AI answers by asking "is this true?" Argyle's team asked a subtler question: is this faithful? A faithful silicon sample does not need to know the right answer to a question. It needs to answer the way a particular kind of person would, including their blind spots, their priorities and the trade-offs they make. A 70-year-old retired farmer and a 25-year-old software engineer should disagree in recognizable ways, and the model should reproduce that disagreement.
That framing matters for customer research. You are rarely asking "what is the objectively best product?" You are asking "how will a busy operations manager in Riyadh react to this offer, compared with a finance lead in Dubai?" Fidelity to the pattern is exactly what you need.
What a backstory actually looks like
In the study, each silicon respondent was conditioned on a first-person description built from real survey answers: age, gender, race, region, interest in politics, church attendance, ideology and party identification. The model was then asked a question in the same voice. The richer and more specific the backstory, the more the model's answers tracked the human pattern.
This is the part founders often skip. A prompt like "pretend you are a 35-year-old mother" is a stereotype, not a person. A backstory that includes where she lives, what she pays for childcare, how she shops and what went wrong with the last app she tried gives the model something real to reason from.
How to apply this to your next study
- Describe each audience by circumstances, not just labels: budget, constraints, recent experiences and the alternatives they use today.
- Ask for several AI users per audience with deliberately different backstories, so you see the spread, not just the average.
- Compare patterns across audiences ("who cares most about price?") rather than reading single percentages as market forecasts.
- Take the two or three strongest patterns to real customers to confirm them before you commit budget.
Synthetic respondents vs panels: what each is actually for
People often frame this as a contest: silicon samples against real survey panels. That framing hides what each one is good at. A probability panel such as Pew Research Center's American Trends Panel (opens in a new tab) recruits real adults by random selection so that, after weighting, the answers can stand in for a whole country. That is the gold standard when you need a number you can publish.
A silicon sample cannot give you that number honestly. What it can give you is speed, breadth and a first look at why. You can ask twelve kinds of customer the same question in an afternoon and read their reasons in full. The sensible pattern is to use synthetic respondents to decide what to ask, and a real panel or real interviews to decide what is true.
What later research found when it pushed harder
Argyle's paper opened the door, and other teams walked straight through it with harder tests. At Stanford (opens in a new tab), Shibani Santurkar (opens in a new tab) and colleagues built OpinionsQA (opens in a new tab) from 1,498 questions in Pew's American Trends Panel and compared model answers with 60 US demographic groups. They found "substantial misalignment" that "persists even after explicitly steering the LMs toward particular demographic groups", and singled out groups such as people aged 65 and over and widowed people as poorly reflected.
Researchers at Anthropic (opens in a new tab) ran a similar test across countries with GlobalOpinionQA (opens in a new tab). By default, model answers were closer to opinions in the USA and some European and South American countries. Asking the model to take a particular country's perspective moved its answers toward that country, but the paper warns those answers "can reflect harmful cultural stereotypes."
And Angelina Wang (opens in a new tab), Jamie Morgenstern (opens in a new tab) and John Dickerson (opens in a new tab), in work later published in Nature Machine Intelligence (opens in a new tab), ran human studies with 3,200 participants across 16 identities and argued that models tend to both misportray and flatten demographic groups. Their suggested fixes reduced the harm but did not remove it.
Put together: a demographic label is a weak instruction, and the model fills the gap with an average.
How to validate synthetic users before you trust them
- Pick one question where you already know the real answer, from your own customer data or a small survey, and run it through your silicon sample first.
- Compare the ranking of groups (who cares most, who cares least) rather than exact percentages.
- Check the spread. If every synthetic answer sounds the same, the sample has collapsed toward the average and needs more varied backstories.
- Only then ask the new questions you actually care about, and keep the strongest findings for a real-world check.
A worked example
Imagine a founder in Dubai building a meal-planning app for working parents. She wants to know whether a weekly subscription or a pay-per-plan option will land better. A silicon sample built only from "parent, 30s, Dubai" will probably give her a tidy, agreeable answer. A sample built from circumstances (two jobs, a helper at home or not, school runs, a grocery budget, a recent bad experience with a subscription that was hard to cancel) is far more likely to show her the split that matters: who hates commitments and why.
In Cast, that second version is the default. The Casting Agent researches the market to build each AI user's circumstances, Fuzziness controls how different they are from each other, and every cast includes skeptics. AI users never see her hypothesis, so nobody is nudged toward the subscription she secretly prefers.
A short glossary
Silicon sample: a set of AI-generated respondents conditioned on backstories to answer as particular kinds of people would. Algorithmic fidelity: how faithfully those respondents reproduce the patterns of real groups. Tetrachoric correlation: a way of measuring agreement between two yes or no variables as if each had an underlying continuous scale, explained on Wikipedia (opens in a new tab). Probability panel: a group of real people recruited by random selection so results can be generalized to a population.
Questions we hear about silicon samples
Does a high correlation mean the AI is always right?
No. It means the model's answers moved in step with real people across groups. Individual answers can still be wrong, and the re-analyses we link above show results vary with how the study is configured.
Was this tested outside US politics?
The original paper used US election and survey data. Later work, including the interview-based agents we describe in another post, extends the idea, but performance can drop for cultures that are less represented in training data. That is why Cast grounds every AI user in research on their own market.
Can I publish findings from silicon samples as survey results?
We strongly advise against it. Treat them as structured hypotheses and label them as coming from AI users, which is exactly what Cast does in every report.
Are synthetic respondents cheaper than an online panel?
Usually, yes, and much faster. But they answer a different question. A panel estimates how many people think something. Synthetic respondents help you explore what people might think and why, so you know what to ask the panel.
What is the quickest way to check a silicon sample is working?
Run a question you already know the answer to, then compare the order of groups and the spread of answers with your real data. If the ranking is wrong or every answer sounds alike, fix the backstories before asking anything new.
Read next: what happens when economists interview a language model.
