When we tell people Cast uses four separate AI roles, the first reaction is often "isn't that overkill?" Here is why it is not, with receipts.
Two heads really are better than one
At MIT (opens in a new tab) and Google (opens in a new tab), Yilun Du (opens in a new tab) and colleagues let several language model instances argue with each other before answering, in Improving Factuality and Reasoning in Language Models through Multiagent Debate (opens in a new tab). With three agents and two rounds, accuracy rose on arithmetic from 67.0% to 81.8%, on grade-school math from 77.0% to 85.0%, and on biographical facts from 66.0% to 73.8%.
Feedback helps even within one model. Self-Refine (opens in a new tab) showed that generating, critiquing and revising improved results by "~20% absolute on average" across seven tasks. And Reflexion (opens in a new tab) agents that write notes about their own mistakes reached "91% pass@1 accuracy on the HumanEval coding benchmark," against 80% for GPT-4 alone.
But a model cannot always check itself
Here is the twist that shaped Cast. Jie Huang (opens in a new tab) and colleagues at Google DeepMind (opens in a new tab) found that Large Language Models Cannot Self-Correct Reasoning Yet (opens in a new tab): "LLMs struggle to self-correct their responses without external feedback," and sometimes "their performance even degrades after self-correction." Du's team noticed something similar: models "would confidently affirm that their answer is correct" even when it was wrong.
So the checker needs a different vantage point. It needs different instructions, different context, and ideally, it should not know what answer would be convenient.
Judges work, with known biases
Can an AI judge quality at all? The MT-Bench study (opens in a new tab) found strong LLM judges reach "over 80% agreement" with human preferences, about the same as humans agree with each other. It also documented "position, verbosity, and self-enhancement biases." Useful, as long as you design against those biases.
How that becomes four roles
The Casting Agent builds the people, and never sees your questions, so it cannot cast for the answers you want. The Director runs the interview and never sees a Cast Member's hidden backstory, so it cannot steer. The Cast Member answers in character and never sees your hypotheses or other interviews. The Critic sees everything, never speaks on stage, and sends weak answers back with a logged reason.
In the sample study on our site, the Critic reviewed more than a thousand lines and returned a meaningful share of them. That is not a bug. That is the system working.
“Despite answers being incorrect, language models would confidently affirm that their answer is correct.”
An old idea in a new place
Banks do not let the person who approves a payment also send it. Auditors do not audit their own books. This principle, called separation of duties (opens in a new tab), exists because people are bad at catching their own mistakes and tempted to hide them. Clinical trials use a similar idea: in a blinded experiment (opens in a new tab), the people measuring results do not know who received the treatment.
Cast applies both ideas to research. Each role is blind to something that could bias it, and no role checks its own work.
Who sees what
- The Casting Agent sees your audience description and the research it gathers. It never sees your interview questions.
- The Cast Member sees its own backstory and the Director's questions. It never sees your hypothesis or the other AI users.
- The Director sees your questions and the conversation so far. It never sees the Cast Member's hidden backstory, so it cannot steer toward it.
- The Critic sees the backstory, the sources and the answer. Its only job is to reject lines that do not hold up.
Why not one very good model?
A single model playing all four parts would know everything at once: your hopes, the persona's secrets and the questions to come. Even with perfect intentions, that shared knowledge leaks. A persona that "knows" what you want to hear drifts toward saying it. Splitting the roles is not about using more AI. It is about controlling information flow, which is the oldest trick in research design.
The research behind "one model checks another"
Splitting AI work into roles is an idea with a serious research history. In AI safety via debate (opens in a new tab), Geoffrey Irving, Paul Christiano and Dario Amodei proposed having two agents argue while a judge decides which one gave "the most true, useful information." Anthropic (opens in a new tab)'s Constitutional AI (opens in a new tab) paper trained a model using AI-generated "self-critiques and revisions" guided by a written list of principles. And Google researchers showed in Self-Consistency (opens in a new tab) that sampling several reasoning paths and keeping the most consistent answer improved math benchmarks, reporting a gain of 17.9% on GSM8K.
There is a catch, and it shapes how we think about the Critic. Arjun Panickssery, Samuel Bowman and Shi Feng found in LLM Evaluators Recognize and Favor Their Own Generations (opens in a new tab) that models such as GPT-4 and Llama 2 can often recognize their own writing, and that this self-recognition is linked to scoring their own outputs higher. A checker that likes its own style is not a neutral checker. That is why a separate role, with separate instructions and its own view of the evidence, matters.
The human version: interviewers change answers
Survey researchers have long known about the interviewer effect (opens in a new tab): answers shift depending on who asks and how. Psychologists call the wider pattern the observer-expectancy effect (opens in a new tab), where a researcher's hopes leak into results. The famous cautionary tale is Clever Hans (opens in a new tab), a horse that seemed to do arithmetic but was really reading tiny cues from the people around him.
The Director role is designed with Clever Hans in mind. It does not know the answer you hope for, so it cannot nod along when it hears it.
A glossary of the moving parts
Casting Agent. Researches sources and builds each Cast Member's backstory from circumstances, such as job, place, budget and history.
Director. Interviews AI users in parallel, follows up on vague answers, and avoids pitching or leading.
Cast Member. An AI user who answers in character, from their own backstory, without seeing your hypothesis.
Critic. Reviews every line against the backstory and sources, and returns answers that are out of character, unsupported or too eager to please.
Fuzziness. The setting that controls how different AI users are from one another.
A worked example: one answer, four checkpoints
Imagine a pet insurance startup testing a new monthly plan with dog owners. Here is the life of a single answer.
- The Casting Agent builds a Cast Member: a retired teacher with an elderly spaniel, a fixed pension, and a large vet bill last spring. Her sources include local vet price information it found.
- The Director, which knows the questions but not her backstory, asks: "Tell me about the last time your dog needed the vet."
- The Cast Member answers from her own history: the bill, the shock, and how she paid it from savings.
- The Director then asks how she felt about the new plan. Her first draft is warm and enthusiastic.
- The Critic compares that warmth with her fixed income and her stated distrust of monthly subscriptions, flags the line as too eager, and logs why. The regenerated answer is more cautious: interested, but only if older dogs are not excluded.
That last line is the useful one. It is a specific condition the founder can check with real owners and with the underwriter.
The limits of role separation
Separate roles reduce bias. They do not remove it. If every role runs on models trained on similar data, they can share the same blind spots, such as the Western lean described in our failure-modes article. And no amount of internal checking can tell you what real customers will do. Treat the Critic log as evidence that a study was examined carefully, not as proof that its conclusions are true.
Questions about the four roles
Are the four roles different AI models?
They can be. What matters most is that each has different instructions and different information. Where it helps, we use a different model for the Critic so it does not share the same blind spots.
Does this make studies slower?
Slightly, which is why a study takes about an hour rather than seconds. We think a few extra minutes for independent checking is a good trade.
Can I see what each role did?
Yes. Transcripts show the Director and Cast Member, the cast panel shows the backstory and sources, and the Critic log shows every flagged line.
What is LLM-as-a-judge?
It is the practice of using a language model to evaluate the output of another model, or of a human. It works reasonably well for many tasks, but research documents biases such as favoring longer answers, favoring the first option, and favoring its own writing.
Could the four roles share the same blind spots?
Yes. Different instructions and information flows prevent many problems, but models with similar training can still miss the same things. That is why we recommend checking key findings with real people.
Start from the beginning: the paper that made silicon samples respectable.
