If you have ever edited someone else's writing, you know you see their mistakes more clearly than your own. Language models are the same, only more so. That simple observation is why Cast has a role whose only job is to doubt.
Why the checker must be separate
Researchers at Google DeepMind (opens in a new tab) put it bluntly in Large Language Models Cannot Self-Correct Reasoning Yet (opens in a new tab): models "struggle to self-correct their responses without external feedback," and sometimes "their performance even degrades after self-correction." Asking the same model to grade its own answer is like asking a student to mark their own exam. So the Critic is a separate role, with separate instructions, and it never speaks on stage.
What the Critic checks
For every line an AI user speaks, the Critic asks four questions. Is this in character? A cautious CFO who suddenly gushes gets flagged. Is any fact checkable, and does it hold? Claims are compared with the AI user's research brief. Was the question leading, or the answer too eager to please? Anthropic (opens in a new tab)'s research on sycophancy (opens in a new tab) shows assistants tend to tell people what they want to hear, so the Critic is trained to be suspicious of enthusiasm. Is it consistent? Answers that contradict something the person said earlier get returned.
What happens to a flagged line
A flagged answer is not deleted and forgotten. It is sent back with the reason, regenerated, and logged. You can see the log. In the sample Fleetwise study you can explore on our site, the Critic reviewed 1,180 lines and returned 83 of them, about 7%. Some were invented statistics. One was a self-described skeptic who suddenly called the idea "a great idea and we would definitely sign." That answer never reached the report.
There is good evidence that this kind of external review works. In Self-Refine (opens in a new tab), adding a feedback and revision step improved outputs by "~20% absolute on average" across seven tasks. And studies of LLM judges (opens in a new tab) found strong models agree with human preferences over 80% of the time, while warning about known biases like favoring longer answers, which is why our Critic scores substance, not length.
What you get
Findings you can defend. When a quote appears in your report, it has survived a reader whose job was to reject it. That is a much higher bar than "the model said so," and it is the bar we think customer research deserves.
Editors, red teams and second readers
Every serious field has a version of the Critic. Newspapers have editors. Security teams hire a red team (opens in a new tab) to attack their own systems. Qualitative researchers measure inter-rater reliability (opens in a new tab) by having two people code the same transcript and comparing results. The common thread is that a second, independent reader catches what the first one cannot see.
Four kinds of line the Critic rejects
- Out of character: a cautious buyer who suddenly becomes enthusiastic, or a junior employee who talks like a CEO.
- Unsupported facts: a price, statistic or claim that contradicts the AI user's sources.
- Too eager to please: praise without reasons, or agreement that arrives before the question is finished.
- Generic filler: answers that could have come from anyone, with no detail from the AI user's life.
Reading the Critic log
The Critic log is one of the most useful parts of a Cast study, and most people skip it. It shows which questions caused the most rejected answers. A question that keeps producing eager answers is probably leading. A question that keeps producing unsupported facts may be asking about something the cast cannot know. Use the log to improve your next questionnaire.
LLM as a judge, explained simply
"LLM as a judge" is the practice of asking one language model to evaluate the output of another. It has become popular because human review is slow and expensive, and because strong models turn out to be surprisingly good graders when they are given clear criteria. One well-known example is G-Eval (opens in a new tab) from Microsoft researchers, which asks a model to reason through explicit evaluation steps before scoring, and found this aligned better with human judgments than older automatic metrics. Anthropic's Constitutional AI (opens in a new tab) work showed a related idea: a model can critique and revise outputs against a written list of principles, rather than relying on a human to label every example.
The Critic in Cast is an LLM judge with a narrow, unglamorous job. It does not decide whether your idea is good. It decides whether each line an AI user speaks deserves to count as evidence.
The known weaknesses of AI judges
AI judges have biases of their own, and pretending otherwise would be a poor start for a role built on doubt. Peiyi Wang and colleagues showed in Large Language Models are not Fair Evaluators (opens in a new tab) that simply changing the order in which two answers appear could swing a judge's verdict, to the point where a weaker model appeared to beat a stronger one on 66 of 80 queries. Arjun Panickssery and colleagues found that LLM evaluators recognize and favor their own generations (opens in a new tab), which is a strong argument for not letting a model mark its own homework.
There is encouraging evidence too. OpenAI (opens in a new tab)'s LLM Critics Help Catch LLM Bugs (opens in a new tab) trained critic models to review code and found their critiques were preferred over human ones in 63% of cases with naturally occurring errors. The same paper is candid that critics can "hallucinate" problems that are not there, and that human and machine teams did better on that front than models alone. That is the balance we aim for: an AI reader that catches a lot, with every decision logged so a human can check it.
How we design around those biases
- Separate roles: the Critic is not the AI user and does not share its instructions, so it is not grading its own voice.
- One line at a time: the Critic judges each answer on its own merits instead of comparing two answers side by side, which avoids the ordering problem.
- A written rubric: character, facts, eagerness and consistency, each with a reason recorded when a line is flagged.
- Visible decisions: flagged lines are kept in a log, so you can disagree with the Critic and see exactly why it acted.
A walk-through: one flagged line
Imagine a study for a meal-kit startup in Dubai. One AI user is a nurse who works night shifts and has said, early in the interview, that she cooks twice a week at most. Later, asked about a premium recipe box, she replies: "I would cook from it every night, it sounds perfect." The Critic flags the answer on two counts. It contradicts what she said about her schedule, and it is enthusiastic without a reason. The line is sent back with that note. The regenerated answer is more like this: "Maybe on my days off. On shift weeks I would let it rot in the fridge, and then I would feel guilty about paying for it." The second version is less flattering and far more useful, because it tells the founder something about pausing and skipping deliveries. The example is hypothetical.
Glossary
LLM judge: a language model used to score or check other model outputs. Rubric: the explicit criteria a judge applies. Position bias: a judge preferring an answer because of where it appears. Self-preference bias: a judge favoring text that resembles its own. Sycophancy: an answer shaped to please the listener rather than to be true.
Limitations worth knowing
The Critic reduces errors, it does not certify truth. It can miss a subtle contradiction, and it can occasionally flag a line that was fine. It also cannot know things the research brief does not contain. That is why we describe its output as a log, not a verdict, and why the best use of a Cast study is to sharpen the questions you take to real customers.
Questions about the Critic
Does the Critic make results more negative?
No. It rejects unsupported answers in both directions. An unfounded complaint is flagged just like unfounded praise.
Can the Critic be wrong?
Yes, which is why flagged lines are logged rather than hidden. You can review any decision and override it.
Why not just use a stronger model and skip the Critic?
Because stronger models are still poor at checking their own work without outside feedback. Independence matters more than raw ability.
What is the difference between LLM as a judge and human review?
An LLM judge applies written criteria quickly and consistently to every line, which no human team could do at the same speed. A human reviewer brings context and judgment the rubric may miss. Cast keeps the Critic's decisions visible so people can do the second part.
Can an AI judge be biased?
Yes. Studies have found judges influenced by answer order, answer length and whether the text resembles their own. Cast reduces these effects by keeping the Critic separate, judging one line at a time and logging every decision.
Read next: why every cast includes skeptics.
