More and more teams use one language model to grade another. A chatbot answers, and an LLM judge reads the answer and says pass or fail. It’s the only practical way to check thousands of free-text answers. But the judge’s verdicts usually go unchecked, so we tested the judge itself.
The short version: on obvious mistakes, judges look almost perfect. On real chatbot mistakes, marked by people, the same judges agreed with the labels only 53–68% of the time. When we read the disagreements, though, the judge was often right and the label wrong.
What we tested
Every item is a grounded question-answering task: a source text, a question, and an answer that should use only that text. The judge must decide whether the answer passes this rubric:
Pass when the answer correctly answers the question and every fact it states is supported by the passage. If the passage does not contain the answer, pass only when the answer says so. Fail when the answer is wrong, incomplete, states unsupported facts, or answers when the passage has no answer.
Two datasets, both balanced between right and wrong answers (so guessing scores 50%):
- Synthetic, 80 answers. Claude wrote 40 short passages about fictional places and products, each with a question. Gemini 3.5 Flash-Lite wrote a correct answer for each question, and Claude wrote a wrong answer with one deliberate mistake: a wrong calculation, a mixed-up entity, a made-up answer, and so on. No human reviewed this set. It’s a controlled test of known mistake types, not a benchmark.
- RAGTruth, 120 answers. Real answers from six chatbots (GPT-3.5, GPT-4, Llama 2 7B/13B/70B, Mistral 7B) to real web-search questions, taken from the public RAGTruth dataset. Human annotators marked every part of each answer that the source text doesn’t support. We sampled 60 clean answers and 15 of each of RAGTruth’s four mistake types.
Two judges, eight prompts. Gemini (gemini-3.8-flash) can explain its verdicts. TypeSafe’s Jev (jev-1.13.0) is a fast classifier that returns a probability instead. For each judge we changed one thing at a time:
- Basic: just “Is this a good answer?”
- Rubric: the rubric above.
- Rubric + reasoning: also “check each fact and explain before the verdict”.
- Rubric + reasoning + examples: also three worked examples.
- Faithfulness only (RAGTruth only): fail only for unsupported or contradicting content. This matches how RAGTruth was labelled.
We ran 2,640 judge calls in total, including repeat runs and a length-bias test.
Easy mistakes versus real ones
- On the synthetic set, every Gemini prompt gave the correct verdict 98–99% of the time, even the one-line “Is this a good answer?”. Jev scored 91–94%.
- On RAGTruth, the same judges agreed with the human labels only 53–68% of the time.
The judges didn’t get worse; the task got harder. A synthetic mistake is one clear error, such as “8 beds” when the text implies 24 − 18 = 6. Real chatbot mistakes are subtler, and, as it turns out, so are the labels.
Where judges struggle
Breaking RAGTruth down by mistake type, the Gemini rubric judge caught 14 of 15 answers with evident unsupported information, but only 8 of 15 subtle contradictions, where one detail is changed. Jev caught 7 of 15.
One answer that all eight judges passed came from Mistral 7B, on “how to loosen a rusted nut”. The text says: “If the nut is near plastic components or other things that heat damages, omit this step.” The six-step answer is otherwise accurate, but it turns that into “be careful when applying heat to avoid damaging nearby plastic components.” “Don’t do it” became “do it carefully”. Even the judges asked to reason step by step described the answer as accurately summarising the passages. They checked that each step existed in the text, not what the text said to do.
The prompt matters only when the task is hard
On the synthetic set the prompt made almost no difference. On RAGTruth it made a big one:
- Basic prompt: Gemini passed 39 of 60 wrong answers (65%). Without a definition of “good”, a fluent answer looks good.
- Rubric: that dropped to 15 of 60 (25%).
- Reasoning and examples added little on top. Gemini is a “thinking” model and already produced about 140 output tokens per verdict even with the basic prompt.
The trade-off: the rubric also made the judge stricter, and it failed 29 of 60 answers the labels called clean. That leads to the most interesting finding.
The answer key can be wrong too
A disagreement with a human label is only a judge error if the label is right. We re-read all 43 answers where Gemini’s reasoning or faithfulness judge disagreed with the label, against the full source text. For the rubric judge, the judge was right in 23 of 32 clear-cut cases, the label in 9, and 4 were debatable. Two examples:
- The label missed an error. A Llama 2 answer about Celexa claims “passage 3 notes that Celexa may cause side effects such as nausea, dizziness, and dry mouth.” Passage 3 mentions no side effects at all. RAGTruth marks the answer as clean; all eight judges failed it.
- The label was too strict. A GPT-3.5 answer says the Industrial Revolution happened “between the late 18th and early 20th centuries”. The text says “the late 1700s and the early 1900s”. The label calls that a contradiction, but it means the same thing. All eight judges passed it.
One caveat: that re-reading was done by Claude, an AI, not by a second human annotator. It’s published in the repository, and every case is visible in the dataset explorer, so you can check it yourself.
Confident is not the same as correct
Jev returns a probability with each verdict, which is tempting to use as a trust signal. It doesn’t work well here:
- Jev’s probabilities for correct and wrong RAGTruth answers overlap heavily. It passed wrong answers with probabilities as high as 0.99.
- On the synthetic set, it caught only 3 of 8 wrong calculations. It passed “8 raised beds” (the correct figure is 6) with a probability of 1.0.
- Jev is about 6× faster than Gemini (about 0.3 s against 1.8 s per verdict), but it classifies rather than calculates.
Good news
- Length bias: wrapping every answer in the same confident filler (“Based on a careful reading of the passage… I have checked it against the passage to make sure it is accurate and complete”) never turned a wrong answer into a pass.
- Consistency: across three identical runs, 99% of synthetic verdicts and 92–94% of RAGTruth verdicts stayed the same.
What to take away
- Test your judge on realistic, hard examples. An easy test set makes every judge look perfect.
- Write a rubric. It was the single biggest improvement.
- Break results down by mistake type. Overall scores hide blind spots like subtle contradictions and arithmetic.
- Read the disagreements. Your labels may need fixing, not your judge.
- Don’t treat confidence as correctness.
Honest limits
- The samples are small: 80 and 120 answers. The full report includes 95% confidence intervals, and a few percentage points is noise.
- It covers one task: short answers grounded in a source text.
- The synthetic set was made entirely by AI (Claude and Gemini), with no human review.
- The label review is an AI’s reading, not a second human annotation.
- The results are for gemini-3.8-flash and jev-1.13.0 in September 2026.
Run it
The repository includes a full write-up with figures and five worked examples, and a dataset explorer. The explorer shows every answer with its source text, the correct verdict and why, and what each judge said, with reasoning or probability.
Requirements: Python 3.12+ and uv. Reading the report and explorer needs no API key; re-running the judges needs Gemini and TypeSafe keys and makes paid calls.
git clone https://github.com/AIEngineerHelp/applied-ai-engineering.git
cd applied-ai-engineering/evaluations-reliability/llm-judge-reliability
uv sync --locked
uv run python -m judgelab fetch-ragtruth # downloads RAGTruth (~36 MB)
uv run python -m judgelab serve # report and explorer at http://127.0.0.1:8765
RAGTruth is downloaded rather than stored in the repository, because its passages come from MS MARCO, whose terms limit use to non-commercial research. Our own judge runs aren’t committed, so to see the per-answer verdicts in the explorer, run the judges yourself with run-all --live (paid calls); the aggregate results are in results/.
How do you check the LLM judges in your own evals: against human labels, spot checks, or not at all? And when your judge and your labels disagree, who usually turns out to be right?