You cannot improve what you cannot measure, and measuring the quality of free-text model output is hard. Human grading is accurate and does not scale; exact-match metrics scale and are useless for anything open-ended. So teams reach for the obvious middle path: use a strong model to grade the outputs of the model under test. LLM as a judge is genuinely useful, and it is also full of traps that produce confident, reproducible, and completely wrong numbers. Knowing the traps is the difference between a real evaluation and a comforting one.
Why the approach works at all
The insight is that judging is easier than generating. Deciding whether an answer is faithful to a source, or which of two summaries is better, is a narrower task than producing the summary in the first place. A capable model is a surprisingly good judge of a bounded, well-specified question — and it runs in seconds for pennies, which means you can grade thousands of outputs on every change instead of sampling a handful by hand.
That scale is the real prize. It turns evaluation from a quarterly human exercise into something you run in CI on every prompt tweak, which is the only way to know whether a change helped or quietly regressed something else. But the judge is only worth running if you trust its scores, and the default setup does not earn that trust.
Position bias: the trap that invalidates most pairwise evals
The most damaging trap is also the least known. When you ask a model “which of these two answers is better, A or B?”, it has a systematic preference for one position — often the first — independent of content. Swap the two answers and the same model frequently picks the other one. It is judging the slot, not the substance.
This quietly ruins any pairwise comparison that presents candidates in a fixed order. Your new prompt looks better because its outputs happened to land in the favoured position. The fix is not optional: run every comparison both ways, A-then-B and B-then-A, and only count a win when the judge picks the same answer in both orderings. Ties and flips get discarded or sent to a human. This doubles your judging cost and it is the price of a number that means anything.
Self-preference: the judge likes its own writing
A related trap: a model tends to rate outputs generated by itself, or by models in its own family, more highly than a neutral grader would. The style feels right to it because it is the style it produces. If you use the same model to generate and to judge, this bias inflates your scores in a way you cannot see from inside the system.
Where the stakes justify it, use a different model family as the judge than the one you are evaluating, or at least be aware that same-family scores skew generous. This matters most when you are comparing your model against a competitor’s — the judge is not neutral, and you should not present its verdict as if it were.
Absolute scores drift; relative comparisons are stabler
Asking a judge to rate an answer “from 1 to 10” feels precise and is mostly noise. The model has no stable internal calibration for what a 7 means versus a 6, the numbers drift between runs, and they cluster — you will find almost everything scored 7 or 8, which tells you nothing. Absolute numeric scores are the least reliable thing you can ask for.
Two things help enormously. First, prefer relative judgments — “is A better than B?” — over absolute ones, because comparison is a question models answer far more consistently than scoring. Second, when you do need an absolute label, use a small rubric with named criteria and concrete definitions rather than a bare number: not “rate 1-10” but “is this answer faithful to the source — yes, partially, or no — and is it complete?” A rubric with explicit anchors turns a vibe into something reproducible.
Weak: "Rate this answer's quality from 1 to 10."
Strong: "Answer YES or NO for each:
- Faithful: every claim is supported by the provided source?
- Complete: addresses all parts of the question?
- Grounded: no information beyond the source?
Then give a one-sentence justification for any NO."Asking for the justification before or alongside the verdict is not decoration — a judge that must state its reasoning is more consistent than one that emits a bare label, for the same reason showing your work catches your own errors.
The step everyone skips: validate the judge against humans
Here is the uncomfortable question that most LLM-as-judge setups never ask: is the judge actually right? A judge that scores confidently but disagrees with human experts is worse than no judge, because it gives you a number to optimise toward that points the wrong way.
Validate it. Have humans grade fifty to a hundred examples, then check how often the judge agrees with them. If agreement is high, you can trust the judge to scale that human judgment. If it is low, your rubric is wrong, your task is too subjective for the model, or you need a stronger judge — and you need to know that before you make a hundred decisions based on its scores. This is a one-time cost per evaluation task and it is the step that separates a real methodology from cargo-culting.
Techniques that make a judge more consistent
Beyond avoiding the traps, a few positive techniques measurably tighten a judge’s agreement with humans, and they are worth applying once the basics are in place.
The most effective is to make the judge reason before it concludes. A judge asked to state its evaluation against each rubric criterion, and only then give a verdict, is more consistent than one that emits a bare label — for the same reason a person forced to show their working catches their own mistakes. Order matters here: the reasoning must come before the verdict in the output, because a model that states its conclusion first tends to rationalise it afterward rather than reason toward it. This costs output tokens, which is a real expense at scale, but it buys reliability that a one-word answer cannot.
Few-shot examples help too, especially for a subjective rubric. Showing the judge two or three worked examples — an answer that should score YES on faithfulness with a note on why, one that should score NO — anchors its interpretation of your criteria to yours. Without them, the judge applies its own implicit standard, which may drift from what you actually mean by “complete” or “grounded.” A handful of examples pins the definition down far more reliably than more words of instruction.
Finally, temperature. Run the judge at a low or zero temperature, because you want its evaluation to be reproducible, not creative. A judge at high temperature gives different verdicts on the same input across runs, which makes your evaluation itself noisy and undermines the whole point of measuring. Determinism is a feature here in a way it rarely is for generation. These three — reason-then-verdict, a few anchoring examples, and low temperature — are cheap to add and each moves agreement with human graders in the right direction, which is the only metric that ultimately matters for a judge.
Where it belongs, and where it does not
Use an LLM judge for what it is good at: bounded, well-specified quality questions at a scale humans cannot match — faithfulness of a RAG answer to its sources, relative quality between two prompt versions, presence or absence of a required element. It shines as a regression gate in CI, catching quality drops on every change, and it composes naturally with the tracing in our LLM observability guide, which is where you collect the outputs to grade in the first place.
Do not use it as the final word on anything high-stakes, subjective, or adversarial. It is a fast, cheap, biased approximation of human judgment — extraordinarily useful when you respect that description and dangerous when you forget it. And when the thing you are really evaluating is retrieval quality rather than generation quality, the judge is aiming at the wrong target; the diagnosis in our guide to fixing RAG retrieval is the better tool for that failure.
Run comparisons both ways, prefer relative to absolute, use a rubric, judge with a different model family where it matters, and validate against humans before you trust a single number. Do those five things and the judge earns its place. Skip them and you have built a machine for generating confident, wrong, reproducible scores — which is worse than having no scores at all, because it feels like rigor.