The 3 ways to grade LLM outputs automatically (and when each one fails)

If you want to evaluate prompt outputs without reading every single one, there are basically three grader types:

1. Deterministic graders. Exact match, regex, JSON schema checks, small scripts.

• Best for: structured output, extraction, classification.
• Fails when: quality is subjective. You can't regex "is this summary good".

2. LLM-as-judge. A model grades the output against criteria you define.

• Best for: tone, helpfulness, correctness of free text.
• Fails when: you're not explicit. Always spot-check it against your own judgment first, and give it explicit criteria. A vague judge is a useless judge.

3. Reference graders. Compare output against an expected answer.

• Best for: tasks with a known good answer (Q&A, transformations).
• Fails when: many different outputs are equally valid.

The practical setup that works for me is deterministic checks for structure and LLM-judge for quality, on the same run. Cheap checks filter the obvious failures, the judge handles nuance.

Ever since I started learning and applying this stuff, the output quality has increased massively.