An eval is a test for an AI feature: a fixed set of tasks, a way to grade each answer, and a score you can compare before and after a change. Here is how golden sets, graders, LLM judges and regression gates work, read from Anthropic's and OpenAI's own guidance.
You tweak one sentence in your system prompt. Three test questions look better. You ship it. A week later a customer says the bot got worse at something you never tried. Welcome to whack-a-mole.
An eval is the workshop jig that fixes this. Instead of eyeballing three answers, you run the same fifty and count. This page reads what Anthropic, OpenAI, Hamel Husain and Eugene Yan have published and puts the useful parts in order. If you are still writing the prompt itself, start with prompting is writing a spec. If your prompt is getting long, context engineering for vibe coders comes first.
What is an LLM eval?
An eval is a test for an AI system. Anthropic's engineering team defines it this way: "give an AI an input, then apply grading logic to its output to measure success." The difference from an ordinary unit test is that the same input can produce a different answer each time, so evals lean on grading and repeated tries.
Anthropic uses a small vocabulary. It helps to know it, because every tool borrows it.
Task: one test with defined inputs and success criteria. Also called a test case.
Trial: one attempt at a task. You run several, because outputs vary.
Grader: logic that scores some part of the result. A task can have several.
Transcript: the full record of a trial: outputs, tool calls, reasoning.
Outcome: the final state of the world. Anthropic's example: a flight agent may say "Your flight has been booked," but the outcome is whether a reservation exists in the database.
Suite: a collection of tasks that measure one capability, such as refunds and cancellations for a support bot.
OpenAI points out that "evals" can mean three different things: public benchmarks that compare models (MMLU is one), standard numeric scores such as ROUGE, and tests you write for your own application. This page is about the third kind. A benchmark tells you which model is strong in general. Your eval tells you whether your prompt works on your users' questions.
How do you know a prompt change made it better?
You compare scores on the same tasks, before and after, with the same graders. That is the whole idea. The old prompt is the baseline. If the new prompt scores higher without dropping tasks that used to pass, it is better by your own definition of better.
Anthropic describes what life is like without this. The breaking point "often comes when users report the agent feels worse after changes, and the team is 'flying blind.'" Debugging turns reactive: wait for a complaint, reproduce it, fix it, "hope nothing else regressed." Hamel Husain calls the early version of this approach vibe checks, "which are useful, but not enough." OpenAI lists "vibe-based evals" as an anti-pattern: using "it seems like it's working" as your strategy.
Two practical points keep the comparison honest:
Change one thing at a time. If you swap the model and the prompt together, you will not know which one moved the score.
Run each task more than once. A task that passed once may fail on the next run. Anthropic runs multiple trials per task for exactly this reason. More on the math in the section on flaky scores below.
A first eval in five steps, condensed by aliteq from Anthropic's and OpenAI's guidance, read 3 Oct 2026. · aliteq research
What goes in a golden set?
A golden set is your fixed list of test tasks with known-good answers or clear pass rules. Anthropic's advice is to start small: "20-50 simple tasks drawn from real failures is a great start." Early on, each change has a big, visible effect, so a small sample is enough to see it.
Where do the tasks come from? Anthropic says to begin with what you already test by hand: the checks you run before each release and the things users try most. If you are in production, mine the bug tracker and the support queue, and prioritize by user impact. OpenAI adds that a dataset can mix synthetic, domain-specific, production and historical data, and should include typical cases, edge cases and adversarial cases.
No users yet? Hamel Husain says you do not need to wait. He uses an LLM to generate synthetic test inputs, then reads the results himself.
Here is the shape of a tiny golden set. It uses OpenAI's own docs example, a bot that sorts IT tickets into Hardware, Software or Other. The first row's ticket text comes from the docs. The other rows are our illustration, not run against any model.
{"ticket": "My monitor won't turn on - help!", "expected": "Hardware"}
{"ticket": "Excel crashes every time I open the budget file", "expected": "Software"}
{"ticket": "Who do I ask about parking passes?", "expected": "Other"}
{"ticket": "returns", "expected": "Other"}
Four things make a golden set trustworthy:
Unambiguous tasks. Anthropic's test: "two domain experts would independently reach the same pass/fail verdict." If they would argue, rewrite the task.
A reference answer for each task. A known working output proves the task can be solved and the grader is set up right. Anthropic adds that a 0% pass rate across many trials is "most often a signal of a broken task, not an incapable agent."
Both directions. Test where a behavior should happen and where it should not. Anthropic's example is web search: if you only test that the model searches when it should, you may build one that searches for everything. Their set included weather questions (search) and "who founded Apple?" (answer from memory).
Messy inputs. OpenAI's edge-case list includes multiple questions in one message, typos, a one-word request such as "returns," other languages, and users who try to override your instructions.
Which graders should you use?
Use the cheapest grader that can judge the answer. Anthropic groups graders into three types: code-based, model-based and human. Its recommendation is deterministic graders where possible, LLM graders where necessary, and humans "judiciously" for validation. Most suites mix them.
Grader types as Anthropic's engineering post groups them, read 3 Oct 2026. · aliteq research
Code graders check things a program can check: an exact or regex match, a test that passes, a linter, a tool call with the right arguments, or the state of the database afterward. Hamel Husain's example is a regex that fails any reply that leaks a raw UUID. They are fast and repeatable. Their weakness is brittleness: a correct answer worded differently can fail.
Model graders use another LLM to score the answer against a rubric. They handle open-ended text and tone. The cost is that they are non-deterministic and, in Anthropic's words, need calibration with human graders.
Human graders are the gold standard and the slowest. Use them to build your labels and to check the judge, not to grade every run.
Two grading habits from Anthropic are worth copying:
Grade the result, not the route. Checking an exact sequence of tool calls is "too rigid." Agents find valid approaches you did not plan for, so grade what they produced.
Give partial credit. A support bot that identifies the problem and verifies the customer but fails to issue the refund is better than one that fails at the start. Your score should show that.
What goes wrong with LLM-as-judge?
An LLM judge is cheap and scales, but it has known biases and needs a human check. OpenAI names two: position bias, where the order of two answers changes the verdict, and verbosity bias, where longer answers win. Eugene Yan's survey adds a third, self-enhancement bias: judges prefer answers written by themselves.
Here is how the sources suggest working around each problem.
Position bias
What the sources say to do
Prefer pairwise comparison or pass/fail over open scores, as OpenAI recommends, and watch for order effects
Verbosity bias
What the sources say to do
Control for answer length, per OpenAI
Self-enhancement
What the sources say to do
Eugene Yan flags that judges favor their own output; check who wrote the answer you are grading
Anthropic: give the judge a way out, such as returning "Unknown"
One judge for everything
Anthropic: grade each dimension with its own isolated judge
Never checked
Track agreement with human labels, and keep checking it
Yan's survey also separates the scoring styles. Direct scoring (one answer, one verdict) suits objective checks such as "is this faithful to the source." Pairwise comparison suits subjective qualities such as tone, and he reports it gives more stable results closer to human annotations.
How do you check a judge?
Label a batch yourself, run the judge on the same batch, and compare. Hamel Husain used a spreadsheet: the model's answer, the judge's critique and verdict, then his colleague's own verdict on 25 to 50 examples at a time. He rewrote the judge prompt until it agreed with the human, and kept re-checking over time.
One warning from his post matters. Raw agreement can mislead when your data is lopsided. If 95% of answers are good, a judge that always says "good" agrees with you 95% of the time. He recommends measuring precision and recall separately instead. Precision asks how many of the judge's "bad" calls were truly bad. Recall asks how many truly bad answers the judge caught.
OpenAI's guidance fits: once the judge is "faster, cheaper, and consistently agrees with human annotations," scale it up. Not before.
Why is one run not enough?
AI answers vary, so a task that passed once can fail the next time. Anthropic gives two metrics. pass@k is the chance of at least one success in k tries. pass^k is the chance that all k tries succeed. They move in opposite directions as k grows.
Anthropic's example: an agent with a 75% per-trial success rate that runs three trials passes all three with probability (0.75)^3, about 42%. We extended the same arithmetic below, assuming each try is independent.
The 75% example is Anthropic's. The other values are aliteq arithmetic (pass^k = 0.75^k, pass@k = 1 - 0.25^k) assuming independent tries. · aliteq research
Which to use depends on the product. For a tool where one good answer is enough, such as suggesting code you will review, pass@k fits. For a customer-facing agent that must behave every time, Anthropic says pass^k is the one that matters.
This is also why a single before-and-after run misleads. A two-point gain on one run can be luck. Run each task several times, and if the change is small, expect to need more tasks to see it. Anthropic notes that mature agents "may need larger, more difficult evals to detect smaller effects."
What is regression gating?
A regression gate is a check that blocks a change when the suite shows it broke something that used to work. Anthropic separates two kinds of eval. Capability evals ask what the agent can do well and should start at a low pass rate, a hill to climb. Regression evals ask whether it still handles everything it used to and should sit near 100%.
The lifecycle runs like this. You climb on the capability set. When tasks pass reliably, they "graduate" into the regression set, which runs continuously to catch drift. Anthropic notes that automated evals can run on every commit, and calls them the first line of defense on each agent change and model upgrade. OpenAI describes the same idea as continuous evaluation: run evals on every change and grow the set over time.
Here is one gate rule, as our illustration, untested:
Run the regression set on every change to the prompt, tools or model. Block the merge if any task that passed in the last accepted run now fails more often than it failed before, or if the overall pass rate drops below the agreed floor.
How strict should the floor be? Hamel Husain says that unlike traditional unit tests you don't necessarily need 100%, because "your pass rate is a product decision." A typo-checker and a refund bot deserve different floors.
Watch for the opposite problem too. An eval at 100% still guards against backsliding but no longer tells you whether you are improving. Anthropic calls this saturation. When a suite is saturated, add harder tasks.
How do you stop the eval itself from lying?
Read the transcripts. Anthropic is blunt: "we do not take eval scores at face value until someone digs into the details of the eval and reads some transcripts." A failure should seem fair. If you cannot tell whether the agent erred or the grader was wrong, the eval is not ready.
Anthropic's cautionary tale: Opus 4.5 first scored 42% on CORE-Bench, until a researcher found rigid grading that penalized "96.12" when the grader expected "96.124991...", plus ambiguous specs and tasks that could not be reproduced exactly. After fixes, the score jumped to 95%. The model had not changed. The test had.
Other traps from the same post:
Shared state. Start every trial from a clean environment. Anthropic saw Claude gain an unfair edge by reading git history left over from earlier trials.
Cheating. Design graders so passing requires solving the problem, not exploiting a loophole.
One-sided sets. Test both when a behavior should and should not happen.
Skipping people. OpenAI lists "ignoring human feedback" as an anti-pattern. Calibrate automated scores against human judgment.
Evals also sit inside a bigger picture. Anthropic compares layers of understanding to the Swiss cheese model from safety engineering: automated evals, production monitoring, A/B tests, user feedback and human review each catch what the others miss.
Which tools run evals?
The concepts matter more than the tool. Anthropic's appendix names Harbor, Braintrust, LangSmith, Langfuse and Arize (Phoenix and AX) as frameworks, and adds that many teams "roll their own eval framework, or just use simple evaluation scripts as a starting point." Its advice is to pick a framework quickly, then put your energy into the tasks and graders.
One timing note for OpenAI users. OpenAI's deprecations page says it announced the end of its Evals platform on June 3, 2026. Existing evals become read-only on October 31, 2026, and the dashboard and API are scheduled to shut down on November 30, 2026. OpenAI links a cookbook guide on moving from OpenAI Evals to Promptfoo. We have not tried that path. Your golden set and graders, kept as plain files, move between tools. That is a good reason to keep them in your own repo.
Your first eval this week
Write down 20 real inputs where your feature failed or nearly failed. Pull them from bug reports and support messages.
For each one, write the answer or rule that counts as a pass. Hand the list to a colleague. If you would disagree on a verdict, rewrite that task.
Add code checks first: exact match, format, a forbidden string. Add an LLM judge only for what code cannot check.
Grade 25 outputs by hand and compare with the judge. Fix the judge prompt until they agree on the bad answers, not just overall.
Run the set on every prompt change, several tries per task. Read the failures before you read the score.
If you build with an AI coding tool, you can ask it to draft the first golden set from your bug list. Check every row yourself, since a wrong expected answer quietly teaches the wrong lesson. For how coding agents differ from the apps they build, see vibe coding vs agentic engineering.
Quick answers
What does eval mean in AI?
Short for evaluation. In AI engineering it means a test for an AI system: give it an input, grade the output, and measure success. OpenAI separates public benchmarks, standard numeric scores, and the tests you write for your own application.
How many test cases do I need?
Anthropic says 20 to 50 simple tasks drawn from real failures is a great start, because early changes have big effects. Mature systems that need to detect small improvements need larger and harder sets.
Can I use an LLM to grade another LLM?
Yes, and many teams do. Both OpenAI and Anthropic say to calibrate the judge against human labels first. Known biases include position bias and verbosity bias, so use clear rubrics, pairwise or pass/fail verdicts, and an "Unknown" option.
Do evals replace manual testing and user feedback?
No. Anthropic treats evals as one layer next to production monitoring, A/B tests, user feedback and reading transcripts. Hamel Husain says to keep looking at your data, and to sample more selectively as the system matures.
What is the difference between a benchmark and an eval?
A benchmark is a public test that compares models in general, such as MMLU. An eval in the sense used here is a test built from your own tasks, so it tells you whether your prompt or agent works for your users.
Does an eval need a 100% pass rate?
Not always. Hamel Husain says the pass rate is a product decision based on the failures you can tolerate. Anthropic expects regression suites to sit near 100% and capability suites to start low.
