LLM evaluation before launch: a practical 7-step guide

LLM evaluation is the practice of testing an AI feature against a fixed set of realistic examples, with written pass criteria, so you know how often it is right before customers find out. A launch-ready evaluation combines typical, edge and adversarial cases, automated grading where possible, human review where it matters, and a rerun of the whole set on every change. The seven steps below follow the published guidance from OpenAI and Anthropic and can be started this week with a spreadsheet.

Key takeaways

• Write specific, measurable success criteria first; 'it works well' cannot be tested.
• Build a test set from real traffic that includes typical, edge and adversarial cases, and keep it growing after launch.
• Grade automatically where you can (exact match, code checks, an LLM judge validated against human labels) and keep humans for high-stakes judgements.
• Test safety and privacy failures, such as prompt injection and personal data leaking, alongside accuracy.
• Rerun the full evaluation on every prompt, model or data change, because LLM output is non-deterministic and providers retire models.

What does evaluating an LLM feature actually involve?

Evaluating an LLM feature means measuring its behaviour on a representative test set against criteria you wrote down in advance. Unlike ordinary unit tests, the same input can produce different outputs, so you measure pass rates across many examples rather than checking a single expected answer. OpenAI's guidance describes the process in five stages: define the objective, collect a dataset, define metrics, run and compare, and continuously evaluate. The seven steps below expand that into actions.

Step 1: How do you define success criteria?

Start by writing criteria that are specific and measurable. Anthropic's guidance suggests criteria that are specific, measurable, achievable and relevant, for example "accurate sentiment classification" instead of "good performance". It also notes that most use cases need several dimensions, such as task fidelity, consistency, tone, privacy preservation, latency and price.

For a support assistant, a usable set might be: answers grounded in the help centre in at least 95% of sampled cases, no invented policies, escalates billing disputes every time, median response under four seconds. The numbers are yours to choose; what matters is that they are written down before you look at results.

Step 2: Where should the test examples come from?

Use real inputs wherever you can. OpenAI recommends combining sources such as production data, historical logs, domain-specific and human-curated data, and making sure the set includes typical cases, edge cases, and adversarial cases. It flags a common anti-pattern: eval sets that do not reproduce real production traffic.

Anthropic adds edge cases worth covering explicitly: irrelevant or missing input, overly long input, harmful user input, and ambiguous cases where humans would also disagree. A first set of 100 to 200 examples is enough to start; each example needs the input and either a reference answer or a rubric.

If your examples contain customer data, the usual GDPR rules apply. Ireland's Data Protection Commission reminds organisations to set retention rules for personal data used with AI systems; pseudonymise test data where you can.

Step 3: How should outputs be graded?

Grade automatically wherever the criterion allows it, and reserve people for judgements that need expertise. The main options:

Method	Good for	Watch out for
Exact match or code check	Classifications, extracted fields, JSON format, tool calls	Only works when there is one right answer
Similarity metrics	Summaries and paraphrases against a reference	Rewards wording, not correctness
LLM as a judge	Tone, groundedness, rubric scoring at scale	Position and verbosity bias
Human review	High-stakes, expert or ambiguous cases	Slow and costly

OpenAI notes that human evaluation gives the highest quality but is slow and expensive, and warns that LLM judges show position bias and verbosity bias. It recommends pass/fail or pairwise grading and validating the judge against human labels before scaling up. Anthropic suggests it is generally best practice to evaluate with a different model from the one that generated the output, and argues that more automated questions beat fewer hand-graded ones.

Step 4: What safety and privacy failures should you test?

Test the failures that would embarrass you or harm a customer, not just accuracy. The OWASP Top 10 for LLM applications is a useful checklist: it includes prompt injection, sensitive information disclosure, system prompt leakage, misinformation and unbounded consumption. Add adversarial examples for each that applies, such as a pasted document containing hidden instructions, a request for another customer's data, or an attempt to extract your system prompt. For agents that take actions, include cases where the right behaviour is to stop and ask a person; our guide to human-in-the-loop AI explains how to choose those points.

Step 5: How do you run and compare versions?

Run the whole set for each candidate (prompt version, model or retrieval setup) and compare results in one table. Record pass rate per criterion, cost per task and latency, then read the failures, not just the score. Failures usually cluster: missing documents, ambiguous instructions or a format the model keeps getting wrong. Fix the cause and rerun. If the fix requires better knowledge rather than better behaviour, our comparison of RAG vs fine-tuning helps you choose the remedy.

Because outputs vary between runs, run each example more than once for criteria close to your threshold, and treat small differences between versions with caution.

Step 6: How do you decide the feature is ready to launch?

Launch when every criterion meets its written threshold on the current set and nobody on the team can name a known failure that would cause real harm. Set the bar per criterion, because a 90% pass rate may be fine for tone but unacceptable for refunds. Document the results, the set version and the known limitations; that record is also useful evidence for data protection and AI Act documentation. The wider operational items, such as monitoring and rollback, are in our AI feature launch checklist.

Step 7: What happens to evaluation after launch?

Evaluation becomes continuous. OpenAI recommends running evals on every change, monitoring for new non-deterministic behaviour and growing the set over time. Feed real failures from production back into the set, rerun it when your provider updates or retires a model, and sample live conversations for human review each week.

Tooling changes too. OpenAI's own guide states that its hosted Evals platform becomes read-only for existing users on 31 October 2026 and is scheduled to shut down on 30 November 2026, so keep your test sets and grading code in your own repository where you can move them between tools and providers.

What is the checklist to follow today?

Copy this list into your project tracker:

• [ ] Success criteria written, each with a number and an owner
• [ ] 100 to 200 test examples from real inputs, labelled typical, edge or adversarial
• [ ] Personal data in test examples minimised or pseudonymised, with a retention rule
• [ ] Automated graders for every criterion that allows one
• [ ] LLM judge (if used) checked against at least a sample of human labels
• [ ] Safety cases for prompt injection, data disclosure and prompt leakage
• [ ] Results table comparing versions on quality, cost and latency
• [ ] Launch thresholds agreed and met, known limitations documented
• [ ] Evaluation rerun automatically on every prompt, model or data change
• [ ] Weekly sample of live outputs reviewed, failures added to the set

Frequently asked questions

How many examples do I need to evaluate an LLM feature?

There is no official minimum. A first set of 100 to 200 realistic examples usually surfaces the main failure patterns, and Anthropic's guidance favours more automatically graded examples over fewer hand-graded ones. Grow the set from real failures after launch.

Can I use an LLM to grade another LLM?

Yes, and both OpenAI and Anthropic describe it, but validate it first. Compare the judge's verdicts with human labels on a sample, prefer pass/fail or pairwise comparisons, and watch for bias towards longer answers or answers shown first.

Are public benchmarks enough to choose a model?

No. Benchmarks compare models in general, while your feature depends on your data, prompts and users. Use benchmarks to shortlist models, then run your own evaluation set to decide.

How often should I rerun my LLM evaluation?

On every change to prompts, models, retrieval data or tools, and at least whenever your provider announces a model update or retirement. Automate the run so it happens before each deployment, like other tests.

Who should own LLM evaluation in a small team?

The product owner should own the success criteria and thresholds, and an engineer should own the test harness. Domain experts, such as support leads, should label examples and review samples, because they know what a wrong answer costs.

If you would like help setting up an evaluation before your AI feature goes live, book a free 30-minute call.

Cover photo by RDNE Stock project on Pexels.