LLM Evaluation and Benchmarking
An evaluation capability built on your own task: scored test sets, calibrated automatic judging, and a regression suite that runs before every release instead of after every complaint.
Ask most teams how good their LLM feature is and you get an anecdote. Ask how good it was last month and you get a different anecdote. Without evaluation, every change is a guess, every regression is discovered by a customer, and every model upgrade is an act of faith.
LLM evaluation and benchmarking is the practice of measuring a language model system against a curated set of representative tasks with agreed correct outcomes, using a mix of deterministic checks, calibrated model-based judging and human review, so that quality can be compared across prompts, models and releases as a number rather than an impression.
Why public benchmarks will not tell you what you need
Leaderboards measure general capability on public tasks. Your users do not ask public questions, and the correlation between a headline benchmark score and performance on your particular workload is weaker than the marketing implies. Worse, public sets leak into training data over time, so a score can improve without the underlying capability moving at all.
The only benchmark that predicts how a system will behave in your product is one built from your traffic, judged by people who know what a correct answer looks like in your domain. Everything in this engagement follows from that.
What we build
The golden set
A few hundred representative cases drawn from real usage, with correct outputs agreed by your subject experts. We deliberately over-sample the awkward cases: ambiguous questions, requests that should be refused, inputs with missing information, and the specific failures that have already caused complaints. A set made only of easy questions produces a comfortable score and no information.
Deterministic checks first
Anything that can be checked with code is checked with code: schema validity, required fields, forbidden content, citation presence, numeric ranges, latency and token cost. These are cheap, fast and completely reliable, and they catch a surprising share of real regressions before any judgement is needed.
Calibrated model-based judging
For the rest, a strong model scores outputs against a rubric. The step teams skip is calibration: we measure how often the judge agrees with your human experts on a sample, tune the rubric until agreement is high, and report that agreement rate alongside every result. An uncalibrated judge is a random number generator with a confident tone.
Human review where it counts
A defined sample is reviewed by people, permanently, not only during the build. This keeps the judge honest as the system and the traffic drift, and it is the only way to catch the failures the rubric was never written to notice.
The regression gate
All of it runs in your pipeline on every change to a prompt, a model version, a retrieval setting or the application code, with a threshold that blocks a release when quality drops. This is the deliverable that changes behaviour: after it exists, nobody edits a prompt on a Friday and hopes.
Evaluate the system, not the model
Users experience retrieval, prompting, tools and generation as one thing. We score end to end, then attribute failures to the stage that caused them, because a system-level number tells you whether to ship and a stage-level attribution tells you what to fix.
What gets measured
| Dimension | How it is judged | Why it matters |
|---|---|---|
| Correctness | Expert-agreed answers, calibrated judge | The headline number, and the one that must not regress |
| Groundedness | Claims traced to retrieved evidence | Distinguishes a right answer from a lucky one |
| Refusal behaviour | Cases that should not be answered | Confident answers to unanswerable questions are the expensive failure |
| Format compliance | Schema and contract validation | Downstream systems break on malformed output, silently |
| Safety and injection | Adversarial suite | Untrusted input is normal input in production |
| Cost and latency | Tokens and milliseconds per request type | Quality that is unaffordable at volume is not quality |
Where the system is retrieval-based, retrieval is scored separately so a fall in accuracy can be attributed to the index rather than to the model. That separation is described further under RAG system development.
How the engagement runs
Short and self-contained. The output is a capability your team runs, not a report we deliver and leave.
Task definition and sampling
We agree what the system is for, sample real traffic, and identify the failure modes that actually matter to the business.
Golden set construction
Cases curated with your experts, correct outputs agreed and documented, edge cases and refusal cases deliberately included.
Harness and judge calibration
Deterministic checks written, rubric drafted, judge calibrated against human scores until agreement is acceptable and reported.
Baseline and model comparison
The current system scored, and candidate models or configurations benchmarked on the same set with cost and latency alongside quality.
CI integration and handover
The suite wired into your pipeline with a release gate, and a session on maintaining the set as the product changes.
What you receive
A measurement capability that outlives the engagement, with the first set of numbers already in it.
Golden evaluation set
Curated cases with expert-agreed outputs, documented rationale and coverage of edge and refusal cases.
Scoring harness
Deterministic checks and rubric-based judging, runnable locally and in CI, with results stored over time.
Judge calibration report
Agreement rate between the automated judge and your experts, and where it disagrees.
Baseline scorecard
Current system performance across every dimension, with cost and latency.
Model comparison
Candidate models and configurations scored on your task, with the trade-offs stated plainly.
Release gate
CI integration with a threshold that blocks a regression, and the runbook for when it fires.
Is this the right engagement?
Worth being direct. LLM Evaluation and Benchmarking is the wrong spend in some situations, and those are listed rather than buried.
Good fit if
- An LLM feature is in production and nobody can say whether last week's change made it better.
- You are choosing between models and want the decision made on your task rather than a leaderboard.
- A quality regression has already reached customers once.
- A regulator, auditor or enterprise buyer has asked how you assure quality.
- You are about to fine-tune and need a defensible baseline first.
Choose something else if
- Nothing is built yet. Evaluation measures a system; there has to be one.
- Subject experts cannot be made available to agree correct answers. The set cannot be built without them.
- You want a one-off audit rather than an ongoing practice. We can do that, but the value is in the gate.
- The real problem is retrieval quality and you already know it. Go straight to RAG development.
Frequently asked questions
Marked up with FAQPage schema so these answers can surface directly in search results and inside AI assistant responses.
What does LLM evaluation actually measure?
Correctness against expert-agreed answers, groundedness in retrieved evidence, refusal behaviour on questions that should not be answered, format compliance, resistance to adversarial input, and cost and latency per request type. Each is reported separately, because a system can be accurate and unaffordable, or cheap and unsafe.
Can a model be used to judge another model's output?
Yes, and it is the only affordable way to score at volume, but only once it has been calibrated. We measure agreement between the judge and your human experts and publish that rate with every result. Without calibration a model judge is confident noise, which is worse than no measurement at all.
How big does the evaluation set need to be?
A few hundred well-chosen cases usually beats several thousand scraped ones. What matters is coverage of the failure modes that carry business consequence, not raw volume. The set then grows over time as production incidents are folded back into it.
How does this help when we upgrade models?
It turns a leap of faith into an afternoon. Run the new model against the same set, compare quality, cost and latency, and see exactly which request types improved and which regressed. Most teams discover the upgrade is a clear win in some areas and a quiet loss in others.
Who maintains the evaluation set afterwards?
Your team, which is the point. We hand over the set, the harness, the rubric and the calibration method, and set up the habit of adding every production failure as a new case. An evaluation set that stops growing stops being useful within a couple of quarters.
Often paired with this
Most clients combine two or three engagements from the Generative AI & LLM Engineering pillar. These are the ones that most often run immediately before or after.
Prompt and Context Engineering
Prompts treated as versioned, tested software: context budgets, regression suites and measured improvement rather than trial and error.
Read more →RAG System Development
Retrieval augmented generation built as a measured retrieval problem, with chunking, hybrid search, reranking and evaluation.
Read more →LLM Fine-Tuning and Customisation
Fine-tuning when prompting and retrieval have genuinely run out, with curated data, a measured baseline and a clear verdict.
Read more →Is this the right engagement?
Tell us what you are trying to build. If a different service fits better, or if you do not need us at all, we will say so.