Model Distillation and Inference Optimisation
Cutting the cost and latency of a working language model system without giving up the quality it already has, using routing, caching, quantisation and distillation, each change scored before it ships.
A generative AI feature that was affordable in pilot is often alarming at scale. The instinct is to move to a cheaper model and accept a quality drop. That is rarely necessary. In most systems the majority of the spend is going somewhere that has nothing to do with the model's capability.
Inference optimisation is the practice of reducing the cost and latency of a language model system while holding quality at an agreed floor, through request routing, caching, context reduction, batching, quantisation and, where it pays, distilling a large model's behaviour into a smaller one trained specifically on the task.
Where the money actually goes
Before optimising anything we instrument the system and account for spend by request type. The pattern that emerges is remarkably consistent across engagements:
| Cost driver | What we typically find | The lever |
|---|---|---|
| Oversized context | Retrieved passages and history that add tokens but not accuracy | Rerank and prune, compress history |
| Uniform model choice | Trivial requests served by the most expensive model available | Route by request class |
| Repeated work | The same question, prefix or document processed many times | Response and prefix caching |
| Retry storms | Malformed output parsed, failed and regenerated | Constrained decoding and schema validation |
| Unbatched serving | Self-hosted GPUs sitting idle between requests | Continuous batching, higher utilisation |
| Output length | Verbose answers nobody reads to the end | Output contracts and length limits |
None of these require accepting a worse model. Several improve latency and quality at the same time, because a tighter context and a constrained output are easier for a model to get right.
How the optimisation runs
Set the quality floor first
Every optimisation is a trade unless it is measured. We start from your evaluation set, or build one, and fix the score below which no change ships regardless of what it saves. Where no such set exists, that work is the substance of LLM evaluation and benchmarking.
Route by request class
Production traffic is not uniform. A large share of requests are simple classifications, short rewrites or lookups that a much cheaper model handles at the same measured quality. We classify traffic, route accordingly, and keep the strongest model for the cases that genuinely need it, with a fallback when the cheap path scores badly on a live check.
Cache what repeats
Exact-match caching for repeated questions, semantic caching for near-duplicates with a similarity threshold tuned against the evaluation set, and prefix caching for the long, stable instruction blocks that begin every request. In systems with a shared knowledge base the repetition rate is usually far higher than the team expects.
Quantise and batch where you own the serving
On self-hosted deployments the dominant number is accelerator utilisation. Continuous batching, paged attention and appropriate quantisation frequently multiply throughput on the same hardware. Quality is measured at each quantisation level rather than assumed, and the serving architecture is covered further under private and self-hosted LLM deployment.
Distil when the task is stable
Where a narrow, high-volume task is well served by a large model, that model can teach a much smaller one. We generate training data from the strong model's outputs on real inputs, tune a small model on it, and deploy only if it clears the quality floor. The result often runs at a fraction of the cost, and can move on device entirely, as covered under small language models and on-device AI.
Latency and cost are not the same optimisation
Batching improves throughput and can worsen the latency an individual user feels. Caching improves both. Routing usually improves both. We ask which one you are actually optimising for before we start, because the answer changes the design and the two are frequently confused in the brief.
What a realistic outcome looks like
We do not quote a headline saving before seeing a system, because the honest answer depends entirely on how much waste is present. What we do commit to is the shape of the result:
- A per-request-type cost breakdown you did not have before, which usually reframes the conversation on its own.
- A ranked list of changes with the measured saving and the measured quality effect of each, so you choose what to take.
- A quality floor that no shipped change falls below, enforced in your pipeline rather than in a promise.
- A projection of unit cost at ten times current volume, which is the number that decides whether the feature can be rolled out.
Where the wider financial picture matters, this pairs naturally with AI ROI analysis and business case.
How the engagement runs
Instrument, then measure, then change. Nothing ships without a score attached.
Instrumentation and baseline
Token accounting per request type, latency percentiles and current quality scores captured as the baseline everything is compared against.
Opportunity analysis
Spend attributed to drivers, repetition and routing potential quantified, and a ranked list of candidate changes with estimated savings.
Implementation
Routing, caching, context reduction, output contracts and, where applicable, quantisation and batching, each measured against the floor.
Distillation, if it pays
A small model trained on the strong model's outputs for a narrow task, deployed only if it clears the quality floor.
Handover
Dashboards, the quality gate and the maintenance guidance for keeping unit cost flat as traffic grows.
What you receive
A cheaper, faster system and the instrumentation to keep it that way as usage grows.
Cost and latency baseline
Spend and latency attributed per request type, with the repetition and routing analysis behind it.
Optimisation backlog
Ranked changes with measured saving and measured quality effect for each, including the ones we advise against.
Routing layer
Request classification and model routing with a live quality check and fallback.
Caching layer
Exact, semantic and prefix caching with tuned thresholds and invalidation rules.
Distilled model, where applicable
Weights, training data and the comparison against the model it replaces.
Cost dashboard and gate
Ongoing visibility of unit cost and a quality gate that blocks a regression from shipping.
Is this the right engagement?
Worth being direct. Model Distillation and Inference Optimisation is the wrong spend in some situations, and those are listed rather than buried.
Good fit if
- A generative AI feature is live and its cost per request is a problem or about to become one.
- Latency is hurting adoption and you do not want to trade away quality to fix it.
- You are self-hosting and suspect accelerator utilisation is poor.
- A narrow, high-volume task is being served by an expensive general model.
- You need a credible unit cost projection before a wider rollout.
Choose something else if
- Nothing is in production yet. Optimise a real traffic pattern, not an imagined one.
- There is no evaluation set and none can be built, so no change can be shown to be safe.
- Quality is the actual complaint. Fix that first with prompt and context engineering or RAG development.
- Volume is small. Engineering time will cost more than the savings for some time yet.
Frequently asked questions
Marked up with FAQPage schema so these answers can surface directly in search results and inside AI assistant responses.
How much can inference optimisation save?
It depends entirely on how much waste the system currently carries, and we will not quote a number before instrumenting yours. What is consistent is where the waste hides: oversized context, uniform model choice for non-uniform traffic, uncached repetition and unconstrained output. Systems that have never been measured tend to have a lot of all four.
Will optimisation reduce quality?
Not below the floor you set, because that floor is enforced by the evaluation gate rather than by judgement. Several of the changes improve quality as a side effect, since a shorter, better-ordered context and a constrained output format are easier for a model to handle correctly.
What is model distillation?
Training a smaller model to reproduce a larger model's behaviour on a specific task, using the large model's outputs on real inputs as training data. It works well for narrow, stable tasks and poorly for open-ended ones, and it is worth doing when volume is high enough that the ongoing saving justifies owning another model.
Do we have to self-host to benefit?
No. Routing, caching, context reduction and output contracts apply equally to API-based systems and usually account for most of the saving. Quantisation and batching only apply where you control the serving stack.
How do we stop costs creeping back up?
The same way you stop quality regressing: instrumentation and a gate. Unit cost per request type is reported continuously and treated as a metric with a threshold, so a change that quietly doubles context length is caught in the pipeline rather than in the following month's invoice.
Often paired with this
Most clients combine two or three engagements from the Generative AI & LLM Engineering pillar. These are the ones that most often run immediately before or after.
Private and Self-Hosted LLM Deployment
Open-weight models served inside your own tenancy or data centre, with honest GPU sizing and throughput economics.
Read more →Small Language Models and On-Device AI
Task-specific small models that run on a phone, laptop or edge device, with low latency and no data leaving it.
Read more →LLM Evaluation and Benchmarking
Golden sets, calibrated judges and a regression suite in CI, so quality is a number that moves rather than an opinion.
Read more →Is this the right engagement?
Tell us what you are trying to build. If a different service fits better, or if you do not need us at all, we will say so.