Inference Optimisation and Latency Tuning
Making each prediction faster and cheaper without quietly making it worse, starting from a profile of where the time and the money actually go rather than from a list of techniques.
Almost every inference optimisation engagement begins with a team ready to quantise the model, and almost every profile shows the model is not the problem. It is a serial retrieval step, an unbatched queue, a missing cache on requests that repeat all day, or a prompt carrying four thousand tokens of context nobody has read since launch.
Inference optimisation reduces the latency and cost of serving predictions, through profiling to locate the actual bottleneck and then applying techniques — batching, caching, quantisation, distillation, compilation, routing — each validated against a defined accuracy floor.
Set the accuracy floor before touching anything
Every meaningful optimisation trades something. Deciding what may be traded, and by how much, is the first conversation rather than the last:
- The accuracy floor. The measured quality below which the system is not acceptable, agreed with the business owner and not with the engineering team alone.
- The latency target. At the ninety-fifth and ninety-ninth percentiles, because the average is not the experience anyone complains about.
- What the cost reduction is worth. A thirty per cent saving that costs two points of accuracy is a business decision, not a technical one.
- The evaluation set that decides it. Fixed, representative, and used to verify every change before it stands.
Profile first, always
We have yet to run an engagement where the first assumption about the bottleneck was correct. Profiling costs a few days and routinely redirects the entire effort — most often from the model to the surrounding pipeline, where the fix is simpler and the saving larger.
The techniques, roughly in order of how often they help
| Technique | What it does | Typical effect |
|---|---|---|
| Caching | Reuses results for repeated or similar requests | Large where traffic repeats; free accuracy |
| Batching | Groups concurrent requests through the hardware together | Major throughput gain; slight latency cost |
| Prompt reduction | Removes context that no longer earns its tokens | Direct cost and latency saving on LLM workloads |
| Quantisation | Serves at lower numeric precision | Substantial speed and memory gain; small accuracy risk |
| Compilation and runtime tuning | Optimises the model graph for the target hardware | Meaningful speedup; no accuracy change |
| Model routing | Sends easy requests to a smaller model | Large cost saving where difficulty varies |
| Distillation | Trains a small model to imitate a large one | Largest saving; largest effort |
| Streaming responses | Returns tokens as generated | Perceived latency only; often the cheapest win |
The last row is worth taking seriously. Where a user is reading the output, streaming the first tokens quickly changes the experience more than shaving a second off total generation, and it costs almost nothing to implement.
Where the bottleneck usually is
In the pipeline, not the model
Retrieval, reranking, tool calls and post-processing frequently dominate. Running independent steps in parallel rather than in sequence is the most common single improvement we make, and it changes no model behaviour at all.
In the context length
For language model workloads, cost and latency scale with tokens. Prompts accumulate instructions and examples that were added for a failure mode long since fixed, and pruning them against the evaluation suite is fast, safe and immediately visible on the bill.
In the absence of batching
Accelerators are parallel hardware, and serving one request at a time wastes most of the capacity. Dynamic batching in the serving layer trades a few milliseconds of queueing for several times the throughput, which usually also resolves the cost question.
In cold starts
An endpoint that scales to zero is cheap and slow to wake. Where a small share of requests hit a cold start, the ninety-ninth percentile latency is terrible while the average looks fine, and the fix is provisioned concurrency rather than a faster model.
In doing hard work for easy requests
Where request difficulty varies, routing straightforward cases to a smaller or cheaper model and reserving the large one for genuinely hard ones is often the largest saving available. It requires a reliable difficulty signal and an evaluation set to prove nothing was lost.
How the engagement runs
The accuracy floor is agreed before profiling, so every later change has a bar to clear.
Baseline and floor
Current latency, throughput, cost and accuracy measured; the accuracy floor and latency target agreed in writing.
Profiling
Time and cost attributed per step, so the real bottleneck is identified rather than assumed.
Implementation
Techniques applied in order of value, each verified against the accuracy floor and latency target.
Load verification
Improvements confirmed at production concurrency and under spike, not only in isolation.
Handover
Before-and-after report, the evaluation harness and guidance for keeping the gains.
What you receive
Measured reductions in latency and cost per request, each verified not to have cost accuracy.
Baseline measurement
Latency percentiles, throughput, cost per request and accuracy before any change.
Profile report
Time and cost attributed per pipeline step, with the real bottleneck identified.
Implemented optimisations
Applied in order of value, each verified against the agreed accuracy floor.
Verification results
Before-and-after latency, cost and accuracy at production concurrency.
Evaluation harness
Reusable, so future changes can be checked against the same bar.
Maintenance guidance
How to keep the gains as models, prompts and traffic change.
Is this the right engagement?
Worth being direct. Inference Optimisation and Latency Tuning is the wrong spend in some situations, and those are listed rather than buried.
Good fit if
- Inference latency is a user complaint or breaches an internal target.
- Cost per request makes the economics of a feature marginal.
- Throughput is limited and scaling out is the only current answer.
- Language model context has grown and nobody has pruned it.
- Request difficulty varies widely and everything is served by the largest model.
Choose something else if
- Performance is acceptable and the pressure is infrastructure cost overall. See GPU cost optimisation.
- No accuracy floor can be agreed, which leaves every trade-off unresolvable.
- The model must run on device. See edge AI deployment.
- The serving layer does not exist yet. See model deployment and serving.
Frequently asked questions
Marked up with FAQPage schema so these answers can surface directly in search results and inside AI assistant responses.
What actually causes slow inference?
Usually not the model. In most engagements the dominant cost is elsewhere in the pipeline — serial retrieval, unbatched requests, missing caching, an oversized prompt, or cold starts. Profiling takes a few days and reliably redirects the work to somewhere simpler and more valuable than the model itself.
Does quantisation reduce accuracy?
It can, which is why it is verified against a fixed evaluation set and an agreed accuracy floor rather than assumed to be free. In practice the loss at common precisions is often negligible for the task at hand, but 'often negligible' is not a substitute for measuring it on your model and your data.
How much can inference cost realistically be reduced?
It depends entirely on the starting point, which is why we measure before promising. Where caching, batching and prompt reduction have never been applied, the reductions are usually substantial; where the pipeline is already tuned, the remaining gains are smaller and harder. The baseline report tells you which situation you are in.
What is model distillation?
Training a smaller model to reproduce the behaviour of a larger one on your specific task. It offers the largest sustained saving and requires the most effort, including a good dataset of the large model's outputs. It is worth it at high volume on a stable task, and rarely worth it otherwise. See model optimisation.
Can you optimise without changing our model?
Frequently, yes, and it is where we start. Caching, batching, parallelising pipeline steps, compilation and prompt reduction all leave model behaviour untouched, which means no accuracy risk to argue about. Techniques that alter the model come later, once the free improvements are exhausted.
Often paired with this
Most clients combine two or three engagements from the MLOps, LLMOps and AI Infrastructure pillar. These are the ones that most often run immediately before or after.
GPU Infrastructure and Cost Optimisation
Utilisation profiling, right-sizing, scheduling and purchasing strategy, measured before anything is changed.
Read more →Model Deployment and Serving
Serving infrastructure sized to your real latency and throughput, with staged rollout and rehearsed rollback.
Read more →Edge AI Deployment
Models sized to real hardware, offline operation, safe over-the-air updates and fleet-wide monitoring.
Read more →Is this the right engagement?
Tell us what you are trying to build. If a different service fits better, or if you do not need us at all, we will say so.