Model Deployment and Serving
Getting a model out of the notebook and into production with the latency, throughput and failure behaviour the application genuinely requires, and a rollback that has been rehearsed rather than documented.
The gap between a model that works and a model that serves is mostly made of unglamorous questions. What happens on the tenth concurrent request. What the consuming application does when inference times out. Whether the version that answered a query last Tuesday can be identified. None of them are interesting and all of them are load-bearing.
Model deployment and serving is the infrastructure that exposes a trained model to the systems that use it, with defined latency and throughput characteristics, versioning, scaling behaviour, failure handling and the ability to roll back to a previous version safely.
Choosing the serving pattern
| Pattern | How it works | Suits |
|---|---|---|
| Batch scoring | Scheduled scoring of a set, results written to a store | Predictions consumed later; cheapest and simplest |
| Real-time endpoint | Synchronous request and response | Predictions needed within a user interaction |
| Streaming inference | Scoring events as they arrive | Continuous flows; see streaming |
| Embedded in application | Model runs inside the consuming service | Very low latency, small models, no network hop |
| Edge deployment | Model runs on device or on site | Connectivity, privacy or latency constraints; see edge AI |
Most organisations reach for a real-time endpoint by default and would be better served by batch scoring, which is cheaper, simpler and easier to reason about. The question that settles it is whether the prediction is needed inside a user interaction or merely soon.
How we build serving
Set the requirements as numbers
Latency at the ninety-fifth and ninety-ninth percentiles, sustained throughput, peak throughput, and the availability target. Averages hide the experience users complain about, and a target expressed as 'fast' cannot be designed against or verified.
Decide what happens when inference fails
Timeouts, retries, circuit breaking, and a defined fallback: a cached prediction, a default value, a rules-based answer or a visible degradation. An application whose behaviour on model failure was never specified will discover it during an incident.
Version the endpoint, not just the model
Every response carries the model version that produced it, and previous versions remain deployable. This is what makes rollback a command rather than a rebuild, and what lets you answer a question about a decision made three months ago.
Roll out in stages, with a shadow first where it matters
Shadow deployment scores live traffic without acting on the result, so a new version can be compared against the incumbent on real inputs. Then a canary share, then full traffic, with automatic rollback on error or latency thresholds.
Scale to the real load pattern
Autoscaling configured against measured behaviour, with attention to cold start, which is the usual reason a scaled-out endpoint still misses its latency target during a spike. Where GPUs are involved, the cost of idle capacity has to be designed for deliberately. See GPU cost optimisation.
Log predictions for the questions that come later
Inputs, outputs, model version, latency and timestamp, retained with attention to what may lawfully be stored. This log is the raw material for monitoring, for debugging and for the audit conversation that eventually happens.
The application contract
Serving is a contract with the systems that consume it, and the contract has to be explicit or it will be assumed incorrectly:
- The interface. Request and response schema, versioned, with a compatibility policy so the model can change without breaking callers.
- The latency guarantee. What the caller may assume, and what it must do when that is exceeded.
- Confidence, where the decision needs it. So the application can route uncertain cases to a person rather than acting on a weak prediction.
- The failure contract. What is returned when the model is unavailable, and whether the caller degrades or blocks.
Written down at design time, this prevents the most common production surprise: an application that silently treats a fallback value as a real prediction.
How the engagement runs
Latency and failure behaviour are specified as numbers before any infrastructure is chosen.
Requirements
Latency percentiles, throughput, availability target and failure behaviour agreed with the consuming teams.
Design
Serving pattern, scaling approach, versioning and rollback strategy decided and costed.
Build
Endpoint or batch pipeline implemented with versioning, prediction logging and the failure contract.
Load and failure testing
Performance verified at peak and under spike; failure injection and rollback rehearsed.
Rollout and handover
Shadow then staged rollout, with runbook and working sessions for the owning team.
What you receive
A serving layer that meets a stated latency target and can be rolled back in one step.
Deployed serving infrastructure
Endpoint or batch pipeline with autoscaling configured to measured load.
Interface specification
Versioned request and response schema with a compatibility policy for callers.
Staged rollout capability
Shadow, canary and full rollout with automatic rollback thresholds.
Prediction logging
Inputs, outputs, version, latency and timestamp, retained lawfully.
Load test results
Measured latency at p95 and p99 and behaviour under spike, against the agreed target.
Runbook
Deployment, rollback, failure handling, scaling and cost levers.
Is this the right engagement?
Worth being direct. Model Deployment and Serving is the wrong spend in some situations, and those are listed rather than buried.
Good fit if
- A model is validated and needs to serve production traffic.
- An existing endpoint misses its latency target or falls over under spike.
- Rollback is currently a rebuild rather than a command.
- The consuming application has no defined behaviour when inference fails.
- You cannot identify which model version produced a past prediction.
Choose something else if
- No model exists yet. See custom ML models.
- The requirement is the whole operational pipeline. See MLOps implementation.
- The model must run on device or on site. See edge AI deployment.
- The model is fast enough and the real problem is inference cost. See inference optimisation.
Frequently asked questions
Marked up with FAQPage schema so these answers can surface directly in search results and inside AI assistant responses.
Should we serve predictions in real time or in batch?
Batch wherever the prediction is needed soon rather than inside a user interaction — it is cheaper, simpler and far easier to operate. Real-time endpoints are justified when a person or a transaction is waiting on the answer. A large share of real-time deployments we are asked to fix should have been batch jobs.
How do you handle traffic spikes?
Autoscaling configured against measured load rather than guessed load, with explicit attention to cold start, which is the usual reason an endpoint still misses its target during the spike it scaled for. Load testing to peak and beyond is part of the build rather than something done after the first bad day.
What happens if the model service goes down?
Whatever you decided it should, which is why the failure contract is agreed at design time: a cached prediction, a default, a rules-based fallback or visible degradation. The dangerous version is an application that silently treats a fallback as a real prediction, and that is what specifying it prevents.
Can we test a new model on real traffic without risk?
Yes, through shadow deployment: the new version scores live requests and its outputs are recorded and compared, but nothing acts on them. It is the safest way to validate on real inputs, and it is followed by a canary share before full traffic.
Do you deploy on our cloud or yours?
Yours. Models, data and logs stay within your accounts and your regions, which matters for data residency and for the audit conversation later. We work across the major clouds and on-premises; see cloud AI setup where the environment itself needs building.
Often paired with this
Most clients combine two or three engagements from the MLOps, LLMOps and AI Infrastructure pillar. These are the ones that most often run immediately before or after.
Inference Optimisation and Latency Tuning
Profiling first, then quantisation, batching, caching and routing, each verified against an accuracy bar.
Read more →Model Monitoring and Drift Detection
Drift, delayed-label accuracy and segment-level performance, with alerts that lead to a decision.
Read more →MLOps Pipeline Implementation
Versioned data and models, automated retraining, tested deployment and a rollback anyone on the team can run.
Read more →Is this the right engagement?
Tell us what you are trying to build. If a different service fits better, or if you do not need us at all, we will say so.