Small Language Models and On-Device AI
Compact, task-specific language models that run on a laptop, handset or edge device, chosen when latency, unit cost or the fact that data must never leave the device decide the architecture.
A great deal of what organisations use frontier models for is not frontier work. Classifying a support ticket, extracting five fields from a form, redacting a name, drafting a two-line reply: these are narrow, high-volume tasks where a model a hundredth of the size does the job at a hundredth of the cost, on hardware you already own.
Small language models are compact models, typically in the range of a few hundred million to about ten billion parameters, specialised for a narrow task and small enough to run on consumer hardware, a phone or an edge device. On-device AI is the engineering that makes them usable there: quantisation, runtime selection, memory and battery management, and a fallback path when the device cannot cope.
Why a small language model is often the better engineering choice
- Latency. No network round trip. Interactive features feel instantaneous in a way that a remote call cannot match, and they keep working on a bad connection.
- Unit cost. Inference on hardware you already own is close to free at the margin, which changes what is worth automating at all.
- Privacy. Data that never leaves the device removes an entire category of compliance argument, rather than mitigating it.
- Availability. The feature works on a plane, in a factory, in a hospital basement and during a provider outage.
- Predictability. No rate limits, no deprecation notices, no pricing changes mid-quarter.
The trade is capability. A small model will not reason across a long, ambiguous document. It will classify, extract, rewrite, redact, route and summarise short content extremely well, and those tasks are a large share of what is actually being paid for today.
How we build on-device AI
Prove the task is small enough
We start by scoring a candidate small model against your task with the same evaluation set used for the frontier baseline. If the gap is unacceptable, the honest answers are distillation, a hybrid design, or not doing it on device at all, and we would rather reach that conclusion in week two.
Specialise, do not generalise
Small models earn their keep when they do one thing. We fine-tune on the narrow task, usually with parameter-efficient methods, and where examples are short we generate more, as described under LLM fine-tuning and synthetic data generation. Distilling from a larger model that already performs the task well is often the fastest route, covered under model distillation and inference optimisation.
Quantise for the target device
Four-bit and eight-bit quantisation decide whether the model fits in memory alongside your application. We measure quality at each level rather than assuming the loss is negligible, and we test on the oldest device you intend to support rather than the newest, because that is where the product actually fails.
Choose the runtime deliberately
The runtime determines whether you can reach the accelerator on the device, how the model is packaged and how large the binary becomes. We select for the platforms you ship to, measure cold start, tokens per second, memory ceiling, thermal behaviour and battery draw under sustained use, and design the packaging and update path around the result.
Design the fallback
Some devices will be too old, too full or too hot. The product needs a defined behaviour for that case: degrade to a server call where policy permits, degrade to a simpler feature where it does not, and tell the user something truthful either way.
Battery and thermals are product decisions
A model that runs beautifully for ninety seconds and then throttles is a support problem, not a feature. We measure sustained performance and heat, not just a first benchmark, and we size the model to what the device can hold for as long as the user will realistically use it.
Where small models fit best
| Task | Why a small model suits it | Typical placement |
|---|---|---|
| Classification and routing | Narrow label space, short inputs, very high volume | On device or in-process |
| Field extraction from forms | Structured output, repetitive patterns | Edge or on-premise |
| Redaction before egress | Data must be cleaned before it can leave | On device, always |
| Short rewriting and tone | Bounded task, immediate feedback expected | On device |
| Offline assistance | Connectivity cannot be assumed | On device, with sync |
| Long-document reasoning | Poor fit; capability gap is real | Server-side model |
Most products end up with a routing layer: the small model handles the common cases locally and escalates the rest. Getting that boundary right is usually the difference between a feature that feels fast and one that feels unreliable.
How the engagement runs
Feasibility on your actual hardware is settled early, because that is what the whole design depends on.
Task fit and baseline
The task is scored against a frontier baseline and a candidate small model, and target devices and their constraints are documented.
Specialisation
Fine-tuning or distillation on the narrow task, with quality tracked against the baseline at each step.
Quantisation and runtime
Quality measured at each quantisation level, runtime selected, and cold start, throughput, memory, thermals and battery measured on real devices.
Integration and fallback
Packaging, update path, routing between local and server inference, and the degradation behaviour for constrained devices.
Handover
Model artefacts, build pipeline, device benchmarks and the maintenance plan for future model and OS versions.
What you receive
A model that runs on the hardware you actually ship to, with the numbers to prove it.
Specialised small model
Trained or distilled for your task, quantised, with weights and training configuration.
Device benchmark report
Cold start, tokens per second, memory, thermal and battery behaviour across your supported device range.
Quality comparison
Small model against the frontier baseline on your evaluation set, per request type.
Runtime integration
Packaging, loading, streaming and update path for each target platform.
Routing and fallback design
When work stays local, when it escalates, and what the user sees when neither is possible.
Maintenance plan
How to retrain, requantise and revalidate when the base model or the OS moves.
Is this the right engagement?
Worth being direct. Small Language Models and On-Device AI is the wrong spend in some situations, and those are listed rather than buried.
Good fit if
- Data must not leave the device for privacy, contractual or regulatory reasons.
- The task is narrow, high volume and currently expensive per call.
- The feature has to work with poor connectivity or none at all.
- Interactive latency matters enough that a network round trip is noticeable.
- You ship software to hardware you control or know well.
Choose something else if
- The task needs reasoning over long, ambiguous documents. The capability gap is real.
- Your device fleet is unknown or extremely varied, and the oldest devices cannot be characterised.
- Volume is low, in which case an API call is simpler and cheaper to own.
- You need a private deployment rather than an on-device one. See private LLM deployment.
Frequently asked questions
Marked up with FAQPage schema so these answers can surface directly in search results and inside AI assistant responses.
What counts as a small language model?
In practice, anything small enough to run acceptably on the hardware you are targeting, which today means roughly a few hundred million to ten billion parameters after quantisation. The useful definition is device-relative rather than absolute, which is why we benchmark on your oldest supported device rather than quoting a parameter count.
Can a small model match a frontier model?
On a narrow, well-specified task, frequently yes, particularly after fine-tuning or distillation on that task. Across open-ended reasoning, no, and any supplier who tells you otherwise is selling something. The engagement establishes which of those two situations you are in before the build.
Which devices can run on-device AI?
Recent phones, laptops and many edge devices with a neural accelerator run quantised small models comfortably. The constraint is usually memory shared with your application rather than raw compute, and thermal behaviour under sustained use rather than a peak benchmark.
Does on-device inference remove our privacy obligations?
It removes the transfer, which is the largest part of the argument, but not the obligation. You still process personal data, still need a lawful basis and still need to handle what the model outputs and what is cached. It makes the compliance story much simpler; it does not make it disappear.
How do updates work once a model ships to devices?
Models are versioned and distributed like any other asset, with a rollback path and a minimum supported version. We design the update mechanism during the build, because retrofitting model distribution to a shipped application is considerably harder than planning it.
Often paired with this
Most clients combine two or three engagements from the Generative AI & LLM Engineering pillar. These are the ones that most often run immediately before or after.
Model Distillation and Inference Optimisation
Cost and latency reduced through routing, caching, quantisation and distillation, with quality held to an agreed floor.
Read more →LLM Fine-Tuning and Customisation
Fine-tuning when prompting and retrieval have genuinely run out, with curated data, a measured baseline and a clear verdict.
Read more →Private and Self-Hosted LLM Deployment
Open-weight models served inside your own tenancy or data centre, with honest GPU sizing and throughput economics.
Read more →Is this the right engagement?
Tell us what you are trying to build. If a different service fits better, or if you do not need us at all, we will say so.