AI API and Middleware Development
The service layer between your applications and your models: a versioned contract, authentication, rate limiting, caching and cost control, so that every team is not integrating with a model provider on its own terms.
The pattern appears in most organisations within a year of their first AI project. Four teams have integrated with a model provider independently, each with its own key, prompt handling, retry logic and error behaviour. Nobody can attribute cost, nothing is cached, and changing provider means four separate migrations.
AI middleware is an internal service layer that sits between consuming applications and underlying models, providing a stable versioned interface, authentication and authorisation, rate limiting and quotas, caching, cost attribution, guardrails and observability, so AI capability can be consumed as a governed platform capability.
What the middleware layer provides
| Concern | Without middleware | With it |
|---|---|---|
| Provider credentials | Distributed across teams and repositories | Held once, never exposed to consumers |
| Cost attribution | One provider invoice | Per consuming team, feature and request |
| Rate limiting | One team's spike degrades everyone | Quotas per consumer, with fair queueing |
| Caching | Rarely implemented | Shared across all consumers |
| Provider change | Every team migrates separately | One change behind a stable contract |
| Guardrails | Implemented inconsistently or not at all | Applied uniformly at the boundary |
| Observability | Per team, if at all | One trace and cost view across everything |
Caching alone frequently justifies the layer. Where several applications ask similar questions of the same content, a shared cache removes a substantial share of requests, and no individual team would have built it.
How we build it
Design the contract for consumers, not for the model
The interface exposes capabilities your applications need — classify this, summarise this, answer from this corpus — rather than passing prompts through. That is what lets the implementation change without every consumer changing with it, and it is the whole point of the layer.
Version explicitly, deprecate on a schedule
Consumers depend on behaviour, not only on shape. Versioning covers model and prompt changes too, with a stated deprecation window, because a silent behaviour change behind a stable schema is the failure this layer is meant to prevent.
Authenticate and authorise per consumer
Each application authenticates as itself, with its own quota and permissions. Where the underlying capability touches user-specific data, the user's identity and permissions have to travel with the request rather than being collapsed into a service account.
Quota and rate limit by consumer
So one team's load test does not exhaust the shared budget or degrade a customer-facing feature. Limits are set with consuming teams and enforced with clear errors rather than silent failures.
Cache deliberately
Exact-match caching for repeated requests, and semantic caching for similar ones where the use case tolerates it. Cache keys must include anything affecting the answer — including the user's permissions, or the cache becomes a leak.
Attribute cost at the request
Tokens and spend recorded per consumer, feature and request, so the provider invoice can be explained and optimisation can be targeted. See LLMOps and inference optimisation.
Fail in a defined way
Timeouts, retries, circuit breaking and a stated fallback, so a provider incident degrades consumers predictably instead of producing four different failure behaviours across four applications.
Build this when the second team needs AI, not the first
For a single application, middleware is overhead. The moment a second team integrates independently, the duplication starts and the cost attribution problem begins. Building it at that point is far cheaper than untangling five direct integrations eighteen months later.
Where it sits
- Above the models, whether hosted, privately deployed or your own trained models served through model deployment.
- Below the applications, which consume capabilities rather than integrating with providers.
- Beside your API gateway, reusing existing authentication, logging and traffic management rather than duplicating them.
- Feeding observability, so a single trace and cost view covers everything the organisation does with AI. See AI observability.
How the engagement runs
The contract is agreed with the consuming teams before implementation begins.
Requirements
Existing integrations inventoried; consuming teams' needs, volumes and latency requirements gathered.
Contract design
Capability-level interface, versioning and deprecation policy agreed with consumers.
Build
Service, authentication, quotas, caching, guardrails, cost attribution and failure behaviour implemented.
Migration
Existing direct integrations moved behind the layer, one consumer at a time, with behaviour reconciled.
Handover
Consumer documentation, runbook and working sessions with platform engineering.
What you receive
One governed integration point, with cost attributable and provider changes made once.
Deployed middleware service
Capability-level API with authentication, quotas, caching and defined failure behaviour.
Versioned contract
Interface and behaviour versioning with a stated deprecation policy.
Cost attribution
Tokens and spend per consumer, feature and request, reconcilable to the provider invoice.
Guardrails
Input and output checks applied uniformly at the boundary, tested against bypass attempts.
Observability integration
Traces and metrics feeding your existing telemetry rather than a separate system.
Consumer documentation
How teams integrate, what limits apply, and how the interface will change.
Is this the right engagement?
Worth being direct. AI API and Middleware Development is the wrong spend in some situations, and those are listed rather than buried.
Good fit if
- More than one team is integrating with model providers independently.
- Provider costs cannot be attributed to a team or feature.
- Provider credentials are distributed across repositories.
- You want to change model or provider without every team migrating.
- Guardrails and logging need to apply consistently across all AI use.
Choose something else if
- One application uses AI and no second consumer is expected; this is overhead.
- The requirement is evaluation and release gating. See LLMOps.
- The requirement is serving your own trained models. See model deployment.
- No platform team will own the service after handover.
Frequently asked questions
Marked up with FAQPage schema so these answers can surface directly in search results and inside AI assistant responses.
When is AI middleware worth building?
When the second team needs AI. For a single application it is pure overhead; the moment two teams integrate independently you get duplicated logic, distributed credentials, unattributable cost and no shared caching. Building the layer at that point is far cheaper than untangling five direct integrations later.
Does this add latency?
A small amount for the extra hop, typically a few milliseconds, and it frequently reduces total latency because shared caching serves a meaningful share of requests without touching a model at all. The net effect is usually positive, and it is measured during the build rather than asserted.
Can it let us switch model providers easily?
It makes it one change instead of many, which is the real benefit. It does not make switching free — models behave differently, so a provider change still has to be validated against an evaluation suite before it ships. See LLMOps for that half of the problem.
How does caching work without leaking data between users?
The cache key includes everything that affects the answer, including the requesting user's permissions where the capability touches user-specific data. Caching on the prompt alone is a common shortcut and a reliable way to serve one user content belonging to another.
Should this be part of our existing API gateway?
Usually it should sit behind it and reuse it, rather than duplicating authentication, logging and traffic management. What the AI layer adds on top is model routing, prompt and configuration management, token-level cost attribution, semantic caching and AI-specific guardrails, none of which a general gateway provides.
Often paired with this
Most clients combine two or three engagements from the AI Integration and Process Automation pillar. These are the ones that most often run immediately before or after.
AI Integration into CRM, ERP and HRMS
AI inside the systems people already use, respecting the permission model and surviving vendor upgrades.
Read more →Legacy System Modernisation with AI
AI-assisted comprehension, rule extraction and test generation, with every finding verified before it is trusted.
Read more →Internal Knowledge Management AI
Answers across your internal systems with source, date and permissions respected — and stale content actively removed.
Read more →Is this the right engagement?
Tell us what you are trying to build. If a different service fits better, or if you do not need us at all, we will say so.