LLMOps and Prompt Versioning
The operational layer under a language model application: versioned prompts, an evaluation suite that decides whether a change ships, and enough cost and quality instrumentation to switch models deliberately rather than hopefully.
Language model applications have a specific and unpleasant failure pattern. Someone improves a prompt on Friday, three other behaviours quietly get worse, and nobody finds out until a customer complains in March. Without an evaluation suite, every prompt change is an uncontrolled experiment on production.
LLMOps is the operational practice for applications built on large language models: versioning prompts and configurations, evaluating changes against a fixed suite before release, tracking cost, latency and quality per version, and managing model and provider changes without regression.
Why this is not simply MLOps
| Concern | Classic MLOps | LLMOps |
|---|---|---|
| The versioned artefact | Model weights from a training run | Prompt, configuration, retrieval settings and model choice |
| What 'correct' means | Compared against a labelled ground truth | Often judged, with several acceptable answers |
| The change that breaks things | Retraining or data drift | A prompt edit, or a provider updating a model beneath you |
| Cost behaviour | Mostly fixed after training | Per request, and volatile with prompt length and traffic |
| Regression risk | Measurable on a test set | Invisible without a deliberately built evaluation suite |
| Failure mode | Accuracy declines | Confident, fluent, wrong |
The third row is the one that surprises organisations. A hosted model can change under a stable version label, and an application that was reliable on Monday behaves differently on Thursday with no change on your side. The evaluation suite is what turns that from a mystery into an alert.
What we build
Prompts as versioned artefacts
Prompts, system messages, tool definitions, retrieval parameters and model settings held in version control with review, not embedded in application code or edited in a console. Every production response is attributable to an exact configuration version.
An evaluation suite that actually gates
A fixed set of cases with expected behaviour, run automatically on every change, covering the ordinary path, known failure modes, adversarial inputs and safety boundaries. A change that regresses the suite does not ship. Building the suite with your experts is the substantial part of this engagement and the part that pays back longest. See LLM evaluation and testing.
Cost and latency tracked per version
Tokens in and out, cost per request and per feature, latency at p95 and p99, all attributed to prompt version. A prompt that improves quality by three per cent and increases cost by sixty is a decision, and it should be made with the numbers visible.
Provider and model abstraction, used sparingly
Enough abstraction to change model or provider without rewriting the application, and no more, because heavy abstraction layers obscure the model-specific behaviour that determines quality. Model changes are validated through the evaluation suite before any switch is made.
Production feedback captured as evaluation cases
User corrections, thumbs-down, escalations and support tickets routed back into the evaluation suite. This is how the suite stays representative rather than becoming a museum of the failure modes you had at launch.
Guardrails and their own tests
Input and output checks for the things that must never happen in your domain, versioned and tested like everything else. Guardrails that were never tested against deliberate attempts to bypass them are decorative.
Twenty good cases beat two hundred careless ones
Teams postpone building an evaluation suite because they imagine it is an enormous undertaking. It is not. Twenty to fifty cases covering your real usage and known failures make every subsequent change measurable, and the suite grows naturally from production feedback afterwards.
What changes once it exists
- Prompt changes stop being frightening. The suite says whether the change is an improvement, so iteration accelerates rather than being avoided.
- Model upgrades become a decision with evidence. Run the suite against the new model, read the quality and cost difference, decide.
- Provider incidents are detected rather than reported by users. Continuous evaluation on a sample catches silent behaviour changes.
- Cost becomes attributable. Per feature and per version, so optimisation targets the right thing. See inference optimisation.
- Regressions are caught before release rather than in a customer conversation three months later.
How the engagement runs
The evaluation suite is built first, because nothing else can be measured without it.
Assessment and case collection
Current setup reviewed; real usage, failure modes and edge cases collected from production and from your experts.
Evaluation suite
Cases written with expected behaviour and judging criteria, automated and baselined against the current system.
Versioning and instrumentation
Prompts and configuration moved into version control; cost, latency and quality tracking attributed per version.
Release gating and guardrails
Suite wired into the release path; guardrails implemented and tested against deliberate bypass attempts.
Handover
Feedback loop into the suite established; working sessions until your team ships a gated change themselves.
What you receive
An evaluation suite that gates releases, and the instrumentation to see what each change costs.
Evaluation suite
Cases with expected behaviour and judging criteria, automated and baselined.
Versioned prompt configuration
Prompts, tools, retrieval settings and model choice under review in version control.
Release gating
Suite wired into deployment so a regressing change cannot ship.
Cost and quality dashboard
Tokens, cost, latency and evaluation scores attributed per version and per feature.
Guardrail implementation
Input and output checks, tested against deliberate bypass attempts.
Feedback loop
Production corrections and escalations routed into the suite as new cases.
Is this the right engagement?
Worth being direct. LLMOps and Prompt Versioning is the wrong spend in some situations, and those are listed rather than buried.
Good fit if
- A language model application is in production and prompt changes are made on instinct.
- You cannot tell whether last month's change helped or hurt.
- Model or provider costs are rising and cannot be attributed to a feature.
- You want to change model or provider and need to know what it would break.
- A regression reached customers and nobody could say when it was introduced.
Choose something else if
- No language model application exists yet. Start with LLM application development.
- The workload is trained models rather than language models. See MLOps implementation.
- No expert time is available to define correct behaviour, which leaves nothing to evaluate against.
- The application is a short-lived experiment nobody intends to maintain.
Frequently asked questions
Marked up with FAQPage schema so these answers can surface directly in search results and inside AI assistant responses.
What is LLMOps?
The operational practice for applications built on large language models: versioning prompts and configuration, evaluating every change against a fixed suite before release, tracking cost and latency per version, and managing model changes without regression. It solves the same class of problem as MLOps with entirely different primitives.
How is LLMOps different from MLOps?
MLOps versions training data and model weights and measures accuracy against a labelled test set. LLMOps versions prompts and configuration, and correctness is frequently judged rather than computed because several answers can be acceptable. It also has to cope with the model itself changing beneath you, which does not happen with weights you trained and hold.
How many evaluation cases do we need?
Fewer than teams fear. Twenty to fifty cases covering real usage, known failure modes and safety boundaries make every subsequent change measurable, and the suite grows from production feedback afterwards. The common mistake is postponing it until there is time to do it thoroughly, which never arrives.
How do you handle a provider changing the model underneath us?
By running the evaluation suite continuously on a sample rather than only at release, so a behaviour change is detected as an alert rather than as a customer complaint. Pinning to specific model versions where the provider offers it, and validating any forced migration through the suite before it takes effect.
Can you help us reduce what our LLM application costs?
Yes, and the instrumentation is the prerequisite: cost attributed per feature and per version shows where the money actually goes, which is rarely where teams assume. Once that is visible, prompt reduction, caching, routing simple requests to smaller models and other measures can be targeted. See inference optimisation.
Often paired with this
Most clients combine two or three engagements from the MLOps, LLMOps and AI Infrastructure pillar. These are the ones that most often run immediately before or after.
AI Observability and Tracing
End-to-end tracing of every retrieval, prompt, tool call and token, so a bad answer can be explained.
Read more →Inference Optimisation and Latency Tuning
Profiling first, then quantisation, batching, caching and routing, each verified against an accuracy bar.
Read more →GPU Infrastructure and Cost Optimisation
Utilisation profiling, right-sizing, scheduling and purchasing strategy, measured before anything is changed.
Read more →Is this the right engagement?
Tell us what you are trying to build. If a different service fits better, or if you do not need us at all, we will say so.