AI Observability and Tracing
Seeing inside a language model application or an agent run: every retrieval, prompt, tool call and token, linked into one trace, so that a wrong answer can be explained rather than reproduced by guesswork.
A user reports that the assistant gave a wrong answer on Tuesday. Without tracing, the investigation is an attempt to recreate the conditions from memory. With tracing, it is opening the trace and seeing that retrieval returned the superseded version of the policy document. One of these takes ten minutes.
AI observability is the instrumentation of language model applications and agent systems so that each request produces a complete trace: the inputs, the retrieval performed, the prompts constructed, the model calls made, the tools invoked, the tokens and cost consumed, and the latency of each step.
Why standard application monitoring is not enough
Conventional monitoring answers whether the service responded and how quickly. For an AI system those are the least interesting questions, because the characteristic failure returns a fast, successful, entirely wrong response:
- Multiple steps produce one answer. Retrieval, reranking, several model calls, tool invocations. The failure is in one step and the response looks normal.
- Non-determinism. The same input can produce different behaviour, so reproducing an issue from a description often fails.
- Cost varies per request. By orders of magnitude, depending on context length and how many steps ran, and it is invisible without per-trace attribution.
- Agents make their own decisions. An agent loop may take a different path each run, and without a trace nobody can say which path it took.
- Quality is not an error code. Nothing in the infrastructure knows the answer was wrong.
What we instrument
The full trace, as a tree
One trace per request, with each step as a nested span: retrieval with the documents returned and their scores, the constructed prompt, the model response, tool calls with their arguments and results, and timing on every one. The tree structure is what makes a multi-step failure legible.
Cost and tokens per step
Attributed to feature, user segment and prompt version. Almost every organisation we work with is surprised by where the money goes — commonly a background summarisation step nobody remembered, or a retry loop that fires more often than anyone believed.
Quality signals joined to traces
Thumbs-down, corrections, escalations to a human, abandoned sessions and evaluation scores, linked to the trace that produced them. This is what turns a complaint into a specific, fixable finding rather than a general sense that the system is unreliable.
Latency broken down by step
Users experience total time; engineers can only fix a specific step. Breaking down retrieval, generation and tool time is what reveals that the eight-second response is six seconds of a slow tool call rather than the model.
Privacy decided deliberately
Traces contain user inputs and retrieved content, which frequently include personal or confidential data. Redaction, retention limits and access control are agreed at the start, because trace storage is a data store like any other and is often overlooked in a privacy review.
Alerting on the things that matter here
Error rates and latency, but also cost per request moving, retrieval returning nothing, guardrails firing more often, evaluation scores falling on the continuous sample, and agent runs hitting their step limit.
Agents make this non-optional
A multi-step agent without tracing cannot be debugged in any meaningful sense — you can see the input and the outcome and nothing in between. Where agents are in production, observability is not an enhancement, it is the difference between operating the system and hoping. This engagement provides the tracing layer; AgentOps adds the trajectory evaluation, spend ceilings and circuit breakers that autonomy specifically requires.
What it changes
| Before | After |
|---|---|
| A bad answer is investigated by trying to recreate it | The trace shows which step failed |
| Cost is a single monthly figure | Cost attributed per feature, version and segment |
| Latency complaints are met with guesses | The slow step is identified in minutes |
| Agent behaviour is opaque | Every decision, tool call and result is visible |
| Quality is discussed as a feeling | Quality signals joined to the traces that caused them |
| A regression is noticed weeks later | Alerting on evaluation scores and guardrail rates |
How the engagement runs
Instrumentation follows the real failure modes, so the traces answer questions people actually ask.
Assessment
Application architecture reviewed; the questions the team currently cannot answer are catalogued.
Design and privacy
Trace structure, retention, redaction and access agreed; platform chosen against your existing telemetry stack.
Instrumentation
Tracing implemented across retrieval, prompting, model calls and tools, with cost and token attribution.
Quality signals and alerting
Feedback and evaluation joined to traces; alerts configured on cost, quality and guardrail behaviour.
Handover
Dashboards, investigation runbook and working sessions debugging real production traces together.
What you receive
Traces that answer the questions your team currently cannot, with cost attributed per feature.
Trace instrumentation
End-to-end tracing across retrieval, prompts, model calls and tools, with timing per step.
Cost attribution
Tokens and spend per trace, feature, version and segment.
Quality signal integration
Feedback, escalations and evaluation scores joined to the traces that produced them.
Privacy configuration
Redaction, retention and access control on trace data, agreed and documented.
Alerting
Cost movement, retrieval failure, guardrail rate, evaluation decline and agent step exhaustion.
Investigation runbook
How to go from a user complaint to the responsible step, with worked examples.
Is this the right engagement?
Worth being direct. AI Observability and Tracing is the wrong spend in some situations, and those are listed rather than buried.
Good fit if
- A language model application or agent is in production and failures are hard to explain.
- Costs are rising and cannot be attributed to a feature.
- Users report bad answers that nobody can reproduce.
- Latency is a complaint and the slow step is unknown.
- Agents run multi-step and their behaviour is opaque.
Choose something else if
- The system is a trained model rather than a generative one. See model monitoring.
- Nothing is in production yet; instrument during the build instead.
- The requirement is evaluation and release gating. See LLMOps.
- Trace data cannot be stored under your privacy constraints and no redaction is acceptable.
Frequently asked questions
Marked up with FAQPage schema so these answers can surface directly in search results and inside AI assistant responses.
What is AI observability?
Instrumentation that produces a complete trace of every AI request: inputs, retrieval and what it returned, the prompts built, model calls, tool invocations, tokens, cost and per-step latency. It exists because standard monitoring reports a fast successful response for an answer that was completely wrong.
How is this different from model monitoring?
Model monitoring watches statistical behaviour — drift and accuracy — for trained models. Observability traces individual executions of a generative or agent system so a specific bad response can be explained. Different questions: one asks whether the model is still right in aggregate, the other asks what happened on this request.
Does tracing slow the application down?
Negligibly when implemented properly. Traces are emitted asynchronously and sampling can be applied to high-volume paths while keeping every error and every flagged response. The overhead is far smaller than the cost of debugging without it.
What about personal data in traces?
It is a real concern and it is settled before instrumentation, not after. Traces hold user inputs and retrieved content, so redaction of sensitive fields, retention limits and access control are designed in. Trace storage is a data store and belongs in your privacy assessment like any other.
Which observability platform do you use?
We work with the established LLM observability tools and with general tracing platforms, and choose on what your organisation already runs. Where you have a telemetry stack in place, extending it usually beats adding a separate system your team has to learn and maintain.
Often paired with this
Most clients combine two or three engagements from the MLOps, LLMOps and AI Infrastructure pillar. These are the ones that most often run immediately before or after.
LLMOps and Prompt Versioning
Versioned prompts, an evaluation suite that gates releases, and cost, latency and quality tracked per change.
Read more →Model Monitoring and Drift Detection
Drift, delayed-label accuracy and segment-level performance, with alerts that lead to a decision.
Read more →Inference Optimisation and Latency Tuning
Profiling first, then quantisation, batching, caching and routing, each verified against an accuracy bar.
Read more →Is this the right engagement?
Tell us what you are trying to build. If a different service fits better, or if you do not need us at all, we will say so.