Agentic RAG
Retrieval that plans before it searches, decomposes a hard question into answerable parts, checks what it found and searches again when the evidence is thin, for the questions single-shot retrieval simply cannot reach.
Standard retrieval answers questions whose answer sits in one place. Ask it something that requires comparing two contracts, tracing a change across three policy versions, or checking whether a stated condition actually holds, and it retrieves the most similar-looking passages and produces something confident. Agentic RAG exists for those questions, and it costs more, which is why it should be used deliberately.
Agentic RAG is retrieval augmented generation in which the system plans its own retrieval: decomposing a question into sub-questions, choosing which sources to search, evaluating whether the evidence gathered is sufficient, searching again where it is not, and verifying the answer against the retrieved material before returning it.
When single-shot retrieval is not enough
- Multi-hop questions. The answer requires a fact from one document to locate the relevant part of another.
- Comparison. Two or more sources have to be retrieved and contrasted, not summarised together.
- Aggregation. The answer is a count, a total or a pattern across many documents rather than a passage in one.
- Temporal reasoning. Which version applied at a particular date, and what changed between versions.
- Conditional questions. Whether a rule applies to a specific case, which requires checking the rule and the case separately.
For everything else, and it is usually the majority of traffic, single-shot retrieval is faster, cheaper and easier to reason about. We build both and route between them, because paying agentic prices for a simple lookup is a design error rather than a sophistication.
How agentic RAG works in practice
Plan before retrieving
The system classifies the question, decides whether it needs decomposition at all, and where it does, produces an explicit plan of sub-questions and the sources each should be answered from. The plan is visible in the trace, which is what makes the behaviour debuggable rather than mysterious.
Retrieve per sub-question, with the right tool
Different sub-questions want different retrieval: semantic search over policy text, a filtered query over structured records, a lookup by identifier. Each becomes a scoped tool, so retrieval strategy is a tool-selection problem with the same discipline as any other, as covered under tool and function-calling integration.
Judge sufficiency, then stop or continue
After each round the system assesses whether the evidence answers the sub-question. Thin evidence triggers a reformulated search; repeated failure triggers an honest 'not found in the sources available' rather than an inference dressed as a finding. A hard ceiling on rounds keeps cost and latency bounded.
Verify the answer against the evidence
Before returning, each claim is checked against the passages retrieved, and unsupported claims are removed or flagged. This verification pass is the single most valuable component in the design, and it is what makes agentic RAG defensible for legal, compliance and clinical use.
Route by question type
A classifier sends straightforward questions down the fast single-shot path and reserves the agentic path for questions that need it. Most deployments end up with the majority of traffic on the cheap path, which is what makes the whole thing affordable.
The foundation still has to be good
Agentic retrieval magnifies whatever the underlying index gives it: better recall means better plans, and a weak index means several expensive rounds of finding nothing. We measure and fix baseline retrieval first, as set out in RAG system development, before adding a planning loop on top of it.
What it costs, and what it buys
| Dimension | Single-shot RAG | Agentic RAG |
|---|---|---|
| Latency | Typically one to three seconds | Several seconds to tens of seconds |
| Cost per question | One retrieval, one generation | Several retrievals, planning and verification |
| Multi-hop accuracy | Poor, and confidently so | Materially better, and traceable |
| Explainability | Which passages were used | The plan, each search, and the sufficiency judgement |
| Failure mode | A fluent wrong answer | An explicit 'not found', more often than not |
That last row is why the extra cost is worth paying for high-stakes questions. A system that says it could not find the answer is far more useful to a professional than one that invents a defensible-sounding paragraph.
How the engagement runs
Baseline retrieval is fixed first, then the planning layer is added only where the score says it is needed.
Question taxonomy and baseline
Real questions classified by type, single-shot baseline measured per type, and the subset that genuinely needs planning identified.
Retrieval tools
Sub-question retrieval built as scoped tools across your sources, each measured independently.
Planning and verification
Decomposition, sufficiency judgement, bounded iteration and the verification pass, scored against the multi-hop set.
Routing and cost control
Classifier routing traffic between fast and agentic paths, with latency and cost per question type measured.
Handover
Evaluation sets, traces, routing thresholds and the runbook for tuning them as question patterns change.
What you receive
A system that answers the hard questions and is honest about the ones it cannot.
Question taxonomy
Real questions classified by type with the baseline score for each, showing where planning is justified.
Agentic retrieval pipeline
Planning, sub-question retrieval tools, bounded iteration and sufficiency judgement.
Verification layer
Claim-level checking against retrieved evidence, with unsupported claims removed or flagged.
Routing layer
Classifier and thresholds sending each question down the cheapest path that answers it.
Evaluation suite
Multi-hop and comparison cases with correct answers and acceptable evidence, wired into CI.
Cost and latency profile
Per question type, with the levers available if the mix shifts.
Is this the right engagement?
Worth being direct. Agentic RAG is the wrong spend in some situations, and those are listed rather than buried.
Good fit if
- Users ask comparison, multi-hop or conditional questions that current retrieval answers badly.
- A confident wrong answer carries real professional or regulatory consequence.
- Answers must show not just sources but how the conclusion was reached.
- Baseline retrieval is already decent and the remaining failures are structural rather than about recall.
- Several seconds of latency is acceptable for the hard questions.
Choose something else if
- Basic retrieval quality has never been measured. Start with RAG system development.
- Nearly all questions are simple lookups, where this adds cost and latency for nothing.
- Sub-second latency is a hard requirement on every request.
- The content itself is contradictory and unowned; planning cannot resolve that.
Frequently asked questions
Marked up with FAQPage schema so these answers can surface directly in search results and inside AI assistant responses.
What is agentic RAG?
It is retrieval augmented generation where the system decides how to retrieve rather than issuing one search: it decomposes the question, chooses sources, judges whether the evidence is sufficient, searches again if it is not, and verifies its answer against what it found before returning it.
When is agentic RAG worth the extra cost?
For multi-hop, comparison, aggregation, temporal and conditional questions, and anywhere a confident wrong answer is expensive. For simple lookups it is a waste, which is why we route traffic by question type rather than sending everything down the expensive path.
How much slower is it?
Several times, typically seconds rather than under a second, because it performs multiple retrieval rounds and a verification pass. We bound the number of rounds explicitly and report latency per question type so the trade is visible rather than discovered in production.
Does it eliminate hallucination?
No, and nobody should claim that. It reduces it substantially, mainly through the verification pass and by making 'not found in the available sources' an acceptable and frequent output. Residual risk remains and is documented rather than dismissed.
Do we need standard RAG first?
Yes. Agentic retrieval amplifies the quality of the underlying index in both directions: good recall makes plans effective, poor recall makes them expensive and futile. We measure and fix the foundation before adding a planning loop to it.
Often paired with this
Most clients combine two or three engagements from the AI Agents & Agentic Automation pillar. These are the ones that most often run immediately before or after.
AI Research and Analyst Agents
Research agents that work approved sources, cite every claim and separate finding from inference.
Read more →Custom AI Agent Development
An agent that completes work in your systems, with scoped tools, trajectory evaluation and a circuit breaker.
Read more →AgentOps: Monitoring, Evaluation and Guardrails
The operations layer for production agents: trajectory tracing, evaluation, spend ceilings and a tested kill switch.
Read more →Is this the right engagement?
Tell us what you are trying to build. If a different service fits better, or if you do not need us at all, we will say so.