LLM Security: Prompt Injection and Data Leakage
Designing language model systems on the assumption that the model will eventually be persuaded to do the wrong thing — so that when it is, the damage is bounded by architecture rather than by a filter someone hoped would hold.
Prompt injection has no complete solution. That is not a gap awaiting a vendor's fix — it follows from the fact that instructions and data arrive through the same channel, in the same form, with no reliable way to separate them. Security work therefore has to assume the model can be subverted and limit what follows.
LLM security covers the design and controls that keep a language model application safe when the model itself cannot be trusted to refuse: separating instructions from untrusted content, treating model output as untrusted input to everything downstream, constraining what tools can do, enforcing permissions at retrieval, and containing the consequences of a successful attack.
The principle everything else follows from
Treat the model as a component that can be persuaded, and design so that persuading it is not sufficient to cause harm:
- Model output is untrusted input. Anything downstream — a database, a shell, a browser, an API, a rendered page — must validate it exactly as it would validate input from the public internet.
- Permissions belong to the user, not the application. A service account with broad access turns any injection into access to everything that account can reach.
- Tools are the blast radius. What an attacker can achieve is bounded by what the tools can do, which is a design decision you control.
- Consequential actions need confirmation outside the model's control, so that a subverted model cannot authorise them on the user's behalf.
- Filters reduce frequency, not possibility. They are worth having and they are not a security boundary.
The specific risks and what actually helps
| Risk | How it happens | What helps |
|---|---|---|
| Direct injection | The user instructs the model to ignore its rules | Filtering, then architectural limits on impact |
| Indirect injection | Instructions hidden in content the model reads | Untrusted-content isolation; no direct action from it |
| Data leakage in retrieval | Permissions checked at display, not in the query | Permission filtering inside retrieval itself |
| System prompt extraction | The model is induced to reveal its instructions | Assume it is public; keep no secrets in it |
| Excessive agency | Tools can do more than the task requires | Least privilege per tool, per user, per session |
| Insecure output handling | Model output executed or rendered unchecked | Validate and escape like any untrusted input |
| Training and log leakage | Sensitive data captured in prompts, logs or fine-tuning | Redaction, retention limits, access control on traces |
| Denial of wallet | Attacker drives expensive requests or loops | Rate limits, spend ceilings, loop detection |
The system prompt is not a secret
Sufficiently determined extraction attempts generally succeed, so the correct posture is to assume the system prompt is public. Anything that must remain confidential — credentials, internal URLs, business rules that carry commercial value — belongs outside it, enforced by architecture rather than by instruction.
What we build
Trust boundaries drawn explicitly
Which content is trusted, which is not, and what each is allowed to influence. Content retrieved from documents, the web or user uploads is untrusted by definition, and it must not be able to trigger an action directly.
Permission-aware retrieval
Access control applied inside the retrieval query, so a user can never receive a passage from a document they cannot open. Post-retrieval filtering is a common shortcut and a reliable source of leaks. See vector database design.
Least privilege on tools
Each tool scoped to the minimum it needs, acting with the user's permissions rather than the application's, with destructive operations separated from routine ones and rate-limited independently.
Confirmation outside the model's reach
Consequential actions require a human confirmation the model cannot fabricate or bypass. This is the control that keeps a successful injection from becoming a successful transaction.
Output handling as though it were hostile
Validation, escaping and parameterisation wherever model output reaches a database, a shell, a browser or a rendered page. A great many AI security incidents are ordinary injection vulnerabilities with a model in the middle.
Detection, because prevention is incomplete
Monitoring for injection patterns, unusual tool sequences, anomalous data volumes and cost spikes, with alerts that reach someone. If prevention cannot be complete, detection is not optional. See AI observability.
How the engagement runs
Architecture is assessed before controls are added, because filters cannot fix a trust-boundary problem.
Architecture assessment
Trust boundaries, data flows, tool permissions and retrieval access model mapped as they actually are.
Threat modelling
Realistic attackers and objectives; controls prioritised by blast radius rather than by likelihood alone.
Implementation
Trust separation, permission-aware retrieval, least-privilege tools, confirmation steps and output handling.
Adversarial validation
Controls tested against real attempts rather than assumed to work; see red teaming.
Detection and handover
Monitoring and alerting implemented; runbook and incident process handed over.
What you receive
A system where a successful injection is contained by design rather than by hope.
Architecture assessment
Trust boundaries, data flows, tool permissions and retrieval access as they actually are.
Trust separation
Untrusted content isolated and unable to trigger action directly.
Permission-aware retrieval
Access control enforced inside the query, verified against real entitlements.
Tool permission model
Least privilege per tool, acting as the user, with destructive operations separated.
Output handling
Validation and escaping wherever model output reaches another system.
Detection and incident response
Monitoring for injection, anomalous tool use and cost spikes, with a rehearsed response.
Is this the right engagement?
Worth being direct. LLM Security: Prompt Injection and Data Leakage is the wrong spend in some situations, and those are listed rather than buried.
Good fit if
- A language model application handles sensitive data or takes actions.
- The system reads content from sources you do not control.
- An agent has tool access with real consequences.
- Security has raised concerns and nobody can answer them concretely.
- Guardrails exist but the architecture assumes the model will always refuse.
Choose something else if
- The requirement is testing rather than architecture. See AI red teaming.
- The application has no tools, no sensitive data and no untrusted content.
- You want a filter that guarantees prompt injection is impossible, which does not exist.
- The architecture cannot be changed, which limits the work to detection alone.
Frequently asked questions
Marked up with FAQPage schema so these answers can surface directly in search results and inside AI assistant responses.
Can prompt injection be prevented?
Not completely, and any product claiming otherwise is overselling. Instructions and data arrive through the same channel in the same form, with no reliable way to separate them. The workable posture is to assume the model can be subverted and design so that subverting it is not sufficient to cause harm — limited tool permissions, permission-aware retrieval, confirmation outside the model's control.
What is indirect prompt injection?
Instructions hidden in content the system reads rather than typed by a user — a document, a web page, an email, a calendar entry. It is the most serious structural weakness in retrieval and agent systems because the attacker never touches your interface, and it is why untrusted content must not be able to trigger an action directly.
How do we stop the model leaking data between users?
By enforcing permissions inside the retrieval query rather than filtering after it, and by having tools act with the requesting user's permissions rather than a broad service account. Systems that check access at display time can still leak through a summary, and a service account with wide access turns any injection into access to everything it can reach.
Is our system prompt confidential?
Treat it as public. Determined extraction attempts generally succeed, so anything that must stay confidential — credentials, internal URLs, commercially valuable rules — belongs outside the prompt and enforced by architecture. Protecting it by instruction is not a control.
Do guardrail products solve this?
They reduce how often attacks succeed, which is worth having. They are not a security boundary, because they can be bypassed and they operate on the same ambiguous channel the attack uses. They belong in a design where the architecture already limits what a successful attack can achieve.
Often paired with this
Most clients combine two or three engagements from the AI Governance, Security and Compliance pillar. These are the ones that most often run immediately before or after.
AI Red Teaming and Adversarial Testing
Adversarial testing against your threat model, with reproducible findings and regression tests you keep.
Read more →Data Privacy for AI: GDPR, HIPAA and DPDP
Lawful basis, DPIAs, transfers, and the hard questions — deletion rights and retention inside trained models.
Read more →AI Audit and Third-Party Assurance
Independent assessment against a stated standard, with evidence tested rather than taken on assertion.
Read more →Is this the right engagement?
Tell us what you are trying to build. If a different service fits better, or if you do not need us at all, we will say so.