A security checklist for AI applications
Prompt injection, data leakage and over-permissioned tools — the AI-specific risks that standard appsec review misses.
AI features introduce a genuinely new attack surface. Your existing security review will not catch most of it, because the vulnerability is in what the system is persuaded to do rather than in what it is exploited to run.
Red-team before launch
A day from someone who did not build it. It consistently finds something.
- 1
Try to extract the system prompt
If it leaks, assume everything in it is public.
- 2
Try to reach another user's data
Through retrieval, through tools, through both.
- 3
Plant instructions in content it reads
A document, a web page, a support ticket.
- 4
Try to trigger an action it should refuse
Directly, then by persuasion, then indirectly.
Treat all retrieved content as hostile
If your system reads web pages, uploaded documents or user-generated content, that content can contain instructions. A model cannot reliably distinguish your instructions from instructions embedded in the data it was given. Assume it will follow them and design so that doing so is harmless.
Authorise at the tool, never in the prompt
'Only show data belonging to this user' in a system prompt is not access control. Enforce scoping in the tool implementation against the authenticated session, exactly as you would for any endpoint.
Retrieval must respect permissions
A common and serious failure: one vector index containing documents from every permission level, searched without filtering. The model dutifully surfaces a document the asker was never allowed to see. Filter at query time by the requester's access, and test that you do.
Pre-launch security review for an AI feature
Must hold
- Every tool enforces authorisation independently of the prompt
- Retrieval filters by the requesting user's permissions
- Model output escaped before rendering — it can contain markup
- Rate limits per user and per agent run
Common failures
- "Only show this user's data" written in the system prompt
- One index holding every permission level, searched unfiltered
- Irreversible actions the model can trigger without confirmation
- Logs capturing raw prompts, secrets and personal data included
The checklist
What we run through before an AI feature ships:
- Untrusted content is delimited and the system prompt says to treat it as data
- Every tool enforces authorisation independently of the prompt
- Retrieval filters by the requesting user's permissions
- Output is escaped before rendering — model output can contain markup
- Irreversible actions require explicit confirmation outside the model's control
- Rate limits per user and per agent run, to bound cost and blast radius
- Logs redact secrets and personal data at capture
- A documented path to disable the feature quickly
Red-team before launch
Have someone who did not build it spend a day trying to make it misbehave — extract its prompt, reach another user's data, trigger an action it should refuse. It is a cheap exercise that consistently finds something.