Why this is not ordinary penetration testing

A conventional penetration test hunts for flaws in code and configuration: a missed input check, a broken access control, a credential left in a log. Those flaws still exist in a system that happens to contain a language model, and they still need testing. What is new is that the model reads instructions and data through the same channel. The prompt that tells it what to do and the document it is asked to summarise arrive as the same kind of text, and the model has no reliable way to tell one from the other. Red-teaming an AI system is the discipline of exploiting that confusion on purpose, to see what the model can be talked into and, more importantly, what it can reach once it has been.

Prompt injection, direct and indirect

Prompt injection sits at the top of the OWASP Top 10 for LLM Applications, and for good reason. It is the failure mode that makes the others reachable. There are two shapes worth separating.

• Direct injection is the user talking to the model and persuading it to abandon its intended instructions. The familiar jailbreak, coaxing a model past its own guardrails with a role-play or a contrived hypothetical, is the visible form of this.
• Indirect injection is more dangerous because the attacker never has to be present. The payload is hidden in content the model reads later: a web page it browses, an email it triages, a support ticket it summarises, a document dropped into a shared folder. The instruction arrives disguised as data, and the user who asked an innocent question never sees it.

Indirect injection is where a system that felt safe in a demonstration quietly stops being safe in production, because production is full of text the organisation did not write.

Tricked, or trusted with too much

The most important distinction in AI red-teaming is between a model being tricked and a model having excessive agency. They are often conflated, and they should not be. A model that only produces text can be talked into saying something false, offensive or off-policy. That is a content problem, and it is usually recoverable. A model connected to tools, a mailbox it can send from, a database it can query, a shell it can run, can be talked into doing something. That is an incident.

OWASP names this excessive agency, and it is where prompt injection turns from embarrassing to expensive. The question a red team keeps returning to is not whether the model can be persuaded, because it usually can, but what a persuaded model is permitted to reach. A system that grants the model broad write access, standing credentials and no confirmation step has decided that a single injected instruction is enough to act on the world. Least privilege, scoped permissions and a human check on consequential actions are not model problems at all; they are the architecture around the model, and that is precisely where good red-teaming applies its pressure.

When we red-team a language model, the failures rarely look like classic bugs. They look like a system doing exactly what it was told by someone it should never have trusted. The question that matters is not whether the model can be talked into something, but what it is allowed to reach once it has been.

— Zia Bharwani, CEO of Red Team Partners, an offensive-security firm that runs red team and penetration-testing engagements internationally

Red Team Partners is a sponsor of the Cyber Security Awards. That observation is one practitioner’s framing, offered here alongside the published frameworks rather than in place of them; the substance of this piece rests on the named standards, not on any single firm’s account of them.

Data leakage

Language models leak in ways that are easy to overlook. Sensitive material can surface because it sat in the model’s context: a system prompt containing secrets, retrieved documents a user was never entitled to see, or the contents of one customer’s session bleeding into another through shared state. A red team tests whether the model can be induced to repeat its own instructions, to disclose retrieved records outside the requester’s permissions, or to return training data it should never emit. The lesson from the OWASP list is consistent. Anything placed in a model’s reach should be treated as potentially reachable by whoever is talking to it, and access control belongs at the retrieval and tool layer, not in a politely worded instruction asking the model to keep a secret.

Tool and agent misuse

Agentic systems, where a model plans steps and calls tools to carry them out, widen the target considerably. Each tool is a new capability an attacker might borrow through the model. Indirect injection in a browsed page can drive an agent to exfiltrate data through a tool it was given for a legitimate purpose. Chained calls can escalate: read a file, find a credential, use it. The red team maps the full set of actions the agent can take, then asks what an untrusted instruction could accomplish by combining them. This is the point where the older discipline reasserts itself, because the containment that matters, sandboxing, output validation, allow-lists and revocable scoped credentials, is ordinary security engineering applied to a new kind of caller.

Working from named frameworks

Adversarial testing of AI is easy to do theatrically and hard to do usefully. The difference is structure. The OWASP Top 10 for LLM Applications gives a concrete taxonomy of what to test and a shared vocabulary for reporting it. The NIST AI Risk Management Framework wraps governance and measurement around that work, so findings feed decisions about whether a system should ship. In Europe, ENISA’s threat landscape work and the NCSC’s guidance on the security of AI and machine-learning systems set out how these risks appear in operational settings and what proportionate mitigation looks like. Anchoring a red-team engagement to these named sources keeps it honest. It measures genuine exposure against a public standard, rather than collecting a scrapbook of jailbreaks that impress in a slide and prove little about whether the system is safe to run.

Sources

• OWASP Top 10 for LLM Applications — the reference list of large-language-model risks.
• NIST AI Risk Management Framework — the US framework for governing AI risk.
• ENISA — the EU Agency for Cybersecurity, guidance on AI security.
• NCSC — UK National Cyber Security Centre guidance on securing AI systems.
• Red Team Partners — practitioner commentary (Zia Bharwani).