What a penetration test actually proves (and what it doesn't)

A scan, a test, and a red team are not the same thing

Three activities get filed under the same heading, and treating them as interchangeable is how organisations end up disappointed by work that did exactly what it said it would.

An automated vulnerability scan runs a tool against a system and compares it to a database of known weaknesses. It is fast, cheap, and useful for coverage, and it is right to run one often. But it reports what might be wrong. A scanner cannot tell whether a flagged issue is genuinely reachable, whether a control in front of it makes it harmless, or whether two low-severity findings combine into something serious. It produces possibilities, including false positives it has no way to confirm.

A penetration test is led by a person. Within an agreed scope, the tester attempts to exploit the weaknesses a scan might only suspect, confirms which are real, and chains them together to show what an attacker could actually accomplish. The output is not a list of maybes but a set of demonstrated paths, each with evidence and a judgement about impact. The Penetration Testing Execution Standard (PTES) describes this as a structured sequence, from scoping and intelligence gathering through exploitation to reporting, precisely so the work is repeatable rather than improvised.

A red team engagement is different again. It is goal-led, modelling a specific adversary, and it tests whether an organisation would detect and respond to an intrusion across people, process and technology, often while the defenders are unaware it is under way. A pen test asks how exploitable a system is. A red team asks whether anyone would notice.

What a penetration test proves

Its core value is exploitability. A good test moves a finding from theory to fact: this weakness is real, it can be reached from here, and reaching it lets an attacker do this specific thing. That distinction matters because most systems carry far more theoretical issues than an organisation can fix at once, and a test helps separate the ones that genuinely open a door from the ones that do not.

It also proves reach and chaining. The most valuable findings are rarely a single dramatic flaw. They are sequences: a low-privilege foothold that leads to a misconfiguration, which exposes a credential, which reaches data that should have been out of bounds. OWASP's testing guidance is built around exactly this attacker's-eye view, examining how the parts of an application behave together rather than in isolation. A person following that logic can surface impact that no scanner, checking each item alone, would ever connect.

Finally, a test proves something about accountability. A CREST-accredited assessment, for example, carries a named, qualified tester and a report someone stands behind. That is why the deliverable is evidence a board or an auditor can weigh, not just a tool's output.

What it cannot prove

A penetration test has two structural limits, and understanding them is what separates a useful test from a false sense of safety.

• It is point-in-time. The test describes the system as it stood while the test ran. The next deployment, dependency update, or configuration change can introduce a weakness the report never had the chance to see. The NCSC is explicit that a test is a snapshot, and that its value decays as the system moves on beneath it.
• It is scope-bound. A test proves things about what was in scope and nothing about what was left out. If an application was assessed but the identity provider behind it was not, the report is silent on the identity provider. Scope is a commercial and practical necessity, but it is also the single most common reason a clean report gives more comfort than it should.

Neither limit is a flaw in the method. They are simply the boundaries of what the evidence covers, and a mature security programme reads a report with both firmly in mind: what day was this, and what exactly was in the frame.

The standards that make the work legible

Penetration testing has matured from an artisanal craft into something with published method, and several named sources define what good looks like. PTES sets out the phases a thorough engagement should follow. OWASP maintains widely used testing guidance for web applications, giving testers and clients a shared reference for what was and was not examined. CREST accredits the companies and individuals who carry the work out, so a buyer can check competence rather than take it on trust. And the UK's NCSC publishes guidance on commissioning tests well, including the reminder that a test is one input into managing risk, not the whole of it.

Used together, these give a client a way to judge a test by its method and its scope rather than by the length of its findings list. A short report against a well-chosen scope, run to a recognised standard, is worth more than a long one produced without either.

One practitioner's framing

Against that backdrop of named standards, it is worth hearing how a working tester describes the distinction, as one voice among the sources above rather than the authority on the matter.

A scanner tells you a door might be unlocked. A penetration test walks through it and shows you what is on the other side. The difference that matters is a human deciding what to chain together, and putting their name to every finding.

— Zia Bharwani, CEO of Red Team Partners, an offensive-security firm that runs red team and penetration-testing engagements internationally

Red Team Partners is a sponsor of the Cyber Security Awards.

How to read one well

A report is only as useful as the questions brought to it. Before commissioning, agree a scope that reflects real exposure rather than the parts that are easiest to test, and write down what is deliberately excluded. When the report arrives, read the scope statement and the test dates first, then treat each finding as a claim about a specific path, not a verdict on the whole estate. Retest the fixes rather than assuming a remediation worked. And schedule the next test around change, not only the calendar, since it is change that quietly undoes the assurance the last one gave.

Read this way, a penetration test is one of the more honest instruments in security: it tells you exactly what a capable person could reach, on a defined system, on a defined day, and it is candid about the fact that it cannot tell you more than that.

Sources

• CREST — accreditation body for penetration testing and technical security services.
• OWASP — open web application security testing standards and methodology.
• PTES — the Penetration Testing Execution Standard.
• NCSC — UK National Cyber Security Centre guidance on assurance and testing.
• Red Team Partners — practitioner commentary (Zia Bharwani).