AI penetration testing: what autonomous testing can and cannot do

Earlier automation in security testing followed fixed rules: a scanner sends a known set of requests and matches responses against signatures. AI-driven testing adds an agent that can read what it sees, decide what to try next and adapt, closer to how a human tester works through a target. Most systems combine a language model for reasoning with conventional security tools for the actual requests, plus memory of what has been tried.

The label is used loosely. Some products are scanners with an AI-written summary; others run multi-step attack sequences with little human input. When evaluating one, ask what the agent decides on its own and what is still a fixed script.

• Coverage at scale. An agent can enumerate thousands of endpoints, parameters and hosts without fatigue, and test each one consistently.
• Known weakness classes. Common issues such as injection, cross-site scripting, security misconfiguration and exposed secrets are well represented in public knowledge, and agents test for them reliably.
• Re-testing on change. Running the same checks after every deploy or infrastructure change is tedious for people and cheap for agents. This is where AI makes continuous penetration testing practical.
• Drafting. Agents can draft clear reproduction notes and suggested fixes, saving human time on write-ups.

• Business logic. Knowing that a refund should never exceed the original charge, or that one tenant must never see another's invoices, requires understanding what the application is for. Broken access control and IDOR flaws often hide here.
• Judging impact. An agent may report a technically valid finding with no real-world consequence, or underrate a small issue that becomes serious when chained with another.
• Novel techniques. Models draw on what they have seen. Truly new attack ideas still mostly come from people.
• Confidence without correctness. Language models can describe an exploit that did not actually work, or misread a response as success. This is the main source of AI-specific false positives.
• Context outside the system. Which data matters most, which outage is unacceptable and which risk the business has already accepted are things the agent has to be told.

A finding is only useful if it is true and if someone acts on it. Unvalidated output fails on both counts: false positives waste engineering time, and once a team has chased a few phantom issues, it starts ignoring real ones. Having a skilled person reproduce each attack path before it is reported keeps the signal trustworthy and lets them judge severity in business terms.

An autonomous agent that can attack systems needs tighter guardrails than a human tester, because it acts faster and does not pause to wonder whether something looks fragile. Controls to expect:

If AI systems are part of your own product, they are also a target. The OWASP Top 10 for Large Language Model Applications lists risks such as prompt injection that a test of an AI feature should cover.

The strongest setups give each side the work it is best at. Agents handle breadth and repetition: discovery, systematic testing of known classes and re-running attacks after every change. Humans handle judgment: confirming that a finding is real, chaining weaknesses into attack paths, probing business logic and deciding what matters most to fix. Human findings then feed back into what the agents check from then on.

The same split applies to defense. An agent can draft a fix or hardening change, but an engineer who knows the system reviews and merges it. AI speeds up the loop without taking people out of the decisions that carry risk.

1. Which steps does the agent take on its own, and which require a human decision?
2. Who reproduces a finding before it reaches you, and what evidence comes with it?
3. How is scope enforced in code, not just in the contract?
4. Which actions is the agent blocked from taking, and how is that tested?
5. Where do prompts, responses and any data the agent sees go, and how long are they kept?
6. How are fixes retested, and who decides a finding is closed?

Vague answers to the first two questions are the clearest warning sign. A provider that cannot say who validates findings is likely passing raw model output to you, and that cost lands on your engineers.

AI-driven testing tends to fit teams with a large or fast-changing attack surface, frequent releases, and the capacity to act on a steady flow of validated findings. It fits less well as the only test of a complex, logic-heavy application, or where an auditor expects a named human-led test. Many organizations use both: an AI-assisted ongoing program plus periodic human-led deep dives. Brexa's continuous testing service follows that model, with a human operator reproducing every finding before it is reported.