Strix and PentAGI on a Real Pentest: 296 Findings and Zero Exploitable

⚡ The Gist in 5 Seconds

• Strix and PentAGI, two open-source LLM agents for pentesting, were run against a client's live external perimeter: about 200 active hosts, the GLM 5.3 model.
• PentAGI produced 296 vulnerabilities, including 42 CRITICAL and 50 HIGH — and not one of those 92 findings turned out to be actually exploitable.
• Strix delivered 15 findings with almost no false positives, but inflated severity: after manual verification, 1 MEDIUM, 2 LOW and 12 INFO remained.

🔍 What Happened

The author of a Habr article under the nickname itsstaace compared two open-source pentest agents, Strix and PentAGI, on a real project rather than benchmarks. The scope was a client's external perimeter, about 200 active hosts. The model was GLM 5.3. Both tools run in an isolated Docker sandbox and deploy without any ritual dancing.

Their mechanics are similar: the agent sets its own tasks, runs tools, writes scripts, and assembles a PoC and a report. Then the differences begin. Strix requires a specific target — --target, -t or --target-list; a free-form --instruction does not replace a mandatory target. PentAGI takes the scope straight from the prompt, which is why the author managed to send it on reconnaissance. Strix, however, is stronger on web applications and code, and requires confirming a finding with a PoC.

The numbers diverged dramatically. Strix burned 145.1 million tokens, PentAGI — 31.4 million. Strix's consumption balloons due to repeatedly passing context to dozens of subagents. PentAGI's aggressive context compression saves tokens but risks losing details and looping on large scopes.

Reconnaissance is a separate story. Neither agent can build an asset registry from an abstract scope like an organization name or a network range. Their recon is a byproduct of the run, not an inventory. Full perimeter discovery remains a separate task.

🧠 Our Take

Why this happened now is clear. A subscription to a model that "knows the entire internet" costs about $20 a month, and LLM agents for pentesting are discussed in every chat. The temptation is simple: let the machine poke at the perimeter for the fourth hour while the human opens a beer. The author tested this scenario on live infrastructure and got a cold shower.

What this changes. PentAGI's 296 findings are not a result but a bill for the work. Manually triaging 92 "critical" findings costs more than running a proper scanner. Our assessment: the main damage here is reputational. An agent that drags other people's CVEs and template-based evidence into a report undermines trust in the entire class of tools more than an honest miss does.

Who wins. Commercial scanners: the real 2 CRITICAL findings, a HIGH with an exposed OAuth secret, and a chain to code execution on the same perimeter were found precisely by a paid tool. People skilled in triage also win — their work has become only more needed. Those who lose are the ones who believe in full autonomy: PentAGI recognized that part of the given range belonged to another organization and still kept scanning against the ROE.

⚠️ What We Don't Know

One run, one scope, one model, one author. We don't see the settings, limits, or temperature, nor how much Strix's 145.1 million tokens translated into in dollars. There is no data on how the agents would behave on an internal network or on targets requiring authentication. There is no independent replication either: the figures 296 and 15 come from a single researcher's report and cannot yet be verified externally. Skepticism is more appropriate here than enthusiasm.

Forecast: within a quarter, open-source pentest agents will get built-in ROE enforcement — otherwise they simply won't be allowed onto corporate perimeters.

Disclaimer: one source — one author's analysis on one scope, no independent replications.

Source: Habr