Developer Benchmarks His Own Open-Source Scanner and Finds Two of Four Checks Produce False Positives

An open-source security scanner's author has published a measured account of how often his own tool's findings hold up, and the answer depends heavily on what kind of site it is pointed at. In a post on dev.to, Guangyi Zhao describes running Web-Vuln-Scanner — a project he maintains at github.com/Ezra-Zhao/Web-Vuln-Scanner — against six applications he wrote himself, all served on the 127.0.0.1 loopback address. The headline result, as he frames it, is that two of the tool's four checks produced false positives because neither compares a response against a pre-attack baseline.

The scope of the claim matters as much as the finding. The post is a community publication, not a peer-reviewed study, and the author states plainly that the scanner is at scaffold v0.1. He also states what the tool does not do: stored XSS, blind SQL injection, CSRF and authenticated scanning are listed as TODOs in the source rather than claimed features. The codebase, including the demo app and tests, totals 846 lines.

The pipeline is described as four serial stages: a scope module that validates authorization, a crawler that collects pages, forms and parameterized URLs on the same host, four independent checks, and a report stage that emits console and JSON output. Checks share no state, and the command-line wrapper isolates each one so a single exception cannot abort a scan. The crawler is bounded to same-host links, 20 pages by default, and records only responses served as text/html.

For every check, the decision rule is stated explicitly, and the author contends that the conclusions follow from those rules. An eight-character random token is generated by the reflected XSS check, a payload containing that token is injected into each URL parameter and form field, and the response must contain the token and the entire payload verbatim. Four payloads are sent by the SQL injection check, whose responses are matched against 21 regular expressions spanning six groups of database error signatures, generic phrases like "syntax error" and "database error" among them; no baseline request is ever sent. One response, the base page, is what the headers check examines. Fourteen fixed paths are requested by the sensitive-files check, which flags any HTTP 200 with a non-empty body and does not inspect the content.

The six targets were built to isolate distinct site shapes: the bundled intentionally vulnerable demo app, a carefully written hardened app, a single-page-app shell returning 200 for unknown paths, an article site whose prose discusses database errors, an app with only a POST form that is genuinely injectable, and an API app reflecting input into JSON. The author says the repository ran unmodified, the demo scan was repeated three times, the authorization guard received five dedicated probes, and scripts plus raw JSON were archived so numbers can be re-run.

On the demo app, the scan reported 13 findings across 31 requests, with per-check times between 0.04 and 0.19 seconds and a full local run finishing in under a second. Three repeated runs produced identical results — 13 findings each time, with the same severity distribution of five high, two medium, three low and three info — which the author presents as evidence the pipeline is deterministic. The bundled test suite of 11 tests passed under pytest in 1.61 seconds.

A single demo finding is examined in detail. The SQL injection check targets a parameter on which the XSS check also fired: the payload was concatenated into a SQL statement, an error occurred in the database, and the exception was echoed unescaped inside a pre element on the error page. A true positive is how the author classifies this, while noting that the reflection came through verbose error disclosure, not through normal page rendering, and that the evidence string is supplied by the report without its context — a reporting debt, as he calls it.

The hardened app, written with output escaping, parameterized queries, all five security headers and real 404s, produced zero vulnerability-class findings. Its single finding was an info-level note that the Server header disclosed the Werkzeug version. The author argues this originates at the development server layer rather than in application code, and that readers should judge info items by which layer produced them. He treats the clean result as meaningful: false positives, he writes, are not this pipeline's default state.

They appear on specific shapes. On the single-page-app target, which returns 200 for every unknown path and had nothing planted, the scan reported 15 findings — 14 from sensitive-files, meaning the entire probe table hit, including .env, db.sqlite and .git/HEAD at high severity. The author calls all 14 false positives, a 100 percent false-positive rate for that check, and notes that 14 of 15 findings, or 93 percent of the report, is noise. The cause, he says, is written into the rule: status 200 plus a non-empty body, with content never examined.

The article site produced a different failure with the same root. Its prose legitimately contains the phrases "database error" and "SQL syntax," and its topic parameter feeds no query. The scan still reported a high-severity possible SQL injection, because the first payload's response matched a generic signature that was already present before any attack. The author's diagnosis is that the check asks only whether the thing it looks for is in the response, never whether it was there beforehand.

As boundaries rather than bugs, two coverage gaps are reported. A real, exploitable string-concatenated SQL injection was present in the POST-only target, yet zero findings were reported by the SQL injection check, which tests only GET forms and URL parameters; the report never stated that the form went untested. In addition, that POST form was tested by the XSS check, which confirmed a reflection via the same error-page echo path, but no check was responsible for matching the database error text present in the same traffic. The price of hermetic check separation is how the author describes this: conclusions are answered for by each check individually, and coverage completeness is answered for by nobody.

Entirely outside the crawler lay the JSON endpoint target. A link on the home page pointed to /api/search?q=, which reflects input verbatim into a JSON response; a manual probe confirmed that the token and the full payload appeared in the body. The link was fetched by the crawler, a non-HTML content type was seen, and it was skipped, so as input it was never seen by either injection check. Here the author is careful: in an HTML context, verbatim reflection in an application/json response is not XSS, and as a page it will not be executed by modern browsers, so calling it a missed XSS would overstate the case. Narrower is the supported conclusion — only the HTML portion of an application is seen by the pipeline, and how large that visible portion was is never reported.

The authorization guard passed all five probes: no default target, no scan without confirmation, and an allowlist that actually narrows scope when supplied. The author notes the guard is enforced in code rather than advised in documentation, which he says is not a given among learning-grade scanners.

Ranking the checks by decision method, the author places XSS first for requiring a random token plus the complete payload, which excludes escaped reflection by construction; its weakness is that it cannot express context. Sensitive-path probing ranks last for relying on status code alone. SQL injection sits between, with reasonable payloads undermined by the missing baseline. Headers is called the most deterministic check, with the caveat about the Server header's origin.

The proposed fix order puts baselines ahead of breadth. First come a baseline request for SQL injection and soft-404 sampling for the sensitive-path check; according to the author, that handles both major false-positive sources at once. Second comes content sniffing — an .env hit should contain key-value lines, and a .git/HEAD hit should begin with "ref:" — ranked lower because baseline sampling benefits all 14 probes, while sniffing rules must be written per path. Third comes a not-covered section listing untested POST forms, non-HTML endpoints and pages beyond the crawl cap. Only fourth does he place new checks such as POST SQLi, stored XSS and blind SQLi; his argument is that widening coverage before baselines land manufactures false positives at larger scale.

The underlying judgment is stated as a general principle: a reliable verdict needs exclusive evidence, and false-positive control means attaching a refutation procedure to every verdict class — first proving the signal is absent without the attack, then reporting that it appeared after. Baseline requests, soft-404 samples and random tokens are described as three implementations of that same idea.

For developers and freelancers who run scanners or read their output, the practical takeaway is editorial interpretation rather than a product recommendation: a clean report and a noisy one can come from the same tool, and the difference here tracked site shape rather than code quality. The author's own advice is to treat sensitive-path output on SPA-shaped sites as untrusted as a block, and to read a zero as silence about coverage rather than proof of safety.

The limitations are stated by the author and should be carried forward. All six targets are minimal local apps isolating one shape each, so mixed real-world sites were not measured. The hardened app was written against the checks' own list, which he calls a studying-to-the-test bias. Everything ran over loopback, so timings are local orders of magnitude only, and concurrency and timeout behavior went unmeasured despite a uniform 10-second per-request timeout in the source. The XSS-via-error-page path was not verified at browser level. All targets ran on the Flask/Werkzeug stack, limiting generalization. Open questions include what similarity threshold should define a soft-404 baseline, whether baseline filtering would suppress true positives on sites that echo the same error stack either way, and at what granularity a not-covered list should be reported.