# Researcher's experiment finds 25 program verifiers matched a single verifier's error rate

An open-source research preview reports that a 25-verifier panel of test-oracle programs had a measured effective size of 1.00, and that a common correlation correction overstated the panel's value by 1.66x. The author states no AI review panel was measured.

Canonical URL: https://freelancenews.online/news/study-shows-25-code-reviewers-provide-no-more-verification-than-one
Published: 2026-09-20T22:07:18.287Z
Updated: 2026-09-24T23:21:36.899Z
Source published: 2026-09-20T21:48:15.000Z
Event date: Not established
Review status: source-reviewed
Review method: Automated comparison against retrieved source text; not independent fact-checking.

Editorial note: Updated after automated comparison with the full extracted source text. Attribution, wording and practical context were re-evaluated; this is not an independent fact check.

## Report

An experiment record has been published by an open-source research project named IDKMesh, and it reports that a panel made up of 25 program-based verifiers did no better than a single member of that panel. On dev.to, the author describes the project as an Apache-2.0 research preview built for Python 3.11+, asking how coordination on uncertain goals can happen among humans, AI agents, tools and heterogeneous compute, and how proposals can be turned into verified work. Because the post is a community publication, the figures below are results the author reported, not findings verified independently.

Stated plainly is the central claim: reviewer count does not equal independent evidence count. Under the project's coordination model, generation is treated as cheap while verification is the scarce resource. A bounded Work Unit, which must declare its security bounds, is how work arrives; replaceable workers attempt it; candidates are judged by a separate evaluation plan owned by the verifier; and reproducible evidence then results, including result manifests, verification results and provenance digests. Rather than a green impression, an explicit decision gated on that evidence is what integration becomes.

The post separates two kinds of evidence. The first is a bundled demonstration with an explicit "evidence_class": "synthetic" label. In that example, five invented verifiers with a mean accuracy of 0.8 and a mean pairwise error correlation of 0.0916 produce an effective_votes figure of 1.6944 against a heuristic_n_eff of 3.6588, and two of three seeded known-bad probes were accepted by the panel. The author is explicit that those reviewers are invented and that 1.69 demonstrates arithmetic rather than measuring anything real.

The observed side is Experiment E017. On that panel, every verifier is a program rather than a model: it acts as a partial test oracle, accepting a candidate only if, across all inputs, that candidate agrees with a reference implementation. Those inputs are drawn from a single named region within a problem's input domain — one of tiny, small, large, extreme or duplicate. Five regions times five seeds yields twenty-five verifiers, which are run across a corpus of 72 candidates; hidden tests are executed to produce the ground truth for that corpus.

The author reports that each verifier's Youden J was significantly positive after Bonferroni correction, with a mean accuracy of 0.7956, so the panel worked individually. Error correlation between verifiers was the problem. Pairs in the same declared region had a mean rho of +0.8924; pairs in different declared regions, described as declared independent, had a mean rho of +0.5263; across all pairs the mean rho was +0.5873. The author's reading is that verifiers sharing no declared attribute still shared 53% of their errors, and that a metadata group boundary is not an independence boundary.

The headline result follows: a 25-verifier majority error of 0.2083 against a single verifier's 0.2044, and a measured effective size of 1.00 out of 25 nominal. The standard correlation correction, N/(1+(N-1)rho), produced a heuristic figure of 1.66 — an overstatement of 1.66x relative to the measured value. The author notes the whole measurement ran in about five seconds on one laptop.

The post also reports a separate experiment, E015, testing that heuristic across a parameter grid. According to the author, the formula was exact at rho=0 and rho=1 but wrong in between, and the sign of its error was not fixed. It was conservative where verifiers are weak, which the author calls the cheap place to be conservative, and optimistic where verifiers are accurate and share modest dependence. At p=0.90 and rho=0.125, the author reports the measured effective size saturating at 4.60 against a heuristic asymptote of 8.00, and a balanced error of 0.0125 for the real panel against 0.000891 for an independent nine-verifier panel — fourteen times worse.

Unusually direct is the scope statement, and it matters for how the result should be read. No AI review panel has been measured in the repository, the author writes; E017's verifiers are programs; and an open question — the one the author most wants data on — is whether the same correlation structure appears among LLM reviewers. Described as a research preview rather than production software, the project is not on PyPI, so installation is from a clone.

The practical tool on offer is gate-audit: verdicts that a team has already gathered are taken in, and what the panel is actually worth is reported out. Labelled synthetic serves as the bundled example. As for the experiment the author most wants next, it is framed as aiming gate-audit at verdict logs produced by a real review gate — whether the reviewers are human, LLM judges or CI checks — particularly when the result disagrees with the post.

For working developers, the immediate implication is about how review capacity is budgeted rather than about any specific tool. If verifiers that share no declared attribute still share a majority of their errors, then adding reviewers to a gate may add cost and latency without adding independent evidence, and the metadata used to justify treating reviewers as independent may not describe the actual error structure. The author's own framing is that verification, not generation, is the scarce resource — which suggests auditing an existing gate's verdict history before expanding it.

That implication is editorial interpretation, not a measured result for human or LLM review. The evidence supplied covers program verifiers on a 72-candidate corpus with hidden-test ground truth, and the author explicitly declines to extend it. A team reading this should treat the effective-size arithmetic as a hypothesis to test against its own logs, not as a finding about code review in general.

There are further limitations. The post is a community publication by the project's author, so the experiment records are self-reported even though the author says each number traces to a committed record that can be re-run. The synthetic demonstration and the observed experiment are clearly separated by the author, and the synthetic figures should not be quoted as measurements. The repository is not on PyPI, installation requires a clone and a virtual environment, and the post gives no pricing because there is none to give for an Apache-2.0 research preview.

The most useful conclusion is procedural. The author's own request is for third-party verdict logs, and the honest state of the evidence is that a program-verifier panel collapsed to an effective size of one while a widely used correction overstated it. Until someone runs the same audit on human or LLM review data, the transferable lesson is to measure a gate's effective independence rather than assume it from reviewer count or grouping metadata.

## Key points

- Experiment E017 reports a 25-verifier program panel with a measured effective size of 1.00, a majority error of 0.2083 against a single verifier's 0.2044.
- Verifiers in different declared regions still shared a mean error correlation of +0.5263, which the author reads as a metadata group boundary not being an independence boundary.
- The author states no AI review panel was measured in the repository and that whether LLM reviewers show the same correlation structure is an open question.
- The bundled gate-audit example is labelled synthetic, and the author asks others to run the audit on real human, LLM or CI verdict logs.

## Practical implications — editorial interpretation

Editorial interpretation: before adding reviewers to a gate, teams could audit existing verdict logs for effective independence, since the reported result suggests extra verifiers may add cost and latency without adding evidence. This is not a measured finding for human or LLM review.

## Limitations and unknowns

Community post by the project's author; results are self-reported. The observed experiment used program verifiers, not human reviewers or LLM judges, and the author explicitly declines to generalize. The synthetic demonstration must not be read as a measurement. The project is a research preview, not on PyPI, and installation is from a clone.

## Sources

- [1] dev.to: A 25-verifier panel measured an effective size of 1.00
  https://dev.to/mskazemi/a-25-verifier-panel-measured-an-effective-size-of-100-lgg
  Retrieved: 2026-09-24T23:21:21.315Z

## Claim references

- The author reports that a 25-verifier program panel had a measured effective size of 1.00 and a majority error of 0.2083 versus 0.2044 for a single verifier. [source 1]
- The author states that verifiers in different declared regions still shared a mean error correlation of +0.5263. [source 1]
- The author states that no AI review panel has been measured in the repository and that E017's verifiers are programs. [source 1]
- The bundled gate-audit example is labelled synthetic, and the author says those five reviewers are invented. [source 1]
- The author reports that the correlation heuristic was exact at rho=0 and rho=1 but wrong in between, with a non-fixed error sign. [source 1]
- The author asks others to run gate-audit on verdict logs from real review gates, including human reviewers, LLM judges or CI checks. [source 1]