Detector changes should be tested against both synthetic controls and real repository snapshots. The September 2026 field assessment found false positives and unsupported coverage that prior passing tests did not reveal. Historical results below describe their original samples; they are not a current zero-false-positive guarantee. This page documents the method and the measured results, in the spirit of an honest benchmarks page: what we measured, what we found, and what we do not claim.
The required evaluation process is:
- Build the detector from a published method where one exists (research papers, field-tested heuristics), deterministic and zero-LLM.
- Run it read-only against a corpus of real production repositories — no writes, no config, first-run conditions (the exact scenario a new user hits).
- Measure signal vs noise by hand: every finding is classified as a true positive, a false positive, or inert (pattern absent in that repo).
- Keep, cut, or tune. Detectors that flood are given precision levers (subject binding, caps, scoping, evidence gates) and re-measured; detectors that add no value are removed, not shipped default-off.
- Dogfood. DocGuard guards its own repository in CI. A detector that false-positives on DocGuard itself gets fixed before release (see the examples below).
The R5 candidate passed 1,733 tests on each supported Node release (18, 20, 22, and 24). The extracted npm tarball ran without installed dependencies, exercising the optional-parser fallback, and the evidence manifest/example passed an independent JSON Schema Draft 2020-12 validator.
The complete frozen benchmark ran 24 paired synthetic/public cases across 12 repository groups. Comparison found no new false positives, false negatives, supported-case abstentions, or removed evidence. Persisted runtime observations remain non-comparable by policy; performance claims require five controlled same-session samples.
| Metric | Point estimate | n | Wilson 95% |
|---|---|---|---|
| Finding precision | 1.000 | 12 expected findings, 0 unexpected | 0.757 – 1.000 |
| Finding recall | 1.000 | 12 expected findings, 0 missed | 0.757 – 1.000 |
| Clean-control false-positive case rate | 0.000 | 12 clean controls | 0.000 – 0.243 |
| Accepted repair rate | null | 0 repairs evaluated | null |
Per-detector cells are small: security n=20, structure n=2, todoTracking n=2,
architecture n=0 (null). A cell that small says almost nothing on its own.
Caveat, verbatim from the envelope: Benchmark precision on a deliberately balanced corpus of 12 defect and 12 clean-control cases across 12 repository groups and 12 causal families. This is DocGuard's precision on labelled cases, not the probability that a finding in your repository is real; quote every ratio with its n and Wilson 95% bound.
The envelope's review.measures is benchmark-precision. DocGuard does not
publish a calibration document (P(finding is real) under a base rate): the real
base rate of a stale documented claim is nowhere near the corpus's 50/50 split,
so a probability read off this corpus would mislead. The HIGH/MEDIUM/LOW labels
in guard output are deterministic strata — a validator's check pass-ratio —
and were never calibrated against outcomes. The loader
(benchmarks/lib/baseline.mjs) recomputes every ratio from the retained cases
and rejects an envelope whose numbers or caveat have been edited by hand.
The corpus is six production repositories spanning TypeScript/Next.js SaaS applications, a Python data pipeline, a financial research lab, and a messaging-platform integration — different sizes, doc cultures, and stacks. They are private client/portfolio projects, so results are reported per-repo but unnamed.
Read-only runs on five corpus repos, first-run conditions:
| Detector | Findings across corpus | False positives | Verdict |
|---|---|---|---|
| REF002 — ADR citations in code | 0 (pattern absent) | 0 | Keep — inert where unused, proven on fixtures + self-repo |
| Wikilink validation (XRF001/2) | 0 (evidence gate held) | 0 | Keep — the .obsidian/resolution gate prevented every would-be FP |
| Indirect impact (import-graph) | 7 docs flagged across 2 repos | 0 (chains verified by hand) | Keep — e.g. 74 changed files → 6 explainable doc chains |
| graphify interop (TRC002 evidence) | evidence-only by design | n/a — can only remove warnings | Keep |
Dogfooding caught two real issues before release:
- REF002 initially flagged DocGuard's own source — the validator's doc
comments used realistic
ADR-012examples, and test fixtures cited ADRs. Fix: non-product scoping (tests/fixtures excluded) + digit-free placeholder examples. Shipped with zero self-findings. - The semantic-claim extractor ignored
.docguardignore— an explicitly excluded historical audit contributed 28 of 39 "unverified claims" on DocGuard's own repo, burying the actionable ones. Fixed and regression-tested; the count dropped to 12, and one of those 12 turned out to be real drift (a stale suite-runtime claim in TEST-SPEC.md) — found by the tool, fixed in the same release.
Same method, six-repo corpus. All six detectors shipped default-on after tuning; the levers that mattered:
- Diff-Suspicion (DSP001) needed path/module-only references (basename matching flooded), a generic-token filter (HTTP verbs, CSS words), and a per-doc cap — one repo's route-inventory doc went from 45 raw findings to a capped, reviewable set.
- Reference-Existence (REF001) excludes CLI flags and requires the two-revision gate (present when the doc was written AND absent now) — the two documented false-positive modes from the underlying method (arXiv 2212.01479).
- API-doc-smells (APS001/2) shipped with the deterministic Bloated/Lazy detectors only (reported F1 0.90/0.95 in the source taxonomy) — the LLM-dependent smell classes were left out by design.
- Built-in
node:test, with Node 18/20/22/24 in the supported CI matrix. Per-change validation records contain measured test totals. - Self-guarded: every push runs all 36 validators against DocGuard's own docs; count claims in this README family are machine-governed (Canonical-Sync), so the validator count is checked, not remembered.
- Deterministic core: no LLM calls at validation time, one pinned
dependency (
@babel/parser, with a regex fallback), no network access.
- No recall guarantee. Precision-first means some real drift is missed by design; the recall-maximizing variants of these detectors were tested and rejected because false-positive floods destroy trust faster than misses do.
- No arbitrary prose judgment. Strict
.docguard-evidence.jsondeclarations can verify selected statements against supported local sources. Remaining prose correctness and intent are surfaced as agent tasks (verify --semantic,diagnose), never auto-judged. - Soft by default. The v0.31/v0.32 detectors are
confidence: lowand emit warnings by default. Exit 2 still fails an ordinary shell step; CI warning policy and severity overrides determine enforcement.
Method note: corpus runs are point-in-time (repos evolve); each accuracy release re-runs the sweep. Full per-release detail lives in the CHANGELOG.