Complete private runtime reports and saved-run acceptance - #345

Summary

Private runs already collect loading, MCP, behavior and comparison evidence, but cannot show those facts consistently or replay them after temporary artifacts are removed. This implements split-plan item 24 with one passive runtime projection and a retained display snapshot. Public activation remains item 25.

• Render loading strength, arm/activation coverage, routing/tool/argument/order/handoff/conflict signals, MCP proof/call outcomes, paired intervals and exclusions, eligibility, recorded costs and measured context through the existing six formats. Preserve unknowns, unsupported expectations and omission counts; reuse shared sanitization and table rendering.
• Atomically save a versioned, bounded plugin_runtime_report.json before private cleanup and caller publication. Offline replay uses fixed-file, no-follow, bounded JSON/schema validation, requires no source tree or optional runtime dependencies, and performs no probes or regrading. Storage failures remain advisory and cannot suppress caller publication or cleanup.
• Fix a classification seam exposed by assembled acceptance: comparisons now consume the context's existing canonical cross-component set. The previous signal-only case view strips that flag and incorrectly left five valid cross-component cases out of the comparison split. No statistical formula or eligibility gate is relaxed.

The ordinary reporter/CLI entrypoints remain unchanged. Snapshot replay is an explicit private API; older runs without a snapshot report it unavailable. Private usage and the implementation design are documented in the included contributor notes.

Verification

• Familiar with the contributing guidelines; commits signed off.
• Added focused regressions and completed independent review; both review findings fixed and re-reviewed.
• 98 passed across projection, replay, runtime/static rendering and existing lifecycle checks.
• 6 assembled acceptance cases passed: default service/runner, five cross-component cases × two attempts × four arms, source removal and six-format replay, missing load/trace evidence, publication failures and ordinary-run isolation.
• make lint, changed-file formatting, make build, OSS-boundary scan, Fern check (0 errors, 1 warning), and staged/new-commit secret scans passed.
• Ran full make test: 22,125 passed, 60 skipped, 27 failed with an initially unlocked environment/system Python. The failing test and owning source files are unchanged from the task base. All 27 previously failing cases passed after syncing locked dependencies and selecting the project Python on PATH (85.54s). The original failed full run is retained; this is not a claim that the initial full run was green.
• Exact-wheel base-only installation: all 247 Python/Jinja/JSON package files match checkout, wheel, source archive and installation. Six-format offline replay and three installed CLI help checks passed; optional runtime/provider packages and public activation flags remain absent.
• No credentials, private datasets or proprietary benchmark content added.

Acceptance substitutes external provider/container execution with local deterministic fixtures and runs emitted setup/census scripts. It exercises real discovery, validation, preparation, service/runner orchestration, evidence, comparisons, publication and cleanup. It does not qualify live agents, providers, remote backends or MCP services.

CI full test jobs passed on Python 3.12, 3.13 and 3.14 for the implementation commit. Windows exposed an oversized pytest parameter ID; the follow-up gives the fixture short IDs (21 replay checks passed locally) and the latest CI run is pending. Product code is unchanged by that follow-up.

Review follow-up

Three verified inline findings are fixed in 37bdd247 and their threads are replied to and resolved:

• Enforce the final sanitized replay size in the shared validator, removing the redundant save-only check.
• Publish canonical logical-case coverage alongside attempt coverage, including skipped, partial, early-stopped and recovered cases.
• Retain bounded per-comparison policy exclusions for two-/three-arm runs without a ladder, with duplicate suppression.

Post-merge compatibility: 111 passed, including the real four-arm workflow and saved replay. The extended affected run recorded 1,899 passed, 6 skipped and 4 subprocess deadline failures; all four passed in a serial rerun at the original limits. Lint, builds, installed base-only replay, 247 Python/Jinja/JSON byte comparisons, three CLI help checks and secret scans passed. Independent re-review found no new actionable issue. The concurrent remote merge is preserved; the latest CI run is pending.

Release Impact

• Updated CHANGELOG.md and private API usage notes.
• Public CLI/configuration/detection/exports and installed public workflow acceptance remain item 25.