Summary
The live runtime-parity and token-efficiency reports require positive assistant-message usage for every runtime-pair scenario. Seven standard/live-only flows intentionally execute no agent/provider turn, so their correct zero usage deterministically fails both release reports.
Affected scenarios
plugin-manifest-contract-healthauth-profile-codex-mixed-profilesauth-profile-doctor-migration-safetycodex-plugin-cold-installcodex-plugin-install-racecodex-plugin-pinned-newcodex-plugin-pinned-old
The manifest flow only checks startup/log health. The other six run local auth/plugin lifecycle fixtures. None creates assistant-message usage; synthetic fixture counters are not provider usage.
Expected
Scenario catalog metadata explicitly declares whether assistant-message usage is required. Omission must fail closed to required. A non-applicable scenario must carry a reason, remain subject to runtime/drift/sentinel parity checks, render tokens as N/A, and be excluded from token aggregates.
Required scenarios with zero usage must remain red. A live report containing only N/A scenarios must not produce green token proof.
Actual
No catalog usage-applicability contract exists. Both live reports blanket-require positive usage for every runtime parity result, producing seven deterministic false usage failures.
Proposed bounded fix
- Add discriminated
runtimeParityUsage scenario metadata with an explicit not-applicable reason; omission defaults to assistant-message-required.
- Mark exactly the seven fixture/startup flows above.
- Propagate the resolved policy into the self-describing
RuntimeParityResult summary payload.
- Runtime parity: suppress only missing-usage failure for N/A rows; keep every runtime/drift/sentinel gate.
- Token efficiency: exclude only explicitly N/A rows from aggregates, expose structured reasons, retain required-zero failures, and fail N/A-only live proof.
- Add schema/catalog, runner/summary, parity-report, and token-report regressions.
Evidence
Exact landed release-QA proof 4bc300843d5ec9f071257c2900701f91df44e1c3 produced 0/0 usage for all seven while both runtime cells passed with no drift. Artifacts remain on Testbox tbx_01kx5qan3y2kpr49rd9h4c5ajb; no transcript or secret content is included here.
Summary
The live runtime-parity and token-efficiency reports require positive assistant-message usage for every runtime-pair scenario. Seven standard/live-only flows intentionally execute no agent/provider turn, so their correct zero usage deterministically fails both release reports.
Affected scenarios
plugin-manifest-contract-healthauth-profile-codex-mixed-profilesauth-profile-doctor-migration-safetycodex-plugin-cold-installcodex-plugin-install-racecodex-plugin-pinned-newcodex-plugin-pinned-oldThe manifest flow only checks startup/log health. The other six run local auth/plugin lifecycle fixtures. None creates assistant-message usage; synthetic fixture counters are not provider usage.
Expected
Scenario catalog metadata explicitly declares whether assistant-message usage is required. Omission must fail closed to required. A non-applicable scenario must carry a reason, remain subject to runtime/drift/sentinel parity checks, render tokens as N/A, and be excluded from token aggregates.
Required scenarios with zero usage must remain red. A live report containing only N/A scenarios must not produce green token proof.
Actual
No catalog usage-applicability contract exists. Both live reports blanket-require positive usage for every runtime parity result, producing seven deterministic false usage failures.
Proposed bounded fix
runtimeParityUsagescenario metadata with an explicitnot-applicablereason; omission defaults toassistant-message-required.RuntimeParityResultsummary payload.Evidence
Exact landed release-QA proof
4bc300843d5ec9f071257c2900701f91df44e1c3produced 0/0 usage for all seven while both runtime cells passed with no drift. Artifacts remain on Testboxtbx_01kx5qan3y2kpr49rd9h4c5ajb; no transcript or secret content is included here.