Skip to content

first-hour-20 runtime parity has 3 native-live proof gaps after harness correction #80434

Description

@100yenadmin

TLDR

Resolved as a hard-gate failure by PR #80323's beta.5 correction.

The old body said first-hour-20 had 18 total / 6 pass / 12 fail. That was the pre-correction mock/harness state. The current beta.5 proof shows:

{
  "total": 18,
  "passed": 15,
  "skipped": 3,
  "failed": 0
}

Product impact if OpenClaw moved fully to Codex today: P4 from this issue as filed. The remaining 3 rows are native/live proof gaps, not mock-proven Codex product bugs.

QA impact: P0 resolved for the maintainer mock gate. The gate now has zero hard failures and explicit report-only rows.

Latest Evidence

OpenClaw baseline: v2026.5.10-beta.5
PR: #80323
PR head: 3336dec6419c9cc9a87dc7cfa6f48118ca2d838e
Remote proof run: https://github.com/electricsheephq/openclaw-local-test/actions/runs/25719383976
Confidence tracker: #80936
Artifact: first-hour-20-direct/qa-suite-summary.json

Remote workflow step Run first-hour-20 direct lane completed successfully.

Report-Only Rows

These rows are intentionally not hard-failed in mock mode:

  • Instruction followthrough repo contract: mock-openai cannot exercise Codex-native read/write tools.
  • Approval turn tool followthrough: mock-openai still models approval followthrough as a Pi-style read call; Codex-native approval/read behavior needs native/live proof.
  • Compaction retry after mutating tool: mock-openai cannot create files through Codex-native read/write; compaction replay safety remains a native/live proof lane.

Current Verdict

This issue should stay closed as the stale failing-gate report. Follow remaining proof work in:

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions