Skip to content

fix(codex): persist automatic compaction history#110587

Merged
steipete merged 4 commits into
mainfrom
test/codex-compaction-restart-stress
Jul 18, 2026
Merged

fix(codex): persist automatic compaction history#110587
steipete merged 4 commits into
mainfrom
test/codex-compaction-restart-stress

Conversation

@steipete

Copy link
Copy Markdown
Contributor

Closes #110586

What Problem This Solves

Fixes an issue where long-running Codex app-server sessions would omit completed automatic native compactions from OpenClaw session history, leaving /status and restart/resume diagnostics with an inaccurate count.

Why This Change Was Made

The Codex projector now returns its completed native compaction count through the canonical top-level attempt field consumed by the embedded runner. The live harness also gains concurrent subagent fan-out, large-output compaction stress, explicit physical app-server identity, repeated cold Gateway restarts, same-thread resume checks, persistent count checks, and post-restart continuation.

The harness uses the stable pinned Codex CLI contract. No protocol, schema version, runtime config surface, or compatibility path changes.

User Impact

Codex sessions now retain truthful automatic-compaction history across Gateway restart and thread resume. Maintainers can run one bounded live scenario that stresses concurrent children, large outputs, compaction, cold restart, and continued work.

Evidence

  • Real OpenAI API-key live gateway run using the stable pinned Codex CLI (0.144.5).
  • 8 direct child agents started concurrently and verified to terminal completion, then repeated after restart stress: 16 direct child runs total.
  • Four 300 KB native tool-output turns before restart plus four after restart: 2.4 MB configured output total, with transcript truncation and automatic compaction events verified.
  • Three physical Gateway/Codex app-server replacements. The OpenClaw session and Codex thread IDs stayed stable, lifecycle action was resumed, and the app-server client identity changed each time.
  • Persistent /status compaction count and durable marker recall verified before and after restart; work continued with another fan-out and compaction wave.
  • Focused tests: 223 passed across Docker harness, Gateway helper, and Codex projector shards.
  • Exact-diff pnpm check:changed: all format, generated SDK baseline, five typecheck lanes, core/extensions/scripts lint, import-cycle, storage, runtime-sidecar, pairing, webhook, and live-Docker guards passed on Blacksmith Testbox.
  • Mandatory autoreview: one actionable tuning-range finding fixed; clean rerun with no findings.
  • AI-assisted. I reviewed the implementation, Codex upstream source contract, live behavior, and validation output.

@openclaw-barnacle openclaw-barnacle Bot added docs Improvements or additions to documentation gateway Gateway runtime scripts Repository scripts docker Docker and sandbox tooling extensions: codex size: L maintainer Maintainer-authored PR labels Jul 18, 2026
@steipete steipete self-assigned this Jul 18, 2026
@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. P2 Normal backlog priority with limited blast radius. labels Jul 18, 2026
@clawsweeper

clawsweeper Bot commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed July 18, 2026, 5:19 AM ET / 09:19 UTC.

Summary
Moves Codex automatic-compaction accounting from nested item lifecycle metadata to the top-level attempt result, with projector coverage and expanded live restart/compaction harness documentation.

PR surface: Source +29, Tests +350, Docs +23, Generated 0, Other +12. Total +414 across 11 files.

Reproducibility: yes. from source and the linked report: a completed native compaction is projected under itemLifecycle while the reported persistence consumer reads the top-level attempt field; the PR also describes a live restart/resume reproduction path.

Review metrics: none identified.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #110586
Summary: This PR is the explicit candidate fix for the paired open report describing the same Codex compaction-count persistence mismatch.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🦞 diamond lobster
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • Tighten the PR description with what changed, how it was validated, and any remaining risk.

Risk before merge

  • [P1] Before merge, an owner needs to verify the pinned Codex app-server contract directly and confirm that top-level attempt accounting is the durable session-state boundary across the supported Codex version.

Maintainer options:

  1. Decide the mitigation before merge
    Continue normal maintainer review; ClawSweeper found no patch-correctness issue.
  2. Pause or close
    Do not merge this PR until maintainers decide whether the risk is worth taking.

Next step before merge

  • Continue normal maintainer review; ClawSweeper found no patch-correctness issue.

Maintainer decision needed

  • Question: Can a Codex-area owner directly verify the pinned Codex app-server event contract and confirm that the top-level attempt compactionCount is the correct durable session-state boundary before merge?
  • Rationale: OpenClaw policy makes direct inspection of sibling Codex source a hard prerequisite for a Codex verdict; this restricted review could not perform that inspection, while the PR carries a protected maintainer label.
  • Likely owner: steipete — He is the assigned current Codex-area contributor and supplied the live reproduction and proposed boundary change.
  • Options:
    • Verify contract, then land (recommended): Inspect the pinned Codex app-server source and retain the field relocation if its completed-compaction event semantics match the projector’s persisted attempt result.
    • Hold for boundary redesign: Pause the PR if Codex exposes a different authoritative compaction lifecycle boundary that should be normalized before persistence.

Security
Cleared: The inspected diff changes Codex result projection and opt-in live-test harness controls without adding dependencies, credentials, permissions, artifact execution, or supply-chain sources.

Review details

Do we have a high-confidence way to reproduce the issue?

Yes, from source and the linked report: a completed native compaction is projected under itemLifecycle while the reported persistence consumer reads the top-level attempt field; the PR also describes a live restart/resume reproduction path.

Is this the best way to solve the issue?

Unclear pending the required direct Codex contract check. Moving the count to the top-level attempt result is the narrowest apparent repair because it matches the reported consumer, but that dependency-backed boundary must be confirmed before merge.

AGENTS.md: unclear because the file could not be read completely.

Codex review notes: model internal, reasoning high; reviewed against e1d87e9a9db9.

Label changes

Label changes:

  • add P2: This fixes inaccurate compaction history and status diagnostics for automatically compacting Codex sessions, without evidence of data loss or runtime unavailability.
  • add proof: sufficient: Contributor real behavior proof is sufficient. The PR body provides detailed after-fix live Gateway evidence across compaction, cold restart, thread resume, persisted status, and continued work; redact any private endpoint or credential details if fuller logs are attached.
  • add rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
  • add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The PR body provides detailed after-fix live Gateway evidence across compaction, cold restart, thread resume, persisted status, and continued work; redact any private endpoint or credential details if fuller logs are attached.

Label justifications:

  • P2: This fixes inaccurate compaction history and status diagnostics for automatically compacting Codex sessions, without evidence of data loss or runtime unavailability.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The PR body provides detailed after-fix live Gateway evidence across compaction, cold restart, thread resume, persisted status, and continued work; redact any private endpoint or credential details if fuller logs are attached.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR body provides detailed after-fix live Gateway evidence across compaction, cold restart, thread resume, persisted status, and continued work; redact any private endpoint or credential details if fuller logs are attached.
Evidence reviewed

PR surface:

Source +29, Tests +350, Docs +23, Generated 0, Other +12. Total +414 across 11 files.

View PR surface stats
Area Files Added Removed Net
Source 3 33 4 +29
Tests 4 432 82 +350
Docs 2 23 0 +23
Config 0 0 0 0
Generated 1 1 1 0
Other 1 12 0 +12
Total 11 501 87 +414

What I checked:

  • Targeted persistence-path change: The PR moves completed native compaction accounting onto the projector result’s top-level compactionCount, while removing it from itemLifecycle; the linked bug reports that the embedded runner consumes the top-level field before session persistence. (extensions/codex/src/app-server/event-projector.ts:434, d9ca9d058637)
  • Regression coverage follows the data-shape boundary: The changed projector test now asserts the top-level result count and asserts that itemLifecycle no longer carries the field, directly covering the mismatch described by the linked report. (extensions/codex/src/app-server/event-projector.test.ts:2998, d9ca9d058637)
  • Live behavior evidence: The PR body reports a real API-key Gateway scenario with automatic compaction, three cold Gateway/app-server replacements, same-thread resume, persisted status verification, marker recall, and continued work; the provided checks also record a successful Real behavior proof run. (src/gateway/gateway-codex-harness.live.test.ts:82, d9ca9d058637)
  • Upstream contract gate remains unverified: Repository policy requires direct inspection of the sibling Codex source for exact Codex runtime/protocol behavior before a Codex verdict. The review sandbox rejected all terminal inspection before execution, so no direct ../codex source or local history/provenance check was possible.
  • Linked canonical bug remains open: The PR is explicitly paired with the open bug report describing the same field-shape mismatch; it should remain open until this candidate fix receives review and lands. (d9ca9d058637)

Likely related people:

  • steipete: Authored the three commits in this PR, filed the paired current reproduction, and is assigned to this work; direct current-main history could not be inspected in the restricted review environment. (role: current Codex-area contributor; confidence: low; commits: 70a9c7564ed8, a70107987adc, d9ca9d058637; files: extensions/codex/src/app-server/event-projector.ts, src/gateway/gateway-codex-harness.live.test.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@steipete
steipete force-pushed the test/codex-compaction-restart-stress branch from d9ca9d0 to ddecd3b Compare July 18, 2026 09:28

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cf004b7b35

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines 1176 to 1179
const review = assertGuardianReviewCompleted({
events: deniedResult.events,
label: "ask-back probe",
requireEvents: false,
});

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Restore Guardian event opt-out for ask-back probe

When this live harness is run through scripts/test-live-codex-harness-docker.sh, OPENCLAW_LIVE_CODEX_HARNESS_REQUIRE_GUARDIAN_EVENTS defaults to 1. This call no longer passes requireEvents: false, so if Codex blocks/refuses the risky ask-back prompt before emitting a completed Guardian review (a case the fallback below still handles), assertGuardianReviewCompleted throws before that fallback can validate the ask-back. This makes the Docker Guardian probe fail despite correct behavior; keep this probe tolerant by passing requireEvents: false.

Useful? React with 👍 / 👎.

@steipete
steipete merged commit f14947d into main Jul 18, 2026
116 checks passed
@steipete
steipete deleted the test/codex-compaction-restart-stress branch July 18, 2026 09:37
@steipete

Copy link
Copy Markdown
Contributor Author

Merged via squash.

MaheshBhushan added a commit to MaheshBhushan/openclaw that referenced this pull request Jul 18, 2026
Commit f14947d (openclaw#110587) added a state.client.getInstanceId() call in
startCodexAttemptRuntime but did not update the app-server test fake clients,
which omit getInstanceId. Those tests only run under the changed-file test
selector, so the breakage stayed dormant on main and surfaces on any PR that
touches their import graph (here, a one-line change in attempt-context.ts).

Add getInstanceId to the four fake-client method sets so the changed-graph
Codex app-server shards run instead of throwing 'getInstanceId is not a
function' (which also stalled the run-attempt shards to their no-output kill).

Not related to the memory_recall fix in this PR; bundled only to unblock CI.
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jul 19, 2026
* fix(codex): persist automatic compaction counts

* test(codex): stress compaction and restart recovery

* test(codex): bind compaction waves to fresh counts

* chore: keep Codex release note in PR
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docker Docker and sandbox tooling docs Improvements or additions to documentation extensions: codex gateway Gateway runtime maintainer Maintainer-authored PR P2 Normal backlog priority with limited blast radius. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. scripts Repository scripts size: L status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Codex automatic compactions are omitted from session history

1 participant