Skip to content

Keep Codex compaction on native threads#82027

Merged
steipete merged 3 commits into
mainfrom
codex/codex-native-compaction-only
May 15, 2026
Merged

Keep Codex compaction on native threads#82027
steipete merged 3 commits into
mainfrom
codex/codex-native-compaction-only

Conversation

@pashpashpash

@pashpashpash pashpashpash commented May 15, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Keep Codex app-server compaction on native Codex thread compaction; no context-engine summarizer fallback or primary path for Codex-runtime sessions.
  • Preserve Lossless by treating it as a context engine through plugins.slots.contextEngine, with agent/plugin runtime context passed into its maintenance LLM policy.
  • Add openclaw doctor --fix migration/warnings for stale Codex compaction overrides, legacy Lossless compaction.provider, and legacy codex-app-server runtime aliases.

Verification

Behavior addressed: Fixes #82008 OAuth compaction auth mismatch. Codex OAuth compaction no longer calls the OpenClaw/Public OpenAI Responses summarizer path, while Lossless remains supported through the context-engine slot.

Real environment tested:

Exact steps or command run after this patch:

  • OPENCLAW_VITEST_MAX_WORKERS=1 pnpm test src/commands/doctor/shared/codex-route-warnings.test.ts extensions/codex/src/app-server/compact.test.ts extensions/codex/src/app-server/run-attempt.context-engine.test.ts -- --reporter=verbose
  • node scripts/run-vitest.mjs src/commands/doctor/shared/codex-route-warnings.test.ts extensions/codex/src/app-server/compact.test.ts
  • pnpm plugin-sdk:api:check
  • git diff --check
  • /Users/steipete/Projects/agent-scripts/skills/codex-review/scripts/codex-review --mode branch
  • (cd ../codex/codex-rs && cargo test -p codex-app-server thread_compact_start_triggers_compaction_and_returns_empty_response)

Evidence after fix:

  • Focused tests passed: doctor 59; Codex app-server/context-engine 22.
  • Additional review-run focused test passed: 72 tests across doctor + Codex compaction.
  • Plugin SDK API baseline check passed.
  • Codex review final run reported no accepted/actionable findings.
  • Codex app-server source confirms thread/compact/start takes threadId, submits Op::Compact, returns {}, and emits contextCompaction item started/completed notifications; the targeted sibling Codex test passed.
  • Testbox runs above show before failure and after passes.

Observed result after fix:

  • Codex app-server compaction starts native thread compaction and never invokes contextEngine.compact() as fallback/primary summarizer.
  • OpenClaw's Codex adapter waits for the Codex-native completion signals that the sibling Codex app-server contract documents and tests: contextCompaction item completion, with deprecated thread/compacted still tolerated.
  • Stale OpenClaw compaction summarizer overrides warn at runtime and are removed by doctor when safe, including legacy codex-app-server runtime alias configs.
  • Legacy Lossless config migrates to plugins.slots.contextEngine = "lossless-claw" with bounded summaryModel policy; unsafe mixed/conflicting configs stay intact with manual warnings.
  • Migrated or blocked openai-codex/* Lossless summary models keep Codex runtime policy after canonicalizing to openai/*, including inherited default summary models and per-agent local compaction-only refs.
  • Doctor keeps whole-agent runtime pins when only compaction-only refs are canonicalized and suppresses ignored-override warnings for pins it will clear during route repair.
  • Runtime ignored-override warnings report inherited provider paths at their source path.
  • Lossless active context engine does not produce ignored-override warnings, including inherited provider configs.

What was not tested:

  • No live installed Lossless package against a real user install in this PR pass; covered by contract/unit tests, local project fixtures, Testbox lanes, and sibling Codex app-server contract verification.

@clawsweeper

clawsweeper Bot commented May 15, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge.

Summary
The PR routes Codex app-server compaction through native Codex thread compaction, adds doctor migration/warnings for stale Codex/Lossless compaction config, exports a context-engine owner helper, and updates Codex docs/changelog.

Reproducibility: yes. Current main has a high-confidence source-level reproduction path: an owning context engine can compact before native Codex compaction, and the linked issue plus PR proof show the OAuth api.responses.write failure mode.

Real behavior proof
Sufficient (logs): The PR body includes structured before/after Testbox runs and the discussion includes redacted live beta-run logs showing the original public Responses/OAuth scope failure no longer appears after the patch.

Next step before merge
Protected maintainer label plus live lifecycle/accounting observations require maintainer merge/release judgment; I found no narrow automated repair finding to queue.

Security
Cleared: The diff changes Codex routing, doctor repair logic, docs, changelog, SDK export surface, and tests without adding dependencies, workflows, install scripts, secret access, or downloaded code execution paths.

Review details

Best possible solution:

Land or revise this native Codex compaction fix after maintainers decide whether to fold the lifecycle/accounting observations into this PR or track them as explicit follow-ups.

Do we have a high-confidence way to reproduce the issue?

Yes. Current main has a high-confidence source-level reproduction path: an owning context engine can compact before native Codex compaction, and the linked issue plus PR proof show the OAuth api.responses.write failure mode.

Is this the best way to solve the issue?

Yes for the central bug. Making Codex app-server native compaction the single owner matches the documented runtime contract; the observed lifecycle/accounting gaps need maintainer scoping rather than an automated cleanup close.

Acceptance criteria:

  • node scripts/run-vitest.mjs extensions/codex/src/app-server/compact.test.ts src/commands/doctor/shared/codex-route-warnings.test.ts src/cli/update-cli.test.ts
  • node scripts/crabbox-wrapper.mjs run --provider blacksmith-testbox --shell -- "pnpm check:changed"
  • Live or redacted-log Codex OAuth app-server proof covering WebChat /compact, organic threshold compaction, and /codex compact lifecycle/accounting before release

What I checked:

  • Live PR state: The GitHub API shows this PR is open, unmerged, labeled maintainer, extensions: codex, proof: sufficient, and currently points at head 292d601. (292d601eb8d5)
  • Current-main bug path: On current main, maybeCompactCodexAppServerSession still calls activeContextEngine.compact() before native Codex app-server compaction when the context engine owns compaction, matching the linked OAuth/public Responses fallback problem. (extensions/codex/src/app-server/compact.ts:40, 1e31bd2ac29d)
  • Runtime contract: The Codex harness runtime docs say native thread compaction is delegated to Codex app-server and OpenClaw records native compaction signals, which supports keeping Codex compaction on the native thread path. Public docs: docs/plugins/codex-harness-runtime.md. (docs/plugins/codex-harness-runtime.md:186, 1e31bd2ac29d)
  • PR implementation path: The PR patch removes the context-engine primary compaction result path, warns about ignored OpenClaw summarizer overrides, starts thread/compact/start, then runs context-engine maintenance after successful native compaction. (extensions/codex/src/app-server/compact.ts:39, 3e3a5aa85a2d)
  • Doctor/docs coverage: The PR adds doctor collection/repair logic for unsupported Codex compaction overrides and legacy Lossless compaction config, plus docs explaining that Codex-backed agents should not set compaction.model or compaction.provider. (src/commands/doctor/shared/codex-route-warnings.ts:185, 67ce25deb213)
  • Real behavior proof and follow-up observations: The PR body records before/after Testbox runs and a sibling Codex contract test; the live discussion says the original 401 api.responses.write failure no longer appears with this PR merged locally, while also reporting follow-on lifecycle/accounting issues.

Likely related people:

  • steipete: Recent GitHub path history and the PR commits show repeated work on Codex app-server compaction, client factories, app-server lifecycle, and this PR branch's implementation commits. (role: recent area contributor; confidence: high; commits: f12e9c41fa15, 1e31bd2ac29d, 3e3a5aa85a2d; files: extensions/codex/src/app-server/compact.ts, extensions/codex/src/app-server/run-attempt.ts, src/commands/doctor/shared/codex-route-warnings.ts)
  • pashpashpash: Although also the PR opener, prior merged history shows work on OpenAI-through-Codex defaults and Codex route/doctor migration behavior that this PR extends. (role: route and doctor feature contributor; confidence: medium; commits: 1c3399010815, 02fe0d8978db, 52771b65e27c; files: src/commands/doctor/shared/codex-route-warnings.ts, docs/plugins/codex-harness.md)
  • jalehman: GitHub path history shows Josh Lehman introduced the Codex app-server context-engine lifecycle and later worked on compatible context-engine thread behavior, both adjacent to this compaction routing change. (role: adjacent context-engine owner; confidence: medium; commits: 51186d272543, b55d9fa4660b; files: extensions/codex/src/app-server/compact.ts, extensions/codex/src/app-server/run-attempt.ts)
  • vincentkoc: Recent path history includes Codex auth-order and doctor route-repair work, which intersects the OAuth routing and config migration side of this PR. (role: adjacent Codex auth/doctor contributor; confidence: medium; commits: eb1a0aa574b5, bca670920362; files: extensions/codex/src/app-server/compact.ts, src/commands/doctor/shared/codex-route-warnings.ts)

Remaining risk / open question:

  • A live tester reported that manual WebChat /compact can complete native compaction but leave the active embedded run stuck, and organic compaction may not update compactionCount consistently.
  • The PR body notes no live installed Lossless package was tested; that path is covered by contract/unit tests and Testbox lanes instead.
  • This read-only review did not rerun the PR's tests or Testbox lanes locally.

Codex review notes: model gpt-5.5, reasoning high; reviewed against 1e31bd2ac29d.

@pashpashpash
pashpashpash force-pushed the codex/codex-native-compaction-only branch from 38ed398 to 6af12ba Compare May 15, 2026 06:00
@openclaw-barnacle openclaw-barnacle Bot added docs Improvements or additions to documentation cli CLI command changes commands Command implementations size: L and removed size: M labels May 15, 2026
@steipete
steipete force-pushed the codex/codex-native-compaction-only branch from 6af12ba to 20b4a9a Compare May 15, 2026 08:02
@openclaw-barnacle openclaw-barnacle Bot added size: XL and removed cli CLI command changes size: L labels May 15, 2026
@Conan-Scott

Copy link
Copy Markdown
Contributor

I tested 2026.5.14-beta.1 with this PR merged locally. The original 401 failure appears fixed: I no longer see compaction routed through public openai-responses with the Codex OAuth profile, and there are no api.responses.write scope errors.

I did find a few follow-on lifecycle/accounting issues around the different compaction paths.

WebChat /compact

Running /compact did reach the new Codex app-server wrapper path:

2026-05-15T17:44:18.197+10:00 [agent/embedded] started codex app-server compaction
2026-05-15T17:45:08.396+10:00 [agent/embedded] completed codex app-server compaction
2026-05-15T17:45:08.402+10:00 [ws] ⇄ res ✓ sessions.compact 53111ms

After that, frontend refresh RPCs returned successfully (commands.list, models.list, sessions.list, chat.history), but the active session then appeared stuck as processing / activeWorkKind=embedded_run. Diagnostics repeated:

classification=stalled_agent_run
reason=active_work_without_progress
activeWorkKind=embedded_run
lastProgress=codex_app_server:notification:turn/completed

So the manual /compact path can complete native Codex compaction, but leave the OpenClaw embedded-run lifecycle wedged.

The context meter also did not visibly drop after this manual /compact; it stayed around 45%, though Compactions eventually showed 1.

Organic / threshold compaction

I then forced organic compaction by intentionally pushing the session context up with large read-only tool output.

This behaved differently:

  • The UI visibly showed context dropping mid-turn.
  • Internal status dropped from about 180k/272k (66%) to 34k/272k (12%).
  • The session did not wedge in the same way.
  • But Compactions stayed at 1; it did not increment for the organic compaction.
  • I did not see the same gateway log lines as the manual /compact wrapper path.

So organic compaction appears functionally successful, but it is not reflected in compactionCount, and the logging/accounting path seems different from manual /compact.

/codex compact

I also tried /codex compact. That appears to be a separate direct Codex control path: it starts Codex compaction, but does not seem to use the PR wrapper path, wait for completion, or update OpenClaw accounting in the same way as WebChat /compact.

Current read

This PR looks correct for the original bug: Codex-backed sessions no longer fall back into OpenClaw/public Responses summarization and hit the api.responses.write 401.

The remaining issues seem to be observability/lifecycle/accounting:

  1. Manual WebChat /compact can complete Codex native compaction but leave the active embedded run stuck.
  2. Organic threshold compaction reduces effective context cleanly, but does not increment compactionCount.
  3. UI context/status can be stale or inconsistent across these paths.
  4. /compact, /codex compact, and organic threshold compaction appear to exercise three different bookkeeping/logging paths.

@steipete
steipete force-pushed the codex/codex-native-compaction-only branch from 20b4a9a to 6760999 Compare May 15, 2026 12:08
@clawsweeper clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 15, 2026
@steipete
steipete force-pushed the codex/codex-native-compaction-only branch 2 times, most recently from 3827a6a to 292d601 Compare May 15, 2026 13:44
@steipete
steipete force-pushed the codex/codex-native-compaction-only branch from 292d601 to 8374148 Compare May 15, 2026 14:16
@steipete steipete added the proof: override Maintainer override for the external PR real behavior proof gate. label May 15, 2026
@steipete

Copy link
Copy Markdown
Contributor

Landing proof for rebase merge (not squash):

Behavior addressed: Codex app-server compaction now stays on native Codex thread compaction; OpenClaw context-engine compaction is not used as fallback/primary summarizer for Codex-runtime sessions. Lossless remains supported through the context-engine slot and legacy config is migrated by doctor when safe.

Real environment tested:

Exact steps or command run after this patch:

  • git diff --check origin/main...HEAD
  • pnpm plugin-sdk:api:check
  • OPENCLAW_VITEST_MAX_WORKERS=1 pnpm test src/commands/doctor/shared/codex-route-warnings.test.ts extensions/codex/src/app-server/compact.test.ts extensions/codex/src/app-server/run-attempt.context-engine.test.ts -- --reporter=dot
  • (cd ../codex/codex-rs && cargo test -p codex-app-server thread_compact_start_triggers_compaction_and_returns_empty_response)

Evidence after fix:

  • Rebased stack preserved as three commits; merge method requested: rebase, not squash.
  • Local focused tests passed after latest rebase: 59 doctor tests + 22 Codex extension tests.
  • Plugin SDK API baseline check passed after latest rebase.
  • Codex app-server source/test confirms thread/compact/start submits Op::Compact, returns {}, and emits contextCompaction item started/completed notifications.
  • Codex review final run before rebase reported no accepted/actionable findings; rebased diff only resolved changelog movement.

Observed result after fix:

  • Codex app-server compaction starts native thread compaction and never invokes contextEngine.compact() as fallback/primary summarizer.
  • Stale OpenClaw compaction summarizer overrides warn at runtime and are removed by doctor when safe.
  • Lossless legacy compaction config migrates to plugins.slots.contextEngine = "lossless-claw" with bounded summaryModel policy; unsafe mixed/conflicting configs stay intact with manual warnings.

What was not tested:

  • No live installed Lossless package against a real user install in this final landing pass; covered by contract/unit tests, local fixtures, Testbox lanes, and sibling Codex app-server contract verification.

@steipete
steipete merged commit 1b87ba8 into main May 15, 2026
90 of 94 checks passed
@steipete
steipete deleted the codex/codex-native-compaction-only branch May 15, 2026 14:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

commands Command implementations docs Improvements or additions to documentation extensions: codex maintainer Maintainer-authored PR proof: override Maintainer override for the external PR real behavior proof gate. proof: sufficient ClawSweeper judged the real behavior proof convincing. size: XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Codex OAuth session compaction can call public openai-responses and fail with missing api.responses.write

3 participants