Skip to content

v2026.5.20 still requires local stability hotfixes after auto-update #85852

Description

@chac4l

Summary

After auto-updating to OpenClaw v2026.5.20 (e510042), the running deployment still required multiple local installed-runtime hotfixes to keep Codex/ACP/cron workflows stable.

Some of the underlying reports are already open, and one related fix was merged upstream, but the released package installed by auto-update did not contain the full effective behavior needed in the active runtime. This makes it easy for the individual issues to look handled while users still need to patch the installed build manually after updating.

This report intentionally omits hostnames, channel IDs, session IDs, run IDs, project/repository names, local filesystem paths, tokens, and private operational content.

Environment

  • OpenClaw: 2026.5.20 (e510042)
  • @openclaw/codex: 2026.5.20
  • @openclaw/acpx: 2026.5.20
  • Runtime shape: Linux/systemd gateway install with Codex app-server and ACP/ACPX enabled
  • Post-hotfix gateway health: live

Local hotfixes that had to be re-applied after update

1. Stale hashed bundle import after auto-update

Related: #85844

Observed shape:

Cannot find module './run-attempt-<old-hash>.js'
Require stack:
- <codex harness bundle>

The on-disk package had the new hashed chunk, while an active/cached module graph still referenced the old chunk. Local mitigation was a compatibility shim for the removed chunk name plus a gateway restart.

Expected upstream behavior: auto-update should either restart/reexec after replacing hashed chunks, keep old chunks until the old process exits, or verify that dynamic import targets referenced by active entrypoints still exist.

2. Codex app-server terminal-idle timeout remains effectively hardcoded

Related: #85242

The installed runtime still needed the local environment-driven terminal-idle timeout behavior so longer scheduled jobs are not silently capped by the internal Codex terminal-idle watchdog. The local mitigation honors an environment override for the terminal-idle watchdog.

Expected upstream behavior: derive the terminal-idle watchdog from the effective run timeout or expose a supported config/env setting, and make the error message name the real limiting timeout.

3. Codex app-server startup-close retry window is still too small

Related: #83959

The installed runtime still used a short immediate startup-close retry loop. A local mitigation increased the startup-close retry budget and added bounded backoff between attempts, so replacement app-server warm-up does not fail a scheduled task before user work starts.

Expected upstream behavior: retry startup-close failures with a bounded warm-up window and clear diagnostics when the retry budget is actually exhausted.

4. ACPX prompt failures still need bounded pre-side-effect retry

Related: #85207

The active @openclaw/acpx package is now version-aligned and includes stderr-tail diagnostic enrichment, but the installed runtime still needed a local bounded retry around transient prompt failures before any prompt-side effects were observed.

The mitigation retries only transient/generic ACP prompt failures before assistant output, tool calls, or session-side effects are detected. It preserves the hard failure once the turn has produced side effects.

Expected upstream behavior: the maintained ACPX runtime should include bounded pre-side-effect retry for transient Codex ACP prompt failures, while keeping no-retry semantics after visible side effects.

5. Progress-only child completion contract not present in installed release

Related: #83863, fixed upstream by #85110

Although #83863 is closed and #85110 was merged, the installed v2026.5.20 runtime still did not contain the effective required-completion contract in the active ACP/subagent bundles. Required child completions could still be finalized as plain success when the captured text was only progress-style output.

Local mitigation re-applied the progress-only required-completion classifier and maps missing/progress-only required completion, or exhausted required completion delivery, to terminalOutcome=blocked with a clear terminal summary.

Expected upstream behavior: the release artifact installed by auto-update should include the merged completion contract, or the issue should remain open until a released package contains it.

Related issue that did not require a code hotfix here

Related: #85526

The local prompt/config mitigation for side-effect-then-timeout cron work remained in place, but the root runtime behavior still appears related: scheduled Codex turns can create an external artifact and then time out before final assistant delivery. This report is mainly about installed-runtime hotfixes that had to be re-applied after the update.

Verified fixed in this release

Related: #83836

The invalid_encrypted_content replay mitigation appears present in the installed runtime: reasoning replay metadata and encrypted-content stripping are present in request assembly. No local re-application was needed for that issue in this pass.

Validation after local hotfixes

After re-applying the local mitigations:

  • syntax checks passed on the modified installed-runtime bundles;
  • Codex/ACPX runtime imports succeeded;
  • gateway restarted cleanly and health returned live;
  • a direct Codex agent smoke run completed through codex/gpt-5.5 with no fallback;
  • post-restart logs did not show new occurrences of:
    • Cannot find module './run-attempt-<old-hash>.js'
    • ACP_TURN_FAILED
    • AcpRuntimeError
    • invalid_encrypted_content
    • client is closed
    • Request timed out before a response was generated
    • startup retry exhaustion

Suggested durable fix

Please treat this as a release/regression consolidation rather than five unrelated user-local patches:

  1. Verify which commits are actually included in the v2026.5.20 auto-update artifact.
  2. Backport or release the merged fix(codex): block progress-only completions [AI-assisted] #85110 behavior in the package served by auto-update.
  3. Land durable fixes for Auto-update can leave running gateway with stale hashed bundle imports #85844, Codex app-server terminal idle timeout is hardcoded and reports misleading config guidance #85242, Codex app-server startup retries can exhaust before replacement server is ready #83959, and Codex ACP gateway path can surface bare ACP_TURN_FAILED when active acpx plugin misses runtime hardening #85207.
  4. Add a release smoke/invariant test that checks the installed artifact, not only source main:
    • no active entrypoint references missing hashed chunks;
    • Codex terminal-idle timeout override or derived timeout is honored;
    • app-server startup-close retry has bounded backoff;
    • ACPX transient prompt failure retries only before side effects;
    • required progress-only ACP/subagent completions become blocked, not plain success.

The key user-facing problem is that auto-update can make the deployment look current while still requiring manual runtime patches for known stability defects.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High-priority user-facing bug, regression, or broken workflow.clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.clawsweeper:needs-live-reproClawSweeper needs live local, crabbox, or manual validation to confirm this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.impact:crash-loopCrash, hang, restart loop, or process-level availability failure.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.issue-rating: 🐚 platinum hermitGood issue quality with a plausible reproduction path needing some confirmation.

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions