Skip to content

runtime: fail closed on plan stalls and retry session-delta sync#62989

Closed
ykq wants to merge 6 commits into
openclaw:mainfrom
ykq:codex/transactional-reclaim-source-patch
Closed

runtime: fail closed on plan stalls and retry session-delta sync#62989
ykq wants to merge 6 commits into
openclaw:mainfrom
ykq:codex/transactional-reclaim-source-patch

Conversation

@ykq

@ykq ykq commented Apr 8, 2026

Copy link
Copy Markdown

Summary

  • fail closed when a turn emits plan/control progress and then stalls without concrete work
  • retry session-delta memory sync after fetch failures instead of hammering and dropping pending work
  • add focused tests for both behaviors

Why

Discord and main-session turns were freezing after control-plane events like update_plan, leaving sessions stuck in processing with no visible reply. Separately, memory session-delta sync was repeatedly logging fetch failed without preserving pending work for retry.

Verification

  • node scripts/run-vitest.mjs run --config vitest.auto-reply-reply.config.ts src/auto-reply/reply/dispatch-from-config.test.ts
  • node scripts/run-vitest.mjs run --config vitest.extension-memory.config.ts extensions/memory-core/src/memory/manager.sync-errors-do-not-crash.test.ts
  • NODE_OPTIONS=--max-old-space-size=8192 pnpm exec tsc -p tsconfig.json --noEmit

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7781256e03

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment on lines +335 to +336
if (concreteProgressSeen || controlPlaneStallTimer || ctx.CommandSource === "native") {
return;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Reset stall timer on repeated plan updates

The stall guard never extends its deadline after the first onPlanUpdate("update") event because armControlPlaneStallTimer returns early whenever a timer already exists. In runs that continue emitting plan updates for longer than 45 seconds before the first tool/block output, this will incorrectly abort an active turn and send the "Turn stalled" error even though control-plane progress is still being produced. Re-arming (or refreshing) the timer on each update would avoid these false positives.

Useful? React with 👍 / 👎.

@greptile-apps

greptile-apps Bot commented Apr 8, 2026

Copy link
Copy Markdown
Contributor

Greptile encountered an error while reviewing this PR. Please reach out to [email protected] for assistance.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2fd3041a84

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment on lines +318 to +320
const controlPlaneAbort = createAbortControllerWithForwardedSignal(
params.replyOptions?.abortSignal,
);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Ensure forwarded abort listeners are always cleaned up

createAbortControllerWithForwardedSignal is called before the plugin-bound early-return paths, but controlPlaneAbort.cleanup() only runs in the later finally block. When replyOptions.abortSignal is provided and execution returns early (for example, plugin-bound handled/declined/error paths), the listener added to the upstream signal is never removed, so long-lived channel abort signals can accumulate listeners and leak memory over many messages.

Useful? React with 👍 / 👎.

@openclaw-barnacle openclaw-barnacle Bot added the agents Agent runtime and tooling label Apr 9, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c436a4e7e0

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment on lines +235 to +236
if (typeof params.entry.cleanupCompletedAt === "number") {
params.entry.cleanupCompletedAt = undefined;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve completed cleanup state for orphaned runs

markOrphanedRunForCleanup always clears cleanupCompletedAt, which reopens runs that were already fully cleaned up/announced on previous boots. On the next restore those runs are treated as pending again (because pending-descendant logic keys off cleanupCompletedAt), so they can be retried and re-announced as new orphan errors, overwriting prior terminal state and producing duplicate parent-facing failure messages. This should skip or preserve completed cleanup records instead of resetting them.

Useful? React with 👍 / 👎.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 32f77e39fc

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/logging/diagnostic.ts
const now = Date.now();
pruneDiagnosticSessionStates(now, true);
for (const [, state] of diagnosticSessionStates) {
if (state.state === "processing" && !sessionHasLiveRun(state)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Limit stale-session reconciliation to tracked run sources

This loop unconditionally flips any processing session to idle when sessionHasLiveRun cannot find it in the embedded/reply registries, but not every dispatch path registers there (for example plugin inbound dispatch calls dispatchReplyFromConfig directly in src/plugin-sdk/inbound-reply-dispatch.ts and still marks the session as processing). In those channels, any turn that runs past one heartbeat interval is reclassified as idle mid-flight, which suppresses stuck-session diagnostics and skews queue/state telemetry for active work.

Useful? React with 👍 / 👎.

Comment on lines +74 to +76
if (normalized === "main") {
return { sessionKey: "agent:main:main" };
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Avoid hardcoding main-lane fail-close to agent:main:main

resolveSessionRefForLane("main") always maps to agent:main:main, but the global main lane is shared by runs from other sessions (for example embedded runs enqueue their global step on main in src/agents/pi-embedded-runner/run.ts). A FailoverError from one of those runs will incorrectly mark the main session idle and decrement its queue depth, while leaving the real session state unchanged, so fail-closed diagnostics point at the wrong session.

Useful? React with 👍 / 👎.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2a826ce4ba

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment on lines +399 to +403
} catch (err) {
const message = String(err);
this.providerUnavailableReason = message;
log.warn(`memory search degrading to keyword-only after query embedding failure: ${message}`);
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve keyword fallback when embedding lookup throws

When embedQueryWithTimeout fails, this branch swallows the exception and leaves queryVec empty, but the downstream if (!hybrid.enabled || !this.fts.enabled || !this.fts.available) path still returns vector-only results. In configurations with query.hybrid.enabled = false (vector-first mode), a transient embedding outage now produces silent empty searches instead of either falling back to keyword results or surfacing an error, which can hide relevant memory hits during provider incidents.

Useful? React with 👍 / 👎.

Comment on lines +488 to +490
if (this.sessionWatchTimer) {
return;
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Enforce retry backoff even with pending debounce timer

The fetch-failure retry path exits early whenever sessionWatchTimer already exists, but that same timer is also used for normal 5s session-delta debounce. Under active transcript traffic during a fetch outage, new updates keep this timer populated, so the 60s retry delay is skipped and retries continue at debounce cadence, causing repeated failed sync attempts and log churn instead of true backoff.

Useful? React with 👍 / 👎.

@clawsweeper

clawsweeper Bot commented Apr 27, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs real behavior proof before merge. Reviewed June 5, 2026, 1:01 AM ET / 05:01 UTC.

Summary
Review failed before ClawSweeper could summarize the requested change.

PR surface: Source +245, Tests +339. Total +584 across 17 files.

Reproducibility: unclear. The review failed before ClawSweeper could establish a reproduction path.

Review metrics: none identified.

Merge readiness
Overall: 🌊 off-meta tidepool
Proof: 🌊 off-meta tidepool
Patch quality: 🌊 off-meta tidepool
Result: rating does not apply to this item.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Risk before merge

  • [P1] No close action taken because the review did not complete.

Maintainer options:

  1. Decide the mitigation before merge
    Retry the Codex review after fixing the execution failure.
  2. Pause or close
    Do not merge this PR until maintainers decide whether the risk is worth taking.

Next step before merge

  • [P1] Review did not complete, so no work-lane recommendation was made.
Review details

Best possible solution:

Retry the Codex review after fixing the execution failure.

Do we have a high-confidence way to reproduce the issue?

Unclear. The review failed before ClawSweeper could establish a reproduction path.

Is this the best way to solve the issue?

Unclear. Retry the review first so ClawSweeper can evaluate the actual issue and fix direction.

AGENTS.md: unclear because the file could not be read completely.

Codex review notes: model gpt-5.5, reasoning high; reviewed against e0018382eb00.

Label changes

Label changes:

  • add rating: 🌊 off-meta tidepool: Overall readiness is 🌊 off-meta tidepool; proof is 🌊 off-meta tidepool and patch quality is 🌊 off-meta tidepool.
  • remove P1: Current review triage priority is none.
  • remove rating: 🧂 unranked krab: Current PR rating is rating: 🌊 off-meta tidepool, so this older rating label is no longer current.
  • remove merge-risk: 🚨 session-state: Current PR review selected no merge-risk labels.
  • remove merge-risk: 🚨 message-delivery: Current PR review selected no merge-risk labels.
  • remove merge-risk: 🚨 availability: Current PR review selected no merge-risk labels.
  • remove status: 📣 needs proof: Current PR status no longer selects a status label.

Label justifications:

  • rating: 🌊 off-meta tidepool: Overall readiness is 🌊 off-meta tidepool; proof is 🌊 off-meta tidepool and patch quality is 🌊 off-meta tidepool.
Evidence reviewed

PR surface:

Source +245, Tests +339. Total +584 across 17 files.

View PR surface stats
Area Files Added Removed Net
Source 10 285 40 +245
Tests 7 348 9 +339
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 17 633 49 +584

What I checked:

  • failure reason: codex execution failed.
  • codex failure detail: Codex review failed for this PR with exit 1.
  • codex stdout: Per-item Codex failure; continuing with the rest of the shard.

Likely related people:

  • unknown: Codex failed before it could trace repository history. (role: review did not complete; confidence: low)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P1 High-priority user-facing bug, regression, or broken workflow. merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. merge-risk: 🚨 availability 🚨 May cause crashes, hangs, restart loops, stalls, or process outages. labels May 19, 2026
@openclaw-barnacle openclaw-barnacle Bot added the triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. label May 19, 2026
@clawsweeper

clawsweeper Bot commented May 20, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper PR egg

🎁 Pass real behavior proof to wake the egg and unlock a hatchable treat.

Where did the egg go?
  • The egg game starts only after the PR passes the real-behavior proof check.
  • Before that, no creature or rarity is rolled. The treat waits for real proof.
  • This is still just collectible flavor: proof affects review readiness, not creature quality.

@openclaw-barnacle

Copy link
Copy Markdown

This pull request has been automatically marked as stale due to inactivity.
Please add updates or it will be closed.

@openclaw-barnacle openclaw-barnacle Bot added the stale Marked as stale due to inactivity label Jun 5, 2026
@clawsweeper clawsweeper Bot added rating: 🌊 off-meta tidepool PR readiness rating does not apply to this item. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jun 5, 2026
@ykq ykq closed this Jun 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling extensions: memory-core Extension: memory-core merge-risk: 🚨 availability 🚨 May cause crashes, hangs, restart loops, stalls, or process outages. merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. P1 High-priority user-facing bug, regression, or broken workflow. rating: 🌊 off-meta tidepool PR readiness rating does not apply to this item. size: L stale Marked as stale due to inactivity triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant