runtime: fail closed on plan stalls and retry session-delta sync#62989
runtime: fail closed on plan stalls and retry session-delta sync#62989ykq wants to merge 6 commits into
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7781256e03
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| if (concreteProgressSeen || controlPlaneStallTimer || ctx.CommandSource === "native") { | ||
| return; |
There was a problem hiding this comment.
Reset stall timer on repeated plan updates
The stall guard never extends its deadline after the first onPlanUpdate("update") event because armControlPlaneStallTimer returns early whenever a timer already exists. In runs that continue emitting plan updates for longer than 45 seconds before the first tool/block output, this will incorrectly abort an active turn and send the "Turn stalled" error even though control-plane progress is still being produced. Re-arming (or refreshing) the timer on each update would avoid these false positives.
Useful? React with 👍 / 👎.
|
Greptile encountered an error while reviewing this PR. Please reach out to [email protected] for assistance. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2fd3041a84
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| const controlPlaneAbort = createAbortControllerWithForwardedSignal( | ||
| params.replyOptions?.abortSignal, | ||
| ); |
There was a problem hiding this comment.
Ensure forwarded abort listeners are always cleaned up
createAbortControllerWithForwardedSignal is called before the plugin-bound early-return paths, but controlPlaneAbort.cleanup() only runs in the later finally block. When replyOptions.abortSignal is provided and execution returns early (for example, plugin-bound handled/declined/error paths), the listener added to the upstream signal is never removed, so long-lived channel abort signals can accumulate listeners and leak memory over many messages.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c436a4e7e0
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| if (typeof params.entry.cleanupCompletedAt === "number") { | ||
| params.entry.cleanupCompletedAt = undefined; |
There was a problem hiding this comment.
Preserve completed cleanup state for orphaned runs
markOrphanedRunForCleanup always clears cleanupCompletedAt, which reopens runs that were already fully cleaned up/announced on previous boots. On the next restore those runs are treated as pending again (because pending-descendant logic keys off cleanupCompletedAt), so they can be retried and re-announced as new orphan errors, overwriting prior terminal state and producing duplicate parent-facing failure messages. This should skip or preserve completed cleanup records instead of resetting them.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 32f77e39fc
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| const now = Date.now(); | ||
| pruneDiagnosticSessionStates(now, true); | ||
| for (const [, state] of diagnosticSessionStates) { | ||
| if (state.state === "processing" && !sessionHasLiveRun(state)) { |
There was a problem hiding this comment.
Limit stale-session reconciliation to tracked run sources
This loop unconditionally flips any processing session to idle when sessionHasLiveRun cannot find it in the embedded/reply registries, but not every dispatch path registers there (for example plugin inbound dispatch calls dispatchReplyFromConfig directly in src/plugin-sdk/inbound-reply-dispatch.ts and still marks the session as processing). In those channels, any turn that runs past one heartbeat interval is reclassified as idle mid-flight, which suppresses stuck-session diagnostics and skews queue/state telemetry for active work.
Useful? React with 👍 / 👎.
| if (normalized === "main") { | ||
| return { sessionKey: "agent:main:main" }; | ||
| } |
There was a problem hiding this comment.
Avoid hardcoding main-lane fail-close to agent:main:main
resolveSessionRefForLane("main") always maps to agent:main:main, but the global main lane is shared by runs from other sessions (for example embedded runs enqueue their global step on main in src/agents/pi-embedded-runner/run.ts). A FailoverError from one of those runs will incorrectly mark the main session idle and decrement its queue depth, while leaving the real session state unchanged, so fail-closed diagnostics point at the wrong session.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2a826ce4ba
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| } catch (err) { | ||
| const message = String(err); | ||
| this.providerUnavailableReason = message; | ||
| log.warn(`memory search degrading to keyword-only after query embedding failure: ${message}`); | ||
| } |
There was a problem hiding this comment.
Preserve keyword fallback when embedding lookup throws
When embedQueryWithTimeout fails, this branch swallows the exception and leaves queryVec empty, but the downstream if (!hybrid.enabled || !this.fts.enabled || !this.fts.available) path still returns vector-only results. In configurations with query.hybrid.enabled = false (vector-first mode), a transient embedding outage now produces silent empty searches instead of either falling back to keyword results or surfacing an error, which can hide relevant memory hits during provider incidents.
Useful? React with 👍 / 👎.
| if (this.sessionWatchTimer) { | ||
| return; | ||
| } |
There was a problem hiding this comment.
Enforce retry backoff even with pending debounce timer
The fetch-failure retry path exits early whenever sessionWatchTimer already exists, but that same timer is also used for normal 5s session-delta debounce. Under active transcript traffic during a fetch outage, new updates keep this timer populated, so the 60s retry delay is skipped and retries continue at debounce cadence, causing repeated failed sync attempts and log churn instead of true backoff.
Useful? React with 👍 / 👎.
|
Codex review: needs real behavior proof before merge. Reviewed June 5, 2026, 1:01 AM ET / 05:01 UTC. Summary PR surface: Source +245, Tests +339. Total +584 across 17 files. Reproducibility: unclear. The review failed before ClawSweeper could establish a reproduction path. Review metrics: none identified. Merge readiness Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch. Risk before merge
Maintainer options:
Next step before merge
Review detailsBest possible solution: Retry the Codex review after fixing the execution failure. Do we have a high-confidence way to reproduce the issue? Unclear. The review failed before ClawSweeper could establish a reproduction path. Is this the best way to solve the issue? Unclear. Retry the review first so ClawSweeper can evaluate the actual issue and fix direction. AGENTS.md: unclear because the file could not be read completely. Codex review notes: model gpt-5.5, reasoning high; reviewed against e0018382eb00. Label changesLabel changes:
Label justifications:
Evidence reviewedPR surface: Source +245, Tests +339. Total +584 across 17 files. View PR surface stats
What I checked:
Likely related people:
What the crustacean ranks mean
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics. How this review workflow works
|
|
ClawSweeper PR egg 🎁 Pass real behavior proof to wake the egg and unlock a hatchable treat. Where did the egg go?
|
|
This pull request has been automatically marked as stale due to inactivity. |
Summary
Why
Discord and main-session turns were freezing after control-plane events like
update_plan, leaving sessions stuck inprocessingwith no visible reply. Separately, memory session-delta sync was repeatedly loggingfetch failedwithout preserving pending work for retry.Verification