feat(cron): surface fallback progress#87343
Conversation
|
Codex review: needs changes before merge. Reviewed June 24, 2026, 4:07 PM ET / 20:07 UTC. Summary PR surface: Source +161, Tests +491, Docs +165, Other +172. Total +989 across 27 files. Reproducibility: yes. from source inspection: if Gateway drain begins before the first runEmbeddedAgent session/global enqueue, runWithModelFallback withholds the continuation option for i === 0 and enqueueCommandInLane rejects. I did not run tests because this review was read-only. Review metrics: 1 noteworthy metric.
Stored data model Merge readiness Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch. Rank-up moves:
Risk before merge
Maintainer options:
Copy recommended automerge instructionNext step before merge
Security Review findings
Review detailsBest possible solution: Land only after continuation covers the first already-admitted attempt as well as later fallback/live-switch attempts, ordinary new enqueues still reject during drain, cron task progress remains sanitized, and the branch is refreshed against current main. Do we have a high-confidence way to reproduce the issue? Yes, from source inspection: if Gateway drain begins before the first runEmbeddedAgent session/global enqueue, runWithModelFallback withholds the continuation option for i === 0 and enqueueCommandInLane rejects. I did not run tests because this review was read-only. Is this the best way to solve the issue? No; the PR is close, but the continuation guard is too narrow for the admitted-run boundary it is trying to protect. The maintainable fix is to pass continuation for the first already-admitted attempt, or deliberately abort on that drain error, while preserving fail-closed behavior for unrelated new work. Full review comments:
Overall correctness: patch is incorrect AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning high; reviewed against f0f5da0e3941. Label changesLabel justifications:
Evidence reviewedPR surface: Source +161, Tests +491, Docs +165, Other +172. Total +989 across 27 files. View PR surface stats
Acceptance criteria:
What I checked:
Likely related people:
What the crustacean ranks mean
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics. How this review workflow works
|
|
ClawSweeper PR egg 🎁 Pass real behavior proof to wake the egg and unlock a hatchable treat. Where did the egg go?
|
This comment was marked as spam.
This comment was marked as spam.
84fbdc2 to
2aa4dd2
Compare
2aa4dd2 to
af7757d
Compare
|
@clawsweeper re-review Rebased and proof updated.
Please re-run the review against the new head. |
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: af7757dd8a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const detail = params.step.fallbackStepFromFailureDetail | ||
| ? ` detail=${params.step.fallbackStepFromFailureDetail}` | ||
| : ""; | ||
| return `model fallback: ${params.step.fallbackStepFromModel}${target}${reason} outcome=${params.step.fallbackStepFinalOutcome}${detail}`; |
There was a problem hiding this comment.
Sanitize fallback details before task progress
When a fallback chain succeeds after a prior provider failure, the success step can carry fallbackStepFromFailureDetail from lastPreviousAttempt.error, which comes from the raw describeFailoverError().rawError path rather than the redacted observation preview used for failed steps. Interpolating it here persists that raw provider error into the cron task progressSummary, which is exposed by openclaw tasks show --json and Gateway task APIs; provider errors can include request payloads, tokens, or local paths. Please redact/truncate the detail before writing it to task progress.
Useful? React with 👍 / 👎.
af7757d to
7cfea05
Compare
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
1ca0923 to
a8088d4
Compare
a8088d4 to
b624936
Compare
|
@clawsweeper re-review Updated after rebasing onto the latest main and fixing the ClawSweeper P2 sanitizer finding.
|
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b6249362d3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const attemptRunOptions: ModelFallbackRunOptions | undefined = | ||
| params.allowGatewayDrainingContinuation === true && i > 0 && attempts.length > 0 | ||
| ? { ...runOptions, allowGatewayDrainingContinuation: true } | ||
| : runOptions; |
There was a problem hiding this comment.
Permit drain continuation after live model switch redirects
When an already-active embedded/cron attempt hits a pending live model switch after markGatewayDraining(), the LiveSessionModelSwitchError branch below redirects by setting i and continue without appending to attempts, so this new guard leaves allowGatewayDrainingContinuation unset for the redirected candidate. That candidate then re-enters runEmbeddedAgent, whose session/global enqueues reject during drain unless the flag is present, so this still fails with GatewayDrainingError for live-switch redirects even though the logical turn was already admitted.
Useful? React with 👍 / 👎.
|
@clawsweeper re-review Updated after rebasing again onto latest
|
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
|
@clawsweeper re-review Updated after fixing the latest P2 live-switch redirect continuation gap.
|
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a894a559da
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| params.allowGatewayDrainingContinuation === true && | ||
| i > 0 && | ||
| (attempts.length > 0 || liveSwitchRedirectContinuationPending); |
There was a problem hiding this comment.
Preserve continuation on the first admitted attempt
When a cron/manual run is already executing and gateway drain flips before the first embedded attempt reaches runEmbeddedAgent's session/global queue enqueues, this guard withholds allowGatewayDrainingContinuation for i === 0. enqueueCommandInLane then rejects the primary with GatewayDrainingError; runWithModelFallback treats that local drain error as a failed primary candidate and either skips straight to a fallback model or fails when no fallback is configured, even though no provider was tried. Pass the continuation option to the first attempt for already-admitted runs, or abort on GatewayDrainingError, so restart drain does not change the model chain.
Useful? React with 👍 / 👎.
|
This pull request has been automatically marked as stale due to inactivity. |
|
ClawSweeper status: review started. I am starting a fresh review of this pull request: feat(cron): surface fallback progress This is item 1/1 in the current shard. Shard 3/25. This placeholder means the worker is alive and reading the current context. I will edit this same comment with the actual review when the claws are done clicking. Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted. |
|
Closing due to inactivity. |
Summary
LiveSessionModelSwitchErrorredirects so later embedded fallback candidates stay admitted after Gateway drain starts.cron task ledgersandcommand lane snapshots).upstream/mainand keep the inherited migration selection compatibility alias explicitly deprecated so architecture/deprecated-JSDoc passes.Verification
node scripts/run-vitest.mjs src/agents/model-fallback.test.ts src/process/command-queue.test.ts src/cron/isolated-agent/run-executor.task-progress.test.ts src/agents/model-fallback.run-embedded.e2e.test.ts(4 shards, 133 tests)pnpm exec tsx scripts/repro/cron-fallback-progress-proof.tsnode scripts/run-tsgo.mjs -p test/tsconfig/tsconfig.core.test.json --incremental --tsBuildInfoFile .artifacts/tsgo-cache/core-test.tsbuildinfopnpm lint --threads=8pnpm check:architecturenode scripts/docs-list.js./node_modules/.bin/oxfmt --check --threads=1 src/agents/model-fallback.ts src/agents/model-fallback.run-embedded.e2e.test.tsgit diff --checkgit diff --check HEAD~1..HEADReal behavior proof
LiveSessionModelSwitchErrorredirects an already-started fallback chain to a later candidate after Gateway drain starts, the redirected embedded candidate now receivesallowGatewayDrainingContinuation=true. The focused embedded e2e covers this by marking drain in the primary attempt, throwing a live-switch redirect to groq, verifying groq succeeds, and verifying new work is still rejected during drain.0x9d ... 0x9c) is stripped bystripAnsi, and compact console sanitization removes the full C1 control range (0x80-0x9f) before cron progress is persisted. The cron progress regression injects C1 OSC in provider/model/fallback detail and asserts no C1/control bytes or C1 OSC payload remain./private/tmp/openclaw-pr-87343-rebase; PR heada894a559da2d5549282e28a08cf05fe7656d1a92; rebased ontoupstream/main@1a3ce7c2a8da266cd0bb2844d44ba5af3942908b; proof used isolated tempOPENCLAW_STATE_DIR=/var/folders/_x/34p90_ts62zcl8g4bvfkyjsw0000gn/T/openclaw-pr-87343-proof-6OmK7F.pnpm exec tsx scripts/repro/cron-fallback-progress-proof.ts.allowGatewayDrainingContinuation=true, reachedanthropic/claude-haiku-3-5, and returnedfallback ok. A new enqueue during drain was still rejected withGatewayDrainingError.