Skip to content

feat(cron): surface fallback progress#87343

Closed
riosbotchen-source wants to merge 1 commit into
openclaw:mainfrom
riosbotchen-source:codex/active-task-observability-hotfix-main
Closed

feat(cron): surface fallback progress#87343
riosbotchen-source wants to merge 1 commit into
openclaw:mainfrom
riosbotchen-source:codex/active-task-observability-hotfix-main

Conversation

@riosbotchen-source

@riosbotchen-source riosbotchen-source commented May 27, 2026

Copy link
Copy Markdown

Summary

  • Stop already-admitted model fallback chains from losing their embedded cron fallback candidate when Gateway drain begins; new queue enqueues still reject during drain.
  • Carry the drain-continuation state through LiveSessionModelSwitchError redirects so later embedded fallback candidates stay admitted after Gateway drain starts.
  • Add queue pressure fields to model fallback observations and surface cron primary/fallback progress into detached task run status.
  • Sanitize fallback failure detail and persisted cron progress fields before task status/CLI surfaces can read them, including ANSI, OSC, C0, C1, and DEL control-character coverage.
  • Strip 8-bit C1 OSC sequences in the shared ANSI helper and remove the full C1 control range in compact console sanitization.
  • Replace the undefined docs example with source-backed OpenClaw artifacts (cron task ledgers and command lane snapshots).
  • Rebase onto latest upstream/main and keep the inherited migration selection compatibility alias explicitly deprecated so architecture/deprecated-JSDoc passes.
  • Thread cron task run ids through manual/isolated execution, add a reproducible runtime proof script, and document the production AI governance checklist.

Verification

  • PASS: node scripts/run-vitest.mjs src/agents/model-fallback.test.ts src/process/command-queue.test.ts src/cron/isolated-agent/run-executor.task-progress.test.ts src/agents/model-fallback.run-embedded.e2e.test.ts (4 shards, 133 tests)
  • PASS: pnpm exec tsx scripts/repro/cron-fallback-progress-proof.ts
  • PASS: node scripts/run-tsgo.mjs -p test/tsconfig/tsconfig.core.test.json --incremental --tsBuildInfoFile .artifacts/tsgo-cache/core-test.tsbuildinfo
  • PASS: pnpm lint --threads=8
  • PASS: pnpm check:architecture
  • PASS: node scripts/docs-list.js
  • PASS: ./node_modules/.bin/oxfmt --check --threads=1 src/agents/model-fallback.ts src/agents/model-fallback.run-embedded.e2e.test.ts
  • PASS: git diff --check
  • PASS: git diff --check HEAD~1..HEAD

Real behavior proof

  • Behavior addressed: Already-active cron/background work can continue its configured model fallback chain after Gateway drain begins, including the embedded runner's session/global queue shape. New command enqueues still reject during drain. Cron-style task progress records primary/fallback model status plus queue active/queued/draining state, with fallback failure detail and terminal-control-bearing progress fields sanitized before persistence.
  • Redirect repair after review: if a LiveSessionModelSwitchError redirects an already-started fallback chain to a later candidate after Gateway drain starts, the redirected embedded candidate now receives allowGatewayDrainingContinuation=true. The focused embedded e2e covers this by marking drain in the primary attempt, throwing a live-switch redirect to groq, verifying groq succeeds, and verifying new work is still rejected during drain.
  • Sanitizer repair after review: C1 OSC (0x9d ... 0x9c) is stripped by stripAnsi, and compact console sanitization removes the full C1 control range (0x80-0x9f) before cron progress is persisted. The cron progress regression injects C1 OSC in provider/model/fallback detail and asserts no C1/control bytes or C1 OSC payload remain.
  • Real environment tested: Local macOS source checkout at /private/tmp/openclaw-pr-87343-rebase; PR head a894a559da2d5549282e28a08cf05fe7656d1a92; rebased onto upstream/main@1a3ce7c2a8da266cd0bb2844d44ba5af3942908b; proof used isolated temp OPENCLAW_STATE_DIR=/var/folders/_x/34p90_ts62zcl8g4bvfkyjsw0000gn/T/openclaw-pr-87343-proof-6OmK7F.
  • Exact steps or command run after this patch: pnpm exec tsx scripts/repro/cron-fallback-progress-proof.ts.
  • Evidence after fix:
[proof] stateDir=/var/folders/_x/34p90_ts62zcl8g4bvfkyjsw0000gn/T/openclaw-pr-87343-proof-6OmK7F
[proof] task created runtime=cron runId=pr-87343-cron-proof
[proof] initial progressSummary=model primary=openai/gpt-4.1-mini; queue active=0 queued=0 draining=no
[proof] active queue before fallback={"lane":"main","queuedCount":0,"activeCount":1,"maxConcurrent":1,"draining":false,"generation":0}
[proof] attempt provider=openai model=gpt-4.1-mini allowGatewayDrainingContinuation=false
[proof] gateway drain marked while embedded queue work is active
[proof] fallback step={"fallbackStepType":"fallback_step","fallbackStepFromModel":"openai/gpt-4.1-mini","fallbackStepToModel":"anthropic/claude-haiku-3-5","fallbackStepFromFailureReason":"rate_limit","fallbackStepFromFailureDetail":"primary rate limited","fallbackStepChainPosition":1,"fallbackStepFinalOutcome":"next_fallback","fallbackStepQueueActive":1,"fallbackStepQueueQueued":0,"fallbackStepQueueDraining":true}
[proof] recorded progressSummary=model fallback: openai/gpt-4.1-mini -> anthropic/claude-haiku-3-5 reason=rate_limit outcome=next_fallback detail=primary rate limited; queue active=1 queued=0 draining=yes
[proof] attempt provider=anthropic model=claude-haiku-3-5 allowGatewayDrainingContinuation=true
[proof] fallback step={"fallbackStepType":"fallback_step","fallbackStepFromModel":"openai/gpt-4.1-mini","fallbackStepToModel":"anthropic/claude-haiku-3-5","fallbackStepFromFailureReason":"rate_limit","fallbackStepFromFailureDetail":"primary rate limited","fallbackStepChainPosition":2,"fallbackStepFinalOutcome":"succeeded","fallbackStepQueueActive":1,"fallbackStepQueueQueued":0,"fallbackStepQueueDraining":true}
[proof] recorded progressSummary=model fallback: openai/gpt-4.1-mini -> anthropic/claude-haiku-3-5 reason=rate_limit outcome=succeeded detail=primary rate limited; queue active=1 queued=0 draining=yes
[proof] active task result=anthropic/claude-haiku-3-5:fallback ok
[proof] new enqueue rejected with GatewayDrainingError
[proof] final task progressSummary=model fallback: openai/gpt-4.1-mini -> anthropic/claude-haiku-3-5 reason=rate_limit outcome=succeeded detail=primary rate limited; queue active=1 queued=0 draining=yes
[proof] PASS embedded queue-shaped fallback continued during drain; new enqueue still rejected
  • Observed result after fix: The primary embedded queue-shaped attempt marked Gateway drain while active; the fallback attempt then ran with allowGatewayDrainingContinuation=true, reached anthropic/claude-haiku-3-5, and returned fallback ok. A new enqueue during drain was still rejected with GatewayDrainingError.
  • What was not tested: A live external-provider cron run was not executed; the proof script uses production model fallback, command queue, and detached task runtime APIs with a deterministic local provider callback. GitHub CI and ClawSweeper re-review are pending on the pushed head.

@openclaw-barnacle openclaw-barnacle Bot added docs Improvements or additions to documentation gateway Gateway runtime agents Agent runtime and tooling size: M triage: mock-only-proof Candidate: PR proof only shows tests, mocks, snapshots, lint, typecheck, or CI. labels May 27, 2026
@clawsweeper

clawsweeper Bot commented May 27, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs changes before merge. Reviewed June 24, 2026, 4:07 PM ET / 20:07 UTC.

Summary
The PR adds cron model fallback progress reporting, queue-pressure observations, a Gateway-drain continuation flag through fallback and embedded runners, terminal sanitization, focused tests, a proof script, and related docs.

PR surface: Source +161, Tests +491, Docs +165, Other +172. Total +989 across 27 files.

Reproducibility: yes. from source inspection: if Gateway drain begins before the first runEmbeddedAgent session/global enqueue, runWithModelFallback withholds the continuation option for i === 0 and enqueueCommandInLane rejects. I did not run tests because this review was read-only.

Review metrics: 1 noteworthy metric.

  • Queue-drain exception surface: 1 internal enqueue option added. The new option changes which work can enter command lanes while Gateway drain is active, so availability behavior needs focused review before merge.

Stored data model
Persistent data-model change detected: migration/backfill/repair: docs/help/testing.md, persistent cache schema: docs/help/testing.md, serialized state: src/cron/service/state.ts. Confirm migration or upgrade compatibility proof before merge.

Merge readiness
Overall: 🦐 gold shrimp
Proof: 🦞 diamond lobster
Patch quality: 🦐 gold shrimp
Result: needs maintainer review before merge.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • [P2] Fix the first-attempt drain-continuation edge and add the pre-first-enqueue regression.
  • Rebase or otherwise resolve the current merge conflict against main.

Risk before merge

  • [P1] A restart drain that starts before the first embedded session/global enqueue can still make an already-admitted cron/manual run skip or fail its intended primary attempt before any provider is tried.
  • [P1] GitHub currently reports the PR branch as conflicting with main, so maintainers need a rebase or merge-result validation before trusting final CI and review.

Maintainer options:

  1. Fix admitted-run continuation (recommended)
    Carry the continuation option into the first already-admitted cron/manual embedded attempt, or abort on GatewayDrainingError, then add the pre-first-enqueue drain regression.
  2. Refresh merge result
    Rebase or otherwise resolve the current GitHub conflict so CI and review cover the actual merge result.
Copy recommended automerge instruction
@clawsweeper automerge

Special instructions:
Fix the first-attempt Gateway-drain continuation edge for already-admitted cron/manual embedded runs, keep ordinary new enqueues rejected during drain, add focused regression coverage for drain before the first embedded enqueue, and refresh the branch against current main.

Next step before merge

  • [P2] A narrow mechanical repair can adjust the continuation guard, add one focused regression, and refresh mergeability without choosing a new product direction.

Security
Cleared: The diff does not add dependencies, workflow execution, secret handling, or new third-party code execution; the terminal-control handling changes are defensive sanitization.

Review findings

  • [P2] Preserve continuation for the first admitted attempt — src/agents/model-fallback.ts:1517-1518
Review details

Best possible solution:

Land only after continuation covers the first already-admitted attempt as well as later fallback/live-switch attempts, ordinary new enqueues still reject during drain, cron task progress remains sanitized, and the branch is refreshed against current main.

Do we have a high-confidence way to reproduce the issue?

Yes, from source inspection: if Gateway drain begins before the first runEmbeddedAgent session/global enqueue, runWithModelFallback withholds the continuation option for i === 0 and enqueueCommandInLane rejects. I did not run tests because this review was read-only.

Is this the best way to solve the issue?

No; the PR is close, but the continuation guard is too narrow for the admitted-run boundary it is trying to protect. The maintainable fix is to pass continuation for the first already-admitted attempt, or deliberately abort on that drain error, while preserving fail-closed behavior for unrelated new work.

Full review comments:

  • [P2] Preserve continuation for the first admitted attempt — src/agents/model-fallback.ts:1517-1518
    When an already-admitted cron/manual run opts into drain continuation, this guard still withholds allowGatewayDrainingContinuation from the primary embedded attempt because i > 0 is required. If Gateway drain flips before that first runEmbeddedAgent session/global enqueue, enqueueCommandInLane rejects and the fallback loop can skip or fail the primary without trying the configured model. Pass continuation for the first admitted attempt, or abort on that drain error, and cover the pre-first-enqueue drain case.
    Confidence: 0.9

Overall correctness: patch is incorrect
Overall confidence: 0.86

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against f0f5da0e3941.

Label changes

Label justifications:

  • P2: This is a normal-priority cron/agent availability and observability improvement with a limited but real restart-drain edge still open.
  • merge-risk: 🚨 availability: The PR changes queue admission during Gateway drain, and the current guard can still fail already-admitted work in a shutdown race.
  • rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦞 diamond lobster and patch quality is 🦐 gold shrimp.
  • status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Sufficient (terminal): The PR body includes copied terminal output from running the proof script in a local source checkout against production fallback, queue, and task APIs; that proof is sufficient for the covered after-fix behavior, while the separate pre-first-enqueue edge remains a code finding.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR body includes copied terminal output from running the proof script in a local source checkout against production fallback, queue, and task APIs; that proof is sufficient for the covered after-fix behavior, while the separate pre-first-enqueue edge remains a code finding.
Evidence reviewed

PR surface:

Source +161, Tests +491, Docs +165, Other +172. Total +989 across 27 files.

View PR surface stats
Area Files Added Removed Net
Source 17 177 16 +161
Tests 4 496 5 +491
Docs 5 166 1 +165
Config 0 0 0 0
Generated 0 0 0 0
Other 1 172 0 +172
Total 27 1011 22 +989

Acceptance criteria:

  • [P2] node scripts/run-vitest.mjs src/agents/model-fallback.test.ts src/process/command-queue.test.ts src/cron/isolated-agent/run-executor.task-progress.test.ts src/agents/model-fallback.run-embedded.e2e.test.ts.
  • [P2] pnpm exec tsx scripts/repro/cron-fallback-progress-proof.ts.
  • [P1] node scripts/run-tsgo.mjs -p test/tsconfig/tsconfig.core.test.json --incremental --tsBuildInfoFile .artifacts/tsgo-cache/core-test.tsbuildinfo.
  • [P1] pnpm lint --threads=8.
  • [P1] pnpm check:architecture.

What I checked:

  • PR guard still withholds continuation from first candidate: The PR only sets per-attempt allowGatewayDrainingContinuation when params.allowGatewayDrainingContinuation is true, i > 0, and an earlier attempt or live-switch redirect exists, so the first embedded attempt cannot receive the drain continuation option. (src/agents/model-fallback.ts:1515, a894a559da2d)
  • Cron opts the logical run into continuation: The cron executor sets allowGatewayDrainingContinuation on runWithModelFallback, but runEmbeddedAgent only receives the flag from per-attempt runOptions, which the current guard omits for i === 0. (src/cron/isolated-agent/run-executor.ts:299, a894a559da2d)
  • Embedded queue admission depends on the propagated flag: runEmbeddedAgent copies allowGatewayDrainingContinuation into both session and global lane enqueue options, so a missing flag still reaches the normal Gateway drain rejection path. (src/agents/embedded-agent-runner/run.ts:502, a894a559da2d)
  • Current main rejects new enqueues during drain: Current main rejects enqueueCommandInLane whenever gatewayDraining is true, which makes the PR's new exception the only way to admit selected continuation work during drain. (src/process/command-queue.ts:481, f0f5da0e3941)
  • Existing PR coverage misses the pre-first-enqueue race: The added embedded tests mark drain inside the first embedded attempt and cover live-switch redirect, but they do not mark drain before the first embedded session/global enqueue. (src/agents/model-fallback.run-embedded.e2e.test.ts:499, a894a559da2d)
  • Live GitHub mergeability check: GitHub reports head a894a55 as mergeable CONFLICTING with mergeStateStatus DIRTY against base 12a5691. (a894a559da2d)

Likely related people:

  • steipete: History shows substantial prior work on model fallback, command queue drain handling, cron scheduling, and isolated cron runner structure. (role: major feature/history contributor; confidence: high; commits: 734bb6b4fd, c397a02c9a, f9409cbe43; files: src/agents/model-fallback.ts, src/process/command-queue.ts, src/cron/service/timer.ts)
  • vincentkoc: Recent main history touches model fallback performance seams, isolated cron lazy-loading seams, and command queue/logging runtime areas that this PR modifies. (role: recent area contributor; confidence: high; commits: 93ce76afe3, 21d850dd66, 28787985c4; files: src/agents/model-fallback.ts, src/cron/isolated-agent/run-executor.ts, src/process/command-queue.ts)
  • Altay: History ties this person to adding fallback error observations, which this PR extends with queue-pressure fields and cron progress projection. (role: adjacent fallback-observation contributor; confidence: medium; commits: 531e8362b1, 0669b0ddc2, 6e962d8b9e; files: src/agents/model-fallback.ts, src/agents/model-fallback-observation.ts)
  • Han Yang: History shows prior live-session model-switch failover work, adjacent to the PR's redirect-continuation path. (role: adjacent live-switch behavior contributor; confidence: medium; commits: 547154865b; files: src/agents/model-fallback.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P2 Normal backlog priority with limited blast radius. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 availability 🚨 May cause crashes, hangs, restart loops, stalls, or process outages. labels May 27, 2026
@clawsweeper

clawsweeper Bot commented May 27, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper PR egg

🎁 Pass real behavior proof to wake the egg and unlock a hatchable treat.

Where did the egg go?
  • The egg game starts only after the PR passes the real-behavior proof check.
  • Before that, no creature or rarity is rolled. The treat waits for real proof.
  • This is still just collectible flavor: proof affects review readiness, not creature quality.

@BingqingLyu

This comment was marked as spam.

@clawsweeper clawsweeper Bot added rating: 🌊 off-meta tidepool PR readiness rating does not apply to this item. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels May 29, 2026
@riosbotchen-source
riosbotchen-source force-pushed the codex/active-task-observability-hotfix-main branch from 84fbdc2 to 2aa4dd2 Compare June 4, 2026 15:13
@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. and removed rating: 🌊 off-meta tidepool PR readiness rating does not apply to this item. labels Jun 4, 2026
@riosbotchen-source
riosbotchen-source force-pushed the codex/active-task-observability-hotfix-main branch from 2aa4dd2 to af7757d Compare June 4, 2026 15:24
@riosbotchen-source
riosbotchen-source marked this pull request as ready for review June 4, 2026 15:52

Copy link
Copy Markdown
Author

@clawsweeper re-review

Rebased and proof updated.

  • PR head: af7757dd8a41f54a70013b2d706ad2d16b4febb1
  • Rebase base used for local proof: upstream/main@7385c611fcf3e696d2d7bb171905ac33522aad8f
  • Local proof passed: pnpm install --frozen-lockfile; targeted run-vitest (3 shards, 111 tests); core-test run-tsgo; targeted oxfmt --check; git diff --check HEAD~1..HEAD; node scripts/docs-list.js
  • GitHub Workflow Sanity completed successfully on this head; CI/CodeQL/OpenGrep were skipped by workflow rules.

Please re-run the review against the new head.

@clawsweeper

clawsweeper Bot commented Jun 4, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: af7757dd8a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/cron/isolated-agent/run-executor.ts Outdated
Comment on lines +137 to +140
const detail = params.step.fallbackStepFromFailureDetail
? ` detail=${params.step.fallbackStepFromFailureDetail}`
: "";
return `model fallback: ${params.step.fallbackStepFromModel}${target}${reason} outcome=${params.step.fallbackStepFinalOutcome}${detail}`;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Sanitize fallback details before task progress

When a fallback chain succeeds after a prior provider failure, the success step can carry fallbackStepFromFailureDetail from lastPreviousAttempt.error, which comes from the raw describeFailoverError().rawError path rather than the redacted observation preview used for failed steps. Interpolating it here persists that raw provider error into the cron task progressSummary, which is exposed by openclaw tasks show --json and Gateway task APIs; provider errors can include request payloads, tokens, or local paths. Please redact/truncate the detail before writing it to task progress.

Useful? React with 👍 / 👎.

@riosbotchen-source
riosbotchen-source force-pushed the codex/active-task-observability-hotfix-main branch from af7757d to 7cfea05 Compare June 4, 2026 16:08
@openclaw-barnacle openclaw-barnacle Bot added scripts Repository scripts and removed size: M labels Jun 4, 2026
@clawsweeper

clawsweeper Bot commented Jun 4, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

@openclaw-barnacle openclaw-barnacle Bot added proof: supplied External PR includes structured after-fix real behavior proof. and removed triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels Jun 4, 2026
@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jun 4, 2026
@riosbotchen-source
riosbotchen-source force-pushed the codex/active-task-observability-hotfix-main branch from 1ca0923 to a8088d4 Compare June 4, 2026 17:29
@openclaw-barnacle openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 4, 2026
@riosbotchen-source
riosbotchen-source force-pushed the codex/active-task-observability-hotfix-main branch from a8088d4 to b624936 Compare June 4, 2026 17:33
@riosbotchen-source

Copy link
Copy Markdown
Author

@clawsweeper re-review

Updated after rebasing onto the latest main and fixing the ClawSweeper P2 sanitizer finding.

  • PR head: b6249362d3eb0afecec24f011c44c2eb4b342a59
  • Base used locally: upstream/main@fbac4a2ec77c84fb8c69f6498e6b1a6dec589c2d
  • Fixed C1 OSC / full C1 terminal-control handling before cron progress persistence: stripAnsi now strips 8-bit OSC (0x9d ... 0x9c), and sanitizeForConsole removes 0x80-0x9f.
  • Added regression coverage with C1 OSC in provider/model/fallback progress detail.
  • Re-ran: targeted Vitest shards, proof script, tsgo, lint, architecture check, docs-list, oxfmt, and committed diff whitespace check.

@clawsweeper

clawsweeper Bot commented Jun 4, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b6249362d3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +1514 to +1517
const attemptRunOptions: ModelFallbackRunOptions | undefined =
params.allowGatewayDrainingContinuation === true && i > 0 && attempts.length > 0
? { ...runOptions, allowGatewayDrainingContinuation: true }
: runOptions;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Permit drain continuation after live model switch redirects

When an already-active embedded/cron attempt hits a pending live model switch after markGatewayDraining(), the LiveSessionModelSwitchError branch below redirects by setting i and continue without appending to attempts, so this new guard leaves allowGatewayDrainingContinuation unset for the redirected candidate. That candidate then re-enters runEmbeddedAgent, whose session/global enqueues reject during drain unless the flag is present, so this still fails with GatewayDrainingError for live-switch redirects even though the logical turn was already admitted.

Useful? React with 👍 / 👎.

@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. labels Jun 4, 2026
@riosbotchen-source

Copy link
Copy Markdown
Author

@clawsweeper re-review

Updated after rebasing again onto latest upstream/main to clear the dirty/merge-conflict state.

  • PR head: d01deef094aaaa9ab2786df117ea98b213f667ac
  • Base used locally: upstream/main@1a3ce7c2a8da266cd0bb2844d44ba5af3942908b
  • Rebase conflict was limited to src/gateway/server-cron.ts; resolved by keeping current main module comment plus the PR cron task-run/progress wiring.
  • Re-ran: targeted Vitest shards, proof script, tsgo, lint, architecture check, docs-list, oxfmt, and committed diff whitespace check.

@clawsweeper

clawsweeper Bot commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

@riosbotchen-source

Copy link
Copy Markdown
Author

@clawsweeper re-review

Updated after fixing the latest P2 live-switch redirect continuation gap.

  • PR head: a894a559da2d5549282e28a08cf05fe7656d1a92
  • Base used locally: upstream/main@1a3ce7c2a8da266cd0bb2844d44ba5af3942908b
  • Carry allowGatewayDrainingContinuation through later-candidate LiveSessionModelSwitchError redirects, even when attempts is still empty.
  • Added embedded e2e coverage where the primary attempt marks Gateway drain, throws a live-switch redirect to groq, the redirected candidate succeeds, and new queue work remains rejected.
  • Re-ran: targeted Vitest shards, proof script, tsgo, lint, architecture check, docs-list, oxfmt, and diff whitespace checks.

@clawsweeper

clawsweeper Bot commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a894a559da

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +1516 to +1518
params.allowGatewayDrainingContinuation === true &&
i > 0 &&
(attempts.length > 0 || liveSwitchRedirectContinuationPending);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve continuation on the first admitted attempt

When a cron/manual run is already executing and gateway drain flips before the first embedded attempt reaches runEmbeddedAgent's session/global queue enqueues, this guard withholds allowGatewayDrainingContinuation for i === 0. enqueueCommandInLane then rejects the primary with GatewayDrainingError; runWithModelFallback treats that local drain error as a failed primary candidate and either skips straight to a fallback model or fails when no fallback is configured, even though no provider was tried. Pass the continuation option to the first attempt for already-admitted runs, or abort on GatewayDrainingError, so restart drain does not change the model chain.

Useful? React with 👍 / 👎.

@openclaw-barnacle

Copy link
Copy Markdown

This pull request has been automatically marked as stale due to inactivity.
Please add updates or it will be closed.

@clawsweeper

clawsweeper Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper status: review started.

I am starting a fresh review of this pull request: feat(cron): surface fallback progress This is item 1/1 in the current shard. Shard 3/25.

This placeholder means the worker is alive and reading the current context. I will edit this same comment with the actual review when the claws are done clicking.

Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted.

@openclaw-barnacle

Copy link
Copy Markdown

Closing due to inactivity.
If you believe this PR should be revived, post in #clawtributors on Discord to talk to a maintainer.
That channel is the escape hatch for high-quality PRs that get auto-closed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling commands Command implementations docs Improvements or additions to documentation gateway Gateway runtime merge-risk: 🚨 availability 🚨 May cause crashes, hangs, restart loops, stalls, or process outages. P2 Normal backlog priority with limited blast radius. proof: sufficient ClawSweeper judged the real behavior proof convincing. proof: supplied External PR includes structured after-fix real behavior proof. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. scripts Repository scripts size: L stale Marked as stale due to inactivity status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants