Skip to content

fix: retry transient recurring cron failures#87871

Merged
steipete merged 2 commits into
mainfrom
codex/retry-transient-cron-failover
May 29, 2026
Merged

fix: retry transient recurring cron failures#87871
steipete merged 2 commits into
mainfrom
codex/retry-transient-cron-failover

Conversation

@steipete

Copy link
Copy Markdown
Contributor

Summary

  • retry recurring cron jobs on transient model errors before waiting for the next scheduled slot
  • share transient retry decision logic between one-shot and recurring cron jobs
  • keep recurring cadence anchored after retry success or retry exhaustion
  • document cron retry behavior and add regression coverage

Proof

  • node scripts/run-vitest.mjs run --config test/vitest/vitest.cron.config.ts src/cron/schedule-identity.test.ts src/cron/service/timer.regression.test.ts
  • node scripts/run-vitest.mjs run --config test/vitest/vitest.tui.config.ts src/tui/gateway-chat.test.ts
  • pnpm -s test:changed
  • env OPENCLAW_TESTBOX=0 pnpm -s check:changed
  • .agents/skills/autoreview/scripts/autoreview --mode local --base origin/main

@openclaw-barnacle openclaw-barnacle Bot added docs Improvements or additions to documentation gateway Gateway runtime size: M maintainer Maintainer-authored PR labels May 29, 2026
@clawsweeper

clawsweeper Bot commented May 29, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs real behavior proof before merge. Reviewed May 28, 2026, 11:49 PM ET / 03:49 UTC.

Summary
The PR adds shared transient retry logic so recurring cron jobs can retry before their next scheduled slot, updates cron retry docs/changelog text, and adds regression coverage.

PR surface: Source +95, Tests +178, Docs +6, Other -3. Total +276 across 6 files.

Reproducibility: yes. source-reproducible: current main applies transient retry policy only to one-shot jobs while recurring errors use normal error backoff. I did not run tests because this was a read-only review.

Review metrics: 1 noteworthy metric.

  • Cron retry config semantics: 1 existing config surface broadened. cron.retry changes from one-shot-only current behavior/docs to recurring retry behavior, so upgrade and operator expectations need explicit review.

Merge readiness
Overall: 🧂 unranked krab
Proof: 🧂 unranked krab
Patch quality: 🦐 gold shrimp
Result: blocked until real behavior proof from a real setup is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • Remove the CHANGELOG.md Unreleased entry and keep release-note context in the PR body or squash message.
  • Update src/config/schema.help.ts and src/config/types.cron.ts so cron.retry help matches recurring retry behavior.
  • [P1] Add redacted real behavior proof showing a recurring cron job retrying after a transient error in a real setup.

Proof guidance:

  • [P1] Needs real behavior proof before merge: The PR body lists tests and checks only; before merge, add redacted live output, logs, terminal screenshot, recording, or artifact showing a recurring cron transient retry after the patch, then update the PR body to trigger review or ask a maintainer to comment @clawsweeper re-review.

Risk before merge

  • [P1] Existing cron.retry settings would begin affecting recurring jobs after upgrade, which can change retry cadence and model-call behavior for existing recurring automations.
  • [P1] The PR currently has only test/check proof, so maintainers do not yet have real post-patch cron behavior evidence for the changed runtime path.

Maintainer options:

  1. Align the contract before merge (recommended)
    Remove the release-owned changelog edit and update schema help/type comments so every user-facing cron.retry surface describes the recurring retry behavior.
  2. Accept the broader retry semantics
    Maintainers can intentionally accept that existing cron.retry settings now control recurring jobs too once the proof and help text make that upgrade behavior explicit.
  3. Pause for an opt-in design
    If recurring retries should not become the default for existing cron.retry users, pause this branch and design a separate opt-in policy before changing runtime behavior.

Next step before merge

  • [P1] Human handling is needed because the PR has a protected maintainer label, a compatibility-sensitive retry semantics change, blocking review findings, and missing real behavior proof.

Security
Cleared: No concrete security or supply-chain concern found; the diff does not change dependencies, lockfiles, workflows, permissions, secrets handling, or downloaded code execution.

Review findings

  • [P2] Remove the hand-written changelog entry — CHANGELOG.md:5-9
  • [P2] Align cron retry help text with recurring retries — docs/gateway/configuration-reference.md:1275
Review details

Best possible solution:

Land only after removing the changelog edit, aligning schema help/type docs with the recurring retry contract, and adding redacted real behavior proof or a maintainer proof override for the cron retry path.

Do we have a high-confidence way to reproduce the issue?

Yes, source-reproducible: current main applies transient retry policy only to one-shot jobs while recurring errors use normal error backoff. I did not run tests because this was a read-only review.

Is this the best way to solve the issue?

No, not merge-ready as-is: the runtime approach is plausible, but the release-owned changelog edit, stale config help/type docs, and missing real behavior proof need resolution first.

Full review comments:

  • [P2] Remove the hand-written changelog entry — CHANGELOG.md:5-9
    This normal bug-fix PR adds an Unreleased block, but the repo rules say CHANGELOG.md is release-owned and normal PR fixes should carry release-note context in the PR body, squash message, or direct commit context. Leaving this entry mixes hand-written content into generated release notes, so remove the CHANGELOG.md change.
    Confidence: 0.95
  • [P2] Align cron retry help text with recurring retries — docs/gateway/configuration-reference.md:1275
    This doc change broadens cron.retry to recurring jobs, but src/config/schema.help.ts and src/config/types.cron.ts still describe the same config as one-shot-only. Users consulting config help or exported config types would get the old contract, so update those surfaces with the recurring retry semantics before merge.
    Confidence: 0.88

Overall correctness: patch is incorrect
Overall confidence: 0.88

AGENTS.md: found and applied where relevant.

Codex review notes: model gpt-5.5, reasoning high; reviewed against e7fb8cabb681.

Label changes

Label changes:

  • add P2: This is a bounded cron reliability improvement with blocking review items but limited blast radius.
  • add merge-risk: 🚨 compatibility: Merging would make existing cron.retry settings affect recurring jobs, changing upgraded users' retry cadence and model-call behavior.
  • add rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🧂 unranked krab and patch quality is 🦐 gold shrimp.
  • add status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs real behavior proof before merge: The PR body lists tests and checks only; before merge, add redacted live output, logs, terminal screenshot, recording, or artifact showing a recurring cron transient retry after the patch, then update the PR body to trigger review or ask a maintainer to comment @clawsweeper re-review.

Label justifications:

  • P2: This is a bounded cron reliability improvement with blocking review items but limited blast radius.
  • merge-risk: 🚨 compatibility: Merging would make existing cron.retry settings affect recurring jobs, changing upgraded users' retry cadence and model-call behavior.
  • rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🧂 unranked krab and patch quality is 🦐 gold shrimp.
  • status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs real behavior proof before merge: The PR body lists tests and checks only; before merge, add redacted live output, logs, terminal screenshot, recording, or artifact showing a recurring cron transient retry after the patch, then update the PR body to trigger review or ask a maintainer to comment @clawsweeper re-review.
Evidence reviewed

PR surface:

Source +95, Tests +178, Docs +6, Other -3. Total +276 across 6 files.

View PR surface stats
Area Files Added Removed Net
Source 2 134 39 +95
Tests 1 180 2 +178
Docs 2 8 2 +6
Config 0 0 0 0
Generated 0 0 0 0
Other 1 1 4 -3
Total 6 323 47 +276

What I checked:

  • Repository policy read: The full root AGENTS.md was read; it marks CHANGELOG.md as release-owned and says normal fixes should put release-note context in the PR body, squash message, or commit instead. (AGENTS.md:192, e7fb8cabb681)
  • Scoped guides read: The relevant docs and scripts scoped AGENTS.md files were read; docs changes are governed by the docs guide and script changes by the scripts guide. Public docs: docs/AGENTS.md. (docs/AGENTS.md:1, e7fb8cabb681)
  • Current main behavior: Current main still documents cron retry as one-shot-only and its recurring error branch applies generic error backoff instead of the transient retry policy. (src/cron/service/timer.ts:1006, e7fb8cabb681)
  • PR runtime change: The PR adds a shared retry decision helper and uses it in the recurring job error branch to schedule a transient retry before the normal next scheduled run when the retry backoff lands first. (src/cron/service/timer.ts:1052, 89fbe0dc7239)
  • Regression coverage: The branch adds tests for recurring transient retry, exhausted recurring retries falling back to the normal schedule, and preserving every-cadence after retry success. (src/cron/service/timer.regression.test.ts:478, 89fbe0dc7239)
  • Changelog edit: The PR adds an Unreleased changelog entry despite the release-owned changelog policy; an existing Codex review comment also flagged this as a P2 issue. (CHANGELOG.md:5, 89fbe0dc7239)

Likely related people:

  • steipete: Recent current-main history shows Peter Steinberger carrying the cron timer/config documentation import and recent cron delivery ergonomics work; he also authored this PR branch. (role: recent area contributor; confidence: high; commits: d5bbf3033c9f, 2209faef405f, cd0bc5195389; files: src/cron/service/timer.ts, docs/gateway/configuration-reference.md, src/config/schema.help.ts)
  • samzong: Recent current-main history includes adjacent cron schedule identity work in a touched file, which may help review schedule identity side effects. (role: adjacent contributor; confidence: medium; commits: c237de552ac3; files: src/cron/schedule-identity.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

…-cron-failover

# Conflicts:
#	src/cron/schedule-identity.ts
#	src/tui/gateway-chat.test.ts
@openclaw-barnacle openclaw-barnacle Bot added the scripts Repository scripts label May 29, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cd0bc51953

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread CHANGELOG.md
Comment on lines +5 to +9
## Unreleased

### Fixes

- Cron: retry recurring jobs after transient model rate limits before waiting for the next scheduled slot.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Remove normal fix from the changelog

This is a normal bug-fix commit, but the repo rules in AGENTS.md say CHANGELOG.md is release-owned and should not be edited for normal PRs or direct fixes; release notes should instead go in the PR body/squash/direct commit context. Leaving this Unreleased entry makes release generation mix hand-written and generated changelog content.

Useful? React with 👍 / 👎.

@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P2 Normal backlog priority with limited blast radius. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. labels May 29, 2026
@steipete
steipete merged commit 00c9f81 into main May 29, 2026
205 of 215 checks passed
@steipete
steipete deleted the codex/retry-transient-cron-failover branch May 29, 2026 03:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Improvements or additions to documentation gateway Gateway runtime maintainer Maintainer-authored PR merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. P2 Normal backlog priority with limited blast radius. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. scripts Repository scripts size: M status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant