fix(cron): stop fallback attempts when cron budget is exhausted#52365
fix(cron): stop fallback attempts when cron budget is exhausted#52365imwyvern wants to merge 2 commits into
Conversation
Greptile SummaryThis PR fixes a real production bug where a cron job's fallback model was allocated ~0ms of budget when the primary model consumed most of the cron timeout, resulting in an immediate spurious failure attributed to the fallback. The fix propagates the outer cron deadline (
Confidence Score: 5/5
Prompt To Fix All With AIThis is a comment left during a code review.
Path: src/cron/service.issue-regressions.test.ts
Line: 1115-1119
Comment:
**Flaky test: exact `Date.now()` equality**
`wallNow` is captured before the async function call, while `deadlineAtMs` inside `executeJobCoreWithTimeout` is set to a second `Date.now()` invocation. If even 1 ms elapses between the two calls (very possible under CI load), the `.toBe()` assertion fails. A range check or fake timers would make this deterministic:
```suggestion
const wallNow = Date.now();
const result = await executeJobCoreWithTimeout(state, cronJob);
expect(result).toMatchObject({ status: "ok", summary: "done" });
// Allow a small window for the two Date.now() calls to differ
expect(observedDeadlineAtMs).toBeGreaterThanOrEqual(wallNow + timeoutSeconds * 1000);
expect(observedDeadlineAtMs).toBeLessThan(wallNow + timeoutSeconds * 1000 + 500);
```
Alternatively, mock `Date.now` with `vi.spyOn(Date, "now").mockReturnValue(wallNow)` to make both calls return the same value and keep the exact `.toBe` check.
How can I resolve this? If you propose a fix, please make it concise.Reviews (1): Last reviewed commit: "fix(cron): stop fallback attempts when c..." | Re-trigger Greptile |
| const wallNow = Date.now(); | ||
| const result = await executeJobCoreWithTimeout(state, cronJob); | ||
|
|
||
| expect(result).toMatchObject({ status: "ok", summary: "done" }); | ||
| expect(observedDeadlineAtMs).toBe(wallNow + timeoutSeconds * 1000); |
There was a problem hiding this comment.
Flaky test: exact
Date.now() equality
wallNow is captured before the async function call, while deadlineAtMs inside executeJobCoreWithTimeout is set to a second Date.now() invocation. If even 1 ms elapses between the two calls (very possible under CI load), the .toBe() assertion fails. A range check or fake timers would make this deterministic:
| const wallNow = Date.now(); | |
| const result = await executeJobCoreWithTimeout(state, cronJob); | |
| expect(result).toMatchObject({ status: "ok", summary: "done" }); | |
| expect(observedDeadlineAtMs).toBe(wallNow + timeoutSeconds * 1000); | |
| const wallNow = Date.now(); | |
| const result = await executeJobCoreWithTimeout(state, cronJob); | |
| expect(result).toMatchObject({ status: "ok", summary: "done" }); | |
| // Allow a small window for the two Date.now() calls to differ | |
| expect(observedDeadlineAtMs).toBeGreaterThanOrEqual(wallNow + timeoutSeconds * 1000); | |
| expect(observedDeadlineAtMs).toBeLessThan(wallNow + timeoutSeconds * 1000 + 500); |
Alternatively, mock Date.now with vi.spyOn(Date, "now").mockReturnValue(wallNow) to make both calls return the same value and keep the exact .toBe check.
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/cron/service.issue-regressions.test.ts
Line: 1115-1119
Comment:
**Flaky test: exact `Date.now()` equality**
`wallNow` is captured before the async function call, while `deadlineAtMs` inside `executeJobCoreWithTimeout` is set to a second `Date.now()` invocation. If even 1 ms elapses between the two calls (very possible under CI load), the `.toBe()` assertion fails. A range check or fake timers would make this deterministic:
```suggestion
const wallNow = Date.now();
const result = await executeJobCoreWithTimeout(state, cronJob);
expect(result).toMatchObject({ status: "ok", summary: "done" });
// Allow a small window for the two Date.now() calls to differ
expect(observedDeadlineAtMs).toBeGreaterThanOrEqual(wallNow + timeoutSeconds * 1000);
expect(observedDeadlineAtMs).toBeLessThan(wallNow + timeoutSeconds * 1000 + 500);
```
Alternatively, mock `Date.now` with `vi.spyOn(Date, "now").mockReturnValue(wallNow)` to make both calls return the same value and keep the exact `.toBe` check.
How can I resolve this? If you propose a fix, please make it concise.There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 93927ee1ab
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (beforeAttemptResult?.type === "stop") { | ||
| logModelFallbackDecision({ | ||
| decision: "stop_before_candidate", | ||
| runId: params.runId, | ||
| requestedProvider: params.provider, | ||
| requestedModel: params.model, | ||
| candidate, | ||
| attempt: i + 1, | ||
| total: candidates.length, | ||
| reason: beforeAttemptResult.reason ?? "timeout", | ||
| error: | ||
| beforeAttemptResult.error ?? | ||
| `Stopped fallback before ${candidate.provider}/${candidate.model}.`, | ||
| nextCandidate: candidates[i + 1], | ||
| isPrimary, | ||
| requestedModelMatched: requestedModel, | ||
| fallbackConfigured: hasFallbackCandidates, | ||
| previousAttempts: attempts, | ||
| }); | ||
| break; |
There was a problem hiding this comment.
Preserve the last real error when a budget stop follows skips
When beforeAttempt returns stop here, the loop just breaks and falls through to throwFallbackFailureSummary(). That only rethrows the prior error when attempts.length <= 1, so any skipped candidate recorded earlier in this same run (for example the existing cooldown/auth skip branches) turns the result into a synthesized All models failed ... summary instead of the original primary failure. In a chain like “primary fails, next fallback is skipped, later fallback is stopped for low cron budget”, this change still masks the real error the commit is trying to preserve.
Useful? React with 👍 / 👎.
6d3f4d7 to
3433417
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 34334173d5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (lastError !== undefined) { | ||
| throw lastError; | ||
| } | ||
| break; |
There was a problem hiding this comment.
Propagate stop errors when all prior candidates were skipped
When beforeAttempt stops here after only skip-only entries have accumulated (for example, the requested model is skipped by the cooldown/auth branch earlier in runWithModelFallback, and the cron budget gate fires on the next runnable fallback), lastError is still undefined. This break then falls through to throwFallbackFailureSummary(), so callers see a synthesized All models failed ... error instead of the explicit stop message from beforeAttempt. In that cron path the new budget-stop behavior is still masked by a misleading summary.
Useful? React with 👍 / 👎.
3433417 to
d628daa
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d628daad75
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const beforeAttemptResult = await params.beforeAttempt?.({ | ||
| candidate, | ||
| attempt: i + 1, | ||
| total: candidates.length, | ||
| previousAttempts: attempts, | ||
| isPrimary, | ||
| requestedModelMatched: requestedModel, | ||
| fallbackConfigured: hasFallbackCandidates, | ||
| }); |
There was a problem hiding this comment.
Run the stop hook before marking cooldown probes
When beforeAttempt stops a near-deadline cron run on a cooldowned provider, the code above has already called markProbeAttempt() for that candidate before any request is sent. Because lastProbeAttempt is global and throttles probes for 30 seconds, the next fallback run can incorrectly skip a real cooldown-recovery probe even though this run never actually probed the provider. Moving the stop check ahead of the cooldown probe bookkeeping would avoid burning probe slots on aborted attempts.
Useful? React with 👍 / 👎.
d628daa to
1bbc38a
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1bbc38af88
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const minRemainingMs = resolveCronFallbackMinRemainingMs(cronTimeoutMs); | ||
| if (remainingMs >= minRemainingMs) { |
There was a problem hiding this comment.
Base the fallback guard on the actual agent timeout
For cron jobs that rely on agents.defaults.timeoutSeconds instead of payload.timeoutSeconds, timeoutMs in this function can be much smaller than the cron safety timeout. The new guard uses cronTimeoutMs here, so those jobs still require 30s remaining before any fallback, even when each model attempt is capped to 5–10s. A primary failure in the last 20–25s will now skip fallbacks that still had enough budget to complete under the configured agent timeout.
Useful? React with 👍 / 👎.
| git branch -D fix/subagent-timeout-partial-results 2>/dev/null && echo "Deleted local" || echo "Already gone" | ||
| git push origin --delete fix/subagent-timeout-partial-results 2>/dev/null && echo "Deleted remote" || echo "Already gone from remote" |
There was a problem hiding this comment.
Remove the unconditional remote branch deletion
This helper always deletes fix/subagent-timeout-partial-results at the end, even though that branch is not in BRANCHES and the script never verifies that it was merged. Running the script to rebase the listed branches will also remove that remote branch from origin, which is destructive if the PR is still open or another maintainer is still using it.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1bbc38af88
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const minRemainingMs = resolveCronFallbackMinRemainingMs(cronTimeoutMs); | ||
| if (remainingMs >= minRemainingMs) { |
There was a problem hiding this comment.
Base the fallback guard on the actual agent timeout
For cron jobs that rely on agents.defaults.timeoutSeconds instead of payload.timeoutSeconds, timeoutMs in this function can be much smaller than the cron safety timeout. The new guard uses cronTimeoutMs here, so those jobs still require 30s remaining before any fallback, even when each model attempt is capped to 5–10s. A primary failure in the last 20–25s will now skip fallbacks that still had enough budget to complete under the configured agent timeout.
Useful? React with 👍 / 👎.
| git branch -D fix/subagent-timeout-partial-results 2>/dev/null && echo "Deleted local" || echo "Already gone" | ||
| git push origin --delete fix/subagent-timeout-partial-results 2>/dev/null && echo "Deleted remote" || echo "Already gone from remote" |
There was a problem hiding this comment.
Remove the unconditional remote branch deletion
This helper always deletes fix/subagent-timeout-partial-results at the end, even though that branch is not in BRANCHES and the script never verifies that it was merged. Running the script to rebase the listed branches will also remove that remote branch from origin, which is destructive if the PR is still open or another maintainer is still using it.
Useful? React with 👍 / 👎.
1bbc38a to
d31660a
Compare
d31660a to
9cab596
Compare
9cab596 to
97875ec
Compare
97875ec to
a4e231e
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a4e231e921
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
a4e231e to
3f552d9
Compare
d5591c3 to
008791f
Compare
|
Rebased onto Fixed the cron regression test parse error by removing the duplicate Added the PR body CI status for |
008791f to
d29f806
Compare
d29f806 to
f7bb1fa
Compare
|
P2 follow-up resolved.
Verification:
|
9be062d to
2374411
Compare
|
ClawSweeper PR egg 🎁 Pass real behavior proof to wake the egg and unlock a hatchable treat. Where did the egg go?
|
2374411 to
8d90cb5
Compare
8d90cb5 to
92496f1
Compare
|
Heads up: this PR needs to be updated against current |
|
@imwyvern thanks for the PR. ClawSweeper is still waiting on real behavior proof before this can move forward. Useful proof can be a screenshot, short video, terminal output, copied live output, linked artifact, or redacted logs that show the changed behavior after the fix. Please redact private tokens, phone numbers, private endpoints, customer data, and anything else sensitive. Once proof is added to the PR body or a comment, ClawSweeper or a maintainer can re-check it. |
Problem
When a cron job's primary model times out near the cron deadline, the fallback model is started but immediately killed by the same cron timeout — receiving effectively 0ms to make an LLM request. The fallback never gets a fair chance to respond, and the user sees a misleading
cron: job execution timed outerror attributed to the fallback model.Observed in production logs (2026-03-22):
The cron timeout wraps the entire fallback chain. When the primary model consumes most of the budget, the fallback model is allocated ~0ms and fails immediately.
Solution
Propagate the cron deadline (
deadlineAtMs) fromexecuteJobCoreWithTimeoutthrough the cron service state down torunCronIsolatedAgentTurn.Add a
beforeAttempthook torunWithModelFallbackthat is called before each candidate attempt. If it returns{ type: 'stop' }, the fallback chain stops and preserves the original error from the previous attempt.Check remaining budget in the cron layer's
beforeAttemptcallback. If less thanmin(30s, timeout/4)remains before the cron deadline, the fallback is skipped with a clear message:This ensures:
beforeAttempthook is generic and reusable for other pre-attempt checksChanges
src/agents/model-fallback.ts— AddbeforeAttempthook +stop_before_candidatedecision typesrc/cron/service/timer.ts— Compute and passdeadlineAtMsto job executionsrc/cron/isolated-agent/run.ts— UsebeforeAttemptto check remaining cron budgetsrc/cron/service/state.ts+src/gateway/server-cron.ts— ThreaddeadlineAtMsthrough interfacesTests
model-fallback.test.ts— beforeAttempt stop behaviorrun.fallback-time-budget.test.ts— Cron deadline propagation + fallback budget exhaustionservice.issue-regressions.test.ts— Deadline passed to isolated runsReal behavior proof
Behavior or issue addressed: Cron isolated agent-turn jobs should not start a model fallback when the primary attempt has consumed almost the whole cron deadline and the fallback would be killed immediately by the same timeout.
Real environment tested: Local OpenClaw checkout on branch
fix/cron-fallback-timeoutat008791fa10, macOS host, Node 22. The proof command uses the same fallback budget formula added in this branch and the cron job shape from the affected isolated agent-turn surface.Exact steps or command run after this patch: Ran
node .artifacts/pr-52365-real-behavior-proof.mjslocally after the patch, then ranpnpm test src/cron/isolated-agent/run.fallback-time-budget.test.ts -- --reporter=verboseas supplemental coverage for the same gate.Evidence after fix: Terminal output from the local reproduction command:
Supplemental targeted test output:
Observed result after fix: With only 100ms left before the cron deadline and a 15,000ms minimum fallback budget, the after-fix gate returns
fallbackAttempted=falseand reportsSkipping fallback: only 100ms remain before cron timeout (need at least 15000ms).The before behavior had nobeforeAttemptbudget gate, so the fallback candidate was started even though it could only receive about 0ms.What was not tested: Live Anthropic/OpenAI API calls were not run for this proof; the local reproduction covers the cron fallback budget decision and the targeted test covers the runtime hook path with mocked provider execution.