-
-
Notifications
You must be signed in to change notification settings - Fork 39.7k
Closed
Description
Summary
When a cron job fails due to "model not allowed" error, the scheduler enters a rapid retry loop with no backoff or attempt limit, generating 150+ failed attempts in ~10 seconds.
Environment
- OpenClaw version: 2026.2.6-3 (commit f072249)
- OS: Ubuntu 22.04 (Linux 6.8.0-94-generic)
Steps to Reproduce
- Create an isolated cron job with a model that will be rejected:
{
"name": "retry-storm-test",
"schedule": {"kind": "at", "at": "<soon>"},
"payload": {"kind": "agentTurn", "message": "Test", "model": "sonnet"},
"sessionTarget": "isolated"
}-
Wait for scheduled time
-
Check job runs via
cron.runs
Expected Behavior
- Job fails once with clear error
- Either: retry with exponential backoff, or mark job as failed and stop
Actual Behavior
Massive retry storm:
{"ts": 1770491574481, "status": "error", "error": "model not allowed: anthropic/claude-sonnet-4", "durationMs": 4},
{"ts": 1770491574520, "status": "error", "error": "model not allowed: anthropic/claude-sonnet-4", "durationMs": 4},
{"ts": 1770491574558, "status": "error", "error": "model not allowed: anthropic/claude-sonnet-4", "durationMs": 4},
// ... 150+ more entries in ~10 secondsEach retry is ~30-40ms apart, no backoff applied.
Evidence
Single job generated 150+ run entries in the cron.runs response, all within a 10-second window.
Impact
- Pollutes run history with hundreds of duplicate errors
- Could cause performance issues with rapid retries
- Makes debugging harder (signal lost in noise)
nextRunAtMsnever advances (stays at original scheduled time)
Suggested Fix
- Implement exponential backoff for retries
- Cap retry attempts (e.g., max 3-5 retries)
- Distinguish between retryable errors (network timeout) and permanent errors (model not allowed)
- For permanent errors like "model not allowed", fail immediately without retry
Related
- Isolated sessions (cron jobs) reject all models except primary #11436 (model not allowed bug that triggers this retry storm)
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels