Skip to content

cron: allow retries for local model preflight#82145

Closed
cthornsburg wants to merge 5 commits into
openclaw:mainfrom
cthornsburg:feat/cron-local-preflight-retry
Closed

cron: allow retries for local model preflight#82145
cthornsburg wants to merge 5 commits into
openclaw:mainfrom
cthornsburg:feat/cron-local-preflight-retry

Conversation

@cthornsburg

@cthornsburg cthornsburg commented May 15, 2026

Copy link
Copy Markdown

Summary

  • add configurable retries for isolated cron local-model provider preflight
  • expose cron.modelPreflight.timeoutMs, maxAttempts, and retryDelayMs
  • document the settings for sleeping/cold-starting Ollama, vLLM, SGLang, LM Studio, and other local providers
  • keep defaults unchanged: one 2500ms probe and no retry delay
  • enforce one shared 55s preflight deadline across the full fallback candidate chain

Why

Isolated cron jobs run a lightweight preflight before starting an agent turn for local/private model providers. Current upstream can advance from an unreachable local primary to configured fallback model candidates, but each local candidate still gets only one default 2.5s preflight probe.

That still leaves the cold-start/sleeping-provider case: a local endpoint can become healthy moments after the single probe fails. This PR keeps the fallback behavior intact while letting operators configure a short wake-up window before cron advances to a fallback or marks the run skipped.

The configured per-endpoint retry window is validated at or below 55s, and the runtime now shares one 55s deadline across the complete preflight candidate chain so multi-local fallback paths stay below cron's isolated-agent setup watchdog.

Review fix

After ClawSweeper noted that the original 55s cap applied per endpoint, this branch now:

  • computes one preflight deadline before walking candidates
  • passes that same deadline to every candidate preflight
  • clamps each local probe timeout and retry delay to the remaining shared budget
  • skips additional local endpoint probes once the shared budget is exhausted
  • still allows remote/cloud fallback candidates to be selected immediately

Supplemental verification

  • corepack pnpm install --frozen-lockfile
  • node scripts/run-vitest.mjs src/cron/isolated-agent/model-preflight.runtime.test.ts src/cron/isolated-agent.model-preflight.test.ts src/cron/isolated-agent/run-fallback-policy.test.ts src/config/config-misc.test.ts - 113 tests passed across focused config and cron shards
  • node scripts/run-vitest.mjs src/commands/agent-via-gateway.test.ts - 55 tests passed, covering the previously failing unrelated CI shard locally
  • pnpm lint --threads=8 - passed
  • pnpm exec oxfmt --check docs/cli/cron.md docs/gateway/configuration-reference.md docs/providers/ollama.md src/config/config-misc.test.ts src/config/schema.help.ts src/config/schema.labels.ts src/config/types.cron.ts src/config/zod-schema.ts src/cron/isolated-agent/model-preflight.runtime.test.ts src/cron/isolated-agent/model-preflight.runtime.ts src/cron/isolated-agent.model-preflight.test.ts src/cron/isolated-agent/run.ts - passed
  • git diff --check - passed
  • pnpm tsgo:core - passed

pnpm check currently passes all guards except the local npm-shrinkwrap.json is stale guard. The ready-for-review CI run on this branch previously passed the shrinkwrap check, so I did not include unrelated generated dependency metadata in this cron patch.

Real behavior proof

Behavior addressed: A sleeping or cold-starting local model provider can miss cron's default single 2.5s preflight probe even though it becomes reachable moments later. Configured retries should let that same local candidate wake before cron advances to a fallback or skips the run.

Real environment tested: macOS local OpenClaw source checkout rebased onto upstream main, Node v22.22.0, pnpm v11.2.2. A real loopback OpenAI-compatible HTTP endpoint was started after an 800ms delay. The actual preflightCronModelProvider runtime used its real network fetch and SSRF-guard path against http://127.0.0.1:<port>/v1/models.

Exact steps or command run after this patch: Ran a pnpm exec tsx -e live harness that started the local HTTP endpoint after 800ms and invoked preflightCronModelProvider twice: first with default settings, then with { timeoutMs: 500, maxAttempts: 2, retryDelayMs: 1000 }.

Evidence after fix: Copied live console output from the real delayed endpoint run:

{"label":"default-single-attempt","result":{"status":"unavailable","provider":"delayed","model":"delayed-local-model","baseUrl":"http://127.0.0.1:54323/v1","retryAfterMs":300000,"reason":"Agent cron job uses delayed/delayed-local-model but the local provider endpoint is not reachable at http://127.0.0.1:54323/v1. Skipping this cron run after 1 preflight attempt; OpenClaw will retry the provider preflight on a later scheduled run. Last error: TypeError: fetch failed"},"elapsedMs":159,"serverHits":0}
{"label":"configured-retry-window","result":{"status":"available"},"elapsedMs":1019,"serverHits":1}

Observed result after fix: With defaults, the endpoint was reported unavailable before it woke, confirming the cold-start issue still exists. With the configured retry window, the first connection failed while the endpoint was offline, cron waited, the endpoint started, and the next probe succeeded with status: "available" and one observed /models request.

What was not tested: A separate real Ollama daemon was not stopped and restarted during this run. The proof used a real delayed local OpenAI-compatible endpoint to exercise the same production preflight runtime and network path deterministically.

@openclaw-barnacle openclaw-barnacle Bot added docs Improvements or additions to documentation gateway Gateway runtime size: S triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels May 15, 2026
@clawsweeper

clawsweeper Bot commented May 15, 2026

Copy link
Copy Markdown
Contributor

Thanks for the contribution. ClawSweeper proposes closing this for now: the implementation may be reasonable, but passing review and proof does not establish that OpenClaw should add this product surface.

Close this PR for now: the branch is a reasonable, proof-backed implementation, but its unique remaining value is adding new core cron preflight retry/config policy after current main already fixed fallback-before-skip behavior, so it needs maintainer-sponsored product direction rather than staying open as an external landing candidate.

Root-cause cluster
Relationship: canonical
Canonical: #82145
Summary: This PR is the canonical live item for cron local-preflight retry knobs; related fallback-chain and auth-preflight items overlap the runtime but own different behavior.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

This is a proposal only until the separate default-off apply policy is enabled and all live maintainer-signal checks pass. A maintainer can sponsor the direction, request a narrower version, or apply clawsweeper:human-review to keep it open.

Review details

Best possible solution:

Keep current main's fallback-before-skip behavior, and add cron local-preflight retry knobs only if a maintainer sponsors this core config surface or narrows it to an approved policy.

Do we have a high-confidence way to reproduce the issue?

Not applicable as a broken existing-contract bug. Current main has a fixed single local-provider probe, and the PR body supplies live delayed-endpoint output demonstrating the proposed retry behavior.

Is this the best way to solve the issue?

No, not as an unsponsored core change. The implementation is a reasonable bounded design, but the remaining decision is whether OpenClaw should expose cron.modelPreflight knobs now that fallback-before-skip is fixed on main.

Security review:

Security review cleared: Cleared: the diff adds cron config/runtime/docs/tests only, keeps probes on the existing SSRF-guarded fetch path, and introduces no dependency, workflow, package, secret, or supply-chain surface.

AGENTS.md: found and applied where relevant.

What I checked:

Likely related people:

  • osolmaz: The live PR is assigned to osolmaz, and osolmaz merged the adjacent cron preflight fallback-chain PR that changed the same decision surface. (role: assigned reviewer and adjacent merger; confidence: high; commits: 7a381b807e23; files: src/cron/isolated-agent/run.ts, src/cron/isolated-agent/run-fallback-policy.ts, src/cron/isolated-agent/run-executor.ts)
  • chen-zhang-cs-code: Authored the merged cron preflight fallback-chain PR that current main uses before this retry-config PR. (role: feature-history contributor; confidence: high; commits: 7a381b807e23; files: src/cron/isolated-agent/run.ts, src/cron/isolated-agent/run-fallback-policy.ts, src/cron/isolated-agent.model-preflight.test.ts)
  • steipete: GitHub commit history shows steipete introduced the local-provider cron preflight runtime and adjacent Ollama docs. (role: original runtime contributor; confidence: high; commits: a66605bf23a7, 93e75f646fff, 1e7510ae103e; files: src/cron/isolated-agent/model-preflight.runtime.ts, src/cron/isolated-agent/run.ts, docs/providers/ollama.md)
  • vincentkoc: Recent GitHub path history shows multiple cron runtime and config-adjacent changes around isolated cron behavior. (role: recent adjacent contributor; confidence: medium; commits: e573b751bf4b, efbefceb0e2e, 0a338147a52f; files: src/cron/isolated-agent/run.ts, src/config/zod-schema.ts)

Codex review notes: model internal, reasoning high; reviewed against 44b4a0ac0598.

@osolmaz osolmaz self-assigned this May 28, 2026
@cthornsburg
cthornsburg force-pushed the feat/cron-local-preflight-retry branch from b3d8d32 to 4aaae6b Compare June 8, 2026 13:11
@openclaw-barnacle openclaw-barnacle Bot added proof: supplied External PR includes structured after-fix real behavior proof. and removed triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels Jun 8, 2026
@cthornsburg
cthornsburg marked this pull request as ready for review June 8, 2026 13:21
@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. P2 Normal backlog priority with limited blast radius. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 availability 🚨 May cause crashes, hangs, restart loops, stalls, or process outages. labels Jun 8, 2026
@cthornsburg
cthornsburg force-pushed the feat/cron-local-preflight-retry branch from 4aaae6b to a02407e Compare June 8, 2026 14:19
@openclaw-barnacle openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 8, 2026
@cthornsburg

Copy link
Copy Markdown
Author

@clawsweeper re-review

Addressed the chain-budget finding in a02407e: the fallback walk now shares one 55s deadline, each probe/delay is clamped to the remaining budget, and multi-local fallback regression coverage was added.

@clawsweeper

clawsweeper Bot commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. labels Jun 8, 2026
@clawsweeper clawsweeper Bot added status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. labels Jun 13, 2026
@openclaw-barnacle openclaw-barnacle Bot removed the stale Marked as stale due to inactivity label Jun 14, 2026
@openclaw-barnacle

Copy link
Copy Markdown

This assigned pull request has been automatically marked as stale after being open for 27 days.
Please add updates or it will be closed.

@openclaw-barnacle openclaw-barnacle Bot added the stale Marked as stale due to inactivity label Jun 14, 2026
@clawsweeper clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. and removed rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Jun 15, 2026
@cthornsburg
cthornsburg force-pushed the feat/cron-local-preflight-retry branch from 416af08 to 9a8fd3d Compare June 15, 2026 15:52
@cthornsburg

Copy link
Copy Markdown
Author

@clawsweeper re-review

Addressed the remaining deadline-clamped cache finding in 9a8fd3de8 and rebased the branch onto current main.

A probe that receives less than its configured timeout because the shared chain deadline is near no longer writes an unavailable result to the endpoint-health cache. The focused regression test proves that a timeout-clamped failure is probed normally and succeeds on the next preflight call.

Verification after rebase:

  • node scripts/run-vitest.mjs src/cron/isolated-agent/model-preflight.runtime.test.ts src/cron/isolated-agent.model-preflight.test.ts src/cron/isolated-agent/run-fallback-policy.test.ts src/config/config-misc.test.ts - 120 tests passed across 4 files
  • pnpm exec oxfmt --check on all 12 touched files - passed
  • pnpm lint --threads=8 - passed
  • pnpm tsgo:core - passed
  • git diff --check origin/main...HEAD - passed

@openclaw-barnacle openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 15, 2026
@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. labels Jun 15, 2026
@clawsweeper

clawsweeper Bot commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

@clawsweeper

clawsweeper Bot commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

🦞👀
ClawSweeper picked this up.

Command router queued. I will update this comment with the next step.

@openclaw-barnacle openclaw-barnacle Bot removed the stale Marked as stale due to inactivity label Jun 16, 2026
@openclaw-barnacle

Copy link
Copy Markdown

This assigned pull request has been automatically marked as stale after being open for 27 days.
Please add updates or it will be closed.

@openclaw-barnacle openclaw-barnacle Bot added the stale Marked as stale due to inactivity label Jun 16, 2026
@openclaw-barnacle

Copy link
Copy Markdown

Closing due to inactivity.
If you believe this PR should be revived, post in #clawtributors on Discord to talk to a maintainer.
That channel is the escape hatch for high-quality PRs that get auto-closed.

@openclaw-barnacle openclaw-barnacle Bot closed this Jul 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Improvements or additions to documentation gateway Gateway runtime merge-risk: 🚨 availability 🚨 May cause crashes, hangs, restart loops, stalls, or process outages. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. P2 Normal backlog priority with limited blast radius. proof: sufficient ClawSweeper judged the real behavior proof convincing. proof: supplied External PR includes structured after-fix real behavior proof. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. size: M stale Marked as stale due to inactivity status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants