Skip to content

Commit 7fefc5f

Browse files
849261680velvet-shark
849261680
and
velvet-shark
authored
fix: cron stream stalls fail over before job timeout (#96096)
* fix(agents): cap cron stream idle stalls * fix(agents): preserve cron hostname timeout * fix: bound cron idle timeout local exceptions * fix: bound cron idle timeout local exceptions --------- Co-authored-by: Radek Sienkiewicz <[email protected]>
1 parent 19707cc commit 7fefc5f

4 files changed

Lines changed: 268 additions & 14 deletions

File tree

‎docs/concepts/agent-loop.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -167,7 +167,7 @@ surfaces, while Codex native hooks remain a separate lower-level Codex mechanism
167167
- Agent runtime: `agents.defaults.timeoutSeconds` default 172800s (48 hours); enforced in `runEmbeddedAgent` abort timer.
168168
- Cron runtime: isolated agent-turn `timeoutSeconds` is owned by cron. The scheduler starts that timer when execution begins, aborts the underlying run at the configured deadline, then runs bounded cleanup before recording the timeout so a stale child session cannot keep the lane stuck.
169169
- Session liveness diagnostics: with diagnostics enabled, `diagnostics.stuckSessionWarnMs` classifies long `processing` sessions that have no observed reply, tool, status, block, or ACP progress. Active embedded runs, model calls, and tool calls report as `session.long_running`; owned silent model calls also stay `session.long_running` until `diagnostics.stuckSessionAbortMs` so slow or non-streaming providers are not reported as stalled too early. Active work with no recent progress reports as `session.stalled`; owned model calls switch to `session.stalled` at or after the abort threshold, and ownerless stale model/tool activity is not hidden as long-running. `session.stuck` is reserved for recoverable stale session bookkeeping, including idle queued sessions with stale ownerless model/tool activity. Stale session bookkeeping releases the affected session lane immediately after recovery gates pass; stalled embedded runs are abort-drained only after `diagnostics.stuckSessionAbortMs` (default: at least 5 minutes and 3x the warning threshold) so queued work can resume without cutting off merely slow runs. Recovery emits structured requested/completed outcomes, and diagnostic state is marked idle only if the same processing generation is still current. Repeated `session.stuck` diagnostics back off while the session remains unchanged.
170-
- Model idle timeout: OpenClaw aborts a model request when no response chunks arrive before the idle window. `models.providers.<id>.timeoutSeconds` extends this idle watchdog for slow local/self-hosted providers, but it is still bounded by any lower `agents.defaults.timeoutSeconds` or run-specific timeout because those control the whole agent run. Otherwise OpenClaw uses `agents.defaults.timeoutSeconds` when configured, capped at 120s by default. Cron-triggered cloud model runs with no explicit model or agent timeout use the same default idle watchdog; cron-triggered local or self-hosted model runs disable the implicit watchdog unless an explicit timeout is configured, so slow local providers should set `models.providers.<id>.timeoutSeconds`.
170+
- Model idle timeout: OpenClaw aborts a model request when no response chunks arrive before the idle window. `models.providers.<id>.timeoutSeconds` extends this idle watchdog for slow local/self-hosted providers, but it is still bounded by any lower `agents.defaults.timeoutSeconds` or run-specific timeout because those control the whole agent run. Otherwise OpenClaw uses `agents.defaults.timeoutSeconds` when configured, capped at 120s by default. Cron-triggered cloud model runs with no explicit model or agent timeout use the same default idle watchdog; with an explicit cron run timeout, cloud model stream stalls are capped at 60s so configured model fallbacks can run before the outer cron deadline. Cron-triggered local or self-hosted model runs disable the implicit watchdog unless an explicit timeout is configured, and explicit cron run timeouts remain the idle window for local/self-hosted providers, so slow local providers should set `models.providers.<id>.timeoutSeconds`.
171171
- Provider HTTP request timeout: `models.providers.<id>.timeoutSeconds` applies to that provider's model HTTP fetches, including connect, headers, body, SDK request timeout, total guarded-fetch abort handling, and model stream idle watchdog. Use this for slow local/self-hosted providers such as Ollama before raising the whole agent runtime timeout, and keep the agent/runtime timeout at least as high when the model request needs to run longer.
172172

173173
## Where things can end early

‎src/agents/embedded-agent-runner/run/attempt.ts‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3129,7 +3129,11 @@ export async function runEmbeddedAttempt(
31293129
trigger: params.trigger,
31303130
runTimeoutMs: resolvedRunTimeoutMs,
31313131
modelRequestTimeoutMs: (params.model as { requestTimeoutMs?: number }).requestTimeoutMs,
3132-
model: params.model as { baseUrl?: string },
3132+
model: {
3133+
baseUrl: params.model.baseUrl,
3134+
id: params.modelId,
3135+
provider: params.provider,
3136+
},
31333137
});
31343138
if (idleTimeoutMs > 0) {
31353139
activeSession.agent.streamFn = streamWithIdleTimeout(

‎src/agents/embedded-agent-runner/run/llm-idle-timeout.test.ts‎

Lines changed: 148 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ import type { StreamFn } from "../../runtime/index.js";
1212
import { resolveLlmIdleTimeoutMs, streamWithIdleTimeout } from "./llm-idle-timeout.js";
1313

1414
const DEFAULT_LLM_IDLE_TIMEOUT_MS = 120_000;
15+
const CRON_LLM_IDLE_TIMEOUT_MS = 60_000;
1516

1617
describe("resolveLlmIdleTimeoutMs", () => {
1718
it("returns default when config is undefined", () => {
@@ -41,8 +42,153 @@ describe("resolveLlmIdleTimeoutMs", () => {
4142
expect(resolveLlmIdleTimeoutMs({ runTimeoutMs: 30_000 })).toBe(30_000);
4243
});
4344

44-
it("honors explicit cron run timeouts as the idle watchdog ceiling", () => {
45-
expect(resolveLlmIdleTimeoutMs({ trigger: "cron", runTimeoutMs: 600_000 })).toBe(600_000);
45+
it("caps explicit cron run timeouts so stream stalls can reach model fallbacks", () => {
46+
expect(resolveLlmIdleTimeoutMs({ trigger: "cron", runTimeoutMs: 600_000 })).toBe(
47+
CRON_LLM_IDLE_TIMEOUT_MS,
48+
);
49+
});
50+
51+
it("uses shorter explicit cron run timeouts as the idle watchdog ceiling", () => {
52+
expect(resolveLlmIdleTimeoutMs({ trigger: "cron", runTimeoutMs: 30_000 })).toBe(30_000);
53+
});
54+
55+
it("honors explicit cron run timeouts for local provider model calls", () => {
56+
expect(
57+
resolveLlmIdleTimeoutMs({
58+
trigger: "cron",
59+
runTimeoutMs: 600_000,
60+
model: { baseUrl: "http://127.0.0.1:11434" },
61+
}),
62+
).toBe(600_000);
63+
});
64+
65+
it.each([
66+
["ollama", "http://ollama-host:11434"],
67+
["ollama-beelink", "http://ollama-host:11434"],
68+
["lmstudio", "http://lmstudio-box:1234/v1"],
69+
["lmstudio-mac", "http://lmstudio-box:1234/v1"],
70+
["vllm", "http://vllm-rig:8000/v1"],
71+
["sglang", "http://sglang-rig:30000/v1"],
72+
])(
73+
"honors explicit cron run timeouts for self-hosted provider %s hostname %s",
74+
(provider, baseUrl) => {
75+
expect(
76+
resolveLlmIdleTimeoutMs({
77+
trigger: "cron",
78+
runTimeoutMs: 600_000,
79+
model: { provider, baseUrl },
80+
}),
81+
).toBe(600_000);
82+
},
83+
);
84+
85+
it("honors explicit cron run timeouts for explicit local host aliases", () => {
86+
expect(
87+
resolveLlmIdleTimeoutMs({
88+
trigger: "cron",
89+
runTimeoutMs: 600_000,
90+
model: { baseUrl: "http://host.docker.internal:11434" },
91+
}),
92+
).toBe(600_000);
93+
});
94+
95+
it("honors explicit cron run timeouts for custom local provider markers on bare hostnames", () => {
96+
const cfg = {
97+
models: {
98+
providers: {
99+
gpu: {
100+
baseUrl: "http://gpu-box:8000/v1",
101+
api: "openai-completions",
102+
apiKey: "custom-local",
103+
models: [],
104+
},
105+
"local-ollama": {
106+
baseUrl: "http://ollama-box:11434",
107+
api: "ollama",
108+
apiKey: "ollama-local",
109+
models: [],
110+
},
111+
},
112+
},
113+
} as unknown as OpenClawConfig;
114+
115+
expect(
116+
resolveLlmIdleTimeoutMs({
117+
cfg,
118+
trigger: "cron",
119+
runTimeoutMs: 600_000,
120+
model: { provider: "gpu", baseUrl: "http://gpu-box:8000/v1" },
121+
}),
122+
).toBe(600_000);
123+
expect(
124+
resolveLlmIdleTimeoutMs({
125+
cfg,
126+
trigger: "cron",
127+
runTimeoutMs: 600_000,
128+
model: { provider: "local-ollama", baseUrl: "http://ollama-box:11434" },
129+
}),
130+
).toBe(600_000);
131+
});
132+
133+
it("honors explicit cron run timeouts for provider-owned local services on bare hostnames", () => {
134+
const cfg = {
135+
models: {
136+
providers: {
137+
ds4: {
138+
baseUrl: "http://ds4-box:8000/v1",
139+
api: "openai-completions",
140+
localService: {
141+
command: "/opt/ds4/ds4-server",
142+
healthUrl: "http://ds4-box:8000/v1/models",
143+
},
144+
models: [],
145+
},
146+
},
147+
},
148+
} as unknown as OpenClawConfig;
149+
150+
expect(
151+
resolveLlmIdleTimeoutMs({
152+
cfg,
153+
trigger: "cron",
154+
runTimeoutMs: 600_000,
155+
model: { provider: "ds4", baseUrl: "http://ds4-box:8000/v1" },
156+
}),
157+
).toBe(600_000);
158+
});
159+
160+
it.each([
161+
["openai", "openai/gpt-5.5", "http://api:8080/v1"],
162+
["custom-proxy", "custom-proxy/gpt-5.5", "http://gateway:4000/v1"],
163+
["ollama-cloud", "ollama-cloud/kimi-k2.6", "http://ollama-host:11434"],
164+
])(
165+
"keeps the cron stall cap for cloud provider %s routed through single-label host %s",
166+
(provider, id, baseUrl) => {
167+
expect(
168+
resolveLlmIdleTimeoutMs({
169+
trigger: "cron",
170+
runTimeoutMs: 600_000,
171+
model: { provider, id, baseUrl },
172+
}),
173+
).toBe(CRON_LLM_IDLE_TIMEOUT_MS);
174+
},
175+
);
176+
177+
it("keeps the cron stall cap for remote or cloud hostnames", () => {
178+
expect(
179+
resolveLlmIdleTimeoutMs({
180+
trigger: "cron",
181+
runTimeoutMs: 600_000,
182+
model: { provider: "openai", id: "openai/gpt-5.5", baseUrl: "https://api.openai.com/v1" },
183+
}),
184+
).toBe(CRON_LLM_IDLE_TIMEOUT_MS);
185+
expect(
186+
resolveLlmIdleTimeoutMs({
187+
trigger: "cron",
188+
runTimeoutMs: 600_000,
189+
model: { provider: "ollama", id: "ollama/gpt-oss:cloud", baseUrl: "http://ollama-host" },
190+
}),
191+
).toBe(CRON_LLM_IDLE_TIMEOUT_MS);
46192
});
47193

48194
it("disables the idle watchdog when an explicit run timeout disables timeouts", () => {

‎src/agents/embedded-agent-runner/run/llm-idle-timeout.ts‎

Lines changed: 114 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,16 @@ import type { EmbeddedRunTrigger } from "./params.js";
1818
* Default idle timeout for LLM streaming responses in milliseconds.
1919
*/
2020
const DEFAULT_LLM_IDLE_TIMEOUT_MS = 120_000;
21+
// Cron has its own outer watchdog; stream stalls must fail early enough for
22+
// the existing model fallback chain to try the next configured candidate.
23+
const CRON_LLM_IDLE_TIMEOUT_MS = 60_000;
24+
const LOCAL_PROVIDER_AUTH_MARKERS = new Set(["custom-local", "ollama-local"]);
25+
const SELF_HOSTED_PROVIDER_ID_PREFIXES = ["ollama", "lmstudio", "vllm", "sglang", "llama-cpp"];
26+
27+
type IdleTimeoutProviderConfig = {
28+
apiKey?: unknown;
29+
localService?: unknown;
30+
};
2131

2232
/**
2333
* Detects loopback / private-network / `.local` base URLs. Local providers
@@ -37,11 +47,9 @@ const DEFAULT_LLM_IDLE_TIMEOUT_MS = 120_000;
3747
* matched, mirroring the SSRF-policy helper in
3848
* `src/cron/isolated-agent/model-preflight.runtime.ts`.
3949
* - DNS-resolved local aliases (e.g. an `/etc/hosts` entry mapping a custom
40-
* hostname to a private IP) are not detected: classification keys on
41-
* `URL.hostname` so resolution would have to happen here, and adding
42-
* sync/async DNS to the watchdog hot path is disproportionate. Affected
43-
* users can use the IP directly or set
44-
* `models.providers.<id>.timeoutSeconds` explicitly.
50+
* hostname to a private IP) are not detected for the implicit watchdog opt-out:
51+
* classification keys on `URL.hostname` so resolution would have to happen
52+
* here, and adding sync/async DNS to the watchdog hot path is disproportionate.
4553
*/
4654
function isLocalProviderBaseUrl(baseUrl: string): boolean {
4755
let host: string;
@@ -95,6 +103,82 @@ function isLocalProviderBaseUrl(baseUrl: string): boolean {
95103
);
96104
}
97105

106+
function isExplicitLocalHostnameBaseUrl(baseUrl: string): boolean {
107+
let host: string;
108+
try {
109+
host = new URL(baseUrl).hostname.toLowerCase();
110+
} catch {
111+
return false;
112+
}
113+
114+
if (
115+
host === "docker.orb.internal" ||
116+
host === "host.docker.internal" ||
117+
host === "host.orb.internal"
118+
) {
119+
return true;
120+
}
121+
return false;
122+
}
123+
124+
function isBareProviderHostnameBaseUrl(baseUrl: string): boolean {
125+
let host: string;
126+
try {
127+
host = new URL(baseUrl).hostname.toLowerCase();
128+
} catch {
129+
return false;
130+
}
131+
132+
if (host.includes(".") || host.includes(":")) {
133+
return false;
134+
}
135+
return /^[a-z0-9](?:[a-z0-9-]*[a-z0-9])?$/.test(host);
136+
}
137+
138+
function isSelfHostedProviderId(provider: string | undefined): boolean {
139+
const normalized = provider?.trim().toLowerCase();
140+
if (!normalized || normalized === "ollama-cloud") {
141+
return false;
142+
}
143+
return SELF_HOSTED_PROVIDER_ID_PREFIXES.some(
144+
(prefix) => normalized === prefix || normalized.startsWith(`${prefix}-`),
145+
);
146+
}
147+
148+
function findConfiguredProviderConfig(
149+
cfg: OpenClawConfig | undefined,
150+
provider: string | undefined,
151+
): IdleTimeoutProviderConfig | undefined {
152+
const normalizedProvider = provider?.trim().toLowerCase();
153+
if (!normalizedProvider) {
154+
return undefined;
155+
}
156+
const providers = cfg?.models?.providers as
157+
| Record<string, IdleTimeoutProviderConfig | undefined>
158+
| undefined;
159+
const exact = providers?.[normalizedProvider];
160+
if (exact) {
161+
return exact;
162+
}
163+
return Object.entries(providers ?? {}).find(
164+
([key]) => key.trim().toLowerCase() === normalizedProvider,
165+
)?.[1];
166+
}
167+
168+
function hasLocalProviderAuthMarker(apiKey: unknown): boolean {
169+
return typeof apiKey === "string" && LOCAL_PROVIDER_AUTH_MARKERS.has(apiKey.trim().toLowerCase());
170+
}
171+
172+
function hasConfiguredLocalProviderSignal(params: {
173+
cfg: OpenClawConfig | undefined;
174+
provider: string | undefined;
175+
}): boolean {
176+
const providerConfig = findConfiguredProviderConfig(params.cfg, params.provider);
177+
return Boolean(
178+
providerConfig?.localService || hasLocalProviderAuthMarker(providerConfig?.apiKey),
179+
);
180+
}
181+
98182
function isOllamaCloudModel(model: { id?: string; provider?: string } | undefined): boolean {
99183
const rawModelId = model?.id;
100184
if (typeof rawModelId !== "string") {
@@ -134,6 +218,22 @@ export function resolveLlmIdleTimeoutMs(params?: {
134218
const hasExplicitRunTimeout =
135219
typeof runTimeoutMs === "number" && Number.isFinite(runTimeoutMs) && runTimeoutMs > 0;
136220
const runTimeoutIsNoTimeout = hasExplicitRunTimeout && runTimeoutMs >= MAX_TIMER_TIMEOUT_MS;
221+
const baseUrl = params?.model?.baseUrl;
222+
const isLocalProvider =
223+
typeof baseUrl === "string" && baseUrl.length > 0 && isLocalProviderBaseUrl(baseUrl);
224+
const isLocalRuntimeModel = isLocalProvider && !isOllamaCloudModel(params?.model);
225+
const isExplicitLocalHostnameRuntimeModel =
226+
typeof baseUrl === "string" &&
227+
baseUrl.length > 0 &&
228+
isExplicitLocalHostnameBaseUrl(baseUrl) &&
229+
!isOllamaCloudModel(params?.model);
230+
const isSelfHostedHostnameRuntimeModel =
231+
typeof baseUrl === "string" &&
232+
baseUrl.length > 0 &&
233+
isBareProviderHostnameBaseUrl(baseUrl) &&
234+
(isSelfHostedProviderId(params?.model?.provider) ||
235+
hasConfiguredLocalProviderSignal({ cfg: params?.cfg, provider: params?.model?.provider })) &&
236+
!isOllamaCloudModel(params?.model);
137237
const timeoutBounds = [
138238
runTimeoutIsNoTimeout ? undefined : runTimeoutMs,
139239
hasExplicitRunTimeout ? undefined : agentTimeoutMs,
@@ -174,7 +274,14 @@ export function resolveLlmIdleTimeoutMs(params?: {
174274
return 0;
175275
}
176276
if (params?.trigger === "cron") {
177-
return clampTimeoutMs(runTimeoutMs);
277+
if (
278+
isLocalRuntimeModel ||
279+
isExplicitLocalHostnameRuntimeModel ||
280+
isSelfHostedHostnameRuntimeModel
281+
) {
282+
return clampTimeoutMs(runTimeoutMs);
283+
}
284+
return clampTimeoutMs(Math.min(runTimeoutMs, CRON_LLM_IDLE_TIMEOUT_MS));
178285
}
179286
return clampImplicitTimeoutMs(runTimeoutMs);
180287
}
@@ -190,10 +297,7 @@ export function resolveLlmIdleTimeoutMs(params?: {
190297
// baseUrl pointing at loopback / private-network / `.local`. Ollama cloud
191298
// models are still hosted remotely even when proxied through local Ollama, so
192299
// keep the cloud watchdog for `*:cloud` model ids.
193-
const baseUrl = params?.model?.baseUrl;
194-
const isLocalProvider =
195-
typeof baseUrl === "string" && baseUrl.length > 0 && isLocalProviderBaseUrl(baseUrl);
196-
if (isLocalProvider && !isOllamaCloudModel(params?.model)) {
300+
if (isLocalRuntimeModel) {
197301
return 0;
198302
}
199303

0 commit comments

Comments
 (0)