Skip to content

Commit cf37b17

Browse files
velvet-sharkvelvet-shark
authored
fix: bound cron idle timeout local exceptions
1 parent 0c45991 commit cf37b17

3 files changed

Lines changed: 180 additions & 11 deletions

File tree

‎docs/concepts/agent-loop.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -167,7 +167,7 @@ surfaces, while Codex native hooks remain a separate lower-level Codex mechanism
167167
- Agent runtime: `agents.defaults.timeoutSeconds` default 172800s (48 hours); enforced in `runEmbeddedAgent` abort timer.
168168
- Cron runtime: isolated agent-turn `timeoutSeconds` is owned by cron. The scheduler starts that timer when execution begins, aborts the underlying run at the configured deadline, then runs bounded cleanup before recording the timeout so a stale child session cannot keep the lane stuck.
169169
- Session liveness diagnostics: with diagnostics enabled, `diagnostics.stuckSessionWarnMs` classifies long `processing` sessions that have no observed reply, tool, status, block, or ACP progress. Active embedded runs, model calls, and tool calls report as `session.long_running`; owned silent model calls also stay `session.long_running` until `diagnostics.stuckSessionAbortMs` so slow or non-streaming providers are not reported as stalled too early. Active work with no recent progress reports as `session.stalled`; owned model calls switch to `session.stalled` at or after the abort threshold, and ownerless stale model/tool activity is not hidden as long-running. `session.stuck` is reserved for recoverable stale session bookkeeping, including idle queued sessions with stale ownerless model/tool activity. Stale session bookkeeping releases the affected session lane immediately after recovery gates pass; stalled embedded runs are abort-drained only after `diagnostics.stuckSessionAbortMs` (default: at least 5 minutes and 3x the warning threshold) so queued work can resume without cutting off merely slow runs. Recovery emits structured requested/completed outcomes, and diagnostic state is marked idle only if the same processing generation is still current. Repeated `session.stuck` diagnostics back off while the session remains unchanged.
170-
- Model idle timeout: OpenClaw aborts a model request when no response chunks arrive before the idle window. `models.providers.<id>.timeoutSeconds` extends this idle watchdog for slow local/self-hosted providers, but it is still bounded by any lower `agents.defaults.timeoutSeconds` or run-specific timeout because those control the whole agent run. Otherwise OpenClaw uses `agents.defaults.timeoutSeconds` when configured, capped at 120s by default. Cron-triggered cloud model runs with no explicit model or agent timeout use the same default idle watchdog; cron-triggered local or self-hosted model runs disable the implicit watchdog unless an explicit timeout is configured, so slow local providers should set `models.providers.<id>.timeoutSeconds`.
170+
- Model idle timeout: OpenClaw aborts a model request when no response chunks arrive before the idle window. `models.providers.<id>.timeoutSeconds` extends this idle watchdog for slow local/self-hosted providers, but it is still bounded by any lower `agents.defaults.timeoutSeconds` or run-specific timeout because those control the whole agent run. Otherwise OpenClaw uses `agents.defaults.timeoutSeconds` when configured, capped at 120s by default. Cron-triggered cloud model runs with no explicit model or agent timeout use the same default idle watchdog; with an explicit cron run timeout, cloud model stream stalls are capped at 60s so configured model fallbacks can run before the outer cron deadline. Cron-triggered local or self-hosted model runs disable the implicit watchdog unless an explicit timeout is configured, and explicit cron run timeouts remain the idle window for local/self-hosted providers, so slow local providers should set `models.providers.<id>.timeoutSeconds`.
171171
- Provider HTTP request timeout: `models.providers.<id>.timeoutSeconds` applies to that provider's model HTTP fetches, including connect, headers, body, SDK request timeout, total guarded-fetch abort handling, and model stream idle watchdog. Use this for slow local/self-hosted providers such as Ollama before raising the whole agent runtime timeout, and keep the agent/runtime timeout at least as high when the model request needs to run longer.
172172

173173
## Where things can end early

‎src/agents/embedded-agent-runner/run/llm-idle-timeout.test.ts‎

Lines changed: 103 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -63,21 +63,117 @@ describe("resolveLlmIdleTimeoutMs", () => {
6363
});
6464

6565
it.each([
66-
"http://ollama-host:11434",
67-
"http://lmstudio-box:1234/v1",
68-
"http://vllm-rig:8000/v1",
69-
"http://sglang-rig:30000/v1",
70-
"http://host.docker.internal:11434",
71-
])("honors explicit cron run timeouts for self-hosted provider hostname %s", (baseUrl) => {
66+
["ollama", "http://ollama-host:11434"],
67+
["ollama-beelink", "http://ollama-host:11434"],
68+
["lmstudio", "http://lmstudio-box:1234/v1"],
69+
["lmstudio-mac", "http://lmstudio-box:1234/v1"],
70+
["vllm", "http://vllm-rig:8000/v1"],
71+
["sglang", "http://sglang-rig:30000/v1"],
72+
])(
73+
"honors explicit cron run timeouts for self-hosted provider %s hostname %s",
74+
(provider, baseUrl) => {
75+
expect(
76+
resolveLlmIdleTimeoutMs({
77+
trigger: "cron",
78+
runTimeoutMs: 600_000,
79+
model: { provider, baseUrl },
80+
}),
81+
).toBe(600_000);
82+
},
83+
);
84+
85+
it("honors explicit cron run timeouts for explicit local host aliases", () => {
7286
expect(
7387
resolveLlmIdleTimeoutMs({
7488
trigger: "cron",
7589
runTimeoutMs: 600_000,
76-
model: { baseUrl },
90+
model: { baseUrl: "http://host.docker.internal:11434" },
7791
}),
7892
).toBe(600_000);
7993
});
8094

95+
it("honors explicit cron run timeouts for custom local provider markers on bare hostnames", () => {
96+
const cfg = {
97+
models: {
98+
providers: {
99+
gpu: {
100+
baseUrl: "http://gpu-box:8000/v1",
101+
api: "openai-completions",
102+
apiKey: "custom-local",
103+
models: [],
104+
},
105+
"local-ollama": {
106+
baseUrl: "http://ollama-box:11434",
107+
api: "ollama",
108+
apiKey: "ollama-local",
109+
models: [],
110+
},
111+
},
112+
},
113+
} as unknown as OpenClawConfig;
114+
115+
expect(
116+
resolveLlmIdleTimeoutMs({
117+
cfg,
118+
trigger: "cron",
119+
runTimeoutMs: 600_000,
120+
model: { provider: "gpu", baseUrl: "http://gpu-box:8000/v1" },
121+
}),
122+
).toBe(600_000);
123+
expect(
124+
resolveLlmIdleTimeoutMs({
125+
cfg,
126+
trigger: "cron",
127+
runTimeoutMs: 600_000,
128+
model: { provider: "local-ollama", baseUrl: "http://ollama-box:11434" },
129+
}),
130+
).toBe(600_000);
131+
});
132+
133+
it("honors explicit cron run timeouts for provider-owned local services on bare hostnames", () => {
134+
const cfg = {
135+
models: {
136+
providers: {
137+
ds4: {
138+
baseUrl: "http://ds4-box:8000/v1",
139+
api: "openai-completions",
140+
localService: {
141+
command: "/opt/ds4/ds4-server",
142+
healthUrl: "http://ds4-box:8000/v1/models",
143+
},
144+
models: [],
145+
},
146+
},
147+
},
148+
} as unknown as OpenClawConfig;
149+
150+
expect(
151+
resolveLlmIdleTimeoutMs({
152+
cfg,
153+
trigger: "cron",
154+
runTimeoutMs: 600_000,
155+
model: { provider: "ds4", baseUrl: "http://ds4-box:8000/v1" },
156+
}),
157+
).toBe(600_000);
158+
});
159+
160+
it.each([
161+
["openai", "openai/gpt-5.5", "http://api:8080/v1"],
162+
["custom-proxy", "custom-proxy/gpt-5.5", "http://gateway:4000/v1"],
163+
["ollama-cloud", "ollama-cloud/kimi-k2.6", "http://ollama-host:11434"],
164+
])(
165+
"keeps the cron stall cap for cloud provider %s routed through single-label host %s",
166+
(provider, id, baseUrl) => {
167+
expect(
168+
resolveLlmIdleTimeoutMs({
169+
trigger: "cron",
170+
runTimeoutMs: 600_000,
171+
model: { provider, id, baseUrl },
172+
}),
173+
).toBe(CRON_LLM_IDLE_TIMEOUT_MS);
174+
},
175+
);
176+
81177
it("keeps the cron stall cap for remote or cloud hostnames", () => {
82178
expect(
83179
resolveLlmIdleTimeoutMs({

‎src/agents/embedded-agent-runner/run/llm-idle-timeout.ts‎

Lines changed: 76 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,13 @@ const DEFAULT_LLM_IDLE_TIMEOUT_MS = 120_000;
2121
// Cron has its own outer watchdog; stream stalls must fail early enough for
2222
// the existing model fallback chain to try the next configured candidate.
2323
const CRON_LLM_IDLE_TIMEOUT_MS = 60_000;
24+
const LOCAL_PROVIDER_AUTH_MARKERS = new Set(["custom-local", "ollama-local"]);
25+
const SELF_HOSTED_PROVIDER_ID_PREFIXES = ["ollama", "lmstudio", "vllm", "sglang", "llama-cpp"];
26+
27+
type IdleTimeoutProviderConfig = {
28+
apiKey?: unknown;
29+
localService?: unknown;
30+
};
2431

2532
/**
2633
* Detects loopback / private-network / `.local` base URLs. Local providers
@@ -96,7 +103,7 @@ function isLocalProviderBaseUrl(baseUrl: string): boolean {
96103
);
97104
}
98105

99-
function isSelfHostedProviderHostnameBaseUrl(baseUrl: string): boolean {
106+
function isExplicitLocalHostnameBaseUrl(baseUrl: string): boolean {
100107
let host: string;
101108
try {
102109
host = new URL(baseUrl).hostname.toLowerCase();
@@ -111,12 +118,67 @@ function isSelfHostedProviderHostnameBaseUrl(baseUrl: string): boolean {
111118
) {
112119
return true;
113120
}
121+
return false;
122+
}
123+
124+
function isBareProviderHostnameBaseUrl(baseUrl: string): boolean {
125+
let host: string;
126+
try {
127+
host = new URL(baseUrl).hostname.toLowerCase();
128+
} catch {
129+
return false;
130+
}
131+
114132
if (host.includes(".") || host.includes(":")) {
115133
return false;
116134
}
117135
return /^[a-z0-9](?:[a-z0-9-]*[a-z0-9])?$/.test(host);
118136
}
119137

138+
function isSelfHostedProviderId(provider: string | undefined): boolean {
139+
const normalized = provider?.trim().toLowerCase();
140+
if (!normalized || normalized === "ollama-cloud") {
141+
return false;
142+
}
143+
return SELF_HOSTED_PROVIDER_ID_PREFIXES.some(
144+
(prefix) => normalized === prefix || normalized.startsWith(`${prefix}-`),
145+
);
146+
}
147+
148+
function findConfiguredProviderConfig(
149+
cfg: OpenClawConfig | undefined,
150+
provider: string | undefined,
151+
): IdleTimeoutProviderConfig | undefined {
152+
const normalizedProvider = provider?.trim().toLowerCase();
153+
if (!normalizedProvider) {
154+
return undefined;
155+
}
156+
const providers = cfg?.models?.providers as
157+
| Record<string, IdleTimeoutProviderConfig | undefined>
158+
| undefined;
159+
const exact = providers?.[normalizedProvider];
160+
if (exact) {
161+
return exact;
162+
}
163+
return Object.entries(providers ?? {}).find(
164+
([key]) => key.trim().toLowerCase() === normalizedProvider,
165+
)?.[1];
166+
}
167+
168+
function hasLocalProviderAuthMarker(apiKey: unknown): boolean {
169+
return typeof apiKey === "string" && LOCAL_PROVIDER_AUTH_MARKERS.has(apiKey.trim().toLowerCase());
170+
}
171+
172+
function hasConfiguredLocalProviderSignal(params: {
173+
cfg: OpenClawConfig | undefined;
174+
provider: string | undefined;
175+
}): boolean {
176+
const providerConfig = findConfiguredProviderConfig(params.cfg, params.provider);
177+
return Boolean(
178+
providerConfig?.localService || hasLocalProviderAuthMarker(providerConfig?.apiKey),
179+
);
180+
}
181+
120182
function isOllamaCloudModel(model: { id?: string; provider?: string } | undefined): boolean {
121183
const rawModelId = model?.id;
122184
if (typeof rawModelId !== "string") {
@@ -160,10 +222,17 @@ export function resolveLlmIdleTimeoutMs(params?: {
160222
const isLocalProvider =
161223
typeof baseUrl === "string" && baseUrl.length > 0 && isLocalProviderBaseUrl(baseUrl);
162224
const isLocalRuntimeModel = isLocalProvider && !isOllamaCloudModel(params?.model);
225+
const isExplicitLocalHostnameRuntimeModel =
226+
typeof baseUrl === "string" &&
227+
baseUrl.length > 0 &&
228+
isExplicitLocalHostnameBaseUrl(baseUrl) &&
229+
!isOllamaCloudModel(params?.model);
163230
const isSelfHostedHostnameRuntimeModel =
164231
typeof baseUrl === "string" &&
165232
baseUrl.length > 0 &&
166-
isSelfHostedProviderHostnameBaseUrl(baseUrl) &&
233+
isBareProviderHostnameBaseUrl(baseUrl) &&
234+
(isSelfHostedProviderId(params?.model?.provider) ||
235+
hasConfiguredLocalProviderSignal({ cfg: params?.cfg, provider: params?.model?.provider })) &&
167236
!isOllamaCloudModel(params?.model);
168237
const timeoutBounds = [
169238
runTimeoutIsNoTimeout ? undefined : runTimeoutMs,
@@ -205,7 +274,11 @@ export function resolveLlmIdleTimeoutMs(params?: {
205274
return 0;
206275
}
207276
if (params?.trigger === "cron") {
208-
if (isLocalRuntimeModel || isSelfHostedHostnameRuntimeModel) {
277+
if (
278+
isLocalRuntimeModel ||
279+
isExplicitLocalHostnameRuntimeModel ||
280+
isSelfHostedHostnameRuntimeModel
281+
) {
209282
return clampTimeoutMs(runTimeoutMs);
210283
}
211284
return clampTimeoutMs(Math.min(runTimeoutMs, CRON_LLM_IDLE_TIMEOUT_MS));

0 commit comments

Comments
 (0)