You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 1dd45f8
Browse filesBrowse the repository at this point in the historyBrowse files
feat(sandbox,router,cli): honor route timeout on streaming chunk-idle
The L7 inference proxy's per-chunk idle timeout was a hardcoded 120s,
ignoring `route.timeout`. Workloads with SSE gaps longer than 120s
(large prefills, tool-call-buffering parsers like qwen3_coder) were
aborted even when the backend was healthy and the operator had
explicitly raised `--timeout`.
Derive the effective per-chunk idle timeout from `route.timeout`,
floored at 120s to avoid regressing default behavior. Operators raise
it with the existing `openshell inference set --timeout <secs>` flag —
no new CLI surface. The floor keeps the current default intact when
operators don't tune the route.
- openshell-router: add `DEFAULT_CHUNK_IDLE_TIMEOUT` +
`effective_chunk_idle_timeout()` in config.rs, plus a
`chunk_idle_timeout` field on `StreamingProxyResponse`. Populated in
both the live and mock response paths.
- openshell-sandbox: proxy.rs reads `resp.chunk_idle_timeout` instead
of the deleted constant. OCSF truncation event now includes the
effective value so operators can see which timeout fired.
- openshell-cli: clarify `--timeout` help text on `inference set` /
`inference update` — also raises the streaming chunk-idle floor.
- Tests: pure-function tests for `effective_chunk_idle_timeout` and
integration tests exercising both the floor and the above-floor
scaling on live + mock streaming paths.
Refs: NVIDIA#866
0 commit comments