Repository navigation
Description
Summary
On a serve node, the OpenAI ingress (:9337) serializes all HTTP — including control-plane reads (/v1/models) and the lock-free-looking health path — behind the current in-flight inference. A single large-prompt turn on a big model makes the ingress unresponsive to any HTTP for the full duration of that turn, even though the process is healthy and the TCP listener keeps accepting instantly.
This breaks supervision, readiness, and discovery for any embedder that polls the ingress: a busy node is indistinguishable from a dead one over HTTP.
Evidence (black-box, mesh-llm v0.74.0, e60b2fe)
Standalone node, single GPU (Apple M5 Max), model unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M:
mesh-llm serve --model unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M --port 9437 --console 3141
Idle: every endpoint answers in <10 ms.
During one ~30s inference (randomized ~30k-token prompt so the prefix/KV cache cannot hit), fired concurrently:
| concurrent request | result |
|---|---|
GET /v1/models |
27.0 s, HTTP 200 |
POST /v1/chat/completions (max_tokens:1) |
28.8 s, HTTP 200 |
tcp_connect(:9437) throughout |
~0 ms |
Both requests queued behind the in-flight turn and completed the instant it finished — they were never errors, just head-of-line blocked. A probe with any timeout < the turn length reads the node as down.
Repeated with a probe loop at a 3 s timeout: /v1/models returned 000 (timeout) for 5 consecutive probes across the prefill window, then 200 immediately after.
Root cause (from source, e60b2fe)
skippy-server's request handler holds Arc<std::sync::Mutex<RuntimeState>> across the whole compute:
// crates/skippy-server/src/http.rs (message handler)
{
let mut runtime = runtime.lock().expect("runtime lock poisoned");
for _ in 0..request.max_new_tokens {
current = runtime.decode(&request.session_id, current)?; // + prefill above
generated_token_ids.push(current);
}
}and the mesh ingress /v1/models handler needs node state via async locks that contend with the same request lifecycle (crates/mesh-llm-host-runtime/src/network/openai/ingress.rs::handle_models_list_request → models_being_served().await / all_served_model_descriptors().await). Net effect: control-plane reads cannot make progress while a turn holds the runtime.
A stack sample during the stall showed 17/19 tokio workers parked and one worker inside llama_context::process_ubatch holding the runtime mutex — i.e. the control plane is blocked on the compute path, not starved for threads.
Impact
Any embedder supervising a node over the ingress (e.g. Buzz desktop's shared-compute watchdog + startup readiness probe) will periodically classify a healthy, busy, or still-loading node as dead. We've worked around it on our side by treating a bound TCP port as "alive" and never restarting on HTTP unresponsiveness — but that means we currently cannot detect a genuinely wedged-but-bound node, because there is no liveness signal that bypasses the inference lock.
Requested fix (ranked)
- Lock-free
GET /live(and/ready) on the ingress that never touchesRuntimeStateor the inference path —/live= process/listener up;/ready= an atomic lifecycle/inflight snapshot (Idle|Downloading|Loading|Ready,inflight_requests). This is the smallest change that makes supervision correct. - Serve
/v1/modelsand/api/statusfrom an immutable/ArcSwapsnapshot that the inference mutex never guards, so catalog/status reads don't head-of-line-block behind a turn. - (Not requested — noted only) Serializing inference behind one turn is acceptable for a single-GPU node; serializing the control plane behind it is the bug.
Happy to test any of these against the repro above.
Activity
michaelneale commentedon Jul 31, 2026
Update: I implemented the proposed lock-free
/healthand it does NOT fix this. Sharing so nobody re-implements the dud.I built the "add a lock-free
/live//healthon the ingress" fix (ranked #1 above) on top ofmainand tested it live against a real prefill. It stalls exactly like/v1/models. So the head-of-line block is upstream of request dispatch, not in the endpoint handlers — a new route on the same ingress server is not enough.What I built
Added a
GET /health|/healthz|/live|/livezbranch as the first thing inmaybe_handle_control_request(beforeis_models_list_request, before any proxy-to-backend), returning{"status":"ok"}via the puresend_json_ok(formats a string, writes the socket, touches no node/runtime/model state). Confirmed compiled into the binary (/healthz//livezstrings present; host-runtime recompiled).Live result (patched build, standalone
serve, gemma-4-E4B, randomized ~30k-token prompt → cache-miss full prefill)Idle:
During the ~21s prefill:
Immediately after the prefill finished, both returned to <1ms/200. The
/healthroute is correct (works idle), it simply never gets served during a turn.What this tells us
A stack sample during the stall showed ~21 tokio workers parked (idle), ~3 in
llama/ggml/decode. So it is not worker-thread starvation, and it is not theRuntimeStatemutex I speculated about earlier (a lock-free handler that never takes it still stalls). The connection is accepted at the kernel level (hence instanttcp_connect) but the request does not reach the dispatch branch while a turn is in flight — the block is in the accept→per-connection path (accept-loop scheduling, or the per-connection preamblenode.plugin_manager().await/read_http_request_*), i.e. beforemaybe_handle_control_request.I was not able to localize the exact upstream point without adding tracing spans and rebuilding, which I'll leave to maintainers who can instrument it faster. But the practical conclusions are firm:
tcp_connectis the only reliable "process alive" signal — it stays ~0ms throughout a turn. Any HTTP probe (including/health) will time out during normal inference and must not be read as death. (That is the mitigation we shipped on the Buzz side.)Happy to run more experiments against the repro if a maintainer wants a specific hypothesis tested.
michaelneale commentedon Aug 1, 2026
Root cause located:
RuntimeStateis held under astd::sync::Mutexfor the whole decode loopFollowing up my previous comment (where the lock-free
/healthroute I added did not fix this). I captured a stack sample of a real node mid-stall and demangled it. This is more specific and more actionable than the "head-of-line blocking" framing I opened with.The contention, from a live sample
Sampled during a ~21s prefill/decode on a
servenode. One of the node's own internal loops was caught blocked:Thread census at that instant:
llama/ggml/decode (holding the lock)Mutex<RuntimeState>::lock75 idle workers. So this is not executor starvation or a thread-pool shortage — it is straightforward contention on a single mutex.
Why
crates/skippy-server/src/http.rstakes the guard outside the generation loop and holds it across every token:The same
RuntimeStatemutex is whatSkippyRuntimeHandle::status()needs. So for the full duration of a turn, anything that wants runtime status/usage/readiness blocks — including the node's own dashboard loop, as sampled above.Honest limits of this evidence
/v1/modelsalso takesnode.state/plugin_managerasync locks, and the accept path is a third candidate. The black-box symptom is certain and reproducible; the precise chain for HTTP is not yet pinned. That's the remaining piece for whoever picks this up, and adding a tracing span around the lock acquisition inSkippyRuntimeHandle::status+ the ingress preamble would settle it quickly.Suggested direction (superseding my original ranked list)
RuntimeStateacross the whole decode loop. Acquire per token, or move generation behind a channel/actor so the state lock is only taken for short reads. This is the actual defect.ArcSwapor a small atomic set: lifecycle, inflight, ctx-used) that the inference path publishes to, so control-plane reads never touch the inference mutex at all./liveroute" — is not sufficient on its own, as I demonstrated; please don't start there.Repro is cheap if useful:
mesh-llm servea large model, fire a randomized ~30k-token prompt (cache-miss), and poll any status endpoint with a 3s timeout while it runs.michaelneale commentedon Aug 1, 2026
Opened #1129 with a fix for the status path.
Root cause turned out to be narrower and more concrete than my earlier comments:
SkippyRuntimeHandle::status()locked the sameMutex<RuntimeState>that the decode loop holds for a whole generation, purely to readsession_stats(). The dashboard tick calls into it every 250ms from an async task, so the blocking acquire also parked tokio workers for the length of a turn.Measured on the repro from this issue (gemma-4-E4B, randomized ~30k-token prompt, ~25s prefill):
/v1/models/api/statusInference unaffected; status stays accurate mid-turn. Note this also clears
/v1/models, which uses async locks rather than that mutex — consistent with the blocking acquire having starved executor threads rather than/v1/modelscontending on the mutex directly.Also worth recording that my earlier suggestion in this thread — add a lock-free
/healthroute on the ingress — did not work; I built and live-tested it and it stalled identically. The lock-free route was treating the symptom at the wrong layer.fix(mesh): stop restarting a busy or loading shared-compute node (#3909)
fix(skippy): never block status reads on the inference lock (Mesh-LLM…
fix(mesh): stop restarting a busy or loading shared-compute node (blo…