Skip to content

Ingress control plane (/v1/models, health) head-of-line blocks behind in-flight inference #1126

Closed

Description

@micspiral

Summary

On a serve node, the OpenAI ingress (:9337) serializes all HTTP — including control-plane reads (/v1/models) and the lock-free-looking health path — behind the current in-flight inference. A single large-prompt turn on a big model makes the ingress unresponsive to any HTTP for the full duration of that turn, even though the process is healthy and the TCP listener keeps accepting instantly.

This breaks supervision, readiness, and discovery for any embedder that polls the ingress: a busy node is indistinguishable from a dead one over HTTP.

Evidence (black-box, mesh-llm v0.74.0, e60b2fe)

Standalone node, single GPU (Apple M5 Max), model unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M:

mesh-llm serve --model unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M --port 9437 --console 3141

Idle: every endpoint answers in <10 ms.

During one ~30s inference (randomized ~30k-token prompt so the prefix/KV cache cannot hit), fired concurrently:

concurrent request result
GET /v1/models 27.0 s, HTTP 200
POST /v1/chat/completions (max_tokens:1) 28.8 s, HTTP 200
tcp_connect(:9437) throughout ~0 ms

Both requests queued behind the in-flight turn and completed the instant it finished — they were never errors, just head-of-line blocked. A probe with any timeout < the turn length reads the node as down.

Repeated with a probe loop at a 3 s timeout: /v1/models returned 000 (timeout) for 5 consecutive probes across the prefill window, then 200 immediately after.

Root cause (from source, e60b2fe)

skippy-server's request handler holds Arc<std::sync::Mutex<RuntimeState>> across the whole compute:

// crates/skippy-server/src/http.rs  (message handler)
{
    let mut runtime = runtime.lock().expect("runtime lock poisoned");
    for _ in 0..request.max_new_tokens {
        current = runtime.decode(&request.session_id, current)?;   // + prefill above
        generated_token_ids.push(current);
    }
}

and the mesh ingress /v1/models handler needs node state via async locks that contend with the same request lifecycle (crates/mesh-llm-host-runtime/src/network/openai/ingress.rs::handle_models_list_request → models_being_served().await / all_served_model_descriptors().await). Net effect: control-plane reads cannot make progress while a turn holds the runtime.

A stack sample during the stall showed 17/19 tokio workers parked and one worker inside llama_context::process_ubatch holding the runtime mutex — i.e. the control plane is blocked on the compute path, not starved for threads.

Impact

Any embedder supervising a node over the ingress (e.g. Buzz desktop's shared-compute watchdog + startup readiness probe) will periodically classify a healthy, busy, or still-loading node as dead. We've worked around it on our side by treating a bound TCP port as "alive" and never restarting on HTTP unresponsiveness — but that means we currently cannot detect a genuinely wedged-but-bound node, because there is no liveness signal that bypasses the inference lock.

Requested fix (ranked)

  1. Lock-free GET /live (and /ready) on the ingress that never touches RuntimeState or the inference path — /live = process/listener up; /ready = an atomic lifecycle/inflight snapshot (Idle|Downloading|Loading|Ready, inflight_requests). This is the smallest change that makes supervision correct.
  2. Serve /v1/models and /api/status from an immutable/ArcSwap snapshot that the inference mutex never guards, so catalog/status reads don't head-of-line-block behind a turn.
  3. (Not requested — noted only) Serializing inference behind one turn is acceptable for a single-GPU node; serializing the control plane behind it is the bug.

Happy to test any of these against the repro above.

Activity

  1. michaelneale commented on Jul 31, 2026

    @michaelneale
    Collaborator

    Update: I implemented the proposed lock-free /health and it does NOT fix this. Sharing so nobody re-implements the dud.

    I built the "add a lock-free /live//health on the ingress" fix (ranked #1 above) on top of main and tested it live against a real prefill. It stalls exactly like /v1/models. So the head-of-line block is upstream of request dispatch, not in the endpoint handlers — a new route on the same ingress server is not enough.

    What I built

    Added a GET /health|/healthz|/live|/livez branch as the first thing in maybe_handle_control_request (before is_models_list_request, before any proxy-to-backend), returning {"status":"ok"} via the pure send_json_ok (formats a string, writes the socket, touches no node/runtime/model state). Confirmed compiled into the binary (/healthz//livez strings present; host-runtime recompiled).

    Live result (patched build, standalone serve, gemma-4-E4B, randomized ~30k-token prompt → cache-miss full prefill)

    Idle:

    /health     -> 200 in 0.0007s
    /v1/models  -> 200 in 0.001s
    

    During the ~21s prefill:

    tcp_connect(:9447) -> 0.2ms  OK        (process alive, port bound)
    GET /health        -> 3s TIMEOUT (000) (my lock-free route)
    GET /v1/models     -> 3s TIMEOUT (000)
    

    Immediately after the prefill finished, both returned to <1ms/200. The /health route is correct (works idle), it simply never gets served during a turn.

    What this tells us

    A stack sample during the stall showed ~21 tokio workers parked (idle), ~3 in llama/ggml/decode. So it is not worker-thread starvation, and it is not the RuntimeState mutex I speculated about earlier (a lock-free handler that never takes it still stalls). The connection is accepted at the kernel level (hence instant tcp_connect) but the request does not reach the dispatch branch while a turn is in flight — the block is in the accept→per-connection path (accept-loop scheduling, or the per-connection preamble node.plugin_manager().await / read_http_request_*), i.e. before maybe_handle_control_request.

    I was not able to localize the exact upstream point without adding tracing spans and rebuilding, which I'll leave to maintainers who can instrument it faster. But the practical conclusions are firm:

    1. A lock-free route on the existing OpenAI ingress server does not solve this. A trustworthy liveness signal needs a genuinely independent accept path (separate listener/task, ideally its own runtime), or the upstream serialization itself must be fixed so control-plane connections aren't held behind an in-flight turn.
    2. For supervisors today: tcp_connect is the only reliable "process alive" signal — it stays ~0ms throughout a turn. Any HTTP probe (including /health) will time out during normal inference and must not be read as death. (That is the mitigation we shipped on the Buzz side.)

    Happy to run more experiments against the repro if a maintainer wants a specific hypothesis tested.

  2. michaelneale commented on Aug 1, 2026

    @michaelneale
    Collaborator

    Root cause located: RuntimeState is held under a std::sync::Mutex for the whole decode loop

    Following up my previous comment (where the lock-free /health route I added did not fix this). I captured a stack sample of a real node mid-stall and demangled it. This is more specific and more actionable than the "head-of-line blocking" framing I opened with.

    The contention, from a live sample

    Sampled during a ~21s prefill/decode on a serve node. One of the node's own internal loops was caught blocked:

    mesh_llm_host_runtime::runtime::startup_handles::startup_local_model_loop
      → runtime::dashboard::refresh_dashboard_context_usage
      → runtime::local::LocalRuntimeModelHandle::ctx_used_tokens
      → inference::skippy::SkippyModelHandle::status
      → skippy_server::embedded::SkippyRuntimeHandle::status
      → std::sync::Mutex<skippy_server::runtime_state::RuntimeState>::lock
      → pthread_mutex_firstfit_lock_slow
      → pthread_mutex_firstfit_lock_wait        ← BLOCKED
    

    Thread census at that instant:

    count
    threads in llama/ggml/decode (holding the lock) 7
    threads blocked on Mutex<RuntimeState>::lock 1 (+2 more std-mutex waiters)
    tokio workers parked/idle 75

    75 idle workers. So this is not executor starvation or a thread-pool shortage — it is straightforward contention on a single mutex.

    Why

    crates/skippy-server/src/http.rs takes the guard outside the generation loop and holds it across every token:

    {
        let mut runtime = runtime.lock().expect("runtime lock poisoned");
        for _ in 0..request.max_new_tokens {
            current = runtime.decode(&request.session_id, current)?;
            generated_token_ids.push(current);
        }
    }

    The same RuntimeState mutex is what SkippyRuntimeHandle::status() needs. So for the full duration of a turn, anything that wants runtime status/usage/readiness blocks — including the node's own dashboard loop, as sampled above.

    Honest limits of this evidence

    • I proved the contention for the status path (sampled, demangled, above).
    • I did not prove the same lock is what stalls the HTTP ingress path specifically — my sample caught 0 HTTP-serve frames (no request was in a handler at sample time). /v1/models also takes node.state/plugin_manager async locks, and the accept path is a third candidate. The black-box symptom is certain and reproducible; the precise chain for HTTP is not yet pinned. That's the remaining piece for whoever picks this up, and adding a tracing span around the lock acquisition in SkippyRuntimeHandle::status + the ingress preamble would settle it quickly.

    Suggested direction (superseding my original ranked list)

    1. Don't hold RuntimeState across the whole decode loop. Acquire per token, or move generation behind a channel/actor so the state lock is only taken for short reads. This is the actual defect.
    2. Serve status/usage/readiness from a snapshot (ArcSwap or a small atomic set: lifecycle, inflight, ctx-used) that the inference path publishes to, so control-plane reads never touch the inference mutex at all.
    3. My earlier suggestion — "add a lock-free /live route" — is not sufficient on its own, as I demonstrated; please don't start there.

    Repro is cheap if useful: mesh-llm serve a large model, fire a randomized ~30k-token prompt (cache-miss), and poll any status endpoint with a 3s timeout while it runs.

  3. michaelneale commented on Aug 1, 2026

    @michaelneale
    Collaborator

    Opened #1129 with a fix for the status path.

    Root cause turned out to be narrower and more concrete than my earlier comments: SkippyRuntimeHandle::status() locked the same Mutex<RuntimeState> that the decode loop holds for a whole generation, purely to read session_stats(). The dashboard tick calls into it every 250ms from an async task, so the blocking acquire also parked tokio workers for the length of a turn.

    Measured on the repro from this issue (gemma-4-E4B, randomized ~30k-token prompt, ~25s prefill):

    during prefill before after
    /v1/models 3009ms timeout 0.5–9ms · 200
    /api/status 3002ms timeout 2.4–3.4ms · 200

    Inference unaffected; status stays accurate mid-turn. Note this also clears /v1/models, which uses async locks rather than that mutex — consistent with the blocking acquire having starved executor threads rather than /v1/models contending on the mutex directly.

    Also worth recording that my earlier suggestion in this thread — add a lock-free /health route on the ingress — did not work; I built and live-tested it and it stalled identically. The lock-free route was treating the symptom at the wrong layer.

  4. added a commit that references this issue on Aug 1, 2026
    7900758
  5. added theissue type on Sep 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions