Skip to content

Serve record stays "ready" through an in-process engine relaunch, so a supervisor cannot tell the relaunch from a runaway load #700

Description

@pcvelz

Splash 1.6.0 (first measured on 1.4.0), M4 Pro 48 GB, macOS 27.2, Qwen3.8-27B, Splash behind a supervisor.

What a supervisor in front of Splash does, and what it reads. On one Mac several engines serve several models to several clients, and a supervisor sits in front of them: it swaps models so the one asked for is loaded and the others are not; it admits requests against memory, holding a request while a load would not fit, waiting for memory before it starts a load, or picking another model; it stops a child whose memory grows while that child should be serving, because on a 48 GB Mac that growth is how a second model sneaks in; and it restarts a child that exited. Every one of those decisions is read from the engines' phases: loading, ready, failed. A relaunch the server starts inside its own ready phase is invisible to all of them: the supervisor reserves no memory for it, holds no requests for it, and reads its growth as a runaway.

What the record says. <runtime>/serve-PORT.json follows the first start: starting, loading, warming, ready, failed. After ready nothing writes it again: on_warming only fires before the first ready (runtime.py, _first_ready), and ready is written once in main. So an engine that fails after it served and relaunches in-process loads a whole model under a record that still says ready, same pid. /status does say it: transport.recovering is true from the failure until the new engine is ready.

What that looks like. Per-second memory history of the supervisor, 2026-10-24:

time what happened
12:54:15 Splash: Engine failed · native loop did not answer status within 30 s
12:54:30 to 12:54:33 old engine exits: wired 38.5 to 10.7 GB
12:54:35 to 12:54:37 in-process relaunch loads: wired 7.2 to 13.3 GB, record says ready
12:54:37 brake stops it: growth 11.3 GB in the window
12:55:39 to 12:56:10 control: the same load, started by the supervisor itself, growth 11.1 GB, brake disarmed, no trip
13:01:27 to 13:01:48 second failure, same sequence, stopped again

Both relaunches were stopped; the identical load started by the supervisor went through. The only difference is who started the load and what the record said during it.

Proposal. Let the record follow the engine's life, not only its first start: at a relaunch write recovering (or the same loading and warming phases) and ready again when the relaunched engine serves, same pid. /status already carries this as transport.recovering, so this is the file-side twin for a follower that does not poll HTTP, the way the desktop app follows the file. A supervisor that polls can read /status meanwhile, and ours will.

With this, #617 (--engine-failure exit) is not needed: the server keeps the restart decision, the supervisor only needs to see it. I'll say so there.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions