Skip to content

Fix a cold start failing when the model server loses its port to a passing connection#543

Merged
tobocop2 merged 1 commit into
mainfrom
fix/engine-port-bind-retry
Jul 16, 2026
Merged

Fix a cold start failing when the model server loses its port to a passing connection#543
tobocop2 merged 1 commit into
mainfrom
fix/engine-port-bind-retry

Conversation

@tobocop2

Copy link
Copy Markdown
Owner

Problem

A model server could fail its first request with "The model server is not responding and no healthy replica is available" on a machine with busy ports. The fleet picks each member's port at start, but llama-swap only spawns the member on its first request, so the port sits unclaimed for the whole wait and a passing connection can take it. The server exits "couldn't bind HTTP server socket", and the client wrote the replica off as dead even though the next spawn binds fine seconds later. This failed a release verify run's ingest.

Solution

When the dead member's captured output shows the bind failure, classify the death as transient instead of dead-replica, so the existing busy-retry backoff re-drives llama-swap's spawn. Every other exit reason keeps the failover path.

The fleet picks each member's port at start, but llama-swap only spawns
the member on its first request, so the port sits unclaimed for the
whole lazy wait and a passing ephemeral connection can take it. The
server then exits "couldn't bind HTTP server socket", llama-swap masks
it as "exited prematurely", and the client classified every such death
as CONNECTION: the sole replica got written off as unhealthy and the
request failed with "no healthy replica", even though llama-swap
re-spawns on the next request and normally binds fine (the occupier is
gone within seconds).

When the upstream's captured tail shows the bind failure, classify the
death as SERVER instead, so the existing retry_on_busy backoff re-drives
the spawn. Any other exit reason keeps the CONNECTION failover path.
@tobocop2
tobocop2 merged commit 24d893c into main Jul 16, 2026
11 of 17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant