Wait out a cold embed replica instead of dropping files on 429#535
Merged
Conversation
A bulk 'lilbee add' immediately after pulling an embedder silently dropped
documents when the embedding llama-server was still warming: every embed
request returned HTTP 429 ('replicas may still be warming'), and after the
short interactive retry budget (~80s) each file was written to skip_reasons.json
as a terminal failure. A cold-start burst of many files dropped whichever ones
did not warm within that window, leaving a silently partial index where a query
for a skipped document returns a false grounded refusal.
The fleet already holds the invariant that a model whose load llama-swap will
wait out can never time out the client first, but the embed busy-retry violated
it: llama-swap keeps a still-loading replica alive for the cold-load budget
(>=600s), while the embed path gave up after ~80s. Give the EMBED-role client a
cold-load deadline sized to that same budget, so a bulk ingest waits the replica
out rather than dropping files. Rerank and the other roles keep the short
interactive budget.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Running
lilbee add <dir>right after pulling an embedder silently dropped files while the embed server was still warming.Every dropped file hit HTTP 429 ("replicas may still be warming") and got written off as a permanent failure. You end up with a partial index and no error, so a query for a dropped file returns "no information" even though it's there.
Cause
The embed retry gave up after ~80s. But llama-swap keeps a loading replica alive for much longer (600s+), so we were dropping files the server was still busy loading.
Fix
Let the embed retry wait out the full load window instead of quitting early. The first request absorbs the warmup, then the rest of the batch goes through.
Rerank still fails fast, so a query never hangs on a cold rerank server.