Skip to content

fix(stream): expired model lock masks account lock, and dropped generated images (#3516, #3521) - #3879

Open
ntdatt812 wants to merge 3 commits into
decolua:masterfrom
ntdatt812:bundle/stream-integrity
Open

ntdatt812 wants to merge 3 commits into
decolua:masterfrom
ntdatt812:bundle/stream-integrity

Conversation

@ntdatt812

Copy link
Copy Markdown
Contributor

Two stream-path fixes that were sitting in separate PRs (#3516, #3521), rebased onto current master. Independent commits; either can be dropped.

fix(fallback) (#3516) — an expired per-model lock masked an active account lock. The per-model entry was checked first and, once its TTL had passed, the code returned "not locked" without consulting the account-level lock that was still live, so requests were sent to an account the router had already taken out of rotation.

fix(stream) (#3521) — generated images were dropped from OpenAI-format streams. The delta handler forwarded content and reasoning fields and ignored the image parts, so a model that returned an image mid-stream produced text-only output with no error.

Not included: #3542

I tried to bundle #3542 (record the turn when a client hangs up mid-stream) and it no longer applies. master has since grown its own finalizeStream() in open-sse/utils/stream.js, called from transform on terminal events and from flush, which overlaps that PR's finishStream(). The remaining gap is real — there is still no cancel callback on the transformer, so a client that disconnects before any terminal event still records nothing — but closing it now means rewriting against the new structure rather than replaying the old patch. I will send that separately rather than force a stale diff through here.

Verification

Run with vitest 3 and the repo config — both parts matter:

npx vitest@3 run --config tests/vitest.config.js <files>

Without --config the @/ alias does not resolve (Failed to load url @/lib/dataDir.js). And vitest 2.x cannot run anything that reaches open-sse/translator/index.js: it throws TypeError: register is not a function at the top-level register(...) calls, because index.js deliberately relies on function-declaration hoisting to survive the import cycle while Vite 5's SSR transform turns the named import into a call-time property access. I confirmed that failure on clean master and on master as of 15 Aug, so it is not caused by these changes.

Every fix was checked by reverting it and confirming which test dies. No claim here rests on a test that passes for another reason.

…count lock

isModelLockActive picked the per-model key on the truthiness of the string:

  const expiry = connection[key] || connection[MODEL_LOCK_ALL];

so a stale `modelLock_<model>` won over a still-active `modelLock___all`. The
connection then read as free for exactly the model that had just failed, while
still reading as locked for every other model:

  modelLock_claude-fable-5 = <1 minute ago>
  modelLock___all          = <next month>

  isModelLockActive(conn, "claude-fable-5")  -> false   (expected true)
  isModelLockActive(conn, "some-other-model") -> true
  getEarliestModelLockUntil(conn)             -> next month

That shape is reachable and already documented by the suite: a plain GitHub 402
sets a two-minute per-model lock, and monthly exhaustion sets an account-wide one
until the next UTC month. Two minutes later the first has expired and the second
has not. Nothing clears the stale key either — the lazy cleanup only runs after a
successful request, which an exhausted account never gets.

Judge each key on its own expiry, which is the rule the sibling
getEarliestModelLockUntil already applies when it skips expired entries. An
unparseable timestamp is treated as no lock, so a malformed value cannot pin a
connection forever.
`hasValuableContent` gates every chunk on the streaming path
(open-sse/utils/stream.js:315). Its OpenAI branch listed content,
reasoning_content, tool_calls, finish_reason and role — but not images.

A generated image arrives on its own chunk with nothing else in the delta, so
every one of them was discarded:

  delta={"role":"assistant"}                              kept
  delta={"images":[{"type":"image_url","image_url":{...}}]}  DROPPED

The translator emits exactly that shape (translator/response/gemini-to-openai.js:105)
and the repo's golden snapshot locks it under "image output (inlineData ->
delta.images)" — so the filter was throwing away a payload the project had
already committed to producing.

Streaming an image-generating model through /v1/chat/completions therefore
returned a well-formed but empty assistant message. Non-streaming is unaffected.

An empty `images` array stays filtered, so a stub chunk is not promoted to
content.
Review follow-up on this PR. The per-key expiry treatment asked for was already
there -- `getEarliestModelLockUntil` skips expired entries -- but the two helpers
diverge on something else: scope.

`isModelLockActive(c, model)` is model-scoped. `getEarliestModelLockUntil(c)` is
connection-wide, so on a connection locked for two models it returns the unrelated
model's expiry. Both retry-timing call sites in auth.js used it, so a connection
locked for `claude-fable-5` until 10:00 while holding a lock for another model until
09:30 advised a retry at 09:30. The retry hits the same lock and fails identically:
one wasted upstream request per caller, surfaced as a second rate-limit error. The
debug line beside it printed `modelLocked(<model>) until <other model's expiry>`.

`getModelLockUntil(connection, model)` is the time-valued twin of
`isModelLockActive`: same two keys, same `getModelLockKey` normalisation so a null or
absent model resolves account-wide in both, and the **later** of the two gating
expiries, because the connection frees up only once both have passed.

`getEarliestModelLockUntil` is left alone. It is the right question for a
per-connection cooldown badge, and widening it on argument arity would reintroduce
the undefined-versus-null divergence this file already had one of.

Also adds the stream-end image test asked for. The existing one calls
`hasValuableContent` directly, which proves the predicate and nothing about the
stream consulting it; these drive a real Gemini->OpenAI transform with the image part
as the last thing the provider sends, where no later chunk can carry it instead.

Mutation-checked: min instead of max, scanning every `modelLock_*` key, and dropping
the account-wide gate each turn a test red. Reverting the `delta.images` line turns
all four stream tests red.
@ntdatt812
ntdatt812 force-pushed the bundle/stream-integrity branch from 3fb3ba6 to b75fcdd Compare October 29, 2026 03:05
@ntdatt812

Copy link
Copy Markdown
Contributor Author

Both addressed in b75fcdd9, rebased onto 1fe586bb (v0.5.112). Thanks for running it — one of the two asks landed somewhere slightly different from where you aimed it, and I'd rather say so than quietly patch something else.

The images test — exactly as asked

stream-keeps-trailing-generated-image.test.js, 4 tests. The existing one called hasValuableContent directly, which proves the predicate and nothing about the stream that consults it. These drive a real Gemini→OpenAI transform with the image part as the last thing the provider sends:

case asserts
text, then image last the image reaches the client and the text is still there
image is the whole response no text, no finish chunk — the case where a drop leaves a well-formed empty message
image then finish chunk present once, not duplicated
two images back to back at the end both, in order

Reverting the delta.images line turns all four red. I checked that rather than assuming it.

The lock helpers — the divergence is scope, not expiry

I went to add the per-key expiry treatment and found it is already there: getEarliestModelLockUntil has if (t <= now) continue. So that part needed nothing.

But you were pointing at something real one level over. isModelLockActive(c, model) is model-scoped; getEarliestModelLockUntil(c) is connection-wide, so on a connection locked for two models it returns the unrelated model's expiry — and both retry-timing call sites in auth.js used it:

const lockedConns = connections.filter(c => isModelLockActive(c, model));
const expiries = lockedConns.map(c => getEarliestModelLockUntil(c)).filter(Boolean);

A connection locked for claude-fable-5 until 10:00 while holding another model's lock until 09:30 therefore advised a retry at 09:30. The retry hits the same lock and fails identically — one wasted upstream request per caller, surfaced to the user as a second rate-limit error. The debug line beside it printed modelLocked(<model>) until <other model's expiry>, which is how I convinced myself it was real.

getModelLockUntil(connection, model) is the time-valued twin of isModelLockActive: the same two keys, the same getModelLockKey normalisation so a null or absent model resolves account-wide in both, and the later of the two gating expiries — the connection frees up only once both have passed, so returning the earlier one is the same bug in a different direction.

I did not widen getEarliestModelLockUntil, deliberately. "When does anything on this account free up" is the right question for a per-connection cooldown badge, and making the scope depend on whether a second argument was passed would reintroduce exactly the undefined-versus-null divergence this PR already fixes one of. Both of its callers wanted the model-scoped question and now ask for it by name.

5 new tests in model-lock-precedence.test.js, including one that walks 8 lock shapes and asserts getModelLockUntil(...) !== null equals isModelLockActive(...) for every one — so the two cannot drift apart again without a red test.

Mutation-checked, since the subtle wrong version here passes a naive test:

Mutation Result
none 10 / 10 pass
Math.min instead of Math.max 1 fail
scan every modelLock_* key (i.e. be the old helper) 1 fail
drop the account-wide gate 2 fail

Verification

82 failing tests on this branch and the same names on master-with-these-tests — comm on the sorted name lists is empty in the branch-only direction, so this adds no failure. (First run writes 138 snapshots, so only the second run's names are usable.) The 11 extra failures in the master-source run are these new tests plus the existing image one, all red without the fix, which is the bulk version of the mutation table above.

Also re-checked against current master before claiming the commits are still needed: hasValuableContent still has no images branch, and isModelLockActive is still connection[key] || connection[MODEL_LOCK_ALL]. Neither has been fixed upstream in the meantime.

The three commits remain independent — the new one is fix(fallback): scope retry timing… and can be dropped on its own if you would rather take it separately.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant