Skip to content

Commit c108489

Browse files
chethanukbaptmont
andauthored
fix(llminternal): guard RunLive errChan sends so abandoned producers exit on reconnect (#1167)
* fix(llminternal): guard RunLive errChan sends so abandoned producers exit on reconnect RunLive's reader and sender goroutines did bare sends on the unbuffered, per-attempt errChan. Once the consumer loop stops reading that channel -- after a resumable-error reconnect, a parent context cancellation, or any other teardown -- cleanup() closes the connection, the producer's in-flight call fails, and its send blocks forever, leaking one goroutine (plus its captured connection state) per occurrence. Long-lived /run_live servers accumulate these across sessions and reconnects. Guard all three sends with the same select-on-connCtx idiom the function already uses for eventsChan. cleanup() always cancels connCtx before the next attempt (and before returning), so a guarded send is guaranteed to unblock. connCtx, not ctx: ctx outlives reconnects and would not unblock an abandoned producer. The regression test fakes the Gemini Live wire protocol over a local websocket server and fails deterministically on the unfixed code with goroutines parked in 'chan send' at the bare-send sites. Fixes #1152 * test(llminternal): cover the SendRealtime guard and stop the leak detector failing open Addresses review feedback on the RunLive errChan leak fix. runLiveStacks sized its buffer once at 1 MiB and never checked for truncation. runtime.Stack truncates silently, so a dump larger than the buffer would under-count parked goroutines and let a real leak pass -- the wrong failure direction for a leak detector. Grow until the whole dump fits. Add a case driving RealtimeInput across a connection loss. Reverting only the SendRealtime guard left every existing case green, so that third errChan send site was uncovered; nothing in the suite populated LiveRequest.RealtimeInput. Each of the three guards is now caught individually when reverted. Rename "non-resumable error terminates" and record why it passes with or without the guards: the sender exits through the existing connCtx.Done() arm, so nothing is stranded. It pins terminal-path behaviour rather than guarding the leak regression. --------- Co-authored-by: westerberg <[email protected]>
1 parent d1fd187 commit c108489

2 files changed

Lines changed: 409 additions & 3 deletions

File tree

‎internal/llminternal/base_flow.go‎

Lines changed: 15 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -370,6 +370,9 @@ func (f *Flow) RunLive(ctx agent.InvocationContext) (agent.LiveSession, iter.Seq
370370
}
371371

372372
eventsChan := make(chan *session.Event)
373+
// Producers must guard sends with connCtx: once a reconnect (or any
374+
// teardown) abandons this unbuffered channel, cleanup's cancelConn is
375+
// what unblocks them (issue #1152).
373376
errChan := make(chan error)
374377

375378
// Send preprocessed content directly to model if any exists after early preprocessing
@@ -386,7 +389,10 @@ func (f *Flow) RunLive(ctx agent.InvocationContext) (agent.LiveSession, iter.Seq
386389
for {
387390
resp, err := liveConn.Recv(connCtx)
388391
if err != nil {
389-
errChan <- err
392+
select {
393+
case errChan <- err:
394+
case <-connCtx.Done():
395+
}
390396
return
391397
}
392398
if resp != nil {
@@ -427,7 +433,10 @@ func (f *Flow) RunLive(ctx agent.InvocationContext) (agent.LiveSession, iter.Seq
427433
}
428434
if req.Content != nil {
429435
if err := liveConn.SendContent(connCtx, req.Content); err != nil {
430-
errChan <- err
436+
select {
437+
case errChan <- err:
438+
case <-connCtx.Done():
439+
}
431440
return
432441
}
433442
}
@@ -436,7 +445,10 @@ func (f *Flow) RunLive(ctx agent.InvocationContext) (agent.LiveSession, iter.Seq
436445
sess.audioMgr.CacheInput(ctx, blob.Data, blob.MIMEType)
437446
}
438447
if err := liveConn.SendRealtime(connCtx, req.RealtimeInput); err != nil {
439-
errChan <- err
448+
select {
449+
case errChan <- err:
450+
case <-connCtx.Done():
451+
}
440452
return
441453
}
442454
}

0 commit comments

Comments
 (0)