Summary
When a burst of inbound messages hits the followup queue cap while the head item is mid-delivery (checked out by an awaited run(next) but not yet removed from the shared items array), applyQueueDropPolicy can select that same in-flight item as an overflow victim. With the default dropPolicy: "summarize", the in-flight item ends up recorded in the overflow summary as dropped even though it is still being delivered and will complete normally a moment later, producing a contradictory record where the same message is both answered and reported as unanswered-due-to-overflow.
Environment
- Commit: 66e4f48 (origin/main at time of filing)
- Surface:
src/utils/queue-helpers.ts (applyQueueDropPolicy, drainNextQueueItem), consumed by src/auto-reply/reply/queue/enqueue.ts (enqueueFollowupRun) and src/auto-reply/reply/queue/drain.ts (collect-merge drain path); affects any channel using the default followup reply queue (default cap 20, default dropPolicy summarize, see src/auto-reply/reply/queue/state.ts:40-41).
Steps to reproduce
- A followup queue has one item (
m1) already checked out for delivery: drainNextQueueItem read items[0] and is awaiting run(m1). Per queue-helpers.ts:180-192, m1 is not removed from the shared items array until run resolves.
- While that await is still pending, seven more inbound messages arrive and each call
enqueueFollowupRun synchronously (src/auto-reply/reply/queue/enqueue.ts:94-197), which calls applyQueueDropPolicy against the cap (queue cap 3 in the minimal repro below, but any cap reproduces this once inbound volume plus in-flight delay reaches it).
applyQueueDropPolicy always splices from the front of items by count (queue-helpers.ts:115-116: const dropCount = items.length - cap + 1; const dropped = items.splice(0, dropCount);), with no check for whether items[0] is the item currently checked out by the in-flight drain. It selects m1 (still mid-delivery) as one of the drop victims.
onDrop runs for m1 immediately: with dropPolicy: "summarize" it is appended to the overflow summary lines (enqueue.ts:136-139); with any other dropPolicy it calls completeFollowupRunLifecycle(m1) immediately (enqueue.ts:141-143).
- The original awaited
run(m1) then resolves normally a moment later and m1 is delivered as usual.
Expected
An item that is already checked out and being delivered by an in-flight drainNextQueueItem (or the collect-merge drainGroup path in drain.ts) should never be selectable as an overflow-drop victim. It should be delivered exactly once and should never also appear in the overflow-summary record or trigger an early lifecycle completion.
Actual
The in-flight item is delivered normally AND is recorded as dropped/complete before that delivery finishes. This is not hypothetical: the repository's own existing test at src/utils/queue-helpers.test.ts, describe("drainNextQueueItem"), test "keeps overflow survivors when the queue mutates during an awaited drain" (lines 167-215), already builds exactly this scenario and passes today with assertions that show the contradiction directly:
expect(delivered).toEqual(["m1", "m6", "m7", "m8"]);
expect(dropped).toEqual(["m1", "m2", "m3", "m4", "m5"]);
Note "m1" appears in both delivered and dropped. This test currently passes on main, meaning current shipped behavior deliberately produces this contradictory outcome and the suite treats it as expected. The contradiction itself is the bug: the assertions are internally consistent with the current implementation, but the implementation is user-facing-incorrect. Fixing this will require both a logic change in applyQueueDropPolicy/drainNextQueueItem and updating this test's assertions to the corrected expected values once fixed.
Root cause
drainNextQueueItem keeps the item it is running physically present in the shared items array for the entire await run(next) duration, removing it only after the run resolves via removeQueuedItemsByRef (queue-helpers.ts:180-192) - this by-reference removal itself is the fix from PR #91450, which closed a worse bug (items.shift() removing whatever was at index 0 by the time the await resolved, causing total silent loss of a different message). The same in-flight-retention pattern holds for the collect-merge path in drain.ts: activeGroupItems stay present in queue.items for the whole await drainGroup() (drain.ts:1169-1195), removed only afterward by consumeAdmittedGroup or completeGroup.
Meanwhile enqueueFollowupRun (enqueue.ts:94-197) is fully synchronous and runs on every inbound message with no check on queue.draining, mutating the very same items array reference returned by getFollowupQueue for as long as the queue exists (src/auto-reply/reply/queue/state.ts:100-138, existing queue branch at lines 101-109 returns the same object).
applyQueueDropPolicy (queue-helpers.ts:101-130) always splices overflow victims from the front by count with zero awareness that index 0 may currently be checked out for delivery:
const dropCount = params.queue.items.length - cap + 1;
const dropped = params.queue.items.splice(0, dropCount);
params.onDrop?.(dropped);
onDrop in enqueueFollowupRun then either pushes the in-flight item straight into the overflow summary (enqueue.ts:136-139, the default summarize policy per state.ts:41) or calls completeFollowupRunLifecycle on it immediately (enqueue.ts:141-143), while the real run for that same item object is still executing and will still deliver it a few lines later.
Impact by policy:
- Default
dropPolicy: "summarize": the in-flight item's content is recorded as part of "[Queue overflow] Dropped N messages" (queue-helpers.ts:232-253) and that summary text is fed into a later prompt, even though the same item was already fully answered. This is silently incorrect with no log or warning anywhere.
- Non-summarize
dropPolicy: completeFollowupRunLifecycle fires on the in-flight item before its real run finishes, permanently marking that lifecycle complete in the completedFollowupLifecycles WeakSet (src/auto-reply/reply/queue/types.ts:178, 201-208). The real completion call later in src/auto-reply/reply/followup-runner.ts:551 or :1763 becomes a silent no-op because the WeakSet already has the lifecycle, so hooks such as Gateway's completeQueuedChatTurn (wired at src/gateway/server-methods/chat.ts:4851) fire too early and never fire again when the run actually finishes.
Preconditions
A burst of inbound messages reaches or exceeds the queue cap (default 20, or lower if operator-configured) while a longer-running agent turn is in flight and checked out at index 0 of the shared items array. This is realistic during any longer agent turn combined with a message burst on a tightly-capped channel.
Sibling surfaces
The collect-merge drain path in drain.ts shares the same in-flight-retention pattern (activeGroupItems stay in queue.items through await drainGroup()), so the same class of overlap is reachable there too, not only through the single-item drainNextQueueItem path exercised by the cited test.
Related work
PR #91450 (merged) fixed a related but more severe sibling bug: drainNextQueueItem used to remove the drained item with items.shift(), which could remove the wrong item entirely (silent total loss of a different, still-queued message) if the array had been mutated during the await. That PR's fix, changing removal to by-reference (removeQueuedItemsByRef), is exactly what keeps the in-flight item physically present in items for the drop-policy code to mistakenly select in the scenario described here. The PR's own diff added the queue-helpers.test.ts test cited above to lock in its fix, and that same test's assertions are the evidence for this remaining, distinct issue. PR #91450 does not address the overlap between drop-policy eviction and in-flight delivery; this is a follow-up gap, not a regression from that change.
Real behavior proof
Behavior addressed: applyQueueDropPolicy can select an item that is currently checked out for in-flight delivery, so it ends up simultaneously delivered and recorded as an overflow drop.
Real environment tested: real production functions applyQueueDropPolicy and drainNextQueueItem from src/utils/queue-helpers.ts, executed directly (no framework harness, no mocks) via tsx against the checkout at commit 66e4f48 (origin/main).
Exact steps or command run after this patch: node_modules/.bin/tsx repro.ts, where repro.ts imports applyQueueDropPolicy and drainNextQueueItem directly from src/utils/queue-helpers.ts and reproduces the same sequence as the scenario above: one item (m1) drained and held mid-await, seven more items (m2..m8) enqueued through applyQueueDropPolicy against a cap of 3 while m1 is still in flight, then the gate is released and remaining items drained.
Evidence after fix:
{
"delivered": ["m1", "m6", "m7", "m8"],
"dropped": ["m1", "m2", "m3", "m4", "m5"],
"remainingQueueItems": [],
"messagesRecordedAsBothDeliveredAndDropped": ["m1"]
}
Observed result after fix: m1 is present in both the delivered list and the dropped/overflow list, produced directly by driving the real applyQueueDropPolicy and drainNextQueueItem functions with no patch applied, confirming the same message is simultaneously reported as answered and as dropped-due-to-overflow on current origin/main.
What was not tested: the collect-merge drain path in drain.ts (activeGroupItems / drainGroup) was inspected by reading code only and not independently reproduced with a standalone run; the completeFollowupRunLifecycle early-completion effect on Gateway's completeQueuedChatTurn hook was traced by reading code only and not reproduced end to end through a live gateway session.
Summary
When a burst of inbound messages hits the followup queue cap while the head item is mid-delivery (checked out by an awaited
run(next)but not yet removed from the shareditemsarray),applyQueueDropPolicycan select that same in-flight item as an overflow victim. With the defaultdropPolicy: "summarize", the in-flight item ends up recorded in the overflow summary as dropped even though it is still being delivered and will complete normally a moment later, producing a contradictory record where the same message is both answered and reported as unanswered-due-to-overflow.Environment
src/utils/queue-helpers.ts(applyQueueDropPolicy,drainNextQueueItem), consumed bysrc/auto-reply/reply/queue/enqueue.ts(enqueueFollowupRun) andsrc/auto-reply/reply/queue/drain.ts(collect-merge drain path); affects any channel using the default followup reply queue (default cap 20, default dropPolicysummarize, seesrc/auto-reply/reply/queue/state.ts:40-41).Steps to reproduce
m1) already checked out for delivery:drainNextQueueItemreaditems[0]and is awaitingrun(m1). Perqueue-helpers.ts:180-192,m1is not removed from the shareditemsarray untilrunresolves.enqueueFollowupRunsynchronously (src/auto-reply/reply/queue/enqueue.ts:94-197), which callsapplyQueueDropPolicyagainst the cap (queue cap 3 in the minimal repro below, but any cap reproduces this once inbound volume plus in-flight delay reaches it).applyQueueDropPolicyalways splices from the front ofitemsby count (queue-helpers.ts:115-116:const dropCount = items.length - cap + 1; const dropped = items.splice(0, dropCount);), with no check for whetheritems[0]is the item currently checked out by the in-flight drain. It selectsm1(still mid-delivery) as one of the drop victims.onDropruns form1immediately: withdropPolicy: "summarize"it is appended to the overflow summary lines (enqueue.ts:136-139); with any otherdropPolicyit callscompleteFollowupRunLifecycle(m1)immediately (enqueue.ts:141-143).run(m1)then resolves normally a moment later andm1is delivered as usual.Expected
An item that is already checked out and being delivered by an in-flight
drainNextQueueItem(or the collect-mergedrainGrouppath indrain.ts) should never be selectable as an overflow-drop victim. It should be delivered exactly once and should never also appear in the overflow-summary record or trigger an early lifecycle completion.Actual
The in-flight item is delivered normally AND is recorded as dropped/complete before that delivery finishes. This is not hypothetical: the repository's own existing test at
src/utils/queue-helpers.test.ts,describe("drainNextQueueItem"), test"keeps overflow survivors when the queue mutates during an awaited drain"(lines 167-215), already builds exactly this scenario and passes today with assertions that show the contradiction directly:Note
"m1"appears in bothdeliveredanddropped. This test currently passes onmain, meaning current shipped behavior deliberately produces this contradictory outcome and the suite treats it as expected. The contradiction itself is the bug: the assertions are internally consistent with the current implementation, but the implementation is user-facing-incorrect. Fixing this will require both a logic change inapplyQueueDropPolicy/drainNextQueueItemand updating this test's assertions to the corrected expected values once fixed.Root cause
drainNextQueueItemkeeps the item it is running physically present in the shareditemsarray for the entireawait run(next)duration, removing it only after the run resolves viaremoveQueuedItemsByRef(queue-helpers.ts:180-192) - this by-reference removal itself is the fix from PR #91450, which closed a worse bug (items.shift()removing whatever was at index 0 by the time the await resolved, causing total silent loss of a different message). The same in-flight-retention pattern holds for the collect-merge path indrain.ts:activeGroupItemsstay present inqueue.itemsfor the wholeawait drainGroup()(drain.ts:1169-1195), removed only afterward byconsumeAdmittedGrouporcompleteGroup.Meanwhile
enqueueFollowupRun(enqueue.ts:94-197) is fully synchronous and runs on every inbound message with no check onqueue.draining, mutating the very sameitemsarray reference returned bygetFollowupQueuefor as long as the queue exists (src/auto-reply/reply/queue/state.ts:100-138, existing queue branch at lines 101-109 returns the same object).applyQueueDropPolicy(queue-helpers.ts:101-130) always splices overflow victims from the front by count with zero awareness that index 0 may currently be checked out for delivery:onDropinenqueueFollowupRunthen either pushes the in-flight item straight into the overflow summary (enqueue.ts:136-139, the defaultsummarizepolicy perstate.ts:41) or callscompleteFollowupRunLifecycleon it immediately (enqueue.ts:141-143), while the realrunfor that same item object is still executing and will still deliver it a few lines later.Impact by policy:
dropPolicy: "summarize": the in-flight item's content is recorded as part of"[Queue overflow] Dropped N messages"(queue-helpers.ts:232-253) and that summary text is fed into a later prompt, even though the same item was already fully answered. This is silently incorrect with no log or warning anywhere.dropPolicy:completeFollowupRunLifecyclefires on the in-flight item before its real run finishes, permanently marking that lifecycle complete in thecompletedFollowupLifecyclesWeakSet (src/auto-reply/reply/queue/types.ts:178, 201-208). The real completion call later insrc/auto-reply/reply/followup-runner.ts:551or:1763becomes a silent no-op because the WeakSet already has the lifecycle, so hooks such as Gateway'scompleteQueuedChatTurn(wired atsrc/gateway/server-methods/chat.ts:4851) fire too early and never fire again when the run actually finishes.Preconditions
A burst of inbound messages reaches or exceeds the queue cap (default 20, or lower if operator-configured) while a longer-running agent turn is in flight and checked out at index 0 of the shared items array. This is realistic during any longer agent turn combined with a message burst on a tightly-capped channel.
Sibling surfaces
The collect-merge drain path in
drain.tsshares the same in-flight-retention pattern (activeGroupItemsstay inqueue.itemsthroughawait drainGroup()), so the same class of overlap is reachable there too, not only through the single-itemdrainNextQueueItempath exercised by the cited test.Related work
PR #91450 (merged) fixed a related but more severe sibling bug:
drainNextQueueItemused to remove the drained item withitems.shift(), which could remove the wrong item entirely (silent total loss of a different, still-queued message) if the array had been mutated during the await. That PR's fix, changing removal to by-reference (removeQueuedItemsByRef), is exactly what keeps the in-flight item physically present initemsfor the drop-policy code to mistakenly select in the scenario described here. The PR's own diff added thequeue-helpers.test.tstest cited above to lock in its fix, and that same test's assertions are the evidence for this remaining, distinct issue. PR #91450 does not address the overlap between drop-policy eviction and in-flight delivery; this is a follow-up gap, not a regression from that change.Real behavior proof
Behavior addressed: applyQueueDropPolicy can select an item that is currently checked out for in-flight delivery, so it ends up simultaneously delivered and recorded as an overflow drop.
Real environment tested: real production functions
applyQueueDropPolicyanddrainNextQueueItemfromsrc/utils/queue-helpers.ts, executed directly (no framework harness, no mocks) viatsxagainst the checkout at commit 66e4f48 (origin/main).Exact steps or command run after this patch:
node_modules/.bin/tsx repro.ts, whererepro.tsimportsapplyQueueDropPolicyanddrainNextQueueItemdirectly fromsrc/utils/queue-helpers.tsand reproduces the same sequence as the scenario above: one item (m1) drained and held mid-await, seven more items (m2..m8) enqueued throughapplyQueueDropPolicyagainst a cap of 3 whilem1is still in flight, then the gate is released and remaining items drained.Evidence after fix:
Observed result after fix: m1 is present in both the delivered list and the dropped/overflow list, produced directly by driving the real applyQueueDropPolicy and drainNextQueueItem functions with no patch applied, confirming the same message is simultaneously reported as answered and as dropped-due-to-overflow on current origin/main.
What was not tested: the collect-merge drain path in drain.ts (activeGroupItems / drainGroup) was inspected by reading code only and not independently reproduced with a standalone run; the completeFollowupRunLifecycle early-completion effect on Gateway's completeQueuedChatTurn hook was traced by reading code only and not reproduced end to end through a live gateway session.