Skip to content

Gateway: duplicate/stale response when new message arrives during post-processing (interrupt race condition) #2483

Description

@Voxvienne

Summary

When a user sends a new message shortly after the agent has already sent its response, the bot sometimes sends a duplicate response to the previous message instead of (or in addition to) answering the new one. This occurs even with gaps of several minutes between messages.

Symptom

Example chronology from Telegram (translated from German):

12:18  User:  "Hey, are we running on MiniMax 2.7 or 2.5?"
12:19  KIRO:  "MiniMax 2.7. It's also stated in the system prompt of this session:
               'Model: MiniMax-M2.7'. That's the current model since March 19."   ← correct ✓
12:19  User:  "Ok. I'd also like to understand why we often end up in this
               desynchronized state in the chat here..."
12:20  KIRO:  "MiniMax 2.7. You're on the current model since March 19 —
               updated from 2.5."                    ← duplicate of the 12:18 answer! ✗
12:20  User:  "That's exactly what I mean — you just answered my previous
               message twice. Why does that happen?"
12:21  KIRO:  [eventually answers the actual 12:19 question about the desync issue]

The bot re-answered the previous question ("which model?") instead of the new one ("why does desync happen?"). The log confirms the interrupt fired:

[Telegram] ⚡ New message while session agent:main:telegram:dm:7836010240 is active - triggering interrupt
[Telegram] 📨 Processing queued message from interrupt

Root Cause

There are two independent interrupt-processing pipelines that can activate for the same pending message under certain timing conditions:

Pipeline A — run.py interrupt monitor (line ~4906)

# Inside _run_agent, polls every 200ms:
if hasattr(adapter, 'has_pending_interrupt') and adapter.has_pending_interrupt(session_key):
    pending_event = adapter.get_pending_message(session_key)  # pops from base._pending_messages
    agent.interrupt(pending_text)

After the executor returns, if result.interrupted, this leads to a recursive _run_agent call (line ~4979) with:

updated_history = result.get("messages", history)  # ← may not include completed response
return await self._run_agent(message=pending, history=updated_history, ...)

Pipeline B — base.py pending check (line ~1088)

# After _message_handler(event) returns:
if session_key in self._pending_messages:
    pending_event = self._pending_messages.pop(session_key)
    await self._process_message_background(pending_event, session_key)

The Race Window

The session remains active (_active_sessions[session_key] is set) well after the response is sent to the user — while Honcho memory writes, SQLite transcript appends, and other async post-processing are still running. This window can be 10–30 seconds.

If a new message arrives during this window:

  1. base.py stores it in _pending_messages and sets the interrupt event
  2. The run_in_executor call (agent thread) may have already returned
  3. The interrupt monitor task is not yet cancelled (cancel happens in finally at line ~4992)
  4. The monitor fires, pops the message from _pending_messages, and may call agent.interrupt() on an already-finished agent
  5. result.get("interrupted") is False → Pipeline A skips the recursive call
  6. _pending_messages is now empty (popped by the monitor)
  7. Pipeline B's check finds nothing → the new message is silently dropped

Alternatively, if the interrupt fires during the executor run but result.messages does not yet include the completed assistant response (because the history is built up incrementally), Pipeline A's recursive _run_agent processes the new message with stale history — causing the LLM to re-answer the unanswered previous question.

Impact

  • Duplicate responses to previous messages
  • New messages occasionally silently dropped
  • Occurs with any message gap, not just rapid successive messages
  • Platform: Telegram (likely affects all platforms using the interrupt mechanism)

Suggested Fix

The two pipelines need to coordinate. A minimal fix:

In gateway/run.py (~line 4940): After run_in_executor returns, regardless of interrupted state, always attempt to drain any pending message that arrived during the tail of the run:

# After executor returns — drain any pending that arrived during post-processing
# (handles the case where the interrupt monitor already fired but result.interrupted=False)
result = result_holder[0]
adapter = self.adapters.get(source.platform)

pending = None
if result and result.get("interrupted") and adapter:
    pending_event = adapter.get_pending_message(session_key) if session_key else None
    if pending_event:
        pending = pending_event.text
    elif result.get("interrupt_message"):
        pending = result.get("interrupt_message")

# NEW: also check for messages that arrived after the executor finished
# but before the interrupt monitor was cancelled
if not pending and adapter:
    late_pending = adapter.get_pending_message(session_key)
    if late_pending:
        pending = late_pending.text

In gateway/platforms/base.py (~line 1088): Ensure the pending check uses the same history that includes the completed response — or defer to the run.py recursive path entirely and remove the duplicate processing:

# Only process pending if run.py didn't already handle it via recursive _run_agent
# (i.e., it wasn't consumed by get_pending_message during the interrupt monitor phase)
if session_key in self._pending_messages:
    pending_event = self._pending_messages.pop(session_key)
    # ... existing code

The deeper fix is to make _active_sessions[session_key] inactive immediately after sending the response (not after all post-processing), so new messages arriving during post-processing get a fresh background task rather than triggering the interrupt path.

Environment

  • hermes-agent: 0.3.0
  • Platform: Telegram (python-telegram-bot, long-polling)
  • Gateway: gateway/run.py + gateway/platforms/base.py + gateway/platforms/telegram.py

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions