Skip to content

fix(realtime): restart the downlink pump after it crashes - #3732

Merged
qbc2016 merged 1 commit into
agentscope-ai:mainfrom
konlue:fix/realtime-downlink-crash-recovery
Nov 12, 2026
Merged

qbc2016 merged 1 commit into
agentscope-ai:mainfrom
konlue:fix/realtime-downlink-crash-recovery

Conversation

@konlue

@konlue konlue commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

AgentScope Version

2.0.12dev (main @ 782bd43)

Description

RealtimeAgent's downlink pump (_pump_downlink in src/agentscope/agent/_realtime/_agent.py) runs without any exception guard: if the model's event stream raises (a provider SDK stream error) or _on_model_event throws, the exception escapes the while True and the pump task dies. _on_downlink_done detects the crash and clears _connected/_connected_event so that, per its docstring, the "next input reconnects" — but it leaves the finished task in self._downlink, and connect() only creates a new pump when that reference is None (the only place it is reset is close()). Every reconnect after a crash therefore succeeds — audio is pushed, the session opens — while nobody pumps the new session's events: the agent goes deaf for the rest of its lifetime, which is exactly the "silently dead session" the callback's docstring says it surfaces instead (#3731).

Changes — the crash branch of _on_downlink_done now also clears the dangling reference, so the next connect() starts a fresh pump:

self._downlink = None

How to test

New case in the existing RealtimeAgentTest (tests/realtime_agent_test.py), driven by the file's ScriptedModel/FakeTransport fixtures:

  • test_crashed_downlink_pump_is_replaced_on_reconnect — a CrashingSessionModel whose first session's events() raises, whose second session behaves: fails on main (the reconnect succeeds but the summary is empty — no events are ever pumped from the new session, and agent._downlink.done() is still true), passes with this fix (the second session's reply is delivered and the live pump is back);
  • the existing reconnect tests (test_provider_timeout_reconnects_on_next_audio, test_old_downlink_cannot_disconnect_new_session, etc.) are untouched and still pass.
python -m pytest tests/realtime_agent_test.py
# 44 passed.

Fixes #3731

Checklist

  • An issue has been created for this PR
  • I have read the CONTRIBUTING.md
  • Docstrings are in Google style
  • Related documentation has been updated (e.g. links, examples, etc.) in documentation repository
  • Code is ready for review

The downlink pump runs without any exception guard, so a provider
stream error or a handler exception kills the task. _on_downlink_done
detects the crash and clears _connected so the next input reconnects,
but it leaves the finished task in self._downlink — and connect() only
creates a new pump when that reference is None. Every reconnect after
a crash therefore succeeds without pumping the new session's events:
the agent goes deaf for the rest of its lifetime.

Clear the dangling reference in the crash branch so the next connect()
starts a fresh pump, and pin the recovery with a regression test that
crashes the first session's event stream and asserts the second
session is pumped.
@github-actions github-actions Bot added the pr/awaiting-review Waiting for maintainers to review this PR label Nov 4, 2026

@qbc2016 qbc2016 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@qbc2016
qbc2016 merged commit 9d65f4e into agentscope-ai:main Nov 12, 2026
9 checks passed
@github-actions github-actions Bot removed the pr/awaiting-review Waiting for maintainers to review this PR label Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: RealtimeAgent never restarts the downlink pump after it crashes — reconnect succeeds but the agent stops receiving model events

2 participants