fix(realtime): keep attached blocks when merging a split user turn - #3241

Fixes #3237

PR Title Format

fix(realtime): keep attached blocks when merging a split user turn — conventional commit, fix type, lowercase description.

AgentScope Version

2.0.10.dev0 (main)

Description

Summary

When endpointing splits the user's speech and the second half of the transcript arrives inside the TurnAggregator merge window, RealtimeAgent._merge_user rebuilds the previous user turn from its concatenated text alone, dropping every non-text block of that turn — e.g. a DataBlock carrying an image the user attached between the halves. The merge now appends the text to the turn and leaves the attached blocks in place.

Why

_merge_user runs whenever a settled transcript continues the previous user turn (src/agentscope/agent/_realtime/_agent.py:826-828), and its docstring says it appends text to that turn. The implementation instead replaced the turn's whole content:

previous = context[-1].get_text_content() or ""
context[-1].content = [TextBlock(text=f"{previous}{text}")]

User messages are documented and validated to carry DataBlocks (_assert_user_content_blocks, src/agentscope/message/_base.py:32-38; the UserMsg docstring states that only TextBlock and DataBlock are allowed for user messages), and RealtimeAgent.send appends any Msg it receives to state.context (src/agentscope/agent/_realtime/_agent.py:528). So the attached image is silently removed from the conversation context once the second half of the split utterance settles — no error, and the loss persists for the rest of the session.

Msg.append_event already defines the project's pattern for merging streamed text into a block list: TOOL_RESULT_TEXT_DELTA appends to the last block only if it is a text block, otherwise appends a new TextBlock (src/agentscope/message/_base.py:432-435). _merge_user now follows the same pattern.

Changes

• src/agentscope/agent/_realtime/_agent.py: _merge_user no longer rebuilds the previous user turn's content. If the turn's last block is a TextBlock, the text is appended to it (unchanged behavior for the plain split-transcript case); otherwise a new TextBlock is appended, and every existing block — DataBlocks included — stays in place. An empty content list still produces a single TextBlock, as before.
• tests/realtime_agent_test.py: two tests in RealtimeAgentTranscriptionTest — the halves of a split transcript still merge into one text block, and a DataBlock attached between the halves survives the merge with the second half's text appended after it (both use TurnAggregator(merge_window_ms=60_000) so they do not depend on wall-clock timing).

No public API change, no breaking change.

Testing

New regression test on the unpatched tree (fails — the DataBlock is dropped):

PYTHONPATH=src python3 -m pytest "tests/realtime_agent_test.py::RealtimeAgentTranscriptionTest::test_split_transcript_merge_keeps_attached_blocks" -v

FAILED tests/realtime_agent_test.py::RealtimeAgentTranscriptionTest::test_split_transcript_merge_keeps_attached_blocks
E       AssertionError: Lists differ: [['text'], ['text']] != [['text'], ['text', 'data', 'text']]
E
E       First differing element 1:
E       ['text']
E       ['text', 'data', 'text']
E
E       - [['text'], ['text']]
E       + [['text'], ['text', 'data', 'text']]
================== 1 failed, 1 passed, 31 deselected in 2.86s ===================

With the fix, both new tests pass:

PYTHONPATH=src python3 -m pytest tests/realtime_agent_test.py -k split_transcript_merge -v
tests/realtime_agent_test.py::RealtimeAgentTranscriptionTest::test_split_transcript_merge_joins_the_halves PASSED [ 50%]
tests/realtime_agent_test.py::RealtimeAgentTranscriptionTest::test_split_transcript_merge_keeps_attached_blocks PASSED [100%]
======================= 2 passed, 31 deselected in 8.25s =======================

Realtime agent module tests, before (origin/main) vs after:

PYTHONPATH=src python3 -m pytest tests/realtime_agent_test.py
31 passed   # origin/main
33 passed   # with this change (the two new tests)

Tests covering the realtime agent and its turn aggregator:

PYTHONPATH=src python3 -m pytest tests/realtime_agent_test.py tests/realtime_aggregator_test.py -q
36 passed   # with this change; 34 on origin/main

tests/tui_test.py (the only other file referencing RealtimeAgent) cannot be collected in this environment on either tree: ModuleNotFoundError: No module named 'textual' (the tui extra is not installed).

Lint and type checks with the versions pinned in .pre-commit-config.yaml:

black --line-length=79 --check src/agentscope/agent/_realtime/_agent.py tests/realtime_agent_test.py
All done! 2 files would be left unchanged.
flake8 --extend-ignore=E203 <same files>      # no findings
pylint <pre-commit args> <same files>         # rated 10.00/10 on both files
mypy <pre-commit args> <same files>           # Success: no issues found in 2 source files

Checklist

• An issue has been created for this PR ([Bug]: RealtimeAgent split-transcript merge replaces the previous user turn's content, dropping attached DataBlocks #3237)
• I have read the CONTRIBUTING.md
• Docstrings are in Google style
• Related documentation has been updated (e.g. links, examples, etc.) in documentation repository — no user-facing API change; the fix aligns the code with the documented behavior that user messages carry text and data blocks
• Code is ready for review