fix(channels): split Telegram messages by UTF-16 code units, not code points by Lesereingrape · Pull Request #6067 · bytedance/deer-flow

Lesereingrape · 2026-09-29T17:12:38Z

Why

Trigger: both outbound size checks in the Telegram adapter count Python code points (len(text), and text[:4096]-style slicing), but Telegram measures its 4096-character message limit in UTF-16 code units, where every non-BMP character — most emoji, CJK Extension B — costs 2.

Telegram's own definitions use that unit: textEntity in TDLib documents offset_/length_ as "Offset/Length of the entity, in UTF-16 code units" (https://core.telegram.org/tdlib/docs/classtd_1_1td__api_1_1text_entity.html), and python-telegram-bot documents MessageEntity.offset the same way. The "characters" of MessageLimit.MAX_TEXT_LENGTH = 4096 are those units.

Pain: a reply that is under 4096 code points but over 4096 UTF-16 units is handed to sendMessage as one oversized chunk. The API rejects it with a deterministic 400 Message is too long, the shared channel retry policy repeats that non-retryable error three times with backoff, and _send_with_retry then re-raises — out of send(), so the chunks after the failing one are never sent either. The chat is left with the frozen streaming preview and no final answer. Emoji-heavy agent replies (the common case for the IM users of this project) hit this at only ~2049 emoji.

Reproducible on main (ed09efdb):

chunks = TelegramChannel._split_message("\U0001f600" * 3000)  # 3000 code points
len(chunks)                                     # 1
len(chunks[0].encode("utf-16-le")) // 2         # 6000, i.e. > 4096

The same mismatch affects the streamed preview: display[: 4095] + "…" is a length check plus a truncation in code points, so the preview can be up to 8191 units and the first send_message of a stream fails too.

What changed

Outbound Telegram text is chunked so that each message is at most 4096 UTF-16 code units, and a non-BMP character is never split across two messages.
The streaming preview truncation budgets the same unit, and its … now takes one of the 4096 units instead of adding a 4097th.
Text made only of BMP characters behaves exactly as before — "a" * 4096 + "b" * 100 still splits into ["a" * 4096, "b" * 100], and empty text still yields [""].

Deliberately out of scope: the Rich Message gate (_can_send_rich, TELEGRAM_MAX_RICH_MESSAGE_LENGTH) also counts code points, but a failed rich send already falls back to the plain-text path above, so it degrades instead of losing the reply.

Surface area

Backend API — endpoint / SSE event / request-response shape under backend/app (outbound Telegram delivery in backend/app/channels/telegram.py)

Screenshots / Recording

N/A — no frontend change.

Bug fix verification

Test path that reproduces the bug: backend/tests/test_telegram_message_length.py
Did it go red on main and green on this branch? yes
- main (ed09efdb): 4 failed, 1 passed
- this branch: 5 passed
- The one test that passes on both is test_split_message_still_slices_bmp_text_at_the_limit, which is the guard that BMP behaviour is unchanged.

Validation

Run locally from backend/ on a uv sync --locked environment:

PYTHONPATH=. uv run pytest tests/test_telegram_message_length.py → 5 passed on the branch; 4 failed / 1 passed with the fix reverted
PYTHONPATH=. uv run pytest tests/test_channels.py -k "telegram or Telegram" → 67 passed (existing streaming/splitting tests unchanged, including test_stream_update_truncates_long_text and test_final_message_splits_long_text)
PYTHONPATH=. uv run pytest tests/test_telegram_channel_connections.py → 5 passed
uv run ruff check backend/app/channels/telegram.py backend/tests/test_telegram_message_length.py and uv run ruff format --check on the same two files → clean

AI assistance

Tool(s) used: Qoder CLI agent, running the /contributor workflow.

How you used it: the agent read the adapter's size checks, drafted the fix and the regression tests, and ran them locally against main and against the branch; the diff is the agent's, the verification numbers above are from its local runs.

(left unticked deliberately: an agent account filed this PR, so this attestation can only be given by the human owner of the account, not by the agent.)

… points

willem-bd · 2026-09-30T00:18:53Z

-            display = display[: TELEGRAM_MAX_MESSAGE_LENGTH - 1] + "…"
+        if _utf16_length(display) > TELEGRAM_MAX_MESSAGE_LENGTH:
+            # Leave room for the ellipsis in the same unit the limit is measured in.
+            display = self._split_message(display, limit=TELEGRAM_MAX_MESSAGE_LENGTH - 1)[0] + "…"


[P2] Bound work when truncating cumulative stream previews. For every oversized streaming update, this calls _split_message on the entire cumulative reply and discards every chunk but the first. Together with the new full-text _utf16_length check, a preview that stopped changing after 4096 units still scans and materializes the complete, growing answer on each update, before the existing throttle/unchanged-text checks. The manager can publish an update every 60 new characters, so long replies incur quadratic total work and transient allocations on the async channel path. Stop counting after the first over-limit unit and construct only the first bounded chunk for the preview.

Pushed b1e18b6. _send_stream_update now clips through a new _first_utf16_chunk(text, limit), which returns at the first character that does not fit and never walks past the limit, and the ellipsis is appended to that bounded prefix: no full-text measurement and no full-text split per update. _utf16_length is gone, the per-character rule lives in one _utf16_width helper shared by the splitter and the clip.

Output is unchanged: for 4095/4096/4097 BMP characters, 3000 emoji, an alternating emoji/ASCII reply and a 200k-character reply the clipped string is identical to what the previous code produced. Width evaluations on the 200k case drop from two full passes (~400k) to 8,193 (two scans bounded by the limit).

test_stream_update_clips_without_measuring_the_whole_reply counts _utf16_width calls through a wrapper and asserts the total stays within 2 * (limit + 1), so the bound is enforced by a test rather than only described.

willem-bd

Reviewed at head d594d41: the core fix is correct. The UTF-16 accounting in _split_message is right (each char is atomic, so a non-BMP char can never straddle a boundary; empty input still yields [""]; pure-BMP text slices at exactly 4096 as before, which the preserved test_channels.py BMP assertions and the new regression test pin down). The stream-preview math checks out too: a <=4095-unit first chunk plus the 1-unit U+2026 ellipsis stays within 4096 units. The keyword-only limit default keeps the signature backward compatible for the three existing call sites, and the plain-text path sets no parse_mode, so there is no markdown-entity-spanning concern; the Rich Message 32768 code-point gate is reasonably left out of scope since a rich failure falls back to this now-correct plain path. Coverage is good, including the emoji-heavy end-to-end send() path. Two new minor findings inline (test-oracle independence, grapheme-boundary documentation); the per-update truncation cost is already tracked in the existing thread.

willem-bd · 2026-09-30T00:34:38Z

+
+
+def _utf16_length(text: str) -> int:
+    return sum(2 if ord(char) > 0xFFFF else 1 for char in text)


Every length assertion in this file is checked with a test-local _utf16_length that is a character-for-character copy of the production formula in telegram.py, so if the shared rule were wrong (e.g. a bad 0xFFFF threshold), all five tests would still pass. Consider asserting against an independent oracle instead — len(chunk.encode("utf-16-le")) // 2, which the PR description itself uses in its repro snippet — at least for the two all(_utf16_length(chunk) <= ...) budget assertions. Related nit: the production helper _utf16_length is never imported here, so it is only exercised indirectly through the send paths.

Fixed in b1e18b6. The duplicated helper is gone: every budget assertion now measures with utf16_units(text) = len(text.encode("utf-16-le")) // 2, the codec oracle the PR description uses in its repro.

Both production helpers are also exercised directly (from app.channels import telegram, the module-attribute style the other channel tests use). test_utf16_width_agrees_with_the_codec pins a BMP letter, an accented Latin char, a CJK char, U+200D, an emoji and a reserved astral code point against the codec, so a wrong threshold in the shared rule now fails on its own instead of being ratified by a copy of itself. _first_utf16_chunk gets its own fits/does-not-fit cases, including the astral character that straddles the boundary.

willem-bd · 2026-09-30T00:34:38Z

+        units = 0
+        for char in text:
+            width = 2 if ord(char) > 0xFFFF else 1
+            if units + width > limit:


Nit, not a blocker: the split boundary can still land inside a user-perceived character — between a base character and a combining mark, mid ZWJ emoji sequence (a family flag-style sequence of three astral code points costs 6 units), or between the two regional indicators that form a flag emoji. Each chunk stays within the UTF-16 budget so Telegram accepts it, and this is strictly no worse than the old code-point slicing, but a one-line docstring note (or a follow-up that prefers breaking before a ZWJ run or an odd trailing regional indicator) would set expectations for rendering artifacts at chunk seams. Fine to defer.

Took the docstring option in b1e18b6: _split_message now says a split can fall inside a combining mark, a ZWJ emoji sequence or the second regional indicator of a flag, and that every chunk stays within budget, so a seam is a rendering artifact rather than a rejected send.

I left the break-before-ZWJ / odd-trailing-regional-indicator rule out of this PR: it moves chunk boundaries for text that already sends successfully, which reads like a separate change with its own tests.

Every cumulative stream update measured the full answer and then split the full answer to keep only its first chunk, so a long reply cost two complete passes per update on the async channel path. _first_utf16_chunk stops at the first character that does not fit, and the preview is built from that bounded prefix. The tests now measure chunk length with the UTF-16 codec instead of a copy of the production formula, cover the width helper and the bounded clip directly, and fail if the preview stops being bounded by the limit.

fix(channels): split Telegram messages by UTF-16 code units, not code…

d594d41

… points

github-actions Bot added area:backend Gateway / runtime / core backend under backend/ risk:medium Medium risk: regular code changes size/M PR changes 100-300 lines labels Sep 29, 2026

willem-bd reviewed Sep 30, 2026

View reviewed changes

willem-bd mentioned this pull request Sep 30, 2026

fix(channels): preserve content when splitting long Discord messages #6065

Closed

willem-bd reviewed Sep 30, 2026

View reviewed changes

This was referenced Sep 30, 2026

fix(channels): keep the delimiter newline when splitting long Discord replies #6062

Merged

fix(discord): keep the newline at the message-split boundary (GH#6055) #6064

Closed

WillemJiang approved these changes Oct 1, 2026

View reviewed changes

WillemJiang added this to the 2.2.0 milestone Oct 1, 2026

WillemJiang merged commit 8df4286 into bytedance:main Oct 1, 2026
22 of 23 checks passed

msm3107 mentioned this pull request Oct 2, 2026

Audyt po pilotażu: limity Telegrama (UTF-16), bezpieczna oferta, /wplata, blokady bazy, prywatność msm3107/GUNB-TOOL#4

Merged

Navigation Menu

Repository navigation

fix(channels): split Telegram messages by UTF-16 code units, not code points - #6067

fix(channels): split Telegram messages by UTF-16 code units, not code points#6067
WillemJiang merged 2 commits into
bytedance:mainfrom
Lesereingrape:fix/telegram-utf16-message-limit

Lesereingrape commented Sep 29, 2026

Uh oh!

willem-bd Sep 30, 2026

Uh oh!

Lesereingrape Sep 30, 2026

Uh oh!

willem-bd left a comment

Uh oh!

willem-bd Sep 30, 2026

Uh oh!

Lesereingrape Sep 30, 2026

Uh oh!

willem-bd Sep 30, 2026

Uh oh!

Lesereingrape Sep 30, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants



		def _utf16_length(text: str) -> int:
		return sum(2 if ord(char) > 0xFFFF else 1 for char in text)

Conversation

Lesereingrape commented Sep 29, 2026

Why

What changed

Surface area

Screenshots / Recording

Bug fix verification

Validation

AI assistance

Uh oh!

willem-bd Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

Lesereingrape Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

willem-bd left a comment

Choose a reason for hiding this comment

Uh oh!

willem-bd Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

Lesereingrape Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

willem-bd Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

Lesereingrape Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants