fix(channels/sidecars): recover per-message reply correlation via ChannelUser.librefang_user across 6 sidecars#5439
Merged
Conversation
…nnelUser.librefang_user across mastodon, nextcloud, reddit, twitch, bluesky, rocketchat Same root cause as the dingtalk (#5423) and qq (#5431) fixes. All six sidecars read cmd.thread_id in on_send to recover a per-message correlation key (mastodon: status_id; nextcloud: msg id; reddit: parent fullname; twitch: tag-id; bluesky: at-URI; rocketchat: tmid). But the daemon's bridge only sets cmd.thread_id when BOTH: * the channel config has [channels.X.overrides] threading = true (defaults to false; verified at crates/librefang-channels/src/bridge.rs:2968-2973), AND * the sidecar declares the "thread" capability (verified at crates/librefang-channels/src/sidecar.rs:1372-1395 — without it, send_in_thread degrades to plain send which hardcodes thread_id: None at line 1203). All six declare `capabilities: list = []`. Production operators with default `threading = false` (and even with threading=true, because the cap-gate fails first) get cmd.thread_id = None on every reply. Net effect, by severity (ranked by user-visible damage): HIGH: reddit — every reply RAISED RuntimeError("missing parent fullname"); RuntimeError caught by SDK's bare-except on_command wrapper and logged to stderr only. Operator saw no message land. Bot looked healthy. MED: nextcloud / rocketchat — bot's "thread reply" posted at channel root with no link back to the trigger. The docstrings of both modules explicitly advertise this as their headline improvement over Rust parity; the modules in fact shipped the same bug. MED: bluesky — reply intent became top-level skeet, no thread context. Visible to all followers' feeds (not just the thread participants). LOW: mastodon — reply degraded to top-level toot; message still landed but UI lost the in-reply-to context. Bonus pre-existing semantic bug ALSO fixed: parse surfaced status.in_reply_to_id (the PARENT the mention was responding to) as the reply target instead of status.id (the mention itself); the bot would have replied to the wrong account even if the round-trip worked. LOW: twitch — chat UI lost the @reply-parent-msg-id badge; plain PRIVMSG still delivered the text. The bugs were not caught by the test suites because every on_send test fabricated cmd.thread_id="..." on a stub — the realistic daemon shape (thread_id=None, user.librefang_user=<id>) was uncovered. Fix (uniform across all six): - parse_X_event: also stash the correlation key in ChannelUser.librefang_user (which the bridge round-trips bytewise through serde regardless of capabilities/overrides — verified at sidecar.rs:766 inbound, :1204 outbound). Keep the existing thread_id assignment for the forward-compat threading=true path. - on_send: read cmd.user["librefang_user"] FIRST, fall back to cmd.thread_id. Sanity guard — reject URL-shaped or whitespace- bearing values (librefang_user is shared across channels; dingtalk puts a sessionWebhook URL there, telegram puts an @username — we must not POST replies with cross-channel garbage as the correlation key). - Reddit gets the strongest guard: enforce t{1,3,4,5}_ prefix. - Bluesky gets the strongest guard: enforce at:// prefix. Tests (per adapter — at minimum one regression guard each): - test_on_send_recovers_X_from_user_librefang_user drives the realistic daemon shape and asserts the outbound payload contains the correlation key. Each fails on the pre-fix code. - reddit gets a second test pinning the prefix-guard (rejects URL-shaped librefang_user, falls back to thread_id). - mastodon parse-test rewritten — the pre-fix test pinned the buggy behaviour (asserted thread_id == in_reply_to_id which was the wrong target). Verification: cd sdk/python && pytest tests/test_{mastodon,nextcloud, reddit,twitch,bluesky,rocketchat}_adapter.py — 343 passed (was 336; +7 regression guards across 6 adapters, -1 pre-fix wrong-behaviour assertion deleted via rewrite). Pre-existing structural issue NOT fixed here (out of scope): - SDK runtime.py:245-247 has a bare-except on on_command that swallowed Reddit's RuntimeError silently. The immediate fix here is to not raise in the first place (no production path now reaches the missing-parent branch), but the bare-except itself is a footgun — future raises from on_send will also vanish to stderr only. Worth a follow-up: re-raise after logging so the bridge can record the failure and trigger the agent's error-handling path. Same investigation pattern: dingtalk #5423 first surfaced this bug class via post-PR self-review; qq #5431 confirmed it generalised; this PR closes out the remaining six known cases.
This was referenced May 21, 2026
Merged
houko
added a commit
that referenced
this pull request
May 21, 2026
…r across sidecar restart (#5448) The in-process `_user_context_tokens[user_id]` cache populated by inbound iLink callbacks vanishes on sidecar restart (any operator upgrade, debug pass, or supervisor respawn). Without the context_token, the bot's first reply after restart posts with an empty token — iLink may reject it or post out-of-thread. Fix: parse_wechat_msg also surfaces `context_token` via `ChannelUser.librefang_user`, which the bridge round-trips bytewise through serde (verified at crates/librefang-channels/src/sidecar.rs:766 inbound, :1204 outbound) so it survives serde + restart cleanly. The in-memory cache stays as the freshness signal (the latest token Lark/ iLink issued is more current than whichever inbound the daemon happens to round-trip back) and takes precedence in on_send. Tests: - New test_on_send_recovers_context_token_from_user_librefang_user_when_cache_cold simulates post-restart (cache empty, cmd.user.librefang_user carries the token) and asserts on_send recovers it. - New test_on_send_ignores_url_shaped_librefang_user pins the cross- channel pollution guard (librefang_user is shared across channels — dingtalk puts a sessionWebhook URL there, telegram puts @username). - cd sdk/python && pytest tests/test_wechat_adapter.py — 63 passed (was 61; +2 regression guards). Surfaces from the #5439-followup audit chain (dingtalk #5423 → qq #5431 → omnibus #5439 → THIS). Same library_user routing pattern, distinct symptom — restart-survivability rather than cap-gate stripping.
houko
added a commit
that referenced
this pull request
May 21, 2026
… sidecar restart (#5449) Same bug class as the wechat restart-fragility hotfix: the in-process `_pending_req_ids[user_id]` cache vanishes on sidecar restart, so the bot's first reply after restart degrades from `aibot_respond_msg` (passive, free) to `aibot_send_msg` (active, quota-burning, blocked for unlicensed bots and rate-limited hard for licensed ones). Fix: parse_wecom_event also surfaces req_id via `ChannelUser.librefang_user`. on_send's `_enqueue_text` learns a `req_id_hint` keyword that falls back to it when the in-memory cache is empty. Cache stays as the primary path because it holds the FRESHEST req_id (subsequent inbounds clobber the dict but the round-tripped librefang_user is whichever inbound the daemon happens to surface — possibly stale). Tests: - test_on_send_recovers_req_id_from_user_librefang_user_when_cache_cold drives the post-restart shape (cache empty, librefang_user carries req_id), asserts on_send emits `aibot_respond_msg` with the recovered req_id rather than degrading to `aibot_send_msg`. - test_on_send_in_memory_cache_wins_over_librefang_user pins the cache-first precedence (freshness invariant). - test_on_send_ignores_url_shaped_librefang_user pins the cross- channel pollution guard. - cd sdk/python && pytest tests/test_wecom_adapter.py — 65 passed (was 62; +3 regression guards). Surfaces from the #5439-followup audit chain.
houko
added a commit
that referenced
this pull request
May 21, 2026
The reader() coroutine in librefang.sidecar.runtime swallows exceptions from `adapter.on_command(cmd)` with a bare-except that logged only `error=str(e)`. This silence let four consecutive production reply-routing regressions ship with no warning sign: - #5417 dingtalk (sessionWebhook cache key mismatch) - #5431 qq (cmd.thread_id never set without `thread` cap) - #5439 reddit (raised RuntimeError on every reply; vanished here) - #5439 nextcloud/rocketchat/twitch/bluesky/mastodon (same class) Each took at least one post-merge self-review pass to surface because the original failure logged as `on_command failed error="missing parent fullname"` with no clue about WHICH adapter or WHICH line raised. Operators didn't notice because the bot silently looked healthy — every adapter just stopped replying. The fix is structural-but-conservative: keep the bare-except (we cannot raise out of reader() without taking down the entire sidecar process for one bad command) but add traceback + error_type + cmd_type as structured fields so debuggers can grep the offending adapter / line directly. The richer long-term fix — surfacing on_command failures as a structured JSON-RPC error response so the bridge correlates the failure back to the original cmd id and the agent's error-handling lane fires — needs a protocol change (Response.error field) and is tracked in a follow-up issue. No tests added: the change is purely log-content shape — it doesn't alter exception handling behaviour, just adds context. The protocol-level fix in the follow-up issue is what needs new test coverage.
Merged
2 tasks
houko
added a commit
that referenced
this pull request
May 21, 2026
…e miss (closes #5452) `parse_notification` stashed the AT Protocol `{root, parent}` reply struct in an in-process `_thread_cache` LRU keyed by notification URI; `_post_status` looked it up on outbound. After ANY sidecar restart between the inbound mention and the bot's reply, the cache was empty, the lookup returned None, and the reply posted as a top-level skeet visible to every follower's feed instead of staying threaded. The single-string `cmd.user.librefang_user` carrier added in #5439 only round-trips the URI. AT Protocol needs 4 fields (`root.uri`, `root.cid`, `parent.uri`, `parent.cid`) which don't fit in one string, so the #5439-style librefang_user fix doesn't structurally work here. This commit (option-1 from the issue — bluesky-only, narrow blast radius) adds: * **`_recover_reply_ref`**: on `_thread_cache` miss in `_post_status`, issue one `app.bsky.feed.getPosts?uris=<uri>` XRPC call to re-fetch the post's cid + its existing `record.reply` chain, reconstruct `{root, parent}` using the same root-resolution rules `_compute_reply_ref` uses on inbound, re-populate `_thread_cache` for subsequent chunks. Recovery failure → WARN naming the URI + degrade to top-level skeet (same as pre-fix, but with operator visibility). * **Transient-failure handling on the recovery path** so an extra failure mode doesn't permanently downgrade a recoverable reply: - **401**: invalidate `_access_jwt` and re-fetch a fresh token via `_get_token` (same pattern `_post_status`'s own 401 path uses at line ~487/693), retry the XRPC once. - **429**: honour `Retry-After` via the existing `_retry_after_secs` helper (same shape verify_credentials / poll / createRecord already use at lines 292/331/493), sleep, retry once. Without either retry, a session rotation OR a transient bsky rate limit on getPosts would silently downgrade every queued reply after a sidecar reboot to top-level skeet — even though the caller's subsequent createRecord refresh / retry would have saved them. * **Sanity guard on the input URI**: must start with `at://`. A misrouted `librefang_user` from another channel (dingtalk sessionWebhook URL, telegram `@username`) returns None without any HTTP call, no cache write. * **Defensive guards on the response shape**: missing `posts` key, empty list, non-dict entry, missing/empty `cid` — every malformed shape surfaces as None so the caller degrades cleanly rather than crashing on `.get()` against a non-dict. * **WARN message broadened** to "post deletion, instance unreachable, rate limit, or auth issue" (was "post deletion / auth issue" — too narrow, missed the 5xx / 429 / network-blip cases). * **`_LruCache` made thread-safe.** `_post_status` runs in `run_in_executor(None, ...)` which uses the default ThreadPoolExecutor — concurrent `put` / `get` from worker threads is real. Without a lock the `move_to_end + assignment + popitem` sequences race on OrderedDict's internal linked list. Added `threading.Lock` guarding `get`/`put`/`__len__`. The hammer-from- 8-threads test pins this; remove the lock and the cap-violation assertion fires under stress. The option-3 fix from #5452 (widen `SidecarSendParams` with a `metadata: HashMap<String, Value>` bytewise-round-tripped field, which would close the class for bluesky + wechat + wecom + any future sidecar needing >1 field of correlation state) is a separate protocol change spanning Rust + Python SDK + every sidecar adapter — deliberately not in scope here. Verification: - `cd sdk/python && pytest tests/test_bluesky_adapter.py` — 53 passed (was 44; +9 regression guards): * recovery happy-path: cache miss → getPosts → reply ref attached to createRecord body; cache re-populated. * deep-thread root preservation: post-with-reply's `record.reply.root` becomes the recovered root, not the immediate parent. * multi-chunk re-uses recovered ref: 3-chunk reply with cache miss = 1 session + 1 getPosts + 3 createRecord (5 calls total, not 7). * recovery failure → top-level skeet WARN. * cross-channel garbage rejected without any HTTP call. * 401 → invalidate session → _get_token re-creates → retry XRPC once with fresh bearer. * 429 → sleep `Retry-After` → retry once. * malformed responses (empty posts, non-dict entry, missing cid, no posts key) → None. * LruCache thread-safety: 8 threads × 200 puts + interleaved reads; cache cap respected, no exceptions raised. - `cd sdk/python && pytest tests/` — full sidecar suite 1859 passed (no regression in any of the other 26 adapters). - Pre-existing `test_post_status_cold_cache_falls_back_to_unthreaded` renamed + updated to assert the getPosts re-fetch fires + 404s + then degrades.
houko
added a commit
that referenced
this pull request
May 21, 2026
…e miss (closes #5452) (#5471) `parse_notification` stashed the AT Protocol `{root, parent}` reply struct in an in-process `_thread_cache` LRU keyed by notification URI; `_post_status` looked it up on outbound. After ANY sidecar restart between the inbound mention and the bot's reply, the cache was empty, the lookup returned None, and the reply posted as a top-level skeet visible to every follower's feed instead of staying threaded. The single-string `cmd.user.librefang_user` carrier added in #5439 only round-trips the URI. AT Protocol needs 4 fields (`root.uri`, `root.cid`, `parent.uri`, `parent.cid`) which don't fit in one string, so the #5439-style librefang_user fix doesn't structurally work here. This commit (option-1 from the issue — bluesky-only, narrow blast radius) adds: * **`_recover_reply_ref`**: on `_thread_cache` miss in `_post_status`, issue one `app.bsky.feed.getPosts?uris=<uri>` XRPC call to re-fetch the post's cid + its existing `record.reply` chain, reconstruct `{root, parent}` using the same root-resolution rules `_compute_reply_ref` uses on inbound, re-populate `_thread_cache` for subsequent chunks. Recovery failure → WARN naming the URI + degrade to top-level skeet (same as pre-fix, but with operator visibility). * **Transient-failure handling on the recovery path** so an extra failure mode doesn't permanently downgrade a recoverable reply: - **401**: invalidate `_access_jwt` and re-fetch a fresh token via `_get_token` (same pattern `_post_status`'s own 401 path uses at line ~487/693), retry the XRPC once. - **429**: honour `Retry-After` via the existing `_retry_after_secs` helper (same shape verify_credentials / poll / createRecord already use at lines 292/331/493), sleep, retry once. Without either retry, a session rotation OR a transient bsky rate limit on getPosts would silently downgrade every queued reply after a sidecar reboot to top-level skeet — even though the caller's subsequent createRecord refresh / retry would have saved them. * **Sanity guard on the input URI**: must start with `at://`. A misrouted `librefang_user` from another channel (dingtalk sessionWebhook URL, telegram `@username`) returns None without any HTTP call, no cache write. * **Defensive guards on the response shape**: missing `posts` key, empty list, non-dict entry, missing/empty `cid` — every malformed shape surfaces as None so the caller degrades cleanly rather than crashing on `.get()` against a non-dict. * **WARN message broadened** to "post deletion, instance unreachable, rate limit, or auth issue" (was "post deletion / auth issue" — too narrow, missed the 5xx / 429 / network-blip cases). * **`_LruCache` made thread-safe.** `_post_status` runs in `run_in_executor(None, ...)` which uses the default ThreadPoolExecutor — concurrent `put` / `get` from worker threads is real. Without a lock the `move_to_end + assignment + popitem` sequences race on OrderedDict's internal linked list. Added `threading.Lock` guarding `get`/`put`/`__len__`. The hammer-from- 8-threads test pins this; remove the lock and the cap-violation assertion fires under stress. The option-3 fix from #5452 (widen `SidecarSendParams` with a `metadata: HashMap<String, Value>` bytewise-round-tripped field, which would close the class for bluesky + wechat + wecom + any future sidecar needing >1 field of correlation state) is a separate protocol change spanning Rust + Python SDK + every sidecar adapter — deliberately not in scope here. Verification: - `cd sdk/python && pytest tests/test_bluesky_adapter.py` — 53 passed (was 44; +9 regression guards): * recovery happy-path: cache miss → getPosts → reply ref attached to createRecord body; cache re-populated. * deep-thread root preservation: post-with-reply's `record.reply.root` becomes the recovered root, not the immediate parent. * multi-chunk re-uses recovered ref: 3-chunk reply with cache miss = 1 session + 1 getPosts + 3 createRecord (5 calls total, not 7). * recovery failure → top-level skeet WARN. * cross-channel garbage rejected without any HTTP call. * 401 → invalidate session → _get_token re-creates → retry XRPC once with fresh bearer. * 429 → sleep `Retry-After` → retry once. * malformed responses (empty posts, non-dict entry, missing cid, no posts key) → None. * LruCache thread-safety: 8 threads × 200 puts + interleaved reads; cache cap respected, no exceptions raised. - `cd sdk/python && pytest tests/` — full sidecar suite 1859 passed (no regression in any of the other 26 adapters). - Pre-existing `test_post_status_cold_cache_falls_back_to_unthreaded` renamed + updated to assert the getPosts re-fetch fires + 404s + then degrades.
houko
added a commit
that referenced
this pull request
May 22, 2026
…gs (#5450) * fix(sdk): include traceback + cmd_type when on_command bare-except logs The reader() coroutine in librefang.sidecar.runtime swallows exceptions from `adapter.on_command(cmd)` with a bare-except that logged only `error=str(e)`. This silence let four consecutive production reply-routing regressions ship with no warning sign: - #5417 dingtalk (sessionWebhook cache key mismatch) - #5431 qq (cmd.thread_id never set without `thread` cap) - #5439 reddit (raised RuntimeError on every reply; vanished here) - #5439 nextcloud/rocketchat/twitch/bluesky/mastodon (same class) Each took at least one post-merge self-review pass to surface because the original failure logged as `on_command failed error="missing parent fullname"` with no clue about WHICH adapter or WHICH line raised. Operators didn't notice because the bot silently looked healthy — every adapter just stopped replying. The fix is structural-but-conservative: keep the bare-except (we cannot raise out of reader() without taking down the entire sidecar process for one bad command) but add traceback + error_type + cmd_type as structured fields so debuggers can grep the offending adapter / line directly. The richer long-term fix — surfacing on_command failures as a structured JSON-RPC error response so the bridge correlates the failure back to the original cmd id and the agent's error-handling lane fires — needs a protocol change (Response.error field) and is tracked in a follow-up issue. No tests added: the change is purely log-content shape — it doesn't alter exception handling behaviour, just adds context. The protocol-level fix in the follow-up issue is what needs new test coverage. * chore(codegen): auto-regenerate openapi.json + sdk + schema baselines [skip ci] * chore(codegen): auto-regenerate openapi.json + sdk + schema baselines [skip ci] --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Omnibus runtime hotfix for 6 sidecars — same bug class as dingtalk #5423 and qq #5431.
Each of mastodon / nextcloud / reddit / twitch / bluesky / rocketchat reads `cmd.thread_id` in `on_send` to recover a per-message correlation key, but the daemon's bridge only sets `cmd.thread_id` when BOTH `[overrides] threading = true` AND the sidecar declares the `thread` capability. All six declare `capabilities: list = []`, so in production every reply lost the correlation.
Severity ranking
Fix (uniform across all six)
Tests
Each new test drives the realistic daemon shape (`thread_id=None`, `user.librefang_user=`) and asserts the outbound payload contains the correct correlation key. Each fails on the pre-fix code — real regression guards, not just coverage.
Pre-existing structural issue NOT fixed here
`sdk/python/librefang/sidecar/runtime.py:245-247` has a bare-except on `on_command` that swallowed Reddit's `RuntimeError` silently. The immediate fix here is to not raise in the first place (no production path now reaches the missing-parent branch), but the bare-except itself is a footgun — future raises from `on_send` will also vanish to stderr only. Worth a follow-up: re-raise after logging so the bridge can record the failure and trigger the agent's error-handling path.
Investigation chain
The same pattern (sidecars cargo-cult `cmd.thread_id` from each other without re-deriving whether the daemon would actually populate it) keeps reappearing. Worth documenting in `sdk/python/AGENTS.md` so the next sidecar author doesn't ship a 4th iteration.