feat(channels)!: migrate dingtalk from in-process adapter to sidecar (Stream mode only)#5417
Conversation
…(Stream mode only)
BREAKING — Webhook mode is removed. DingTalk moves from a 1,276-line
in-process Rust adapter to a Python sidecar
(librefang.sidecar.adapters.dingtalk). The Stream-mode bot path is
preserved verbatim with four improvements (messageId dedupe via the
shared SeenSet, heartbeat/send coexistence on one socket via a stdlib
queue.Queue, 429 Retry-After honoured on every outbound POST, 15 s
explicit urlopen timeout on every HTTP call — Rust only had it on the
gateway registration). Legacy Webhook mode (HTTP POST callback + HMAC-
SHA256 signature) is NOT ported — Stream mode is the DingTalk-
documented modern default (needs no public IP, no port), simpler to
operate, and a strict superset of Webhook for restricted-egress
deployments. Operators on Webhook mode must re-create the robot in
the DingTalk Open Platform with stream subscription enabled and
migrate credentials from DINGTALK_ACCESS_TOKEN + DINGTALK_SECRET to
DINGTALK_APP_KEY + DINGTALK_APP_SECRET.
Tests: cd sdk/python && pytest tests/test_dingtalk_adapter.py — 61 passed
cargo check --workspace --lib — clean
cargo clippy --workspace --lib --tests -- -D warnings — clean
Stdlib feasibility verified up front: all DingTalk crypto is HMAC-
SHA256 + SHA256 + Base64, all in Python stdlib (hmac / hashlib /
base64). No third-party crypto deps needed; the sidecar SDK's
stdlib-only contract is preserved across all 20 reference adapters.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e96c447a17
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| msg_id = cmd.channel_id or "" | ||
| if msg_id: | ||
| with self._session_lock: | ||
| session_webhook = self._session_webhooks.pop(msg_id, None) |
There was a problem hiding this comment.
Resolve sessionWebhook using the same key on send
_run_session stores session_webhook under the inbound message_id, but on_send looks it up with cmd.channel_id. In the sidecar bridge, cmd.channel_id comes from ChannelUser.platform_id (not platform_message_id), and this adapter never sets channel_id in protocol.message, so normal replies use the sender/chat identity and miss the cache. The result is "no sessionWebhook ... dropping" and outbound replies are discarded for regular DingTalk conversations.
Useful? React with 👍 / 👎.
…r.librefang_user (#5423) PR #5417's on_send tried to recover the per-message sessionWebhook URL in two ways, both wrong: 1. _session_webhooks[cmd.channel_id] — cache keyed by inbound message_id, but the daemon's KernelBridgeAdapter sets cmd.channel_id = user.platform_id (the sender's staffId), so this lookup always misses. 2. cmd.user["session_webhook"] — a key that does not exist in the ChannelUser serialization (grep `crates/` confirms zero hits). Net effect: every outbound reply from a DingTalk agent hit the "no sessionWebhook for message; dropping" warn-and-drop branch. Mirror the deleted Rust adapter's pattern: store the URL in ChannelUser.librefang_user on inbound emit (dingtalk.rs:357), then recover via cmd.user.librefang_user in on_send (was stream_reply_url(user) in the Rust path, dingtalk.rs:233-238). The field gets round-tripped intact through the daemon's bridge code, so the recovery path is reliable without per-process caching. Also reject non-HTTP librefang_user values up-front — the field is shared across channels (Telegram puts the username there, etc.) and we must not POST replies to garbage destinations. Tests: - New test_on_send_recovers_session_webhook_from_user_librefang_user asserts the daemon-shaped (channel_id=staffId, user.librefang_user=url) path works. This test fails on the original PR #5417 sidecar — regression guard for future refactors. - New test_on_send_ignores_non_http_librefang_user pins the HTTP-scheme filter. - Renamed test_on_send_falls_back_to_user_session_webhook (which asserted the dead `user.session_webhook` key path) — that path no longer exists. - cd sdk/python && pytest tests/test_dingtalk_adapter.py — 62 passed Drive-by doc cleanup (CLAUDE.md "fix what you found"): - 4 env-var tables in configuration/{page.mdx,security/page.mdx} + zh equivalents still listed DINGTALK_ACCESS_TOKEN and DINGTALK_SECRET as live webhook-mode credentials; the validator reading them was removed in #5417. Operators following those tables would set vars the sidecar ignores. - The zh integrations/channels/integrations/page.mdx rewritten to drop the now-removed Webhook setup section (the EN version was already rewritten in #5417; the zh one was not, leaving the bilingual docs out of sync). - Validator-table dingtalk row in configuration/page.mdx + zh equivalent updated to point at the sidecar's own env check. Tracking: surfaced by post-merge self-review of #5417.
Surfaces from the post-#5445 audit (6th in the dingtalk/qq/omnibus review chain — first verdict that wasn't BROKEN/NEEDS_HOTFIX). All three nits are doc/UX cleanups, none are runtime regressions: 1. **Docstring claim #3 was wrong** — said "429 Retry-After honoured on every outbound POST", but only Cloud API uses `_cloud_post_with_retry`; the gateway path calls `_http_request` directly and raises on any non-2xx. Soften the claim to "Cloud-API outbound POSTs only" + explain when gateway-side retry would matter (operators proxying the local Baileys gateway behind a rate-limiting reverse-proxy). 2. **WHATSAPP_VERIFY_TOKEN unset failed silently** — Meta's subscription handshake returned 403 with no log line pointing back to the missing env var; operators saw "subscription failed" in the Meta dashboard with nothing in their daemon logs. Now warns at __init__, matching the WHATSAPP_APP_SECRET pattern already there. 3. **WHATSAPP_GROUP_POLICY is dead config** — Cloud API webhook payloads don't surface a group/conversation distinction (we hardcode `is_group=False` at `_handle_post_webhook`), and gateway mode delegates inbound entirely to the Node Baileys gateway which never calls back into the sidecar's `should_handle_message` filter. So `WHATSAPP_GROUP_POLICY` has zero effect today. Mark the schema field with an explicit "(currently inert)" note in the label so operators don't waste time setting it. Kept the field itself for forward-compat — Meta has been rolling out group-chat support to the Cloud API gradually and we want the schema to be ready when it lands. Drive-by: dropped the misleading send-voice docstring claim (lines 22-23 + 36-39 promised a `{gateway}/message/send-voice` multipart upload route that doesn't exist in code; rationale comment about the daemon's ChannelContent::Voice always carrying a URL at the dispatch boundary explains why we don't need it). Test: - New test_get_verify_empty_self_token_rejects asserts BOTH the __init__ warn fires AND the handshake fails closed. Regression guard for #2 — without this an attacker who guesses the empty hub.verify_token could subscribe their own callback URL. - cd sdk/python && pytest tests/test_whatsapp_adapter.py — 79 passed (was 78; +1 regression guard). This closes the audit chain. WhatsApp itself is structurally clean — natural routing key (phone) is preserved as channel_id both directions, the bug family from #5417 → #5449 doesn't apply.
The reader() coroutine in librefang.sidecar.runtime swallows exceptions from `adapter.on_command(cmd)` with a bare-except that logged only `error=str(e)`. This silence let four consecutive production reply-routing regressions ship with no warning sign: - #5417 dingtalk (sessionWebhook cache key mismatch) - #5431 qq (cmd.thread_id never set without `thread` cap) - #5439 reddit (raised RuntimeError on every reply; vanished here) - #5439 nextcloud/rocketchat/twitch/bluesky/mastodon (same class) Each took at least one post-merge self-review pass to surface because the original failure logged as `on_command failed error="missing parent fullname"` with no clue about WHICH adapter or WHICH line raised. Operators didn't notice because the bot silently looked healthy — every adapter just stopped replying. The fix is structural-but-conservative: keep the bare-except (we cannot raise out of reader() without taking down the entire sidecar process for one bad command) but add traceback + error_type + cmd_type as structured fields so debuggers can grep the offending adapter / line directly. The richer long-term fix — surfacing on_command failures as a structured JSON-RPC error response so the bridge correlates the failure back to the original cmd id and the agent's error-handling lane fires — needs a protocol change (Response.error field) and is tracked in a follow-up issue. No tests added: the change is purely log-content shape — it doesn't alter exception handling behaviour, just adds context. The protocol-level fix in the follow-up issue is what needs new test coverage.
Five review-feedback fixes layered on top of the previous commit:
1. Generator script committed to `scripts/gen_sidecar_samples.py`
so future re-runs (new sidecar lands, SCHEMA rotates) don't
have to reconstruct the logic from the PR body. Made
executable. The script's docstring documents the sanitisation
rules and re-run workflow inline.
2. Secret-field SCHEMA hints preserved as trailing comments.
`secret`-type values still render as `"..."` (TOML cleanliness)
but the SCHEMA placeholder, when meaningful, now ships as a
trailing `# <hint>`. Example:
# TWITCH_OAUTH_TOKEN = "..." # oauth:abc123… (prefix auto-added)
# WHATSAPP_APP_SECRET = "..." # optional; (production should always set this)
Operators no longer lose the semantic guidance for the most
security-sensitive fields when the sample collapses the literal.
3. Migration-warning text restored for all 27 adapters that
replaced an in-process channel (every sidecar in this
project — list with PR numbers in
MIGRATED_FROM_IN_PROCESS). Each block now carries:
# Migrated from in-process to sidecar in #<NNN>. Old
# `[channels.<name>]` blocks are no longer recognised.
so an operator upgrading a pre-migration config gets the
explicit "your old block won't parse" signal at the matching
adapter, not just buried in CHANGELOG. Fixes the regression
where the 4 pre-existing blocks' original migration notes
were silently lost.
4. Two-path adapters (whatsapp / wechat) gain a "Requires
EITHER / OR" hint above their env table — their SCHEMAs mark
every var optional because there are two valid configuration
paths (Cloud API vs Baileys for whatsapp; pre-supplied token
vs QR-login for wechat) but the adapter still SystemExit(2)s
with zero config. The hint tells operators which path to
pick. Similarly ntfy carries a privacy note that
`NTFY_SERVER_URL` defaults to the public ntfy.sh server
unless overridden.
5. "(out-of-process sidecar)" suffix stripped from per-block
headers. The section header already documents this for all
27 entries; the per-block repetition was noise.
6. Reordering in `librefang.toml.example`: the generic
"Sidecar channel adapters (out-of-process, any language)"
section that documents protocol-level meta-fields
(`restart`, `restart_initial_backoff_ms`, `message_buffer`,
`overflow`, …) now sits ABOVE the 27 per-adapter blocks.
Operators read the protocol context first, then the specific
instances — matches the natural information-flow order an
operator scanning the file expects.
7. CHANGELOG entry added under `[Unreleased] > Changed`.
Documents the sample fill-in for operators who track release
notes.
8. Also caught + added missing dingtalk migration PR (#5417)
to MIGRATED_FROM_IN_PROCESS; previous revision had 26/27.
Verification:
- `tomllib.load()` on both files — clean
- `cargo xtask schema-check gen && cargo xtask schema-check check`
— baseline regenerated, matches
- 27/27 migration notes present in generated output
- Script is reproducible and idempotent
#5464) * docs(config): fill in [[sidecar_channels]] samples for all 27 sidecar adapters `librefang.toml.example` and `init_default_config.toml` only sampled 4 sidecars (telegram / discord / slack / wechat) — the other 23 have existed since the channel-migration project (#5224 → #5459) but were never added to either config template. Operators running `librefang init` or copy-pasting from the example file had no in-tree guidance for bluesky / dingtalk / email / feishu / google_chat / gotify / line / mastodon / matrix / mattermost / nextcloud / ntfy / qq / reddit / rocketchat / signal / teams / twitch / webex / webhook / wecom / whatsapp / zulip. Each adapter now has a commented `[[sidecar_channels]]` block listing its required env vars (extracted from the adapter's own SCHEMA via `python3 -m librefang.sidecar.adapters.<name> --describe`, so the sample stays in lockstep with the actual adapter contract). Blocks include up to 2 commonly-tuned optional env vars as inline hints; the full inventory is one --describe invocation away. Sanitisations applied during generation: - Secret-type values render as `"..."` — the dashboard SCHEMA hint often contains free-text prose that doesn't belong in a TOML literal. Operators put the real secret in `~/.librefang/secrets.env`, not the sample. - Text placeholders with `"` are escaped. - Description sentences split on `. ` (period + space) so "Rocket.Chat" doesn't truncate to "Rocket". Also drops the stale `[channels.whatsapp]` block from librefang.toml.example — whatsapp is sidecar-only since the migration and the in-process `phone_number_id_env` field no longer exists. Generated content (336 lines) is byte-deterministic from the adapter SCHEMAs — easy to regenerate if a new sidecar lands or an existing one rotates its env vars. The shared "Sidecar channel adapters (out-of-process, any language)" section that documents the protocol meta-fields (restart, backoff, message_buffer) is preserved — it's orthogonal to per-adapter env vars and applies to all 27 sidecars uniformly. Verification: - `python3 -c "import tomllib; tomllib.load(open(p, 'rb'))"` on both sample files — clean (the new content is all `#` comments, parser ignores them). - `cargo xtask schema-check gen && cargo xtask schema-check check` — config.sha256 baseline regenerated and matches. - No Rust source changes; `init_default_config.toml` is `include_str!`'d into `librefang-cli` so the binary picks up the new template on next build automatically. * docs(config): address review feedback on sidecar-sample fill-in Five review-feedback fixes layered on top of the previous commit: 1. Generator script committed to `scripts/gen_sidecar_samples.py` so future re-runs (new sidecar lands, SCHEMA rotates) don't have to reconstruct the logic from the PR body. Made executable. The script's docstring documents the sanitisation rules and re-run workflow inline. 2. Secret-field SCHEMA hints preserved as trailing comments. `secret`-type values still render as `"..."` (TOML cleanliness) but the SCHEMA placeholder, when meaningful, now ships as a trailing `# <hint>`. Example: # TWITCH_OAUTH_TOKEN = "..." # oauth:abc123… (prefix auto-added) # WHATSAPP_APP_SECRET = "..." # optional; (production should always set this) Operators no longer lose the semantic guidance for the most security-sensitive fields when the sample collapses the literal. 3. Migration-warning text restored for all 27 adapters that replaced an in-process channel (every sidecar in this project — list with PR numbers in MIGRATED_FROM_IN_PROCESS). Each block now carries: # Migrated from in-process to sidecar in #<NNN>. Old # `[channels.<name>]` blocks are no longer recognised. so an operator upgrading a pre-migration config gets the explicit "your old block won't parse" signal at the matching adapter, not just buried in CHANGELOG. Fixes the regression where the 4 pre-existing blocks' original migration notes were silently lost. 4. Two-path adapters (whatsapp / wechat) gain a "Requires EITHER / OR" hint above their env table — their SCHEMAs mark every var optional because there are two valid configuration paths (Cloud API vs Baileys for whatsapp; pre-supplied token vs QR-login for wechat) but the adapter still SystemExit(2)s with zero config. The hint tells operators which path to pick. Similarly ntfy carries a privacy note that `NTFY_SERVER_URL` defaults to the public ntfy.sh server unless overridden. 5. "(out-of-process sidecar)" suffix stripped from per-block headers. The section header already documents this for all 27 entries; the per-block repetition was noise. 6. Reordering in `librefang.toml.example`: the generic "Sidecar channel adapters (out-of-process, any language)" section that documents protocol-level meta-fields (`restart`, `restart_initial_backoff_ms`, `message_buffer`, `overflow`, …) now sits ABOVE the 27 per-adapter blocks. Operators read the protocol context first, then the specific instances — matches the natural information-flow order an operator scanning the file expects. 7. CHANGELOG entry added under `[Unreleased] > Changed`. Documents the sample fill-in for operators who track release notes. 8. Also caught + added missing dingtalk migration PR (#5417) to MIGRATED_FROM_IN_PROCESS; previous revision had 26/27. Verification: - `tomllib.load()` on both files — clean - `cargo xtask schema-check gen && cargo xtask schema-check check` — baseline regenerated, matches - 27/27 migration notes present in generated output - Script is reproducible and idempotent * docs(config): surface `pip install librefang-sdk` prerequisite in sidecar samples Operators following the sample config to set up a sidecar channel hit a confusing failure mode at daemon boot when `librefang-sdk` isn't installed in the interpreter the daemon picked: WARN sidecar --describe failed; discovery card will have no form fields adapter="telegram" error=describe exited 1: ... ModuleNotFoundError: No module named 'librefang' The error repeats once per adapter in SIDECAR_CATALOG (27× on a clean install) because daemon boot calls `--describe` on every known adapter to populate the dashboard's Add-Channel form. The sample's existing header documents secrets handling and SCHEMA introspection but never told operators that `python3 -m librefang.sidecar.adapters.<name>` requires the SDK to be importable from the daemon's interpreter. Now it does, prominently — with three details that prior reports surfaced as common confusion: 1. `pip install librefang-sdk` (PyPI package name) OR `pip install -e sdk/python/` from a source checkout. 2. Multi-interpreter footgun: mise / pyenv / conda commonly pick a different `python3` for the daemon than for the operator's shell. The sample now says so and gives a one-line verification command. 3. Verification snippet: `python3 -c 'import librefang.sidecar; print(librefang.__file__)'` from the same shell that starts librefang. Also tags the regeneration command (`scripts/gen_sidecar_samples.py`) as "for librefang maintainers only — operators can ignore this" so an operator who copies the sample file as their config and tries to follow every command in the header doesn't waste time looking for a librefang source checkout. This is a docs-only change. Both sample files re-validated as clean TOML; `cargo xtask schema-check gen` refreshes the config baseline to match. The companion runtime + discovery rewrite (translating the daemon's `ModuleNotFoundError` into the SAME actionable install hint operators see in the sample header) ships as PR #5465.
…gs (#5450) * fix(sdk): include traceback + cmd_type when on_command bare-except logs The reader() coroutine in librefang.sidecar.runtime swallows exceptions from `adapter.on_command(cmd)` with a bare-except that logged only `error=str(e)`. This silence let four consecutive production reply-routing regressions ship with no warning sign: - #5417 dingtalk (sessionWebhook cache key mismatch) - #5431 qq (cmd.thread_id never set without `thread` cap) - #5439 reddit (raised RuntimeError on every reply; vanished here) - #5439 nextcloud/rocketchat/twitch/bluesky/mastodon (same class) Each took at least one post-merge self-review pass to surface because the original failure logged as `on_command failed error="missing parent fullname"` with no clue about WHICH adapter or WHICH line raised. Operators didn't notice because the bot silently looked healthy — every adapter just stopped replying. The fix is structural-but-conservative: keep the bare-except (we cannot raise out of reader() without taking down the entire sidecar process for one bad command) but add traceback + error_type + cmd_type as structured fields so debuggers can grep the offending adapter / line directly. The richer long-term fix — surfacing on_command failures as a structured JSON-RPC error response so the bridge correlates the failure back to the original cmd id and the agent's error-handling lane fires — needs a protocol change (Response.error field) and is tracked in a follow-up issue. No tests added: the change is purely log-content shape — it doesn't alter exception handling behaviour, just adds context. The protocol-level fix in the follow-up issue is what needs new test coverage. * chore(codegen): auto-regenerate openapi.json + sdk + schema baselines [skip ci] * chore(codegen): auto-regenerate openapi.json + sdk + schema baselines [skip ci] --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Summary
Operator action required
Existing `[channels.dingtalk]` block is no longer recognised. Re-declare as `[[sidecar_channels]]`:
```toml
[[sidecar_channels]]
name = "dingtalk"
command = "python3"
args = ["-m", "librefang.sidecar.adapters.dingtalk"]
channel_type = "dingtalk"
[sidecar_channels.env]
DINGTALK_APP_KEY = "dingxxxxxxxx"
DINGTALK_ALLOWED_USERS = "staff-1,staff-2" # optional CSV staffId allowlist
DINGTALK_ACCOUNT_ID = "prod-bot" # optional multi-bot routing
```
Secret via `~/.librefang/secrets.env`: `DINGTALK_APP_SECRET`.
`channel_type = "dingtalk"` preserves the channel-type token so existing routing / `channel_role_mapping` keys keep resolving.
Webhook-mode operators must re-create the robot in the DingTalk Open Platform with stream subscription enabled and migrate credentials from `DINGTALK_ACCESS_TOKEN` + `DINGTALK_SECRET` to `DINGTALK_APP_KEY` + `DINGTALK_APP_SECRET`. Stream mode requires no public IP, so it works in every deployment topology Webhook supported.
What's deleted
What's added
Test plan
Stdlib feasibility audit
All DingTalk crypto uses HMAC-SHA256 + SHA256 + Base64 + URL encoding — all in Python stdlib (`hmac` / `hashlib` / `base64` / `urllib.parse`). The sidecar SDK's stdlib-only contract is preserved across all 20 reference adapters; no third-party crypto deps were considered.