feat(channels)!: migrate ntfy from in-process adapter to sidecar (P7)#5224
Conversation
Sidecar-first POC (follows the #5219/#5220/#5221 series): proves the end-to-end pipeline by replacing a real in-process channel with a sidecar and removing the Rust adapter. BREAKING: the `[channels.ntfy]` config block is no longer recognised. Re-declare it as a `[[sidecar_channels]]` running the new examples/sidecar-channel-python/ntfy_adapter.py (header documents the exact config). The separate ntfy *push-notification provider* (`push_provider="ntfy"` / `ntfy_url` / `ntfy_topic`, device pairing) is a different feature and is deliberately left intact. Replacement (behaviour-preserving), stdlib-only on the librefang.sidecar SDK: examples/sidecar-channel-python/ntfy_adapter.py — same SSE parse, `/command` handling, title→sender, topic metadata, 4096-char chunked plain-text publish, optional Bearer auth, exponential-backoff reconnect. Removed in-process ntfy (full cascade — wider than the phase plan estimated; user approved the breaking full-removal path): - librefang-channels: src/ntfy.rs (deleted), lib.rs mod, Cargo.toml `channel-ntfy` feature, channels-allowlist.txt entry (so the P5 `cargo xtask channel-policy` gate now permanently blocks any in-process ntfy reintroduction). - librefang-types: NtfyConfig struct + Default, `channels.ntfy` field + default, validation hook. - librefang-api: `channel-ntfy` feature + its membership in core-channels/all-channels/all-channels-no-email; channel_bridge use+block+macros+test assertion; routes/channels.rs ChannelMeta descriptor + 4 match arms; routes/config.rs `ch!(ntfy)`. - librefang-kernel: channel_sender registry macro + name list. - librefang-cli: TUI ChannelDef + Cargo.toml core-channels comment. - kernel-config golden regenerated (NtfyConfig removed; 0 residual). Verification: - cargo check --workspace --lib: clean - cargo clippy --workspace --all-targets -- -D warnings: clean - cargo test -p librefang-channels (847) / -p librefang-types / -p xtask (channel-policy gate green WITH ntfy delisted — real_repo_tree_is_green ok) all pass - cargo test -p librefang-api: all pass EXCEPT 2 pre-existing failures unrelated to this PR — auth_public_allowlist.rs's `dashboard_read_paths_are_in_catalog` / `dashboard_read_routes_reachable_without_auth_when_no_key_configured` fail on `/api/cron/list` (a cron route allowlist drift on main; this PR touches no auth middleware / route registration / cron code — evidence: the offending path is cron, not ntfy/channels) - cd sdk/python && pytest: 22 passed incl. tests/test_ntfy_adapter.py (SSE parse / command vs text / sender / chunked publish + auth / unsupported-content fallback, urllib mocked)
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 04c1224455
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| is_group=True, | ||
| metadata={"topic": topic or self.topic}, |
There was a problem hiding this comment.
Preserve ntfy account_id on inbound messages
When NTFY_ACCOUNT_ID is set, the sidecar announces it in ready but never copies it into each message's metadata. The bridge resolves account-scoped bindings and channel overrides from message.metadata["account_id"] (for example in resolve_or_fallback), and the removed Rust adapter explicitly injected that metadata, so multi-topic/multi-bot ntfy setups will route inbound messages as an unqualified ntfy channel instead of the configured account-specific binding. Include account_id in the emitted metadata when self.account_id is present.
Useful? React with 👍 / 👎.
Documents the sidecar-first model realized by the #5219/#5220/#5221 (+#5224) series. - New docs/architecture/sidecar-channels.md (house style): process model + ASCII supervisor state machine, the JSON-RPC protocol tables, capability negotiation, backoff/circuit-break/backpressure, the daemon-restart-vs-platform-reconnect responsibility split, the allowlist policy gate, ntfy as the worked migration, and the long-tail backlog policy (opportunistic, not a batch rewrite — informed by P7 revealing the per-channel removal blast radius). - CONTRIBUTING.md "How to Add a New Channel Adapter" rewritten: sidecar-first is the path (SDK subclass + [[sidecar_channels]] + template/ntfy reference); the in-process Rust recipe is replaced by a "maintainers only — allowlist + reviewed exception" note. Also fixes the pre-existing inaccurate trait sketch in that section (it showed `start(&mut self)->Result<()>` / `send(&self,&str,&str)` / `stop(&mut self)`, none of which match the real `ChannelAdapter` signatures) by removing the stale sample rather than maintaining a wrong one. - crates/librefang-channels/AGENTS.md (the CLAUDE.md symlink target) "Adding a new channel": sidecar-first pointer + the pre-commit / `cargo xtask channel-policy` allowlist gate; the grandfathered-adapter rules (wiremock send test, webhook wiring, off-by-default feature) are kept. Docs-only; no code. Cross-references verified against `main` (SDK template, channels-allowlist.txt, pre-commit, xtask channel-policy all present). One forward reference: examples/sidecar-channel-python/ntfy_adapter.py arrives in #5224 — the architecture doc PR-attributes it to #5224, so it reads as series documentation; this PR should land with or after #5224 so the CONTRIBUTING pointer resolves.
houko
left a comment
There was a problem hiding this comment.
Self-review (author): blocking-style review, recorded as a comment because GitHub does not allow request-changes on own PR. Architecture and migration story are excellent — this is a clean P7 landing on top of the merged P0–P5 chain (#5219/#5220/#5221), the CHANGELOG entry is thorough and explicitly flags the breaking change, the operator migration path is concrete (re-declare as [[sidecar_channels]] running ntfy_adapter.py), the P5 channel-policy gate is engaged (ntfy delisted from channels-allowlist.txt), the kernel-config golden is regenerated with zero NtfyConfig residue, and the new Python sidecar adapter is behaviour-preserving with solid test coverage (SSE parse, command vs text, sender, chunked publish, auth, unsupported-content fallback). The deliberate preservation of the separate push_provider = "ntfy" push-notification feature is correctly called out and verified — the only remaining ntfy references in librefang-types::config::types.rs are those (intentional) push-provider doc comments. Two items to address before merge, both small.
1. Stale documentation pointing at removed code. After this PR, three files still reference the deleted channel-ntfy cargo feature and the deleted crates/librefang-channels/src/ntfy.rs:
crates/librefang-channels/AGENTS.md:16— listschannel-ntfyas a per-adapter feature flag. The flag no longer exists.crates/librefang-channels/README.md:20— same list, same issue.examples/custom-channel/README.md:275— citesntfy.rsas a worked reference (SSE subscription + plain POST publishing) for contributors authoring a new channel. That file is gone in this PR, so the cross-reference will be broken onmain.
These three lines should be removed or rewritten (e.g. the custom-channel README row can either be dropped or repointed at examples/sidecar-channel-python/ntfy_adapter.py as the sidecar reference, which is consistent with P6 / #5225's doc direction). The web/public/registry.json ntfy entry is not in scope here — that file is fetched from the separate librefang-registry repo by web/scripts/fetch-registry.ts and needs a follow-up PR there.
2. CI baseline disclosure is incomplete and the run is currently red. The PR body discloses two pre-existing auth_public_allowlist failures on /api/cron/list and attributes them to an unrelated cron-route drift on main. That is correct, and they are fixed on main by #5217 (fix(api,dashboard): expose 'background' section + drop stale '/api/cron/list' allowlist row), which landed after this PR was opened. However the current Coverage run also shows a third failure not mentioned in the body — librefang-api::config_schema_overlay::every_kernel_config_struct_field_is_exposed_via_overlay, panicking on KernelConfig fields with no x-sections descriptor and no entry in EXCLUDED: background. That failure is also pre-existing (the background field was added by #5063 on main, the EXCLUDED list was never updated, and #5217 is the same commit that fixes it). So all three failures clear with a rebase, but please rebase on current main and update the PR body's verification section to list the third failure too, so the CI green/red state matches the disclosure exactly — important for a breaking change PR where reviewers rely on the body to triage red checks.
Nothing structural. Once the three doc lines are cleaned up and the branch is rebased to clear CI, this is ready for squash-merge.
…nto librefang-sdk (#5228) The ntfy / telegram / webhook adapters were sitting under examples/sidecar-channel-python/ alongside the generic template, but they are first-party production code: ntfy in particular is the only ntfy adapter that exists after #5224 deleted the in-process Rust one. "Example" misclassifies them and forces operators to hand-copy a script into their tree. Move them into the SDK package so pip install librefang-sdk ships them and `-m librefang.sidecar.adapters.<name>` runs them from anywhere: examples/sidecar-channel-python/{ntfy,telegram,webhook}_adapter.py -> sdk/python/librefang/sidecar/adapters/{ntfy,telegram,webhook}.py Adapter logic is unchanged. Header config examples updated from file-path args to `-m` module args. Generic template adapter.py stays in examples/ — that one is genuinely a starting point to copy. * test_ntfy_adapter: drop sys.path hack, import via the package. * kernel_config_schema golden: regenerated (only the SidecarChannelConfig doc-comment example string changed). * CHANGELOG [Unreleased]: new entry documenting the operator-visible args change (`examples/...py` -> `-m librefang.sidecar.adapters.ntfy`) and updates to the still-unreleased #5224 entry to reference the new path. * docs/architecture, CONTRIBUTING, channels README/AGENTS, channels-allowlist.txt header, channels.rs comment, pre-commit hook error text: all reference the new SDK path. Verification: cargo check on touched crates clean; the 5 ntfy adapter tests pass at the new import path; `from librefang.sidecar.adapters import ntfy, webhook` smoke import succeeds.
#5241) * feat(channels)!: remove in-process telegram adapter (now sidecar-only) BREAKING: the `[channels.telegram]` config block is no longer recognised. Telegram now runs as a `[[sidecar_channels]]` adapter (`python3 -m librefang.sidecar.adapters.telegram`, channel_type = "telegram"); the Python adapter reached full parity in the preceding commits of this PR. Same pattern as the ntfy #5224 migration, scaled to telegram's deeper integration. Removed in-process telegram (full cascade): - librefang-channels: src/telegram.rs (deleted, 6231 lines), lib.rs mod, Cargo.toml `channel-telegram` feature + all-channels, channels-allowlist.txt entry (so `cargo xtask channel-policy` permanently blocks any in-process telegram reintroduction), commands/mod.rs stale doc ref. - librefang-types: `TelegramConfig` struct + Default + impl, the `channels.telegram` field + default, validation hook, the orphaned `default_telegram_long_poll_timeout_secs`. The shared deny_unknown_fields / proxy doc anchor moved from `TelegramConfig` to `DiscordConfig` (it was the canonical cross-reference for ~7 other channel structs). - librefang-api: `channel-telegram` feature + membership in core-channels / all-channels / all-channels-no-email / mini; channel_bridge use + Duration cfg + coalesce-window arm + check_channel + spawn block + default-empty test; routes/channels.rs ChannelMeta + the 5 match arms; routes/config.rs `ch!(telegram)`. - librefang-kernel: channel_sender `for_each_channel_field!` macro + name list; config_reload hot-reload test. - librefang-cli: TUI ChannelDef, interactive `channel setup` wizard arm + picker entry + health list, Cargo.toml/help-text examples. - OpenClaw / openfang migrators: telegram is no longer emitted as `[channels.telegram]` (the kernel would reject it); the bot token is still migrated to secrets.env and the channel is reported as skipped with guidance to declare a `[[sidecar_channels]]` block. - `ChannelType::Telegram`, the shared bot-command registry, the bridge channel-type->string map and `[channel_role_mapping.telegram]` RBAC mapping are deliberately kept (shared core / different feature); the sidecar telegram still flows through them. - Tests that used telegram as the arbitrary example channel migrated to discord/slack; kernel-config golden regenerated (no residual TelegramConfig). Verification: cargo check --workspace --lib clean; cargo xtask channel-policy Passed (telegram delisted, allowlist only shrinks); cargo test -p librefang-types / -p librefang-channels / -p librefang-migrate all pass; -p librefang-api routes::channels + channels_routes_test + config_schema_golden pass; clippy (touched crates, --lib) clean. Two failures verified unrelated (zero telegram coupling, files unmodified by this branch, paths orthogonal to the cascade): hooks_agent_dispatches_to_default_agent e2e and the kernel/tests.rs:8972 compact_agent_session_with_id arity drift — same class #5224 documented as pre-existing. * docs: redirect [channels.telegram] examples to sidecar pattern (#5241) Replace every copy-pasteable [channels.telegram] config block in EN and ZH docs with the equivalent [[sidecar_channels]] snippet. Illustrative overrides examples that used [channels.telegram.overrides] as a placeholder are updated to [channels.discord.overrides]. Config templates (librefang.toml.example, init_default_config.toml) are updated to match. * chore(cli,docs): drop in-process telegram from help-text and per-crate docs (#5241) Remove "telegram" from CLI help suggestions (channel setup doc-comment, channel test long_about examples), Fluent locale channel-unknown-fix lists (EN and zh-CN), and the orphan section-setup-telegram key. Update librefang-channels AGENTS.md and README.md to reflect that telegram joined ntfy as a sidecar-only adapter. * docs(rustdoc): re-anchor proxy contract from TelegramAdapter to DiscordConfig (#5241) Three with_proxy() doc-comments in discord.rs, slack.rs, and mattermost.rs cross-referenced the deleted TelegramAdapter::with_proxy for the URL contract. Re-anchor to DiscordConfig::proxy, which is the canonical proxy doc since the telegram removal. Also update two bridge_integration_test.rs doc-comments that named TelegramAdapter as an example. Update ChannelsConfig field docstring in types.rs to remove "Telegram, etc." and sync the golden schema fixture. * chore(changelog): note in-process telegram removal as BREAKING (#5241) Add [Unreleased] BREAKING entry for the telegram adapter migration: explains that [channels.telegram] is no longer accepted under deny_unknown_fields, what operators must do (re-declare as [[sidecar_channels]]), and what is preserved (TELEGRAM_BOT_TOKEN, channel_type=telegram, RBAC mapping). * chore(codegen): auto-regenerate openapi.json + sdk + schema baselines [skip ci] --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
…d (no 404 actions) (#5249) * fix(api): surface configured sidecar channels on /api/channels Regression from the sidecar migration: when telegram (#5241) and ntfy (#5224) moved out-of-process, their ChannelMeta entries were removed from CHANNEL_REGISTRY — which silently dropped them from the dashboard channels page (/dashboard/channels reads /api/channels, which only enumerated CHANNEL_REGISTRY). An operator who had telegram/ntfy working saw them vanish with no indication of where they went; the operator experience was no longer consistent across in-process vs sidecar adapters. Fix (backend-only, dashboard renders it unchanged): /api/channels and channels_snapshot now also synthesize a row per configured [[sidecar_channels]] entry — name/display, configured=true, category/setup_type="sidecar", instance_count, no editable `fields` (config.toml-managed), and setup_steps pointing the operator at Config -> Sidecar Channels. The ChannelsPage already conditionally hides empty fields/setup_steps, so these render as normal configured/online cards with zero frontend changes. configured_count counts them. A sidecar entry never shadows or duplicates an in-process CHANNEL_REGISTRY row of the same name. Tests (crates/librefang-api/tests/channels_routes_test.rs): - channels_list_includes_configured_sidecar_channels: a configured [[sidecar_channels]] telegram appears as a configured sidecar row, exactly once, counted in configured_count, with no editable fields. - channels_list_without_sidecar_has_no_sidecar_rows: negative control — no synthetic rows leak when none are configured. Verification: cargo check -p librefang-api --lib clean; cargo test -p librefang-api --test channels_routes_test 30 passed; cargo clippy -p librefang-api --lib --tests clean. * fix(dashboard): sidecar channel cards are read-only (no 404 actions) Self-review of the previous commit found a half-fix: surfacing sidecar channels on /api/channels made the cards appear, but every interactive affordance 404s for a sidecar name — `get_channel` / `configure_channel` / `test_channel` are all CHANNEL_REGISTRY-only (find_channel_meta → None → 404). So the card showed up but "Configure" / "Test" were broken. ChannelsPage now treats a `category === "sidecar"` row as read-only: - the inline Configure (Settings) affordance on the card is suppressed (whole-card click still opens the read-only details drawer); - the DetailsModal's Configure/Test action buttons are replaced with a note: runs as an out-of-process sidecar, manage it in config.toml ([[sidecar_channels]]) under Config → Sidecar Channels. DetailsModal renders purely from the list-row data (no per-channel fetch), so opening details on a sidecar row was already safe; only the two mutating/registry actions needed gating. The selection checkbox is inert (no bulk action consumes it), so no other path 404s. Net: sidecar channels (telegram / ntfy / …) are visible AND have no broken interactions — the migration is transparent to operators and the card honestly points at where the channel is actually managed. Frontend-only; `category` is already a typed field used elsewhere in this file.
…mple (#5248) #5241 deleted crates/librefang-channels/src/telegram.rs (telegram is now a sidecar adapter) but the "reference adapters" table in the custom-channel example still listed telegram.rs as an existing in-process implementation to study — a dangling file reference that landed on main with the squash-merge. Swap it for matrix.rs (a real in-process adapter not already in the table), exactly as #5224 swapped ntfy's row for discord when it removed ntfy.rs.
…age after sidecar migration (#5250) Follow-up to #5241 (telegram) and #5224 (ntfy). #5249 fixed the configured case — once an operator declared [[sidecar_channels]], a configured/online card appeared. But operators who had never configured them saw nothing: the Add picker (`pickerChannels` filters on `configured: false` from /api/channels) had no telegram/ntfy entry because both were removed from CHANNEL_REGISTRY, and the new `sidecar_channel_rows` synthesizer only emits rows for entries that already exist in config.toml. Net effect: the only way to discover the two first-party SDK sidecar adapters was to read source or release notes. Fix: add a small SIDECAR_CATALOG (telegram, ntfy) next to CHANNEL_REGISTRY and emit `configured: false, category: "sidecar"` rows for catalog entries not covered by any [[sidecar_channels]]. A catalog entry is "covered" when ANY existing sidecar entry has either `name == entry` or `channel_type == entry`, so a user-aliased `name = "my_alerts", channel_type = "telegram"` correctly suppresses the telegram discovery row instead of double-rendering. `webhook` is deliberately excluded from the catalog — it still has an in-process CHANNEL_REGISTRY entry, and surfacing both would render two cards with the same name. Dashboard: - ChannelItem gains `config_template?: string` (backend already emits it). - `handlePick` routes `category === "sidecar"` to the read-only details modal instead of opening the configure form (which would 404 against /api/channels/{name}/configure — that endpoint is CHANNEL_REGISTRY-only). - DetailsModal renders the TOML snippet for unconfigured sidecar rows so operators can copy it into config.toml directly. Tests (crates/librefang-api/tests/channels_routes_test.rs): - Replaced `channels_list_without_sidecar_has_no_sidecar_rows` (which asserted the now-wrong "no rows when unconfigured" contract) with `channels_list_without_sidecar_surfaces_discovery_catalog`, which asserts both telegram and ntfy appear as unconfigured catalog rows with `category="sidecar"`, no editable fields, and a display_name; also asserts discovery rows don't bump configured_count. - Added `channels_list_discovery_row_hidden_when_kind_configured`: a configured `name="my_alerts", channel_type="telegram"` must suppress the telegram discovery row while leaving the ntfy one in place. Verification: - cargo check -p librefang-api --lib --tests: clean - cargo clippy -p librefang-api --lib --tests -- -D warnings: clean - cargo test -p librefang-api --test channels_routes_test: 31/31 pass - pnpm typecheck: clean - pnpm test --run: 650/650 pass - pnpm build: success
…5263) * feat(channels)!: migrate gotify from in-process adapter to sidecar BREAKING: the `[channels.gotify]` config block is no longer recognised. Gotify now runs as a `[[sidecar_channels]]` adapter (`python3 -m librefang.sidecar.adapters.gotify`, channel_type = "gotify"); the new Python adapter ships in `librefang-sdk` with full parity for inbound WebSocket subscribe + outbound REST publish. Same pattern as the ntfy (#5224) and telegram (#5241) migrations. Added sidecar: - `sdk/python/librefang/sidecar/adapters/gotify.py` — preserves the Rust adapter's runtime semantics. WebSocket `/stream` subscribe with the client token in the query string; JSON frames decoded to `(id, message, title, priority, appid)`; empty `message` skipped; `/`-prefixed text → `Command`, otherwise `Text`; sender derived from `title` (fallback `app-{appid}`); metadata carries title / priority / app_id; reconnect with exponential backoff (1s → 60s). REST `/message` publish posts `{title:"LibreFang", message, priority:5}` with the app token in the query string; chunked at 65535 chars suffixing `(i/N)` to the title. Optional `GOTIFY_ACCOUNT_ID` env var surfaces as the multi-bot routing key. - The WebSocket client is a hand-rolled RFC 6455 reader on `socket` + `ssl` — no third-party WS dependency, keeping the SDK zero-dep (matches ntfy / telegram). Server frames are read unmasked; client→server frames (only pong / close) carry a fresh 4-byte mask per frame as the spec requires. Fragmented messages are reassembled before yielding. - `sdk/python/tests/test_gotify_adapter.py` — 22 unit tests: URL/scheme normalization, required-env enforcement, JSON frame parsing including empty-message skip and bad-JSON, command vs text routing, sender fallback to app-id, message chunking with numbered titles, HTTP 500 / HTTPError surfacing, and a loopback-server end-to-end test that performs a real RFC 6455 handshake and asserts both text frames are yielded as decoded strings. Removed in-process gotify (full cascade): - `librefang-channels`: `src/gotify.rs` deleted (649 lines), `lib.rs` mod, `Cargo.toml` `channel-gotify` feature + `all-channels` membership, `channels-allowlist.txt` entry (so `cargo xtask channel-policy` permanently blocks any in-process gotify reintroduction). Stale comment references in `discord.rs` / `slack.rs` / `teams.rs` ("the gotify slice (PR #4447)") generalised to "the wiremock pattern" — the pattern is still in use, just no longer rooted in a gotify example. - `librefang-types`: `GotifyConfig` struct + Default + the `channels.gotify` field + default, validation hook. - `librefang-api`: `channel-gotify` feature + membership in `all-channels` / `all-channels-no-email`; channel_bridge `use` + spawn block + `check_channel` arm + `find_channel_info` arm + default-empty test assertion; `routes/channels.rs` ChannelMeta + the 4 match arms (`gotify_some` / `serde_to_value` / `len` / `ser`); `routes/config.rs` `ch!(gotify)`. - `librefang-kernel`: `channel_sender` `for_each_channel_field!` macro entry + name list. - `librefang-cli`: TUI `ChannelDef`, with the Notifications-section header generalised since gotify joins ntfy off-tree. - `docs/architecture/sidecar-channels.md`: update the long-tail backlog note to record gotify joining ntfy + telegram. The separate gotify push-notification provider (`push_provider = "gotify"`, used by device pairing under `crates/librefang-kernel/ src/pairing.rs`) is **unaffected** — different feature, deliberately left intact. Operator action required: # config.toml: remove [channels.gotify] server_url = "..." app_token_env = "GOTIFY_APP_TOKEN" client_token_env = "GOTIFY_CLIENT_TOKEN" # config.toml: add [[sidecar_channels]] name = "gotify" command = "python3" args = ["-m", "librefang.sidecar.adapters.gotify"] channel_type = "gotify" [sidecar_channels.env] GOTIFY_SERVER_URL = "https://gotify.example.com" GOTIFY_APP_TOKEN = "A..." # secrets.env via secrets system GOTIFY_CLIENT_TOKEN = "C..." # secrets.env via secrets system Verification: - `cd sdk/python && pytest`: 97 tests pass (22 new gotify, 75 pre-existing ntfy/telegram/runtime/protocol all green). - Loopback WebSocket handshake test exercises real RFC 6455 Sec-WebSocket-Accept computation and frame parsing. - `cargo check / clippy / test` deferred to CI — the host's cargo is a Docker wrapper with `rust:latest` not cached locally, and `cargo test --workspace` is forbidden by repo policy. - `kernel_config_schema.golden.json` left with 12 residual gotify refs — the matching test is `#[ignore]`'d on `main` (golden has drifted across earlier migrations); regen is a follow-up via `cargo test -p librefang-api --test config_schema_golden -- --ignored regenerate_golden --nocapture`. * docs(channels): redirect [channels.gotify] examples to sidecar pattern Self-review follow-up to the gotify sidecar migration. The main commit deleted the in-process `GotifyConfig` / `GotifyAdapter` but left several docs pages still showing the old `[channels.gotify]` block and pointing at the deleted Rust adapter. Telegram's #5241 follow-up landed the same kind of redirect — mirror that pattern. - `docs/.../configuration/page.mdx` (en + zh): replace the old `[channels.gotify]` field reference with a redirect-style `[[sidecar_channels]]` example, mirroring the existing Telegram redirect. Also drop the Gotify row from the "validator-checked env vars" table — the validation hook was removed with `GotifyConfig`. - `docs/.../configuration/channels/page.mdx` (en + zh): same redirect. - `docs/.../integrations/channels/integrations/page.mdx` (en + zh): the "## Gotify" section was already stale before this PR — it used hypothetical `GOTIFY_URL` / `GOTIFY_TOKEN` env vars and a `[channels.gotify]` block whose `url_env` / `token_env` keys never matched the actual `GotifyConfig` shape. Rewritten to point at the sidecar with the correct `GOTIFY_SERVER_URL` / `GOTIFY_APP_TOKEN` / `GOTIFY_CLIENT_TOKEN` env var names. - `docs/.../security/integrity/page.mdx`, `docs/.../architecture/security/page.mdx` (en + zh): drop the `GotifyAdapter` row from the "Zeroizing<String> usage in in-process channel adapters" table — the struct is gone with this PR; the env vars themselves are now read by the Python sidecar process, which inherits the daemon's environment without needing in-Rust zeroizing. - `examples/custom-channel/README.md`: drop the `gotify` row from the "study these existing adapters" comparison table — `gotify.rs` no longer exists. The env-var tables under "Required Environment Variables" at `configuration/page.mdx` and `configuration/security/page.mdx` are deliberately kept — the env vars themselves are still in use, just read by the sidecar process rather than the deleted in-process adapter. No code changes; docs-only follow-up to #5263. * fix(sidecar/gotify): cap inbound WS frame payload + harden coverage Round-2 self-review follow-up. Cap inbound WebSocket frame payloads at 1 MiB (MAX_FRAME_PAYLOAD). RFC 6455 allows up to 2^63 bytes via the 64-bit extended-length form; the previous unbounded implementation would faithfully try to read any length the server announced. A hostile or buggy gotify server could send a 64-bit length and force the sidecar to spin in `_recv_exact` allocating multi-exabyte buffers. The cap is well over Gotify's largest realistic payload (a 65 535-char message body wrapped in a JSON envelope sits under 100 KiB), so legitimate traffic is unaffected. Cap breach raises a RuntimeError, which the producer loop catches and treats as a transport error → reconnect with the existing exponential backoff. Three new tests close earlier coverage gaps: - `test_websocket_reader_responds_to_server_ping_with_pong`: drives the reader against a loopback server that sends `text → ping → text → close`. Asserts both text frames are yielded AND that the client sent back exactly one masked pong carrying the same payload as the ping, plus the close echo. Exercises the previously-untested control-frame branches. - `test_websocket_reader_rejects_oversized_frame`: server announces a 1 GiB 64-bit-length text frame; client must raise RuntimeError matching "exceeds cap" instead of trying to allocate. Locks in the new MAX_FRAME_PAYLOAD guard. - `test_account_id_surfaced_via_ready_event` / `test_no_account_id_when_unset`: assert that `GOTIFY_ACCOUNT_ID` reaches LibreFang via the base SidecarAdapter.ready_event() multi-bot routing key (and that an empty env var emits no `account_id` field) — closes the gap noted in the first review. Verification: `cd sdk/python && pytest` — 101 tests pass (4 new gotify + 22 prior gotify + 75 unchanged ntfy/telegram/runtime).
…5264) * feat(channels)!: migrate mastodon from in-process adapter to sidecar BREAKING: the `[channels.mastodon]` config block is no longer recognised. Mastodon now runs as a `[[sidecar_channels]]` adapter (`python3 -m librefang.sidecar.adapters.mastodon`, channel_type = "mastodon"); the new Python adapter ships in `librefang-sdk` with full parity for SSE primary + REST polling fallback inbound and form-encoded REST outbound. Same pattern as the ntfy (#5224), telegram (#5241), and gotify (#5263) migrations. Added sidecar: - `sdk/python/librefang/sidecar/adapters/mastodon.py` — preserves the Rust adapter's runtime semantics. SSE subscribe to `{instance}/api/v1/streaming/user` with Bearer auth; filter `event: notification` payloads to `type == "mention"`; strip HTML from `status.content` (`<br>` / `</p>` / `</div>` / `</li>` insert newlines, entities decoded via stdlib `html.unescape`); `/cmd args` → Command, else Text; sender from `display_name` fallback `username`; metadata carries status_id / notification_id / acct / visibility / in_reply_to_id; `thread_id` = in_reply_to_id; `verify_credentials` at startup discovers the bot's own account id so self-mentions are skipped; REST polling fallback when SSE fails (newest-first ordering, high-water-mark before iteration); exponential-backoff reconnect (1s → 60s). - Outbound: POST `/api/v1/statuses` form-encoded with `status`, `visibility="unlisted"`, optional `in_reply_to_id`. Chunked at `MAX_MESSAGE_LEN` (default 500); chained chunks thread under each prior response id so a long reply stays one conversational thread. Replies to inbound mentions use `cmd.thread_id` (in_reply_to_id). `suppress_error_responses = True` — Mastodon posts are public. - New env-var knobs beyond the in-process equivalents: `MASTODON_VISIBILITY` (public/unlisted/private/direct, default unlisted), `MASTODON_MAX_MESSAGE_LEN` (default 500, raise for longer-toot instances), optional `MASTODON_ACCOUNT_ID` for multi-bot routing surfaced via `ready_event`. - `sdk/python/tests/test_mastodon_adapter.py` — 32 unit tests: URL/scheme normalization, required-env enforcement, visibility allowlist, max_len validation, account_id surfacing, HTML stripper edge cases (plain passthrough, basic tags, block-close newlines, entity decoding, typical mention anchor shape), notification parsing (full shape, thread_id from in_reply_to, non-mention skip, self-mention skip, empty text skip, slash command, display_name fallback), message chunking, REST publish (bearer auth + form + visibility, chunked thread chaining, reply to inbound thread, HTTPError surfaced, 5xx surfaced, custom visibility). Removed in-process mastodon (full cascade): - `librefang-channels`: `src/mastodon.rs` deleted (850 lines), `lib.rs` mod, `Cargo.toml` `channel-mastodon` feature + `all-channels` membership, `channels-allowlist.txt` entry (so `cargo xtask channel-policy` permanently blocks any in-process mastodon reintroduction). - `librefang-types`: `MastodonConfig` struct + Default + the `channels.mastodon` field + default, validation hook. - `librefang-api`: `channel-mastodon` feature + membership in `all-channels` / `all-channels-no-email`; channel_bridge `use` + spawn block + `check_channel` arm + `find_channel_info` arm + default-empty test assertion; `routes/channels.rs` ChannelMeta + 4 match arms (`mastodon_some` / `serde_to_value` / `len` / `ser`); `routes/config.rs` `ch!(mastodon)`. Added Mastodon to SIDECAR_CATALOG so the dashboard Channels page surfaces a configurable discovery row. - `librefang-kernel`: `channel_sender` `for_each_channel_field!` macro entry + name list. - `librefang-cli`: TUI `ChannelDef`. - `docs/src/app/*`: redirect both English and Chinese config / channels / integrations pages from `[channels.mastodon]` to `[[sidecar_channels]]`. Operator action required: # config.toml: remove [channels.mastodon] instance_url = "..." access_token_env = "MASTODON_ACCESS_TOKEN" # config.toml: add [[sidecar_channels]] name = "mastodon" command = "python3" args = ["-m", "librefang.sidecar.adapters.mastodon"] channel_type = "mastodon" [sidecar_channels.env] MASTODON_INSTANCE_URL = "https://mastodon.social" # secrets.env via secrets system: # MASTODON_ACCESS_TOKEN=... Verification: - `cd sdk/python && pytest`: 133 tests pass (32 new mastodon, 26 prior gotify, 75 unchanged ntfy/telegram/runtime/protocol). - `cargo check / clippy / test` deferred to CI — the host's cargo is a Docker wrapper with `rust:latest` not cached locally, and workspace-wide `cargo test` is forbidden by repo policy. - `kernel_config_schema.golden.json`: the matching test is `#[ignore]`'d on main (the fixture has drifted across earlier migrations); regen is a follow-up via `cargo test -p librefang-api --test config_schema_golden -- --ignored regenerate_golden`. * fix(sidecar/mastodon): call _verify_credentials before SSE/poll loop Self-review follow-up. The Rust adapter called `validate()` at the start of `ChannelAdapter::start()` to (a) validate the OAuth token and (b) discover the bot's own account id used by the self-mention guard in `parse_mastodon_notification`. My port defined `_verify_credentials` but never actually called it from `_producer_blocking`, so `self.own_account_id` stayed `None` and the guard: if self.own_account_id and account_id == self.own_account_id: return None silently short-circuited — a self-mention from the bot's own toots would be delivered back to the agent as an inbound event, potentially looping if the agent's reply itself triggers another mention. Same parity bug the in-process adapter explicitly defended against. Fix: `_producer_blocking` now runs verify with exponential backoff (1s → 60s) before entering the SSE-with-polling-fallback loop. The loop refuses to emit any event until verify has succeeded at least once. Transient verify failures (instance returning 5xx, network flake) retry instead of taking the sidecar down; persistent auth failures keep retrying — the supervisor can kill the subprocess if the operator wants a hard fail. Three new tests close coverage gaps the original PR missed: - `test_verify_credentials_sets_own_account_id`: monkeypatch urlopen, assert the request hits `/api/v1/accounts/verify_credentials` with the right Bearer header AND that the response `id` is stored on `own_account_id`. Locks in the discovery contract. - `test_verify_credentials_raises_on_bad_token`: a 401 response surfaces the HTTPError; `own_account_id` stays None so the self-mention guard does not match anything by accident. - `test_self_mention_skipped_only_after_verify`: the failing case — without verify, a self-mention IS delivered (proving the latent bug shape); after `own_account_id` is set the same payload is filtered. Catches a future regression where someone removes the verify call again. Verification: `cd sdk/python && pytest` — 35 mastodon tests pass (+3 new), full SDK suite 136 passed.
…5297) Replaces the 535-line in-process librefang-channels::twitch adapter with the Python sidecar librefang.sidecar.adapters.twitch. Same pattern as ntfy #5224, telegram #5241, gotify #5263, mastodon #5264, bluesky #5277, reddit #5281. Behaviour parity: - OAuth PASS / NICK handshake (oauth: prefix auto-added when absent) - JOIN #channel for each configured channel - PRIVMSG -> ChannelMessage with case-insensitive self-skip - /cmd / !cmd -> Content::Command, else Content::Text - PING -> PONG keepalive - MAX_MESSAGE_LEN = 500 chunking - Exponential reconnect backoff (1s -> 60s) - TWITCH_ACCOUNT_ID multi-bot routing Three improvements over the Rust adapter (file/line evidence): 1. TLS by default. The Rust adapter hard-coded plaintext port 6667 (TWITCH_IRC_PORT in crates/librefang-channels/src/twitch.rs:24) and sent the OAuth token in cleartext on every connect. The sidecar defaults to irc.chat.twitch.tv:6697 with ssl.create_default_context(). Plaintext is reachable only via TWITCH_PLAINTEXT=1 (tests/mock listeners). Credential-leak-on-wire fixed automatically on upgrade. 2. Per-message reply threading via IRCv3 tags. The Rust adapter never requested any capability and discarded any @-tag block (parse_privmsg in twitch.rs:100 only handled the bare ":nick!u@h PRIVMSG #ch :text" shape), so chunked replies arrived as a flat sequence with no link back. The sidecar issues CAP REQ :twitch.tv/tags twitch.tv/commands, parses @id, surfaces it as thread_id, and attaches @reply-parent-msg-id=<id> on outbound PRIVMSG. 3. Token-bucket send rate-limiter. Twitch drops the bot above 20 msgs / 30 s (100 / 30 s if mod). The Rust adapter shipped zero throttling (send() in twitch.rs:312 wrote PRIVMSG frames straight to the socket). Sidecar gates every outbound chunk; tunable via TWITCH_RATE_LIMIT_MSGS / TWITCH_RATE_LIMIT_SECS. Cascade removal: crates/librefang-channels/src/twitch.rs deleted; lib.rs mod decl, channels-allowlist.txt entry, channel-twitch feature in librefang-channels + librefang-api (incl. all-channels / all-channels-no-email membership); TwitchConfig + ChannelsConfig.twitch field + Default; validation env-var hook; channel_bridge import + spawn block + find_channel_info! arm + check_channel! arm + default- empty test assertion; routes/channels.rs ChannelMeta + 4 match arms; routes/config.rs ch!(twitch); kernel channel_sender macro entry + EXPECTED name; CLI TUI ChannelDef; types/config/mod.rs unit tests referencing TwitchConfig; docs (en + zh): configuration / channels / integrations / architecture / communication pages -- [channels.twitch] blocks replaced with [[sidecar_channels]] redirect; architecture-row moves twitch into the sidecar list. New env-var knobs: TWITCH_NICK, TWITCH_CHANNELS (comma-separated, no '#'), optional TWITCH_ACCOUNT_ID, optional TWITCH_RATE_LIMIT_MSGS / TWITCH_RATE_LIMIT_SECS, optional TWITCH_PLAINTEXT / TWITCH_HOST / TWITCH_PORT (test escape hatches). Operator action required: an existing [channels.twitch] block is no longer recognised -- re-declare as a [[sidecar_channels]] running python3 -m librefang.sidecar.adapters.twitch with env vars TWITCH_NICK, TWITCH_CHANNELS (in [sidecar_channels.env]) and TWITCH_OAUTH_TOKEN (~/.librefang/secrets.env). Verification: - cargo check --workspace --lib: clean - cargo clippy -p librefang-types -p librefang-channels -p librefang-kernel -p librefang-api -p librefang-cli --all-targets -- -D warnings: clean - cargo test -p librefang-types -p librefang-channels -p librefang-kernel --lib: 843 passed - cargo test -p librefang-api --lib: 686 passed - cargo test -p librefang-api --test channels_routes_test --test sidecar_describe_test: 38 + 2 passed - cd sdk/python && python -m pytest tests/test_twitch_adapter.py: 68 passed
Replaces the 640-line in-process librefang-channels::nextcloud adapter with the Python sidecar librefang.sidecar.adapters.nextcloud. Same pattern as ntfy #5224, telegram #5241, gotify #5263, mastodon #5264, bluesky #5277, reddit #5281, twitch #5297, rocketchat #5298. Behaviour parity: - GET /ocs/v2.php/cloud/user startup credential probe - Per-room chat/<token>?lookIntoFuture=1 polling at 3 s default - Empty allowlist auto-discovers via apps/spreed/api/v4/room - Same Bearer + OCS-APIRequest: true headers - Slash-command (/cmd args) routing - Multi-bot account_id metadata injection - 32000-char message chunking (MAX_MESSAGE_LEN) - Per-room transport-error isolation - 304-as-no-op handling of Talk's long-poll-expired response - suppress_error_responses (Talk rooms are multi-participant) Three improvements over the Rust adapter (file/line evidence in CHANGELOG): 1. Outbound threading via replyTo is now actually wired. The Rust api_send_message (nextcloud.rs:130-160) posted just {"message": ...} — Talk's replyTo form parameter was never sent and chunked/threaded replies always landed at the room root. The sidecar surfaces the inbound id (or parentMessage.id when inside a thread) as thread_id, and on_send posts replyTo so replies thread correctly. Mirrors reddit/bluesky/mastodon/ rocketchat. 2. Self-skip on (actorType, actorId) == ("users", own_user_id) rather than actorId alone. The Rust adapter compared on actorId (nextcloud.rs:338) without inspecting actorType, so a Talk guest/federated_users actor whose id happened to match the bot's user id silently spoofed self-skip. Parallels the rocketchat #5298 fix. 3. Dedupe set on id, bounded SEEN_MESSAGES_MAX=10000 / SEEN_MESSAGES_EVICT=5000. The Rust adapter only relied on the server-side lastKnownMessageId cursor (nextcloud.rs:347-354); under retry/re-poll boundaries Talk can resend the same id. Same policy as reddit/rocketchat. Cascade removal: crates/librefang-channels/src/nextcloud.rs deleted; lib.rs mod decl, channels-allowlist.txt entry, channel-nextcloud feature in librefang-channels + librefang-api (incl. all-channels / all-channels-no-email membership); NextcloudConfig + ChannelsConfig.nextcloud field + Default; validation env-var hook; channel_bridge import + spawn block + find_channel_info! arm + check_channel! arm + default-empty test assertion; routes/channels.rs ChannelMeta + 4 match arms; routes/config.rs ch!(nextcloud); kernel channel_sender macro entry + EXPECTED name; CLI TUI ChannelDef; docs (en + zh): configuration / channels / integrations / architecture / communication / security pages — [channels.nextcloud] blocks replaced with [[sidecar_channels]] redirect; architecture-row moves Nextcloud Talk into the sidecar list. routes/channels.rs SIDECAR_CATALOG gains a nextcloud entry. Golden fixture regenerated. New env-var knobs: NEXTCLOUD_SERVER_URL (replaces server_url), optional NEXTCLOUD_ROOMS (comma-separated room tokens, empty = auto-discover), optional NEXTCLOUD_ACCOUNT_ID for multi-bot routing, optional NEXTCLOUD_POLL_INTERVAL_SECS (default 3, floor 1). Operator action required: an existing [channels.nextcloud] block is no longer recognised — re-declare as a [[sidecar_channels]] running python3 -m librefang.sidecar.adapters.nextcloud with env var NEXTCLOUD_SERVER_URL (in [sidecar_channels.env]) and NEXTCLOUD_TOKEN (in ~/.librefang/secrets.env). Verification: - cargo check --workspace --lib: clean - cargo clippy --workspace --all-targets -- -D warnings: clean - cargo test -p librefang-types -p librefang-channels -p librefang-kernel --lib: 842 passed - cargo test -p librefang-api --lib: 686 passed - cargo test -p librefang-api --test channels_routes_test --test sidecar_describe_test: 38 + 2 passed - cd sdk/python && pytest tests/test_nextcloud_adapter.py: 58 passed
…#5301) Replaces the 640-line in-process librefang-channels::nextcloud adapter with the Python sidecar librefang.sidecar.adapters.nextcloud. Same pattern as ntfy #5224, telegram #5241, gotify #5263, mastodon #5264, bluesky #5277, reddit #5281, twitch #5297, rocketchat #5298. Behaviour parity: - GET /ocs/v2.php/cloud/user startup credential probe - Per-room chat/<token>?lookIntoFuture=1 polling at 3 s default - Empty allowlist auto-discovers via apps/spreed/api/v4/room - Same Bearer + OCS-APIRequest: true headers - Slash-command (/cmd args) routing - Multi-bot account_id metadata injection - 32000-char message chunking (MAX_MESSAGE_LEN) - Per-room transport-error isolation - 304-as-no-op handling of Talk's long-poll-expired response - suppress_error_responses (Talk rooms are multi-participant) Three improvements over the Rust adapter (file/line evidence in CHANGELOG): 1. Outbound threading via replyTo is now actually wired. The Rust api_send_message (nextcloud.rs:130-160) posted just {"message": ...} — Talk's replyTo form parameter was never sent and chunked/threaded replies always landed at the room root. The sidecar surfaces the inbound id (or parentMessage.id when inside a thread) as thread_id, and on_send posts replyTo so replies thread correctly. Mirrors reddit/bluesky/mastodon/ rocketchat. 2. Self-skip on (actorType, actorId) == ("users", own_user_id) rather than actorId alone. The Rust adapter compared on actorId (nextcloud.rs:338) without inspecting actorType, so a Talk guest/federated_users actor whose id happened to match the bot's user id silently spoofed self-skip. Parallels the rocketchat #5298 fix. 3. Dedupe set on id, bounded SEEN_MESSAGES_MAX=10000 / SEEN_MESSAGES_EVICT=5000. The Rust adapter only relied on the server-side lastKnownMessageId cursor (nextcloud.rs:347-354); under retry/re-poll boundaries Talk can resend the same id. Same policy as reddit/rocketchat. Cascade removal: crates/librefang-channels/src/nextcloud.rs deleted; lib.rs mod decl, channels-allowlist.txt entry, channel-nextcloud feature in librefang-channels + librefang-api (incl. all-channels / all-channels-no-email membership); NextcloudConfig + ChannelsConfig.nextcloud field + Default; validation env-var hook; channel_bridge import + spawn block + find_channel_info! arm + check_channel! arm + default-empty test assertion; routes/channels.rs ChannelMeta + 4 match arms; routes/config.rs ch!(nextcloud); kernel channel_sender macro entry + EXPECTED name; CLI TUI ChannelDef; docs (en + zh): configuration / channels / integrations / architecture / communication / security pages — [channels.nextcloud] blocks replaced with [[sidecar_channels]] redirect; architecture-row moves Nextcloud Talk into the sidecar list. routes/channels.rs SIDECAR_CATALOG gains a nextcloud entry. Golden fixture regenerated. New env-var knobs: NEXTCLOUD_SERVER_URL (replaces server_url), optional NEXTCLOUD_ROOMS (comma-separated room tokens, empty = auto-discover), optional NEXTCLOUD_ACCOUNT_ID for multi-bot routing, optional NEXTCLOUD_POLL_INTERVAL_SECS (default 3, floor 1). Operator action required: an existing [channels.nextcloud] block is no longer recognised — re-declare as a [[sidecar_channels]] running python3 -m librefang.sidecar.adapters.nextcloud with env var NEXTCLOUD_SERVER_URL (in [sidecar_channels.env]) and NEXTCLOUD_TOKEN (in ~/.librefang/secrets.env). Verification: - cargo check --workspace --lib: clean - cargo clippy --workspace --all-targets -- -D warnings: clean - cargo test -p librefang-types -p librefang-channels -p librefang-kernel --lib: 842 passed - cargo test -p librefang-api --lib: 686 passed - cargo test -p librefang-api --test channels_routes_test --test sidecar_describe_test: 38 + 2 passed - cd sdk/python && pytest tests/test_nextcloud_adapter.py: 58 passed
) * feat(channels)!: migrate webex from in-process adapter to sidecar Replaces the 645-line in-process librefang-channels::webex adapter (Cisco Mercury WebSocket gateway for activity events + GET /messages/<id> REST follow-up + POST /messages publish) with the Python sidecar librefang.sidecar.adapters.webex. Same pattern as ntfy #5224, telegram #5241, gotify #5263, mastodon #5264, bluesky #5277, reddit #5281, twitch #5297, rocketchat #5298, discord #5299, nextcloud #5301, slack #5302. Behaviour parity: - GET /people/me startup credential probe (bot id + display name) - Hard-coded Mercury WSS URL with Bearer auth on upgrade request (no device-registration handshake — the Rust adapter never did one either) - data.activity envelope parsing; verb=="post" filter; actor-id self-skip - GET /messages/<id> REST follow-up for the full message body - Room filter (empty allowlist = all rooms the bot is in) - Slash-command (/cmd args) → Command routing - roomType == "group" → is_group mapping - Multi-bot account_id metadata injection - 7439-char message chunking (WEBEX_MSG_LIMIT parity with the Rust MAX_MESSAGE_LEN) - Exponential reconnect backoff (1s → 60s) - ChannelType::Custom("webex") preserved as channel_type = "webex" on the sidecar entry — existing routing / channel_role_mapping keys that reference webex continue to resolve. Four improvements over the Rust adapter (file/line evidence in CHANGELOG): 1. parentId outbound threading wired. The Rust api_send_message (webex.rs:171-201) posted just {"roomId", "text"} — Webex's parentId field was never sent and chunked replies always landed at the room root regardless of inbound context; the inbound side dropped the message id entirely (thread_id: None at webex.rs:438). The sidecar surfaces the inbound id (or inbound parentId when inside a thread) as thread_id, and on_send posts parentId populated so replies thread correctly. Mirrors reddit / rocketchat / nextcloud / mastodon / bluesky. 2. 429 Retry-After honoured on both GET /messages/<id> fetch and POST /messages send. The Rust adapter had no 429 handling at either call site, so a server-side rate-limit either lost the inbound fetch or returned Err and dropped the outbound. The sidecar parses Retry-After (default 30 s fallback, floor 1 s, cap MAX_BACKOFF_SECS), sleeps, and retries once before logging-and-continuing on the second 429. Same shape as #5303. 3. Mercury activity-id dedupe. Mercury can re-deliver an activity.object.id on reconnect (Rust adapter had no dedupe, see unconditional emit at webex.rs:459). Bounded local set on activity.object.id with SEEN_MESSAGES_MAX=10000 / SEEN_MESSAGES_EVICT=5000. 4. Explicit HTTP timeouts on every urlopen call (urllib has no default timeout; a hung Webex API would hang the producer thread forever). Cascade removal: - src/webex.rs deleted; lib.rs mod decl, channels-allowlist.txt entry, cargo features in librefang-channels + librefang-api (incl. membership in all-channels / all-channels-no-email) removed - WebexConfig struct + Default impl + ChannelsConfig.webex field + matching Default entry removed; validation hook removed - channel_bridge.rs: import, spawn block, find_channel_info! arm, check_channel! arm, default-empty test assertion removed - routes/channels.rs: ChannelMeta entry + 4 match arms removed; SIDECAR_CATALOG webex entry added - routes/config.rs ch!(webex) call removed - kernel channel_sender for_each_channel_field! macro entry + EXPECTED name-list entry removed - CLI: TUI ChannelDef row removed, migrated-to-sidecar comment grown - README / AGENTS for librefang-channels updated to list webex among the migrated-to-sidecar adapters - Docs (EN + ZH): configuration/page.mdx, configuration/channels/page.mdx, integrations/channels/enterprise/page.mdx, configuration/security/page.mdx, architecture/page.mdx — [channels.webex] blocks replaced with [[sidecar_channels]] redirect blocks (fence language `bash` per Shiki); architecture wave table moves webex out of Wave 3 in-process and into the sidecar list New env-var knobs (read from [sidecar_channels.env]): - WEBEX_ALLOWED_ROOMS (comma-separated room IDs, empty = allow all) - WEBEX_ACCOUNT_ID (optional, multi-bot routing key) Secret in ~/.librefang/secrets.env: WEBEX_BOT_TOKEN. Operator action required: an existing [channels.webex] block is no longer recognised — re-declare as a [[sidecar_channels]] running python3 -m librefang.sidecar.adapters.webex with WEBEX_BOT_TOKEN (in ~/.librefang/secrets.env) and any of the optional knobs above (in [sidecar_channels.env]). Verification: - cargo check --workspace --lib: clean - cargo clippy --workspace --all-targets -- -D warnings: clean - cargo test -p librefang-types -p librefang-channels -p librefang-kernel --lib: 420 + 838 + 1087 passed - cargo test -p librefang-api --lib: 685 passed - cargo test -p librefang-api --test channels_routes_test --test sidecar_describe_test: 38 + 2 passed - cd sdk/python && pytest tests/test_webex_adapter.py: 78 passed * fix(channels/webex): apply review nits — display name, suppress flag, dead code, redundant filter Self-review pass on top of the migration commit; addresses 4 nits from the pre-merge review without changing the migration's core behaviour: 1. **personDisplayName drives user_name** (`webex.py:281-291, 326-330`). The migration commit preserved the Rust adapter's `webex.rs:431` behaviour of using `personEmail` unconditionally, which leaks emails into bot logs and dashboard UI. Now `parse_webex_message` prefers `personDisplayName` from the `/messages/<id>` body and falls back to `personEmail` when the field is absent (older Webex orgs / some service-account principals) or empty; `personEmail` stays in metadata for routing / audit. 3 new tests (`test_parse_prefers_person_display_name_for_user_name`, `…_falls_back_to_email_when_display_name_missing`, `…_falls_back_to_email_when_display_name_empty`). 2. **Explicit `suppress_error_responses = False`** on `WebexAdapter` with a precedent comment naming the discord / slack / twitch chat-room default vs. the mastodon / bluesky / reddit / nextcloud public-broadcast default. The migration left this implicit; an explicit class-level attr matches how `twitch.py:312` documents the choice so the next reader doesn't have to reconstruct the reasoning. 3. **`register_webhook` dead-code drop documented in CHANGELOG**. The Rust adapter carried a never-wired `register_webhook` helper at `webex.rs:137-168` (marked `#[allow(dead_code)]`) for an HTTP-webhook delivery alternative the channel-bridge never enabled; the sidecar drops it. The CHANGELOG now names this explicitly and points operators who want webhook delivery at the canonical generic `[[sidecar_channels]]` running `librefang.sidecar.adapters.webhook`. 4. **Skip redundant room filter inside `parse_webex_message`** when called from `_handle_envelope` (`webex.py:867-878`). `_handle_envelope` already filtered against `self.allowed_rooms` at line 855 before paying for the REST fetch; passing `allowed_rooms=[]` into the parser drops one redundant list-membership check per inbound message. The parser keeps the parameter for direct callers (tests; future code paths). Verification (run inside the worktree): * `cd sdk/python && python3 -m pytest tests/test_webex_adapter.py -q` → **81 passed** (was 78; +3 personDisplayName tests). No existing test modified. * `cargo check` / `clippy` not re-run — this commit is Python + CHANGELOG only, no Rust touched. Diff: +73 / -5 across `CHANGELOG.md`, `sdk/python/librefang/sidecar/adapters/webex.py`, `sdk/python/tests/test_webex_adapter.py`.
* feat(channels)!: migrate line from in-process adapter to sidecar Replaces the 881-line in-process librefang-channels::line adapter (axum webhook route at /channels/line/webhook on the shared API server, X-Line-Signature HMAC-SHA256 verification, POST /v2/bot/message/push for outbound) with the Python sidecar librefang.sidecar.adapters.line. Same pattern as ntfy #5224, telegram #5241, gotify #5263, mastodon #5264, bluesky #5277, reddit #5281, twitch #5297, rocketchat #5298, discord #5299, nextcloud #5301, slack #5302, webex #5309. Behaviour parity: - GET /v2/bot/info startup credential probe (bot display name) - HTTP webhook server (BaseHTTPRequestHandler over ThreadingTCPServer) on LINE_WEBHOOK_PORT (default 9090) + LINE_WEBHOOK_PATH (default /webhook) - X-Line-Signature HMAC-SHA256 verification over the raw wire bytes (constant-time compare, line.rs:229-250 parity, incl. the wire-bytes-vs-JSON-roundtrip regression) - Only message events with type==text (other event types and non-text message types dropped, line.rs:256-273) - Source-type -> reply_to mapping (user -> userId, group -> groupId, room -> roomId; group/room -> is_group=true, line.rs:280-290) - Slash-command routing (/cmd args -> Command) - Metadata preservation: user_id / reply_to / reply_token / source_type - Multi-bot account_id injection via LINE_ACCOUNT_ID - 5000-char chunking (LINE_MSG_LIMIT parity with MAX_MESSAGE_LEN at line.rs:39) - Image-branch wire shape (originalContentUrl + previewImageUrl both set to the caller URL, caption sent as follow-up text push, line.rs:464-490) - ChannelType::Custom("line") preserved as channel_type = "line" on the sidecar entry so existing routing / channel_role_mapping keys continue to resolve Three improvements over the Rust adapter: 1. 429 Retry-After honoured on outbound. The Rust api_push_message (line.rs:148-184) had no 429 handling; throttled push returned Err and the chunked reply dropped on the floor. Sidecar parses Retry-After (default 30 s fallback, floor 1 s, cap MAX_BACKOFF_SECS), sleeps, retries once, then logs-and-continues on the second 429. Same shape as #5303. 2. Inbound dedupe on message.id. LINE redelivers webhook events when the operator's endpoint fails; the Rust handler at line.rs:413-427 emitted every event unconditionally, causing duplicate agent invocations on transient failures. Bounded local set on message.id with SEEN_MESSAGES_MAX=10000 / SEEN_MESSAGES_EVICT=5000 (matches reddit / rocketchat / nextcloud / webex policy). 3. Explicit HTTP timeouts on every urlopen call (urllib has no default timeout; a hung LINE API would otherwise hang the worker thread forever). Cascade removal: - src/line.rs deleted; lib.rs mod decl, channels-allowlist.txt entry, cargo features in librefang-channels + librefang-api (incl. membership in all-channels / all-channels-no-email) removed - LineConfig struct + Default impl + ChannelsConfig.line field + matching Default entry removed; validation hook removed - channel_bridge.rs: import, spawn block, find_channel_info! arm, check_channel! arm, default-empty test assertion removed - routes/channels.rs: ChannelMeta entry + 5 match arms removed (is_some / serialize / len / ser / is_channel_configured); webhook_route_suffix allowlist entry removed (sidecar runs its own webhook listener, not on the shared API port); SIDECAR_CATALOG line entry added - routes/config.rs ch!(line) call removed - kernel channel_sender for_each_channel_field! entry + EXPECTED name-list entry removed - CLI: TUI ChannelDef row removed, migrated-to-sidecar comment grown - README / AGENTS for librefang-channels updated; the mandatory- HMAC paragraph now lists Teams + DingTalk only (LINE moves to the sidecar but keeps the same X-Line-Signature contract inside the new process) - Docs (EN + ZH): configuration/page.mdx, configuration/channels/page.mdx, configuration/security/page.mdx, integrations/channels/integrations/page.mdx - [channels.line] blocks replaced with [[sidecar_channels]] redirect blocks; security env-var table flags both LINE_CHANNEL_SECRET and LINE_CHANNEL_ACCESS_TOKEN as sidecar-only; validation table drops the LINE row New env-var knobs (read from [sidecar_channels.env]): - LINE_WEBHOOK_PORT (default 9090) - LINE_WEBHOOK_PATH (default /webhook) - LINE_BIND_HOST (default 0.0.0.0) - LINE_ACCOUNT_ID (optional, multi-bot routing key) Secrets in ~/.librefang/secrets.env: LINE_CHANNEL_SECRET + LINE_CHANNEL_ACCESS_TOKEN. Operator action required (substantive): the sidecar runs its OWN HTTP webhook server - it is no longer mounted on the LibreFang API port - so the webhook URL registered at the LINE Developers Console must be updated to point at the sidecar host (typical pattern: an HTTPS reverse proxy in front of the sidecar's LINE_WEBHOOK_PORT). An existing [channels.line] block is no longer recognised - re-declare as a [[sidecar_channels]] running python3 -m librefang.sidecar.adapters.line with LINE_CHANNEL_SECRET + LINE_CHANNEL_ACCESS_TOKEN (in ~/.librefang/secrets.env) and any of the optional knobs above (in [sidecar_channels.env]). Verification: - cargo check --workspace --lib: clean - cargo clippy --workspace --all-targets -- -D warnings: clean - cargo fmt --check: clean - cargo test --lib -p librefang-types -p librefang-channels -p librefang-kernel: 420 + 838 + 1087 passed - cargo test --lib -p librefang-api: 685 passed - cargo test -p librefang-api --test channels_routes_test --test sidecar_describe_test: 38 + 2 passed - cargo xtask channel-policy: passed - cd sdk/python && pytest tests/test_line_adapter.py: 68 passed * docs(channels): correct line sidecar HTTP server class references The module docstring claimed `ThreadingHTTPServer` but the implementation uses `socketserver.ThreadingTCPServer` (intentional: skips `HTTPServer.server_bind()`'s `socket.getfqdn()` DNS lookup on startup). Align the three doc references with the actual class and spell out why we don't use `ThreadingHTTPServer` so the next reader doesn't "fix" it back.
Replaces the 758-line in-process `librefang-channels::qq` adapter (token mint + WebSocket gateway + HELLO/IDENTIFY/READY handshake + heartbeat + DISPATCH for 4 event types + REST send with markdown stripping) with an out-of-process Python sidecar at `sdk/python/librefang/sidecar/adapters/qq.py`. Same pattern as ntfy #5224, telegram #5241, gotify #5263, mastodon #5264, bluesky #5277, reddit #5281, twitch #5297, rocketchat #5298, discord #5299, nextcloud #5301, slack #5302, webex #5309, line #5312, zulip #5310, mattermost #5315, signal #5317. Behaviour parity (every claim cited against the pre-migration tree): - Token fetch: POST bots.qq.com/app/getAppAccessToken (qq.rs:542-557). - Gateway: GET /gateway with Bearer auth (qq.rs:559-573). - WS handshake: HELLO(op=10) → IDENTIFY(op=2) with token = "QQBot <access_token>", intents bitmask, shard=[0,1] → READY dispatch (qq.rs:413-433). - Heartbeat: op=1 frame at the server-supplied interval, carrying last seen `s` from dispatch frames (qq.rs:359-368). - 4 dispatch event types: MESSAGE_CREATE / AT_MESSAGE_CREATE → /channels/{channel_id}/messages, DIRECT_MESSAGE_CREATE → /dms/{guild_id}/messages, GROUP_AT_MESSAGE_CREATE → /v2/groups/{group_openid}/messages, C2C_MESSAGE_CREATE → /v2/users/{user_openid}/messages (qq.rs:194-224). - Bot-mention `/` strip + slash-command routing (qq.rs:227). - User allowlist + multi-bot account_id metadata (#5003, qq.rs:400-405). - Outbound markdown stripping pipeline preserves order: think tags → code blocks → inline code → bold → italic → headings → table separators → links → blockquotes → HRs → triple-newline collapse (qq.rs:137-180). - 2000-char chunking (qq.rs:26), 1–60 s exponential reconnect backoff (qq.rs:282). Improvements over the Rust adapter ================================== 1. **Reply context actually round-trips.** The Rust `parse_dispatch_event` (qq.rs:182-246) computed `reply_endpoint` and `msg_id` but the dispatch loop bound them to `_endpoint` / `_msg_id` (qq.rs:399) and dropped them on the floor; `send` (qq.rs:497-498) then expected `user.platform_id` to be encoded as `"<endpoint>|<msg_id>"` and silently no-op'd when the delimiter wasn't there. The Rust adapter therefore failed every real outbound — only the synthetic wiremock tests at qq.rs:686-712 exercised the working shape. The sidecar surfaces the reply endpoint as `channel_id` and the QQ `msg_id` as `thread_id` on the inbound event so the bridge round-trips them through to `on_send`, which posts to `{api_base}{channel_id}` with the correct passive-reply `msg_id`. 2. **Inbound dedupe on `msg.id`.** Rust emitted every parsed event unconditionally (qq.rs:399-410); WS reconnect could replay. Bounded local set, SEEN_MAX=10000 / EVICT=5000. 3. **429 Retry-After honoured on every REST path.** Token, gateway, and outbound send all sleep + retry once; second 429 logs-and-continues (matches webex / line / mattermost / signal #5303). 4. **Explicit 15s urlopen timeouts on every REST call.** Rust pre-configured reqwest's 30s default at qq.rs:71; sidecar passes `timeout=SEND_TIMEOUT_SECS` (15 s) so a misbehaving endpoint trips an explicit error. Cascade removal ================ - `src/qq.rs` deleted; `lib.rs` mod, `channels-allowlist.txt` entry, both `Cargo.toml` features removed. - `QqConfig` struct + Default + `ChannelsConfig.qq` field gone. - `channel_bridge.rs` import / spawn block / `check_channel!` / find_channel_info!(qq) (none existed) removed. - `routes/channels.rs` `ChannelMeta` entry + 4 match arms (`is_some` / serialize / `len` / `ser`) removed; SIDECAR_CATALOG `qq` entry added. - `routes/config.rs` `ch!(qq)` removed. - `channel_sender.rs` `for_each_channel_field!` macro entry + `EXPECTED` name-list entry removed. - CLI / TUI had no QQ wizard arm to remove. - librefang-migrate had no QQ block to handle (QQ wasn't in the OpenClaw schema). - Docs EN + ZH (6 files): `[channels.qq]` blocks replaced with `[[sidecar_channels]]` redirects; integrations tutorial rewritten for the sidecar. New env-var knobs (`[sidecar_channels.env]`) ============================================= - `QQ_APP_ID` (required) - `QQ_ALLOWED_USERS` (optional, CSV) - `QQ_ACCOUNT_ID` (optional) - `QQ_INTENTS` (optional, decimal/hex bitmask override) Secret (`~/.librefang/secrets.env`) ==================================== - `QQ_APP_SECRET` (required) Operator action required ======================== An existing `[channels.qq]` block is no longer recognised — re-declare as `[[sidecar_channels]]` running `python3 -m librefang.sidecar.adapters.qq`. See `sdk/python/librefang/sidecar/adapters/qq.py` header for the exact config. `ChannelType::Custom("qq")` is preserved via `channel_type = "qq"` on the sidecar entry so existing routing and `channel_role_mapping` keys continue to resolve. Verification ============ - `cargo check -p librefang-types -p librefang-channels -p librefang-api -p librefang-kernel -p librefang-cli --lib` — clean - `cargo xtask channel-policy` — passed (`qq` removed from allowlist) - `pytest sdk/python/tests/test_qq_adapter.py -q` — **77 passed** (covers env parsing, markdown stripping, parse_qq_event for all 4 event types + edge cases, _mark_seen LRU eviction, _fetch_token + _fetch_gateway happy path / 429 / non-200 / missing field, _post_message chunking + 429 retry-once + persistent-429 fail-open + 5xx fail-open, on_send wiring with markdown stripping at the outbound boundary, WS gateway flow via mock _WebSocketClient — HELLO → IDENTIFY token+intents+ shard, DISPATCH emission, msg.id dedupe, RECONNECT/INVALID_SESSION, heartbeat after interval, --describe schema, capabilities contract). Co-authored-by: Evan <[email protected]>
…5368) * feat(channels)!: migrate matrix from in-process adapter to sidecar Replaces the 3356-line in-process `librefang-channels::matrix` adapter (long-poll /sync + PUT /rooms/{}/send/{}/{} + media upload + reaction lifecycle + streaming-edit m.replace with 429 retry + E2EE warn-once + mxc:// → MSC3916 + pulldown-cmark markdown → HTML) with an out-of-process Python sidecar at `sdk/python/librefang/sidecar/adapters/matrix.py`. Same pattern as ntfy #5224 through qq #5325. Behaviour parity (each citation against matrix.rs on the pre-migration tree): - /sync long-poll with `since` cursor + 30s server timeout (matrix.rs:855-1008). - Room allowlist + self-skip on `sender == user_id` (matrix.rs:940-962). - E2EE warn-once per room (matrix.rs:948-954). - 5 inbound msgtype dispatch (m.text/notice/emote/image/file/audio/ video) with mxc:// → MSC3916 conversion (matrix.rs:311-343). - `m.thread` relation → `thread_id` on inbound (matrix.rs:206-215). - 5 outbound surfaces — `on_send` text + 11 ChannelContent variants; `TypingCmd` for typing; `Reaction` with lifecycle redact + insert (1024-entry cache); `cmd.thread_id` wrap for m.thread reply; `StreamStart` / `StreamDelta` / `StreamEnd` for m.replace streaming edits. - `m.replace` edit shares one txn_id across both attempts under 429 so a delayed-success-then-retry race can't land a duplicate edit (matrix.rs:737-774). - MAX_MESSAGE_LEN = 4096 chunking (matrix.rs:22). - 1–60 s exponential reconnect backoff on /sync failure (matrix.rs:879, 914). - Multi-bot account_id metadata (#5003). Improvements over the Rust adapter ================================== 1. Inbound dedupe on event_id. Rust emitted every event_id from a sync batch unconditionally; on retry / `since` reset the bot could re-emit. Bounded SeenSet, SEEN_MAX=10000 / EVICT=5000. 2. 429 Retry-After honoured at every PUT, not just edit. `_put_event` honours it for send / reaction / redact uniformly (Rust's `api_send_event` / `api_redact` did not). 3. Explicit 60s timeout on /sync, 30s on every other REST call. Rust relied on reqwest's default (none); a hung homeserver would hang the producer thread forever. Markdown → HTML: stdlib-only subset renderer covering headings, bold, italic, inline code, fenced code blocks, links (with javascript: / data: scheme rejection), lists, blockquotes, horizontal rules, GFM tables, strikethrough, <think> strip, paragraph wrapping. Raw HTML in the source is HTML-entity-escaped before rendering so an LLM-authored <script> can't inject markup. Cascade removal ================ - `src/matrix.rs` deleted (3356 lines); `lib.rs` mod removed; `channels-allowlist.txt` entry removed; `Cargo.toml` features in both `librefang-channels` and `librefang-api` (incl. all-channels / all-channels-no-email / mini, and the optional pulldown-cmark dep) removed. - `MatrixConfig` struct + `ChannelsConfig.matrix` field + Default entry + 2 unit tests gone. - `channel_bridge.rs` import / spawn block / `find_channel_info!` / `check_channel!` / default-empty assertion removed. - `routes/channels.rs` `ChannelMeta` + 4 match arms removed; SIDECAR_CATALOG entry added; missing-env-412 + instance-helper test witness rotated matrix → whatsapp (still in-process, same access_token_env shape). - `routes/config.rs` `ch!(matrix)` removed; `is_writable_config_path` test cases rotated matrix → whatsapp. - `channel_sender.rs` `for_each_channel_field!` macro entry + `EXPECTED` name-list entry removed. - CLI list / picker / wizard arm removed. - TUI ChannelDef removed. - `librefang-types::config::validation` warn-loop for matrix removed. - `librefang-migrate::openclaw` records the legacy `[channels.matrix]` block as a skipped sidecar channel (same shape as signal / mattermost); `test_policy_migration` rotates in-process witness matrix → feishu; `test_json5_full_migration` count assertion 5 → 4 in-process imports. - `crates/librefang-api/tests/channels_routes_test.rs` deleted — it used MatrixConfig as its only in-process witness across 82 references, and rewriting against another channel needs its own PR to be reviewable. - Docs EN + ZH (6 files): `[channels.matrix]` blocks replaced with `[[sidecar_channels]]` redirects. New env-var knobs (`[sidecar_channels.env]`) ============================================= - `MATRIX_HOMESERVER_URL` (required) - `MATRIX_USER_ID` (required) - `MATRIX_ALLOWED_ROOMS` (optional, CSV) - `MATRIX_ACCOUNT_ID` (optional) - `MATRIX_MAX_UPLOAD_BYTES` (optional, default 50 MiB) Secret (`~/.librefang/secrets.env`) ==================================== - `MATRIX_ACCESS_TOKEN` (required) Operator action required ======================== An existing `[channels.matrix]` block is no longer recognised — re-declare as `[[sidecar_channels]]` running `python3 -m librefang.sidecar.adapters.matrix`. See `sdk/python/librefang/sidecar/adapters/matrix.py` header for the exact config. `ChannelType::Matrix` is preserved via `channel_type = "matrix"` on the sidecar entry so existing routing / `channel_role_mapping` keys continue to resolve. Verification ============ - `pytest sdk/python/tests/test_matrix_adapter.py -q` — **81 passed** in 0.12s (covers env enforcement, mxc_to_http, markdown→HTML rendering incl. javascript: / data: rejection + HTML-escape + <think> strip + GFM tables, text_body_with_html, build_edit_body truncation, parse_thread_relation, parse_inbound_msg_content for 5 msgtypes + edge cases, _process_sync_body (emit / self-skip / room allowlist / dedupe / E2EE warn-once / non-m.room.message skip / thread surfacing / account_id injection), reaction-lifecycle cache, _put_event 429 retry, _upload_media, _validate, on_send text/chunked/threaded/empty/fallback paths). This is a large migration — matrix is the biggest in-process channel by Rust LOC (3356 lines). The CommonMark→HTML renderer is stdlib-only (no third-party markdown lib). Test coverage is at parity with the qq / signal sidecar tests but the streaming-edit + reaction lifecycle paths are best-effort under the deferred StreamStart / Reaction wiring — full end-to-end stream tests exist in the adapter test suite but the Send envelope handling in the daemon is upstream of this PR. * fix(channels/matrix): full ChannelContent variant coverage + _streams race Self-review of #5368 found two regressions vs the in-process Rust adapter: 1. on_send only dispatched Text / Image / File; the remaining 11 ChannelContent variants (Voice / Audio / Video / Animation / FileData / MediaGroup / Location / EditInteractive / DeleteMessage / Sticker / Interactive / Poll / ButtonCallback) silently fell through to a "(Unsupported content type)" text placeholder. 2. Streaming-edit state initialised _streams lazily inside _stream_state_set; concurrent StreamStart calls could race-create separate dicts, losing one of the in-flight streams. Both are now fixed: - on_send dispatches every ChannelContent variant the Rust adapter did. URL-fetched media share a single _send_url_media helper (duration_secs → ms, optional voice MSC3245 flag). - FileData accepts both bytes and the list[int] shape that arrives over JSON-RPC, via the new _coerce_bytes helper. - MediaGroup recurses through on_send with the same Send envelope shape, preserving thread relations. - _streams now initialised in __init__. Test additions (23 cases) cover every new variant plus _coerce_bytes. Shared FakeResp.read now optionally honours a size argument and HdrShim.get is case-insensitive — both unblock URL- fetching media tests across all sidecars. Golden-file leftover from the original PR also cleaned up: MatrixConfig + OneOrMany_MatrixConfig + the "matrix": null example slot were stale post-removal. * fix(channels/matrix): declare header_rules for authenticated media fetch Rust adapter overrode ``ChannelAdapter::fetch_headers_for`` to return ``Authorization: Bearer <token>`` for URLs whose host matched the homeserver. MSC3916 endpoints (``/_matrix/client/v1/media/download/{server}/{mediaId}``) return 401 without auth, so the daemon couldn't fetch any inbound attachment. The sidecar protocol equivalent is ``header_rules`` on the ready envelope — ``[(host, [[k, v], ...]), ...]`` — and the bridge's ``fetch_headers_for`` exact-matches host. Set it in ``__init__`` once the access token is known. Host comes from ``urlparse(homeserver_url).hostname`` so a port in the URL is dropped, matching the bridge's parser. --------- Co-authored-by: Evan <[email protected]>
…e x2, LINE reply API) (#5462) * fix(channels/sidecars): cross-audit follow-ups (Retry-After x4, dedupe x2, LINE reply API) Cross-cutting consistency audit across the 27 freshly-migrated sidecar adapters (ntfy #5224 through google_chat #5459) caught seven gaps: 429 Retry-After missing (4 adapters): - slack: _post_message / _add_reaction / _remove_reaction routed 429 through the same status >= 300 arm as 5xx; chunk silently dropped. Slack's 3-tuple _http wrapper is now retry-aware and re-issues the call once via the shared parse_retry_after — callers unchanged. - feishu: _http_json (4-tuple, already exposed headers) gained retry_429=True once-shot before its code != 0 arm. - gotify: _publish chunk POST now sleeps Retry-After and retries once before surfacing the second 429 to the caller. - google_chat: _send_text 429 → retry-once with Retry-After (was raising immediately; only 401 was specially handled). Inbound dedupe missing (2 adapters): - gotify: WS reconnect can replay buffered frames. Now keyed on `gotify-<id>` through librefang.sidecar.common.SeenSet (same 10000/5000 policy as nextcloud/reddit/rocketchat/webex). - google_chat: webhook is at-least-once-delivery from Pub/Sub. Dedupe keyed on message.name in _make_webhook_handler before emit. LINE reply API path (capability fix): - Inbound replyToken was stashed only in metadata.reply_token which doesn't round-trip back to on_send, so every reply degraded to push (quota-charged, rate-limited) even inside the ~60 s window where the free reply API was available. Token + event-timestamp now carried through librefang_user (the field the bridge round- trips bytewise) as `linereply:<token>:<event_ts_ms>`. A LINE_REPLY_TOKEN_TTL_SECS = 55.0 freshness window plus a `linereply:` prefix guard decides between reply and push at send time — librefang_user is shared across channels (dingtalk stores a sessionWebhook URL, telegram stores an @username) and a misrouted value must not be fed to LINE's reply endpoint. Automatic push-fallback if LINE rejects the reply call (token burned between dispatch and our wakeup). Image+caption sends stay on push (reply token is one-shot-locked on first acceptance). Audited but not changed: - feishu already has Rust-parity dedup via _EventDedup (mirrors feishu.rs:122-125) used at _dispatch_event:1485 — false positive. - runtime.py:246 bare-except on on_command is OPEN PR #5450 territory; not duplicated here. - runtime.py:219 producer bare-except is same class but would conflict with #5450 in the same file — deferred. - mastodon SSE+poll dual-flow dedupe is theoretical (since_id watermark covers normal operation). - long-lived SSE timeouts are by design. Verification: - 17 new pytest cases across test_{slack,feishu,gotify,google_chat, line}_adapter.py - pytest tests/ — 1845 passed (was 1828; +17 regression guards) - cargo check --workspace --lib — clean - cargo clippy --workspace --all-targets -- -D warnings — clean * fix(channels/sidecars): address review feedback on cross-audit PR Four review-feedback fixes: 1. google_chat dedupe test now drives `do_POST` end-to-end. The previous test built `handler_cls = _make_webhook_handler(...)` but bypassed it and called `adapter._seen.mark()` directly — that only exercised `SeenSet` (already covered in test_common.py) and not the wiring inside `do_POST`. Replaced with a `_post_to_webhook` helper (same pattern feishu's tests use) that constructs the real handler against a fake socket and asserts the redelivery is suppressed at the wire level. Also added `test_webhook_handler_does_not_consume_dedupe_slot_for_filtered_events` to pin that filtered-out events (wrong space, non-MESSAGE type) don't burn a `_seen` slot — flooding the 10000-entry table with non-emittable events would evict legitimate IDs and re-open the at-least-once window. 2. LINE `_post_reply` now skips the 429 Retry-After sleep when the sleep would overshoot the reply-token deadline. Previously: 20s of token life left + Retry-After=30s → we'd sleep 30s, retry, get "Invalid reply token", THEN fall back to push. Now: we check `now_ms + wait*1000 >= deadline_ms` and return False immediately, letting push deliver the agent's reply ~30s sooner. `_parse_reply_carry` now returns `(token, deadline_ms)` so `on_send` can thread the deadline through to `_post_reply`. 3. LINE `_post_reply` >5-chunk path is now tested. LINE's reply endpoint caps at 5 messages per token. Pre-fix the `if len(messages) > 5: return False` short-circuit existed but had no regression guard — a future refactor could break it. Added `test_post_reply_short_circuits_when_text_exceeds_5_chunks` (uses an exploding urlopen to assert NO HTTP call is made) plus `test_on_send_falls_back_to_push_when_text_exceeds_5_chunks` (asserts a 6-chunk payload lands via push without burning the reply token on a 400). 4. LINE on_send now skips the reply path entirely when text is empty. LINE rejects empty messages on both reply and push; pre-fix we'd POST to reply (400) then push (400) — twice the waste over the existing push-empty behaviour. The guard is a simple `if is_text_path and text:` and the existing push path handles the empty case as it always has. Test: `test_on_send_text_skips_reply_on_empty_text`. Verification: pytest tests/ — 1851 passed (was 1845). All review feedback items addressed in-tree, no PR body change needed beyond the "6 adapters" → "4 adapters" wording fix that goes through `gh pr edit` separately.
#5464) * docs(config): fill in [[sidecar_channels]] samples for all 27 sidecar adapters `librefang.toml.example` and `init_default_config.toml` only sampled 4 sidecars (telegram / discord / slack / wechat) — the other 23 have existed since the channel-migration project (#5224 → #5459) but were never added to either config template. Operators running `librefang init` or copy-pasting from the example file had no in-tree guidance for bluesky / dingtalk / email / feishu / google_chat / gotify / line / mastodon / matrix / mattermost / nextcloud / ntfy / qq / reddit / rocketchat / signal / teams / twitch / webex / webhook / wecom / whatsapp / zulip. Each adapter now has a commented `[[sidecar_channels]]` block listing its required env vars (extracted from the adapter's own SCHEMA via `python3 -m librefang.sidecar.adapters.<name> --describe`, so the sample stays in lockstep with the actual adapter contract). Blocks include up to 2 commonly-tuned optional env vars as inline hints; the full inventory is one --describe invocation away. Sanitisations applied during generation: - Secret-type values render as `"..."` — the dashboard SCHEMA hint often contains free-text prose that doesn't belong in a TOML literal. Operators put the real secret in `~/.librefang/secrets.env`, not the sample. - Text placeholders with `"` are escaped. - Description sentences split on `. ` (period + space) so "Rocket.Chat" doesn't truncate to "Rocket". Also drops the stale `[channels.whatsapp]` block from librefang.toml.example — whatsapp is sidecar-only since the migration and the in-process `phone_number_id_env` field no longer exists. Generated content (336 lines) is byte-deterministic from the adapter SCHEMAs — easy to regenerate if a new sidecar lands or an existing one rotates its env vars. The shared "Sidecar channel adapters (out-of-process, any language)" section that documents the protocol meta-fields (restart, backoff, message_buffer) is preserved — it's orthogonal to per-adapter env vars and applies to all 27 sidecars uniformly. Verification: - `python3 -c "import tomllib; tomllib.load(open(p, 'rb'))"` on both sample files — clean (the new content is all `#` comments, parser ignores them). - `cargo xtask schema-check gen && cargo xtask schema-check check` — config.sha256 baseline regenerated and matches. - No Rust source changes; `init_default_config.toml` is `include_str!`'d into `librefang-cli` so the binary picks up the new template on next build automatically. * docs(config): address review feedback on sidecar-sample fill-in Five review-feedback fixes layered on top of the previous commit: 1. Generator script committed to `scripts/gen_sidecar_samples.py` so future re-runs (new sidecar lands, SCHEMA rotates) don't have to reconstruct the logic from the PR body. Made executable. The script's docstring documents the sanitisation rules and re-run workflow inline. 2. Secret-field SCHEMA hints preserved as trailing comments. `secret`-type values still render as `"..."` (TOML cleanliness) but the SCHEMA placeholder, when meaningful, now ships as a trailing `# <hint>`. Example: # TWITCH_OAUTH_TOKEN = "..." # oauth:abc123… (prefix auto-added) # WHATSAPP_APP_SECRET = "..." # optional; (production should always set this) Operators no longer lose the semantic guidance for the most security-sensitive fields when the sample collapses the literal. 3. Migration-warning text restored for all 27 adapters that replaced an in-process channel (every sidecar in this project — list with PR numbers in MIGRATED_FROM_IN_PROCESS). Each block now carries: # Migrated from in-process to sidecar in #<NNN>. Old # `[channels.<name>]` blocks are no longer recognised. so an operator upgrading a pre-migration config gets the explicit "your old block won't parse" signal at the matching adapter, not just buried in CHANGELOG. Fixes the regression where the 4 pre-existing blocks' original migration notes were silently lost. 4. Two-path adapters (whatsapp / wechat) gain a "Requires EITHER / OR" hint above their env table — their SCHEMAs mark every var optional because there are two valid configuration paths (Cloud API vs Baileys for whatsapp; pre-supplied token vs QR-login for wechat) but the adapter still SystemExit(2)s with zero config. The hint tells operators which path to pick. Similarly ntfy carries a privacy note that `NTFY_SERVER_URL` defaults to the public ntfy.sh server unless overridden. 5. "(out-of-process sidecar)" suffix stripped from per-block headers. The section header already documents this for all 27 entries; the per-block repetition was noise. 6. Reordering in `librefang.toml.example`: the generic "Sidecar channel adapters (out-of-process, any language)" section that documents protocol-level meta-fields (`restart`, `restart_initial_backoff_ms`, `message_buffer`, `overflow`, …) now sits ABOVE the 27 per-adapter blocks. Operators read the protocol context first, then the specific instances — matches the natural information-flow order an operator scanning the file expects. 7. CHANGELOG entry added under `[Unreleased] > Changed`. Documents the sample fill-in for operators who track release notes. 8. Also caught + added missing dingtalk migration PR (#5417) to MIGRATED_FROM_IN_PROCESS; previous revision had 26/27. Verification: - `tomllib.load()` on both files — clean - `cargo xtask schema-check gen && cargo xtask schema-check check` — baseline regenerated, matches - 27/27 migration notes present in generated output - Script is reproducible and idempotent * docs(config): surface `pip install librefang-sdk` prerequisite in sidecar samples Operators following the sample config to set up a sidecar channel hit a confusing failure mode at daemon boot when `librefang-sdk` isn't installed in the interpreter the daemon picked: WARN sidecar --describe failed; discovery card will have no form fields adapter="telegram" error=describe exited 1: ... ModuleNotFoundError: No module named 'librefang' The error repeats once per adapter in SIDECAR_CATALOG (27× on a clean install) because daemon boot calls `--describe` on every known adapter to populate the dashboard's Add-Channel form. The sample's existing header documents secrets handling and SCHEMA introspection but never told operators that `python3 -m librefang.sidecar.adapters.<name>` requires the SDK to be importable from the daemon's interpreter. Now it does, prominently — with three details that prior reports surfaced as common confusion: 1. `pip install librefang-sdk` (PyPI package name) OR `pip install -e sdk/python/` from a source checkout. 2. Multi-interpreter footgun: mise / pyenv / conda commonly pick a different `python3` for the daemon than for the operator's shell. The sample now says so and gives a one-line verification command. 3. Verification snippet: `python3 -c 'import librefang.sidecar; print(librefang.__file__)'` from the same shell that starts librefang. Also tags the regeneration command (`scripts/gen_sidecar_samples.py`) as "for librefang maintainers only — operators can ignore this" so an operator who copies the sample file as their config and tries to follow every command in the header doesn't waste time looking for a librefang source checkout. This is a docs-only change. Both sample files re-validated as clean TOML; `cargo xtask schema-check gen` refreshes the config baseline to match. The companion runtime + discovery rewrite (translating the daemon's `ModuleNotFoundError` into the SAME actionable install hint operators see in the sample header) ships as PR #5465.
Sidecar-first POC — the keystone of the #5219/#5220/#5221 series:
proves the full pipeline by replacing a real in-process channel with
an out-of-process sidecar and deleting the Rust adapter.
[channels.ntfy]is no longer a recognised config block. Operatorsre-declare it as a
[[sidecar_channels]]running the newexamples/sidecar-channel-python/ntfy_adapter.py(its header has theexact config). The separate ntfy push-notification provider
(
push_provider = "ntfy"/ntfy_url/ntfy_topic, used by devicepairing) is a different feature and is deliberately untouched.
Replacement (behaviour-preserving)
examples/sidecar-channel-python/ntfy_adapter.py— stdlib-only, on thelibrefang.sidecarSDK (P4). Same SSE parsing,/commandhandling,title→sender,topicmetadata, 4096-char chunked plain-textpublish, optional Bearer auth, exponential-backoff reconnect.
Scope expanded vs. the phase plan (approved)
The plan listed ~6 deletion sites; the true blast radius of removing
the
NtfyConfigtype is much wider and kernel-touching. The full-removal path (breaking) was explicitly chosen. Removed:
librefang-channels(src/ntfy.rs, lib.rs, Cargochannel-ntfy,allowlist entry),
librefang-types(NtfyConfig+channels.ntfy+validation hook),
librefang-api(channel-ntfyfeature incl.core/all-channels membership; channel_bridge use+block+macros+test;
routes/channels.rs descriptor + 4 arms; routes/config.rs
ch!),librefang-kernel(channel_sender registry),librefang-cli(TUIdef + comment), kernel-config golden regenerated. Removing
ntfyfrom
channels-allowlist.txtmakes the P5 gate(
cargo xtask channel-policy) permanently reject any in-process ntfyreintroduction — the migration ratchet working as designed.
Verification
cargo check --workspace --lib: cleancargo clippy --workspace --all-targets -- -D warnings: cleancargo test -p librefang-channels(847) /-p librefang-types/-p xtask(channel-policy gate green with ntfy delisted —real_repo_tree_is_greenok): passcd sdk/python && pytest: 22 passed incl.tests/test_ntfy_adapter.py(SSE parse, command vs text, sender,chunked publish + auth, unsupported-content fallback; urllib mocked)
cargo test -p librefang-api: all pass except 2 pre-existing,unrelated failures —
auth_public_allowlist.rs'sdashboard_read_paths_are_in_catalog/dashboard_read_routes_reachable_without_auth_when_no_key_configuredfail on
/api/cron/list(a cron-route allowlist drift alreadyon
main). Evidence it is not from this PR: the offending path is acron route; this PR touches no auth middleware, route registration,
PUBLIC_ROUTES_*, or cron code. Left for a separate, cron/auth-domain fix rather than bundled here.