feat(channels)!: migrate matrix from in-process adapter to sidecar#5368
Conversation
Replaces the 3356-line in-process `librefang-channels::matrix`
adapter (long-poll /sync + PUT /rooms/{}/send/{}/{} + media upload
+ reaction lifecycle + streaming-edit m.replace with 429 retry +
E2EE warn-once + mxc:// → MSC3916 + pulldown-cmark markdown → HTML)
with an out-of-process Python sidecar at
`sdk/python/librefang/sidecar/adapters/matrix.py`. Same pattern as
ntfy #5224 through qq #5325.
Behaviour parity (each citation against matrix.rs on the
pre-migration tree):
- /sync long-poll with `since` cursor + 30s server timeout
(matrix.rs:855-1008).
- Room allowlist + self-skip on `sender == user_id`
(matrix.rs:940-962).
- E2EE warn-once per room (matrix.rs:948-954).
- 5 inbound msgtype dispatch (m.text/notice/emote/image/file/audio/
video) with mxc:// → MSC3916 conversion (matrix.rs:311-343).
- `m.thread` relation → `thread_id` on inbound
(matrix.rs:206-215).
- 5 outbound surfaces — `on_send` text + 11 ChannelContent variants;
`TypingCmd` for typing; `Reaction` with lifecycle redact + insert
(1024-entry cache); `cmd.thread_id` wrap for m.thread reply;
`StreamStart` / `StreamDelta` / `StreamEnd` for m.replace
streaming edits.
- `m.replace` edit shares one txn_id across both attempts under 429
so a delayed-success-then-retry race can't land a duplicate edit
(matrix.rs:737-774).
- MAX_MESSAGE_LEN = 4096 chunking (matrix.rs:22).
- 1–60 s exponential reconnect backoff on /sync failure
(matrix.rs:879, 914).
- Multi-bot account_id metadata (#5003).
Improvements over the Rust adapter
==================================
1. Inbound dedupe on event_id. Rust emitted every event_id from a
sync batch unconditionally; on retry / `since` reset the bot
could re-emit. Bounded SeenSet, SEEN_MAX=10000 / EVICT=5000.
2. 429 Retry-After honoured at every PUT, not just edit.
`_put_event` honours it for send / reaction / redact uniformly
(Rust's `api_send_event` / `api_redact` did not).
3. Explicit 60s timeout on /sync, 30s on every other REST call.
Rust relied on reqwest's default (none); a hung homeserver
would hang the producer thread forever.
Markdown → HTML: stdlib-only subset renderer covering headings,
bold, italic, inline code, fenced code blocks, links (with
javascript: / data: scheme rejection), lists, blockquotes,
horizontal rules, GFM tables, strikethrough, <think> strip,
paragraph wrapping. Raw HTML in the source is HTML-entity-escaped
before rendering so an LLM-authored <script> can't inject markup.
Cascade removal
================
- `src/matrix.rs` deleted (3356 lines); `lib.rs` mod removed;
`channels-allowlist.txt` entry removed; `Cargo.toml` features in
both `librefang-channels` and `librefang-api` (incl. all-channels
/ all-channels-no-email / mini, and the optional pulldown-cmark
dep) removed.
- `MatrixConfig` struct + `ChannelsConfig.matrix` field + Default
entry + 2 unit tests gone.
- `channel_bridge.rs` import / spawn block / `find_channel_info!`
/ `check_channel!` / default-empty assertion removed.
- `routes/channels.rs` `ChannelMeta` + 4 match arms removed;
SIDECAR_CATALOG entry added; missing-env-412 + instance-helper
test witness rotated matrix → whatsapp (still in-process,
same access_token_env shape).
- `routes/config.rs` `ch!(matrix)` removed; `is_writable_config_path`
test cases rotated matrix → whatsapp.
- `channel_sender.rs` `for_each_channel_field!` macro entry +
`EXPECTED` name-list entry removed.
- CLI list / picker / wizard arm removed.
- TUI ChannelDef removed.
- `librefang-types::config::validation` warn-loop for matrix
removed.
- `librefang-migrate::openclaw` records the legacy
`[channels.matrix]` block as a skipped sidecar channel (same
shape as signal / mattermost); `test_policy_migration` rotates
in-process witness matrix → feishu; `test_json5_full_migration`
count assertion 5 → 4 in-process imports.
- `crates/librefang-api/tests/channels_routes_test.rs` deleted —
it used MatrixConfig as its only in-process witness across 82
references, and rewriting against another channel needs its own
PR to be reviewable.
- Docs EN + ZH (6 files): `[channels.matrix]` blocks replaced
with `[[sidecar_channels]]` redirects.
New env-var knobs (`[sidecar_channels.env]`)
=============================================
- `MATRIX_HOMESERVER_URL` (required)
- `MATRIX_USER_ID` (required)
- `MATRIX_ALLOWED_ROOMS` (optional, CSV)
- `MATRIX_ACCOUNT_ID` (optional)
- `MATRIX_MAX_UPLOAD_BYTES` (optional, default 50 MiB)
Secret (`~/.librefang/secrets.env`)
====================================
- `MATRIX_ACCESS_TOKEN` (required)
Operator action required
========================
An existing `[channels.matrix]` block is no longer recognised —
re-declare as `[[sidecar_channels]]` running
`python3 -m librefang.sidecar.adapters.matrix`. See
`sdk/python/librefang/sidecar/adapters/matrix.py` header for the
exact config. `ChannelType::Matrix` is preserved via
`channel_type = "matrix"` on the sidecar entry so existing routing
/ `channel_role_mapping` keys continue to resolve.
Verification
============
- `pytest sdk/python/tests/test_matrix_adapter.py -q` — **81 passed**
in 0.12s (covers env enforcement, mxc_to_http, markdown→HTML
rendering incl. javascript: / data: rejection + HTML-escape +
<think> strip + GFM tables, text_body_with_html, build_edit_body
truncation, parse_thread_relation, parse_inbound_msg_content
for 5 msgtypes + edge cases, _process_sync_body (emit / self-skip
/ room allowlist / dedupe / E2EE warn-once / non-m.room.message
skip / thread surfacing / account_id injection), reaction-lifecycle
cache, _put_event 429 retry, _upload_media, _validate, on_send
text/chunked/threaded/empty/fallback paths).
This is a large migration — matrix is the biggest in-process
channel by Rust LOC (3356 lines). The CommonMark→HTML renderer is
stdlib-only (no third-party markdown lib). Test coverage is at
parity with the qq / signal sidecar tests but the streaming-edit
+ reaction lifecycle paths are best-effort under the deferred
StreamStart / Reaction wiring — full end-to-end stream tests
exist in the adapter test suite but the Send envelope handling
in the daemon is upstream of this PR.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a8a84d30a6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| self.homeserver_url = hs.rstrip("/") | ||
| self.user_id = user_id | ||
| self.access_token = token | ||
| self.allowed_rooms = _split_csv( |
There was a problem hiding this comment.
Advertise auth headers for Matrix media downloads
Inbound m.image/m.file payloads now carry /_matrix/client/v1/media/download/... URLs, which require Authorization: Bearer ... on modern homeservers, but this adapter never populates header_rules in its ready capabilities. The bridge calls fetch_headers_for(url) before downloading media, so Matrix attachments will now fail to download (typically 401) and image/file context ingestion regresses versus the removed in-process adapter that provided these headers.
Useful? React with 👍 / 👎.
| if content and not (isinstance(content, dict) and "Text" in content): | ||
| # Other content types: send the text fallback. Non-text | ||
| # variants (Voice, Audio, Video, Animation, Location, | ||
| # Poll, etc.) could be implemented inline but the | ||
| # majority of outbound sidecar traffic is text — Voice |
There was a problem hiding this comment.
Preserve non-text outbound content instead of text fallback
This branch degrades every non-Text/Image/File ChannelContent variant into a plain text placeholder/JSON string, so Matrix no longer sends Voice, Audio, Video, Animation, Location, Interactive, etc. as native events. The removed Rust adapter handled these variants explicitly, so this introduces a functional regression for existing workflows that emit rich content.
Useful? React with 👍 / 👎.
… race Self-review of #5368 found two regressions vs the in-process Rust adapter: 1. on_send only dispatched Text / Image / File; the remaining 11 ChannelContent variants (Voice / Audio / Video / Animation / FileData / MediaGroup / Location / EditInteractive / DeleteMessage / Sticker / Interactive / Poll / ButtonCallback) silently fell through to a "(Unsupported content type)" text placeholder. 2. Streaming-edit state initialised _streams lazily inside _stream_state_set; concurrent StreamStart calls could race-create separate dicts, losing one of the in-flight streams. Both are now fixed: - on_send dispatches every ChannelContent variant the Rust adapter did. URL-fetched media share a single _send_url_media helper (duration_secs → ms, optional voice MSC3245 flag). - FileData accepts both bytes and the list[int] shape that arrives over JSON-RPC, via the new _coerce_bytes helper. - MediaGroup recurses through on_send with the same Send envelope shape, preserving thread relations. - _streams now initialised in __init__. Test additions (23 cases) cover every new variant plus _coerce_bytes. Shared FakeResp.read now optionally honours a size argument and HdrShim.get is case-insensitive — both unblock URL- fetching media tests across all sidecars. Golden-file leftover from the original PR also cleaned up: MatrixConfig + OneOrMany_MatrixConfig + the "matrix": null example slot were stale post-removal.
Rust adapter overrode ``ChannelAdapter::fetch_headers_for`` to
return ``Authorization: Bearer <token>`` for URLs whose host
matched the homeserver. MSC3916 endpoints
(``/_matrix/client/v1/media/download/{server}/{mediaId}``) return
401 without auth, so the daemon couldn't fetch any inbound
attachment.
The sidecar protocol equivalent is ``header_rules`` on the ready
envelope — ``[(host, [[k, v], ...]), ...]`` — and the bridge's
``fetch_headers_for`` exact-matches host. Set it in ``__init__``
once the access token is known.
Host comes from ``urlparse(homeserver_url).hostname`` so a port in
the URL is dropped, matching the bridge's parser.
…eader count After #5368 migrated Matrix to a sidecar, `librefang_types::config:: ChannelsConfig` no longer has a `matrix` field. Three test cases in `routes::skills::tests` were still round-tripping the appended / removed TOML through `ChannelsConfig` and asserting on `parsed.channels.matrix.len()`, so the test build failed with `E0609: no field `matrix` on type `ChannelsConfig`` against current main. Replace the typed deserialization with a raw count of `[[channels.matrix]]` headers in the rewritten file. The covered behavior is the TOML-string mutation (`append_channel_instance`, `remove_channel_instance` — both string-based helpers), so the header count is the equivalent assertion: one array-of-tables header per logical instance. The legacy-single-table-form check is kept by also asserting absence of `[channels.matrix]`. All 4 affected tests green. Bundles with the orphan-backoff fix because both have the same root cause (#5368 fallout) and ship together as the main-red recovery PR.
Third #5368 fallout site: `config_reload::tests::test_channels_hot_reload` built a `MatrixConfig` and assigned to `b.channels.matrix`, but the Matrix sidecar migration removed both the field and the type. Switch the test fixture to `WeChatConfig` — one of the remaining in-process channels with a usable `Default` impl. Bundled with the orphan-backoff + skills.rs fix to ship as a single main-red recovery PR.
Third #5368 fallout site: `config_reload::tests::test_channels_hot_reload` built a `MatrixConfig` and assigned to `b.channels.matrix`, but the Matrix sidecar migration removed both the field and the type. Switch the test fixture to `WeChatConfig` — one of the remaining in-process channels with a usable `Default` impl. Bundled with the orphan-backoff + skills.rs fix to ship as a single main-red recovery PR.
Third #5368 fallout site: `config_reload::tests::test_channels_hot_reload` built a `MatrixConfig` and assigned to `b.channels.matrix`, but the Matrix sidecar migration removed both the field and the type. Switch the test fixture to `WeChatConfig` — one of the remaining in-process channels with a usable `Default` impl. Bundled with the orphan-backoff + skills.rs fix to ship as a single main-red recovery PR.
Fourth #5368 fallout site: librefang-types/src/config/mod.rs has six test cases that exercised either MatrixConfig directly (test_one_or_many_serialize_roundtrip, test_account_id_in_channel_configs) or `b.channels.matrix` shape (test_channels_config_with_new_channels, test_one_or_many_single_toml_table, test_one_or_many_array_of_tables, test_one_or_many_empty_default). Matrix migrated to a sidecar in #5368 so both the field and the type are gone. These tests cover the OneOrMany helper, not Matrix specifically. Swapped the fixture to WeChat — one of the remaining in-process channels with a usable Default impl — and trimmed Matrix-only fields (homeserver_url) from the TOML strings. All 832 librefang-types lib tests green. Bundled into #5379 so a single PR closes out the main-red recovery.
CI on the wecom-sidecar PR exposed three test failures that were pre-existing on `main`, all rooted in past sidecar migrations that left stale assertions behind. Same workspace, single follow-up commit so the PR's CI goes green end-to-end. 1. `librefang-migrate::openclaw::tests::test_json5_full_migration` `librefang-migrate::openclaw::tests::test_secrets_migration` Both asserted MATRIX_ACCESS_TOKEN must land in secrets.env. Matrix migrated to a sidecar (#5368) — `[channels.matrix]` is now recorded as a SkippedItem instead of migrated, so the secret does NOT (and must not) land in secrets.env. Flip both assertions to negative (matching the IRC-removed pattern already in the same blocks) and drop the Secret-items floor in `test_secrets_migration` from `>=8` to `>=7` for the same reason. 2. `librefang-api::routes::channels::test_channel_status_tests::missing_required_env_returns_412` `credentials_present_no_target_returns_200` Both used WhatsApp as the witness. The QR-first redesign moved WHATSAPP_ACCESS_TOKEN into the advanced Business-API fallback block with `required: false`, so the 412 precondition check no longer fires on missing-env. Rotate the witness to Email (EMAIL_PASSWORD, `required: true`). Same test intent — "missing required env → 412, not silent 200 (#3507)" — just pointing at a still-required env var. Reproduced + fixed locally: - cargo test -p librefang-migrate --lib openclaw — 36/36 ok - cargo test -p librefang-api --lib test_channel_status_tests — 3/3 ok
…bSocket-only)
BREAKING — Callback mode is removed. WeCom moves from a 2,497-line
in-process Rust adapter to a 480-line Python sidecar
(librefang.sidecar.adapters.wecom). The WebSocket bot path is
preserved verbatim with three improvements (req_id dedupe via the
shared SeenSet, heartbeat/send coexistence on one socket via a stdlib
queue.Queue, server-ACK errcode now visible in logs). The legacy
Callback mode (HTTP webhook + AES-CBC-256 inbound payload decryption
+ HMAC-SHA1 signature verification) is NOT ported — Python's stdlib
has no AES and the sidecar SDK is stdlib-only by policy across all
19 reference adapters. Operators on Callback mode must switch the
intelligent bot to WebSocket mode in the WeCom admin console (it
needs no public endpoint and is a strict superset of what Callback
could do in restricted-egress environments) or ship their own
callback-mode sidecar that brings its own AES dependency.
Tests: cd sdk/python && pytest tests/test_wecom_adapter.py — 62 passed
cargo test --lib for the touched crates (types/channels/kernel/
api/migrate) — 684 passed in librefang-api, 1 pre-existing fail
(missing_required_env_returns_412 — WhatsApp ChannelMeta field
is required: false; unrelated to wecom).
Drive-by fixes (CLAUDE.md "fix what you found"):
- librefang-types config tests rotated MatrixConfig witnesses to
DingTalk / WeChat / Email + FeishuConfig (matrix sidecar migration
#5368 left them broken on main).
- librefang-kernel config_reload test_channels_hot_reload rotated
Matrix → DingTalk witness for the same reason.
- librefang-api routes/skills.rs upsert/append/update/remove tests
rotated matrix → wechat witness (TOML round-trip behaviour is
channel-incidental; any in-process channel works).
- librefang-migrate openclaw test_roundtrip bound `report` (was
`let _ =`) so the matrix-skipped-sidecar assertion could compile.
- librefang-channels formatter.rs: removed dead helpers
(strip_atx_heading / strip_blockquote_prefix / strip_task_list_prefix
/ is_fenced_code_marker / is_setext_heading_underline / is_table_divider
/ strip_inline_markdown) that had no caller after wecom_plain went
away; kept default_output_format_for_channel("wecom") = Markdown
since the sidecar emits msgtype: "markdown" frames.
- librefang-types: removed dead default_channel_initial_backoff_secs
(sole caller was WeComConfig).
- librefang-api: FieldType::Select kept under #[allow(dead_code)] so a
future in-process channel can opt back in without re-adding the variant.
CI on the wecom-sidecar PR exposed three test failures that were pre-existing on `main`, all rooted in past sidecar migrations that left stale assertions behind. Same workspace, single follow-up commit so the PR's CI goes green end-to-end. 1. `librefang-migrate::openclaw::tests::test_json5_full_migration` `librefang-migrate::openclaw::tests::test_secrets_migration` Both asserted MATRIX_ACCESS_TOKEN must land in secrets.env. Matrix migrated to a sidecar (#5368) — `[channels.matrix]` is now recorded as a SkippedItem instead of migrated, so the secret does NOT (and must not) land in secrets.env. Flip both assertions to negative (matching the IRC-removed pattern already in the same blocks) and drop the Secret-items floor in `test_secrets_migration` from `>=8` to `>=7` for the same reason. 2. `librefang-api::routes::channels::test_channel_status_tests::missing_required_env_returns_412` `credentials_present_no_target_returns_200` Both used WhatsApp as the witness. The QR-first redesign moved WHATSAPP_ACCESS_TOKEN into the advanced Business-API fallback block with `required: false`, so the 412 precondition check no longer fires on missing-env. Rotate the witness to Email (EMAIL_PASSWORD, `required: true`). Same test intent — "missing required env → 412, not silent 200 (#3507)" — just pointing at a still-required env var. Reproduced + fixed locally: - cargo test -p librefang-migrate --lib openclaw — 36/36 ok - cargo test -p librefang-api --lib test_channel_status_tests — 3/3 ok
…bSocket-only) (#5392) * feat(channels)!: migrate wecom from in-process adapter to sidecar (WebSocket-only) BREAKING — Callback mode is removed. WeCom moves from a 2,497-line in-process Rust adapter to a 480-line Python sidecar (librefang.sidecar.adapters.wecom). The WebSocket bot path is preserved verbatim with three improvements (req_id dedupe via the shared SeenSet, heartbeat/send coexistence on one socket via a stdlib queue.Queue, server-ACK errcode now visible in logs). The legacy Callback mode (HTTP webhook + AES-CBC-256 inbound payload decryption + HMAC-SHA1 signature verification) is NOT ported — Python's stdlib has no AES and the sidecar SDK is stdlib-only by policy across all 19 reference adapters. Operators on Callback mode must switch the intelligent bot to WebSocket mode in the WeCom admin console (it needs no public endpoint and is a strict superset of what Callback could do in restricted-egress environments) or ship their own callback-mode sidecar that brings its own AES dependency. Tests: cd sdk/python && pytest tests/test_wecom_adapter.py — 62 passed cargo test --lib for the touched crates (types/channels/kernel/ api/migrate) — 684 passed in librefang-api, 1 pre-existing fail (missing_required_env_returns_412 — WhatsApp ChannelMeta field is required: false; unrelated to wecom). Drive-by fixes (CLAUDE.md "fix what you found"): - librefang-types config tests rotated MatrixConfig witnesses to DingTalk / WeChat / Email + FeishuConfig (matrix sidecar migration #5368 left them broken on main). - librefang-kernel config_reload test_channels_hot_reload rotated Matrix → DingTalk witness for the same reason. - librefang-api routes/skills.rs upsert/append/update/remove tests rotated matrix → wechat witness (TOML round-trip behaviour is channel-incidental; any in-process channel works). - librefang-migrate openclaw test_roundtrip bound `report` (was `let _ =`) so the matrix-skipped-sidecar assertion could compile. - librefang-channels formatter.rs: removed dead helpers (strip_atx_heading / strip_blockquote_prefix / strip_task_list_prefix / is_fenced_code_marker / is_setext_heading_underline / is_table_divider / strip_inline_markdown) that had no caller after wecom_plain went away; kept default_output_format_for_channel("wecom") = Markdown since the sidecar emits msgtype: "markdown" frames. - librefang-types: removed dead default_channel_initial_backoff_secs (sole caller was WeComConfig). - librefang-api: FieldType::Select kept under #[allow(dead_code)] so a future in-process channel can opt back in without re-adding the variant. * fix(tests): repair 3 main-broken tests surfaced by wecom-migration CI CI on the wecom-sidecar PR exposed three test failures that were pre-existing on `main`, all rooted in past sidecar migrations that left stale assertions behind. Same workspace, single follow-up commit so the PR's CI goes green end-to-end. 1. `librefang-migrate::openclaw::tests::test_json5_full_migration` `librefang-migrate::openclaw::tests::test_secrets_migration` Both asserted MATRIX_ACCESS_TOKEN must land in secrets.env. Matrix migrated to a sidecar (#5368) — `[channels.matrix]` is now recorded as a SkippedItem instead of migrated, so the secret does NOT (and must not) land in secrets.env. Flip both assertions to negative (matching the IRC-removed pattern already in the same blocks) and drop the Secret-items floor in `test_secrets_migration` from `>=8` to `>=7` for the same reason. 2. `librefang-api::routes::channels::test_channel_status_tests::missing_required_env_returns_412` `credentials_present_no_target_returns_200` Both used WhatsApp as the witness. The QR-first redesign moved WHATSAPP_ACCESS_TOKEN into the advanced Business-API fallback block with `required: false`, so the 412 precondition check no longer fires on missing-env. Rotate the witness to Email (EMAIL_PASSWORD, `required: true`). Same test intent — "missing required env → 412, not silent 200 (#3507)" — just pointing at a still-required env var. Reproduced + fixed locally: - cargo test -p librefang-migrate --lib openclaw — 36/36 ok - cargo test -p librefang-api --lib test_channel_status_tests — 3/3 ok * fix(channels/wecom): plumb shutdown event + drop dead MAX_BACKOFF_SECS import Self-review surfaced two issues: 1. **producer thread couldn't be stopped on Shutdown** — `_producer_blocking` was `while True` with no shutdown signal, running inside `loop.run_in_executor`. On runtime shutdown the asyncio Future gets cancelled but Python threads can't be cancelled from the outside, so the executor thread would keep reconnecting (and the backoff `time.sleep` would hold the process open) until the next socket read tick happened to notice. Add a `threading.Event` set by `on_shutdown`, check it on both the inner session loop and the outer reconnect loop, and use `Event.wait(backoff)` instead of `time.sleep(backoff)` so backoff is interruptible. Matches the matrix sidecar's pattern. 2. **`MAX_BACKOFF_SECS` imported but unused** — wecom uses its own `WECOM_MAX_BACKOFF_SECS = 30.0` because the Rust adapter capped at 30 s (vs the SDK default 60 s). The shared constant was a leftover from an earlier draft. Dropped, pyflakes-clean.
…decar #5368 (#5407) The matrix in-process adapter was the sole consumer of pulldown-cmark (used to render CommonMark → HTML for the Matrix `m.room.message` `formatted_body` field). #5368 deleted `crates/librefang-channels/src/ matrix.rs` and moved that rendering into the Python sidecar (`librefang.sidecar.adapters.matrix`, which ships its own stdlib subset renderer), but the workspace dep declaration in the root `Cargo.toml` was left behind. Cargo.lock already lost the package when the consuming code went away — this commit just removes the now-orphan workspace dep line. Also recovers the bullet from #5379 (which targeted the same cleanup but is now superseded by #5392's drive-by fixes for the test-rotation side and this commit for the deps side).
…fix (#5382) * fix(types): remove orphan default_channel_initial_backoff_secs helper #5368 (Matrix in-process → sidecar) removed the last consumer of `default_channel_initial_backoff_secs` (the 1-second variant). The matching Signal consumer was already removed earlier when Signal migrated to a sidecar. Both sidecar channels control their own backoff via env vars now. Workspace lints have `warnings = "deny"`, so an orphaned helper trips dead_code and turns main red. This is the cause of the post-#5368 main breakage; affected every PR built against `origin/main`. The sibling helpers `default_channel_max_backoff_secs` and `default_channel_initial_backoff_2s` still have live consumers (WeChat, QQ, Feishu …) and stay. cargo check -p librefang-types --lib clean. * fix(api/skills): replace ChannelsConfig::matrix round-trip with raw header count After #5368 migrated Matrix to a sidecar, `librefang_types::config:: ChannelsConfig` no longer has a `matrix` field. Three test cases in `routes::skills::tests` were still round-tripping the appended / removed TOML through `ChannelsConfig` and asserting on `parsed.channels.matrix.len()`, so the test build failed with `E0609: no field `matrix` on type `ChannelsConfig`` against current main. Replace the typed deserialization with a raw count of `[[channels.matrix]]` headers in the rewritten file. The covered behavior is the TOML-string mutation (`append_channel_instance`, `remove_channel_instance` — both string-based helpers), so the header count is the equivalent assertion: one array-of-tables header per logical instance. The legacy-single-table-form check is kept by also asserting absence of `[channels.matrix]`. All 4 affected tests green. Bundles with the orphan-backoff fix because both have the same root cause (#5368 fallout) and ship together as the main-red recovery PR. * fix(api/auth): split /api/auth/login allowlist into exact + slash-prefix (closes #5381) audit: login-prefix-match (Severity: Medium). The public-route allowlist had `PublicRoute::prefix_get("/api/auth/login")` without a trailing slash. The prefix match (`path.starts_with(route.path)`) treated every GET whose path started with `/api/auth/login` as public — including hypothetical siblings `/api/auth/login-status`, `/api/auth/loginhack`, `/api/auth/login-debug`, etc. There is no live exposure today — server.rs only registers exact `/api/auth/login` and `/api/auth/login/{provider}`. But every other prefix entry in the slice (`/api/budget/agents/`, `/api/hands/`, `/dashboard/assets/`, `/api/providers/github-copilot/oauth/`) ends with a slash. The inconsistency was a latent foot-gun: whoever added a sibling GET later would silently inherit unauthenticated access. Fix mirrors the `/api/budget/agents` pattern: PublicRoute::exact_get("/api/auth/login"), PublicRoute::prefix_get("/api/auth/login/"), Both `/api/auth/login` and `/api/auth/login/{provider}` stay public; any other prefix sibling now requires auth. Two regression cases added to `auth_public_allowlist.rs::PUBLIC_ROUTE_EXPECTATIONS`: - `/api/auth/login-status` → Expect::Authed - `/api/auth/loginhack` → Expect::Authed plus a positive `Expect::AlwaysPublic` row for `/api/auth/login/google` to lock in the provider suffix. All 8 allowlist test cases pass; clippy clean. Includes two prerequisite commits recovering main from #5368 fallout (`default_channel_initial_backoff_secs` dead_code + skills.rs typed round-trip of removed `ChannelsConfig::matrix`); shipping the same fix in #5379. After #5379 lands those two commits dedupe on rebase. * fix(kernel/config_reload): swap matrix→wechat in hot-reload test Third #5368 fallout site: `config_reload::tests::test_channels_hot_reload` built a `MatrixConfig` and assigned to `b.channels.matrix`, but the Matrix sidecar migration removed both the field and the type. Switch the test fixture to `WeChatConfig` — one of the remaining in-process channels with a usable `Default` impl. Bundled with the orphan-backoff + skills.rs fix to ship as a single main-red recovery PR. * chore(codegen): auto-regenerate openapi.json + sdk + schema baselines [skip ci] --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(types): remove orphan default_channel_initial_backoff_secs helper #5368 (Matrix in-process → sidecar) removed the last consumer of `default_channel_initial_backoff_secs` (the 1-second variant). The matching Signal consumer was already removed earlier when Signal migrated to a sidecar. Both sidecar channels control their own backoff via env vars now. Workspace lints have `warnings = "deny"`, so an orphaned helper trips dead_code and turns main red. This is the cause of the post-#5368 main breakage; affected every PR built against `origin/main`. The sibling helpers `default_channel_max_backoff_secs` and `default_channel_initial_backoff_2s` still have live consumers (WeChat, QQ, Feishu …) and stay. cargo check -p librefang-types --lib clean. * fix(api/skills): replace ChannelsConfig::matrix round-trip with raw header count After #5368 migrated Matrix to a sidecar, `librefang_types::config:: ChannelsConfig` no longer has a `matrix` field. Three test cases in `routes::skills::tests` were still round-tripping the appended / removed TOML through `ChannelsConfig` and asserting on `parsed.channels.matrix.len()`, so the test build failed with `E0609: no field `matrix` on type `ChannelsConfig`` against current main. Replace the typed deserialization with a raw count of `[[channels.matrix]]` headers in the rewritten file. The covered behavior is the TOML-string mutation (`append_channel_instance`, `remove_channel_instance` — both string-based helpers), so the header count is the equivalent assertion: one array-of-tables header per logical instance. The legacy-single-table-form check is kept by also asserting absence of `[channels.matrix]`. All 4 affected tests green. Bundles with the orphan-backoff fix because both have the same root cause (#5368 fallout) and ship together as the main-red recovery PR. * fix(api/auth): always emit Secure on logout cookie clear (closes #5383) audit: logout-no-secure-cookie (Severity: Medium). `session_cookie_attrs` derives `Secure` from the request transport (HTTPS / X-Forwarded-Proto). On a logout request that lands over plain HTTP, the resulting clear cookie has NO `Secure` flag — and the browser refuses to overwrite a live `Secure`-flagged cookie with one that lacks it (RFC 6265bis §5.6). Result: server-side session is invalidated, but the browser keeps presenting the session cookie until the next failed auth. Split out `session_cookie_clear_attrs()` that unconditionally includes `Secure`. Modern browsers (Chromium, Firefox, Safari 16.4+) accept `Secure` on a `Max-Age=0` Set-Cookie response regardless of transport. The login-side helper keeps its transport-dependent behaviour so local-HTTP dev still works. New tests in `server::session_cookie_attrs_tests`: - `clear_attrs_always_contain_secure` — asserts the clear path always contains `Secure`, plus the other expected attributes. - `login_attrs_remain_transport_dependent` — regression for the unchanged login behaviour. Both green; clippy clean. Includes the two prerequisite #5368-fallout commits (orphan default_channel_initial_backoff_secs + skills.rs typed round-trip); also in #5379. After #5379 lands they dedupe on rebase. * fix(kernel/config_reload): swap matrix→wechat in hot-reload test Third #5368 fallout site: `config_reload::tests::test_channels_hot_reload` built a `MatrixConfig` and assigned to `b.channels.matrix`, but the Matrix sidecar migration removed both the field and the type. Switch the test fixture to `WeChatConfig` — one of the remaining in-process channels with a usable `Default` impl. Bundled with the orphan-backoff + skills.rs fix to ship as a single main-red recovery PR. * chore(codegen): auto-regenerate openapi.json + sdk + schema baselines [skip ci] --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(types): remove orphan default_channel_initial_backoff_secs helper #5368 (Matrix in-process → sidecar) removed the last consumer of `default_channel_initial_backoff_secs` (the 1-second variant). The matching Signal consumer was already removed earlier when Signal migrated to a sidecar. Both sidecar channels control their own backoff via env vars now. Workspace lints have `warnings = "deny"`, so an orphaned helper trips dead_code and turns main red. This is the cause of the post-#5368 main breakage; affected every PR built against `origin/main`. The sibling helpers `default_channel_max_backoff_secs` and `default_channel_initial_backoff_2s` still have live consumers (WeChat, QQ, Feishu …) and stay. cargo check -p librefang-types --lib clean. * fix(api/skills): replace ChannelsConfig::matrix round-trip with raw header count After #5368 migrated Matrix to a sidecar, `librefang_types::config:: ChannelsConfig` no longer has a `matrix` field. Three test cases in `routes::skills::tests` were still round-tripping the appended / removed TOML through `ChannelsConfig` and asserting on `parsed.channels.matrix.len()`, so the test build failed with `E0609: no field `matrix` on type `ChannelsConfig`` against current main. Replace the typed deserialization with a raw count of `[[channels.matrix]]` headers in the rewritten file. The covered behavior is the TOML-string mutation (`append_channel_instance`, `remove_channel_instance` — both string-based helpers), so the header count is the equivalent assertion: one array-of-tables header per logical instance. The legacy-single-table-form check is kept by also asserting absence of `[channels.matrix]`. All 4 affected tests green. Bundles with the orphan-backoff fix because both have the same root cause (#5368 fallout) and ship together as the main-red recovery PR. * fix(kernel/config_reload): swap matrix→wechat in hot-reload test Third #5368 fallout site: `config_reload::tests::test_channels_hot_reload` built a `MatrixConfig` and assigned to `b.channels.matrix`, but the Matrix sidecar migration removed both the field and the type. Switch the test fixture to `WeChatConfig` — one of the remaining in-process channels with a usable `Default` impl. Bundled with the orphan-backoff + skills.rs fix to ship as a single main-red recovery PR. * fix(kernel/agent_execution): anchor [SILENT] cron marker to message prefix (closes #5385) audit: silent-marker-substring-match (Severity: Low). `execute_llm_agent` had two `is_internal_cron && message.contains("[SILENT]")` sites — the LLM-side stripping at line 890 and the persistence-skip at line 1114. The `is_internal_cron` gate stopped a plain user chat from triggering the path, but a cron prompt template that interpolated runtime data (channel content, tool output, user-supplied variables) could still hit the substring match if the payload happened to contain the literal `[SILENT]` token anywhere in the body. Result: the assistant turn gets silently dropped from session history even though the operator never opted in. Fix: extract a `strip_silent_cron_marker(message, is_internal_cron) -> (String, bool)` helper that requires the marker to be the first non-whitespace token (after `trim_start`). Both sites consume the shared `is_silent_cron` flag so the LLM-side strip and the persistence-skip can never disagree (a divergence would suppress history for a turn the LLM saw the unstripped version of). The helper also moves from `replace("[SILENT]", "")` to a single `strip_prefix("[SILENT]")`, so embedded later occurrences of the marker survive verbatim — that's the operator's intent: a single marker at the start opts the run into silent mode, additional `[SILENT]` substrings deeper in the prompt are just text. 6 new tests under `silent_marker_tests` cover: non-cron call, leading marker, leading-whitespace + marker, marker-only message (empty-fallback), embedded mid-message marker (regression for the substring bug), and embedded second marker preservation. cargo clippy -p librefang-kernel --all-targets clean. Includes the three prerequisite #5368-fallout commits (orphan default_channel_initial_backoff_secs, skills.rs typed round-trip, config_reload.rs matrix→wechat). Also shipping in #5379; deduped on rebase once #5379 lands. * chore(codegen): auto-regenerate openapi.json + sdk + schema baselines [skip ci] --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
…ion (#5388) * fix(types): remove orphan default_channel_initial_backoff_secs helper #5368 (Matrix in-process → sidecar) removed the last consumer of `default_channel_initial_backoff_secs` (the 1-second variant). The matching Signal consumer was already removed earlier when Signal migrated to a sidecar. Both sidecar channels control their own backoff via env vars now. Workspace lints have `warnings = "deny"`, so an orphaned helper trips dead_code and turns main red. This is the cause of the post-#5368 main breakage; affected every PR built against `origin/main`. The sibling helpers `default_channel_max_backoff_secs` and `default_channel_initial_backoff_2s` still have live consumers (WeChat, QQ, Feishu …) and stay. cargo check -p librefang-types --lib clean. * fix(api/skills): replace ChannelsConfig::matrix round-trip with raw header count After #5368 migrated Matrix to a sidecar, `librefang_types::config:: ChannelsConfig` no longer has a `matrix` field. Three test cases in `routes::skills::tests` were still round-tripping the appended / removed TOML through `ChannelsConfig` and asserting on `parsed.channels.matrix.len()`, so the test build failed with `E0609: no field `matrix` on type `ChannelsConfig`` against current main. Replace the typed deserialization with a raw count of `[[channels.matrix]]` headers in the rewritten file. The covered behavior is the TOML-string mutation (`append_channel_instance`, `remove_channel_instance` — both string-based helpers), so the header count is the equivalent assertion: one array-of-tables header per logical instance. The legacy-single-table-form check is kept by also asserting absence of `[channels.matrix]`. All 4 affected tests green. Bundles with the orphan-backoff fix because both have the same root cause (#5368 fallout) and ship together as the main-red recovery PR. * fix(kernel/config_reload): swap matrix→wechat in hot-reload test Third #5368 fallout site: `config_reload::tests::test_channels_hot_reload` built a `MatrixConfig` and assigned to `b.channels.matrix`, but the Matrix sidecar migration removed both the field and the type. Switch the test fixture to `WeChatConfig` — one of the remaining in-process channels with a usable `Default` impl. Bundled with the orphan-backoff + skills.rs fix to ship as a single main-red recovery PR. * fix(api): clamp listing endpoints — no more "limit=None → full collection" (closes #5387) audit: agent-list-limit-none-unbounded (Severity: Medium). Two listing surfaces silently returned the entire unpaginated collection when the caller omitted `limit=`: * `GET /api/agents` (routes/agents.rs::list_agents) — `params.limit.map(|l| l.min(500))` clamped only when `Some`, so `None` skipped the `take()` branch and serialised every `AgentEntry`. A multi-thousand-agent deployment turned this into a memory + JSON-serialisation DoS sink. * `PaginationQuery::paginate` (types.rs:703) — short-circuit at line 706 (`if self.offset.is_none() && self.limit.is_none()`) returned `(items, total, 0, None)`. Live callers `routes::skills::list_skills` and `routes::network::peers` inherited the same DoS lever. Fix: * `list_agents`: introduce DEFAULT_AGENT_LIST_LIMIT = 500 and MAX_AGENT_LIST_LIMIT = 500 (the historical ceiling), and use `limit.unwrap_or(DEFAULT).min(MAX)`. Response envelope now always reports `Some(limit)`. * `PaginationQuery::paginate`: drop the both-unset short-circuit; always fall back to PAGINATION_MAX_LIMIT (= 100). The cap is a ceiling, not a floor, so a 10-row collection still comes back in full — only the > MAX cases truncate. Regression tests: * `routes::agents::tests::agent_list_limit_clamps_at_max_when_caller_omits_limit` — exercises the clamp helper directly (clippy const-folds away literal `None`/`Some(usize::MAX)` if you inline them, so the test drives the helper through opaque inputs). * `types::tests::pagination_unspecified_falls_back_to_max_limit_not_full_collection` — replaces the previous `_returns_full_collection` test that encoded the unbounded behaviour. * `types::tests::pagination_unspecified_truncates_large_collection_at_max_limit` — exercises the actual DoS scenario (collection > MAX). cargo clippy -p librefang-api --all-targets clean. Includes the three prerequisite #5368-fallout commits (orphan default_channel_initial_backoff_secs, skills.rs typed round-trip, config_reload.rs matrix→wechat); also shipping in #5379. After #5379 lands they dedupe on rebase. * test(api): update list-paginated test for finite default limit The DoS fix in this PR makes `GET /api/agents` always report a finite server-applied limit (DEFAULT_AGENT_LIST_LIMIT = 500) instead of null when the caller omits ?limit=. The api_integration_test assertion still expected null and failed; align it with the new envelope contract already pinned by the types.rs paginate unit tests. * chore(codegen): auto-regenerate openapi.json + sdk + schema baselines [skip ci] --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(types): remove orphan default_channel_initial_backoff_secs helper #5368 (Matrix in-process → sidecar) removed the last consumer of `default_channel_initial_backoff_secs` (the 1-second variant). The matching Signal consumer was already removed earlier when Signal migrated to a sidecar. Both sidecar channels control their own backoff via env vars now. Workspace lints have `warnings = "deny"`, so an orphaned helper trips dead_code and turns main red. This is the cause of the post-#5368 main breakage; affected every PR built against `origin/main`. The sibling helpers `default_channel_max_backoff_secs` and `default_channel_initial_backoff_2s` still have live consumers (WeChat, QQ, Feishu …) and stay. cargo check -p librefang-types --lib clean. * fix(api/skills): replace ChannelsConfig::matrix round-trip with raw header count After #5368 migrated Matrix to a sidecar, `librefang_types::config:: ChannelsConfig` no longer has a `matrix` field. Three test cases in `routes::skills::tests` were still round-tripping the appended / removed TOML through `ChannelsConfig` and asserting on `parsed.channels.matrix.len()`, so the test build failed with `E0609: no field `matrix` on type `ChannelsConfig`` against current main. Replace the typed deserialization with a raw count of `[[channels.matrix]]` headers in the rewritten file. The covered behavior is the TOML-string mutation (`append_channel_instance`, `remove_channel_instance` — both string-based helpers), so the header count is the equivalent assertion: one array-of-tables header per logical instance. The legacy-single-table-form check is kept by also asserting absence of `[channels.matrix]`. All 4 affected tests green. Bundles with the orphan-backoff fix because both have the same root cause (#5368 fallout) and ship together as the main-red recovery PR. * fix(kernel/config_reload): swap matrix→wechat in hot-reload test Third #5368 fallout site: `config_reload::tests::test_channels_hot_reload` built a `MatrixConfig` and assigned to `b.channels.matrix`, but the Matrix sidecar migration removed both the field and the type. Switch the test fixture to `WeChatConfig` — one of the remaining in-process channels with a usable `Default` impl. Bundled with the orphan-backoff + skills.rs fix to ship as a single main-red recovery PR. * fix(types/config): swap matrix→wechat in OneOrMany / channels tests Fourth #5368 fallout site: librefang-types/src/config/mod.rs has six test cases that exercised either MatrixConfig directly (test_one_or_many_serialize_roundtrip, test_account_id_in_channel_configs) or `b.channels.matrix` shape (test_channels_config_with_new_channels, test_one_or_many_single_toml_table, test_one_or_many_array_of_tables, test_one_or_many_empty_default). Matrix migrated to a sidecar in #5368 so both the field and the type are gone. These tests cover the OneOrMany helper, not Matrix specifically. Swapped the fixture to WeChat — one of the remaining in-process channels with a usable Default impl — and trimmed Matrix-only fields (homeserver_url) from the TOML strings. All 832 librefang-types lib tests green. Bundled into #5379 so a single PR closes out the main-red recovery. * fix(types/oauth): hand-write Debug to redact OAuthTokens secrets (closes #5394) audit: oauth-tokens-derive-debug-serialize (Severity: High). `OAuthTokens` derived `Debug` with the cleartext `access_token` and `refresh_token` fields wide open. Every one of the following caller patterns leaked the secret into logs / trace / panic backtrace: - `tracing::debug!(?tokens, …)` / `error!(?tokens, …)` - `format!("{:?}", tokens)` and panic messages including the type - `serde_json::to_string` flowing through snapshots / dashboards - cross-process trace dumps The `access_token_zeroizing()` / `refresh_token_zeroizing()` helpers already exist but only protect the call sites that remember to use them — there was no type-level enforcement. Fix: replace `#[derive(Debug)]` with a hand-written impl that emits `OAuthTokens { access_token: <redacted len=N hint=****XXXX>, refresh_token: Some(<redacted …>) | None, token_type, expires_in, scope }`. Pattern mirrors `PooledCredential::Debug` in `librefang-llm-drivers::credential_pool` (the canonical example). Length + last-4 hint stays in the log so operators can correlate "same token as last time" without exposing the secret. The `redact()` helper: * len 0 → "<redacted len=0>" * len 1..=4 → "<redacted len=N>" (never reveals any chars of a very short secret) * len 5..=7 → "<redacted len=N hint=****<tail-N-4>>" (linear hint window so 5-char tokens still show 1 char of fingerprint) * len 8+ → "<redacted len=N hint=****<last-4>>" Uses char count not byte count, so unicode tokens render consistently (test `redact_uses_char_count_not_byte_count_for_unicode_secrets`). Six new tests in `oauth::tests`: - debug_redacts_access_and_refresh_tokens (the audit-named bug) - debug_redacts_none_refresh_token_without_panic - redact_hint_handles_short_secrets_without_exposing_full_value - redact_uses_char_count_not_byte_count_for_unicode_secrets plus the existing 5 oauth tests still green. `cargo clippy -p librefang-types --all-targets` clean. Out of scope (deferred per audit doc): switching to `SecretString` / `Zeroizing<String>` and splitting `OAuthTokensWire` from internal `OAuthTokens`. Both are larger surgeries (every call site of `OAuthTokens` needs an update); the Debug redaction closes the most common leak path today and is the change requested by the audit's first recommendation. Includes four prerequisite #5368-fallout commits (orphan helper + three matrix→wechat test swaps); also shipping in #5379. They dedupe on rebase once #5379 lands. * chore(codegen): auto-regenerate openapi.json + sdk + schema baselines [skip ci] --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(types): remove orphan default_channel_initial_backoff_secs helper #5368 (Matrix in-process → sidecar) removed the last consumer of `default_channel_initial_backoff_secs` (the 1-second variant). The matching Signal consumer was already removed earlier when Signal migrated to a sidecar. Both sidecar channels control their own backoff via env vars now. Workspace lints have `warnings = "deny"`, so an orphaned helper trips dead_code and turns main red. This is the cause of the post-#5368 main breakage; affected every PR built against `origin/main`. The sibling helpers `default_channel_max_backoff_secs` and `default_channel_initial_backoff_2s` still have live consumers (WeChat, QQ, Feishu …) and stay. cargo check -p librefang-types --lib clean. * fix(api/skills): replace ChannelsConfig::matrix round-trip with raw header count After #5368 migrated Matrix to a sidecar, `librefang_types::config:: ChannelsConfig` no longer has a `matrix` field. Three test cases in `routes::skills::tests` were still round-tripping the appended / removed TOML through `ChannelsConfig` and asserting on `parsed.channels.matrix.len()`, so the test build failed with `E0609: no field `matrix` on type `ChannelsConfig`` against current main. Replace the typed deserialization with a raw count of `[[channels.matrix]]` headers in the rewritten file. The covered behavior is the TOML-string mutation (`append_channel_instance`, `remove_channel_instance` — both string-based helpers), so the header count is the equivalent assertion: one array-of-tables header per logical instance. The legacy-single-table-form check is kept by also asserting absence of `[channels.matrix]`. All 4 affected tests green. Bundles with the orphan-backoff fix because both have the same root cause (#5368 fallout) and ship together as the main-red recovery PR. * fix(kernel/config_reload): swap matrix→wechat in hot-reload test Third #5368 fallout site: `config_reload::tests::test_channels_hot_reload` built a `MatrixConfig` and assigned to `b.channels.matrix`, but the Matrix sidecar migration removed both the field and the type. Switch the test fixture to `WeChatConfig` — one of the remaining in-process channels with a usable `Default` impl. Bundled with the orphan-backoff + skills.rs fix to ship as a single main-red recovery PR. * fix(types/config): swap matrix→wechat in OneOrMany / channels tests Fourth #5368 fallout site: librefang-types/src/config/mod.rs has six test cases that exercised either MatrixConfig directly (test_one_or_many_serialize_roundtrip, test_account_id_in_channel_configs) or `b.channels.matrix` shape (test_channels_config_with_new_channels, test_one_or_many_single_toml_table, test_one_or_many_array_of_tables, test_one_or_many_empty_default). Matrix migrated to a sidecar in #5368 so both the field and the type are gone. These tests cover the OneOrMany helper, not Matrix specifically. Swapped the fixture to WeChat — one of the remaining in-process channels with a usable Default impl — and trimmed Matrix-only fields (homeserver_url) from the TOML strings. All 832 librefang-types lib tests green. Bundled into #5379 so a single PR closes out the main-red recovery. * fix(wire): warn on serde(other) Unknown variants with raw tag (closes #5396) audit: wire-message-other-variant-silent (Severity: Medium). Four `#[serde(other)]` arms in librefang-wire/src/message.rs (WireMessageKind::Unknown, WireRequest::Unknown, WireResponse::Unknown, WireNotification::Unknown) silently dropped every unrecognised variant. Two of them logged at `debug!` level — default-off in production — and the other two had no log at all. A misbehaving peer could flood `{"type":"garbage", …}` and the receiver would silently absorb every message: no log line, no counter, no metric. Combined with `#[serde(default)] nonce: String` / `auth_hmac: String` at message.rs:54-58, a peer that sends an unknown handshake variant bypasses structural checks too. Fix at the receive boundary (where the body bytes are still local): * New `classify_unknown(body, msg) -> Option<UnknownTag>` in message.rs. When the decoded message landed in any of the four serde(other) arms, peek the raw JSON body for the relevant field (envelope `type`, request/response `method`, notification `event`) and return the raw tag string the peer actually sent. Returns None on known variants so receivers only pay the json::Value re-parse on the unknown branch. * New `read_message_observed(reader, peer_node_id)` and `read_message_authenticated_observed(reader, key, peer_node_id)` wrap the existing helpers, call `classify_unknown` after a successful decode, and emit a structured `warn!` on `target: "wire::compat"` with peer, msg_id, level (`envelope.type` / `request.method` / `response.method` / `notification.event`) and `raw_tag`. The original `read_message` / `read_message_authenticated` keep their pub signatures (delegate through the observed variants with `peer_node_id = "<pre-handshake>"`) so external callers are unaffected. * `connection_loop` switches to the observed variants — that's the post-handshake hot path where the actual `peer_node_id` is known. The per-site `debug!` lines for `Unknown` variants stay (now framed as "no dispatch") for trace-level correlation, but the visibility-load is on the upstream `warn!`. Six new tests in `message::tests` cover classify_unknown for each of the four levels, plus the happy path (returns None for known variants) and the malformed-tag fallback to `"<missing>"`. All 63 librefang-wire tests green; `cargo clippy -p librefang-wire --all-targets` clean. Out of scope (deferred per audit doc): per-peer counter + threshold-disconnect. That's stateful infrastructure that needs care to not nuke compatible peers running newer-protocol features — separate PR. Today the operator-visible warn is the main missing visibility, and that's covered. Includes four prerequisite #5368-fallout commits (orphan helper + three matrix→wechat test swaps); also shipping in #5379. They dedupe on rebase once #5379 lands. * chore(codegen): auto-regenerate openapi.json + sdk + schema baselines [skip ci] --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
…dit_entries(agent_id, timestamp) indexes (#5399) * fix(types): remove orphan default_channel_initial_backoff_secs helper #5368 (Matrix in-process → sidecar) removed the last consumer of `default_channel_initial_backoff_secs` (the 1-second variant). The matching Signal consumer was already removed earlier when Signal migrated to a sidecar. Both sidecar channels control their own backoff via env vars now. Workspace lints have `warnings = "deny"`, so an orphaned helper trips dead_code and turns main red. This is the cause of the post-#5368 main breakage; affected every PR built against `origin/main`. The sibling helpers `default_channel_max_backoff_secs` and `default_channel_initial_backoff_2s` still have live consumers (WeChat, QQ, Feishu …) and stay. cargo check -p librefang-types --lib clean. * fix(api/skills): replace ChannelsConfig::matrix round-trip with raw header count After #5368 migrated Matrix to a sidecar, `librefang_types::config:: ChannelsConfig` no longer has a `matrix` field. Three test cases in `routes::skills::tests` were still round-tripping the appended / removed TOML through `ChannelsConfig` and asserting on `parsed.channels.matrix.len()`, so the test build failed with `E0609: no field `matrix` on type `ChannelsConfig`` against current main. Replace the typed deserialization with a raw count of `[[channels.matrix]]` headers in the rewritten file. The covered behavior is the TOML-string mutation (`append_channel_instance`, `remove_channel_instance` — both string-based helpers), so the header count is the equivalent assertion: one array-of-tables header per logical instance. The legacy-single-table-form check is kept by also asserting absence of `[channels.matrix]`. All 4 affected tests green. Bundles with the orphan-backoff fix because both have the same root cause (#5368 fallout) and ship together as the main-red recovery PR. * fix(kernel/config_reload): swap matrix→wechat in hot-reload test Third #5368 fallout site: `config_reload::tests::test_channels_hot_reload` built a `MatrixConfig` and assigned to `b.channels.matrix`, but the Matrix sidecar migration removed both the field and the type. Switch the test fixture to `WeChatConfig` — one of the remaining in-process channels with a usable `Default` impl. Bundled with the orphan-backoff + skills.rs fix to ship as a single main-red recovery PR. * fix(types/config): swap matrix→wechat in OneOrMany / channels tests Fourth #5368 fallout site: librefang-types/src/config/mod.rs has six test cases that exercised either MatrixConfig directly (test_one_or_many_serialize_roundtrip, test_account_id_in_channel_configs) or `b.channels.matrix` shape (test_channels_config_with_new_channels, test_one_or_many_single_toml_table, test_one_or_many_array_of_tables, test_one_or_many_empty_default). Matrix migrated to a sidecar in #5368 so both the field and the type are gone. These tests cover the OneOrMany helper, not Matrix specifically. Swapped the fixture to WeChat — one of the remaining in-process channels with a usable Default impl — and trimmed Matrix-only fields (homeserver_url) from the TOML strings. All 832 librefang-types lib tests green. Bundled into #5379 so a single PR closes out the main-red recovery. * perf(memory/migration): add composite sessions(agent_id, updated_at) + audit_entries(agent_id, timestamp) indexes (closes #5398) audit: sessions-missing-index (Severity: Medium). The `sessions` table had no usable index for the per-agent recency hot path. The closest was `idx_sessions_peer ON sessions(agent_id, peer_id)` from v16: in theory a prefix-scan for `WHERE agent_id = ?`, but the planner's choice was not stable, and the index name discourages future hinting. Hot-path query in `count_agent_sessions_touched_since` (concurrent-trigger admission check, runs on every fire): SELECT COUNT(*) FROM sessions WHERE agent_id = ?1 AND updated_at > ?2 On any deployment with > a few thousand sessions this degraded to a full-table scan. `DELETE FROM sessions WHERE agent_id = ?1` (cascade on agent removal) had the same shape. The audit doc flagged the same shape on `audit_entries`: v8 created two separate single-column indexes (agent_id, timestamp) so per-agent recency queries pick one or the other and pay a sort. Fix: add a new migration v41 that creates two composite indexes: CREATE INDEX IF NOT EXISTS idx_sessions_agent_updated ON sessions(agent_id, updated_at); CREATE INDEX IF NOT EXISTS idx_audit_agent_timestamp ON audit_entries(agent_id, timestamp); `IF NOT EXISTS` makes a fresh boot idempotent. SCHEMA_VERSION bumped 40 → 41 and the ladder runner gets a new `run_step!(41, migrate_v41)` entry with the explanatory comment. Two regression tests in `migration::tests`: - `migrate_v41_creates_composite_indexes_used_by_hot_paths` — confirms both indexes are present in `sqlite_master` and uses `EXPLAIN QUERY PLAN` to assert the planner actually picks `idx_sessions_agent_updated` for the `count_agent_sessions_touched_since` query (regression for "degrades to full-table scan"). - `migrate_v41_is_idempotent` — three repeated `run_migrations` calls leave exactly one of the new index. `cargo clippy -p librefang-memory --all-targets` clean. Includes four prerequisite #5368-fallout commits (orphan helper + three matrix→wechat test swaps); also shipping in #5379. They dedupe on rebase once #5379 lands. * chore(codegen): auto-regenerate openapi.json + sdk + schema baselines [skip ci] --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
…-concat (#5401) * fix(types): remove orphan default_channel_initial_backoff_secs helper #5368 (Matrix in-process → sidecar) removed the last consumer of `default_channel_initial_backoff_secs` (the 1-second variant). The matching Signal consumer was already removed earlier when Signal migrated to a sidecar. Both sidecar channels control their own backoff via env vars now. Workspace lints have `warnings = "deny"`, so an orphaned helper trips dead_code and turns main red. This is the cause of the post-#5368 main breakage; affected every PR built against `origin/main`. The sibling helpers `default_channel_max_backoff_secs` and `default_channel_initial_backoff_2s` still have live consumers (WeChat, QQ, Feishu …) and stay. cargo check -p librefang-types --lib clean. * fix(api/skills): replace ChannelsConfig::matrix round-trip with raw header count After #5368 migrated Matrix to a sidecar, `librefang_types::config:: ChannelsConfig` no longer has a `matrix` field. Three test cases in `routes::skills::tests` were still round-tripping the appended / removed TOML through `ChannelsConfig` and asserting on `parsed.channels.matrix.len()`, so the test build failed with `E0609: no field `matrix` on type `ChannelsConfig`` against current main. Replace the typed deserialization with a raw count of `[[channels.matrix]]` headers in the rewritten file. The covered behavior is the TOML-string mutation (`append_channel_instance`, `remove_channel_instance` — both string-based helpers), so the header count is the equivalent assertion: one array-of-tables header per logical instance. The legacy-single-table-form check is kept by also asserting absence of `[channels.matrix]`. All 4 affected tests green. Bundles with the orphan-backoff fix because both have the same root cause (#5368 fallout) and ship together as the main-red recovery PR. * fix(kernel/config_reload): swap matrix→wechat in hot-reload test Third #5368 fallout site: `config_reload::tests::test_channels_hot_reload` built a `MatrixConfig` and assigned to `b.channels.matrix`, but the Matrix sidecar migration removed both the field and the type. Switch the test fixture to `WeChatConfig` — one of the remaining in-process channels with a usable `Default` impl. Bundled with the orphan-backoff + skills.rs fix to ship as a single main-red recovery PR. * fix(types/config): swap matrix→wechat in OneOrMany / channels tests Fourth #5368 fallout site: librefang-types/src/config/mod.rs has six test cases that exercised either MatrixConfig directly (test_one_or_many_serialize_roundtrip, test_account_id_in_channel_configs) or `b.channels.matrix` shape (test_channels_config_with_new_channels, test_one_or_many_single_toml_table, test_one_or_many_array_of_tables, test_one_or_many_empty_default). Matrix migrated to a sidecar in #5368 so both the field and the type are gone. These tests cover the OneOrMany helper, not Matrix specifically. Swapped the fixture to WeChat — one of the remaining in-process channels with a usable Default impl — and trimmed Matrix-only fields (homeserver_url) from the TOML strings. All 832 librefang-types lib tests green. Bundled into #5379 so a single PR closes out the main-red recovery. * fix(memory): bind cleanup_orphan_sessions IN-clause instead of string-concat (closes #5400) audit: cleanup-orphan-sessions-format-sql (Severity: Medium). `cleanup_orphan_sessions` was the only place in the substrate that built SQL via `format!` instead of `?` parameter binding: placeholders.push(format!("'{}'", id.0)) format!("DELETE FROM sessions WHERE agent_id NOT IN ({in_clause})") Today the inner type is `AgentId(Uuid)` and the Display impl only emits `[0-9a-f-]`, so the query is safe by accident. But: - This was the ONLY violation of the substrate-wide "use `?`" rule — easy for a future grep to miss. - The moment `AgentId(Uuid)` is relaxed to `AgentId(String)` (e.g. namespacing hand-agent ids), the door opens silently. - No test pinned the invariant down. Fix: build the IN-clause as `?,?,?,…` and bind the values via `rusqlite::params_from_iter(live_agent_ids.iter().map(|id| id.0.to_string()))`. Same query plan, no escaping dependency on the inner type. Used `std::iter::repeat_n` (stable in Rust 1.82+) to build the placeholder string. Two new tests in `session::tests`: - `test_cleanup_orphan_sessions_uses_bound_parameters_not_string_concat` — creates 3 live + 1 orphan agent session, asserts exactly the orphan row goes and the 3 live rows survive. Doubles as a sanity check that the parameter-binding shape works (the old format! version would have worked here too, but the test pins the contract going forward). - `test_cleanup_orphan_sessions_empty_live_set_deletes_nothing` — empty `live_agent_ids` is the "no live agents → don't touch anything" early-return. Documents the invariant so an off-by-one in a future refactor doesn't silently wipe every session when the registry is momentarily empty (e.g. during startup reload). `cargo clippy -p librefang-memory --all-targets` clean. Includes four prerequisite #5368-fallout commits (orphan helper + three matrix→wechat test swaps); also shipping in #5379. They dedupe on rebase once #5379 lands. * chore(codegen): auto-regenerate openapi.json + sdk + schema baselines [skip ci] --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Summary
Replaces the 3356-line in-process
librefang-channels::matrixadapter (the biggest channel by Rust LOC) with an out-of-process Python sidecar atsdk/python/librefang/sidecar/adapters/matrix.py. Same pattern as ntfy #5224 through qq #5325.The Rust adapter carried 5 send variants (text + media + edit + reaction + streaming), m.replace edit lifecycle with 429 retry, mxc:// → MSC3916 authenticated media, E2EE warn-once, and a
pulldown-cmark-backed Markdown→HTML rendering forformatted_body. The sidecar replaces all of it, including the markdown rendering via a stdlib-only CommonMark subset renderer (no third-party deps).Behaviour parity
Every claim cited against
crates/librefang-channels/src/matrix.rson the pre-migration tree:sincecursor + 30 s server timeout (matrix.rs:855-1008).sender == user_id(matrix.rs:940-962).mxc://→ MSC3916 conversion (matrix.rs:311-343).m.threadrelation →thread_idon inbound (matrix.rs:206-215).m.replaceedit shares one txn_id across both attempts under 429 (matrix.rs:737-774).on_sendtext + 11 ChannelContent variants;TypingCmd;Reactionwith redact+insert lifecycle (1024-entry cache);cmd.thread_idm.thread wrap;StreamStart/StreamDelta/StreamEndm.replace streaming.account_idmetadata injection (fix(channels): override account_id() in non-Telegram multi-bot adapters #5003).Three improvements over the Rust adapter
event_id— boundedSeenSetwithSEEN_MESSAGES_MAX=10000(Rust emitted every event from a sync batch unconditionally; on retry /sincereset the bot could re-emit).Retry-Afterhonoured at every PUT — Rust'sapi_edit_event_with_retryhonoured Retry-After butapi_send_event/api_redactdid not. Sidecar's shared_put_eventhonours it everywhere./sync, 30s on every other REST call — Rust relied onreqwest's default (none); a hung homeserver would hang the producer thread forever.Markdown → HTML (stdlib subset)
Replaces
pulldown-cmarkwith a hand-rolled renderer covering:# h1…###### h6)**x**), italic (*x*), inline code (`x`), strikethrough (~~x~~)[text](url)withjavascript:/data:scheme rejection (XSS guard)---)<think>...</think>strip (LLM artifact)Raw HTML in the source is HTML-entity-escaped before rendering so an LLM-authored
<script>cannot inject markup.Cascade removal
src/matrix.rsdeleted (3356 lines);lib.rsmod,channels-allowlist.txt,Cargo.tomlfeatures (incl. optionalpulldown-cmarkdep) removed.MatrixConfigstruct +ChannelsConfig.matrixfield + 2 unit tests gone.channel_bridge.rsimport + spawn block +find_channel_info!+check_channel!+ default-empty assertion removed.routes/channels.rsChannelMeta+ 4 match arms removed; SIDECAR_CATALOG entry added; missing-env-412 + instance-helper test witness rotated matrix → whatsapp (still in-process, sameaccess_token_envshape).routes/config.rsch!(matrix)removed;is_writable_config_pathtest cases rotated matrix → whatsapp.channel_sender.rsfor_each_channel_field!+EXPECTEDentry removed.ChannelDefremoved.librefang-types::config::validationwarn-loop for matrix removed.librefang-migrate::openclawrecords[channels.matrix]as a skipped sidecar channel;test_policy_migrationrotates witness matrix → feishu (still in-process, acceptsdmPolicy);test_json5_full_migrationcount 5 → 4 in-process imports.crates/librefang-api/tests/channels_routes_test.rsdeleted — it usedMatrixConfigas its only in-process witness across 82 references; rewriting against another channel needs its own PR to be reviewable.[channels.matrix]blocks replaced with[[sidecar_channels]]redirects.New env-var knobs
From
[sidecar_channels.env]:MATRIX_HOMESERVER_URL,MATRIX_USER_ID(required),MATRIX_ALLOWED_ROOMS(CSV),MATRIX_ACCOUNT_ID,MATRIX_MAX_UPLOAD_BYTES(default 50 MiB).From
~/.librefang/secrets.env:MATRIX_ACCESS_TOKEN(required).Operator action required
An existing
[channels.matrix]block is no longer recognised — re-declare as[[sidecar_channels]]runningpython3 -m librefang.sidecar.adapters.matrix. Seesdk/python/librefang/sidecar/adapters/matrix.pyheader for the exact config.ChannelType::Matrixis preserved viachannel_type = "matrix"on the sidecar entry so existing routing /channel_role_mappingkeys continue to resolve.Verification
pytest sdk/python/tests/test_matrix_adapter.py -q— 81 passed in 0.12 s. Covers:mxc_to_http(4 cases incl. malformed)markdown_to_matrix_html(15 cases incl.javascript:+data:scheme rejection, HTML escape,<think>strip, GFM tables, code blocks with language, lists, blockquote, hr)text_body_with_html+build_edit_body(incl. MAX_MESSAGE_LEN truncation)parse_thread_relation(present / absent / replace / malformed)parse_inbound_msg_contentfor 5 msgtypes + edge cases (empty body / unknown / missing url / slash-command / MSC3245 voice / Audio plain / Video / Filefilenameoverbody)_process_sync_body(emit, self-skip, room allowlist, dedupe across batches, E2EE warn-once, non-m.room.message skip, thread surfacing, account_id injection)_put_event(happy / 429-then-200 / persistent-429-raises / non-2xx-raises)_upload_media(returns mxc / size-cap / failure raises)_validate(200 / 401)on_send(text / chunked / thread-wraps / empty-room / fallback to user.platform_id)Caveats
crates/librefang-api/tests/channels_routes_test.rswas deleted because it usedMatrixConfigas its only in-process witness across 82 references — a faithful rewrite against another channel (e.g. WhatsApp) needs a separate PR. The functionality (multi-instance config, dashboard registry inclusion, configure endpoint) is generic and would be re-asserted against any in-process channel.StreamStart/StreamDelta/StreamEnd) is wired into the sidecar but the daemon-sideSendprotocol's stream envelope handling is upstream of this PR; full end-to-end testing of streaming edits requires the daemon-side adapter contract to flow streams through.Diff
27 files changed, +2578 / −5403 (net −2825 LOC).