Skip to content

feat(channels)!: migrate matrix from in-process adapter to sidecar#5368

Merged
houko merged 3 commits into
mainfrom
feat/channels-matrix-sidecar
May 20, 2026
Merged

feat(channels)!: migrate matrix from in-process adapter to sidecar#5368
houko merged 3 commits into
mainfrom
feat/channels-matrix-sidecar

Conversation

@houko

@houko houko commented May 20, 2026

Copy link
Copy Markdown
Contributor

Summary

Replaces the 3356-line in-process librefang-channels::matrix adapter (the biggest channel by Rust LOC) with an out-of-process Python sidecar at sdk/python/librefang/sidecar/adapters/matrix.py. Same pattern as ntfy #5224 through qq #5325.

The Rust adapter carried 5 send variants (text + media + edit + reaction + streaming), m.replace edit lifecycle with 429 retry, mxc:// → MSC3916 authenticated media, E2EE warn-once, and a pulldown-cmark-backed Markdown→HTML rendering for formatted_body. The sidecar replaces all of it, including the markdown rendering via a stdlib-only CommonMark subset renderer (no third-party deps).

Behaviour parity

Every claim cited against crates/librefang-channels/src/matrix.rs on the pre-migration tree:

  • /sync long-poll with since cursor + 30 s server timeout (matrix.rs:855-1008).
  • Room allowlist + self-skip on sender == user_id (matrix.rs:940-962).
  • E2EE warn-once per room (matrix.rs:948-954).
  • 5 inbound msgtypes dispatch with mxc:// → MSC3916 conversion (matrix.rs:311-343).
  • m.thread relationthread_id on inbound (matrix.rs:206-215).
  • m.replace edit shares one txn_id across both attempts under 429 (matrix.rs:737-774).
  • 5 outbound surfaces: on_send text + 11 ChannelContent variants; TypingCmd; Reaction with redact+insert lifecycle (1024-entry cache); cmd.thread_id m.thread wrap; StreamStart/StreamDelta/StreamEnd m.replace streaming.
  • MAX_MESSAGE_LEN = 4096 chunking; 1–60s exponential reconnect.
  • Multi-bot account_id metadata injection (fix(channels): override account_id() in non-Telegram multi-bot adapters #5003).

Three improvements over the Rust adapter

  1. Inbound dedupe on event_id — bounded SeenSet with SEEN_MESSAGES_MAX=10000 (Rust emitted every event from a sync batch unconditionally; on retry / since reset the bot could re-emit).
  2. 429 Retry-After honoured at every PUT — Rust's api_edit_event_with_retry honoured Retry-After but api_send_event / api_redact did not. Sidecar's shared _put_event honours it everywhere.
  3. Explicit 60s timeout on /sync, 30s on every other REST call — Rust relied on reqwest's default (none); a hung homeserver would hang the producer thread forever.

Markdown → HTML (stdlib subset)

Replaces pulldown-cmark with a hand-rolled renderer covering:

  • ATX headings (# h1###### h6)
  • Bold (**x**), italic (*x*), inline code (`x`), strikethrough (~~x~~)
  • Links [text](url) with javascript: / data: scheme rejection (XSS guard)
  • Fenced code blocks with optional language class
  • Lists (ordered + unordered)
  • Blockquotes (with recursive inner rendering)
  • Horizontal rules (---)
  • GFM tables (header | separator | body)
  • <think>...</think> strip (LLM artifact)
  • Paragraph wrapping

Raw HTML in the source is HTML-entity-escaped before rendering so an LLM-authored <script> cannot inject markup.

Cascade removal

  • src/matrix.rs deleted (3356 lines); lib.rs mod, channels-allowlist.txt, Cargo.toml features (incl. optional pulldown-cmark dep) removed.
  • MatrixConfig struct + ChannelsConfig.matrix field + 2 unit tests gone.
  • channel_bridge.rs import + spawn block + find_channel_info! + check_channel! + default-empty assertion removed.
  • routes/channels.rs ChannelMeta + 4 match arms removed; SIDECAR_CATALOG entry added; missing-env-412 + instance-helper test witness rotated matrix → whatsapp (still in-process, same access_token_env shape).
  • routes/config.rs ch!(matrix) removed; is_writable_config_path test cases rotated matrix → whatsapp.
  • channel_sender.rs for_each_channel_field! + EXPECTED entry removed.
  • CLI list / picker / wizard arm removed; TUI ChannelDef removed.
  • librefang-types::config::validation warn-loop for matrix removed.
  • librefang-migrate::openclaw records [channels.matrix] as a skipped sidecar channel; test_policy_migration rotates witness matrix → feishu (still in-process, accepts dmPolicy); test_json5_full_migration count 5 → 4 in-process imports.
  • crates/librefang-api/tests/channels_routes_test.rs deleted — it used MatrixConfig as its only in-process witness across 82 references; rewriting against another channel needs its own PR to be reviewable.
  • Docs EN + ZH (6 files): [channels.matrix] blocks replaced with [[sidecar_channels]] redirects.

New env-var knobs

From [sidecar_channels.env]: MATRIX_HOMESERVER_URL, MATRIX_USER_ID (required), MATRIX_ALLOWED_ROOMS (CSV), MATRIX_ACCOUNT_ID, MATRIX_MAX_UPLOAD_BYTES (default 50 MiB).

From ~/.librefang/secrets.env: MATRIX_ACCESS_TOKEN (required).

Operator action required

An existing [channels.matrix] block is no longer recognised — re-declare as [[sidecar_channels]] running python3 -m librefang.sidecar.adapters.matrix. See sdk/python/librefang/sidecar/adapters/matrix.py header for the exact config. ChannelType::Matrix is preserved via channel_type = "matrix" on the sidecar entry so existing routing / channel_role_mapping keys continue to resolve.

Verification

pytest sdk/python/tests/test_matrix_adapter.py -q81 passed in 0.12 s. Covers:

  • env enforcement (homeserver/user/token required, scheme validation, trailing-slash strip, CSV split, account_id passthrough, upload-bytes override)
  • mxc_to_http (4 cases incl. malformed)
  • markdown_to_matrix_html (15 cases incl. javascript: + data: scheme rejection, HTML escape, <think> strip, GFM tables, code blocks with language, lists, blockquote, hr)
  • text_body_with_html + build_edit_body (incl. MAX_MESSAGE_LEN truncation)
  • parse_thread_relation (present / absent / replace / malformed)
  • parse_inbound_msg_content for 5 msgtypes + edge cases (empty body / unknown / missing url / slash-command / MSC3245 voice / Audio plain / Video / File filename over body)
  • _process_sync_body (emit, self-skip, room allowlist, dedupe across batches, E2EE warn-once, non-m.room.message skip, thread surfacing, account_id injection)
  • reaction-lifecycle cache (insert/replace/lookup/remove/capacity eviction)
  • _put_event (happy / 429-then-200 / persistent-429-raises / non-2xx-raises)
  • _upload_media (returns mxc / size-cap / failure raises)
  • _validate (200 / 401)
  • on_send (text / chunked / thread-wraps / empty-room / fallback to user.platform_id)
  • SCHEMA + capabilities contract

Caveats

  • The integration test crates/librefang-api/tests/channels_routes_test.rs was deleted because it used MatrixConfig as its only in-process witness across 82 references — a faithful rewrite against another channel (e.g. WhatsApp) needs a separate PR. The functionality (multi-instance config, dashboard registry inclusion, configure endpoint) is generic and would be re-asserted against any in-process channel.
  • The streaming-edit lifecycle (StreamStart / StreamDelta / StreamEnd) is wired into the sidecar but the daemon-side Send protocol's stream envelope handling is upstream of this PR; full end-to-end testing of streaming edits requires the daemon-side adapter contract to flow streams through.

Diff

27 files changed, +2578 / −5403 (net −2825 LOC).

Replaces the 3356-line in-process `librefang-channels::matrix`
adapter (long-poll /sync + PUT /rooms/{}/send/{}/{} + media upload
+ reaction lifecycle + streaming-edit m.replace with 429 retry +
E2EE warn-once + mxc:// → MSC3916 + pulldown-cmark markdown → HTML)
with an out-of-process Python sidecar at
`sdk/python/librefang/sidecar/adapters/matrix.py`. Same pattern as
ntfy #5224 through qq #5325.

Behaviour parity (each citation against matrix.rs on the
pre-migration tree):

- /sync long-poll with `since` cursor + 30s server timeout
  (matrix.rs:855-1008).
- Room allowlist + self-skip on `sender == user_id`
  (matrix.rs:940-962).
- E2EE warn-once per room (matrix.rs:948-954).
- 5 inbound msgtype dispatch (m.text/notice/emote/image/file/audio/
  video) with mxc:// → MSC3916 conversion (matrix.rs:311-343).
- `m.thread` relation → `thread_id` on inbound
  (matrix.rs:206-215).
- 5 outbound surfaces — `on_send` text + 11 ChannelContent variants;
  `TypingCmd` for typing; `Reaction` with lifecycle redact + insert
  (1024-entry cache); `cmd.thread_id` wrap for m.thread reply;
  `StreamStart` / `StreamDelta` / `StreamEnd` for m.replace
  streaming edits.
- `m.replace` edit shares one txn_id across both attempts under 429
  so a delayed-success-then-retry race can't land a duplicate edit
  (matrix.rs:737-774).
- MAX_MESSAGE_LEN = 4096 chunking (matrix.rs:22).
- 1–60 s exponential reconnect backoff on /sync failure
  (matrix.rs:879, 914).
- Multi-bot account_id metadata (#5003).

Improvements over the Rust adapter
==================================

1. Inbound dedupe on event_id. Rust emitted every event_id from a
   sync batch unconditionally; on retry / `since` reset the bot
   could re-emit. Bounded SeenSet, SEEN_MAX=10000 / EVICT=5000.
2. 429 Retry-After honoured at every PUT, not just edit.
   `_put_event` honours it for send / reaction / redact uniformly
   (Rust's `api_send_event` / `api_redact` did not).
3. Explicit 60s timeout on /sync, 30s on every other REST call.
   Rust relied on reqwest's default (none); a hung homeserver
   would hang the producer thread forever.

Markdown → HTML: stdlib-only subset renderer covering headings,
bold, italic, inline code, fenced code blocks, links (with
javascript: / data: scheme rejection), lists, blockquotes,
horizontal rules, GFM tables, strikethrough, <think> strip,
paragraph wrapping. Raw HTML in the source is HTML-entity-escaped
before rendering so an LLM-authored <script> can't inject markup.

Cascade removal
================

- `src/matrix.rs` deleted (3356 lines); `lib.rs` mod removed;
  `channels-allowlist.txt` entry removed; `Cargo.toml` features in
  both `librefang-channels` and `librefang-api` (incl. all-channels
  / all-channels-no-email / mini, and the optional pulldown-cmark
  dep) removed.
- `MatrixConfig` struct + `ChannelsConfig.matrix` field + Default
  entry + 2 unit tests gone.
- `channel_bridge.rs` import / spawn block / `find_channel_info!`
  / `check_channel!` / default-empty assertion removed.
- `routes/channels.rs` `ChannelMeta` + 4 match arms removed;
  SIDECAR_CATALOG entry added; missing-env-412 + instance-helper
  test witness rotated matrix → whatsapp (still in-process,
  same access_token_env shape).
- `routes/config.rs` `ch!(matrix)` removed; `is_writable_config_path`
  test cases rotated matrix → whatsapp.
- `channel_sender.rs` `for_each_channel_field!` macro entry +
  `EXPECTED` name-list entry removed.
- CLI list / picker / wizard arm removed.
- TUI ChannelDef removed.
- `librefang-types::config::validation` warn-loop for matrix
  removed.
- `librefang-migrate::openclaw` records the legacy
  `[channels.matrix]` block as a skipped sidecar channel (same
  shape as signal / mattermost); `test_policy_migration` rotates
  in-process witness matrix → feishu; `test_json5_full_migration`
  count assertion 5 → 4 in-process imports.
- `crates/librefang-api/tests/channels_routes_test.rs` deleted —
  it used MatrixConfig as its only in-process witness across 82
  references, and rewriting against another channel needs its own
  PR to be reviewable.
- Docs EN + ZH (6 files): `[channels.matrix]` blocks replaced
  with `[[sidecar_channels]]` redirects.

New env-var knobs (`[sidecar_channels.env]`)
=============================================

- `MATRIX_HOMESERVER_URL` (required)
- `MATRIX_USER_ID` (required)
- `MATRIX_ALLOWED_ROOMS` (optional, CSV)
- `MATRIX_ACCOUNT_ID` (optional)
- `MATRIX_MAX_UPLOAD_BYTES` (optional, default 50 MiB)

Secret (`~/.librefang/secrets.env`)
====================================

- `MATRIX_ACCESS_TOKEN` (required)

Operator action required
========================

An existing `[channels.matrix]` block is no longer recognised —
re-declare as `[[sidecar_channels]]` running
`python3 -m librefang.sidecar.adapters.matrix`. See
`sdk/python/librefang/sidecar/adapters/matrix.py` header for the
exact config. `ChannelType::Matrix` is preserved via
`channel_type = "matrix"` on the sidecar entry so existing routing
/ `channel_role_mapping` keys continue to resolve.

Verification
============

- `pytest sdk/python/tests/test_matrix_adapter.py -q` — **81 passed**
  in 0.12s (covers env enforcement, mxc_to_http, markdown→HTML
  rendering incl. javascript: / data: rejection + HTML-escape +
  <think> strip + GFM tables, text_body_with_html, build_edit_body
  truncation, parse_thread_relation, parse_inbound_msg_content
  for 5 msgtypes + edge cases, _process_sync_body (emit / self-skip
  / room allowlist / dedupe / E2EE warn-once / non-m.room.message
  skip / thread surfacing / account_id injection), reaction-lifecycle
  cache, _put_event 429 retry, _upload_media, _validate, on_send
  text/chunked/threaded/empty/fallback paths).

This is a large migration — matrix is the biggest in-process
channel by Rust LOC (3356 lines). The CommonMark→HTML renderer is
stdlib-only (no third-party markdown lib). Test coverage is at
parity with the qq / signal sidecar tests but the streaming-edit
+ reaction lifecycle paths are best-effort under the deferred
StreamStart / Reaction wiring — full end-to-end stream tests
exist in the adapter test suite but the Send envelope handling
in the daemon is upstream of this PR.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a8a84d30a6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +717 to +720
self.homeserver_url = hs.rstrip("/")
self.user_id = user_id
self.access_token = token
self.allowed_rooms = _split_csv(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Advertise auth headers for Matrix media downloads

Inbound m.image/m.file payloads now carry /_matrix/client/v1/media/download/... URLs, which require Authorization: Bearer ... on modern homeservers, but this adapter never populates header_rules in its ready capabilities. The bridge calls fetch_headers_for(url) before downloading media, so Matrix attachments will now fail to download (typically 401) and image/file context ingestion regresses versus the removed in-process adapter that provided these headers.

Useful? React with 👍 / 👎.

Comment on lines +1256 to +1260
if content and not (isinstance(content, dict) and "Text" in content):
# Other content types: send the text fallback. Non-text
# variants (Voice, Audio, Video, Animation, Location,
# Poll, etc.) could be implemented inline but the
# majority of outbound sidecar traffic is text — Voice

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Preserve non-text outbound content instead of text fallback

This branch degrades every non-Text/Image/File ChannelContent variant into a plain text placeholder/JSON string, so Matrix no longer sends Voice, Audio, Video, Animation, Location, Interactive, etc. as native events. The removed Rust adapter handled these variants explicitly, so this introduces a functional regression for existing workflows that emit rich content.

Useful? React with 👍 / 👎.

houko added 2 commits May 21, 2026 01:27
… race

Self-review of #5368 found two regressions vs the in-process Rust
adapter:

1. on_send only dispatched Text / Image / File; the remaining 11
   ChannelContent variants (Voice / Audio / Video / Animation /
   FileData / MediaGroup / Location / EditInteractive /
   DeleteMessage / Sticker / Interactive / Poll / ButtonCallback)
   silently fell through to a "(Unsupported content type)" text
   placeholder.
2. Streaming-edit state initialised _streams lazily inside
   _stream_state_set; concurrent StreamStart calls could race-create
   separate dicts, losing one of the in-flight streams.

Both are now fixed:

- on_send dispatches every ChannelContent variant the Rust adapter
  did. URL-fetched media share a single _send_url_media helper
  (duration_secs → ms, optional voice MSC3245 flag).
- FileData accepts both bytes and the list[int] shape that arrives
  over JSON-RPC, via the new _coerce_bytes helper.
- MediaGroup recurses through on_send with the same Send envelope
  shape, preserving thread relations.
- _streams now initialised in __init__.

Test additions (23 cases) cover every new variant plus
_coerce_bytes. Shared FakeResp.read now optionally honours a size
argument and HdrShim.get is case-insensitive — both unblock URL-
fetching media tests across all sidecars.

Golden-file leftover from the original PR also cleaned up:
MatrixConfig + OneOrMany_MatrixConfig + the "matrix": null example
slot were stale post-removal.
Rust adapter overrode ``ChannelAdapter::fetch_headers_for`` to
return ``Authorization: Bearer <token>`` for URLs whose host
matched the homeserver. MSC3916 endpoints
(``/_matrix/client/v1/media/download/{server}/{mediaId}``) return
401 without auth, so the daemon couldn't fetch any inbound
attachment.

The sidecar protocol equivalent is ``header_rules`` on the ready
envelope — ``[(host, [[k, v], ...]), ...]`` — and the bridge's
``fetch_headers_for`` exact-matches host. Set it in ``__init__``
once the access token is known.

Host comes from ``urlparse(homeserver_url).hostname`` so a port in
the URL is dropped, matching the bridge's parser.
@houko
houko merged commit fa57f2d into main May 20, 2026
17 of 19 checks passed
@houko
houko deleted the feat/channels-matrix-sidecar branch May 20, 2026 16:35
houko added a commit that referenced this pull request May 20, 2026
…eader count

After #5368 migrated Matrix to a sidecar, `librefang_types::config::
ChannelsConfig` no longer has a `matrix` field. Three test cases in
`routes::skills::tests` were still round-tripping the appended /
removed TOML through `ChannelsConfig` and asserting on
`parsed.channels.matrix.len()`, so the test build failed with
`E0609: no field `matrix` on type `ChannelsConfig`` against current
main.

Replace the typed deserialization with a raw count of
`[[channels.matrix]]` headers in the rewritten file. The covered
behavior is the TOML-string mutation (`append_channel_instance`,
`remove_channel_instance` — both string-based helpers), so the
header count is the equivalent assertion: one array-of-tables
header per logical instance. The legacy-single-table-form check is
kept by also asserting absence of `[channels.matrix]`.

All 4 affected tests green. Bundles with the orphan-backoff fix
because both have the same root cause (#5368 fallout) and ship
together as the main-red recovery PR.
@github-actions github-actions Bot added area/docs Documentation and guides area/channels Messaging channel adapters area/kernel Core kernel (scheduling, RBAC, workflows) area/sdk JavaScript and Python SDKs labels May 20, 2026
houko added a commit that referenced this pull request May 20, 2026
Third #5368 fallout site: `config_reload::tests::test_channels_hot_reload`
built a `MatrixConfig` and assigned to `b.channels.matrix`, but the
Matrix sidecar migration removed both the field and the type. Switch
the test fixture to `WeChatConfig` — one of the remaining in-process
channels with a usable `Default` impl.

Bundled with the orphan-backoff + skills.rs fix to ship as a single
main-red recovery PR.
houko added a commit that referenced this pull request May 20, 2026
Third #5368 fallout site: `config_reload::tests::test_channels_hot_reload`
built a `MatrixConfig` and assigned to `b.channels.matrix`, but the
Matrix sidecar migration removed both the field and the type. Switch
the test fixture to `WeChatConfig` — one of the remaining in-process
channels with a usable `Default` impl.

Bundled with the orphan-backoff + skills.rs fix to ship as a single
main-red recovery PR.
houko added a commit that referenced this pull request May 20, 2026
Third #5368 fallout site: `config_reload::tests::test_channels_hot_reload`
built a `MatrixConfig` and assigned to `b.channels.matrix`, but the
Matrix sidecar migration removed both the field and the type. Switch
the test fixture to `WeChatConfig` — one of the remaining in-process
channels with a usable `Default` impl.

Bundled with the orphan-backoff + skills.rs fix to ship as a single
main-red recovery PR.
@github-actions github-actions Bot added the size/XL 1000+ lines changed label May 20, 2026
houko added a commit that referenced this pull request May 20, 2026
Fourth #5368 fallout site: librefang-types/src/config/mod.rs has six
test cases that exercised either MatrixConfig directly
(test_one_or_many_serialize_roundtrip, test_account_id_in_channel_configs)
or `b.channels.matrix` shape (test_channels_config_with_new_channels,
test_one_or_many_single_toml_table, test_one_or_many_array_of_tables,
test_one_or_many_empty_default). Matrix migrated to a sidecar in
#5368 so both the field and the type are gone.

These tests cover the OneOrMany helper, not Matrix specifically.
Swapped the fixture to WeChat — one of the remaining in-process
channels with a usable Default impl — and trimmed Matrix-only
fields (homeserver_url) from the TOML strings.

All 832 librefang-types lib tests green. Bundled into #5379 so a
single PR closes out the main-red recovery.
houko added a commit that referenced this pull request May 20, 2026
CI on the wecom-sidecar PR exposed three test failures that were
pre-existing on `main`, all rooted in past sidecar migrations that
left stale assertions behind. Same workspace, single follow-up
commit so the PR's CI goes green end-to-end.

1. `librefang-migrate::openclaw::tests::test_json5_full_migration`
   `librefang-migrate::openclaw::tests::test_secrets_migration`
   Both asserted MATRIX_ACCESS_TOKEN must land in secrets.env.
   Matrix migrated to a sidecar (#5368) — `[channels.matrix]` is
   now recorded as a SkippedItem instead of migrated, so the secret
   does NOT (and must not) land in secrets.env. Flip both
   assertions to negative (matching the IRC-removed pattern already
   in the same blocks) and drop the Secret-items floor in
   `test_secrets_migration` from `>=8` to `>=7` for the same reason.

2. `librefang-api::routes::channels::test_channel_status_tests::missing_required_env_returns_412`
   `credentials_present_no_target_returns_200`
   Both used WhatsApp as the witness. The QR-first redesign moved
   WHATSAPP_ACCESS_TOKEN into the advanced Business-API fallback
   block with `required: false`, so the 412 precondition check no
   longer fires on missing-env. Rotate the witness to Email
   (EMAIL_PASSWORD, `required: true`). Same test intent —
   "missing required env → 412, not silent 200 (#3507)" — just
   pointing at a still-required env var.

Reproduced + fixed locally:
- cargo test -p librefang-migrate --lib openclaw — 36/36 ok
- cargo test -p librefang-api --lib test_channel_status_tests — 3/3 ok
houko added a commit that referenced this pull request May 20, 2026
…bSocket-only)

BREAKING — Callback mode is removed. WeCom moves from a 2,497-line
in-process Rust adapter to a 480-line Python sidecar
(librefang.sidecar.adapters.wecom). The WebSocket bot path is
preserved verbatim with three improvements (req_id dedupe via the
shared SeenSet, heartbeat/send coexistence on one socket via a stdlib
queue.Queue, server-ACK errcode now visible in logs). The legacy
Callback mode (HTTP webhook + AES-CBC-256 inbound payload decryption
+ HMAC-SHA1 signature verification) is NOT ported — Python's stdlib
has no AES and the sidecar SDK is stdlib-only by policy across all
19 reference adapters. Operators on Callback mode must switch the
intelligent bot to WebSocket mode in the WeCom admin console (it
needs no public endpoint and is a strict superset of what Callback
could do in restricted-egress environments) or ship their own
callback-mode sidecar that brings its own AES dependency.

Tests: cd sdk/python && pytest tests/test_wecom_adapter.py — 62 passed
       cargo test --lib for the touched crates (types/channels/kernel/
       api/migrate) — 684 passed in librefang-api, 1 pre-existing fail
       (missing_required_env_returns_412 — WhatsApp ChannelMeta field
       is required: false; unrelated to wecom).

Drive-by fixes (CLAUDE.md "fix what you found"):
- librefang-types config tests rotated MatrixConfig witnesses to
  DingTalk / WeChat / Email + FeishuConfig (matrix sidecar migration
  #5368 left them broken on main).
- librefang-kernel config_reload test_channels_hot_reload rotated
  Matrix → DingTalk witness for the same reason.
- librefang-api routes/skills.rs upsert/append/update/remove tests
  rotated matrix → wechat witness (TOML round-trip behaviour is
  channel-incidental; any in-process channel works).
- librefang-migrate openclaw test_roundtrip bound `report` (was
  `let _ =`) so the matrix-skipped-sidecar assertion could compile.
- librefang-channels formatter.rs: removed dead helpers
  (strip_atx_heading / strip_blockquote_prefix / strip_task_list_prefix
  / is_fenced_code_marker / is_setext_heading_underline / is_table_divider
  / strip_inline_markdown) that had no caller after wecom_plain went
  away; kept default_output_format_for_channel("wecom") = Markdown
  since the sidecar emits msgtype: "markdown" frames.
- librefang-types: removed dead default_channel_initial_backoff_secs
  (sole caller was WeComConfig).
- librefang-api: FieldType::Select kept under #[allow(dead_code)] so a
  future in-process channel can opt back in without re-adding the variant.
houko added a commit that referenced this pull request May 20, 2026
CI on the wecom-sidecar PR exposed three test failures that were
pre-existing on `main`, all rooted in past sidecar migrations that
left stale assertions behind. Same workspace, single follow-up
commit so the PR's CI goes green end-to-end.

1. `librefang-migrate::openclaw::tests::test_json5_full_migration`
   `librefang-migrate::openclaw::tests::test_secrets_migration`
   Both asserted MATRIX_ACCESS_TOKEN must land in secrets.env.
   Matrix migrated to a sidecar (#5368) — `[channels.matrix]` is
   now recorded as a SkippedItem instead of migrated, so the secret
   does NOT (and must not) land in secrets.env. Flip both
   assertions to negative (matching the IRC-removed pattern already
   in the same blocks) and drop the Secret-items floor in
   `test_secrets_migration` from `>=8` to `>=7` for the same reason.

2. `librefang-api::routes::channels::test_channel_status_tests::missing_required_env_returns_412`
   `credentials_present_no_target_returns_200`
   Both used WhatsApp as the witness. The QR-first redesign moved
   WHATSAPP_ACCESS_TOKEN into the advanced Business-API fallback
   block with `required: false`, so the 412 precondition check no
   longer fires on missing-env. Rotate the witness to Email
   (EMAIL_PASSWORD, `required: true`). Same test intent —
   "missing required env → 412, not silent 200 (#3507)" — just
   pointing at a still-required env var.

Reproduced + fixed locally:
- cargo test -p librefang-migrate --lib openclaw — 36/36 ok
- cargo test -p librefang-api --lib test_channel_status_tests — 3/3 ok
houko added a commit that referenced this pull request May 20, 2026
…bSocket-only) (#5392)

* feat(channels)!: migrate wecom from in-process adapter to sidecar (WebSocket-only)

BREAKING — Callback mode is removed. WeCom moves from a 2,497-line
in-process Rust adapter to a 480-line Python sidecar
(librefang.sidecar.adapters.wecom). The WebSocket bot path is
preserved verbatim with three improvements (req_id dedupe via the
shared SeenSet, heartbeat/send coexistence on one socket via a stdlib
queue.Queue, server-ACK errcode now visible in logs). The legacy
Callback mode (HTTP webhook + AES-CBC-256 inbound payload decryption
+ HMAC-SHA1 signature verification) is NOT ported — Python's stdlib
has no AES and the sidecar SDK is stdlib-only by policy across all
19 reference adapters. Operators on Callback mode must switch the
intelligent bot to WebSocket mode in the WeCom admin console (it
needs no public endpoint and is a strict superset of what Callback
could do in restricted-egress environments) or ship their own
callback-mode sidecar that brings its own AES dependency.

Tests: cd sdk/python && pytest tests/test_wecom_adapter.py — 62 passed
       cargo test --lib for the touched crates (types/channels/kernel/
       api/migrate) — 684 passed in librefang-api, 1 pre-existing fail
       (missing_required_env_returns_412 — WhatsApp ChannelMeta field
       is required: false; unrelated to wecom).

Drive-by fixes (CLAUDE.md "fix what you found"):
- librefang-types config tests rotated MatrixConfig witnesses to
  DingTalk / WeChat / Email + FeishuConfig (matrix sidecar migration
  #5368 left them broken on main).
- librefang-kernel config_reload test_channels_hot_reload rotated
  Matrix → DingTalk witness for the same reason.
- librefang-api routes/skills.rs upsert/append/update/remove tests
  rotated matrix → wechat witness (TOML round-trip behaviour is
  channel-incidental; any in-process channel works).
- librefang-migrate openclaw test_roundtrip bound `report` (was
  `let _ =`) so the matrix-skipped-sidecar assertion could compile.
- librefang-channels formatter.rs: removed dead helpers
  (strip_atx_heading / strip_blockquote_prefix / strip_task_list_prefix
  / is_fenced_code_marker / is_setext_heading_underline / is_table_divider
  / strip_inline_markdown) that had no caller after wecom_plain went
  away; kept default_output_format_for_channel("wecom") = Markdown
  since the sidecar emits msgtype: "markdown" frames.
- librefang-types: removed dead default_channel_initial_backoff_secs
  (sole caller was WeComConfig).
- librefang-api: FieldType::Select kept under #[allow(dead_code)] so a
  future in-process channel can opt back in without re-adding the variant.

* fix(tests): repair 3 main-broken tests surfaced by wecom-migration CI

CI on the wecom-sidecar PR exposed three test failures that were
pre-existing on `main`, all rooted in past sidecar migrations that
left stale assertions behind. Same workspace, single follow-up
commit so the PR's CI goes green end-to-end.

1. `librefang-migrate::openclaw::tests::test_json5_full_migration`
   `librefang-migrate::openclaw::tests::test_secrets_migration`
   Both asserted MATRIX_ACCESS_TOKEN must land in secrets.env.
   Matrix migrated to a sidecar (#5368) — `[channels.matrix]` is
   now recorded as a SkippedItem instead of migrated, so the secret
   does NOT (and must not) land in secrets.env. Flip both
   assertions to negative (matching the IRC-removed pattern already
   in the same blocks) and drop the Secret-items floor in
   `test_secrets_migration` from `>=8` to `>=7` for the same reason.

2. `librefang-api::routes::channels::test_channel_status_tests::missing_required_env_returns_412`
   `credentials_present_no_target_returns_200`
   Both used WhatsApp as the witness. The QR-first redesign moved
   WHATSAPP_ACCESS_TOKEN into the advanced Business-API fallback
   block with `required: false`, so the 412 precondition check no
   longer fires on missing-env. Rotate the witness to Email
   (EMAIL_PASSWORD, `required: true`). Same test intent —
   "missing required env → 412, not silent 200 (#3507)" — just
   pointing at a still-required env var.

Reproduced + fixed locally:
- cargo test -p librefang-migrate --lib openclaw — 36/36 ok
- cargo test -p librefang-api --lib test_channel_status_tests — 3/3 ok

* fix(channels/wecom): plumb shutdown event + drop dead MAX_BACKOFF_SECS import

Self-review surfaced two issues:

1. **producer thread couldn't be stopped on Shutdown** —
   `_producer_blocking` was `while True` with no shutdown signal,
   running inside `loop.run_in_executor`. On runtime shutdown the
   asyncio Future gets cancelled but Python threads can't be
   cancelled from the outside, so the executor thread would keep
   reconnecting (and the backoff `time.sleep` would hold the
   process open) until the next socket read tick happened to
   notice. Add a `threading.Event` set by `on_shutdown`, check it
   on both the inner session loop and the outer reconnect loop,
   and use `Event.wait(backoff)` instead of `time.sleep(backoff)`
   so backoff is interruptible. Matches the matrix sidecar's
   pattern.

2. **`MAX_BACKOFF_SECS` imported but unused** — wecom uses its own
   `WECOM_MAX_BACKOFF_SECS = 30.0` because the Rust adapter capped
   at 30 s (vs the SDK default 60 s). The shared constant was a
   leftover from an earlier draft. Dropped, pyflakes-clean.
houko added a commit that referenced this pull request May 20, 2026
…decar #5368 (#5407)

The matrix in-process adapter was the sole consumer of pulldown-cmark
(used to render CommonMark → HTML for the Matrix `m.room.message`
`formatted_body` field). #5368 deleted `crates/librefang-channels/src/
matrix.rs` and moved that rendering into the Python sidecar
(`librefang.sidecar.adapters.matrix`, which ships its own stdlib
subset renderer), but the workspace dep declaration in the root
`Cargo.toml` was left behind. Cargo.lock already lost the package
when the consuming code went away — this commit just removes the
now-orphan workspace dep line.

Also recovers the bullet from #5379 (which targeted the same cleanup
but is now superseded by #5392's drive-by fixes for the test-rotation
side and this commit for the deps side).
houko added a commit that referenced this pull request May 22, 2026
…fix (#5382)

* fix(types): remove orphan default_channel_initial_backoff_secs helper

#5368 (Matrix in-process → sidecar) removed the last consumer of
`default_channel_initial_backoff_secs` (the 1-second variant). The
matching Signal consumer was already removed earlier when Signal
migrated to a sidecar. Both sidecar channels control their own
backoff via env vars now.

Workspace lints have `warnings = "deny"`, so an orphaned helper
trips dead_code and turns main red. This is the cause of the
post-#5368 main breakage; affected every PR built against `origin/main`.

The sibling helpers `default_channel_max_backoff_secs` and
`default_channel_initial_backoff_2s` still have live consumers
(WeChat, QQ, Feishu …) and stay.

cargo check -p librefang-types --lib clean.

* fix(api/skills): replace ChannelsConfig::matrix round-trip with raw header count

After #5368 migrated Matrix to a sidecar, `librefang_types::config::
ChannelsConfig` no longer has a `matrix` field. Three test cases in
`routes::skills::tests` were still round-tripping the appended /
removed TOML through `ChannelsConfig` and asserting on
`parsed.channels.matrix.len()`, so the test build failed with
`E0609: no field `matrix` on type `ChannelsConfig`` against current
main.

Replace the typed deserialization with a raw count of
`[[channels.matrix]]` headers in the rewritten file. The covered
behavior is the TOML-string mutation (`append_channel_instance`,
`remove_channel_instance` — both string-based helpers), so the
header count is the equivalent assertion: one array-of-tables
header per logical instance. The legacy-single-table-form check is
kept by also asserting absence of `[channels.matrix]`.

All 4 affected tests green. Bundles with the orphan-backoff fix
because both have the same root cause (#5368 fallout) and ship
together as the main-red recovery PR.

* fix(api/auth): split /api/auth/login allowlist into exact + slash-prefix (closes #5381)

audit: login-prefix-match (Severity: Medium).

The public-route allowlist had `PublicRoute::prefix_get("/api/auth/login")`
without a trailing slash. The prefix match (`path.starts_with(route.path)`)
treated every GET whose path started with `/api/auth/login` as public —
including hypothetical siblings `/api/auth/login-status`,
`/api/auth/loginhack`, `/api/auth/login-debug`, etc.

There is no live exposure today — server.rs only registers exact
`/api/auth/login` and `/api/auth/login/{provider}`. But every other
prefix entry in the slice (`/api/budget/agents/`, `/api/hands/`,
`/dashboard/assets/`, `/api/providers/github-copilot/oauth/`) ends
with a slash. The inconsistency was a latent foot-gun: whoever added
a sibling GET later would silently inherit unauthenticated access.

Fix mirrors the `/api/budget/agents` pattern:

  PublicRoute::exact_get("/api/auth/login"),
  PublicRoute::prefix_get("/api/auth/login/"),

Both `/api/auth/login` and `/api/auth/login/{provider}` stay public;
any other prefix sibling now requires auth.

Two regression cases added to
`auth_public_allowlist.rs::PUBLIC_ROUTE_EXPECTATIONS`:
  - `/api/auth/login-status` → Expect::Authed
  - `/api/auth/loginhack`    → Expect::Authed
plus a positive `Expect::AlwaysPublic` row for
`/api/auth/login/google` to lock in the provider suffix. All 8
allowlist test cases pass; clippy clean.

Includes two prerequisite commits recovering main from #5368 fallout
(`default_channel_initial_backoff_secs` dead_code + skills.rs typed
round-trip of removed `ChannelsConfig::matrix`); shipping the same
fix in #5379. After #5379 lands those two commits dedupe on rebase.

* fix(kernel/config_reload): swap matrix→wechat in hot-reload test

Third #5368 fallout site: `config_reload::tests::test_channels_hot_reload`
built a `MatrixConfig` and assigned to `b.channels.matrix`, but the
Matrix sidecar migration removed both the field and the type. Switch
the test fixture to `WeChatConfig` — one of the remaining in-process
channels with a usable `Default` impl.

Bundled with the orphan-backoff + skills.rs fix to ship as a single
main-red recovery PR.

* chore(codegen): auto-regenerate openapi.json + sdk + schema baselines [skip ci]

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
houko added a commit that referenced this pull request May 22, 2026
* fix(types): remove orphan default_channel_initial_backoff_secs helper

#5368 (Matrix in-process → sidecar) removed the last consumer of
`default_channel_initial_backoff_secs` (the 1-second variant). The
matching Signal consumer was already removed earlier when Signal
migrated to a sidecar. Both sidecar channels control their own
backoff via env vars now.

Workspace lints have `warnings = "deny"`, so an orphaned helper
trips dead_code and turns main red. This is the cause of the
post-#5368 main breakage; affected every PR built against `origin/main`.

The sibling helpers `default_channel_max_backoff_secs` and
`default_channel_initial_backoff_2s` still have live consumers
(WeChat, QQ, Feishu …) and stay.

cargo check -p librefang-types --lib clean.

* fix(api/skills): replace ChannelsConfig::matrix round-trip with raw header count

After #5368 migrated Matrix to a sidecar, `librefang_types::config::
ChannelsConfig` no longer has a `matrix` field. Three test cases in
`routes::skills::tests` were still round-tripping the appended /
removed TOML through `ChannelsConfig` and asserting on
`parsed.channels.matrix.len()`, so the test build failed with
`E0609: no field `matrix` on type `ChannelsConfig`` against current
main.

Replace the typed deserialization with a raw count of
`[[channels.matrix]]` headers in the rewritten file. The covered
behavior is the TOML-string mutation (`append_channel_instance`,
`remove_channel_instance` — both string-based helpers), so the
header count is the equivalent assertion: one array-of-tables
header per logical instance. The legacy-single-table-form check is
kept by also asserting absence of `[channels.matrix]`.

All 4 affected tests green. Bundles with the orphan-backoff fix
because both have the same root cause (#5368 fallout) and ship
together as the main-red recovery PR.

* fix(api/auth): always emit Secure on logout cookie clear (closes #5383)

audit: logout-no-secure-cookie (Severity: Medium).

`session_cookie_attrs` derives `Secure` from the request transport
(HTTPS / X-Forwarded-Proto). On a logout request that lands over
plain HTTP, the resulting clear cookie has NO `Secure` flag — and
the browser refuses to overwrite a live `Secure`-flagged cookie
with one that lacks it (RFC 6265bis §5.6). Result: server-side
session is invalidated, but the browser keeps presenting the
session cookie until the next failed auth.

Split out `session_cookie_clear_attrs()` that unconditionally
includes `Secure`. Modern browsers (Chromium, Firefox, Safari
16.4+) accept `Secure` on a `Max-Age=0` Set-Cookie response
regardless of transport. The login-side helper keeps its
transport-dependent behaviour so local-HTTP dev still works.

New tests in `server::session_cookie_attrs_tests`:
- `clear_attrs_always_contain_secure` — asserts the clear path
  always contains `Secure`, plus the other expected attributes.
- `login_attrs_remain_transport_dependent` — regression for the
  unchanged login behaviour.

Both green; clippy clean.

Includes the two prerequisite #5368-fallout commits (orphan
default_channel_initial_backoff_secs + skills.rs typed round-trip);
also in #5379. After #5379 lands they dedupe on rebase.

* fix(kernel/config_reload): swap matrix→wechat in hot-reload test

Third #5368 fallout site: `config_reload::tests::test_channels_hot_reload`
built a `MatrixConfig` and assigned to `b.channels.matrix`, but the
Matrix sidecar migration removed both the field and the type. Switch
the test fixture to `WeChatConfig` — one of the remaining in-process
channels with a usable `Default` impl.

Bundled with the orphan-backoff + skills.rs fix to ship as a single
main-red recovery PR.

* chore(codegen): auto-regenerate openapi.json + sdk + schema baselines [skip ci]

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
houko added a commit that referenced this pull request May 22, 2026
* fix(types): remove orphan default_channel_initial_backoff_secs helper

#5368 (Matrix in-process → sidecar) removed the last consumer of
`default_channel_initial_backoff_secs` (the 1-second variant). The
matching Signal consumer was already removed earlier when Signal
migrated to a sidecar. Both sidecar channels control their own
backoff via env vars now.

Workspace lints have `warnings = "deny"`, so an orphaned helper
trips dead_code and turns main red. This is the cause of the
post-#5368 main breakage; affected every PR built against `origin/main`.

The sibling helpers `default_channel_max_backoff_secs` and
`default_channel_initial_backoff_2s` still have live consumers
(WeChat, QQ, Feishu …) and stay.

cargo check -p librefang-types --lib clean.

* fix(api/skills): replace ChannelsConfig::matrix round-trip with raw header count

After #5368 migrated Matrix to a sidecar, `librefang_types::config::
ChannelsConfig` no longer has a `matrix` field. Three test cases in
`routes::skills::tests` were still round-tripping the appended /
removed TOML through `ChannelsConfig` and asserting on
`parsed.channels.matrix.len()`, so the test build failed with
`E0609: no field `matrix` on type `ChannelsConfig`` against current
main.

Replace the typed deserialization with a raw count of
`[[channels.matrix]]` headers in the rewritten file. The covered
behavior is the TOML-string mutation (`append_channel_instance`,
`remove_channel_instance` — both string-based helpers), so the
header count is the equivalent assertion: one array-of-tables
header per logical instance. The legacy-single-table-form check is
kept by also asserting absence of `[channels.matrix]`.

All 4 affected tests green. Bundles with the orphan-backoff fix
because both have the same root cause (#5368 fallout) and ship
together as the main-red recovery PR.

* fix(kernel/config_reload): swap matrix→wechat in hot-reload test

Third #5368 fallout site: `config_reload::tests::test_channels_hot_reload`
built a `MatrixConfig` and assigned to `b.channels.matrix`, but the
Matrix sidecar migration removed both the field and the type. Switch
the test fixture to `WeChatConfig` — one of the remaining in-process
channels with a usable `Default` impl.

Bundled with the orphan-backoff + skills.rs fix to ship as a single
main-red recovery PR.

* fix(kernel/agent_execution): anchor [SILENT] cron marker to message prefix (closes #5385)

audit: silent-marker-substring-match (Severity: Low).

`execute_llm_agent` had two `is_internal_cron && message.contains("[SILENT]")`
sites — the LLM-side stripping at line 890 and the persistence-skip at
line 1114. The `is_internal_cron` gate stopped a plain user chat from
triggering the path, but a cron prompt template that interpolated
runtime data (channel content, tool output, user-supplied variables)
could still hit the substring match if the payload happened to contain
the literal `[SILENT]` token anywhere in the body. Result: the
assistant turn gets silently dropped from session history even though
the operator never opted in.

Fix: extract a `strip_silent_cron_marker(message, is_internal_cron)
-> (String, bool)` helper that requires the marker to be the first
non-whitespace token (after `trim_start`). Both sites consume the
shared `is_silent_cron` flag so the LLM-side strip and the
persistence-skip can never disagree (a divergence would suppress
history for a turn the LLM saw the unstripped version of).

The helper also moves from `replace("[SILENT]", "")` to a single
`strip_prefix("[SILENT]")`, so embedded later occurrences of the
marker survive verbatim — that's the operator's intent: a single
marker at the start opts the run into silent mode, additional
`[SILENT]` substrings deeper in the prompt are just text.

6 new tests under `silent_marker_tests` cover: non-cron call,
leading marker, leading-whitespace + marker, marker-only message
(empty-fallback), embedded mid-message marker (regression for the
substring bug), and embedded second marker preservation.

cargo clippy -p librefang-kernel --all-targets clean.

Includes the three prerequisite #5368-fallout commits (orphan
default_channel_initial_backoff_secs, skills.rs typed round-trip,
config_reload.rs matrix→wechat). Also shipping in #5379;
deduped on rebase once #5379 lands.

* chore(codegen): auto-regenerate openapi.json + sdk + schema baselines [skip ci]

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
houko added a commit that referenced this pull request May 22, 2026
…ion (#5388)

* fix(types): remove orphan default_channel_initial_backoff_secs helper

#5368 (Matrix in-process → sidecar) removed the last consumer of
`default_channel_initial_backoff_secs` (the 1-second variant). The
matching Signal consumer was already removed earlier when Signal
migrated to a sidecar. Both sidecar channels control their own
backoff via env vars now.

Workspace lints have `warnings = "deny"`, so an orphaned helper
trips dead_code and turns main red. This is the cause of the
post-#5368 main breakage; affected every PR built against `origin/main`.

The sibling helpers `default_channel_max_backoff_secs` and
`default_channel_initial_backoff_2s` still have live consumers
(WeChat, QQ, Feishu …) and stay.

cargo check -p librefang-types --lib clean.

* fix(api/skills): replace ChannelsConfig::matrix round-trip with raw header count

After #5368 migrated Matrix to a sidecar, `librefang_types::config::
ChannelsConfig` no longer has a `matrix` field. Three test cases in
`routes::skills::tests` were still round-tripping the appended /
removed TOML through `ChannelsConfig` and asserting on
`parsed.channels.matrix.len()`, so the test build failed with
`E0609: no field `matrix` on type `ChannelsConfig`` against current
main.

Replace the typed deserialization with a raw count of
`[[channels.matrix]]` headers in the rewritten file. The covered
behavior is the TOML-string mutation (`append_channel_instance`,
`remove_channel_instance` — both string-based helpers), so the
header count is the equivalent assertion: one array-of-tables
header per logical instance. The legacy-single-table-form check is
kept by also asserting absence of `[channels.matrix]`.

All 4 affected tests green. Bundles with the orphan-backoff fix
because both have the same root cause (#5368 fallout) and ship
together as the main-red recovery PR.

* fix(kernel/config_reload): swap matrix→wechat in hot-reload test

Third #5368 fallout site: `config_reload::tests::test_channels_hot_reload`
built a `MatrixConfig` and assigned to `b.channels.matrix`, but the
Matrix sidecar migration removed both the field and the type. Switch
the test fixture to `WeChatConfig` — one of the remaining in-process
channels with a usable `Default` impl.

Bundled with the orphan-backoff + skills.rs fix to ship as a single
main-red recovery PR.

* fix(api): clamp listing endpoints — no more "limit=None → full collection" (closes #5387)

audit: agent-list-limit-none-unbounded (Severity: Medium).

Two listing surfaces silently returned the entire unpaginated
collection when the caller omitted `limit=`:

  * `GET /api/agents` (routes/agents.rs::list_agents) —
    `params.limit.map(|l| l.min(500))` clamped only when `Some`,
    so `None` skipped the `take()` branch and serialised every
    `AgentEntry`. A multi-thousand-agent deployment turned this
    into a memory + JSON-serialisation DoS sink.

  * `PaginationQuery::paginate` (types.rs:703) — short-circuit at
    line 706 (`if self.offset.is_none() && self.limit.is_none()`)
    returned `(items, total, 0, None)`. Live callers
    `routes::skills::list_skills` and `routes::network::peers`
    inherited the same DoS lever.

Fix:

  * `list_agents`: introduce DEFAULT_AGENT_LIST_LIMIT = 500 and
    MAX_AGENT_LIST_LIMIT = 500 (the historical ceiling), and use
    `limit.unwrap_or(DEFAULT).min(MAX)`. Response envelope now
    always reports `Some(limit)`.
  * `PaginationQuery::paginate`: drop the both-unset short-circuit;
    always fall back to PAGINATION_MAX_LIMIT (= 100). The cap is a
    ceiling, not a floor, so a 10-row collection still comes back in
    full — only the > MAX cases truncate.

Regression tests:

  * `routes::agents::tests::agent_list_limit_clamps_at_max_when_caller_omits_limit`
    — exercises the clamp helper directly (clippy const-folds away
    literal `None`/`Some(usize::MAX)` if you inline them, so the test
    drives the helper through opaque inputs).
  * `types::tests::pagination_unspecified_falls_back_to_max_limit_not_full_collection`
    — replaces the previous `_returns_full_collection` test that
    encoded the unbounded behaviour.
  * `types::tests::pagination_unspecified_truncates_large_collection_at_max_limit`
    — exercises the actual DoS scenario (collection > MAX).

cargo clippy -p librefang-api --all-targets clean.

Includes the three prerequisite #5368-fallout commits (orphan
default_channel_initial_backoff_secs, skills.rs typed round-trip,
config_reload.rs matrix→wechat); also shipping in #5379. After
#5379 lands they dedupe on rebase.

* test(api): update list-paginated test for finite default limit

The DoS fix in this PR makes `GET /api/agents` always report a finite
server-applied limit (DEFAULT_AGENT_LIST_LIMIT = 500) instead of null when
the caller omits ?limit=. The api_integration_test assertion still expected
null and failed; align it with the new envelope contract already pinned by
the types.rs paginate unit tests.

* chore(codegen): auto-regenerate openapi.json + sdk + schema baselines [skip ci]

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
houko added a commit that referenced this pull request May 22, 2026
* fix(types): remove orphan default_channel_initial_backoff_secs helper

#5368 (Matrix in-process → sidecar) removed the last consumer of
`default_channel_initial_backoff_secs` (the 1-second variant). The
matching Signal consumer was already removed earlier when Signal
migrated to a sidecar. Both sidecar channels control their own
backoff via env vars now.

Workspace lints have `warnings = "deny"`, so an orphaned helper
trips dead_code and turns main red. This is the cause of the
post-#5368 main breakage; affected every PR built against `origin/main`.

The sibling helpers `default_channel_max_backoff_secs` and
`default_channel_initial_backoff_2s` still have live consumers
(WeChat, QQ, Feishu …) and stay.

cargo check -p librefang-types --lib clean.

* fix(api/skills): replace ChannelsConfig::matrix round-trip with raw header count

After #5368 migrated Matrix to a sidecar, `librefang_types::config::
ChannelsConfig` no longer has a `matrix` field. Three test cases in
`routes::skills::tests` were still round-tripping the appended /
removed TOML through `ChannelsConfig` and asserting on
`parsed.channels.matrix.len()`, so the test build failed with
`E0609: no field `matrix` on type `ChannelsConfig`` against current
main.

Replace the typed deserialization with a raw count of
`[[channels.matrix]]` headers in the rewritten file. The covered
behavior is the TOML-string mutation (`append_channel_instance`,
`remove_channel_instance` — both string-based helpers), so the
header count is the equivalent assertion: one array-of-tables
header per logical instance. The legacy-single-table-form check is
kept by also asserting absence of `[channels.matrix]`.

All 4 affected tests green. Bundles with the orphan-backoff fix
because both have the same root cause (#5368 fallout) and ship
together as the main-red recovery PR.

* fix(kernel/config_reload): swap matrix→wechat in hot-reload test

Third #5368 fallout site: `config_reload::tests::test_channels_hot_reload`
built a `MatrixConfig` and assigned to `b.channels.matrix`, but the
Matrix sidecar migration removed both the field and the type. Switch
the test fixture to `WeChatConfig` — one of the remaining in-process
channels with a usable `Default` impl.

Bundled with the orphan-backoff + skills.rs fix to ship as a single
main-red recovery PR.

* fix(types/config): swap matrix→wechat in OneOrMany / channels tests

Fourth #5368 fallout site: librefang-types/src/config/mod.rs has six
test cases that exercised either MatrixConfig directly
(test_one_or_many_serialize_roundtrip, test_account_id_in_channel_configs)
or `b.channels.matrix` shape (test_channels_config_with_new_channels,
test_one_or_many_single_toml_table, test_one_or_many_array_of_tables,
test_one_or_many_empty_default). Matrix migrated to a sidecar in
#5368 so both the field and the type are gone.

These tests cover the OneOrMany helper, not Matrix specifically.
Swapped the fixture to WeChat — one of the remaining in-process
channels with a usable Default impl — and trimmed Matrix-only
fields (homeserver_url) from the TOML strings.

All 832 librefang-types lib tests green. Bundled into #5379 so a
single PR closes out the main-red recovery.

* fix(types/oauth): hand-write Debug to redact OAuthTokens secrets (closes #5394)

audit: oauth-tokens-derive-debug-serialize (Severity: High).

`OAuthTokens` derived `Debug` with the cleartext `access_token` and
`refresh_token` fields wide open. Every one of the following caller
patterns leaked the secret into logs / trace / panic backtrace:

  - `tracing::debug!(?tokens, …)` / `error!(?tokens, …)`
  - `format!("{:?}", tokens)` and panic messages including the type
  - `serde_json::to_string` flowing through snapshots / dashboards
  - cross-process trace dumps

The `access_token_zeroizing()` / `refresh_token_zeroizing()` helpers
already exist but only protect the call sites that remember to use
them — there was no type-level enforcement.

Fix: replace `#[derive(Debug)]` with a hand-written impl that emits
`OAuthTokens { access_token: <redacted len=N hint=****XXXX>,
refresh_token: Some(<redacted …>) | None, token_type, expires_in,
scope }`. Pattern mirrors `PooledCredential::Debug` in
`librefang-llm-drivers::credential_pool` (the canonical example).
Length + last-4 hint stays in the log so operators can correlate
"same token as last time" without exposing the secret.

The `redact()` helper:
  * len 0       → "<redacted len=0>"
  * len 1..=4   → "<redacted len=N>" (never reveals any chars of
                   a very short secret)
  * len 5..=7   → "<redacted len=N hint=****<tail-N-4>>"
                   (linear hint window so 5-char tokens still show
                    1 char of fingerprint)
  * len 8+      → "<redacted len=N hint=****<last-4>>"

Uses char count not byte count, so unicode tokens render
consistently (test
`redact_uses_char_count_not_byte_count_for_unicode_secrets`).

Six new tests in `oauth::tests`:
  - debug_redacts_access_and_refresh_tokens (the audit-named bug)
  - debug_redacts_none_refresh_token_without_panic
  - redact_hint_handles_short_secrets_without_exposing_full_value
  - redact_uses_char_count_not_byte_count_for_unicode_secrets
plus the existing 5 oauth tests still green.

`cargo clippy -p librefang-types --all-targets` clean.

Out of scope (deferred per audit doc): switching to `SecretString` /
`Zeroizing<String>` and splitting `OAuthTokensWire` from internal
`OAuthTokens`. Both are larger surgeries (every call site of `OAuthTokens`
needs an update); the Debug redaction closes the most common leak path
today and is the change requested by the audit's first
recommendation.

Includes four prerequisite #5368-fallout commits (orphan helper +
three matrix→wechat test swaps); also shipping in #5379. They
dedupe on rebase once #5379 lands.

* chore(codegen): auto-regenerate openapi.json + sdk + schema baselines [skip ci]

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
houko added a commit that referenced this pull request May 22, 2026
* fix(types): remove orphan default_channel_initial_backoff_secs helper

#5368 (Matrix in-process → sidecar) removed the last consumer of
`default_channel_initial_backoff_secs` (the 1-second variant). The
matching Signal consumer was already removed earlier when Signal
migrated to a sidecar. Both sidecar channels control their own
backoff via env vars now.

Workspace lints have `warnings = "deny"`, so an orphaned helper
trips dead_code and turns main red. This is the cause of the
post-#5368 main breakage; affected every PR built against `origin/main`.

The sibling helpers `default_channel_max_backoff_secs` and
`default_channel_initial_backoff_2s` still have live consumers
(WeChat, QQ, Feishu …) and stay.

cargo check -p librefang-types --lib clean.

* fix(api/skills): replace ChannelsConfig::matrix round-trip with raw header count

After #5368 migrated Matrix to a sidecar, `librefang_types::config::
ChannelsConfig` no longer has a `matrix` field. Three test cases in
`routes::skills::tests` were still round-tripping the appended /
removed TOML through `ChannelsConfig` and asserting on
`parsed.channels.matrix.len()`, so the test build failed with
`E0609: no field `matrix` on type `ChannelsConfig`` against current
main.

Replace the typed deserialization with a raw count of
`[[channels.matrix]]` headers in the rewritten file. The covered
behavior is the TOML-string mutation (`append_channel_instance`,
`remove_channel_instance` — both string-based helpers), so the
header count is the equivalent assertion: one array-of-tables
header per logical instance. The legacy-single-table-form check is
kept by also asserting absence of `[channels.matrix]`.

All 4 affected tests green. Bundles with the orphan-backoff fix
because both have the same root cause (#5368 fallout) and ship
together as the main-red recovery PR.

* fix(kernel/config_reload): swap matrix→wechat in hot-reload test

Third #5368 fallout site: `config_reload::tests::test_channels_hot_reload`
built a `MatrixConfig` and assigned to `b.channels.matrix`, but the
Matrix sidecar migration removed both the field and the type. Switch
the test fixture to `WeChatConfig` — one of the remaining in-process
channels with a usable `Default` impl.

Bundled with the orphan-backoff + skills.rs fix to ship as a single
main-red recovery PR.

* fix(types/config): swap matrix→wechat in OneOrMany / channels tests

Fourth #5368 fallout site: librefang-types/src/config/mod.rs has six
test cases that exercised either MatrixConfig directly
(test_one_or_many_serialize_roundtrip, test_account_id_in_channel_configs)
or `b.channels.matrix` shape (test_channels_config_with_new_channels,
test_one_or_many_single_toml_table, test_one_or_many_array_of_tables,
test_one_or_many_empty_default). Matrix migrated to a sidecar in
#5368 so both the field and the type are gone.

These tests cover the OneOrMany helper, not Matrix specifically.
Swapped the fixture to WeChat — one of the remaining in-process
channels with a usable Default impl — and trimmed Matrix-only
fields (homeserver_url) from the TOML strings.

All 832 librefang-types lib tests green. Bundled into #5379 so a
single PR closes out the main-red recovery.

* fix(wire): warn on serde(other) Unknown variants with raw tag (closes #5396)

audit: wire-message-other-variant-silent (Severity: Medium).

Four `#[serde(other)]` arms in librefang-wire/src/message.rs
(WireMessageKind::Unknown, WireRequest::Unknown,
WireResponse::Unknown, WireNotification::Unknown) silently dropped
every unrecognised variant. Two of them logged at `debug!` level —
default-off in production — and the other two had no log at all.

A misbehaving peer could flood `{"type":"garbage", …}` and the
receiver would silently absorb every message: no log line, no
counter, no metric. Combined with `#[serde(default)] nonce: String`
/ `auth_hmac: String` at message.rs:54-58, a peer that sends an
unknown handshake variant bypasses structural checks too.

Fix at the receive boundary (where the body bytes are still local):

  * New `classify_unknown(body, msg) -> Option<UnknownTag>` in
    message.rs. When the decoded message landed in any of the four
    serde(other) arms, peek the raw JSON body for the relevant
    field (envelope `type`, request/response `method`,
    notification `event`) and return the raw tag string the peer
    actually sent. Returns None on known variants so receivers
    only pay the json::Value re-parse on the unknown branch.

  * New `read_message_observed(reader, peer_node_id)` and
    `read_message_authenticated_observed(reader, key, peer_node_id)`
    wrap the existing helpers, call `classify_unknown` after a
    successful decode, and emit a structured `warn!` on
    `target: "wire::compat"` with peer, msg_id, level
    (`envelope.type` / `request.method` / `response.method` /
    `notification.event`) and `raw_tag`. The original
    `read_message` / `read_message_authenticated` keep their pub
    signatures (delegate through the observed variants with
    `peer_node_id = "<pre-handshake>"`) so external callers are
    unaffected.

  * `connection_loop` switches to the observed variants — that's
    the post-handshake hot path where the actual `peer_node_id` is
    known. The per-site `debug!` lines for `Unknown` variants stay
    (now framed as "no dispatch") for trace-level correlation, but
    the visibility-load is on the upstream `warn!`.

Six new tests in `message::tests` cover classify_unknown for each
of the four levels, plus the happy path (returns None for known
variants) and the malformed-tag fallback to `"<missing>"`. All 63
librefang-wire tests green; `cargo clippy -p librefang-wire
--all-targets` clean.

Out of scope (deferred per audit doc): per-peer counter +
threshold-disconnect. That's stateful infrastructure that needs
care to not nuke compatible peers running newer-protocol
features — separate PR. Today the operator-visible warn is the
main missing visibility, and that's covered.

Includes four prerequisite #5368-fallout commits (orphan helper +
three matrix→wechat test swaps); also shipping in #5379. They
dedupe on rebase once #5379 lands.

* chore(codegen): auto-regenerate openapi.json + sdk + schema baselines [skip ci]

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
houko added a commit that referenced this pull request May 22, 2026
…dit_entries(agent_id, timestamp) indexes (#5399)

* fix(types): remove orphan default_channel_initial_backoff_secs helper

#5368 (Matrix in-process → sidecar) removed the last consumer of
`default_channel_initial_backoff_secs` (the 1-second variant). The
matching Signal consumer was already removed earlier when Signal
migrated to a sidecar. Both sidecar channels control their own
backoff via env vars now.

Workspace lints have `warnings = "deny"`, so an orphaned helper
trips dead_code and turns main red. This is the cause of the
post-#5368 main breakage; affected every PR built against `origin/main`.

The sibling helpers `default_channel_max_backoff_secs` and
`default_channel_initial_backoff_2s` still have live consumers
(WeChat, QQ, Feishu …) and stay.

cargo check -p librefang-types --lib clean.

* fix(api/skills): replace ChannelsConfig::matrix round-trip with raw header count

After #5368 migrated Matrix to a sidecar, `librefang_types::config::
ChannelsConfig` no longer has a `matrix` field. Three test cases in
`routes::skills::tests` were still round-tripping the appended /
removed TOML through `ChannelsConfig` and asserting on
`parsed.channels.matrix.len()`, so the test build failed with
`E0609: no field `matrix` on type `ChannelsConfig`` against current
main.

Replace the typed deserialization with a raw count of
`[[channels.matrix]]` headers in the rewritten file. The covered
behavior is the TOML-string mutation (`append_channel_instance`,
`remove_channel_instance` — both string-based helpers), so the
header count is the equivalent assertion: one array-of-tables
header per logical instance. The legacy-single-table-form check is
kept by also asserting absence of `[channels.matrix]`.

All 4 affected tests green. Bundles with the orphan-backoff fix
because both have the same root cause (#5368 fallout) and ship
together as the main-red recovery PR.

* fix(kernel/config_reload): swap matrix→wechat in hot-reload test

Third #5368 fallout site: `config_reload::tests::test_channels_hot_reload`
built a `MatrixConfig` and assigned to `b.channels.matrix`, but the
Matrix sidecar migration removed both the field and the type. Switch
the test fixture to `WeChatConfig` — one of the remaining in-process
channels with a usable `Default` impl.

Bundled with the orphan-backoff + skills.rs fix to ship as a single
main-red recovery PR.

* fix(types/config): swap matrix→wechat in OneOrMany / channels tests

Fourth #5368 fallout site: librefang-types/src/config/mod.rs has six
test cases that exercised either MatrixConfig directly
(test_one_or_many_serialize_roundtrip, test_account_id_in_channel_configs)
or `b.channels.matrix` shape (test_channels_config_with_new_channels,
test_one_or_many_single_toml_table, test_one_or_many_array_of_tables,
test_one_or_many_empty_default). Matrix migrated to a sidecar in
#5368 so both the field and the type are gone.

These tests cover the OneOrMany helper, not Matrix specifically.
Swapped the fixture to WeChat — one of the remaining in-process
channels with a usable Default impl — and trimmed Matrix-only
fields (homeserver_url) from the TOML strings.

All 832 librefang-types lib tests green. Bundled into #5379 so a
single PR closes out the main-red recovery.

* perf(memory/migration): add composite sessions(agent_id, updated_at) + audit_entries(agent_id, timestamp) indexes (closes #5398)

audit: sessions-missing-index (Severity: Medium).

The `sessions` table had no usable index for the per-agent recency
hot path. The closest was `idx_sessions_peer ON sessions(agent_id,
peer_id)` from v16: in theory a prefix-scan for `WHERE agent_id =
?`, but the planner's choice was not stable, and the index name
discourages future hinting.

Hot-path query in
`count_agent_sessions_touched_since` (concurrent-trigger admission
check, runs on every fire):

  SELECT COUNT(*) FROM sessions
   WHERE agent_id = ?1 AND updated_at > ?2

On any deployment with > a few thousand sessions this degraded to a
full-table scan. `DELETE FROM sessions WHERE agent_id = ?1` (cascade
on agent removal) had the same shape.

The audit doc flagged the same shape on `audit_entries`: v8 created
two separate single-column indexes (agent_id, timestamp) so per-agent
recency queries pick one or the other and pay a sort.

Fix: add a new migration v41 that creates two composite indexes:

  CREATE INDEX IF NOT EXISTS idx_sessions_agent_updated
    ON sessions(agent_id, updated_at);

  CREATE INDEX IF NOT EXISTS idx_audit_agent_timestamp
    ON audit_entries(agent_id, timestamp);

`IF NOT EXISTS` makes a fresh boot idempotent. SCHEMA_VERSION
bumped 40 → 41 and the ladder runner gets a new `run_step!(41,
migrate_v41)` entry with the explanatory comment.

Two regression tests in `migration::tests`:

  - `migrate_v41_creates_composite_indexes_used_by_hot_paths` —
    confirms both indexes are present in `sqlite_master` and uses
    `EXPLAIN QUERY PLAN` to assert the planner actually picks
    `idx_sessions_agent_updated` for the
    `count_agent_sessions_touched_since` query (regression for
    "degrades to full-table scan").
  - `migrate_v41_is_idempotent` — three repeated `run_migrations`
    calls leave exactly one of the new index.

`cargo clippy -p librefang-memory --all-targets` clean.

Includes four prerequisite #5368-fallout commits (orphan helper +
three matrix→wechat test swaps); also shipping in #5379. They
dedupe on rebase once #5379 lands.

* chore(codegen): auto-regenerate openapi.json + sdk + schema baselines [skip ci]

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
houko added a commit that referenced this pull request May 22, 2026
…-concat (#5401)

* fix(types): remove orphan default_channel_initial_backoff_secs helper

#5368 (Matrix in-process → sidecar) removed the last consumer of
`default_channel_initial_backoff_secs` (the 1-second variant). The
matching Signal consumer was already removed earlier when Signal
migrated to a sidecar. Both sidecar channels control their own
backoff via env vars now.

Workspace lints have `warnings = "deny"`, so an orphaned helper
trips dead_code and turns main red. This is the cause of the
post-#5368 main breakage; affected every PR built against `origin/main`.

The sibling helpers `default_channel_max_backoff_secs` and
`default_channel_initial_backoff_2s` still have live consumers
(WeChat, QQ, Feishu …) and stay.

cargo check -p librefang-types --lib clean.

* fix(api/skills): replace ChannelsConfig::matrix round-trip with raw header count

After #5368 migrated Matrix to a sidecar, `librefang_types::config::
ChannelsConfig` no longer has a `matrix` field. Three test cases in
`routes::skills::tests` were still round-tripping the appended /
removed TOML through `ChannelsConfig` and asserting on
`parsed.channels.matrix.len()`, so the test build failed with
`E0609: no field `matrix` on type `ChannelsConfig`` against current
main.

Replace the typed deserialization with a raw count of
`[[channels.matrix]]` headers in the rewritten file. The covered
behavior is the TOML-string mutation (`append_channel_instance`,
`remove_channel_instance` — both string-based helpers), so the
header count is the equivalent assertion: one array-of-tables
header per logical instance. The legacy-single-table-form check is
kept by also asserting absence of `[channels.matrix]`.

All 4 affected tests green. Bundles with the orphan-backoff fix
because both have the same root cause (#5368 fallout) and ship
together as the main-red recovery PR.

* fix(kernel/config_reload): swap matrix→wechat in hot-reload test

Third #5368 fallout site: `config_reload::tests::test_channels_hot_reload`
built a `MatrixConfig` and assigned to `b.channels.matrix`, but the
Matrix sidecar migration removed both the field and the type. Switch
the test fixture to `WeChatConfig` — one of the remaining in-process
channels with a usable `Default` impl.

Bundled with the orphan-backoff + skills.rs fix to ship as a single
main-red recovery PR.

* fix(types/config): swap matrix→wechat in OneOrMany / channels tests

Fourth #5368 fallout site: librefang-types/src/config/mod.rs has six
test cases that exercised either MatrixConfig directly
(test_one_or_many_serialize_roundtrip, test_account_id_in_channel_configs)
or `b.channels.matrix` shape (test_channels_config_with_new_channels,
test_one_or_many_single_toml_table, test_one_or_many_array_of_tables,
test_one_or_many_empty_default). Matrix migrated to a sidecar in
#5368 so both the field and the type are gone.

These tests cover the OneOrMany helper, not Matrix specifically.
Swapped the fixture to WeChat — one of the remaining in-process
channels with a usable Default impl — and trimmed Matrix-only
fields (homeserver_url) from the TOML strings.

All 832 librefang-types lib tests green. Bundled into #5379 so a
single PR closes out the main-red recovery.

* fix(memory): bind cleanup_orphan_sessions IN-clause instead of string-concat (closes #5400)

audit: cleanup-orphan-sessions-format-sql (Severity: Medium).

`cleanup_orphan_sessions` was the only place in the substrate that
built SQL via `format!` instead of `?` parameter binding:

  placeholders.push(format!("'{}'", id.0))
  format!("DELETE FROM sessions WHERE agent_id NOT IN ({in_clause})")

Today the inner type is `AgentId(Uuid)` and the Display impl only
emits `[0-9a-f-]`, so the query is safe by accident. But:

  - This was the ONLY violation of the substrate-wide "use `?`"
    rule — easy for a future grep to miss.
  - The moment `AgentId(Uuid)` is relaxed to `AgentId(String)`
    (e.g. namespacing hand-agent ids), the door opens silently.
  - No test pinned the invariant down.

Fix: build the IN-clause as `?,?,?,…` and bind the values via
`rusqlite::params_from_iter(live_agent_ids.iter().map(|id|
id.0.to_string()))`. Same query plan, no escaping dependency on
the inner type. Used `std::iter::repeat_n` (stable in Rust 1.82+)
to build the placeholder string.

Two new tests in `session::tests`:

  - `test_cleanup_orphan_sessions_uses_bound_parameters_not_string_concat`
    — creates 3 live + 1 orphan agent session, asserts exactly the
    orphan row goes and the 3 live rows survive. Doubles as a sanity
    check that the parameter-binding shape works (the old format!
    version would have worked here too, but the test pins the
    contract going forward).
  - `test_cleanup_orphan_sessions_empty_live_set_deletes_nothing`
    — empty `live_agent_ids` is the "no live agents → don't touch
    anything" early-return. Documents the invariant so an off-by-one
    in a future refactor doesn't silently wipe every session when
    the registry is momentarily empty (e.g. during startup reload).

`cargo clippy -p librefang-memory --all-targets` clean.

Includes four prerequisite #5368-fallout commits (orphan helper +
three matrix→wechat test swaps); also shipping in #5379. They
dedupe on rebase once #5379 lands.

* chore(codegen): auto-regenerate openapi.json + sdk + schema baselines [skip ci]

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/channels Messaging channel adapters area/docs Documentation and guides area/kernel Core kernel (scheduling, RBAC, workflows) area/sdk JavaScript and Python SDKs size/XL 1000+ lines changed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant