Skip to content

Matrix crypto-store: observability for accumulated populated token-hash sibling dirs (discussion) #76613

Description

@eldron81-r2d2

Summary

This is a discussion / hardening request, not a regression report.

resolvePreferredMatrixStorageRoot (extensions/matrix/src/matrix/client/storage.ts:190-287) intentionally enumerates sibling token-hash directories under the same account dir and may pick a stale sibling over the canonical current-token root via a score+mtime heuristic. The behavior is covered by tests (storage.test.ts:483-620) and supports legitimate token-rotation continuity.

The request is to make the side effects of this design observable and documented, so that operators of long-running deployments can detect and clean up accumulated stale dirs. The selection logic itself works correctly — multiple populated dirs sit side by side, and the canonical-selection guards (require deviceId, require sibling to declare matching device) hold up. But there is no signal that they have accumulated, and over time they cause:

  1. Operator confusion ("which store is real?")
  2. Subtle harder-to-debug issues if metadata becomes ambiguous (e.g. partial write to storage-meta.json)
  3. Disk-bloat accumulation
  4. Pending-migration metadata in stale dirs being read by legacy-crypto-restore.ts (the sibling scan there is intentional — looking for legacy-crypto-migration.json — but it makes the multi-dir state surface in unexpected places)

In one observed deployment, three populated stores accumulated side by side over several token rotations, two of them holding state for a device that no longer existed on the homeserver. The active store selection picked the right one, but the existence of stale dirs made debugging unrelated crypto issues noticeably harder.

Affected file

extensions/matrix/src/matrix/client/storage.ts:190-287resolvePreferredMatrixStorageRoot()

Proposal: incremental, non-breaking observability

Primary ask — structured warning on multi-populated state

When sibling enumeration finds more than one populated hash dir under the same account, emit a structured warning. This is non-breaking — selection logic is unchanged.

// (LogService import would need to be added; storage.ts currently does not import it.)
const populatedSiblings = siblingEntries
  .filter(entry => entry.isDirectory() && entry.name !== params.canonicalTokenHash)
  .filter(entry => scoreStorageRoot(path.join(parentDir, entry.name)) > 0);

if (populatedSiblings.length > 0) {
  LogService.warn(
    "MatrixStorage",
    `Multiple populated token-hash dirs under ${parentDir} ` +
    `(canonical: ${params.canonicalTokenHash}, siblings: ${populatedSiblings.map(e => e.name).join(", ")}). ` +
    `Recommend archiving stale dirs to prevent drift after token rotations.`,
  );
}

Alternative: emit through the channel's structured logger if LogService is not the right channel here.

Documentation update

docs/channels/matrix.md (or the appropriate channel doc) could explicitly call out:

  • What happens to the old crypto-store on token rotation
  • That having multiple populated hash dirs is not a supported steady state
  • How to manually clean up siblings (recommend mv … _archive/ over rm -rf)

Possible follow-ups (separate, not part of this primary ask)

  1. Opt-in archive-on-startup. Config option channels.matrix.archiveStaleStores: false (default) that, when true, moves populated sibling dirs to _archive/<timestamp>/ after canonical root is selected. Operators can later remove _archive/ once they've confirmed migration was clean. Worth discussing as a separate issue if the primary observability lands.
  2. openclaw matrix doctor subcommand. Surface multi-populated-store detection and offer cleanup interactively. Better DX than relying on log scraping.
  3. Stricter mode that fails closed on multiple populated dirs. Probably too aggressive as a default — would break legitimate token-rotation workflows.

Real-world example

A deployment tree that triggered this discussion (paths sanitized):

~/.openclaw/matrix/accounts/default/<homeserver>__<userId>/
├── 660b…3921/                    (active, current device, ~380 keys)
├── c2e7…88b6/                    (stale, deleted device, ~26 keys)
└── c2e7…88b6.bak-20260413/       (backup of stale)

plus a sibling 'default.bak-pre-reset-20260417/' with another full mirror.

Active root selection correctly picked 660b…3921. The other dirs were leftovers from earlier resets. Without observability, operator had no signal that cleanup was needed; the only way to find this was a manual find ~/.openclaw/matrix -name 'storage-meta.json'.

Tests

In extensions/matrix/src/matrix/client/storage.test.ts:

  • Multi-populated hash-dirs scenario: assert structured warn-log is emitted (in addition to the existing reuse/canonical assertions)
  • No siblings present: warn-log is not emitted
  • Existing token-rotation tests (it("reuses an existing token-hash storage root for the same device …"), etc.) remain green

Why this is a discussion, not a bug

The current behavior is by design and tested. This issue is about giving operators visibility into multi-store accumulation, not about changing selection logic. If maintainers prefer to defer this entirely or scope it differently (e.g. "doctor command only"), happy to follow that direction.

Environment

  • OpenClaw: source at main 84a30e21 (2026.5.3)
  • Also affects 2026.4.27 shipped via npm
  • Storage path: ~/.openclaw/matrix/accounts/<account>/<homeserver>__<userId>/

AI authorship disclosure

Investigation, code review, and this report drafted with AI assistance (Anthropic Claude + OpenAI Codex). Human-validated against main 84a30e21 source before filing. Filed as part of a series with focused atomic-write and event-driven-persistence issues — all three relate to Matrix crypto-store persistence reliability.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P3Low-priority cleanup, docs, polish, ergonomics, or speculative work.clawsweeper:linked-pr-openClawSweeper found an open linked pull request for this issue.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.impact:otherThis issue has meaningful maintainer-visible impact outside the owned taxonomy.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions