Skip to content

fix(matrix): replace recovered command progress lines#89920

Merged
jesse-merhi merged 8 commits into
openclaw:mainfrom
bdjben:fix/matrix-recovered-tool-progress
Jun 15, 2026
Merged

fix(matrix): replace recovered command progress lines#89920
jesse-merhi merged 8 commits into
openclaw:mainfrom
bdjben:fix/matrix-recovered-tool-progress

Conversation

@bdjben

@bdjben bdjben commented Jun 3, 2026

Copy link
Copy Markdown
Contributor

Summary

  • preserve Matrix command-output progress identity with item/tool call IDs so recovered command output updates the existing draft line
  • prevent stale nested tool failures like run openclaw cron -> run jq (agent) failed from remaining visible after the command recovers
  • add a Matrix monitor regression covering failed nested progress followed by completed command output

Proof

Redacted live behavior that motivated this fix

In a private Matrix room, a tool-progress notice presented a nested tool-chain failure as a live/current failure:

run openclaw cron -> run jq (agent) failed

The private room, sender, Matrix event IDs, cron job ID, exact room label, and surrounding transcript are intentionally omitted. The important user-visible behavior was that Matrix progress rendering made the stale nested failure remain visible even after the underlying work had recovered.

A follow-up check of the underlying cron/run state showed the later run was green, so the remaining bad user experience was Matrix progress state reconciliation: the old failed progress line was not being replaced by later recovered command output.

Here is a screenshot of the same behavior (see the failed message posted after the substantive work which did not actually fail):
Screenshot 2026-06-08 at 4 41 10 PM

Redacted after-fix live Matrix transport proof

First, here is a screenshot of after-fix behavior from the same agent (no more false "failed" message):
Screenshot 2026-06-08 at 4 38 09 PM

Further, I verified the after-fix behavior on 2026-06-07 against a real Matrix homeserver through the Matrix Client-Server API, using the [redacted] bot account in a temporary private proof room with no invites and no user transcript content.

Setup:

  • PR commit: 41512ac1636d28e4e98d25391d4245b825a5c372
  • Matrix setup: real Matrix homeserver, private temporary room, [redacted] bot account
  • Redactions: room ID, original draft event ID, edit event ID, account identifiers, and all server-local identifiers

Readback proof:

{
  "roomId": "[redacted-room-id]",
  "originalDraftEventId": "[redacted-original-event-id]",
  "editEventId": "[redacted-edit-event-id]",
  "beforeDraftBody": "Working\n- `🛠️ redacted command failed`",
  "editOuterBody": "* Working\n- `🛠️ completed`",
  "editNewContentBody": "Working\n- `🛠️ completed`",
  "editRelation": {
    "rel_type": "m.replace",
    "event_id": "[redacted-original-event-id]"
  },
  "assertions": {
    "sameDraftEdited": true,
    "beforeContainsFailed": true,
    "afterContainsCompleted": true,
    "afterOmitsFailed": true,
    "originalRoomHasNoInvites": true
  }
}

This proves the visible Matrix transport operation is an edit of the existing draft event via m.replace, and the edited m.new_content.body contains completed while no longer containing failed.

Matrix handler regression proof

The regression in extensions/matrix/src/matrix/monitor/handler.test.ts uses synthetic Matrix identifiers and the public-safe failure string above.

Reproduction sequence:

  1. Emit a failed command item with stable synthetic IDs:
    • itemId: command-1
    • toolCallId: call-1
    • progressText: run openclaw cron -> run jq (agent) failed
  2. Verify the Matrix draft initially contains failed.
  3. Emit recovered command output for the same synthetic IDs:
    • status: completed
    • exitCode: 0
  4. Verify the edited Matrix draft contains completed.
  5. Verify the edited Matrix draft no longer contains:
    • failed
    • run openclaw cron -> run jq

That proves the recovered command progress replaces the stale failed line instead of leaving a fake live failure in Matrix.

Commands run:

  • pnpm -s exec vitest run --config test/vitest/vitest.extension-matrix.config.ts extensions/matrix/src/matrix/monitor/handler.test.ts -t "replaces recovered Matrix command progress" — 1 passed, 113 skipped
  • pnpm -s exec vitest run --config test/vitest/vitest.extension-matrix.config.ts extensions/matrix/src/matrix/monitor/handler.test.ts — 114 passed
  • git diff --check — passed

Redaction note

No live Matrix room IDs, user IDs, message IDs, cron IDs, room labels, private transcript text, or private job contents are included. The proof uses synthetic identifiers and the minimum public-safe failure string needed to demonstrate the behavior.

Real behavior proof

Behavior addressed: Matrix command-progress drafts used the public itemId for both the command start line and the later command-output line. When those IDs differed for the same command call, the draft kept the stale command-start line instead of replacing it with the completed command-output line.

Real environment tested: OpenClaw Matrix QA on rebased head e220ad3f905, with a real OpenClaw gateway process, Matrix room topology, ghcr.io/matrix-construct/tuwunel:v1.5.1, and mock OpenAI provider. The QA harness captured Matrix room events and verified the preview edit relation.

Exact steps or command run after this patch:

QA_ROOT="$(mktemp -d /tmp/openclaw-matrix-pr-rebased.XXXXXX)"
mkdir -p "$QA_ROOT/home/.openclaw" "$QA_ROOT/state"
printf '{}\n' > "$QA_ROOT/home/.openclaw/openclaw.json"
OPENCLAW_HOME="$QA_ROOT/home" \
OPENCLAW_CONFIG_PATH="$QA_ROOT/home/.openclaw/openclaw.json" \
OPENCLAW_STATE_DIR="$QA_ROOT/state" \
OPENCLAW_QA_MATRIX_CAPTURE_CONTENT=1 \
pnpm openclaw qa matrix \
  --provider-mode mock-openai \
  --profile all \
  --scenario matrix-room-tool-progress-command-preview \
  --output-dir .artifacts/qa-e2e/manual-command-progress-pr-rebased-latest \
  --fail-fast

Evidence after fix:

[matrix-qa] scenario pass matrix-room-tool-progress-command-preview 3.9s
[matrix-qa] suite pass 4/4 total=23.8s
preview body: Clawing

`🛠️ completed; command print text → run sleep 2`
final reply relation: m.replace
final reply target: $iJqknkCMdBm47S0T6Bt6WocytEd8ibv6GVzqbqgNQw8
final reply token matched: yes

Observed result after fix: the Matrix preview contains the completed command line only, and the final Matrix reply replaces the same preview event with m.replace. The stale matrix-command-progress-start command line is gone.

Additional before/after check on the same merge helper:

origin/main d012d29e6f29:
lineCount 2
🛠️ `matrix-command-progress-start`
🛠️ `completed`
RESULT=STALE

PR head e220ad3f905:
lineCount 1
🛠️ `completed`
RESULT=CLEAN

What was not tested: an external production Matrix homeserver was not rerun after the rebase. The PR already includes earlier redacted live Matrix transport evidence, and this rebase rerun used the Matrix QA homeserver plus the real OpenClaw Matrix gateway path.

@openclaw-barnacle openclaw-barnacle Bot added channel: matrix Channel integration: matrix size: S triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels Jun 3, 2026
@clawsweeper

clawsweeper Bot commented Jun 3, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed June 12, 2026, 3:02 AM ET / 07:02 UTC.

Summary
The PR correlates command progress and completion events so recovered commands replace stale progress lines across Matrix and sibling channel previews.

PR surface: Source +249, Tests +929. Total +1178 across 22 files.

Reproducibility: unclear. the PR body provides a credible live Matrix sequence and regression scenario, but the current-main failing path was not independently traced in this incomplete review.

Review metrics: 1 noteworthy metric.

  • Channel behavior surface: 5 channel adapters changed. A shared replacement invariant now affects Discord, Matrix, Teams, Slack, and Telegram preview behavior.

Merge readiness
Overall: 🦐 gold shrimp
Proof: 🦞 diamond lobster
Patch quality: 🦐 gold shrimp
Result: needs maintainer review before merge.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • Complete the shared event-contract and sibling-channel correctness review.

Risk before merge

  • [P1] The shared identity change affects progress replacement semantics across Discord, Matrix, Teams, Slack, and Telegram and needs complete sibling-path compatibility review.

Maintainer options:

  1. Finish cross-channel compatibility review (recommended)
    Verify that correlation replaces only matching command lines while preserving public line IDs and unrelated concurrent progress before merge.
  2. Pause for owner review
    Keep the PR open for the shared streaming and agent-runtime owners if the event identity contract remains ambiguous.

Next step before merge

  • [P2] A maintainer should resume the incomplete deep review; no concrete repair finding is established yet.

Security
Cleared: The supplied diff shows no concrete security or supply-chain change, though the functional review remains incomplete.

Review details

Best possible solution:

Complete the full review of the shared correlation invariant, Codex event contract, all channel callers, scoped policies, tests, current-main behavior, and prepared live proof before deciding whether to merge.

Do we have a high-confidence way to reproduce the issue?

Unclear: the PR body provides a credible live Matrix sequence and regression scenario, but the current-main failing path was not independently traced in this incomplete review.

Is this the best way to solve the issue?

Unclear: shared correlation appears directionally appropriate, but the full event contract and sibling channel behavior were not reviewed deeply enough to establish that this is the best fix.

AGENTS.md: unclear because the file could not be read completely.

Codex review notes: model internal, reasoning high; reviewed against d4819948f37d.

Label changes

Label changes:

  • add merge-risk: 🚨 compatibility: Changing shared progress identity can replace the wrong line or alter existing channel preview behavior if correlation semantics differ across event sources.
  • add rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦞 diamond lobster and patch quality is 🦐 gold shrimp.
  • remove rating: 🦞 diamond lobster: Current PR rating is rating: 🦐 gold shrimp, so this older rating label is no longer current.

Label justifications:

  • P2: This is a normal-priority user-visible progress correction with bounded but multi-channel blast radius.
  • merge-risk: 🚨 compatibility: Changing shared progress identity can replace the wrong line or alter existing channel preview behavior if correlation semantics differ across event sources.
  • rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦞 diamond lobster and patch quality is 🦐 gold shrimp.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The PR body includes redacted after-fix Matrix homeserver readback showing an m.replace edit of the same draft event, with completed present and failed removed.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR body includes redacted after-fix Matrix homeserver readback showing an m.replace edit of the same draft event, with completed present and failed removed.
  • mantis: telegram-visible-proof: Mantis should capture Telegram visible proof. The PR changes visible Telegram command-progress replacement and prepared native Telegram Desktop proof exists for that behavior.
Evidence reviewed

PR surface:

Source +249, Tests +929. Total +1178 across 22 files.

View PR surface stats
Area Files Added Removed Net
Source 13 274 25 +249
Tests 9 939 10 +929
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 22 1213 35 +1178

What I checked:

  • Provided PR context: The supplied diff spans shared channel streaming, agent event forwarding, Matrix QA, and five channel adapters, so a partial review would not support a safe verdict. (src/channels/streaming.ts:230, 2edf75618f4a)

Likely related people:

  • jesse-merhi: Authored the latest five commits refining shared command-progress identity across the affected channel and agent-runtime surfaces. (role: recent area contributor; confidence: high; commits: 9513c78fbea8, af12db860b61, cbae361e5ca6; files: src/channels/streaming.ts, src/auto-reply/reply/agent-runner-execution.ts, extensions/matrix/src/matrix/monitor/handler.ts)
  • bdjben: Reported the live Matrix failure and authored the initial command-progress replacement change with after-fix Matrix proof. (role: behavior reporter and initial patch author; confidence: medium; commits: 74ab486aea95; files: extensions/matrix/src/matrix/monitor/handler.ts, extensions/matrix/src/matrix/monitor/handler.test.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P2 Normal backlog priority with limited blast radius. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. labels Jun 3, 2026
@bdjben

bdjben commented Jun 3, 2026

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jun 3, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

@openclaw-barnacle openclaw-barnacle Bot added the triage: blank-template Candidate: PR template appears mostly untouched. label Jun 7, 2026
@bdjben

bdjben commented Jun 7, 2026

Copy link
Copy Markdown
Contributor Author

Added redacted after-fix live Matrix transport proof to the PR body: real Matrix Client-Server API send + m.replace edit in a private no-invite proof room, showing the edited m.new_content body contains completed and omits failed. Also reran focused handler test, full handler test, and git diff --check.

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jun 7, 2026

Copy link
Copy Markdown
Contributor

🦞👀
ClawSweeper picked this up.

Command router queued. I will update this comment with the next step.

@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jun 7, 2026
@openclaw-barnacle openclaw-barnacle Bot added triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. and removed triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. proof: sufficient ClawSweeper judged the real behavior proof convincing. labels Jun 7, 2026
@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. merge-risk: 🚨 other 🚨 Merging this PR has meaningful risk outside the owned taxonomy. and removed rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. labels Jun 8, 2026
@openclaw-barnacle openclaw-barnacle Bot removed the triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. label Jun 8, 2026
@jesse-merhi jesse-merhi self-assigned this Jun 9, 2026
@jesse-merhi

Copy link
Copy Markdown
Member

/clawsweeper re-review

@openclaw-barnacle openclaw-barnacle Bot added the channel: discord Channel integration: discord label Jun 10, 2026
@clawsweeper clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 12, 2026
@openclaw-barnacle openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 12, 2026
@clawsweeper clawsweeper Bot added rating: 🌊 off-meta tidepool PR readiness rating does not apply to this item. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. mantis: telegram-visible-proof Mantis should capture Telegram visible proof. and removed rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. mantis: telegram-visible-proof Mantis should capture Telegram visible proof. rating: 🌊 off-meta tidepool PR readiness rating does not apply to this item. labels Jun 12, 2026
@openclaw-barnacle openclaw-barnacle Bot removed the triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. label Jun 12, 2026
@clawsweeper clawsweeper Bot removed proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. labels Jun 12, 2026
@jesse-merhi

Copy link
Copy Markdown
Member

Land-ready verification after rebase on origin/main (d012d29e6f29) and PR head e220ad3f905.

Local proof run:

origin/main d012d29e6f29:
lineCount 2
🛠️ `matrix-command-progress-start`
🛠️ `completed`
RESULT=STALE

PR head e220ad3f905:
lineCount 1
🛠️ `completed`
RESULT=CLEAN

Focused tests:

PATH="/Users/jmerhi/.local/share/fnm/node-versions/v22.21.0/installation/bin:$PATH" node scripts/run-vitest.mjs run extensions/discord/src/monitor/message-handler.process.test.ts extensions/matrix/src/matrix/monitor/handler.test.ts extensions/msteams/src/reply-dispatcher.test.ts extensions/slack/src/monitor/message-handler/dispatch.preview-fallback.test.ts extensions/telegram/src/bot-message-dispatch.test.ts extensions/qa-matrix/src/runners/contract/scenarios.test.ts src/auto-reply/reply/agent-runner-execution.test.ts src/auto-reply/reply/followup-runner.test.ts src/plugin-sdk/channel-streaming.test.ts
[test] passed 8 Vitest shards in 71.14s

Matrix QA:

pnpm openclaw qa matrix --provider-mode mock-openai --profile all --scenario matrix-room-tool-progress-command-preview --output-dir .artifacts/qa-e2e/manual-command-progress-pr-rebased-latest --fail-fast
[matrix-qa] scenario pass matrix-room-tool-progress-command-preview 3.9s
[matrix-qa] suite pass 4/4 total=23.8s
preview body: Clawing

`🛠️ completed; command print text → run sleep 2`
final reply relation: m.replace

CI: GitHub reports mergeStateStatus: CLEAN; gh pr checks reports 165 checks with 0 pending/failing/blocking.

Known proof gap: I did not rerun the external production Matrix homeserver proof after the rebase. The PR body has earlier redacted live Matrix transport proof, and this rebase was verified with the Matrix QA homeserver plus the real OpenClaw Matrix gateway path.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

channel: discord Channel integration: discord channel: matrix Channel integration: matrix channel: msteams Channel integration: msteams channel: slack Channel integration: slack channel: telegram Channel integration: telegram mantis: telegram-visible-proof Mantis should capture Telegram visible proof. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. P2 Normal backlog priority with limited blast radius. proof: supplied External PR includes structured after-fix real behavior proof. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. size: XL status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. triage: blank-template Candidate: PR template appears mostly untouched.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants