Skip to content

test(channels): delivery trace harness with feishu and mattermost goldens#104478

Merged
steipete merged 1 commit into
mainfrom
claude/chanref-p0-delivery-traces
Jul 11, 2026
Merged

test(channels): delivery trace harness with feishu and mattermost goldens#104478
steipete merged 1 commit into
mainfrom
claude/chanref-p0-delivery-traces

Conversation

@steipete

@steipete steipete commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

Related: #104318

What Problem This Solves

The channel refactor program needs a conformance oracle: today, each channel's delivery behavior (streaming edits, card entity lifecycles, cancellation, rate-limit recovery, pagination) lives only as code inside deliver callbacks. There is no replayable record of what wire calls a channel makes for a given reply turn — which is exactly what the planned core delivery engines must reproduce before any channel migrates onto them.

Why This Change Was Made

Adds a small deterministic trace harness (src/channels/plugins/contracts/trace/delivery-trace.ts, 310 LOC, exported through the existing channel-contract-testing test subpath): scenario scripts drive a channel's real dispatcher wiring with mocked wire clients under fake timers; ordered in/out events are recorded as committed JSONL goldens; verify mode (default) replays and asserts byte-stable equality; OPENCLAW_TRACE_UPDATE=1 re-records. First two adoptions prove the pattern at both extremes: feishu (5 scenarios — full CardKit entity lifecycle, 429'd preview recovery, overflow pagination, cancel discarding the card) and mattermost (3 scenarios — edit-in-place drafts, boundary rotation, stop-flush cancel). Goldens deliberately pin two current-behavior warts as Phase-4 oracles (feishu cross-block snapshot glue without separator; cancel closes the card with partial content). No prod surface; extension tests drive everything (no core→extension imports). Credential/token values never appear in goldens.

User Impact

None at runtime — test infrastructure. For maintainers: channel delivery behavior is now pinned and replayable; the planned core engines get an executable acceptance spec.

Evidence

  • Double-run golden stability on Testbox tbx_01kx8hw49gvym563sk9b54r321 (record + verify green twice), re-proven post-rebase on tbx_01kx8k1q3spdwp6977bz5hxm2y.
  • Full lanes green: feishu 78 files/1126 tests, mattermost 44/569, contracts lanes (incl. import guardrails over the new core file), complete pnpm check:changed.
  • Autoreview (codex gpt-5.6-sol, xhigh): clean — one scenario-consistency finding accepted and fixed (rate-limit recovery timing), one out-of-scope suggestion (implement backoff) rejected per the binding spec: this PR captures current behavior, Phase 4 owns policy.

Note for reviewers: pnpm test src/channels/plugins/contracts is a no-op in test-projects (dir excluded); the contracts lanes run via test:contracts:channels.

@openclaw-barnacle openclaw-barnacle Bot added channel: mattermost Channel integration: mattermost channel: feishu Channel integration: feishu size: XL maintainer Maintainer-authored PR labels Jul 11, 2026
@clawsweeper clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. P3 Low-priority cleanup, docs, polish, ergonomics, or speculative work. labels Jul 11, 2026
@clawsweeper

clawsweeper Bot commented Jul 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Codex review: needs changes before merge. Reviewed July 11, 2026, 9:40 AM ET / 13:40 UTC.

Summary
Adds a reusable delivery-trace test harness, exports it from the channel contract-testing SDK subpath, and adds five Feishu plus three Mattermost JSONL golden scenarios.

PR surface: Source +425, Tests +703. Total +1128 across 12 files.

Reproducibility: not applicable. This PR adds deterministic test infrastructure rather than reporting a current-main runtime bug. The adapter defect is directly source-verifiable by comparing the proposed mapping with production Mattermost callback ordering.

Review metrics: 1 noteworthy metric.

  • Golden scenarios: 8 added. Five Feishu and three Mattermost traces define the initial lifecycle oracle for the linked channel-refactor program.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #104318
Summary: This PR is an implementation candidate for the linked channel-refactor program's Phase 0 delivery-trace conformance oracle.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🦐 gold shrimp
Proof: 🦞 diamond lobster
Patch quality: 🦐 gold shrimp
Result: needs maintainer review before merge.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • Align the Mattermost boundary transition with production ordering and regenerate the affected golden.

Next step before merge

  • [P2] The remaining blocker is a narrow mechanical test-adapter ordering repair with a clear fixture and focused validation path.

Security
Cleared: The patch is limited to tests, fixtures, and a test SDK barrel, with no dependency, workflow, permission, secret, install, publishing, or production execution changes.

Review findings

  • [P2] Defer the Mattermost boundary flush until the next activity — extensions/mattermost/src/delivery-trace.test.ts:218-223
Review details

Best possible solution:

Represent assistant-message-start as pending state in the Mattermost trace adapter, rotate the preview generation when the next visible activity arrives, and regenerate the affected golden.

Do we have a high-confidence way to reproduce the issue?

Not applicable: this PR adds deterministic test infrastructure rather than reporting a current-main runtime bug. The adapter defect is directly source-verifiable by comparing the proposed mapping with production Mattermost callback ordering.

Is this the best way to solve the issue?

No in its current Mattermost mapping: the generic harness and test-only SDK seam are appropriate, but the adapter must preserve production lifecycle ordering before its goldens can serve as an engine oracle.

Full review comments:

  • [P2] Defer the Mattermost boundary flush until the next activity — extensions/mattermost/src/delivery-trace.test.ts:218-223
    block-final represents assistant-message-start, but production only marks that boundary pending and calls noteBoundary() when the next visible text, reasoning, or tool activity begins. Awaiting it here emits the prior-post PUT before the next partial input, so the golden records the wrong wire ordering. Track a pending boundary, rotate from the next visible activity, and regenerate the golden. This remains from the prior review and was equally visible on the first reviewed head.
    Confidence: 0.96
    Late finding: first raised on code an earlier review cycle already covered.

Overall correctness: patch is incorrect
Overall confidence: 0.96

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 325679a33574.

Label changes

Label justifications:

  • P3: This is maintainer-facing test infrastructure with no direct runtime or user-facing behavior change.
  • rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦞 diamond lobster and patch quality is 🦐 gold shrimp.
  • status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Sufficient (live_output): The PR provides exact-head Testbox record-and-verify output plus Feishu, Mattermost, channel-contract, and changed-check results; no further contributor proof is required for this test-only change.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR provides exact-head Testbox record-and-verify output plus Feishu, Mattermost, channel-contract, and changed-check results; no further contributor proof is required for this test-only change.
Evidence reviewed

PR surface:

Source +425, Tests +703. Total +1128 across 12 files.

View PR surface stats
Area Files Added Removed Net
Source 10 425 0 +425
Tests 2 703 0 +703
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 12 1128 0 +1128

Acceptance criteria:

  • [P1] pnpm test extensions/mattermost/src/delivery-trace.test.ts.
  • [P1] pnpm test extensions/mattermost.
  • [P1] pnpm test:contracts:channels.
  • [P1] pnpm check:changed.

What I checked:

  • Proposed boundary behavior: The Mattermost adapter clears partial state and immediately awaits noteBoundary() when processing block-final. (extensions/mattermost/src/delivery-trace.test.ts:218, 593cea6dee97)
  • Production lifecycle: Current Mattermost code treats assistant-message-start as a pending candidate; enterBlockPreviewActivity() performs noteBoundary() only when the next visible text, reasoning, or tool activity begins a new generation. (extensions/mattermost/src/mattermost/monitor.ts:1771, 325679a33574)
  • Boundary controller effect: Calling noteBoundary() immediately resets streamed-content state and awaits forceNewMessage(), which seals and rotates the current draft generation before the next scripted input. (extensions/mattermost/src/mattermost/draft-stream.ts:62, 325679a33574)
  • Re-review continuity: The live head is still the SHA from the latest completed review, so the prior P2 has not been repaired. (593cea6dee97)
  • Late-finding provenance: The Mattermost boundary logic was unchanged between the first and latest reviewed heads; only formatting changed in that file. (extensions/mattermost/src/delivery-trace.test.ts:218, 593cea6dee97)
  • Program relationship: The linked open umbrella explicitly lists delivery trace recording as Phase 0 behavior-lock work, making this PR a candidate implementation rather than a duplicate or standalone direction.

Likely related people:

  • vincentkoc: Git history and blame connect this contributor to the current Mattermost monitor and draft-stream lifecycle that defines the ordering invariant. (role: introduced current behavior; confidence: high; commits: 6013e78f5a2a, 61f93540b262; files: extensions/mattermost/src/mattermost/monitor.ts, extensions/mattermost/src/mattermost/draft-stream.ts)
  • steipete: Prior merged channel-contract and runtime history plus authorship of the canonical refactor program connect this contributor to the intended conformance-oracle direction. (role: feature owner and program proposer; confidence: high; commits: ab96520bbad5, 2766c27b2aa6, f2d7a825b122; files: src/channels/plugins/types.plugin.ts, src/plugins/runtime/runtime-channel.ts, src/channels/message/reply-pipeline.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.
Review history (2 earlier review cycles)
  • reviewed 2026-07-11T13:02:55.294Z sha 6bd64a2 :: needs real behavior proof before merge. :: [P2] Remove the plugin-private path literal from the core harness
  • reviewed 2026-07-11T13:34:44.997Z sha 593cea6 :: needs changes before merge. :: [P2] Defer the Mattermost boundary flush until the next activity

@steipete
steipete force-pushed the claude/chanref-p0-delivery-traces branch from 6bd64a2 to 593cea6 Compare July 11, 2026 13:28
@clawsweeper clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jul 11, 2026
@steipete
steipete merged commit ba826be into main Jul 11, 2026
109 checks passed
@steipete
steipete deleted the claude/chanref-p0-delivery-traces branch July 11, 2026 13:49
@steipete

steipete commented Jul 11, 2026

Copy link
Copy Markdown
Contributor Author

Merged via squash.

@github-actions github-actions Bot mentioned this pull request Jul 11, 2026
Open
This was referenced Jul 11, 2026
Merged
Merged
Merged
Merged
Merged
Merged
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jul 12, 2026
@steipete
test(channels): delivery trace harness with feishu and mattermost gol…
e57c9b3 
…dens (openclaw#104478)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

channel: feishu Channel integration: feishu channel: mattermost Channel integration: mattermost maintainer Maintainer-authored PR P3 Low-priority cleanup, docs, polish, ergonomics, or speculative work. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. size: XL status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@steipete