fix(qa): record checked-out ref in evidence by vincentkoc · Pull Request #96434 · openclaw/openclaw

vincentkoc · 2026-06-24T13:07:50Z

What Problem This Solves

QA evidence could report a stale environment ref when GITHUB_SHA came from the workflow event or an inherited Crabbox environment instead of the checkout actually under test.

Why This Change Was Made

Evidence now prefers an explicit OPENCLAW_QA_REF, then resolves the checked-out HEAD from the requested repoRoot, then falls back to GITHUB_SHA for non-git/package environments. Suite, test-file, live transport, and UX Matrix producers pass their repo root through the shared resolver.

User Impact

QA artifacts identify the source revision that produced them, including manual/ref-selected and remote Crabbox runs. Explicit OPENCLAW_QA_REF behavior remains unchanged.

Evidence

Reproduced before fix: checkout ae9474b reported inherited GITHUB_SHA bd47995.
After fix: the same probe reports ae9474b and preserves an explicit override.
node scripts/run-vitest.mjs extensions/qa-lab/src/evidence-summary.test.ts test/scripts/qa-ux-matrix-evidence-producer.test.ts extensions/qa-lab/src/live-transports/telegram/telegram-live.runtime.test.ts extensions/qa-lab/src/live-transports/slack/slack-live.runtime.test.ts (77 tests passed)
oxfmt check, oxlint, and git diff --check passed.
autoreview: clean, no accepted/actionable findings.

clawsweeper · 2026-06-24T13:11:23Z

Codex review: needs maintainer review before merge. Reviewed June 24, 2026, 9:40 AM ET / 13:40 UTC.

Summary
The branch centralizes QA evidence ref resolution and threads repoRoot through suite, test-file, live-transport, and UX Matrix evidence producers so artifacts record the checked-out ref before falling back to GITHUB_SHA.

PR surface: Source +47, Tests +24, Other +8. Total +79 across 9 files.

Reproducibility: yes. from source and PR proof: current main records OPENCLAW_QA_REF || GITHUB_SHA, so an inherited event SHA can differ from the checked-out repo under test. I did not run local tests because this review is read-only.

Review metrics: none identified.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🌊 off-meta tidepool
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

[P2] Rerun or clear the SIGKILL broad CI shard if it remains required before merge.

Risk before merge

[P1] Current head has an unstable broad CI shard ending in SIGKILL; no diff-level defect was identified from the log excerpt, but required CI should remain the merge gate.

Maintainer options:

Decide the mitigation before merge
Land the shared resolver after maintainer review and green required checks, preserving OPENCLAW_QA_REF as the highest-precedence override and GITHUB_SHA as the non-git fallback.
Pause or close
Do not merge this PR until maintainers decide whether the risk is worth taking.

Next step before merge

No ClawSweeper repair lane is needed; this maintainer-authored, protected-label PR needs maintainer review and required checks rather than an automated fix.

Security
Cleared: The diff adds a fixed-argument local git rev-parse probe for QA metadata and does not change dependencies, workflows, permissions, secrets, or package resolution.

Review details

Best possible solution:

Land the shared resolver after maintainer review and green required checks, preserving OPENCLAW_QA_REF as the highest-precedence override and GITHUB_SHA as the non-git fallback.

Do we have a high-confidence way to reproduce the issue?

Yes, from source and PR proof: current main records OPENCLAW_QA_REF || GITHUB_SHA, so an inherited event SHA can differ from the checked-out repo under test. I did not run local tests because this review is read-only.

Is this the best way to solve the issue?

Yes, this is the best fix shape: the resolver is centralized in QA evidence construction and callers pass their resolved repo root. Duplicating checkout-ref logic in each producer would be more drift-prone.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against f0be8e7b6ee8.

Label changes

Label justifications:

P3: This is a small QA artifact accuracy fix with limited blast radius and no user runtime compatibility impact.
rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🌊 off-meta tidepool and patch quality is 🐚 platinum hermit.
status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: The external-contributor proof gate does not apply to this maintainer-authored PR; the body still includes before/after probe output and targeted validation commands.

Evidence reviewed

PR surface:

Source +47, Tests +24, Other +8. Total +79 across 9 files.

View PR surface stats

Area	Files	Added	Removed	Net
Source	7	52	5	+47
Tests	1	24	0	+24
Docs	0	0	0	0
Config	0	0	0	0
Generated	0	0	0	0
Other	1	16	8	+8
Total	9	92	13	+79

What I checked:

Repository policy read: Root AGENTS.md plus scoped extensions/AGENTS.md, scripts/AGENTS.md, and test/AGENTS.md were read; the protected maintainer-label and deep PR review rules affected the keep-open routing. (AGENTS.md:1, f0be8e7b6ee8)
Current main records environment ref only: Current main resolves QA evidence environment.ref from OPENCLAW_QA_REF or GITHUB_SHA, which can record an inherited workflow event SHA instead of the checked-out repository revision. (extensions/qa-lab/src/evidence-summary.ts:391, f0be8e7b6ee8)
Patch centralizes checkout ref resolution: The PR adds a shared resolver that preserves explicit OPENCLAW_QA_REF, reads git rev-parse --verify HEAD from the requested repo root, and falls back to GITHUB_SHA when git resolution is unavailable. (extensions/qa-lab/src/evidence-summary.ts:390, c79fb2cddd72)
Patch threads repoRoot through QA producers: The diff passes repoRoot through QA suite artifacts, test-file evidence, Discord/Slack/Telegram/WhatsApp live evidence, and the UX Matrix script producer, covering the central producer surfaces instead of duplicating ref logic per caller. (extensions/qa-lab/src/suite.ts:848, c79fb2cddd72)
Regression coverage added: The PR adds a focused test that sets a stale GITHUB_SHA, passes the current repo root, and expects evidence to record the checked-out HEAD. (extensions/qa-lab/src/evidence-summary.test.ts:124, c79fb2cddd72)
CI status checked: Live PR status shows the head is mergeable but unstable; the failing broad shard ended with SIGKILL after many tests passed, not with a concrete assertion tied to this diff. (c79fb2cddd72)

Likely related people:

RomneyDa: Prior merged commits added and evolved QA evidence artifact output, evidence summary shape, live evidence summaries, and slim evidence mode across the central QA Lab evidence files. (role: feature owner; confidence: high; commits: 4809ac70fa5a, 561b293c7a96, e32929e12cd4; files: extensions/qa-lab/src/evidence-summary.ts, extensions/qa-lab/src/suite.ts, extensions/qa-lab/src/test-file-scenario-runner.ts)
vincentkoc: Recent main history touches the UX Matrix evidence producer and test-file runner, and current main blame for the existing environment resolver points to recent release-closeout history by this contributor. (role: recent area contributor; confidence: high; commits: 0671c0890073, 08f8de3aeec9, 648ef73bde1b; files: extensions/qa-lab/src/evidence-summary.ts, scripts/qa/ux-matrix-evidence-producer.ts, extensions/qa-lab/src/test-file-scenario-runner.ts)
Solvely-Colin: Prior merged work added script-backed QA evidence scenarios and the UX Matrix evidence scenario, which are caller surfaces updated by this PR. (role: adjacent owner; confidence: medium; commits: 591313e80a5e, d5a27b0b9684; files: extensions/qa-lab/src/test-file-scenario-runner.ts, scripts/qa/ux-matrix-evidence-producer.ts, extensions/qa-lab/src/evidence-summary.ts)

What the crustacean ranks mean

🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

vincentkoc · 2026-06-24T14:37:48Z

Merged via squash.

Prepared head SHA: 86b3df6e59b17fb4a1670a2007c03e11cc974e41
Merge commit: d83cd282c602bd61ee54ed7baec9377ef60d309b

Thanks @vincentkoc!

@vincentkoc

Merged via squash. Prepared head SHA: 86b3df6 Co-authored-by: vincentkoc <[email protected]> Co-authored-by: vincentkoc <[email protected]> Reviewed-by: @vincentkoc

@vincentkoc

Merged via squash. Prepared head SHA: 86b3df6 Co-authored-by: vincentkoc <[email protected]> Co-authored-by: vincentkoc <[email protected]> Reviewed-by: @vincentkoc

openclaw-barnacle Bot added scripts Repository scripts extensions: qa-lab size: S maintainer Maintainer-authored PR labels Jun 24, 2026

vincentkoc marked this pull request as ready for review June 24, 2026 13:08

fix(qa): record checked-out ref in evidence

c79fb2c

vincentkoc force-pushed the fix/qa-evidence-ref-origin branch from c5f6cc2 to c79fb2c Compare June 24, 2026 13:20

fix(qa): record checked-out ref in evidence

86b3df6

vincentkoc requested a review from a team as a code owner June 24, 2026 14:27

vincentkoc merged commit d83cd28 into main Jun 24, 2026
153 of 160 checks passed

vincentkoc deleted the fix/qa-evidence-ref-origin branch June 24, 2026 14:37

github-actions Bot mentioned this pull request Jun 24, 2026

📡 Upstream Digest — 2026-06-24 17:48 UTC curtismercier/openclaw-mods#1145

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(qa): record checked-out ref in evidence#96434

fix(qa): record checked-out ref in evidence#96434
vincentkoc merged 2 commits into
mainfrom
fix/qa-evidence-ref-origin

vincentkoc commented Jun 24, 2026

Uh oh!

clawsweeper Bot commented Jun 24, 2026 •

edited

Loading

Uh oh!

Uh oh!

vincentkoc commented Jun 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Uh oh!

Conversation

vincentkoc commented Jun 24, 2026

What Problem This Solves

Why This Change Was Made

User Impact

Evidence

Uh oh!

clawsweeper Bot commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

vincentkoc commented Jun 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

clawsweeper Bot commented Jun 24, 2026 •

edited

Loading