Skip to content

fix(ci): ignore profiling-only resource gate failures#102894

Merged
vincentkoc merged 1 commit into
mainfrom
fix/performance-profiled-release-gate
Jul 9, 2026
Merged

fix(ci): ignore profiling-only resource gate failures#102894
vincentkoc merged 1 commit into
mainfrom
fix/performance-profiled-release-gate

Conversation

@vincentkoc

@vincentkoc vincentkoc commented Jul 9, 2026

Copy link
Copy Markdown
Member

Closes #102888

What Problem This Solves

Fixes an issue where maintainers running the deep-profile release performance lane would get a false shipment blocker when V8 profiling pushed CPU/RSS above normal-path thresholds, even though Kova marked those measurements profiling-affected and baseline-ineligible.

Why This Change Was Made

The trusted Kova report adapter now accepts a non-zero verdict only when every blocking card maps one-to-one to a failed profiled record, every violation is an affected CPU/RSS metric, and there are no error statuses or baseline regressions. Functional, latency, auth, timeout, harness, provider, and ordinary unprofiled resource failures remain blocking.

User Impact

Release validation no longer reruns or blocks on profiler overhead alone. Normal-path performance limits and every non-resource release gate remain strict.

Evidence

@vincentkoc vincentkoc self-assigned this Jul 9, 2026
@openclaw-barnacle openclaw-barnacle Bot added scripts Repository scripts size: M maintainer Maintainer-authored PR labels Jul 9, 2026
@vincentkoc
vincentkoc marked this pull request as ready for review July 9, 2026 14:26
@vincentkoc
vincentkoc marked this pull request as draft July 9, 2026 14:30
@clawsweeper

clawsweeper Bot commented Jul 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed July 9, 2026, 12:13 PM ET / 16:13 UTC.

Summary
The PR expands scripts/lib/kova-report-gate.mjs, the OpenClaw Performance workflow summary text, and script/workflow tests so profiled CPU/RSS-only Kova DO_NOT_SHIP reports can be classified as tolerated.

PR surface: Tests +655, Config 0, Other +440. Total +1095 across 4 files.

Reproducibility: yes. for source-level reproducibility. The linked issue gives exact workflow inputs and failed run links, current main only accepts PARTIAL reports, and pinned Kova emits DO_NOT_SHIP for blocking OpenClaw failure cards.

Review metrics: 1 noteworthy metric.

  • Release-Gate Tolerance Modes: 1 added, 1 retained. The helper can now convert both filtered-partial and profiled-resource-only Kova nonzero verdicts into effective success, so the fail-open boundary deserves maintainer attention.

Stored data model
Persistent data-model change detected: vector/embedding metadata: scripts/lib/kova-report-gate.mjs. Confirm migration or upgrade compatibility proof before merge.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #102888
Summary: This PR is the linked candidate fix for the maintainer-authored deep-profile release-gate issue.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🦞 diamond lobster
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • Refresh or re-check exact-head proof after bringing the branch up to current main.

Risk before merge

  • [P1] The PR intentionally lets a narrow class of nonzero Kova release-gate verdicts become effective success; if the classifier is too broad, release automation could miss a real blocker.
  • [P1] The live PR is mergeable but behind current main, so maintainers should refresh or re-check exact-head proof before landing.

Maintainer options:

  1. Verify Current-Head Gate Behavior (recommended)
    Before merge, verify this head accepts the real profiled-resource-only retry report and still rejects baseline, non-resource, failed-command, cleanup, and error-status reports.
  2. Accept The Narrow Automation Risk
    Maintainers can explicitly own the release-gate tolerance if the operational release benefit outweighs the residual classifier risk.
  3. Pause For Kova Policy Instead
    Pause or close this PR if maintainers prefer solving profiler overhead in Kova thresholds or release policy rather than OpenClaw's report adapter.

Next step before merge

  • [P2] Manual review is appropriate because this maintainer-authored protected PR deliberately changes release-gate automation tolerance rather than needing a mechanical repair.

Maintainer decision needed

  • Question: Should OpenClaw allow the performance workflow to treat Kova profiled-resource-only DO_NOT_SHIP reports as effective success after current-head proof stays green?
  • Rationale: The code path is narrow and tested, but it changes a release-gate fail-open boundary that should be explicitly owned by maintainers rather than cleanup automation.
  • Likely owner: vincentkoc — vincentkoc has the strongest recent history on the OpenClaw workflow/helper plus the pinned Kova source contract used by this PR.
  • Options:
    • Accept After Exact-Head Proof (recommended): Keep this classifier and merge after the branch is refreshed or current-head proof again covers the real retry report plus baseline, non-resource, command, cleanup, and error-status negative cases.
    • Tighten The Classifier First: Ask for additional constraints or real-report fixtures if maintainers want less tolerance around profiled Kova resource failures.
    • Keep Kova Strict: Reject the tolerance and address profiler overhead in Kova thresholds or deep-profile release policy instead.

Security
Cleared: The diff changes a workflow summary string, a local JSON report adapter, and tests without changing permissions, action refs, dependency sources, secrets handling, lockfiles, or downloaded code execution.

Review details

Best possible solution:

Land the centralized adapter change after current-head proof confirms only filtered-partial and profiling-affected CPU/RSS resource failures are tolerated while baseline, functional, latency, harness, timeout, auth, cleanup, and unprofiled failures remain blocking.

Do we have a high-confidence way to reproduce the issue?

Yes for source-level reproducibility. The linked issue gives exact workflow inputs and failed run links, current main only accepts PARTIAL reports, and pinned Kova emits DO_NOT_SHIP for blocking OpenClaw failure cards.

Is this the best way to solve the issue?

Yes. The workflow already centralizes nonzero Kova verdict interpretation in scripts/lib/kova-report-gate.mjs, so a narrow adapter update is cleaner than a workflow-side bypass or broad threshold change.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 4f9a371c9530.

Label changes

Label justifications:

  • P2: This is a focused release-infrastructure bug fix with limited maintainer-facing blast radius rather than an end-user runtime emergency.
  • merge-risk: 🚨 automation: The diff changes release workflow gating logic so a Kova nonzero verdict can be treated as success under a new classifier.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: The external-contributor proof gate is not applicable to this MEMBER PR; the body and follow-up comment nevertheless cite Actions run links, Testbox validation, and a captured retry artifact accepted as profiled-resource-only.
Evidence reviewed

PR surface:

Tests +655, Config 0, Other +440. Total +1095 across 4 files.

View PR surface stats
Area Files Added Removed Net
Source 0 0 0 0
Tests 2 803 148 +655
Docs 0 0 0 0
Config 1 1 1 0
Generated 0 0 0 0
Other 1 510 70 +440
Total 4 1314 219 +1095

What I checked:

  • Current main rejects the reported verdict class: Current main only tolerates Kova reports after gate.verdict === "PARTIAL", so the linked issue's DO_NOT_SHIP profiled resource report remains blocking before this PR. (scripts/lib/kova-report-gate.mjs:27, 4f9a371c9530)
  • Workflow centralizes the nonzero verdict decision: The performance workflow converts a nonzero Kova exit to effective success only when the shared helper accepts the report JSON, so the PR is changing the existing central gate instead of adding a second workflow bypass. (.github/workflows/openclaw-performance.yml:370, 4f9a371c9530)
  • PR adds a narrow profiled-resource classifier: The PR adds evaluateToleratedProfiledKovaReport and the combined evaluateToleratedKovaReport, returning profiled-resource-only only after the stricter classifier accepts the report. (scripts/lib/kova-report-gate.mjs:497, 7e2d287db5fb)
  • Regression tests cover the positive and fail-closed paths: The PR tests the exact profiled-resource acceptance path plus negative cases for command failures, cleanup failures, non-resource violations, baseline regressions, duplicate cards, malformed summaries, and partial-report drift. (test/scripts/kova-report-gate.test.ts:394, 7e2d287db5fb)
  • Kova profiling contract supports the predicate: Pinned Kova source records profilingEnabled, profilingBaselineEligible, and profilingAffectsResourceMeasurements in report measurements, which are the dependency fields the PR checks. (openclaw/Kova/src/evaluator.mjs:807, 4f1460165830)
  • Kova gate emits the DO_NOT_SHIP failure shape: Pinned Kova gate logic marks blocking OpenClaw failure cards as DO_NOT_SHIP and builds cards from record violations and summarized resource measurements, matching the linked issue's failure mode. (openclaw/Kova/src/matrix/gate.mjs:100, 4f1460165830)

Likely related people:

  • vincentkoc: Recent GitHub path history shows repeated work on the OpenClaw performance workflow and Kova report gate helper, and the pinned Kova source ref cited by the issue is authored by vincentkoc. (role: recent area contributor and Kova contract owner; confidence: high; commits: b63317d07e8d, 74f90885f351, cf152fe76e45; files: .github/workflows/openclaw-performance.yml, scripts/lib/kova-report-gate.mjs, test/scripts/kova-report-gate.test.ts)
  • shakkernerd: Kova commit history for src/matrix/gate.mjs shows shakkernerd carried the gate logic that builds DO_NOT_SHIP and openclaw-failure cards consumed by this adapter. (role: upstream Kova gate logic contributor; confidence: medium; commits: a01122521cdd, f271da752e70; files: openclaw/Kova/src/matrix/gate.mjs, openclaw/Kova/src/evaluator.mjs)
  • steipete: OpenClaw history shows adjacent work on the performance workflow and script-test documentation around the same release validation surface. (role: adjacent workflow contributor; confidence: medium; commits: ca1829c3f4ed, 58c663920d04; files: .github/workflows/openclaw-performance.yml, test/scripts/openclaw-performance-workflow.test.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.
Review history (3 earlier review cycles)
  • reviewed 2026-07-09T14:57:08.830Z sha 2bceac1 :: needs maintainer review before merge. :: none
  • reviewed 2026-07-09T15:38:11.883Z sha 7e2d287 :: needs maintainer review before merge. :: none
  • reviewed 2026-07-09T15:45:50.590Z sha 7e2d287 :: needs maintainer review before merge. :: none

@clawsweeper clawsweeper Bot mentioned this pull request Jul 9, 2026
Closed
@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. P2 Normal backlog priority with limited blast radius. merge-risk: 🚨 automation 🚨 May affect CI, automerge, proof capture, label sync, or maintainer automation. labels Jul 9, 2026
@vincentkoc
vincentkoc force-pushed the fix/performance-profiled-release-gate branch from 2bceac1 to 7e2d287 Compare July 9, 2026 15:13
@vincentkoc
vincentkoc marked this pull request as ready for review July 9, 2026 15:33
@vincentkoc

vincentkoc commented Jul 9, 2026

Copy link
Copy Markdown
Member Author

Exact-head validation is complete for 7e2d287db5fb4641608f8c032e118814490a601f.

Root cause: Kova deep profiling marks CPU/RSS as instrumentation-affected and baseline-ineligible, but its evaluator still applies the normal 900 MB RSS threshold. The same candidate stayed at 692-807 MB in normal/live runs and reached 923-925 MB only with CPU, heap, and trace profiling enabled.

Proof:

  • Blacksmith Testbox tbx_01kx3pr5ej845fx4j5j0rt0sdy: 91 focused gate/workflow tests passed.
  • The same exact-head box passed check:changed for the four-file tooling lane with zero lint errors.
  • The captured retry artifact is accepted as profiled-resource-only.
  • Command, timeout, cleanup, identity, baseline, malformed-summary, unrelated-group, and non-resource mutations all remain rejected.
  • Fresh autoreview completed with no actionable findings.

The exception remains narrow: normal and live performance failures still block; only canonical deep-profiled resource-only failures can be tolerated.

Fixes #102888.

@vincentkoc vincentkoc closed this Jul 9, 2026
@vincentkoc vincentkoc reopened this Jul 9, 2026
@vincentkoc
vincentkoc merged commit d18e48c into main Jul 9, 2026
265 of 269 checks passed
@vincentkoc
vincentkoc deleted the fix/performance-profiled-release-gate branch July 9, 2026 16:22
@vincentkoc

vincentkoc commented Jul 9, 2026

Copy link
Copy Markdown
Member Author

Merged via squash.

@vincentkoc vincentkoc mentioned this pull request Jul 9, 2026
Merged
Simon-XYDT pushed a commit to Simon-XYDT/openclaw that referenced this pull request Jul 9, 2026
@vincentkoc @Simon-XYDT
fix(ci): ignore profiling-only resource gate failures (openclaw#102894)
dbdfa9b
@github-actions github-actions Bot mentioned this pull request Jul 9, 2026
Open
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jul 10, 2026
@vincentkoc
fix(ci): ignore profiling-only resource gate failures (openclaw#102894)
d808db8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@vincentkoc vincentkoc

Labels

maintainer Maintainer-authored PR merge-risk: 🚨 automation 🚨 May affect CI, automerge, proof capture, label sync, or maintainer automation. P2 Normal backlog priority with limited blast radius. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. scripts Repository scripts size: XL status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Bug]: Deep-profile release gate treats instrumented RSS as baseline-eligible

1 participant

@vincentkoc