Skip to content

fix(qa): prevent qa smoke ci timeouts under gateway concurrency#99368

Merged
RomneyDa merged 1 commit into
mainfrom
fix/qa-disable-startup-prewarms
Jul 4, 2026
Merged

fix(qa): prevent qa smoke ci timeouts under gateway concurrency#99368
RomneyDa merged 1 commit into
mainfrom
fix/qa-disable-startup-prewarms

Conversation

@RomneyDa

@RomneyDa RomneyDa commented Jul 3, 2026

Copy link
Copy Markdown
Member

What Problem This Solves

Fixes an issue where the QA smoke profile could intermittently time out waiting for gateway readiness or native command responses when multiple isolated gateways ran concurrently on a low-CPU runner.

The observed failures included runtime-inventory-drift-check timing out on channel readiness and native-command-session-target timing out on /stop, while provider-auth prewarm telemetry showed multi-second event-loop stalls.

Why This Change Was Made

Each QA child gateway scheduled optional startup model discovery and a provider-auth worker shortly after readiness. With eight gateways on a two-vCPU runner, those background warmups overlapped and starved foreground gateway work.

QA child gateways now skip the optional initial startup warmups. Production behavior is unchanged, and provider-auth failure invalidation plus failure-triggered rewarming remain enabled so QA retains production runtime semantics after startup.

User Impact

No product behavior changes. QA smoke runs should be more reliable under high concurrency and should be neutral-to-faster because they no longer spend CPU on background cache warming that the deterministic mock scenarios do not require.

Evidence

  • pnpm exec vitest run src/gateway/server-startup.test.ts src/gateway/server-startup-post-attach.test.ts extensions/qa-lab/src/gateway-child.test.ts — 122 tests passed.
  • pnpm check — passed.
  • Four QA suites launched simultaneously, each running runtime-inventory-drift-check and native-command-session-target at concurrency 2 — all eight gateway scenarios passed.
  • Concurrent two-scenario suite wall times were 21.5–22.4 seconds after the final fix.
  • Autoreview initially identified that disabling the whole provider-auth sidecar would also remove failure invalidation; the implementation was corrected to suppress only the startup timer. Final autoreview returned no actionable findings.

@openclaw-barnacle openclaw-barnacle Bot added gateway Gateway runtime extensions: qa-lab size: S maintainer Maintainer-authored PR labels Jul 3, 2026
@RomneyDa RomneyDa changed the title fix(qa): prevent smoke timeouts under gateway concurrency fix(qa): prevent qa smoke ci timeouts under gateway concurrency Jul 3, 2026
@clawsweeper

clawsweeper Bot commented Jul 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed July 3, 2026, 6:28 PM ET / 22:28 UTC.

Summary
The branch adds QA-child environment gates to skip startup model and provider-auth prewarm work, and threads a provider-auth startup switch through gateway post-attach startup with regression coverage.

PR surface: Source +23, Tests +55. Total +78 across 6 files.

Reproducibility: yes. with medium confidence rather than a deterministic local repro. The earlier CI artifact and PR body show intermittent QA smoke timeouts under concurrent gateway startup, and the current source shows the startup warmup timers that the PR disables for QA child gateways.

Review metrics: 2 noteworthy metrics.

  • Internal QA env gates: 2 added. They change QA child gateway startup behavior, so maintainers should notice the automation coverage tradeoff before merge.
  • Named target scenarios: 2 passed in latest QA Smoke CI rerun. The earlier review blocker was a failure in these named smoke paths, and the latest head rerun shows both now passing.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🦞 diamond lobster
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • none.

Risk before merge

  • [P1] The PR adds internal QA child env gates and suppresses optional startup warmups in QA child gateways; that is an automation coverage tradeoff even though production defaults remain enabled and the latest smoke run passed.

Maintainer options:

  1. Land with current smoke proof (recommended)
    Maintainers can accept the QA-only startup-warmup coverage tradeoff because focused tests preserve failure-triggered rewarm and the latest QA Smoke CI rerun passed on the PR head.
  2. Require a final smoke refresh
    If maintainers want extra assurance after any base refresh, rerun QA Smoke CI on the final merge result before landing.

Next step before merge

  • [P2] No repair lane is needed; this protected maintainer PR has no current discrete patch defect, so the remaining action is maintainer merge/label handling.

Security
Cleared: Cleared: the diff does not change dependencies, workflows, package scripts, secrets handling, external artifact execution, or runtime security boundaries.

Review details

Best possible solution:

Keep the QA-only warmup suppression with focused regression tests and fresh full smoke proof, while leaving production startup prewarms enabled by default.

Do we have a high-confidence way to reproduce the issue?

Yes, with medium confidence rather than a deterministic local repro. The earlier CI artifact and PR body show intermittent QA smoke timeouts under concurrent gateway startup, and the current source shows the startup warmup timers that the PR disables for QA child gateways.

Is this the best way to solve the issue?

Yes. The PR is a narrow QA-only mitigation that preserves production defaults and failure-triggered provider-auth rewarm; broader scheduler/stagger fixes were already attempted in related QA smoke work.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 1cab479f7814.

Label changes

Label changes:

  • add rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
  • add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: Maintainer-authored PR, so the external contributor proof gate does not apply; the PR body and live QA Smoke CI run still provide validation evidence.
  • remove rating: 🧂 unranked krab: Current PR rating is rating: 🐚 platinum hermit, so this older rating label is no longer current.
  • remove status: ⏳ waiting on author: Current PR status label is status: 👀 ready for maintainer look.

Label justifications:

  • P2: The PR targets QA Smoke CI reliability under gateway concurrency with limited product-runtime blast radius.
  • merge-risk: 🚨 automation: The diff changes QA child gateway startup behavior and therefore can affect CI/proof behavior even though production startup defaults remain enabled.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: Maintainer-authored PR, so the external contributor proof gate does not apply; the PR body and live QA Smoke CI run still provide validation evidence.
Evidence reviewed

PR surface:

Source +23, Tests +55. Total +78 across 6 files.

View PR surface stats
Area Files Added Removed Net
Source 3 25 2 +23
Tests 3 56 1 +55
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 6 81 3 +78

What I checked:

  • Repository policy read: Root AGENTS.md and the scoped gateway/extensions guides were read fully; their deep PR review, protected maintainer item, and QA/gateway startup guidance apply. (AGENTS.md:24, 1cab479f7814)
  • Current main provider-auth startup behavior: Current main registers the provider-auth failure hook and always schedules the startup prewarm timer from the gateway lifetime sidecar path. (src/gateway/server-startup-post-attach.ts:196, 1cab479f7814)
  • Current main model-prewarm skip precedent: Current main already has an env-gated startup model prewarm skip, which the PR mirrors for provider-auth startup prewarm in QA child gateways. (src/gateway/server-startup-post-attach.ts:642, 1cab479f7814)
  • PR implementation shape: The PR sets the startup model and provider-auth skip env vars only in the QA child runtime environment and adds a startupEnabled switch that returns before scheduling only the provider-auth startup timer. (extensions/qa-lab/src/gateway-child.ts:228, 04ba7d33ea37)
  • Failure-triggered rewarm preserved: The PR's added test asserts that disabling provider-auth startup prewarm still registers the failure hook, clears auth state on failure, and runs a delayed rewarm. (src/gateway/server-startup-post-attach.test.ts:1045, 04ba7d33ea37)
  • Latest target CI proof: The latest QA Smoke CI job on head 04ba7d3 succeeded; the log shows both runtime-inventory-drift-check and native-command-session-target passing, with the final suite summary passed=24 failed=0 total=24. (04ba7d33ea37)

Likely related people:

  • RomneyDa: Authored this branch and recent merged QA smoke/concurrency stabilization PRs that changed the same QA smoke execution area. (role: recent QA smoke and gateway-concurrency contributor; confidence: high; commits: 04ba7d33ea37, 438f208a7666, 58c92df58837; files: extensions/qa-lab/src/gateway-child.ts, src/gateway/server-startup-post-attach.ts, src/gateway/server.impl.ts)
  • Vincent Koc: Git history for the current startup/provider-auth surface includes recent work touching the gateway startup and QA child files now under review. (role: recent gateway startup/provider-auth contributor; confidence: medium; commits: e085fa1a3ffd; files: src/gateway/server-startup-post-attach.ts, src/gateway/server.impl.ts, extensions/qa-lab/src/gateway-child.ts)
  • steipete: Older history shows gateway startup splitting and QA child runtime hardening in the same area, useful as adjacent routing context if the startup boundary needs follow-up. (role: historical gateway and QA startup contributor; confidence: medium; commits: 8de63ca268, a00b01f5ed, 5cf01ac7c1; files: src/gateway/server-startup-post-attach.ts, src/gateway/server.impl.ts, extensions/qa-lab/src/gateway-child.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. P2 Normal backlog priority with limited blast radius. merge-risk: 🚨 automation 🚨 May affect CI, automerge, proof capture, label sync, or maintainer automation. labels Jul 3, 2026
@yaoyi1222 yaoyi1222 mentioned this pull request Jul 3, 2026
Closed
This was referenced Jul 3, 2026
Closed
Merged
@RomneyDa

RomneyDa commented Jul 3, 2026

Copy link
Copy Markdown
Member Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jul 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. labels Jul 3, 2026
This was referenced Jul 4, 2026
Closed
Merged
Merged
@RomneyDa
RomneyDa merged commit 19035bd into main Jul 4, 2026
232 of 242 checks passed
@RomneyDa
RomneyDa deleted the fix/qa-disable-startup-prewarms branch July 4, 2026 03:15
@RomneyDa RomneyDa mentioned this pull request Jul 4, 2026
Merged
@github-actions github-actions Bot mentioned this pull request Jul 4, 2026
Open
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jul 4, 2026
@RomneyDa
fix(qa): avoid startup prewarm contention (openclaw#99368)
e43e453
@0xopaque 0xopaque mentioned this pull request Jul 4, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

extensions: qa-lab gateway Gateway runtime maintainer Maintainer-authored PR merge-risk: 🚨 automation 🚨 May affect CI, automerge, proof capture, label sync, or maintainer automation. P2 Normal backlog priority with limited blast radius. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. size: S status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@RomneyDa