Skip to content

fix(gateway): disable provider auth prewarm by default#86564

Closed
wesleysimplicio wants to merge 1 commit into
openclaw:mainfrom
wesleysimplicio:feat/gateway-disable-provider-auth-prewarm-default-pr
Closed

fix(gateway): disable provider auth prewarm by default#86564
wesleysimplicio wants to merge 1 commit into
openclaw:mainfrom
wesleysimplicio:feat/gateway-disable-provider-auth-prewarm-default-pr

Conversation

@wesleysimplicio

@wesleysimplicio wesleysimplicio commented May 25, 2026 •
edited by vincentkoc
Loading

Copy link
Copy Markdown

Summary

  • Skip the provider-auth startup warm on the normal gateway path unless providerAuthPrewarm.enabled === true.
  • Keep the gateway lifetime provider-auth sidecar installed when the runtime supplies providerAuthPrewarm, so auth-profile failures still clear stale prepared state and schedule a delayed rewarm.
  • Add a regression test for the default hook-only path.

Root Cause

The post-attach startup flow treated providerAuthPrewarm as enabled unless callers explicitly set enabled: false. The normal gateway caller supplies providerAuthPrewarm for live config access, so startup still scheduled provider-auth warm work by default. The previous PR version fixed the startup warm by disabling the entire sidecar, which also removed auth-profile failure invalidation.

Fix

  • Keep enabled: true as the explicit startup-warm opt-in.
  • Treat omitted enabled as hook-only: register the auth failure hook and rewarm sidecar, but do not schedule startup warm.
  • Keep enabled: false as the test/minimal disable path.

Real behavior proof

Behavior addressed: gateway startup no longer kicks off provider-auth warm by default, while auth-profile failures still clear and rewarm prepared provider-auth state.

Real environment tested: AWS Crabbox Linux runner, lease cbx_7966d6c753d7, run run_5d2dd60d1672.

Exact steps or command run after this patch: node scripts/crabbox-wrapper.mjs run --provider aws --label wsl-event-loop-86752-check-changed --idle-timeout 45m --ttl 90m --timing-json --stop-after always --shell -- 'pnpm check:changed'.

Evidence after fix: pnpm check:changed exited 0. The changed gate selected core and coreTests, then ran conflict-marker guard, dependency guards, core typecheck, core test typecheck, changed-file lint, runtime sidecar loader guard, import-cycle check, webhook body guard, and pairing guards.

Observed result after fix: the focused regression path keeps the provider-auth failure hook active without calling warmCurrentProviderAuthStateOffMainThread on startup; triggering the auth failure hook clears current provider-auth state and performs the delayed rewarm.

What was not tested: native Windows and WSL2 project tests did not run. AWS native Windows had no node runtime in PATH (run_a044f871e2b3), the tooling probe also found no node, choco, winget, or pwsh (run_498feccc91fb), AWS WSL2 hydration exited before its marker (run_9230eb79e292), and Azure WSL2 allocation did not produce a lease after more than 10 minutes (run_6a3979ab4680).

Verification

  • node scripts/run-vitest.mjs src/gateway/server-startup-post-attach.test.ts at 187cf1f85e4: 46 passed
  • git diff --check origin/main...HEAD
  • node scripts/run-oxlint.mjs src/gateway/server-startup-post-attach.ts src/gateway/server-startup-post-attach.test.ts
  • autoreview --mode local: clean, no accepted/actionable findings
  • Crabbox AWS changed gate: run_5d2dd60d1672, lease cbx_7966d6c753d7, pnpm check:changed, exit 0

Related PRs / issues

Fixes #86512
Refs #86752

@openclaw-barnacle openclaw-barnacle Bot added gateway Gateway runtime size: XS triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels May 25, 2026
@clawsweeper

clawsweeper Bot commented May 25, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed June 24, 2026, 3:13 PM ET / 19:13 UTC.

Summary
The branch makes gateway provider-auth startup warming explicit opt-in while keeping the auth-profile failure rewarm sidecar and adding a regression test for the hook-only default path.

PR surface: Source +4, Tests +53. Total +57 across 2 files.

Reproducibility: yes. for the central startup-scheduling defect by source inspection: current main and v2026.6.10 pass providerAuthPrewarm without enabled and schedule the delayed warm sweep by default. I did not run a live CPU or latency reproduction in this read-only review.

Review metrics: 1 noteworthy metric.

  • Startup Auth Default: 1 default changed, 0 config keys added. The PR changes normal gateway startup behavior without adding a user-facing setting or migration, so maintainers need to accept the upgrade behavior before merge.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #86512
Summary: This PR is a candidate fix for the macOS v2026.5.22 provider-auth startup CPU/default-prewarm report; sibling items add platform-specific and broader startup-event-loop evidence.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🦞 diamond lobster
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

Risk before merge

  • [P1] Changing startup provider-auth warm from default-on to explicit opt-in can defer prepared auth discovery for upgraded gateways until an auth failure, config reload, logout, or on-demand auth check.
  • [P1] This PR and fix(gateway): avoid default provider auth startup prewarm #88514 carry the same fix shape, so maintainers need to choose and refresh one branch rather than land both.
  • [P2] The narrow default-prewarm change does not prove all linked Docker/WSL2, Telegram, cached-context, embedded-runtime startup, or local RPC timeout symptoms are solved.

Maintainer options:

  1. Choose And Refresh One Branch (recommended)
    Select this PR or fix(gateway): avoid default provider auth startup prewarm #88514 as canonical, rebase or resolve conflicts, and preserve the hook-only default plus explicit-enabled tests before merge.
  2. Accept Lazy Auth Startup
    Merge the refreshed branch only if maintainers intentionally accept moving default provider-auth state population from startup to explicit opt-in or failure-triggered rewarm.
  3. Keep Eager Startup Warm
    Pause or close this direction if maintainers decide eager provider-auth readiness is more important than avoiding unsolicited startup sweep work.

Next step before merge

  • [P2] Human review is needed because the remaining action is maintainer acceptance of a compatibility-sensitive auth-provider startup default plus branch/canonical-PR resolution, not a narrow automated repair.

Security
Cleared: The diff changes internal gateway startup scheduling and a colocated test only; it adds no dependencies, workflows, package metadata, credential storage, or new code-execution surface.

Review details

Best possible solution:

Consolidate on one refreshed hook-only/default-off provider-auth startup branch, preserve explicit enabled coverage and failure-triggered rewarm, and track broader startup latency symptoms separately.

Do we have a high-confidence way to reproduce the issue?

Yes for the central startup-scheduling defect by source inspection: current main and v2026.6.10 pass providerAuthPrewarm without enabled and schedule the delayed warm sweep by default. I did not run a live CPU or latency reproduction in this read-only review.

Is this the best way to solve the issue?

Yes if maintainers accept the compatibility tradeoff: the PR targets the scheduler that creates the default warm, preserves explicit startup warming and auth-profile failure rewarm, and adds focused coverage. It still needs canonical branch selection and refresh before merge.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against eac1d3349c29.

Label changes

Label justifications:

  • P1: The PR targets a user-reported gateway CPU/event-loop regression that can affect real gateway, agent, and channel workflows.
  • merge-risk: 🚨 compatibility: Existing upgraded gateways that relied on eager provider-auth state may see lazy discovery instead of startup-prepared auth state.
  • merge-risk: 🚨 auth-provider: The diff changes when prepared provider-auth state is populated and refreshed for model/auth status paths.
  • merge-risk: 🚨 availability: If expensive provider/auth discovery is deferred to a later interaction, some setups may move startup pressure into first-use latency or stalls.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Override: A maintainer applied proof: override for this PR.
Evidence reviewed

PR surface:

Source +4, Tests +53. Total +57 across 2 files.

View PR surface stats
Area Files Added Removed Net
Source 1 28 24 +4
Tests 1 53 0 +53
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 2 81 24 +57

Acceptance criteria:

  • [P1] node scripts/run-vitest.mjs src/gateway/server-startup-post-attach.test.ts src/agents/model-provider-auth.test.ts -- --run.
  • [P1] node scripts/run-oxlint.mjs src/gateway/server-startup-post-attach.ts src/gateway/server-startup-post-attach.test.ts src/agents/model-provider-auth.ts src/agents/model-provider-auth.test.ts.
  • [P1] git diff --check origin/main...HEAD.
  • [P1] Crabbox gateway RPC proof on the chosen refreshed branch.

What I checked:

Likely related people:

  • vincentkoc: Assigned to this PR, force-pushed the current head, and authored the overlapping maintainer PR with the same default-off startup-prewarm contract. (role: likely follow-up owner; confidence: high; commits: 187cf1f85e40, 165b2c6620e4; files: src/gateway/server-startup-post-attach.ts, src/gateway/server-startup-post-attach.test.ts)
  • sjf: GitHub commit metadata maps the startup provider-auth prewarm optimization to sjf, and the commit message names the one-time startup warm cost tradeoff. (role: introduced behavior; confidence: high; commits: 120a262dbcd1; files: src/agents/model-provider-auth.ts, src/gateway/server-startup-post-attach.ts, src/gateway/server-reload-handlers.ts)
  • samzong: Authored the merged worker-thread provider-auth warm PR that current main uses, which mitigated main-event-loop blocking but left default startup scheduling in place. (role: related mitigation author; confidence: high; commits: 316fd5b62572, 86626d3ed02b; files: src/agents/model-provider-auth.ts, src/agents/model-provider-auth.worker.ts, src/gateway/server-startup-post-attach.ts)
  • steipete: Merged the worker-thread provider-auth mitigation and authored recent gateway startup event-loop stall work adjacent to this scheduling tradeoff. (role: recent adjacent contributor and merger; confidence: medium; commits: 3e8fd4944fe1, 316fd5b62572; files: src/gateway/server-startup-post-attach.ts, src/gateway/server.impl.ts, src/agents/model-provider-auth.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P1 High-priority user-facing bug, regression, or broken workflow. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 auth-provider 🚨 May break OAuth, tokens, provider routing, model choice, or credentials. merge-risk: 🚨 availability 🚨 May cause crashes, hangs, restart loops, stalls, or process outages. labels May 25, 2026
@clawsweeper

clawsweeper Bot commented May 25, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper PR egg

🎁 Pass real behavior proof to wake the egg and unlock a hatchable treat.

Where did the egg go?
  • The egg game starts only after the PR passes the real-behavior proof check.
  • Before that, no creature or rarity is rolled. The treat waits for real proof.
  • This is still just collectible flavor: proof affects review readiness, not creature quality.

@omarshahine

omarshahine commented May 25, 2026

Copy link
Copy Markdown
Contributor

Thanks for the PR. Before this can move forward, please add live proof from the affected surface, not just unit tests, mocked tests, or source inspection.

A useful proof update should include:

  • the exact build/SHA tested
  • the real environment used
  • the command, UI flow, channel flow, provider request, or other live path exercised
  • before/after symptom evidence where applicable
  • the observed result after the patch
  • any remaining proof gaps

Please redact secrets, tokens, phone numbers, and private message content from logs or screenshots.

@wesleysimplicio

wesleysimplicio commented May 26, 2026

Copy link
Copy Markdown
Author

Live proof from the built tree (SHA 017c0c7, pnpm build completed):

Command:
/opt/homebrew/bin/node openclaw.mjs gateway run --dev --allow-unconfigured --force --verbose --ws-log compact --port 19002

Observed runtime result:

  • gateway reached [gateway] ready
  • startup completed and the process stayed healthy until SIGTERM
  • the startup log did not emit provider auth state pre-warmed or provider auth state re-warmed

This is the real runtime execution requested for the provider-auth startup path.

@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. and removed status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels May 29, 2026
@openclaw-barnacle openclaw-barnacle Bot removed the triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. label May 29, 2026
@vincentkoc
vincentkoc force-pushed the feat/gateway-disable-provider-auth-prewarm-default-pr branch from 017c0c7 to 187cf1f Compare May 31, 2026 01:15
@vincentkoc vincentkoc self-assigned this May 31, 2026
@vincentkoc

vincentkoc commented May 31, 2026

Copy link
Copy Markdown
Member

Maintainer update: I rebased and narrowed this PR so the default path skips only the startup provider-auth warm. The auth-profile failure hook and delayed rewarm sidecar stay installed, which addresses the ClawSweeper blocker about dropping stale-auth invalidation.

Verification:

  • node scripts/run-vitest.mjs src/gateway/server-startup-post-attach.test.ts at 187cf1f85e4: 46 passed
  • git diff --check origin/main...HEAD
  • node scripts/run-oxlint.mjs src/gateway/server-startup-post-attach.ts src/gateway/server-startup-post-attach.test.ts
  • autoreview --mode local: clean, no accepted/actionable findings
  • Crabbox AWS changed gate: run_5d2dd60d1672, lease cbx_7966d6c753d7, pnpm check:changed, exit 0

Windows / WSL proof gap:

  • AWS native Windows run_a044f871e2b3: raw image had no node; project code did not run.
  • AWS native Windows tooling probe run_498feccc91fb: no node, choco, winget, or pwsh in PATH; only Windows PowerShell 5.1 was available.
  • AWS WSL2 run_9230eb79e292: WSL2 bootstrap synced, then local Actions hydration exited before writing its marker; project code did not run.
  • Azure WSL2 run_6a3979ab4680: coordinator allocation waited more than 10 minutes without a lease; I stopped the local attempt.

@vincentkoc vincentkoc added proof: override Maintainer override for the external PR real behavior proof gate. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. labels May 31, 2026
@clawsweeper clawsweeper Bot removed proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. labels May 31, 2026
@clawsweeper clawsweeper Bot added the rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. label May 31, 2026
This was referenced Jun 15, 2026
Closed
Closed
Closed
Closed
@openclaw-barnacle

openclaw-barnacle Bot commented Jun 22, 2026

Copy link
Copy Markdown

This assigned pull request has been automatically marked as stale after being open for 27 days.
Please add updates or it will be closed.

@openclaw-barnacle openclaw-barnacle Bot added the stale Marked as stale due to inactivity label Jun 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@vincentkoc vincentkoc

Labels

gateway Gateway runtime merge-risk: 🚨 auth-provider 🚨 May break OAuth, tokens, provider routing, model choice, or credentials. merge-risk: 🚨 availability 🚨 May cause crashes, hangs, restart loops, stalls, or process outages. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. P1 High-priority user-facing bug, regression, or broken workflow. proof: override Maintainer override for the external PR real behavior proof gate. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. size: S stale Marked as stale due to inactivity status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Bug] v2026.5.22 启动后 Gateway CPU 持续 100%+，导致请求延迟飙升 50-100 倍

4 participants

@wesleysimplicio @omarshahine @vincentkoc @RomneyDa