Skip to content

fix(gateway): avoid default provider auth startup prewarm#88514

Closed
vincentkoc wants to merge 1 commit into
mainfrom
fix-wsl-event-loop-starvation-86752
Closed

fix(gateway): avoid default provider auth startup prewarm#88514
vincentkoc wants to merge 1 commit into
mainfrom
fix-wsl-event-loop-starvation-86752

Conversation

@vincentkoc

@vincentkoc vincentkoc commented May 31, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Summary

This PR stops the gateway from running the provider-auth warm sweep by default during startup. The issue report shows Docker/WSL2 gateway event-loop starvation, local RPC timeouts, and a 284s provider-auth prewarm on 2026.5.22; current main had moved the warm path off-main-thread, but still scheduled the warm sweep automatically after ready.

The intended outcome is narrower: default startup should keep the gateway responsive and avoid unsolicited provider-auth sweep work, while explicit callers can still opt into startup warming and auth-profile failure events still trigger a rewarm.

Out of scope: this does not solve every context-size or runtime-startup latency signal from the issue, and native Windows proof here uses an isolated foreground gateway with no live provider credentials or channel accounts.

Reviewers should focus on the startup contract in src/gateway/server-startup-post-attach.ts: default startup no longer schedules provider-auth prewarm, but the failure hook and explicit enabled: true path remain covered.

Linked context

Closes #86752

Related #85999, #86201, #86512, #86073.

Maintainer-requested as part of the Windows / WSL issue sweep from #74163.

Real behavior proof (required for external PRs)

  • Behavior or issue addressed: WSL2/Docker-style and native Windows gateway startup should avoid the default provider-auth warm sweep that can starve gateway RPCs and channel timers.
  • Real environment tested: AWS Crabbox Windows WSL2 lease cbx_be71611ba83d (harbor-lobster), Node v22.22.2, pnpm 11.2.2; AWS Crabbox native Windows lease cbx_8ccfe5ff4900 (native-win-pr74163-aws), Node v24.16.0, pnpm 11.2.2, PowerShell 5.1.20348.4294, Windows 10.0.20348.5139.
  • Exact steps or command run after this patch: started a foreground gateway in WSL2 with token auth, OPENCLAW_GATEWAY_STARTUP_TRACE=1, then ran authenticated gateway status --require-rpc and direct gateway call status against the live gateway. On native Windows, hydrated the PR branch through hydrate-windows-daemon, started an isolated foreground gateway with token auth and OPENCLAW_GATEWAY_STARTUP_TRACE=1, then ran authenticated gateway status --require-rpc --json against the live gateway.
  • Evidence after fix: WSL2 Crabbox run run_76eb41a23a64 reported rpc.ok: true, direct status RPC returned runtime status, gateway reached [gateway] ready, and startup logs did not contain provider auth state pre-warmed. Native Windows Crabbox run run_0b72afe5ff48 reported rpc.ok: true, gateway reached [gateway] ready, startup trace showed eventLoopMax=0.0ms, and startup logs did not contain provider auth state pre-warmed.
  • Observed result after fix: the gateway became RPC-ready and served status requests while skipping default startup provider-auth prewarm on both WSL2 and native Windows proof lanes.
  • What was not tested: live native Windows provider credentials, real channel accounts, and the broader context-size/runtime-startup latency cluster were not exercised by this narrow startup-prewarm change.
  • Before evidence: [Bug]: 2026.5.22 Docker/WSL2 gateway event-loop starvation, 284s provider-auth prewarm, slow Telegram turn, and local RPC timeouts #86752 includes a 2026.5.22 Docker/WSL2 report with about 284s provider-auth prewarm, local node RPC timeouts, delayed Telegram timer firing, and slow Telegram turn latency.

Tests and validation

Commands run:

  • node scripts/run-vitest.mjs src/gateway/server-startup-post-attach.test.ts src/agents/model-provider-auth.test.ts -- --run
  • node scripts/run-oxlint.mjs src/gateway/server-startup-post-attach.ts src/gateway/server-startup-post-attach.test.ts src/agents/model-provider-auth.ts src/agents/model-provider-auth.test.ts
  • git diff --check origin/main..HEAD
  • .agents/skills/autoreview/scripts/autoreview --mode branch --base origin/main
  • AWS Crabbox WSL2 run_101b08aeda52: targeted Vitest and oxlint in WSL2 after hydrating Node/pnpm.
  • AWS Crabbox WSL2 run_76eb41a23a64: authenticated live gateway RPC proof.
  • AWS Crabbox native Windows Actions hydrate 26703936119: hydrate-windows-daemon on branch fix-wsl-event-loop-starvation-86752.
  • AWS Crabbox native Windows run_db0c5343dbb9: node scripts/run-vitest.mjs src/gateway/server-startup-post-attach.test.ts -- --run passed 46 tests.
  • AWS Crabbox native Windows run_0b72afe5ff48: authenticated foreground gateway RPC proof.
  • AWS Crabbox native Windows run_612f3d317f98: pnpm check:changed passed after deepening the shallow checkout so origin/main...HEAD had a merge base.

Regression coverage added:

  • Added coverage that startup can register provider-auth failure rewarm without scheduling default startup prewarm.
  • Existing explicit enabled: true prewarm tests continue to cover the opt-in startup warm path.

What failed before this fix:

  • Current main still scheduled provider-auth startup warm by default after ready, matching the issue's remaining implicated path even after earlier worker-thread mitigations.

If no test was added, why not?

  • A focused regression test was added for the changed startup scheduling contract.

Risk checklist

Did user-visible behavior change? (Yes/No)

Yes. Gateway startup no longer performs default provider-auth prewarm unless explicitly enabled.

Did config, environment, or migration behavior change? (Yes/No)

No config or migration shape changes.

Did security, auth, secrets, network, or tool execution behavior change? (Yes/No)

Yes. This changes when provider-auth state is warmed, but not the auth decision path itself.

What is the highest-risk area?

A provider account state that previously became available from default startup prewarm may now be computed lazily or after an auth-profile failure event.

How is that risk mitigated?

The explicit enabled: true startup warm path remains available and tested, and the auth-profile failure rewarm hook still runs even when default startup prewarm is disabled.

Current review state

What is the next action?

Maintainer review the behavioral tradeoff: default startup responsiveness versus eager provider-auth state. The earlier native Windows proof gap is now covered by native Windows targeted tests, check:changed, and foreground gateway RPC proof.

What is still waiting on author, maintainer, CI, or external proof?

Human maintainer review of the startup behavior tradeoff. The WSL2 proof, native Windows proof, targeted tests, lint, changed gate, diff check, and autoreview are complete.

Which bot or reviewer comments were addressed?

Local autoreview completed cleanly with no accepted/actionable findings. The ClawSweeper native Windows proof caveat has been addressed with the native Windows proof runs listed above.

@vincentkoc vincentkoc self-assigned this May 31, 2026
@openclaw-barnacle openclaw-barnacle Bot added gateway Gateway runtime size: S maintainer Maintainer-authored PR labels May 31, 2026
@clawsweeper

clawsweeper Bot commented May 31, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed June 29, 2026, 6:25 PM ET / 22:25 UTC.

Summary
The PR makes gateway provider-auth startup warming explicit opt-in while keeping the auth-profile failure rewarm sidecar and adding regression coverage for the hook-only default path.

PR surface: Source +4, Tests +53. Total +57 across 2 files.

Reproducibility: yes. for the startup-scheduling defect from source: current main passes providerAuthPrewarm without enabled and schedules startup warm by omission. I did not run a live reproduction of the exact 284s Docker/WSL2 timing in this read-only review.

Review metrics: 1 noteworthy metric.

  • Startup Auth Default: 1 changed default, 0 config keys added. The PR changes normal gateway startup auth behavior without adding a user-facing setting or migration, so maintainers need to accept the upgrade behavior before merge.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #86752
Summary: This PR is the candidate fix for the Docker/WSL2 provider-auth startup-prewarm report; a sibling PR carries the same hook-only/default-off contract for a related high-CPU report, while broader startup event-loop saturation remains partial-overlap work.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🦞 diamond lobster
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • Refresh the branch against current main and clear the CONFLICTING/DIRTY state.
  • [P2] Record maintainer acceptance of the lazy/default-off provider-auth startup contract before merge.

Risk before merge

  • [P1] Changing provider-auth startup warm from default-on to explicit opt-in can defer prepared auth discovery for upgraded gateways until an auth failure, reload, logout, or on-demand auth check.
  • [P1] The PR is draft and GitHub reports CONFLICTING/DIRTY, so maintainers need a careful refresh that preserves current main's gateway startup and provider-auth changes.
  • [P2] The linked Docker/WSL2 report includes broader cached-context, embedded-runtime startup, Telegram latency, and local RPC timeout symptoms that this narrow default-prewarm change does not prove solved.

Maintainer options:

  1. Refresh And Accept Lazy Startup Auth (recommended)
    Refresh the branch against current main and merge only after maintainers explicitly accept moving default provider-auth warming from eager startup work to hook-triggered or explicit opt-in work.
  2. Require Upgrade Auth Proof
    Ask for fresh-install and upgrade proof showing setups that relied on eager prepared provider-auth state still behave acceptably or have a documented explicit opt-in path.
  3. Keep Eager Startup Warm
    Pause or close this approach if maintainers decide first-call/model-auth latency protection is more important than removing unsolicited startup provider-auth sweep work.

Next step before merge

  • [P2] Manual review is needed because the remaining blockers are maintainer acceptance of a compatibility-sensitive auth-provider startup default, canonical branch choice, and a draft/conflicting branch refresh.

Security
Cleared: The diff changes internal gateway startup scheduling and a colocated test; it adds no dependencies, workflows, package metadata, credential storage, or new code-execution surface.

Review details

Best possible solution:

Refresh one canonical branch onto current main, preserve explicit startup warming plus auth-failure rewarm, and merge only after maintainers accept lazy/default-off provider-auth startup while tracking broader startup latency separately.

Do we have a high-confidence way to reproduce the issue?

Yes for the startup-scheduling defect from source: current main passes providerAuthPrewarm without enabled and schedules startup warm by omission. I did not run a live reproduction of the exact 284s Docker/WSL2 timing in this read-only review.

Is this the best way to solve the issue?

Yes, if maintainers accept the compatibility tradeoff. The PR targets the scheduler that creates unsolicited startup warm work, preserves explicit startup warming and auth-profile failure rewarm, and avoids adding a public config or env knob.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 843ad143647e.

Label changes

Label justifications:

  • P1: The PR targets a user-reported gateway startup regression that can affect RPC responsiveness and channel workflows in real deployments.
  • merge-risk: 🚨 compatibility: Existing upgraded gateways that relied on eager provider-auth state may see lazy discovery instead of startup-prepared auth state.
  • merge-risk: 🚨 auth-provider: The diff changes when prepared provider-auth state is populated during gateway startup.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (logs): The PR body includes after-fix WSL2 and native Windows Crabbox gateway RPC/log proof showing RPC readiness while the default provider-auth prewarm marker is absent.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR body includes after-fix WSL2 and native Windows Crabbox gateway RPC/log proof showing RPC readiness while the default provider-auth prewarm marker is absent.
Evidence reviewed

PR surface:

Source +4, Tests +53. Total +57 across 2 files.

View PR surface stats
Area Files Added Removed Net
Source 1 28 24 +4
Tests 1 53 0 +53
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 2 81 24 +57

What I checked:

  • Repository policy applied: Root AGENTS.md plus scoped src/gateway/AGENTS.md and src/agents/AGENTS.md were read fully; the relevant guidance treats gateway hot paths, provider routing, auth state, and startup defaults as compatibility-sensitive review surfaces. (AGENTS.md:23, 843ad143647e)
  • Current main normal caller omits enabled: The production gateway startup path passes providerAuthPrewarm with getConfig only, so omitted enabled currently determines whether startup warming runs. (src/gateway/server.impl.ts:1711, 843ad143647e)
  • Current main schedules startup warm by omission: Current main schedules provider-auth warming whenever enabled is not explicitly false, and the sidecar timer calls warmCurrentProviderAuthStateOffMainThread and logs the pre-warmed marker. (src/gateway/server-startup-post-attach.ts:1303, 843ad143647e)
  • PR implements hook-only default: The diff adds startupWarmEnabled, keeps the sidecar when providerAuthPrewarm is present, and passes startupWarmEnabled only when enabled is true. (src/gateway/server-startup-post-attach.ts:220, 165b2c6620e4)
  • Regression coverage added: The added test covers omitted enabled: it registers the auth-profile failure hook, does not call startup warm after the delay, then clears and rewarms after the hook fires with the latest config. (src/gateway/server-startup-post-attach.test.ts:942, 165b2c6620e4)
  • Latest release still has default startup timer: v2026.6.10 still contains the provider-auth startup timer and pre-warmed log path, so this PR is not obsolete against shipped behavior. (src/gateway/server-startup-post-attach.ts:296, aa69b12d0086)

Likely related people:

  • vincentkoc: Authored this maintainer PR, has recent merged work delaying provider-auth warmup, and is assigned on the live PR. (role: recent gateway/provider-auth startup contributor and likely follow-up owner; confidence: high; commits: 165b2c6620e4, eb5d6c7294b6, 17be26bc4f8b; files: src/gateway/server-startup-post-attach.ts, src/gateway/server-startup-post-attach.test.ts, src/gateway/server.impl.ts)
  • Sarah Fortune: Git history shows the startup provider-auth prewarm and prepared auth-state behavior originated in her May 2026 commits. (role: provider-auth prewarm feature contributor; confidence: high; commits: 4f80cc1943b9, 95343affbbee, 62a330e75216; files: src/agents/model-provider-auth.ts, src/gateway/server-startup-post-attach.ts)
  • samzong: Authored the merged worker-thread provider-auth warm mitigation that current main uses, which reduced main-event-loop blocking but did not make startup warming opt-in. (role: related mitigation author; confidence: high; commits: 316fd5b62572; files: src/agents/model-provider-auth.ts, src/agents/model-provider-auth.worker.ts, src/gateway/server-startup-post-attach.ts)
  • Peter Steinberger: Recent commits adjusted gateway startup event-loop behavior and provider-auth cancellation adjacent to this scheduling tradeoff. (role: recent adjacent startup/auth contributor; confidence: medium; commits: 3e8fd4944fe1, 60e3749de339, caa610214411; files: src/gateway/server-startup-post-attach.ts, src/gateway/server.impl.ts, src/agents/model-provider-auth.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.
Review history (1 earlier review cycle)
  • reviewed 2026-06-21T09:03:22.851Z sha 165b2c6 :: needs maintainer review before merge. :: none

@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. P1 High-priority user-facing bug, regression, or broken workflow. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 auth-provider 🚨 May break OAuth, tokens, provider routing, model choice, or credentials. rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. and removed rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. labels May 31, 2026
This was referenced Jun 15, 2026
Closed
Closed
Closed
@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. and removed rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. labels Jun 15, 2026
This was referenced Jun 20, 2026
Closed
Open
@steipete steipete mentioned this pull request Jul 6, 2026
Merged
@steipete

steipete commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Landed the current-main replay in #100667 as 20e6983bed2.

The landed fix preserves this PR's important contract: normal Gateway startup keeps provider-auth failure rewarming but no longer schedules the broad startup sweep. The replay also removes the later redundant QA-only skip environment surface, adds the changelog entry, and passed focused provider/gateway tests, a full build, live authenticated Gateway RPC proof, the broad changed gate, fresh autoreview, and exact-head hosted CI including QA Smoke.

This branch could not be refreshed directly because maintainer edits are disabled and it had diverged from current main. Vincent's authorship is preserved through Co-authored-by metadata and the changelog thanks @vincentkoc.

Thank you for the careful WSL2 and native Windows proof. For future contributor branches, enabling Allow edits by maintainers lets us refresh and land the original PR instead of replaying it. Closing this PR as superseded by the landed canonical fix.

@steipete steipete closed this Jul 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@vincentkoc vincentkoc

Labels

gateway Gateway runtime maintainer Maintainer-authored PR merge-risk: 🚨 auth-provider 🚨 May break OAuth, tokens, provider routing, model choice, or credentials. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. P1 High-priority user-facing bug, regression, or broken workflow. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. size: S status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Bug]: 2026.5.22 Docker/WSL2 gateway event-loop starvation, 284s provider-auth prewarm, slow Telegram turn, and local RPC timeouts

2 participants

@vincentkoc @steipete