Skip to content

fix(gateway,mac): stop stale protocol mismatch reconnect loop, auto-r…#82876

Closed
loeclos wants to merge 1 commit into
openclaw:mainfrom
loeclos:fix/protocol-mismatch-stale-client-loop
Closed

fix(gateway,mac): stop stale protocol mismatch reconnect loop, auto-r…#82876
loeclos wants to merge 1 commit into
openclaw:mainfrom
loeclos:fix/protocol-mismatch-stale-client-loop

Conversation

@loeclos

@loeclos loeclos commented May 17, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Problem: After rolling back from v2026.5.12 to v2026.5.7 on macOS, a stale CLI-mode client (cli v2026.5.12) continuously reconnects to the gateway producing ~1/sec protocol mismatch spam that persists across gateway restarts and killing visible helper processes
  • Why it matters: The log spam makes deep status/audit/probe behavior noisy and confusing; the retrying incompatible client wastes CPU and connection resources; users have no automated way to clean up stale updater processes left behind by update/rollback
  • What changed: (1) GatewayClient pauses reconnect on protocol mismatch instead of retrying forever; (2) doctor --fix auto-removes stale ai.openclaw.update.* launchd jobs; (3) server rate-limits repeated protocol mismatches per source IP
  • What did NOT change: No changes to gateway protocol negotiation logic, update/rollback mechanism, launchd lifecycle, or existing stale-updater detection surface

Change Type

  • Bug fix
  • Feature
  • Refactor required for the fix
  • Docs
  • Security hardening
  • Chore/infra

Scope

  • Gateway / orchestration
  • Skills / tool execution
  • Auth / tokens
  • Memory / storage
  • Integrations
  • API / contracts
  • UI / DX
  • CI/CD / infra

Linked Issue/PR

Real behavior proof

  • Behavior or issue addressed: Stale client protocol mismatch reconnect loop after rollback
  • Real environment tested: Linux (gateway protocol test suite); macOS-specific launchd remediation passes unit tests (launchd stubs)
  • Exact steps or command run after this patch:
    • pnpm test src/gateway/reconnect-gating.test.ts — 13/13 pass including PROTOCOL_MISMATCH block
    • pnpm test src/daemon/launchd.test.ts — 59/59 pass
    • pnpm test src/commands/doctor-platform-notes.launchctl-env-overrides.test.ts — 6/6 pass
    • pnpm test src/flows/doctor-health-contributions.test.ts — 16/16 pass
    • pnpm test src/cli/daemon-cli/status.gather.test.ts src/cli/daemon-cli/status.print.test.ts — 27/27 pass
    • pnpm test src/gateway/protocol/ — 87/87 pass
  • Evidence after fix: All 140+ relevant tests pass
  • Observed result after fix: GatewayClient pauses reconnect on PROTOCOL_MISMATCH instead of looping; doctor --fix removes stale updater launchd jobs; server suppresses logs after 5 identical mismatches per IP per minute
  • What was not tested: Live macOS rollback reproduction (requires macOS host with launchd)

Root Cause

  • Root cause: GatewayClient.shouldPauseReconnectAfterAuthFailure() did not list PROTOCOL_MISMATCH as a pause-eligible detail code, so any client receiving a protocol mismatch error would retry indefinitely (exponential backoff 1s→30s, never stopping)
  • Missing detection / guardrail: No auto-remediation path in doctor for stale ai.openclaw.update.* launchd jobs — only detection/reporting existed
  • Contributing context: The stale client process from v2026.5.12 survived rollback; without the pause-on-mismatch fix, it kept reconnecting and producing log spam

Regression Test Plan

  • Coverage level that should have caught this:
    • Unit test
    • Seam / integration test
    • End-to-end test
    • Existing coverage already sufficient
  • Target test or file: src/gateway/reconnect-gating.test.tsisNonRecoverableAuthError block test for PROTOCOL_MISMATCH
  • Scenario the test should lock in: A protocol mismatch error causes the client to pause reconnection (non-recoverable auth error)
  • Why this is the smallest reliable guardrail: Single pure function test covering the gateway-client decision point
  • Existing test that already covers this (if any): Similar tests exist for all other auth error detail codes
  • If no new test is added, why not: N/A — test was added

User-visible / Behavior Changes

  • Gateway clients from incompatible versions stop retrying after one protocol mismatch (no more endless log spam)
  • openclaw doctor --fix on macOS now auto-removes stale ai.openclaw.update.* launchd jobs instead of only reporting them
  • Gateway suppresses repeated protocol mismatch log lines from the same IP after 5 occurrences per minute (silent close)

Diagram

Before:
client v2026.5.12 → connect → server v2026.5.7 → protocol mismatch → close(1002)
                                                                    → client scheduleReconnect() → connect → (loop forever)
After (Phase 1):
client v2026.5.12 → connect → server v2026.5.7 → protocol mismatch → close(1002, code:PROTOCOL_MISMATCH)
                                                                    → client shouldPauseReconnect=true → STOP
After (Phase 3):
... → protocol mismatch x5 from same IP/60s → silent close (no log, no error frame)
Security Impact
- 
New permissions/capabilities? No
- 
Secrets/tokens handling changed? No
- 
New/changed network calls? No
- 
Command/tool execution surface changed? No
- 
Data access scope changed? No
Compatibility / Migration
- 
Backward compatible? Yes
- 
Config/env changes? No
- 
Migration needed? No
Risks and Mitigations
- 
Risk: doctor --fix removes a running updater job mid-update
- 
Mitigation: The function only runs during doctor --fix with explicit user opt-in; updater jobs are short-lived by design and removal is safe once the user has confirmed no update is running (already documented in existing guidance)
- 
Risk: Rate limiter could suppress legitimate rapid reconnects from different clients behind the same NAT IP
- 
Mitigation: Threshold is 5/60s — legitimate clients would succeed on first attempt and never hit the protocol mismatch path

…emediate updater jobs, rate-limit server side

- Add PROTOCOL_MISMATCH to ConnectErrorDetailCodes so clients recognize
  the error type and pause reconnect instead of looping forever
- Server now sends code: PROTOCOL_MISMATCH in protocol mismatch error
  response; GatewayClient includes it in shouldPauseReconnect list
- doctor --fix now auto-removes stale ai.openclaw.update.* launchd jobs
- Server-side per-IP rate limiter suppresses logs/error frames after 5
  protocol mismatches per 60s from the same address
@openclaw-barnacle openclaw-barnacle Bot added app: web-ui App: web-ui gateway Gateway runtime commands Command implementations size: S triage: mock-only-proof Candidate: PR proof only shows tests, mocks, snapshots, lint, typecheck, or CI. labels May 17, 2026
@clawsweeper

clawsweeper Bot commented May 17, 2026

Copy link
Copy Markdown
Contributor

Thanks for the context here. I swept through the related work, and this is now duplicate or superseded.

Close as superseded: the linked rollback report was resolved by the merged maintainer PR on current main, while this branch is now conflicting and its remaining auto-removal/rate-limit approach is unproven and not the accepted fix direction.

Canonical path: The linked rollback issue was closed by maintainer comment as fixed by merged PR #82908 and commit 38b3e73.

So I’m closing this here and keeping the remaining discussion on #82908.

Review details

Best possible solution:

Keep the merged gateway diagnostics and rollback troubleshooting as the canonical resolution; pursue reconnect gating or automatic updater-job removal only as a new current-main PR with maintainer approval and live macOS proof.

Do we have a high-confidence way to reproduce the issue?

Partly yes from source and the linked issue logs: the protocol-mismatch and reconnect paths are inspectable, but no live macOS rollback reproduction was established for this branch.

Is this the best way to solve the issue?

No. The accepted current-main fix is diagnostics and manual stale-process cleanup, while this conflicting branch adds unproven automatic removal and rate-limiting behavior that would need a fresh maintainer decision.

Security review:

Security review cleared: No supply-chain or secret-handling change was found; the concrete concern is operational compatibility from launchd job removal, covered as merge risk.

AGENTS.md: found and applied where relevant.

What I checked:

Likely related people:

  • steipete: Authored and merged the current-main diagnostics PR that closed the linked rollback/protocol-mismatch report. (role: canonical fix author and merger; confidence: high; commits: 38b3e7362272; files: docs/gateway/troubleshooting.md, src/gateway/server/ws-connection/message-handler.ts, src/cli/daemon-cli/status.gather.ts)

Codex review notes: model internal, reasoning high; reviewed against f1b8827d20c8; fix evidence: commit 38b3e7362272, main fix timestamp 2026-05-17T06:33:34+01:00.

@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P2 Normal backlog priority with limited blast radius. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. labels Jun 14, 2026
@clawsweeper

clawsweeper Bot commented Jun 14, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper applied the proposed close for this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

app: web-ui App: web-ui commands Command implementations gateway Gateway runtime merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. P2 Normal backlog priority with limited blast radius. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. size: S status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. triage: mock-only-proof Candidate: PR proof only shows tests, mocks, snapshots, lint, typecheck, or CI.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Protocol mismatch spam after rollback: local client still identifies as cli v2026.5.12 against 2026.5.7 gateway

1 participant