Skip to content

fix(gateway): isolate gmail watcher restart and abort handling#82395

Merged
frankekn merged 4 commits into
openclaw:mainfrom
samzong:fix/gmail-watcher-restart-abort
May 16, 2026
Merged

fix(gateway): isolate gmail watcher restart and abort handling#82395
frankekn merged 4 commits into
openclaw:mainfrom
samzong:fix/gmail-watcher-restart-abort

Conversation

@samzong

@samzong samzong commented May 16, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Problem: Gmail post-ready sidecars could run with stale state during hot reload, Tailscale setup could outlive cancellation, and aborting a command could be treated like a clean exit.
  • Why it matters: stale sidecar startup can execute against outdated runtime state, rewrite the Tailscale public target, and incorrect command-status classification hides failures.
  • What changed:
    • Stop queued post-ready sidecars before restarting the Gmail watcher in reload flow.
    • Propagate cancellation from post-ready sidecar stop through Gmail watcher startup into Tailscale status and serve/funnel commands.
    • Track abort-initiated kills separately and normalize their outcome as signal termination.
    • Add tests for ordered sidecar shutdown, Tailscale cancellation, and pre-aborted command execution.
  • What did NOT change (scope boundary):
    • No public schema, provider contracts, or dependency changes.

Change Type (select all)

  • Bug fix
  • Feature
  • Refactor required for the fix
  • Docs
  • Security hardening
  • Chore/infra

Scope (select all touched areas)

  • Gateway / orchestration
  • Skills / tool execution
  • Auth / tokens
  • Memory / storage
  • Integrations
  • API / contracts
  • UI / DX
  • CI/CD / infra

Linked Issue/PR

  • Closes #
  • Related #
  • This PR fixes a bug or regression

Real behavior proof (required for external PRs)

  • Behavior addressed:
    • Sidecars ordering in Gmail watcher restart no longer allows stale post-ready sidecars to survive into fresh config reload.
    • Cancelling a stale Gmail startup now aborts in-flight Tailscale setup before it can apply an old serve/funnel target.
    • Abort-driven command shutdown now resolves as signal termination, not synthesized success.
  • Real environment tested:
    • macOS local OpenClaw source worktree, Node.js v24.14.0.
    • Runtime proof imported the actual src/hooks/gmail-watcher.ts module and used temporary executable shims on PATH to record tailscale and gog process calls without real credentials.
  • Exact steps or command run after this patch:
    • node --import tsx --input-type=module <<'NODE' with a runtime probe that:
      • installs temporary tailscale and gog executables on PATH;
      • starts stale Gmail watcher config for [email protected] with Tailscale target old-target:8788;
      • aborts that startup while the first tailscale status --json command is still in flight;
      • starts fresh Gmail watcher config for [email protected] with Tailscale target new-target:8788;
      • asserts stale result is startup cancelled, old Tailscale serve mutations are 0, and new Tailscale serve mutations are 1.
    • node scripts/run-vitest.mjs src/hooks/gmail-watcher.test.ts src/hooks/gmail-setup-utils.test.ts src/hooks/gmail-watcher-lifecycle.test.ts src/gateway/server-startup-post-attach.test.ts
    • node scripts/run-tsgo.mjs -p tsconfig.core.json --incremental --tsBuildInfoFile .artifacts/tsgo-cache/core.tsbuildinfo
    • node scripts/run-tsgo.mjs -p test/tsconfig/tsconfig.test.src.json --incremental --tsBuildInfoFile .artifacts/tsgo-cache/test-src.tsbuildinfo
    • node scripts/run-oxlint.mjs --tsconfig config/tsconfig/oxlint.core.json src/gateway/server-startup-post-attach.test.ts src/gateway/server-startup-post-attach.ts src/hooks/gmail-setup-utils.test.ts src/hooks/gmail-setup-utils.ts src/hooks/gmail-watcher-lifecycle.test.ts src/hooks/gmail-watcher-lifecycle.ts src/hooks/gmail-watcher.test.ts src/hooks/gmail-watcher.ts
    • pnpm exec oxfmt --check --threads=1 src/gateway/server-startup-post-attach.test.ts src/gateway/server-startup-post-attach.ts src/hooks/gmail-setup-utils.test.ts src/hooks/gmail-setup-utils.ts src/hooks/gmail-watcher-lifecycle.test.ts src/hooks/gmail-watcher-lifecycle.ts src/hooks/gmail-watcher.test.ts src/hooks/gmail-watcher.ts
    • git diff --check
  • Evidence after fix:
    • Runtime proof output:
      OpenClaw Gmail watcher restart abort proof
      stale result: {"started":false,"reason":"startup cancelled"}
      fresh result: {"started":true}
      old tailscale serve mutations: 0
      new tailscale serve mutations: 1
      observed command log:
        tailscale status --json
        tailscale status --json
        tailscale serve --bg --set-path /gmail-pubsub --yes new-target:8788
        gog gmail watch start --account [email protected] --label INBOX --topic projects/proof/topics/gmail
      RESULT: pass - aborted stale startup did not mutate the old Tailscale endpoint; fresh startup configured the new target.
      
    • Targeted Vitest output: Test Files 4 passed (4), Tests 44 passed (44).
    • tsgo core and source-test type checks completed with exit code 0.
    • Targeted oxlint reported Found 0 warnings and 0 errors.
    • Formatting and diff whitespace checks passed.
  • Observed result after fix:
    • Stale cancelled startup returned startup cancelled.
    • No tailscale serve --set-path ... old-target:8788 command ran after cancellation.
    • Fresh watcher startup configured exactly one tailscale serve --set-path /gmail-pubsub --yes new-target:8788 command.
    • runCommandWithTimeout sets abort termination state and does not collapse it into exit-success semantics.
  • What was not tested:
    • Real Gmail API credentials and real Tailscale network mutation against an operator tailnet were not used; the runtime proof used local executable shims to exercise actual OpenClaw watcher and process-control code without secrets.
    • Broad Testbox gate was not run because crabbox was unavailable locally (crabbox not in PATH, ../crabbox/bin/crabbox missing, and scripts/crabbox-wrapper.mjs failed sanity checks).

Root Cause (if applicable)

  • Root cause:
    • Missing sequencing before Gmail watcher restart.
    • Missing cancellation propagation into Tailscale setup.
    • Missing abort-state carry-through in exit normalization.
  • Missing detection / guardrail:
    • No stop-before-start check for post-ready sidecar queue in this path.
    • No regression proving Tailscale setup receives cancellation and avoids stale endpoint mutation.
    • Abort handling only inferred through generic kill state.
  • Contributing context (if known):
    • Timing race during hot reload.

Regression Test Plan (if applicable)

  • Coverage level that should have caught this:
    • Unit test
    • Seam / integration test
    • End-to-end test
    • Existing coverage already sufficient
  • Target test or file:
    • src/gateway/server-reload-handlers.test.ts
    • src/gateway/server-startup-post-attach.test.ts
    • src/hooks/gmail-watcher.test.ts
    • src/hooks/gmail-setup-utils.test.ts
    • src/process/exec.test.ts
  • Scenario the test should lock in:
    • Sidecars are stopped before restart, post-ready stop aborts the active watcher startup signal, Tailscale commands receive the abort signal, and abort path does not resolve as exit success.
  • Why this is the smallest reliable guardrail:
    • The bug is a deterministic cancellation boundary across Gateway sidecar scheduling, Gmail watcher startup, and command execution.
  • Existing test that already covers this (if any):
    • Prior reload and process execution tests covered only part of the sequence.
  • If no new test is added, why not:
    • New tests were needed for Tailscale setup cancellation.

User-visible / Behavior Changes

  • Hot-reload Gmail watcher now clears queued post-ready sidecars before restart.
  • Cancelled stale Gmail startup can no longer continue Tailscale setup and overwrite the public Gmail hook target after a newer watcher starts.
  • Abort-triggered command kill is reported as signal termination with non-success result semantics.

Diagram (if applicable)

N/A

Security Impact (required)

  • New permissions/capabilities? (Yes/No) No
  • Secrets/tokens handling changed? (Yes/No) No
  • New/changed network calls? (Yes/No) No
  • Command/tool execution surface changed? (Yes/No) No
  • Data access scope changed? (Yes/No) No
  • If any Yes, explain risk + mitigation:
    • Not applicable.

Repro + Verification

Environment

  • OS: macOS local shell
  • Runtime/container: local shell, Node.js v24.14.0
  • Model/provider: N/A
  • Integration/channel (if any): Gmail watcher with local tailscale/gog executable shims
  • Relevant config (redacted): local proof config with fake account names, fake hook token values, and no real credentials

Steps

  1. Run the runtime proof command described in the Real behavior proof section.
  2. Confirm stale startup returns startup cancelled.
  3. Confirm command log has no old-target:8788 Tailscale serve mutation.
  4. Confirm command log has exactly one new-target:8788 Tailscale serve mutation.
  5. Run the targeted tests, type checks, lint, format, and diff checks listed above.

Expected

  • No stale sidecar restart race.
  • Cancelled stale Tailscale setup does not mutate the old endpoint.
  • Fresh Gmail watcher startup configures the new endpoint once.
  • Abort kill does not read as successful process exit.

Actual

  • Runtime proof and targeted checks matched expected behavior.

Evidence

Attach at least one:

  • Failing test/log before + passing after
  • Trace/log snippets
  • Screenshot/recording
  • Perf numbers (if relevant)

Human Verification (required)

What you personally verified (not just CI), and how:

  • Verified scenarios:
    • Runtime Gmail watcher restart/abort proof with temporary process shims and copied command output.
    • Unit test verification for sidecar sequencing, Tailscale signal propagation, and abort path.
    • Type/lint/format checks for the touched files.
  • Edge cases checked:
    • Pre-aborted signal path.
    • close/reload ordering.
    • In-flight Tailscale setup cancellation.
  • What you did not verify:
    • Real production Gmail/Tailscale credentials or real tailnet endpoint mutation.
    • Broad Testbox validation, because Crabbox was unavailable in this local worktree.

Review Conversations

  • I replied to or resolved every bot review conversation I addressed in this PR.
  • I left unresolved only the conversations that still need reviewer or maintainer judgment.

Compatibility / Migration

  • Backward compatible? (Yes/No) Yes
  • Config/env changes? (Yes/No) No
  • Migration needed? (Yes/No) No
  • If yes, exact upgrade steps:
    • Not applicable.

Risks and Mitigations

  • Risk: additional cleanup step adds small shutdown overhead in hot-reload path.
    • Mitigation: already using existing lifecycle close helpers; only ordering changed.

@openclaw-barnacle openclaw-barnacle Bot added gateway Gateway runtime size: L triage: mock-only-proof Candidate: PR proof only shows tests, mocks, snapshots, lint, typecheck, or CI. labels May 16, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d940579a14

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/hooks/gmail-watcher.ts Outdated
@clawsweeper

clawsweeper Bot commented May 16, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge.

Summary
The PR adds cancellable Gmail post-ready sidecar handles, passes abort signals through Gmail/Tailscale command startup, normalizes abort-driven process exits, and adds focused gateway, Gmail, and process tests.

Reproducibility: yes. Source inspection shows current main starts Gmail/Tailscale setup without an abort signal, and the PR body provides a runtime probe that exercises the cancellation path with local command shims.

Real behavior proof
Sufficient (terminal): The PR body includes after-fix terminal output from a runtime probe importing the real Gmail watcher module with PATH shims, showing zero stale Tailscale mutations and one fresh mutation.

Next step before merge
This is an active external bug-fix PR with supplied proof and no discrete automation repair finding; maintainers need to inspect the failing CI check and complete normal merge review.

Security
Cleared: The diff narrows existing command execution through abort propagation and does not add dependencies, workflows, permissions, or secret handling.

Review details

Best possible solution:

Land a reviewed version that keeps cancellation in the gateway/Gmail owner paths, preserves abort exit semantics in the shared process helper, and retains focused regression coverage.

Do we have a high-confidence way to reproduce the issue?

Yes. Source inspection shows current main starts Gmail/Tailscale setup without an abort signal, and the PR body provides a runtime probe that exercises the cancellation path with local command shims.

Is this the best way to solve the issue?

Yes. The patch uses the narrow maintainable path: register cancellable post-ready handles, pass AbortSignal through the existing Gmail/Tailscale command calls, and normalize abort exits in the shared process helper without adding config or public contracts.

What I checked:

  • Live PR state: GitHub API reports the PR is open, no longer draft, head c550d5d83ab8f2ba8a83d45970a25cc8a8add376, with labels gateway, size: XL, and proof: supplied. (c550d5d83ab8)
  • Current main lacks Tailscale cancellation: On current main, startGmailWatcher calls ensureTailscaleEndpoint without a signal, and ensureTailscaleEndpoint runs both Tailscale commands with only timeoutMs. (src/hooks/gmail-watcher.ts:195, e650f8930d9d)
  • Current main lacks queued Gmail sidecar stop: The current reload handler restarts Gmail directly after stopGmailWatcher; it has no stopPostReadySidecars hook before restart. (src/gateway/server-reload-handlers.ts:338, e650f8930d9d)
  • PR head stops queued sidecars before Gmail reload: The PR head adds stopPostReadySidecars to reload params and invokes it before importing/stopping/restarting the Gmail watcher. (src/gateway/server-reload-handlers.ts:340, c550d5d83ab8)
  • PR head propagates abort to Gmail/Tailscale commands: The PR head builds a Gmail cancellation signal, passes it into ensureTailscaleEndpoint, and forwards params.signal into both tailscale status and tailscale serve/funnel. (src/hooks/gmail-watcher.ts:274, c550d5d83ab8)
  • PR head normalizes abort-driven process termination: The shared process helper now accepts signal, skips pre-aborted spawns, tracks abort-issued kills separately, removes abort listeners on settle, and reports abort termination as signal. (src/process/exec.ts:331, c550d5d83ab8)

Likely related people:

  • steipete: Recent mainline history includes Gmail Tailscale target support and gateway startup sidecar responsiveness/latency work that overlaps the PR surface. (role: feature owner and recent adjacent owner; confidence: high; commits: 1fe9f648b1f6, 2949171fcc15, 43ababf96b08; files: src/hooks/gmail-watcher.ts, src/hooks/gmail-setup-utils.ts, src/gateway/server-startup-post-attach.ts)
  • vincentkoc: Recent process execution history includes Windows command output and exit-code handling work, which is adjacent to this PR's abort normalization in runCommandWithTimeout. (role: recent process helper contributor; confidence: medium; commits: e6d2c9b08005, 37099dae3e0f; files: src/process/exec.ts)
  • Angfr95: Recent Gmail watcher/setup history includes a Gmail helper PATHEXT shim fix in the same command-execution area touched by this PR. (role: recent Gmail helper contributor; confidence: medium; commits: f126f72d6388; files: src/hooks/gmail-watcher.ts, src/hooks/gmail-setup-utils.ts)
  • samzong: Besides authoring this PR, recent merged mainline history shows gateway startup trace attribution work in the startup/plugin sidecar area. (role: recent gateway startup contributor; confidence: medium; commits: 1d121c1f08a6; files: src/gateway/server-startup-post-attach.ts)

Remaining risk / open question:

  • The PR head currently has a failing checks-node-core GitHub Actions check with only a generic public annotation available in this read-only review.
  • The supplied runtime proof uses local executable shims rather than real Gmail credentials or a real Tailscale tailnet mutation, so maintainers may still want broader live/Testbox validation before landing.

Codex review notes: model gpt-5.5, reasoning high; reviewed against e650f8930d9d.

@samzong
samzong marked this pull request as draft May 16, 2026 02:29
@openclaw-barnacle openclaw-barnacle Bot added size: XL proof: supplied External PR includes structured after-fix real behavior proof. and removed size: L triage: mock-only-proof Candidate: PR proof only shows tests, mocks, snapshots, lint, typecheck, or CI. labels May 16, 2026
@clawsweeper clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 16, 2026
@samzong
samzong force-pushed the fix/gmail-watcher-restart-abort branch from 5ebfdec to c550d5d Compare May 16, 2026 06:32
@openclaw-barnacle openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 16, 2026
@samzong

samzong commented May 16, 2026

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown
Contributor

Codex Review: Didn't find any major issues. Another round soon, please!

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@samzong
samzong marked this pull request as ready for review May 16, 2026 06:38
@clawsweeper clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 16, 2026
@frankekn frankekn self-assigned this May 16, 2026
@openclaw-barnacle openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 16, 2026
@frankekn
frankekn merged commit 2fa853d into openclaw:main May 16, 2026
103 of 108 checks passed
@frankekn

Copy link
Copy Markdown
Contributor

Merged via squash.

Thanks @samzong!

github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request May 24, 2026
…law#82395)

Merged via squash.

Prepared head SHA: 6150284
Co-authored-by: samzong <[email protected]>
Co-authored-by: frankekn <[email protected]>
Reviewed-by: @frankekn
galiniliev pushed a commit to galiniliev/openclaw that referenced this pull request May 25, 2026
…law#82395)

Merged via squash.

Prepared head SHA: 6150284
Co-authored-by: samzong <[email protected]>
Co-authored-by: frankekn <[email protected]>
Reviewed-by: @frankekn
qiaokuan1992 pushed a commit to qiaokuan1992/openclaw that referenced this pull request Jun 2, 2026
…law#82395)

Merged via squash.

Prepared head SHA: 6150284
Co-authored-by: samzong <[email protected]>
Co-authored-by: frankekn <[email protected]>
Reviewed-by: @frankekn
jameslcowan pushed a commit to jameslcowan/openclaw that referenced this pull request Jun 2, 2026
…law#82395)

Merged via squash.

Prepared head SHA: 6150284
Co-authored-by: samzong <[email protected]>
Co-authored-by: frankekn <[email protected]>
Reviewed-by: @frankekn
SYU8384 pushed a commit to SYU8384/openclaw that referenced this pull request Jun 3, 2026
…law#82395)

Merged via squash.

Prepared head SHA: 6150284
Co-authored-by: samzong <[email protected]>
Co-authored-by: frankekn <[email protected]>
Reviewed-by: @frankekn
sablehead pushed a commit to sablehead/openclaw that referenced this pull request Jun 10, 2026
…law#82395)

Merged via squash.

Prepared head SHA: 6150284
Co-authored-by: samzong <[email protected]>
Co-authored-by: frankekn <[email protected]>
Reviewed-by: @frankekn
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gateway Gateway runtime proof: supplied External PR includes structured after-fix real behavior proof. size: XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants