fix(tailscale): retry status json after serve startup by TUARAN · Pull Request #91553 · openclaw/openclaw

TUARAN · 2026-06-09T01:51:57Z

What Problem This Solves

Gateway startup can configure Tailscale Serve successfully and then immediately lose the published hostname because the first tailscale status --json probe hits a transient daemon or macOS service startup race. The Gateway remains running, but it can omit the derived Serve URL/origin and previously emitted a scary command-failure log.

Closes #42798.

Why This Change Was Made

Retry hostname discovery only after this Gateway process actually applies Tailscale Serve. Ordinary getTailnetHostname callers, including the latency-bounded openclaw status scan, remain single-attempt.
Reuse the exact cached Tailscale binary selected for Serve setup, so a retry cannot switch installations.
Use the repository's canonical retryAsync helper for three total attempts with a fixed 500 ms delay.
Retry only the pinned Execa timeout contract (timedOut) and Tailscale's documented/observed local daemon or macOS service readiness failures. Missing binaries, permission errors, malformed JSON, and unrelated failures still stop immediately.
Suppress command-failure output for this best-effort hostname probe; the Gateway's existing Serve status remains the user-facing result.

The maintainer rewrite also removes the one-use retry module from the contributor version and keeps the policy in the owning Tailscale integration module.

User Impact

Tailscale Serve startup now tolerates a brief local daemon/network-extension readiness race and can recover the published hostname and MCP app origin without adding retry latency to status/configuration callers. Persistent transient failures can add at most two 500 ms delays plus the existing per-attempt command timeout.

Contributor: @TUARAN. The rebased commits and maintainer fix commits retain Co-authored-by: TUARAN <[email protected]> credit.

Evidence

Exact head: 97ee0685811caaa0b00035d3b789b53c8188542c

OPENCLAW_HEAVY_CHECK_LOCK_SCOPE=worktree mise exec [email protected] -- node scripts/run-vitest.mjs src/infra/tailscale.test.ts src/gateway/server-tailscale.test.ts — 45 tests passed across 2 shards.
CI=1 OPENCLAW_CHECK_CHANGED_REMOTE_CHILD=1 OPENCLAW_CHANGED_LANES_RAW_SYNC=1 OPENCLAW_HEAVY_CHECK_LOCK_SCOPE=worktree PNPM_CONFIG_VERIFY_DEPS_BEFORE_RUN=false mise exec [email protected] -- node scripts/check-changed.mjs -- src/gateway/server-tailscale.test.ts src/gateway/server-tailscale.ts src/infra/tailscale-status.ts src/infra/tailscale.test.ts src/infra/tailscale.ts — passed full selected core/core-test typecheck, complete core lint, formatting, import-cycle, dependency, boundary, and repository guards. This was the documented trusted-source local fallback after Blacksmith Testbox warmup timed out.
.agents/skills/autoreview/scripts/autoreview --mode branch --base origin/main --stream-engine-output — clean; no accepted/actionable findings.
Tailscale dependency/runtime contract: upstream v1.94.1 and current source report Failed to connect to local Tailscale daemon; the installed Homebrew Tailscale 1.98.9 binary also contains the supported macOS message failed to connect to local Tailscale service; is Tailscale running?. Execa 9.6.1 reports command timeouts with timedOut: true.
Contributor live mechanism proof on the earlier final-head implementation ran three real macOS/Homebrew Serve cycles: each induced a first status timeout, recovered the real tailnet hostname on attempt two, enabled Serve, and completed cleanup. The maintainer rewrite narrows where the same retry behavior is invoked; exact-head regression and contract proof are above.

Not tested on the rewritten head: a fresh root-launchd or isolated userspace Tailscale daemon cycle. Hosted exact-head CI remains required before merge.

clawsweeper · 2026-06-09T01:54:32Z

Codex review: needs maintainer review before merge. Reviewed July 13, 2026, 2:57 AM ET / 06:57 UTC.

Summary
Adds a bounded same-binary retry helper for transient tailscale status --json readiness failures, with regression coverage ensuring permanent command and parsing failures remain single-attempt.

PR surface: Source +52, Tests +51. Total +103 across 3 files.

Reproducibility: yes. for the failure mechanism: the real harness deterministically causes the first production status probe to time out and shows that a second probe succeeds; current main's single-attempt path has no recovery step.

Review metrics: 1 noteworthy metric.

Status retry budget: 1 → 3 attempts; 5s → ~16s maximum. This bounded increase is the primary availability tradeoff maintainers should consciously accept before merge.

Stored data model
Persistent data-model change detected: serialized state: src/infra/tailscale.test.ts. Confirm migration or upgrade compatibility proof before merge.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #42798
Summary: This PR is the active candidate fix for the linked canonical startup-race issue; no separate root cause is indicated.

Members:

canonical: fix(tailscale): startup race condition causes transient 'Command failed: tailscale status --json' errors #42798 - The issue describes the same Serve-startup status race and is explicitly linked for closure by this PR. (github.com)

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🦞 diamond lobster
Proof: 🦞 diamond lobster
Patch quality: 🦞 diamond lobster
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Risk before merge

[P1] A qualifying persistent daemon-readiness failure can increase the status-probe portion of startup from one 5-second attempt to three attempts plus two 500 ms delays, so the worst-case exposure-startup wait becomes approximately 16 seconds.
[P1] Live proof used macOS Homebrew Tailscale in userspace-networking mode rather than a root launchd or Linux daemon; the focused tests cover error classification, but platform-specific CLI wording remains a small residual risk.

Maintainer options:

Land the bounded retry (recommended)
Accept the additional worst-case startup wait because retries are restricted to daemon-readiness failures and final-head live proof demonstrates successful recovery.
Reduce the retry budget
Change to two total attempts if maintainers prefer a lower maximum startup delay at the cost of less tolerance for slow daemon readiness.
Request another platform proof
Pause for a Linux or launchd-managed daemon run if maintainers require confirmation that its timeout and connection-error shapes match the tested classifier.

Next step before merge

No automated repair remains; the prior mechanical blocker is fixed, so the next action is normal exact-head maintainer review and merge handling.

Security
Cleared: The patch adds no dependency, workflow, secret, permission, shell construction, or new executable surface; it reuses the already-detected binary and existing argument array.

Review details

Best possible solution:

Merge the final head after exact-head maintainer review, retaining the bounded transient-only retry and allowing the linked issue to close when the fix lands.

Do we have a high-confidence way to reproduce the issue?

Yes for the failure mechanism: the real harness deterministically causes the first production status probe to time out and shows that a second probe succeeds; current main's single-attempt path has no recovery step.

Is this the best way to solve the issue?

Yes. Retrying the exact status read only for recognized daemon-readiness failures is narrower and more maintainable than adding an unconditional sleep, a new binary-path setting, or broad retries for permanent failures.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against f1076c23d46b.

Label changes

Label changes:

add rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): Final-head live output demonstrates three real Serve recovery cycles with an induced first production timeout, a successful second status read, recovered tailnet hostname, and completed cleanup.
remove rating: 🦐 gold shrimp: Current PR rating is rating: 🦞 diamond lobster, so this older rating label is no longer current.
remove status: ⏳ waiting on author: Current PR status label is status: 👀 ready for maintainer look.

Label justifications:

P2: This fixes a limited-blast-radius Gateway/Tailscale startup reliability and misleading-warning problem without blocking the core runtime.
merge-risk: 🚨 availability: Merging intentionally extends the maximum Tailscale hostname-discovery wait when daemon-readiness failures persist.
rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): Final-head live output demonstrates three real Serve recovery cycles with an induced first production timeout, a successful second status read, recovered tailnet hostname, and completed cleanup.
proof: sufficient: Contributor real behavior proof is sufficient. Final-head live output demonstrates three real Serve recovery cycles with an induced first production timeout, a successful second status read, recovered tailnet hostname, and completed cleanup.

Evidence reviewed

PR surface:

Source +52, Tests +51. Total +103 across 3 files.

View PR surface stats

Area	Files	Added	Removed	Net
Source	2	57	5	+52
Tests	1	51	0	+51
Docs	0	0	0	0
Config	0	0	0	0
Generated	0	0	0	0
Other	0	0	0	0
Total	3	108	5	+103

What I checked:

Changed runtime path: getTailnetHostname delegates status reads to the new helper while preserving the existing binary detection, parsing, hostname selection, and fallback flow. (src/infra/tailscale.ts:134, f58e6810fa4d)
Narrow retry classification: The helper retries only command timeouts represented by a killed SIGTERM process and recognized local-daemon readiness messages; all other command errors are immediately rethrown. (src/infra/tailscale-status.ts:9, f58e6810fa4d)
Regression coverage: Tests cover recovery after daemon-connect and timeout failures, plus single-attempt behavior for missing binaries, permission failures, and malformed JSON. (src/infra/tailscale.test.ts:80, f58e6810fa4d)
Real behavior proof: The PR body records three final-head macOS/Homebrew Serve cycles in which the first production status call timed out, the second recovered the real tailnet hostname, Serve remained enabled, and cleanup completed. (src/infra/tailscale-status.ts:35, f58e6810fa4d)
Current-main necessity: The provided current-main comparison shows only one status --json attempt, so the linked startup race remains unfixed until this or an equivalent change lands; the canonical issue remains open and linked to this PR. (github.com) (src/infra/tailscale.ts:131, f1076c23d46b)
Feature history: Tailscale Serve behavior and its fallback tests have repeatedly been maintained in this file; historical commits include the sudo fallback and subsequent guard work. (git.mineracks.com) (src/infra/tailscale.ts:1, 05b0b82937)

Likely related people:

steipete: Peter Steinberger authored or carried several historical Tailscale Serve hardening and fallback commits in the central runtime file. (git.mineracks.com) (role: feature-history owner and recent area contributor; confidence: high; commits: fd9be79be1, 05b0b82937, e4708b3b99; files: src/infra/tailscale.ts, src/infra/tailscale.test.ts)
sweepies: The earlier guarded sudo-fallback work in this same runtime path was merged with attribution to this contributor, making them relevant to adjacent command-failure semantics. (git.mineracks.com) (role: prior Tailscale fallback contributor; confidence: medium; commits: 05b0b82937; files: src/infra/tailscale.ts, src/infra/tailscale.test.ts)

What the crustacean ranks mean

🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

Review history (4 earlier review cycles)

reviewed 2026-06-21T16:26:48.107Z sha b516c83 :: needs real behavior proof before merge. :: [P2] Suppress non-final status retry logs | [P2] Skip retries for missing Tailscale binaries
reviewed 2026-06-30T20:18:18.143Z sha b516c83 :: needs real behavior proof before merge. :: [P2] Suppress non-final status retry logs | [P2] Skip retries for missing Tailscale binaries
reviewed 2026-07-13T03:17:43.603Z sha ed5a0fe :: needs changes before merge. :: [P2] Skip retries for non-retryable Tailscale failures
reviewed 2026-07-13T03:25:27.148Z sha ed5a0fe :: needs changes before merge. :: [P2] Skip retries for non-retryable Tailscale failures

TUARAN · 2026-07-13T03:14:12Z

Resolved the branch-history conflict by rebuilding this PR from current main and reapplying only the narrow status retry plus its regression test.

Current head: ed5a0feb65bf6b20190e4350964833aa7d8eae9b
GitHub mergeability: clean (mergeable=true)
Surface: 2 files, +53/-5

Verification on the refreshed content:

node_modules/oxfmt/bin/oxfmt --check src/infra/tailscale.ts src/infra/tailscale.test.ts
OPENCLAW_HEAVY_CHECK_LOCK_SCOPE=worktree node scripts/run-vitest.mjs src/infra/tailscale.test.ts src/gateway/server-tailscale.test.ts — 49 tests passed
git diff --check
.agents/skills/autoreview/scripts/autoreview --mode local — clean, no accepted/actionable findings
real macOS/Homebrew Tailscale proof — 3/3 Serve cycles recovered after the first status call failed, resolved the real tailnet hostname on attempt two, and completed cleanup

The old simulated repro script was intentionally dropped; the PR body now records the exact refreshed-head live proof and its limitations.

TUARAN · 2026-07-13T06:15:59Z

Addressed the ClawSweeper P2 and CI LOC guard on final head f58e6810fa4daf111e02115fa4771d52e8788ae4.

Retry only daemon connection failures or status probes terminated by the existing timeout.
Missing binaries, permission failures, and malformed JSON are not retried.
Retry reader moved to src/infra/tailscale-status.ts; main file is 483 lines, helper is 55.
Focused tests: 53 Tailscale/Gateway + 10 Plugin SDK surface tests passed.
Post-split real macOS/Homebrew Tailscale 1.98.8 proof passed 3/3 cycles; each recovered the real hostname on attempt two and completed Serve cleanup.
Final remote runtime/test/caller blobs are byte-identical to the live-tested workspace.
Temporary proof node was logged out and stopped.

The PR body has the refreshed six-field proof. GitHub reports mergeable=true.

clawsweeper · 2026-07-13T06:16:02Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

TUARAN · 2026-07-13T06:51:51Z

@clawsweeper re-review

clawsweeper · 2026-07-13T06:51:54Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

State: Complete
Detail: The targeted re-review published an exact durable review comment and queued an exact-item serialized router run.
Run: https://github.com/openclaw/clawsweeper/actions/runs/29230331623
Updated: 2026-07-13T06:59:35.434Z

Co-authored-by: TUARAN <[email protected]>

clawsweeper · 2026-07-21T15:35:23Z

ClawSweeper status: review started.

I am starting a fresh review of this pull request: fix(tailscale): retry status json after serve startup This is item 1/1 in the current shard. Shard 0/1.

This placeholder means the worker is alive and reading the current context. I will edit this same comment with the actual review when the claws are done clicking.

Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted.

steipete · 2026-07-21T15:44:02Z

Merged via squash.

Prepared head SHA: 97ee0685811caaa0b00035d3b789b53c8188542c
Landed commit: 3a5b2764f8a5e98e574d4e45d6d782048d1306df

* fix(tailscale): restrict status retries * test(tailscale): cover retry eligibility * refactor(tailscale): add status retry helper * refactor(tailscale): split status retry helper * fix(tailscale): scope status retry to serve startup Co-authored-by: TUARAN <[email protected]> * fix(tailscale): recognize app service startup errors Co-authored-by: TUARAN <[email protected]> --------- Co-authored-by: Peter Steinberger <[email protected]>

openclaw-barnacle Bot added scripts Repository scripts size: S triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels Jun 9, 2026

clawsweeper Bot mentioned this pull request Jun 9, 2026

fix(tailscale): startup race condition causes transient 'Command failed: tailscale status --json' errors #42798

Closed

TUARAN force-pushed the codex/issue-42798-tailscale-status-retry branch from b516c83 to ed5a0fe Compare July 13, 2026 03:10

openclaw-barnacle Bot removed the scripts Repository scripts label Jul 13, 2026

TUARAN force-pushed the codex/issue-42798-tailscale-status-retry branch from ed5a0fe to 5a6e0cb Compare July 13, 2026 06:11

steipete self-assigned this Jul 21, 2026

TUARAN and others added 6 commits July 21, 2026 08:26

fix(tailscale): restrict status retries

84222e1

test(tailscale): cover retry eligibility

73c2875

refactor(tailscale): add status retry helper

a2a9ec3

refactor(tailscale): split status retry helper

f7d847b

fix(tailscale): scope status retry to serve startup

7c283e9

Co-authored-by: TUARAN <[email protected]>

fix(tailscale): recognize app service startup errors

97ee068

Co-authored-by: TUARAN <[email protected]>

steipete force-pushed the codex/issue-42798-tailscale-status-retry branch from f58e681 to 97ee068 Compare July 21, 2026 15:32

openclaw-barnacle Bot added the gateway Gateway runtime label Jul 21, 2026

steipete merged commit 3a5b276 into openclaw:main Jul 21, 2026
197 of 200 checks passed

github-actions Bot mentioned this pull request Jul 21, 2026

📡 Upstream Digest — 2026-07-21 17:07 UTC curtismercier/openclaw-mods#1365

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(tailscale): retry status json after serve startup#91553

fix(tailscale): retry status json after serve startup#91553
steipete merged 6 commits into
openclaw:mainfrom
TUARAN:codex/issue-42798-tailscale-status-retry

TUARAN commented Jun 9, 2026 •

edited by steipete

Loading

Uh oh!

clawsweeper Bot commented Jun 9, 2026 •

edited

Loading

Uh oh!

TUARAN commented Jul 13, 2026

Uh oh!

TUARAN commented Jul 13, 2026 •

edited

Loading

Uh oh!

clawsweeper Bot commented Jul 13, 2026 •

edited

Loading

Uh oh!

TUARAN commented Jul 13, 2026

Uh oh!

clawsweeper Bot commented Jul 13, 2026 •

edited

Loading

Uh oh!

clawsweeper Bot commented Jul 21, 2026

Uh oh!

Uh oh!

steipete commented Jul 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

TUARAN commented Jun 9, 2026 • edited by steipete Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Problem This Solves

Why This Change Was Made

User Impact

Evidence

Uh oh!

clawsweeper Bot commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TUARAN commented Jul 13, 2026

Uh oh!

TUARAN commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

clawsweeper Bot commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TUARAN commented Jul 13, 2026

Uh oh!

clawsweeper Bot commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

clawsweeper Bot commented Jul 21, 2026

Uh oh!

Uh oh!

steipete commented Jul 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

TUARAN commented Jun 9, 2026 •

edited by steipete

Loading

clawsweeper Bot commented Jun 9, 2026 •

edited

Loading

TUARAN commented Jul 13, 2026 •

edited

Loading

clawsweeper Bot commented Jul 13, 2026 •

edited

Loading

clawsweeper Bot commented Jul 13, 2026 •

edited

Loading