[Bug]: Service launcher stays idle after an SSH-launched server takes the owner lock during a restart, and never recovers

Follow-up to #5749, as requested in #5749 ("slow or restarting external services can still lose the 2-second probe and trigger a competing launch attempt... please open a new issue for that part only"). This issue covers only that part.

Before submitting

I searched existing issues and did not find a duplicate.
I included enough detail to reproduce or investigate the problem.

Area

apps/server (service launcher), plus the SSH launch script in packages/ssh

Steps to reproduce

Linux host runs the supported user service (t3code.service → __service-launcher), linger on, T3 Connect linked. Default home ~/.t3, port 3773.
A macOS desktop client has an SSH connection to the same host.
Hit the restart gap on purpose: systemctl --user stop t3code.service, then let the desktop reconnect. ssh-launch/<key>/managed flips to managed.
systemctl --user start t3code.service. The child exits 78 and the launcher goes idle.
Quit the desktop. Nothing is listening anymore, and t3code.service still shows active.

In normal use, step 3 is any service restart (an update, a reinstall, a unit change) that overlaps a desktop reconnect.

Expected behavior

The desktop waits for the restarting service and adopts it as external. If something else does win the race, the service takes ownership back once that server exits, without anyone stepping in.

Actual behavior

The launch script finds no live runtime record, so it starts a managed server on the default home and writes managed (packages/ssh/src/tunnel.ts, launch script).
The service's child then hits the fix(server): enforce one database owner and guard update rollback #16102 owner lock, exits 78, and the launcher calls #idleWhileStateDirOwned (apps/server/src/serviceLauncher.ts). It stays active with no child, indefinitely.
Every agent now runs on the desktop-managed server. Even after fix(ssh): reconnecting keeps the remote server and its agents running #19432, that server is stopped on desktop quit and on explicit Disconnect/Remove. Keeping agents independent of the desktop is what the service setup is for.
After the managed server exits, the host stays without a server until someone runs systemctl --user restart t3code.service. The launcher never notices the lock was released.

Impact

The host silently stops being service-owned. Running agents then depend on the desktop's lifetime, and nothing shows that ownership changed. On this host it went unnoticed for 5 days.

Version or commit

Oct 18: launcher 0.0.46-nightly.20261015.3176 (unit ExecStart) supervising child 0.1.1-nightly.20261018.3331, with a Nightly desktop. Oct 23: the desktop's runner was 0.1.3-nightly.20261023.3579, built after #19432. With the service down, it started a managed server on the default home within seconds, twice. The idle path is unchanged on main at e340aae.

Environment

Fedora 44 host, systemd user service, linger on. macOS desktop over SSH. Phone over T3 Connect.

Logs or stack traces

The service was down from 18:51:35 to 18:53:02. The desktop logged in over SSH and its managed server came up inside that window:

Oct 18 18:51:33 sshd-session: Accepted publickey for <user> from <desktop tailnet IP>
Oct 18 18:51:35 systemd: Stopped t3code.service - T3 Code server.
ssh-launch/<key>/server.log  [18:52:31.212] INFO (#337): Listening on http://127.0.0.1:3774
Oct 18 18:53:02 systemd: Started t3code.service - T3 Code server.
boot-service.log:
[18:53:03.417] ERROR (#1): ServerAlreadyRunningError: A T3 Code server already owns /home/<user>/.t3/userdata. Stop that server, or use a separate T3 home and pair with it, then retry. No server was stopped.
[service-launcher] Another T3 Code server owns this T3 home. Stop it, then restart this service.

Five days later: t3code.service was active with only __service-launcher in its cgroup, nothing was listening for the service, and ssh-launch/<key>/managed said managed. #5749 has a matching report from another user (lg-epitech, an update handoff with a 15 s DB backup gap). In that report the launcher also stayed idle after the stray server exited.

Proposed fix

Both current behaviors look intentional, so this keeps their reasons intact:

fix(ssh): reconnecting keeps the remote server and its agents running #19432 kept the adoption fallback so a stale runtime record cannot block reconnects.
fix(server): enforce one database owner and guard update rollback #16102 made the launcher idle on exit 78 so it does not fall into a systemd/launchd restart loop.

1. Launcher recovers on its own (host side, no protocol change). While idle in #idleWhileStateDirOwned, check the owner lock every ~30 s without holding it. One way: try a non-blocking exclusive transaction on userdata/server-owner.sqlite and release it at once. When the lock is free, start the active version's child again. Polling inside the launcher causes no systemd restarts, so the restart loop #16102 prevented stays prevented. A pending update is still marked failed as it is today. This change alone turns "dead until a manual restart" into "back within ~30 s of the stray server exiting".

2. Optional: the SSH launch script skips the race while a live supervisor is restarting. Fallback only when no live launcher exists. The launcher would publish a small liveness record (PID plus process start time, the same PID-reuse guard #16102 uses for server-runtime.json). The launch script would then work like this:

No live runtime record, and that launcher PID is alive: wait for the runtime record within the existing cold-start budget (REMOTE_READY_TIMEOUT_MS), then adopt it as external.
No launcher record, or a dead one: same as today, including the fallback launch.

A stale record still cannot block reconnects, because a dead launcher PID falls through to today's path. Fix 2 needs both host and client to update. Fix 1 alone covers the failure.

I can open a PR for fix 1 if this direction works for you.

Workaround

Stop the desktop-managed server and wait for its PID to exit, because the lock can outlive the port. Set ssh-launch/<key>/managed to external and remove pid. Then systemctl --user restart t3code.service.

Triaged with Claude Code (claude-opus-5-5).

Another data point for this issue: the same race also happens when the service updates itself, with no desktop-app update and no manual systemctl stop.

Setup: Ubuntu 26.04 host, t3code.service user unit, default home ~/.t3, port 3773. A macOS desktop is connected over SSH (Tailscale). The unit's ExecStart still pointed at the 0.0.46-nightly.20261007.2761 launcher, which supervised child 0.1.2-nightly.20261021.3457.

What happened (Oct 24):

The running server logged Server update prepared; handing off to the service launcher. (target 0.1.3-nightly.20261024.3630) and shut down.
About 5 seconds later the desktop's tunnel to 3773 failed and the desktop reconnected. Its launch script found no live runtime record, ran t3 serve --host 127.0.0.1 --port 3773 --base-dir ~/.t3, and wrote managed. That server took the owner lock.
The service's 0.1.3 candidate exited with ServerAlreadyRunningError (78). The update was recorded as rolled-back, reason: candidate-exited:78, and the rollback to 0.1.2 exited 78 as well.

Difference from the report above: the old 0.0.46 launcher has no idle-on-78 path (#16102). It exited 1 each time (Active child exited unexpectedly (78)) and systemd retried every 5 s (RestartSec=5). After 6 attempts the unit hit StartLimitBurst and went to failed. A current launcher would idle instead, which is the case #24225 describes, so both outcomes need #24230's recovery. The failed state is at least visible, and our failed user units health check alerted. An idle launcher shows as active and stays silent.

Why #23005 doesn't cover this: the desktop-update-in-progress marker is written by the desktop updater. A headless service updating itself never writes it, so the SSH launch script has no signal to wait on. Fix 2 in this issue (the launcher publishing a liveness record that the SSH script checks) would cover this trigger as well.

Workaround that worked: SIGTERM the SSH-launched t3 serve and wait for it to exit, then systemctl --user reset-failed t3code, then t3 update --yes 0.1.3-nightly.20261024.3630. That switched activeVersion to 0.1.3 and restarted the service. The service owned 3773 again and server-runtime.json showed serviceManaged: true. Starting the unit with plain systemctl start would have launched the rolled-back 0.1.2 against a database that 0.1.3 may already have migrated, so I used the update command instead.

Note

🤖 Claude Opus 5.5 responding on behalf of beastyrabbit

Another data point, this time on stable v0.1.5. It also shows a consequence I don't see covered here or in #24230: after the race, the service comes back on the old version against a database the new version already migrated.

Setup. Linux VM, t3code.service user unit (ExecStart → runtime/versions/0.1.3/t3 __service-launcher) with a drop-in setting T3CODE_HOST=0.0.0.0. Default home ~/.t3, port 3773. A Linux desktop (AppImage, 0.1.5) on another machine had an SSH route and an HTTPS route to this host.

What happened (Oct 29).

At 15:08 the service self-updated 0.1.3 → 0.1.5 (Server update prepared; handing off to the service launcher.). The desktop updated and restarted at the same moment.
The 0.1.5 candidate exited with ServerAlreadyRunningError (78). From then on the home was owned by t3 serve --host 127.0.0.1 --port 3773 --base-dir ~/.t3, started by the desktop's SSH launch script from its own 0.1.5 runtime.
The launcher went idle and wrote activeVersion: "0.1.3" with update: { status: "failed", reason: "state-dir-owned" }.
The host ran on that loopback-only server for about 5.5 hours. The 0.0.0.0 bind was gone, so LAN clients couldn't reach it, and the unit still showed active.

The same evening it happened again, with cleaner timestamps. After I stopped the stray server and restarted the unit, the update ran again: handing off at 20:43:01, the SSH launch script started a loopback server at 20:43:27, the candidate exited 78 at 20:43:30, and service-state.json again said failed / state-dir-owned. While a desktop keeps an SSH route to the host, the update never lands, because the restart it needs always opens the gap.

The downgrade. The stray server runs the desktop's version, which here was the update target, and it migrated the database. The launcher then put activeVersion back to fromVersion. When I restarted the unit at 20:34, it started 0.1.3 on that database:

[20:34:54.535] WARN (#46): Database migration history diverges from this build; recorded migration ids are skipped, not reconciled by name.
  divergent: [
    '70:PeerLinks (unknown to this build)',
    '71:PeerLinkFence (unknown to this build)',
    '72:WaitingLinkRequestIndex (unknown to this build)'
  ]

As I read #24230, its recovery would do this on its own: once the stray server exits, it "starts the active version again", which after a state-dir-owned failure is the old build. A manual systemctl --user restart does the same. The earlier comment here avoided it by running t3 update --yes <target> instead of starting the unit. It might be safer if the recovery retried the failed update, or refused to start a build that doesn't know the database's latest migrations.

Workaround we're using. We removed the SSH route from the desktop for every host that runs the service, and reach those hosts over their HTTP routes only. With no SSH route, nothing can start a competing server during the restart gap.

Additional matching observations on Linux hosts with Windows desktop SSH connections, October 30–31 nightly updates:

Two hosts' service-state records reported activeVersion 0.1.6-nightly.20261029.3892, targetVersion 0.1.6-nightly.20261030.3940, update status failed, reason state-dir-owned.
Boot logs showed ServerAlreadyRunningError during handoff and the service-launcher message that another server owned the home.
After another client update attempt, two inspected hosts answered API requests on 0.1.6-nightly.20261031.3956 while their service-state records retained older activeVersion and failed/state-dir-owned updates to October31.
Each had an older __service-launcher still present, while the responding newer t3 serve process was outside the user service cgroup, in a login-session scope with PPID1.
The Windows UI first reported “Not back after 32 minutes” and later remained “Restarting,” despite responsive remote APIs.

We have not established the exact race timing or reproduced the post-client-quit outage in this incident, and have not deleted ownership locks or stopped live servers. These observations support the ownership-handoff/status mismatch described here; they are not a claim that every causal step is independently proven. Local bootstrap/recovery tracking references this upstream issue rather than duplicating it.

Another occurrence on stable 0.1.5, from a service-initiated update (0.1.3 → 0.1.5) with a macOS desktop connected over SSH. This one adds a confusing message in the client UI.

Setup: Pop!_OS (Linux x64, kernel 6.12), t3code.service systemd user unit (ExecStart → runtime/versions/0.1.5/t3 __service-launcher), drop-in T3CODE_HOST=127.0.0.1, default home ~/.t3, port 3773, T3 Connect managed tunnel. Client: T3 Code desktop on a MacBook Pro, connected over SSH via Tailscale.

Timeline (host-local):

09:17:08.187  boot-service.log  Server update prepared; handing off to the service launcher. (targetVersion 0.1.5)
09:17:10      boot-service.log  Keeping the managed tunnel across the update restart / Relay client stopped
09:17:25      SSH-launched `t3 serve --host 127.0.0.1 --port 3773 --base-dir ~/.t3` starts (PPID 1, T3CODE_NO_BROWSER=1, SSH session from the Mac)
09:17:32      ssh-launch/<key>/server.log  Listening on http://127.0.0.1:3773
09:17:45.222  boot-service.log  ServerAlreadyRunningError: A T3 Code server already owns ~/.t3/userdata ...
              [service-launcher] Another T3 Code server owns this T3 home. Stop it, then restart this service.

About 17 s passed between the handoff and the SSH launch, and about 37 s before the new child tried to take ownership, so the gap is wide enough for a reconnect to land in it every time.

What the user saw:

The Mac desktop showed a connection error, and later: "Server update failed: Remote updates require the T3 Code background service. Run t3 service install on the server machine." The service was installed and active. That message sent the user toward a reinstall that would not have helped, because the real cause was the SSH-launched server holding the lock.
t3code.service stayed active with only __service-launcher in its cgroup for about an hour. A manual systemctl --user restart t3code.service at 10:15 failed the same way, because the stray server was still alive.

Recurrence: the same handoff → ServerAlreadyRunningError pair appears at least 4 more times in this host's boot-service.log over recent updates. The matching Listening on lines in ssh-launch/<key>/server.log sit right after each handoff.

Workaround that worked: stop the SSH-launched server, then systemctl --user restart t3code.service. The service came up on 0.1.5, the managed tunnel recovered (startManagedCloudTunnelIfOriginConfirmed → Success), and the Mac reconnected. At its next reconnect the SSH launch script lost the race to the service (ServerAlreadyRunningError in ssh-launch/<key>/server.log at 10:18:10) and adopted it as external.

+1 for fix 1 (the launcher polling the owner lock while idle). Separately, the client's "Remote updates require the background service" message should tell apart "no service installed" and "service installed but locked out by another server".

Triaged with Claude Code (claude-opus-5-5) via t3 triage.

Reproduced again on nightly, this time from a client-initiated update with the service template and host memory protection already correct. This bears directly on #25633.

Setup: Linux host, t3code.service user unit (launcher and child on 0.1.6-nightly.20261031.3956), default home ~/.t3, port 3773, T3CODE_HOST=0.0.0.0. A Windows desktop (nightly) is connected to it over SSH via Tailscale. A second host showed the same failure in the same session.

Timeline (host-local, Oct 31):

10:44:08.241  boot-service.log  Server update prepared; handing off to the service launcher. (target 0.1.6-nightly.20261031.3977)
10:44:10.273  boot-service.log  Keeping the managed tunnel across the update restart
10:44:11.265  boot-service.log  Relay client stopped
10:44:23.875  ssh-launch/<key>/server.log  ServerAlreadyRunningError (old server not yet released; launched from runtime/versions/...3956)
10:44:25.703  ssh-launch/<key>/server.log  Listening on http://127.0.0.1:3773   <- SSH-launched 3956 serve now owns the home
10:45:12.176  boot-service.log  ServerAlreadyRunningError from runtime/versions/...3977/t3
              [service-launcher] Another T3 Code server owns this T3 home. Stop it, then restart this service.

The update was recorded as failed, state-dir-owned. The desktop showed the update as stuck.

Observations:

The client that requested the update is the one whose SSH reconnect won the race. The launch script retried about 12 s after its first lock failure and succeeded because the old server had released the lock and the candidate had not taken it yet.
The candidate needed about 61 s from handoff to its ownership attempt. That leaves the SSH reconnect plenty of time every time.
The SSH-launched server runs the old version, so the target of the requested update loses to the version being replaced.
A plain CLI t3 update --yes on a similar host with no client connected passed earlier the same day. That fits the race needing an SSH reconnect and is not evidence of a fix.

On our side we treat #25633 (keep external ownership in the SSH launch path) as the prevention and #25237 as recovery. The only supported workarounds we can see are to close every client, or remove its SSH route, during updates, or to connect over a non-SSH route. If there is a supported way to keep the desktop from launching a server over SSH for an environment that is already service-managed, it would be good to document it here.

Another recurrence, this time with SSH ownership already recorded as external. This is the case #25633 addresses.

Setup: Ubuntu 26.04 host. t3code.service user unit on 0.1.6-nightly.20261031.3956, default home ~/.t3, port 3773. A Windows desktop is connected over SSH, and its ~/.t3/ssh-launch/<key>/managed read external before the update. The desktop's server update targeted 0.1.6-nightly.20261031.3977.

Timeline (UTC, 2026-10-31, from the host logs):

14:44:18.462 boot-service.log: Server update prepared; handing off to the service launcher.
14:44:35.215 ssh-launch/<key>/server.log: Listening on http://127.0.0.1:3773. The desktop's launch script had started its own t3 serve about 17 s after the handoff.
14:46:57.372 boot-service.log: the .3977 candidate exits with ServerAlreadyRunningError. The launcher idles, and service-state.json records update: {status: failed, reason: state-dir-owned, from .3956, target .3977}.
14:47:15.873 A further SSH launch attempt also fails with ServerAlreadyRunningError.

The same sequence occurred earlier on this host (00:00:20 → 00:03:05 EDT). A second Linux host with a Windows SSH client failed the same way on the same update. Both hosts ended up back on the service-owned .3956 with the update recorded as failed.

Source check:

v0.1.6-nightly.20261031.3977 does not change serviceLauncher.ts, packages/ssh or selfUpdate.ts compared with .3956.
On main at 15fe283, #idleWhileStateDirOwned and the launch script's fallback (clear external after a missed readiness probe, then pick_port + t3 serve) are both unchanged.
fix(server): service launcher starts its server again once another server lets go of the T3 home #24230 (launcher recovery) was closed without merging on Oct 29.

Because external was already recorded here, #25633 (keep recorded external ownership through the readiness budget, and never launch a replacement) is the change that would have prevented this occurrence. #27275 would have ended the client's 32-minute "Restarting…" wait. Neither changes launcher recovery after a lost race, so a host stays idle until someone steps in.

Maintainers: could you say whether #25633 is acceptable as the SSH-side fix, and whether launcher recovery (fix 1 here) is still wanted after #24230 was closed? Our workaround for now is to update with every SSH desktop client fully quit. We have not removed locks, killed servers or edited service state.

Recurrence on stable 0.1.7, on the same kind of setup as the Oct 31 report above (Pop!_OS host, t3code.service, macOS desktop 0.1.7 over SSH via Tailscale, T3 Connect linked, default home ~/.t3, port 3773). This adds three things: both attempts to reach 0.1.7 collided, the two servers then collided in the other direction, and the desktop's error pointed at the network.

Timeline (host-local, CET):

Nov 4 09:14:27  boot-service.log  Server update prepared; handing off (target 0.1.7)
Nov 4 09:14:56  ssh-launch/<key>/server.log  Listening on http://127.0.0.1:3773
Nov 4 09:15:09  boot-service.log  ServerAlreadyRunningError; launcher idles

Nov 5 13:37:34  boot-service.log  Server update prepared; handing off (target 0.1.7)
Nov 5 13:37:54  ssh-launch/<key>/server.log  Listening on http://127.0.0.1:3773
Nov 5 13:38:16  boot-service.log  ServerAlreadyRunningError; launcher idles
Nov 5 14:10:48  journal  t3code.service stopped; the SSH-launched server's /ws span ends in the same second
Nov 5 14:10:53  journal  t3code.service started
Nov 5 14:11:00  boot-service.log  Listening on http://127.0.0.1:3773
Nov 5 14:11:03  ssh-launch/<key>/server.log  ServerAlreadyRunningError
Nov 5 14:12:08  server trace  desktop reconnects; ssh-launch `managed` now reads `external`

What's new:

The update was needed twice. The host was rebooted at 18:47 on Nov 4, and later service stack traces show the 0.1.6 runtime, so the first update to 0.1.7 does not appear to have stuck. This matches the version rollback described above on 0.1.5.
The collision can also run in reverse. At 14:11:03 the SSH-launched server lost the owner lock to the service. The desktop was disconnected for about 80 seconds, from 14:10:48 to 14:12:08.
The desktop blamed the network. The user saw the diagnosis below while SSH from the Mac still worked and t3 connect status passed on the host:

T3 Code connection to <host>: T3 Code on <host> did not answer
Cause: timeout
Attempts: 4
Route: Direct
Error: <host> did not respond during connection setup. Your DNS or firewall may be blocking T3 Connect.

The timing of that diagnosis is not fully explained: four attempts should take longer than the 80-second gap if the environment has two routes, and no failed attempts appear in the host trace. I can't confirm it came from this collision.

0.1.7 is the latest stable, #24230 was closed without merging, and #25633 is still open, so stable users have no fix yet.

Triaged with Claude Code (claude-opus-5-5) via t3 triage.

Another reproduction with a Windows desktop connected over SSH to a Linux ARM64 T3 background service, both on Nightly.

During a server update, the managed server stopped for the handoff. An SSH-launched server appeared during that gap, and the update failed with state-dir-owned. The service launcher remained active without its server child, and the desktop showed “restarting”. The manual-update warning returned afterward.

Process and update-state snapshots support an ownership conflict during the handoff; they do not establish the exact internal reconnect decision. Please include Windows-to-Linux SSH connections when testing a fix.

Written by claude-sonnet-5-5 on behalf of Anthony

Same idle-launcher state on macOS (launchd), triggered by a self-update handoff, which surfaces as "needs a manual update" in the update dialog

Adding a second environment. I haven't confirmed the cause of the competing server here. The effects match this issue.

Environment

macOS arm64, background service installed (com.t3tools.t3code.service → t3 __service-launcher), default home ~/.t3, port 3773.
The desktop app (nightly) was also installed. I don't know what started the competing server.
Server 0.1.8-nightly.20261105.4218. The client that showed the symptom was desktop nightly 0.1.8-nightly.20261106.4247 on Windows, connected remotely.

What I saw

The remote update UI listed this host as "1 skipped… needs a manual update" on every attempt.
ps showed t3 serve --host 127.0.0.1 --port 3773 --base-dir ~/.t3 with parent pid 1, not the __service-launcher. The launcher was running, but its state was speculative and it had no child.

boot-service.log showed the sequence below. The same ServerAlreadyRunningError pair appears at three earlier times in the same log:

00:54:08 INFO  Server update prepared; handing off to the service launcher.
00:54:26 (a `t3 serve` with parent 1 starts, on the newly prepared runtime)
00:54:43 ERROR ServerAlreadyRunningError: A T3 Code server already owns <state dir>
[service-launcher] Another T3 Code server owns this T3 home. Stop it, then restart this service.

So the launcher's child lost the owner lock to a server that was started outside it, and the launcher then stayed idle for about 29 hours. A server not running under the launcher doesn't advertise self-update, so the client shows "needs a manual update" (same mechanism as Update banner asks for a manual update on an SSH server the desktop app already updates #25528).

Fix that worked
kill -TERM <stray server pid>, wait for exit, then launchctl kickstart -k gui/<uid>/com.t3tools.t3code.service. The launcher then ran its own server child, and the user could update from the remote UI.

Takeaways

The trigger doesn't need an SSH launch. A self-update handoff plus any other starter racing for the lock is enough.
The launcher never retries after the owner exits, which matches the "stays idle" part of this issue. An idle launcher that logs the failure but doesn't recover is easy to miss for days.
A hint in the update UI would help here, for example "a server not managed by the service owns this home", instead of the generic "manual update".

Triaged with t3 triage (CLI 0.1.7) using Claude Sonnet 5.5 in Claude Code.

Note

🤖 Claude Opus 5.5 responding on behalf of vpsman

Still reproduces on 0.1.8-nightly.20261107.4314, and tunnel.ts is unchanged on main at e455200. One trigger is left that neither open PR covers: an installed service that is stopped.

Setup: a Linux host running t3 service (systemd user unit, linger on) and two desktop clients connected over SSH. I rendered the launch and stop scripts from the tag and ran them over SSH the way a desktop does. They used a separate state key seeded with managed=external and port=3773, like the desktops' record, so the desktops' own record stayed untouched.

With the service running, the launch printed {"remotePort":3773,"serverKind":"external"} in 1.5 s.
systemctl --user stop t3code.service at 17:47:57.39, launch started straight away. At 17:48:02.41 it printed {"remotePort":3773,"serverKind":"managed"}. server-runtime.json named a new t3 serve --host 127.0.0.1 --port 3773 --base-dir ~/.t3 with parent PID 1 and no serviceManaged.
systemctl --user start t3code.service at 17:48:09.69. Eight seconds later the unit read active (running) with NRestarts=0 and no child. boot-service.log had ServerAlreadyRunningError and Another T3 Code server owns this T3 home.
I stopped the stray server with the desktop's stop script at 17:48:36.78. Both desktops reconnected at 17:48:39 and had their own server listening again by 17:48:44. The launcher was still idle at 17:48:57. (The two desktops starting a server at the same moment is a separate bug, [Bug]: Two desktops launching one SSH host at once both start a server, and the record keeps the dead one's PID #32549.)

On the other host, during a client-requested update, the old server handed off at 16:53:15.35 and the new one listened at 16:53:19.17. That's a gap of about 4 s, twice the 2 s reuse probe. No desktop connected in that window, but after an unexpected stop the desktops re-ran the launch within 2 to 3 s.

What the open PRs change, from reading their diffs. I haven't run them.

fix(ssh): preserve external server ownership during updates #25633 waits the full 60 s for the port when the record says external. That covers the ~4 s update swap. When no runtime is alive it still clears the record and starts a managed server, so step 2 still happens, 60 s later. Its description says it does not detect an installed service.
fix(server): restart the service after T3 home ownership clears #25237 restarts the launcher's server once the stray one exits. Step 4 shows the race it then runs: its poll is every 30 s, and the desktops relaunch within seconds.

A proposal for the remaining case: before falling back to t3 serve, the launch checks whether a service is installed for the default home. If it is installed and not running, the launch starts it. Either way it then waits up to REMOTE_READY_TIMEOUT_MS for the service's server and attaches as external. If that server never answers, as with the idle launcher in this issue, the launch fails instead of starting a second server. It never restarts a running service, because a restart mid-update would kill the trial child. Archive-mode hosts have no Node, so the check would be a new __ssh-helper subcommand beside pick-port that reuses BootService.status (installed, base dir) and the manager's start step. The dev runner (T3_NODE_SCRIPT_PATH) would skip it.

This needs no new liveness record, unlike fix 2 above, because the service manager already knows whether the unit is installed. Would you take it as a follow-up to #25633, or should it get its own issue? I can implement it either way.

Still reproduces on today's nightly (launcher 0.1.8-nightly.20261106.4275, update target 0.1.8-nightly.20261108.4367). It's the same handoff race, with two details I haven't seen in this thread: a different desktop started the update, and while the stray server ran, no client without SSH could reach the host.

Setup: Debian 12 VM, t3code.service user unit with linger, default home ~/.t3, and T3CODE_HOST=0.0.0.0, T3CODE_PORT=3774 in service.env. Two macOS nightly desktops. Desktop A had an SSH environment for this host. Desktop B, a phone and peer links from other hosts reached it over HTTPS, through a reverse proxy to <host>:3774.

Timeline (host-local):

16:43:59.663  boot-service.log            Server update prepared; handing off to the service launcher.   (update to 4367, started from desktop B)
16:44:15.198  ssh-launch/<A>/server.log   Listening on http://127.0.0.1:3774                              (desktop A reconnects inside the 16 s gap)
16:44:32.551  boot-service.log            ServerAlreadyRunningError ... No server was stopped.
              service-state.json          update.status "failed", reason "state-dir-owned"
16:56:46.304  ssh-launch/<A>/server.log   Listening on http://127.0.0.1:3774                              (desktop A updated itself and relaunched)
~17:05        SSH environment removed in desktop A; its server exits. Nothing listens on 3774.
              `__service-launcher` stays active with no child.
17:09:36.252  boot-service.log            Listening on http://0.0.0.0:3774                                (only after a manual `t3 update ... --yes`)

Desktop B asked for the update over HTTPS, and desktop A, which only had an SSH route, happened to reconnect inside the handoff gap. The SSH desktop doesn't have to update or restart anything to trigger this.

The SSH-launched server took the service's port (3774) but listened on 127.0.0.1 only, while the service binds 0.0.0.0. From 16:44 to 17:09 the reverse proxy returned 502 for desktop B, the phone and the peer links. Desktop A saw a healthy host, and nothing on the host or in B's UI said the service no longer owned it.

After the stray server exited, the launcher stayed idle until I restarted the service by hand. #25237 covers that part and would have ended the outage around 17:05. #25633 would probably have prevented the takeover at 16:44. Neither is in 4367. Until they land, the workaround I'm using for a host that runs the background service is to add it to desktops by its HTTPS address, never as an SSH environment.

Another variant of this race, hit during the launcher's update handoff rather than a plain restart, with an extra risk.

Environment: Linux host running t3code.service, Windows desktop clients connected over SSH, nightlies 4449 → 4452 / 4465 / 4478 / 4491.

What happens:

An update is requested from the app (server.updateServerWithProgress).
The launcher stops the active server and starts its full cp backup of a 55 GB statev2.sqlite (16–25 min here).
About 10 s later, the desktop client's SSH launch runs t3 serve --host 127.0.0.1 --port 3773 --base-dir ~/.t3 outside the service. It takes server-owner.sqlite and serves, writing to the database while the launcher is still copying it, so the backup is torn.
The trial child exits 78 (ServerAlreadyRunningError). The launcher records failed / state-dir-owned, deletes its backup and idles with "Another T3 Code server owns this T3 home."

We saw this four times (Nov 9 22:45 and 23:28; Nov 10 10:49 and again that evening), and once more during a maintenance window with the service stopped.

Extra risk: if the trial exits with any code other than 78, #returnToPrevious restores the torn copy over a database that the stray server is actively writing.

Fix suggestion:

The guard already exists: olderBootServiceHome refuses an external server while the boot service is active and runtime/.restart-pending exists, and t3 update sets that file. But the launcher's own update handoff never does. Have the launcher set it, or hold the ownership lock, from handoff until the trial child starts.
Desktop SSH launches shouldn't start a server for a home that has a boot service installed. They should wait and reconnect instead (as this issue proposes).

Workaround: run t3 update <version> --yes from a shell outside T3. It sets .restart-pending itself, does no copy, and has been reliable for us (4449 → 4478, 4491 → 4501).

Still reproduces on stable 0.1.9 (service-initiated update 0.1.7 → 0.1.9, triggered from a macOS desktop 0.1.9 that also had an SSH environment for the host). Hit twice in a row, Nov 11 and Nov 12.

Setup: Pop!_OS (Linux x64, kernel 6.12), t3code.service user unit (ExecStart → runtime/versions/0.1.7/t3 __service-launcher), drop-in T3CODE_HOST=127.0.0.1, default home ~/.t3, port 3773, T3 Connect linked. Client: T3 Code desktop 0.1.9 on a MacBook Pro, connected over SSH via Tailscale.

Timeline (host-local, Nov 12):

05:59:23.564  boot-service.log  Server update prepared; handing off to the service launcher (target 0.1.9)
05:59:30      sshd: new session from the Mac; launch script runs `nohup t3 serve --host 127.0.0.1 --port 3773 --base-dir ~/.t3` (0.1.9, PPID 1, session scope, ssh-launch/<key>/managed = managed)
06:00:14.928  boot-service.log  launcher's 0.1.9 child: ServerAlreadyRunningError → "[service-launcher] Another T3 Code server owns this T3 home"
              service-state.json: activeVersion 0.1.7, update failed, reason state-dir-owned

The desktop shows "restarting…" indefinitely.

Two additions:

After the race, the stray SSH-launched 0.1.9 server has already migrated the DB while service-state.json still says activeVersion: 0.1.7, so a plain systemctl --user restart would bring 0.1.7 back on a 0.1.9-migrated database (the same risk noted above for 0.1.5).
t3 service install (the recommended repair) re-triggers the race itself: its stop/start opened another gap, and the still-connected desktop launched a second stray server within ~2 s (06:11:47 pid file → 06:11:51 ServerAlreadyRunningError). Recovery only worked once the desktop was fully quit.

Workaround that worked: fully quit the SSH-connected desktop, stop the stray t3 serve, run t3 service install from the 0.1.9 runtime (moves ExecStart + activeVersion to 0.1.9), then use T3 Connect instead of an SSH environment for that host.

Posted by Claude Code (Opus 5.5) on behalf of the reporter, from a t3 triage session.

Another repro: the stuck update is also visible from a remote client (Windows GUI → macOS service)

What happened: I pressed Update in the Windows desktop app, targeting a macOS service-managed server. At 15:38:11 the server logged Server update prepared; handing off to the service launcher (target 0.1.10-nightly.20261113.4616). About 14 seconds later an unmanaged t3 serve --host 127.0.0.1 --port 3773 --base-dir ~/.t3 (older 0.1.9-nightly.20261110.4452, parent pid 1) took the owner lock. The 0.1.10 server then failed with ServerAlreadyRunningError. runtime/service-state.json recorded update.status: "failed", reason: "state-dir-owned". The launcher printed Another T3 Code server owns this T3 home. Stop it, then restart this service. and stayed idle.

Client symptom: the Windows GUI stayed on "Updating" for 15+ minutes. It was connected to the orphan server, which never reported the failure.

Recovery: stopping the orphan was not enough. The launcher did not retry after the lock was released. I also had to run launchctl kickstart -k gui/<uid>/com.t3tools.t3code.service.

Environment: macOS arm64 service. The launchd unit still points at the 0.1.8-nightly.20261104.4165 launcher (see #17490). The state DB is about 13 GB. The Windows client was the latest nightly.

Unknown: what started the competing t3 serve. I don't know whether the update flow itself or something else on this machine did.

Triage CLI 0.1.9, with the diagnosis done by Claude (Sonnet 5.5) in a Claude Code agent session.

Another occurrence on Linux (x64, kernel 6.8), on builds older than the fix in #25237.

Setup: t3code.service systemd user unit, ExecStart → runtime/versions/0.1.7-nightly.20261101.4048/t3 __service-launcher, default home ~/.t3, port 3773, T3 Connect linked.

What happened:

The service started on 2 Nov 00:12 CET. Its launcher logged ServerAlreadyRunningError / "Another T3 Code server owns this T3 home. Stop it, then restart this service." It then stayed idle for 11 days with NRestarts=0 and no server child.
Meanwhile a 0.1.10-nightly.20261112.4564/t3 serve --host 127.0.0.1 --port 3773 process (parent pid 1) ran in an sshd login session scope (session-*.scope), not in the service cgroup. It was started on 12 Nov 09:51 CET from an SSH connection, most likely the desktop app's SSH environment.
t3 service status (4564) reported "needs an update or repair … Run t3 service install to repair it."
npx [email protected] then failed with "A T3 Code server is already running … Connect to that server, stop it before starting another".
After the SSH-started server was stopped with SIGTERM, the idle 0.1.7 launcher did not take over. Port 3773 stayed closed until the service was reinstalled.

After repair on 0.1.10-nightly.20261113.4630: 0.1.10-nightly.20261113.4630/t3 service install repointed the unit's ExecStart to the 4630 runtime and restarted the service. The launcher started a 4630 t3 serve child inside the service cgroup, ran the pending DB migrations, and the server answered HTTP 200 on port 3773 within seconds (NRestarts=0). So the repair path works. I haven't yet seen whether 4630 recovers on its own if an SSH-started server takes the lock again.

Triaged with Claude Code (Opus 5.5) via t3 triage.

Navigation Menu

Repository navigation

[Bug]: Service launcher stays idle after an SSH-launched server takes the owner lock during a restart, and never recovers #24225

Before submitting

Area

Steps to reproduce

Expected behavior

Actual behavior

Impact

Version or commit

Environment

Logs or stack traces

Proposed fix

Workaround

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Bug]: Service launcher stays idle after an SSH-launched server takes the owner lock during a restart, and never recovers #24225

Description

Before submitting

Area

Steps to reproduce

Expected behavior

Actual behavior

Impact

Version or commit

Environment

Logs or stack traces

Proposed fix

Workaround

Activity

ishanavasthi commented on Oct 25, 2026

beastyrabbit commented on Oct 29, 2026

ViperJuice commented on Oct 31, 2026

jrgong420 commented on Oct 31, 2026

ViperJuice commented on Oct 31, 2026

ViperJuice commented on Oct 31, 2026

firstclassformula commented on Nov 5, 2026

codewarnab commented on Nov 6, 2026

abassaf commented on Nov 6, 2026

vpsman commented on Nov 7, 2026

keskinonur commented on Nov 8, 2026

ViperJuice commented on Nov 11, 2026

firstclassformula commented on Nov 12, 2026

abassaf commented on Nov 13, 2026

linksmith commented on Nov 13, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions