Repository navigation
[Bug]: Remote server updates can't cross a service launcher protocol bump without a shell on the server machine #34502
Description
Before submitting
- I searched existing issues and did not find a duplicate.
- I included enough detail to reproduce or investigate the problem.
Area
apps/server
Steps to reproduce
- Run a server as the background service (
t3 service install) whose launcher is protocol 3. In my case the unit still ran the launcher from0.0.46-nightly.20261003.2610after several successful remote updates had moved the server to0.1.9-nightly.20261111.4501. - From another device (desktop app or app.t3.codes), update that server to
0.1.10-nightly.20261111.4529or any later release withSERVICE_LAUNCHER_PROTOCOL = 4.
Expected behavior
A server that only gets used remotely can be updated remotely, including across a launcher protocol bump. It can do that on its own, or at least through some path that doesn't need a shell on the host.
Actual behavior
The update fails with:
The installed service launcher speaks protocol 3, but [email protected] requires protocol 4. On the server machine, install this exact CLI: `curl -fsSL https://t3.codes/install.sh | T3CODE_VERSION=0.1.10-nightly.20261111.4529 sh`. Check that `t3 --version` prints 0.1.10-nightly.20261111.4529 and that `which -a t3` lists no older t3 (such as an npm or nvm install) first, then run `~/.local/bin/t3 service install` (the installer's default path). ...
The comment in serviceProtocol.ts says this is intentional ("an installed protocol-3 launcher needs one local update or service install with a restart: remote updates replace its child"). Still, from a user's side every remote-only server gets stuck at each protocol bump until someone opens a shell on that machine. Launchers outlive remote updates, so the stuck launcher can be very old: mine was 0.0.46 while the server was on 0.1.9.
There's also a trap: an old npm/nvm global t3 earlier on PATH makes a bare t3 service install reinstall the old version. The message warns about it, but only after the user has already hit it once.
Impact
Major degradation or frequent failure
Version or commit
Server 0.1.9-nightly.20261111.4501 updating to 0.1.10-nightly.20261111.4529. Launcher 0.0.46-nightly.20261003.2610.
Environment
Linux (WSL2), systemd user service, remote update triggered from another device.
Workaround
On the server machine, follow the error message: install the exact CLI with the install script, uninstall any npm global t3, then run ~/.local/bin/t3 service install. When the agent doing this runs under that same server, start it detached, e.g. systemd-run --user ~/.local/bin/t3 service install, so the restart doesn't kill it partway through.
Possible direction
When the staged preflight is blocked only by the launcher protocol, the running server has already downloaded and checked the target runtime. It could then run <staged runtime>/t3 service install outside its own service cgroup (systemd-run --user on Linux, the launchd equivalent on macOS). That rewrites the unit and restarts it onto a current launcher. Bumps that ship before servers have this code would still need the manual step, but later ones wouldn't. It does mean the server rewrites its own unit without a human in the loop, so this may conflict with why it's manual today.
Filed with Claude Code (Opus 5.5).
Activity
Zedyas commentedon Nov 11, 2026
Two more hosts hit this on the same update (0.1.9 nightly → 0.1.10-nightly.20261111.4529), both systemd user services:
On both,
ExecStartstill pointed at the launcher from the stable0.1.0release that ran the originalt3 service install. Both servers were on 0.1.9 builds from the same day, so neither was out of date. Thet3on PATH was0.1.3-nightly.20261023.3579on both, which is the CLI-lag behavior from #14609.The workaround in the error message fixed both. On the bare-metal host, the restart also stopped an Android emulator running under the service, because
KillMode=mixedstops every process in the service cgroup. A remote-safe upgrade path should probably warn before a restart that ends child processes.ishaanko commentedon Nov 11, 2026
I looked at how big a fix would be on
main.Most of it exists already on Linux. When
t3 service installruns inside the service on systemd, it hands itself to a transient unit viasystemd-run(bootService.ts, "handing the install to a transient unit"). That way the restart doesn't kill it partway through. So a server whose update is blocked only by the launcher protocol could run<staged runtime>/t3 service install --base-dir <baseDir>, and the staged CLI already knows how to survive the restart.Rough shape:
blockedBy: "launcher-protocol", so the server can tell this case apart from a too-new database. Today it only returns a text reason. The new field is additive, so mixed versions still decode each other.selfUpdate.ts, handle that case by running the staged runtime'sservice installinstead of failing, and report "restarting onto a new launcher" to the client.Caveats:
handoff, so on macOS a self-install would stop the service and kill itself. It would need a detached launchd job. Windows has no boot service manager (see [Bug]: Remote update fails on Windows hosts with "launcher speaks protocol 3" and only Linux/macOS fix steps #34435).serviceProtocol.tsreads like the manual step was chosen on purpose, so it's worth confirming you want the server rewriting its own unit unattended.Alternative: the launcher could exec into the staged launcher in place instead of rewriting the unit. That avoids the cgroup handoff and works under launchd too, with the unit catching up at the next reconcile. I haven't checked whether the bundled Node runtime supports that.
andrewfree commentedon Nov 11, 2026
Same failure on macOS with a launchd background service (macOS 27.2, Apple Silicon arm64). These facts were collected read-only on 2026-11-11 at 20:00 UTC.
0.1.10-nightly.20261111.4529, already the update target.0.1.9-nightly.20261110.4465. Its/.well-known/t3/environmentdescriptor reportsserverSelfUpdate: "boot-service".ProgramArgumentsand the live launcher process both run<T3 home>/runtime/versions/0.0.46-nightly.20261013.3053/t3 __service-launcher.The server trace has 9 failed server-update calls with the launcher protocol error. The most recent, at 19:59 UTC, includes:
This confirms the failure on launchd while the desktop app on the same machine is already on the target build.
Not tested: the manual repair from the error message has not been run on this host, so I can't say whether it works under launchd.
Investigated with GPT-6.1 Sol via Codex in T3 Code. Reviewed by Claude Fable 5.2 via Claude Agent in T3 Code.
gertjvr commentedon Nov 11, 2026
Another Linux host hit this on the same update (0.1.9-nightly.20261111.4515 → 0.1.10-nightly.20261111.4529). It's a systemd user service on Debian 13, x64 (kernel 6.12).
ExecStartpointed at0.1.6-nightly.20261031.3977/t3 __service-launcher. Every in-app update since then (21 "Server update prepared; handing off to the service launcher" entries inboot-service.log) replaced only the server runtime and left the launcher pinned.service-state.jsonsaidprotocol: 3the whole time.t3on PATH (~/.local/bin/t3) was already0.1.10-nightly.20261111.4529. The curl install step wasn't needed here, but the in-app update still couldn't proceed. A current CLI on PATH doesn't help unless someone runsservice installby hand.~/.local/bin/t3 service installrewrote the unit, restarted onto the 0.1.10 launcher (protocol: 4), and/.well-known/t3/environmentnow reports 0.1.10-nightly.20261111.4529.app-t3code-agents-*.slice/t3code-claude-*.scope), not in the service cgroup, so this isn't onlyKillMode=mixed. The server shutdown takes its agent sessions down with it. It's another reason for a warning before a restart, as suggested above.Investigated with Claude Opus 5.5 via Claude Code (
t3 triage).AojdevStudio commentedon Nov 11, 2026
One more macOS host, and a setup where the fix the error suggests is unsafe.
Environment. macOS 27.3, arm64. Server
0.1.9-nightly.20261110.4478updating to0.1.10-nightly.20261111.4529.service-state.jsonsaidprotocol: 3. Five more hosts of mine were on a protocol 3 launcher too: a second Mac with the same wrapper, one plain launchd Mac, and three Linux systemd user services. Each ran a launcher from a0.1.8nightly while serving a0.1.9build. On the plain systemd and launchd hosts, the workaround in the error fixed it.The suggested fix breaks a custom launchd wrapper. On two Macs I run the server from a small signed
.appwith its own launchd label, and T3'scom.t3tools.t3code.serviceislaunchctl disabled. The reason is Full Disk Access. macOS ties it to a bare binary's full path, and each update moves the launcher to a newruntime/versions/<version>/t3path, so the grant stops applying. A signed app keeps one identity across updates.At tag
v0.1.10-nightly.20261111.4529,t3 service installrunslaunchctl enableand thenlaunchctl bootstraponcom.t3tools.t3code.service(the launchdactivatesteps inapps/server/src/cloud/bootService.ts). That clears the user'sdisableand starts a second launcher beside the wrapper.What worked. The wrapper reads
activeVersionfromruntime/service-state.jsonand execs that runtime'st3 __service-launcher. So the fix was:install.shand run it withT3CODE_VERSION=0.1.10-nightly.20261111.4529. It touches no launchd job.service-state.jsonas{"protocol": 4, "activeVersion": "0.1.10-nightly.20261111.4529"}, the same contentservice installwrites.Both Macs now serve
0.1.10-nightly.20261111.4529with protocol 4. T3's label stays disabled, and Full Disk Access still applies.Suggestion. The boot service
installalready takes{ start: false }(bootService.ts:874), and its tests use it.t3 service installexposes only--allow-downgrade. A--no-startflag that writes the runtime, state, and unit but skipsenableandbootstrapwould give wrapper setups a supported way across a protocol bump. That is separate from the remote-only fix discussed above.Written through
t3 triageby Claude Opus 5.5 in Claude Code.