Skip to content

Commit 890d339

Browse files
authored
Merge pull request #2130 from lidge-jun/fix/systemd-bus-unreachable-disk-check
fix(probe,codex): stop fencing native-main on a probe that could not answer
2 parents ceac592 + 50386c1 commit 890d339

6 files changed

Lines changed: 689 additions & 3 deletions

File tree

Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
# 170 — #2114 and #2108: the fence that could not re-ask
2+
3+
Both reports are the same shape wearing two operating systems: a probe that
4+
could not get an answer, a verdict of `unknown`, and a fence that holds for the
5+
whole process. Every native request 503s and only `ocx restart` clears it.
6+
7+
Shipped as PR #2130, five commits.
8+
9+
## The three changes
10+
11+
**#2114 — the Linux session bus.** `inspectSystemd` called every non-zero exit
12+
`unknown`. Its own comment was right that a non-zero status means the question
13+
never reached the bus — and that is exactly why the verdict was wrong. It is
14+
evidence about the bus, not about who owns this home.
15+
16+
Widening on the exit code alone fails open: with the bus down `systemctl`
17+
cannot see a foreign unit either, so "no answer" would read as "no owner" on a
18+
machine that has one. The classification asks the disk instead, which needs no
19+
bus.
20+
21+
**#2108 (a) — an unaskable WinSW query.** WinSW is optional, and a
22+
scheduler-only install has neither its XML nor its exe on disk. A timed-out
23+
`sc.exe query` still returned `unknown`, which outranks the disk. With both
24+
assets gone there is nothing for a registration to belong to.
25+
26+
**#2108 (b) — the fence re-asks.** `startServer` takes the verdict once and
27+
holds it, which is why waiting never helped and restart always did. Correct for
28+
`foreign-ownership` — a foreign owner is a fact. Wrong for
29+
`ownership-unknown`, which says the probe could not answer.
30+
31+
## What the audit caught, three rounds running
32+
33+
This is the part worth keeping. The functional idea was right in round one; the
34+
**ownership model was wrong three times in a row**, and each time the symptom
35+
was identical — a fence with no way to re-ask, which is #2108 returning by
36+
another route.
37+
38+
| Round | Defect | Why it slipped |
39+
|---|---|---|
40+
| 1 | `inspectSystemdOffline` checked ONE path, so a foreign unit in `~/.local/share/systemd/user` or an XDG override was invisible → `ownership: owned` on a foreign host | The bus-up path never had the hole; I reused its constant, not its coverage |
41+
| 2 | The probe decremented the refcount AND the fence's own `release()` decremented again → a fence another holder owned got lifted | `Math.max(0, ...)` floored it, so it under-counted silently instead of going negative |
42+
| 2 | A spent hook stayed in the map, so no LATER fence could install one | The "do not reset the budget" guard was right; its scope was not |
43+
| 3 | Owner-identity guard was correct but **untested** — replacing it with `if (true)` left every suite green | A third route to the same wedge, with nothing pinning it |
44+
45+
Round 1's fail-open is the one that mattered: I wrote "a unit naming a foreign
46+
home stays present and still blocks" in a commit message, and the auditor proved
47+
it end-to-end as `ownership = owned`. The claim was true only for a unit at the
48+
canonical path.
49+
50+
The fix that finally held is smaller than either attempt: the hook delegates to
51+
the fence's own idempotent `release()`. One fence, one payment, whoever triggers
52+
it. Both earlier versions were modelling the ownership in the wrong place.
53+
54+
## Guards, and the ones that were not guards
55+
56+
Every branch was ablated. Three that looked like guards were not:
57+
58+
- the multi-unit conflict branch — a fail-closed decision with nothing asserting
59+
it; ablating it left the suite green
60+
- "foreign never retries" — double-implemented, so removing either half alone
61+
stayed green; only removing both failed
62+
- the owner-identity check — round 3's finding, above
63+
64+
All three now have tests driven red against the real code.
65+
66+
## Stated rather than hidden
67+
68+
**Locale.** systemd localizes the bus-failure strings, so a non-English host
69+
will not match and keeps the old `unknown`. That fences rather than admits,
70+
which is the safe direction, but the #2114 fix does not reach every affected
71+
user. Forcing `LC_ALL=C` on the probe would remove the caveat and is the
72+
obvious follow-up; it is not done here because it changes every `systemctl`
73+
call the module makes.
74+
75+
**Coverage.** These are Windows and Linux platform paths. The suites that prove
76+
them ran on macOS and on `ssh lidge`; the platform CI legs are the check that
77+
actually matters.
78+
79+
**A hostile `XDG_CONFIG_HOME`** redirects the offline check to an
80+
attacker-chosen directory. The auditor raised it and then cleared it: reaching
81+
that requires controlling the proxy's own environment, at which point
82+
`CODEX_HOME` is equally controllable and the comparison is moot. No privilege
83+
boundary is crossed.
84+
85+
## Coordination with #2029
86+
87+
Draft PR #2029 edits the same function for #1939 and classifies two other bus
88+
messages as `absent`. This branch never touched it or its branch. The two
89+
agree in direction; if it lands first this reconciles rather than replaces.
90+
91+
## Verification
92+
93+
```
94+
full suite on ssh lidge 13,524 pass / 15 skip / 0 fail across 855 files
95+
probe + startup + boundary 96 pass / 0 fail
96+
bun x tsc --noEmit exit 0
97+
bun run privacy:scan exit 0
98+
```
99+
100+
One CI failure was investigated and dismissed on evidence: `keyring ubuntu`
101+
stalled eight minutes on an apt mirror and was cancelled by the job timeout —
102+
an infrastructure fault with no relation to the diff.
103+

‎src/codex/native-profile-startup.ts‎

Lines changed: 96 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ import {
1515
import { withNativeMainExclusiveClaim } from "./native-main-claim";
1616
import { scrubNativeMainAuthTempResidues } from "./native-main-auth-temp";
1717
import { NATIVE_STAGE_SWEEP_INTERVAL_MS } from "./native-profile-stage-store";
18+
import type { NativeCodexOwnership } from "../integrations/native/ownership-preflight";
1819

1920
export type NativeMainStartupGateSnapshot =
2021
| { status: "ready"; homeId: string | null }
@@ -310,6 +311,75 @@ export function startNativeMainStartupLifecycle(
310311
};
311312
}
312313

314+
/**
315+
* How many times a service-ownership fence will re-ask before it stops asking (#2108).
316+
*
317+
* A host that is permanently unaskable must not re-probe on every request forever, and a
318+
* host that recovers usually does so within the first few. The budget belongs to the
319+
* REASON, not to an individual fence: raising a second fence deliberately does not hand
320+
* out a fresh allowance, or a caller looping over fences could spin the probe forever.
321+
* It is dropped when the last fence for that reason releases.
322+
*/
323+
export const NATIVE_MAIN_OWNERSHIP_RETRY_LIMIT = 5;
324+
325+
/** Reprobe hooks for the fences currently held, keyed by the reason they were raised for. */
326+
const serviceOwnershipReprobes = new Map<NativeMainServiceOwnershipBlockReason, ServiceOwnershipReprobe>();
327+
328+
interface ServiceOwnershipReprobe {
329+
readonly probe: () => NativeCodexOwnership;
330+
attempts: number;
331+
/** The fence that installed this hook; only its own release may drop the entry. */
332+
readonly owner: NativeMainStartupLifecycle;
333+
/** Releases the fence that installed this hook, exactly once. */
334+
readonly spend: () => void;
335+
}
336+
337+
/** Test-only: the retry budget is module state and would otherwise leak across tests. */
338+
export function __resetNativeMainOwnershipRetries(): void {
339+
for (const entry of serviceOwnershipReprobes.values()) entry.attempts = 0;
340+
}
341+
342+
/**
343+
* Re-ask whether this host is still unownable, and drop the fence if it is not.
344+
*
345+
* `startServer` takes the ownership verdict once, at boot, and holds it for the process
346+
* lifetime. For `foreign-ownership` that is correct — a foreign owner is a fact, and
347+
* re-asking would only hand a determined caller a second chance. For `ownership-unknown`
348+
* it is wrong: that verdict means the probe could not answer, so waiting cannot help,
349+
* which is precisely why the #2108 reporter had to run `ocx restart` after every reboot.
350+
*
351+
* The re-probe is demand-driven rather than timed: it runs when something asks whether
352+
* native-main is fenced, which is usually a request but is also the background token
353+
* guardian's warmup. It is capped so a permanently unaskable host cannot spin.
354+
*
355+
* The probe is synchronous `spawnSync` with a bounded timeout, and this function is on a
356+
* request path, so the cap is what keeps a wedged host from paying that cost repeatedly.
357+
*/
358+
function reprobeServiceOwnership(reason: NativeMainServiceOwnershipBlockReason): boolean {
359+
if (reason !== "ownership-unknown") return false;
360+
const entry = serviceOwnershipReprobes.get(reason);
361+
if (!entry) return false;
362+
if (entry.attempts >= NATIVE_MAIN_OWNERSHIP_RETRY_LIMIT) return false;
363+
entry.attempts += 1;
364+
let answer: NativeCodexOwnership;
365+
try {
366+
answer = entry.probe();
367+
} catch {
368+
// An inspection that throws is not evidence the host became ownable.
369+
return false;
370+
}
371+
if (answer !== "owned") return false;
372+
// Release through the fence that installed this hook, and only that one.
373+
//
374+
// Several servers can hold a fence for the same reason while only one carries a hook, so
375+
// clearing the shared refcount here would unblock fences this probe never spoke for.
376+
// Decrementing here directly is just as wrong the other way: that fence's own release()
377+
// would then pay a second time for one fence, leaving the count short. Delegating to the
378+
// fence's idempotent release keeps exactly one payment per fence.
379+
entry.spend();
380+
return true;
381+
}
382+
313383
function activeServiceOwnershipBlockReason(): NativeMainServiceOwnershipBlockReason | null {
314384
if ((serviceOwnershipRefs.get("foreign-ownership") ?? 0) > 0) return "foreign-ownership";
315385
if ((serviceOwnershipRefs.get("ownership-unknown") ?? 0) > 0) return "ownership-unknown";
@@ -325,10 +395,11 @@ function serviceOwnershipSnapshot(
325395
/** Close native-main admission without resolving or creating any CODEX_HOME artifacts. */
326396
export function blockNativeMainStartupForUnownedServiceHome(
327397
reason: NativeMainServiceOwnershipBlockReason,
398+
options?: { reprobe?: () => NativeCodexOwnership },
328399
): NativeMainStartupLifecycle {
329400
serviceOwnershipRefs.set(reason, (serviceOwnershipRefs.get(reason) ?? 0) + 1);
330401
let released = false;
331-
return {
402+
const lifecycle: NativeMainStartupLifecycle = {
332403
homeId: null,
333404
settled: Promise.resolve(serviceOwnershipSnapshot(reason)),
334405
async release() {
@@ -337,8 +408,25 @@ export function blockNativeMainStartupForUnownedServiceHome(
337408
const remaining = Math.max(0, (serviceOwnershipRefs.get(reason) ?? 0) - 1);
338409
if (remaining === 0) serviceOwnershipRefs.delete(reason);
339410
else serviceOwnershipRefs.set(reason, remaining);
411+
if (serviceOwnershipReprobes.get(reason)?.owner === lifecycle) {
412+
serviceOwnershipReprobes.delete(reason);
413+
}
340414
},
341415
};
416+
// Do NOT reset an existing budget: keying the reprobe by reason means a caller raising
417+
// fences in a loop would otherwise be handed a fresh allowance each time and could spin
418+
// the probe forever. But once the holder is gone its entry is removed above, so a LATER
419+
// fence installs its own hook — a server started after an earlier probe must not be left
420+
// needing `ocx restart`, which is the very symptom this exists to remove.
421+
if (options?.reprobe && reason === "ownership-unknown" && !serviceOwnershipReprobes.has(reason)) {
422+
serviceOwnershipReprobes.set(reason, {
423+
probe: options.reprobe,
424+
attempts: 0,
425+
owner: lifecycle,
426+
spend: () => { void lifecycle.release(); },
427+
});
428+
}
429+
return lifecycle;
342430
}
343431

344432
export function bindNativeMainStartupLifecycle(server: object, lifecycle: NativeMainStartupLifecycle): void {
@@ -353,7 +441,13 @@ export async function releaseNativeMainStartupLifecycle(server: object): Promise
353441
}
354442

355443
export function isNativeMainTrafficBlocked(): boolean {
356-
return activeServiceOwnershipBlockReason() !== null || snapshot.status === "blocked";
444+
const reason = activeServiceOwnershipBlockReason();
445+
if (reason !== null && reprobeServiceOwnership(reason)) {
446+
// The host became ownable after boot (#2108): the fence lifts here rather than
447+
// waiting for the restart the reporter had to perform by hand.
448+
return activeServiceOwnershipBlockReason() !== null || snapshot.status === "blocked";
449+
}
450+
return reason !== null || snapshot.status === "blocked";
357451
}
358452

359453
/**

‎src/server/index.ts‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -705,6 +705,11 @@ export function startServer(port?: number, deps: StartServerDeps = {}): Server<W
705705
? startNativeMainStartupLifecycle(deps.nativeMainStartup)
706706
: blockNativeMainStartupForUnownedServiceHome(
707707
nativeOwnership.ownership === "foreign" ? "foreign-ownership" : "ownership-unknown",
708+
// #2108: an `unknown` verdict means the probe could not answer, not that this host
709+
// is unownable. Hand the fence a way to re-ask so a host that becomes answerable
710+
// after boot reopens on its own instead of needing `ocx restart`. A `foreign`
711+
// verdict ignores this by design — that one is a fact, not a question.
712+
{ reprobe: () => inspectStartupOwnership(deps).ownership },
708713
)
709714
: {
710715
homeId: null,

‎src/service-manager-probe.ts‎

Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -179,6 +179,86 @@ function unitEnvValue(body: string, key: string): string | null {
179179
return null;
180180
}
181181

182+
/**
183+
* Did `systemctl --user` fail because the session bus could not be reached at all?
184+
*
185+
* These are the shapes reported on #2114 and #1939. The distinction that matters is
186+
* "the question never left the machine" versus "systemd answered and said no" — only
187+
* the former licenses reading the disk instead.
188+
*
189+
* **Locale caveat, stated rather than hidden:** systemd localizes these strings, so a
190+
* non-English host will not match and keeps the old `unknown`. That is the safe
191+
* direction — it fences rather than admits — but it does mean the fix does not reach
192+
* every affected user. Forcing `LC_ALL=C` on the probe would remove the caveat and is
193+
* the obvious follow-up; it is not done here because it changes every systemctl call
194+
* this module makes, not just this branch.
195+
*/
196+
function busUnreachable(stderr: string): boolean {
197+
const err = stderr.trim();
198+
return err.includes("Failed to connect to bus")
199+
|| err.includes("Failed to connect to user scope bus")
200+
|| err.includes("Failed to get D-Bus connection")
201+
|| err.includes("DBUS_SESSION_BUS_ADDRESS")
202+
|| err.includes("System has not been booted with systemd");
203+
}
204+
205+
/**
206+
* Ownership from the unit file alone, for when the bus cannot answer (#2114).
207+
*
208+
* A unit file is proof of installation that does not require a running bus, and the homes
209+
* it names are what ownership is actually decided on. What the disk cannot tell us is
210+
* whether systemd has the unit LOADED, so this reports `registration: "absent"` — the
211+
* honest reading of "no running manager has it" — rather than inventing a live state.
212+
*
213+
* A foreign home therefore still blocks, which is the whole reason this consults the disk
214+
* instead of widening the exit code.
215+
*/
216+
function systemdUserUnitSearchPaths(home: string): string[] {
217+
// systemd's user search path is not one directory. Checking only the canonical one and
218+
// calling the rest absent is a fail-open: with the bus down a foreign unit in any other
219+
// search dir is invisible, and "no answer" would be read as "no owner".
220+
const xdgConfig = process.env.XDG_CONFIG_HOME?.trim();
221+
const xdgData = process.env.XDG_DATA_HOME?.trim();
222+
const dirs = [
223+
xdgConfig ? join(xdgConfig, "systemd", "user") : join(home, ".config", "systemd", "user"),
224+
join(home, ".config", "systemd", "user"),
225+
xdgData ? join(xdgData, "systemd", "user") : join(home, ".local", "share", "systemd", "user"),
226+
join(home, ".local", "share", "systemd", "user"),
227+
];
228+
return [...new Set(dirs)].map(dir => join(dir, `${TASK}.service`));
229+
}
230+
231+
function inspectSystemdOffline(home: string): ServiceManagerInstallation {
232+
const candidates = systemdUserUnitSearchPaths(home);
233+
const found = candidates.filter(path => artifactPresence(path) === "present");
234+
if (candidates.some(path => artifactPresence(path) === "unreadable")) {
235+
return unknown("the session bus is unreachable and a systemd unit could not be read");
236+
}
237+
if (found.length === 0) return { kind: "absent" };
238+
if (found.length > 1) {
239+
return unknown("the session bus is unreachable and more than one systemd unit file claims this proxy");
240+
}
241+
const definitionPath = found[0]!;
242+
let body: string;
243+
try {
244+
body = readFileSync(definitionPath, "utf-8");
245+
} catch (error) {
246+
return unknown(`the session bus is unreachable and the systemd unit could not be read: ${String(error)}`);
247+
}
248+
return {
249+
kind: "present",
250+
claims: [{
251+
backend: "systemd",
252+
definitionPath,
253+
homes: {
254+
codexHome: unitEnvValue(body, "CODEX_HOME"),
255+
opencodexHome: unitEnvValue(body, "OPENCODEX_HOME"),
256+
},
257+
registration: "absent",
258+
}],
259+
};
260+
}
261+
182262
function inspectLaunchd(deps: Required<Pick<ProbeDeps, "run" | "uid" | "home">>): ServiceManagerInstallation {
183263
const definitionPath = join(deps.home, "Library", "LaunchAgents", `${LABEL}.plist`);
184264

@@ -269,6 +349,15 @@ function inspectSystemd(deps: Required<Pick<ProbeDeps, "run" | "home">>): Servic
269349
if (shown.status !== 0) {
270350
// A missing unit still exits ZERO and says not-found; a non-zero status means
271351
// the question never reached the bus.
352+
//
353+
// That is evidence about the BUS, not evidence that a foreign service owns this home
354+
// (#2114). Calling it `unknown` fences native-main for the whole process, so a laptop
355+
// with no session bus answers every native request with a 503 until `ocx restart`.
356+
//
357+
// Widening on the exit code alone would fail open, because with the bus down systemctl
358+
// cannot see a foreign unit either. So ask the disk, which needs no bus, and fall back
359+
// to `unknown` for every other non-zero exit.
360+
if (busUnreachable(shown.stderr)) return inspectSystemdOffline(deps.home);
272361
return unknown(`systemctl show exited ${String(shown.status)}: ${shown.stderr.trim()}`);
273362
}
274363

@@ -729,6 +818,16 @@ function walkWinswChain(
729818
const registration = probeWinswRegistration(deps);
730819

731820
if (xml === "absent" && exe === "absent" && registration === "absent") return { kind: "absent" };
821+
// A query we could not ask is a question about a service that cannot exist: WinSW is an
822+
// optional backend, and with neither its XML nor its exe on disk there is nothing for a
823+
// registration to belong to. Fencing here on an `sc.exe` timeout is one of the two
824+
// triggers behind #2108, where a scheduler-only install answers 503 until `ocx restart`.
825+
//
826+
// The disk outranks the unaskable query only when BOTH assets are gone. Either one
827+
// present means a real install may be there and the old `unknown` still holds.
828+
if (registration === "unknown" && xml === "absent" && exe === "absent") {
829+
return { kind: "absent" };
830+
}
732831
if (registration === "unknown") {
733832
return unknown("the native WinSW service registration could not be verified");
734833
}

0 commit comments

Comments
 (0)