Skip to content

Commit 8dba2b4

Browse files
authored
fix(daemon): compose stale-claim recovery from the dead owner's state dir (#2171)
* fix(daemon): compose stale-claim recovery from the dead owner's state dir (#2168) The daemon's acquire-time reconciliation and startup sweep ran recovery through the daemon's single composed gateway, so settling a foreign state dir's dead claim cleared owned-process records by bare session id in the RECONCILING daemon's own store — a live same-named session (default is common across worktrees) lost its owned-processes.json to a foreign claim's cleanup, while the dead owner's marker survived. Live-reproduced on main. Every production reconciliation path — open-time admission, transient admission, the startup sweep, and the daemonless CLI release — now goes through createOwnerScopedDeviceClaimReconciler: one recovery composition per claim, bound to that claim's recorded state dir (owned-process store and session artifacts included), disposed after the transaction. Gateway state is instance-scoped, so disposal shuts down only the transaction's own handles. * test: prove the startup sweep composes recovery from the claim's state dir Review follow-up on #2171: the unit tests exercised the owner-scoped reconciler directly, so reverting the sweep callsite to the caller-scoped reconciler stayed green. This regression boots the real daemon runtime with a foreign dead claim whose recorded simctl recorder still runs as an orphan, next to a live same-named session record in the daemon's own state dir (purpose outside the startup reaper's selection). It proves the sweep terminates the orphan, settles the foreign recording open -> completed, clears the owner's process record in the OWNER's state dir, and leaves the daemon's own record byte-identical. Observed red against a reverted sweep callsite. * test: prove open-time reconciliation composes recovery from the owner state dir Review follow-ups on #2171: - The acquire/open wiring had only live evidence. The new regression drives a real open through the production request router against a device whose foreign dead claim carries an orphaned recorder and durable recording evidence, next to a live same-named session record under the daemon's own state dir. It proves open-time reconciliation terminates the orphan, settles the foreign recording, clears the owner's process record in the OWNER's state dir, leaves the daemon's own record byte-identical, and hands the claim to the opening session. Observed red against a reverted router callsite. - The startup-sweep regression now shuts its daemon runtime down in finally, so an assertion failure cannot leak the runtime. * test: restore an absent claims-dir override by deletion, not assignment Review follow-up on #2171: both new regressions (and the pre-existing prune-log test they mirrored) restored AGENT_DEVICE_CLAIMS_DIR by plain assignment, which turns an absent previous value into the literal string "undefined" and leaks global test state. Absent values are now restored by deleting the variable.
1 parent d330a67 commit 8dba2b4

8 files changed

Lines changed: 465 additions & 74 deletions

‎src/__tests__/daemon-entrypoint.test.ts‎

Lines changed: 154 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
import assert from 'node:assert/strict';
2+
import { spawn } from 'node:child_process';
23
import crypto from 'node:crypto';
34
import fs from 'node:fs';
45
import http from 'node:http';
@@ -11,7 +12,15 @@ import {
1112
trackDownloadableArtifact,
1213
} from '../daemon/artifact-tracking.ts';
1314
import { runCmdBackground } from '@agent-device/host-kit/command';
14-
import { isProcessAlive, waitForProcessExit } from '@agent-device/host-kit/process';
15+
import {
16+
createOwnedProcessRecordStore,
17+
isProcessAlive,
18+
readProcessCommand,
19+
readProcessStartTime,
20+
waitForProcessExit,
21+
} from '@agent-device/host-kit/process';
22+
import { createDurableResourceEnvelope } from '@agent-device/capture-kit';
23+
import { screenRecordingDurableResource } from '../daemon/screen-recording-session-resource.ts';
1524

1625
import { closeLoopbackServer, listenOnLoopback, waitForHttpOk } from './test-utils/loopback.ts';
1726
import { mkdtempForTestSync } from './test-utils/tmp-dir.ts';
@@ -341,8 +350,151 @@ test('daemon runtime records startup device-claim reconciliation in daemon.log',
341350
await runtime?.shutdown();
342351
assert.equal(exitCode, 0);
343352
} finally {
344-
process.env.AGENT_DEVICE_CLAIMS_DIR = previousClaimsDir;
353+
if (previousClaimsDir === undefined) delete process.env.AGENT_DEVICE_CLAIMS_DIR;
354+
else process.env.AGENT_DEVICE_CLAIMS_DIR = previousClaimsDir;
345355
fs.rmSync(stateDir, { recursive: true, force: true });
346356
fs.rmSync(claimsDir, { recursive: true, force: true });
347357
}
348358
});
359+
360+
test('startup sweep settles a foreign dead owner without touching a live same-named session', async () => {
361+
// #2168: the sweep must compose recovery from the STALE CLAIM's state dir.
362+
// With the daemon-gateway (caller-scoped) reconciler this test goes red: the
363+
// foreign recording cleanup clears session "shared" in THIS daemon's store,
364+
// deleting the live record planted below, and leaves the foreign one behind.
365+
const daemonStateDir = mkdtempForTestSync('agent-device-daemon-sweep-scope-');
366+
const foreignStateDir = mkdtempForTestSync('agent-device-daemon-sweep-foreign-');
367+
const claimsDir = mkdtempForTestSync('agent-device-daemon-sweep-claims-');
368+
const deviceId = 'sweep-scope-udid';
369+
const deviceKey = `local:apple:ios:${deviceId}`;
370+
const previousClaimsDir = process.env.AGENT_DEVICE_CLAIMS_DIR;
371+
process.env.AGENT_DEVICE_CLAIMS_DIR = claimsDir;
372+
373+
const sessionStoreFor = (stateDir: string) => {
374+
const paths = resolveDaemonPaths(stateDir);
375+
return createOwnedProcessRecordStore({
376+
stateDir: paths.baseDir,
377+
sessionsDir: paths.sessionsDir,
378+
resolveSessionDir: (sessionId) => path.join(paths.sessionsDir, sessionId),
379+
});
380+
};
381+
382+
// The dead owner's recorded recorder, still running as an orphan: recovery
383+
// must find it owned-alive so it takes the descriptor-cleanup path, which
384+
// terminates it and clears the owner's process record — through whichever
385+
// owned-process store the sweep composed. A plain sleeper with its real
386+
// observed command and start time satisfies the exact-identity match.
387+
const orphanRecorder = spawn('sleep', ['120'], { stdio: 'ignore' });
388+
await new Promise((resolve) => orphanRecorder.once('spawn', resolve));
389+
const orphanPid = orphanRecorder.pid;
390+
assert.ok(orphanPid);
391+
const orphanMarker = {
392+
pid: orphanPid,
393+
startTime: readProcessStartTime(orphanPid) ?? '',
394+
command: readProcessCommand(orphanPid) ?? '',
395+
};
396+
assert.ok(orphanMarker.startTime.length > 0 && orphanMarker.command.length > 0);
397+
let runtime: Awaited<ReturnType<typeof startDaemonRuntime>> = null;
398+
399+
try {
400+
const foreignSessionDir = path.join(resolveDaemonPaths(foreignStateDir).sessionsDir, 'shared');
401+
const resourcePath = screenRecordingDurableResource.store.resolvePath(foreignSessionDir);
402+
screenRecordingDurableResource.store.write(
403+
resourcePath,
404+
createDurableResourceEnvelope({
405+
resourceKind: 'screen-recording',
406+
sessionId: 'shared',
407+
device: { id: deviceId, family: 'apple', appleOs: 'ios', kind: 'simulator' },
408+
owner: { kind: 'local-family', family: 'apple' },
409+
fence: { token: 'sweep-fence', generation: 1 },
410+
lifecycle: 'open',
411+
descriptor: {
412+
version: 1,
413+
body: {
414+
backend: 'simctl',
415+
outputPath: path.join(foreignSessionDir, 'recording.mp4'),
416+
processes: [orphanMarker],
417+
},
418+
},
419+
}),
420+
);
421+
sessionStoreFor(foreignStateDir).replace({ kind: 'session', sessionId: 'shared' }, [
422+
{ ...orphanMarker, purpose: 'simctl-screen-recording' },
423+
]);
424+
// The live same-named session in THIS daemon's state dir. The purpose is
425+
// outside the startup reaper's selection, so only a wrong-store recovery
426+
// clear can remove it.
427+
sessionStoreFor(daemonStateDir).replace({ kind: 'session', sessionId: 'shared' }, [
428+
{ pid: process.pid, startTime: 'live-marker', command: 'live-probe', purpose: 'test-probe' },
429+
]);
430+
const liveRecordPath = path.join(
431+
resolveDaemonPaths(daemonStateDir).sessionsDir,
432+
'shared',
433+
'owned-processes.json',
434+
);
435+
const liveRecordBefore = fs.readFileSync(liveRecordPath, 'utf8');
436+
fs.writeFileSync(
437+
path.join(claimsDir, `${crypto.createHash('sha256').update(deviceKey).digest('hex')}.json`),
438+
JSON.stringify({
439+
schemaVersion: 1,
440+
deviceKey,
441+
device: { platform: 'ios', id: deviceId, name: 'Sweep Scope iPhone', kind: 'simulator' },
442+
session: 'shared',
443+
workspace: '/worktrees/dead',
444+
stateDir: foreignStateDir,
445+
ownerPid: 999_999_999,
446+
ownerStartTime: 'old-start-time',
447+
ownerToken: 'sweep-scope-token',
448+
createdAtMs: 1,
449+
updatedAtMs: 1,
450+
}),
451+
);
452+
453+
runtime = await startDaemonRuntime({
454+
env: {
455+
...process.env,
456+
AGENT_DEVICE_STATE_DIR: daemonStateDir,
457+
AGENT_DEVICE_DAEMON_SERVER_MODE: 'http',
458+
AGENT_DEVICE_CLAIMS_DIR: claimsDir,
459+
},
460+
exit: () => {},
461+
registerProcessHandlers: false,
462+
stderr: { write: () => {} },
463+
stdout: { write: () => {} },
464+
});
465+
assert.notEqual(runtime, null);
466+
467+
assert.deepEqual(fs.readdirSync(claimsDir), [], 'the foreign dead claim should be reconciled');
468+
const settled = screenRecordingDurableResource.store.read(resourcePath);
469+
assert.equal(settled.status, 'decoded');
470+
if (settled.status === 'decoded') {
471+
assert.equal(settled.envelope.lifecycle, 'completed');
472+
}
473+
assert.equal(
474+
fs.existsSync(path.join(foreignSessionDir, 'owned-processes.json')),
475+
false,
476+
"the dead owner's process record should be cleared in the OWNER's state dir",
477+
);
478+
assert.equal(
479+
fs.readFileSync(liveRecordPath, 'utf8'),
480+
liveRecordBefore,
481+
"the live same-named session's process record must stay byte-identical",
482+
);
483+
484+
assert.equal(
485+
isProcessAlive(orphanPid),
486+
false,
487+
"the dead owner's orphaned recorder should be terminated by recovery",
488+
);
489+
} finally {
490+
await runtime?.shutdown().catch(() => {});
491+
try {
492+
orphanRecorder.kill('SIGKILL');
493+
} catch {}
494+
if (previousClaimsDir === undefined) delete process.env.AGENT_DEVICE_CLAIMS_DIR;
495+
else process.env.AGENT_DEVICE_CLAIMS_DIR = previousClaimsDir;
496+
fs.rmSync(daemonStateDir, { recursive: true, force: true });
497+
fs.rmSync(foreignStateDir, { recursive: true, force: true });
498+
fs.rmSync(claimsDir, { recursive: true, force: true });
499+
}
500+
});

‎src/cli/commands/device-release.ts‎

Lines changed: 13 additions & 57 deletions
Original file line numberDiff line numberDiff line change
@@ -1,74 +1,30 @@
1-
import { SessionStore } from '../../daemon/session-store.ts';
2-
import { resolveDaemonPaths } from '../../daemon/config.ts';
3-
import { createDeviceClaimReconciler } from '../../daemon/device-claim-reconciliation.ts';
1+
import {
2+
createOwnerScopedDeviceClaimReconciler,
3+
type OwnerScopedClaimRecoveryComposer,
4+
} from '../../daemon/device-claim-owner-recovery.ts';
45
import { createDaemonRecoveryPlatformScope } from '../../daemon/platform-request-scope.ts';
56
import {
67
releaseProvenStaleDeviceClaims,
7-
type DeviceClaimReconciler,
88
type DeviceClaimStaleReleaseOutcome,
99
} from '../../daemon/device-claims.ts';
1010
import type { DeviceClaimSelectors } from '../../daemon/device-claim-inspection.ts';
11-
import { createPlatformRuntimeGateway } from '../../platform-runtime.ts';
12-
import { createOwnedProcessRecordStore } from '@agent-device/host-kit/process';
13-
14-
export type StaleClaimRecovery = {
15-
reconcile: DeviceClaimReconciler;
16-
dispose(): Promise<void>;
17-
};
1811

1912
/**
2013
* Daemonless `device release --stale`: settles each provably dead owner
21-
* through the same reconciliation transaction the daemon uses at `open` and
22-
* startup — resources first, claim last — without requiring the (typically
23-
* dead) owner daemon, or any daemon, to be running.
14+
* through the same owner-scoped reconciliation the daemon uses at `open` and
15+
* startup — recovery composed from the stale claim's own state dir, resources
16+
* settled first, claim cleared last — without requiring the (typically dead)
17+
* owner daemon, or any daemon, to be running.
2418
*/
2519
export async function runStaleDeviceClaimRelease(
2620
selectors: DeviceClaimSelectors,
27-
composeRecovery: (stateDir: string) => StaleClaimRecovery = composeStaleClaimRecovery,
21+
composeRecovery?: OwnerScopedClaimRecoveryComposer,
2822
): Promise<DeviceClaimStaleReleaseOutcome[]> {
2923
return await releaseProvenStaleDeviceClaims({
3024
selectors,
31-
reconcile: async (claim) => {
32-
const recovery = composeRecovery(claim.stateDir);
33-
try {
34-
return await recovery.reconcile(claim);
35-
} finally {
36-
await recovery.dispose();
37-
}
38-
},
39-
});
40-
}
41-
42-
/**
43-
* Recovery is composed per claim, bound to the stale owner's recorded state
44-
* dir: the owned-process record store and session artifact paths must be the
45-
* dead owner's, never the caller's. One caller-scoped store would let a
46-
* foreign claim's cleanup clear a same-named live session's records in the
47-
* caller's state dir. Local-only gateway: claims exist only for local
48-
* devices, so provider runtimes stay out of the composition entirely.
49-
*/
50-
function composeStaleClaimRecovery(stateDir: string): StaleClaimRecovery {
51-
const daemonPaths = resolveDaemonPaths(stateDir);
52-
const sessionStore = new SessionStore(daemonPaths.sessionsDir);
53-
const gateway = createPlatformRuntimeGateway({
54-
sessionsDir: daemonPaths.sessionsDir,
55-
ownedProcesses: createOwnedProcessRecordStore({
56-
stateDir: daemonPaths.baseDir,
57-
sessionsDir: daemonPaths.sessionsDir,
58-
resolveSessionDir: (sessionId) => sessionStore.resolveSessionDir(sessionId),
59-
}),
60-
resolveSessionArtifacts: (sessionId) => ({
61-
outputPath: sessionStore.resolveAppLogPath(sessionId),
62-
pidPath: sessionStore.resolveAppLogPidPath(sessionId),
63-
}),
25+
reconcile: createOwnerScopedDeviceClaimReconciler(
26+
createDaemonRecoveryPlatformScope(),
27+
composeRecovery,
28+
),
6429
});
65-
return {
66-
reconcile: createDeviceClaimReconciler({
67-
gateway,
68-
scope: createDaemonRecoveryPlatformScope(),
69-
}),
70-
dispose: async () => {
71-
await gateway.shutdown();
72-
},
73-
};
7430
}
Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
import assert from 'node:assert/strict';
2+
import { test } from 'vitest';
3+
import type { DeviceClaim } from '../device-claims.ts';
4+
import { createOwnerScopedDeviceClaimReconciler } from '../device-claim-owner-recovery.ts';
5+
6+
const scope = {
7+
signal: new AbortController().signal,
8+
diagnostics: { emit: () => {} },
9+
progress: { report: () => {} },
10+
};
11+
12+
function makeClaim(stateDir: string, session: string): DeviceClaim {
13+
return {
14+
schemaVersion: 2,
15+
deviceKey: `local:android:none:${session}`,
16+
device: { family: 'android', id: session, name: 'Pixel', kind: 'emulator' },
17+
session,
18+
workspace: `/worktrees/${session}`,
19+
stateDir,
20+
ownerPid: 999_999_999,
21+
ownerStartTime: 'dead-start',
22+
ownerToken: `${session}-token`,
23+
createdAtMs: 1,
24+
updatedAtMs: 1,
25+
};
26+
}
27+
28+
test('composes one recovery per claim from that claim state dir and disposes it', async () => {
29+
const events: string[] = [];
30+
const reconcile = createOwnerScopedDeviceClaimReconciler(scope, (stateDir) => ({
31+
reconcile: async (claim) => {
32+
assert.equal(claim.stateDir, stateDir);
33+
events.push(`reconcile:${stateDir}`);
34+
return { status: 'reconciled' };
35+
},
36+
dispose: async () => {
37+
events.push(`dispose:${stateDir}`);
38+
},
39+
}));
40+
41+
await reconcile(makeClaim('/state/owner-a', 'shared'));
42+
await reconcile(makeClaim('/state/owner-b', 'shared'));
43+
44+
assert.deepEqual(events, [
45+
'reconcile:/state/owner-a',
46+
'dispose:/state/owner-a',
47+
'reconcile:/state/owner-b',
48+
'dispose:/state/owner-b',
49+
]);
50+
});
51+
52+
test('disposes the composed recovery when reconciliation retains or throws', async () => {
53+
const disposed: string[] = [];
54+
const retainReconcile = createOwnerScopedDeviceClaimReconciler(scope, (stateDir) => ({
55+
reconcile: async () => ({ status: 'retained', reason: 'cleanup-pending' }),
56+
dispose: async () => {
57+
disposed.push(stateDir);
58+
},
59+
}));
60+
const result = await retainReconcile(makeClaim('/state/retained', 'shared'));
61+
assert.deepEqual(result, { status: 'retained', reason: 'cleanup-pending' });
62+
63+
const throwingReconcile = createOwnerScopedDeviceClaimReconciler(scope, (stateDir) => ({
64+
reconcile: async () => {
65+
throw new Error('recovery exploded');
66+
},
67+
dispose: async () => {
68+
disposed.push(stateDir);
69+
},
70+
}));
71+
await assert.rejects(
72+
async () => await throwingReconcile(makeClaim('/state/thrown', 'shared')),
73+
/recovery exploded/,
74+
);
75+
assert.deepEqual(disposed, ['/state/retained', '/state/thrown']);
76+
});

0 commit comments

Comments
 (0)