Skip to content

Commit 2d2aa6c

Browse files
authored
feat: add stale device claim release and dead-end recovery guidance (#2162)
* feat: add stale device claim release and dead-end recovery guidance Close the #1320 recovery loop for claims no daemon can settle on its own: - agent-device device release --stale settles a provably dead owner's durable resources through the same exact-owner reconciliation open and daemon startup use, then clears the claim last — daemonlessly, composing a local-only platform gateway in the CLI process. Live, uncertain, PID-reused, and corrupt claims always fail closed and are reported with the reason. - DEVICE_IN_USE conflicts whose recorded owner provably cannot release (dead or superseded) now carry the exact release command as their recovery instead of a status inspection that dead-ended. - device status --stale now offers the matching release command when provably dead owners are listed. - daemon stop now warns in text output when a claim was orphaned (previously visible only via --json) and names the status/release commands. Part of #1320. * test: cover release refusal branches, text rendering, and orphan warnings Changed-line coverage on the stale-release slice was 69.33% against the 70% gate: the refusal-reason branches, the text-mode outcome rendering with the live-owner hint, and the daemon stop orphaned-claim warning had no tests. Cover them directly; the misnamed-claim-file refusal is also pinned. * test: prove resources gate stale release and the scan-to-lock race stays closed Review follow-ups on #2162: - Two end-to-end CLI regressions run device release --stale through the real local gateway against a dead owner whose state dir holds an attributable durable screen-recording/app-log record: an owner-mismatched record and unreadable resource evidence both retain the claim (app-log-owner-mismatch, app-log-descriptor-invalid), proving cleanup must reach a terminal state before the claim can be deleted. - A deterministic race regression holds the per-device claim lock while a release is in flight, replaces the claim with a successor before releasing the lock, and proves the transaction reports changed without reconciling or touching the successor's claim. * fix: bind stale-claim recovery to the dead owner's state dir Review P1 on #2162: the CLI composed one gateway from the caller's state dir, so recovery for a foreign stale claim could clear the caller's live owned-process record when both used the same session name — Apple recording cleanup clears by session id through the gateway-composed store. Recovery is now composed per claim, with the owned-process record store and session artifact paths bound to the stale claim's recorded state dir, and disposed after each transaction. The regression writes two dead claims with one shared session name in different state dirs and proves each recovery is composed from that claim's own state dir, never the caller's.
1 parent 5fddf1c commit 2d2aa6c

12 files changed

Lines changed: 881 additions & 26 deletions

File tree

‎recording-1788175667275.mp4‎

54.6 KB
Binary file not shown.
Lines changed: 306 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,306 @@
1+
import assert from 'node:assert/strict';
2+
import crypto from 'node:crypto';
3+
import fs from 'node:fs';
4+
import path from 'node:path';
5+
import { test, vi } from 'vitest';
6+
import { readCurrentOwnerIdentity } from '@agent-device/host-kit/process';
7+
import { createDurableResourceEnvelope } from '@agent-device/capture-kit';
8+
import { localRuntimeOwner } from '@agent-device/contracts/platform-runtime';
9+
import { appLogResourceStore } from '../daemon/app-log-resource-store.ts';
10+
import { runStaleDeviceClaimRelease } from '../cli/commands/device-release.ts';
11+
import { runCliCapture } from './cli-capture.ts';
12+
import { mkdtempForTestSync } from './test-utils/tmp-dir.ts';
13+
14+
vi.mock('@agent-device/host-kit/process', async (importOriginal) =>
15+
(await import('./test-utils/host-process-mock.ts')).pinOwnProcessStartTime(importOriginal),
16+
);
17+
18+
function hashedClaimPath(claimsDir: string, deviceKey: string): string {
19+
const hash = crypto.createHash('sha256').update(deviceKey).digest('hex');
20+
return path.join(claimsDir, `${hash}.json`);
21+
}
22+
23+
function writeClaim(
24+
claimsDir: string,
25+
params: {
26+
deviceKey: string;
27+
id: string;
28+
name: string;
29+
session: string;
30+
ownerPid: number;
31+
ownerStartTime: string | null;
32+
stateDir: string;
33+
},
34+
): void {
35+
fs.writeFileSync(
36+
hashedClaimPath(claimsDir, params.deviceKey),
37+
JSON.stringify({
38+
schemaVersion: 1,
39+
deviceKey: params.deviceKey,
40+
device: { platform: 'android', id: params.id, name: params.name, kind: 'emulator' },
41+
session: params.session,
42+
workspace: '/worktrees/release-test',
43+
stateDir: params.stateDir,
44+
ownerPid: params.ownerPid,
45+
ownerStartTime: params.ownerStartTime,
46+
ownerToken: `${params.session}-token`,
47+
createdAtMs: 1,
48+
updatedAtMs: 1,
49+
}),
50+
);
51+
}
52+
53+
test('device release requires --stale and explains how live owners are released', async () => {
54+
const result = await runCliCapture(['device', 'release', '--json']);
55+
assert.equal(result.calls.length, 0);
56+
const payload = JSON.parse(result.stdout || result.stderr);
57+
assert.equal(payload.success, false);
58+
assert.equal(payload.error.code, 'INVALID_ARGS');
59+
assert.match(payload.error.message, /pass --stale to confirm/);
60+
});
61+
62+
test('device release --stale settles a provably dead owner daemonlessly and refuses a live one', async () => {
63+
const claimsDir = mkdtempForTestSync('agent-device-cli-release-');
64+
const stateDir = mkdtempForTestSync('agent-device-cli-release-state-');
65+
const owner = readCurrentOwnerIdentity();
66+
try {
67+
writeClaim(claimsDir, {
68+
deviceKey: 'local:android:none:emulator-5554',
69+
id: 'emulator-5554',
70+
name: 'Dead Pixel',
71+
session: 'dead-session',
72+
ownerPid: 999_999_999,
73+
ownerStartTime: 'old-start-time',
74+
stateDir,
75+
});
76+
writeClaim(claimsDir, {
77+
deviceKey: 'local:android:none:emulator-5556',
78+
id: 'emulator-5556',
79+
name: 'Live Pixel',
80+
session: 'live-session',
81+
ownerPid: owner.pid,
82+
ownerStartTime: owner.startTime,
83+
stateDir: process.cwd(),
84+
});
85+
86+
const result = await runCliCapture(['device', 'release', '--stale', '--json'], {
87+
env: { AGENT_DEVICE_CLAIMS_DIR: claimsDir },
88+
});
89+
assert.equal(result.calls.length, 0);
90+
const payload = JSON.parse(result.stdout);
91+
assert.equal(payload.success, true);
92+
assert.equal(payload.data.released.length, 1);
93+
assert.equal(payload.data.released[0].session, 'dead-session');
94+
assert.equal(payload.data.refused.length, 1);
95+
assert.equal(payload.data.refused[0].session, 'live-session');
96+
assert.equal(payload.data.refused[0].reason, 'live-owner');
97+
assert.equal(
98+
fs.existsSync(hashedClaimPath(claimsDir, 'local:android:none:emulator-5554')),
99+
false,
100+
);
101+
assert.equal(
102+
fs.existsSync(hashedClaimPath(claimsDir, 'local:android:none:emulator-5556')),
103+
true,
104+
);
105+
} finally {
106+
fs.rmSync(claimsDir, { recursive: true, force: true });
107+
fs.rmSync(stateDir, { recursive: true, force: true });
108+
}
109+
});
110+
111+
test('device status --stale offers the exact release command for provably dead owners', async () => {
112+
const claimsDir = mkdtempForTestSync('agent-device-cli-release-');
113+
try {
114+
writeClaim(claimsDir, {
115+
deviceKey: 'local:android:none:emulator-5554',
116+
id: 'emulator-5554',
117+
name: 'Dead Pixel',
118+
session: 'dead-session',
119+
ownerPid: 999_999_999,
120+
ownerStartTime: 'old-start-time',
121+
stateDir: process.cwd(),
122+
});
123+
124+
const result = await runCliCapture(['device', 'status', '--stale'], {
125+
env: { AGENT_DEVICE_CLAIMS_DIR: claimsDir },
126+
});
127+
assert.match(
128+
result.stdout,
129+
/Release provably dead owners with: agent-device device release --stale/,
130+
);
131+
} finally {
132+
fs.rmSync(claimsDir, { recursive: true, force: true });
133+
}
134+
});
135+
136+
test('device release --stale renders per-claim outcomes with a live-owner hint in text mode', async () => {
137+
const claimsDir = mkdtempForTestSync('agent-device-cli-release-');
138+
const stateDir = mkdtempForTestSync('agent-device-cli-release-state-');
139+
const owner = readCurrentOwnerIdentity();
140+
try {
141+
writeClaim(claimsDir, {
142+
deviceKey: 'local:android:none:emulator-5554',
143+
id: 'emulator-5554',
144+
name: 'Dead Pixel',
145+
session: 'dead-session',
146+
ownerPid: 999_999_999,
147+
ownerStartTime: 'old-start-time',
148+
stateDir,
149+
});
150+
writeClaim(claimsDir, {
151+
deviceKey: 'local:android:none:emulator-5556',
152+
id: 'emulator-5556',
153+
name: 'Live Pixel',
154+
session: 'live-session',
155+
ownerPid: owner.pid,
156+
ownerStartTime: owner.startTime,
157+
stateDir: process.cwd(),
158+
});
159+
160+
const result = await runCliCapture(['device', 'release', '--stale'], {
161+
env: { AGENT_DEVICE_CLAIMS_DIR: claimsDir },
162+
});
163+
assert.match(result.stdout, /released android Dead Pixel session=dead-session/);
164+
assert.match(result.stdout, /refused android Live Pixel session=live-session[^\n]*live-owner/);
165+
assert.match(result.stdout, /agent-device daemon stop --state-dir/);
166+
} finally {
167+
fs.rmSync(claimsDir, { recursive: true, force: true });
168+
fs.rmSync(stateDir, { recursive: true, force: true });
169+
}
170+
});
171+
172+
test('device release --stale keeps the claim while durable resources cannot be settled', async () => {
173+
const claimsDir = mkdtempForTestSync('agent-device-cli-release-');
174+
const ownerStateDir = mkdtempForTestSync('agent-device-cli-release-owner-');
175+
try {
176+
writeClaim(claimsDir, {
177+
deviceKey: 'local:android:none:emulator-5554',
178+
id: 'emulator-5554',
179+
name: 'Dead Pixel',
180+
session: 'dead-session',
181+
ownerPid: 999_999_999,
182+
ownerStartTime: 'old-start-time',
183+
stateDir: ownerStateDir,
184+
});
185+
// The dead owner left an attributable durable resource whose recorded
186+
// device is NOT this claim's device: exact-owner recovery refuses it, and
187+
// the claim must survive until the resource reaches a terminal state.
188+
const resourcePath = appLogResourceStore.resolvePath(
189+
path.join(ownerStateDir, 'sessions', 'dead-session'),
190+
);
191+
appLogResourceStore.write(
192+
resourcePath,
193+
createDurableResourceEnvelope({
194+
resourceKind: 'app-log',
195+
sessionId: 'dead-session',
196+
device: { id: 'some-other-device', family: 'android', kind: 'emulator' },
197+
owner: localRuntimeOwner('android'),
198+
fence: { token: 'fence', generation: 1 },
199+
lifecycle: 'open',
200+
descriptor: { version: 1, body: { pid: 123 } },
201+
}),
202+
);
203+
204+
const result = await runCliCapture(['device', 'release', '--stale', '--json'], {
205+
env: { AGENT_DEVICE_CLAIMS_DIR: claimsDir },
206+
});
207+
const payload = JSON.parse(result.stdout);
208+
assert.equal(payload.success, true);
209+
assert.equal(payload.data.released.length, 0);
210+
assert.equal(payload.data.retained.length, 1);
211+
assert.equal(payload.data.retained[0].reason, 'app-log-owner-mismatch');
212+
assert.equal(
213+
fs.existsSync(hashedClaimPath(claimsDir, 'local:android:none:emulator-5554')),
214+
true,
215+
);
216+
assert.equal(fs.existsSync(resourcePath), true);
217+
} finally {
218+
fs.rmSync(claimsDir, { recursive: true, force: true });
219+
fs.rmSync(ownerStateDir, { recursive: true, force: true });
220+
}
221+
});
222+
223+
test('device release --stale keeps the claim when resource evidence is unreadable', async () => {
224+
const claimsDir = mkdtempForTestSync('agent-device-cli-release-');
225+
const ownerStateDir = mkdtempForTestSync('agent-device-cli-release-owner-');
226+
try {
227+
writeClaim(claimsDir, {
228+
deviceKey: 'local:android:none:emulator-5554',
229+
id: 'emulator-5554',
230+
name: 'Dead Pixel',
231+
session: 'dead-session',
232+
ownerPid: 999_999_999,
233+
ownerStartTime: 'old-start-time',
234+
stateDir: ownerStateDir,
235+
});
236+
const resourcePath = appLogResourceStore.resolvePath(
237+
path.join(ownerStateDir, 'sessions', 'dead-session'),
238+
);
239+
fs.mkdirSync(path.dirname(resourcePath), { recursive: true });
240+
fs.writeFileSync(resourcePath, '{');
241+
242+
const result = await runCliCapture(['device', 'release', '--stale', '--json'], {
243+
env: { AGENT_DEVICE_CLAIMS_DIR: claimsDir },
244+
});
245+
const payload = JSON.parse(result.stdout);
246+
assert.equal(payload.data.retained.length, 1);
247+
assert.equal(payload.data.retained[0].reason, 'app-log-descriptor-invalid');
248+
assert.equal(
249+
fs.existsSync(hashedClaimPath(claimsDir, 'local:android:none:emulator-5554')),
250+
true,
251+
);
252+
} finally {
253+
fs.rmSync(claimsDir, { recursive: true, force: true });
254+
fs.rmSync(ownerStateDir, { recursive: true, force: true });
255+
}
256+
});
257+
258+
test('recovery is composed per claim from the stale owner state dir, never the caller', async () => {
259+
const claimsDir = mkdtempForTestSync('agent-device-cli-release-');
260+
const ownerDirA = mkdtempForTestSync('agent-device-cli-release-owner-a-');
261+
const ownerDirB = mkdtempForTestSync('agent-device-cli-release-owner-b-');
262+
try {
263+
// Two dead owners with the SAME session name in different state dirs: the
264+
// adversarial shape from review — one caller-scoped store would clear the
265+
// wrong worktree's records for that session name.
266+
writeClaim(claimsDir, {
267+
deviceKey: 'local:android:none:emulator-5554',
268+
id: 'emulator-5554',
269+
name: 'Dead Pixel A',
270+
session: 'shared-session',
271+
ownerPid: 999_999_999,
272+
ownerStartTime: 'old-start-time',
273+
stateDir: ownerDirA,
274+
});
275+
writeClaim(claimsDir, {
276+
deviceKey: 'local:android:none:emulator-5556',
277+
id: 'emulator-5556',
278+
name: 'Dead Pixel B',
279+
session: 'shared-session',
280+
ownerPid: 999_999_999,
281+
ownerStartTime: 'old-start-time',
282+
stateDir: ownerDirB,
283+
});
284+
process.env.AGENT_DEVICE_CLAIMS_DIR = claimsDir;
285+
286+
const composedFor: string[] = [];
287+
const outcomes = await runStaleDeviceClaimRelease({}, (stateDir) => {
288+
composedFor.push(stateDir);
289+
return {
290+
reconcile: async (claim) => {
291+
assert.equal(claim.stateDir, stateDir);
292+
return { status: 'reconciled' };
293+
},
294+
dispose: async () => {},
295+
};
296+
});
297+
298+
assert.deepEqual([...composedFor].sort(), [ownerDirA, ownerDirB].sort());
299+
assert.equal(outcomes.filter((outcome) => outcome.status === 'released').length, 2);
300+
} finally {
301+
delete process.env.AGENT_DEVICE_CLAIMS_DIR;
302+
fs.rmSync(claimsDir, { recursive: true, force: true });
303+
fs.rmSync(ownerDirA, { recursive: true, force: true });
304+
fs.rmSync(ownerDirB, { recursive: true, force: true });
305+
}
306+
});

‎src/cli-schema/command-overrides.ts‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -46,14 +46,14 @@ const SCHEMA_ONLY_CLI_COMMAND_SCHEMAS = {
4646
},
4747
device: {
4848
text: {
49-
summary: 'Inspect enforced local device ownership without daemon side effects',
49+
summary: 'Inspect and recover enforced local device ownership without a daemon',
5050
description:
51-
'Inspect enforced host-local device ownership claims without starting or contacting a daemon. --stale only inspects proven-stale claims; automatic reclamation occurs during open or daemon startup after exact-owner resource reconciliation.',
51+
'Inspect enforced host-local device ownership claims without starting or contacting a daemon; status --stale only inspects proven-stale claims. release --stale settles a provably dead owner through exact-owner resource reconciliation and clears its claim last — live and uncertain owners always fail closed. Automatic reclamation still occurs during open and daemon startup.',
5252
},
5353
usageOverride:
54-
'device status [--platform <platform>] [--udid <udid>] [--serial <serial>] [--stale]',
54+
'device status|release [--platform <platform>] [--udid <udid>] [--serial <serial>] [--stale]',
5555
listUsageOverride: 'device status',
56-
positionalArgs: ['status'],
56+
positionalArgs: ['status|release'],
5757
allowedFlags: ['stale'],
5858
supportedFlags: ['platform', 'device', 'udid', 'serial'],
5959
},

‎src/cli/commands/__tests__/daemon.test.ts‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -139,3 +139,40 @@ test('reports graceful provider cleanup as unknown when the shutdown report is u
139139
fs.rmSync(stateDir, { recursive: true, force: true });
140140
}
141141
});
142+
143+
test('warns in text output when a graceful stop leaves an orphaned claim', async () => {
144+
const stateDir = mkdtempForTestSync('agent-device-daemon-command-');
145+
mocks.readDaemonStopIdentity.mockReturnValue({ pid: 123, processStartTime: 'start-time' });
146+
mocks.stopDaemon.mockResolvedValue(GRACEFUL_RESULT);
147+
const claim = {
148+
deviceKey: 'local:android:none:emulator-5554',
149+
session: 'default',
150+
platform: 'android',
151+
deviceId: 'emulator-5554',
152+
};
153+
mocks.readDaemonShutdownReport.mockReturnValue({
154+
providerReleases: { released: [], pending: [] },
155+
claims: { released: [], orphaned: [claim], superseded: [] },
156+
});
157+
158+
try {
159+
await daemonCommand({
160+
positionals: ['stop'],
161+
flags: { clean: false, help: false, json: false, stateDir, version: false },
162+
client: {} as never,
163+
});
164+
165+
const [, data, renderHuman] = mocks.writeCommandOutput.mock.calls.at(-1) ?? [];
166+
expect(data).toEqual(
167+
expect.objectContaining({
168+
claimsOrphaned: [claim],
169+
warnings: [expect.stringContaining('was not released cleanly')],
170+
}),
171+
);
172+
const rendered = (renderHuman as () => string)();
173+
expect(rendered).toContain('Ownership of emulator-5554 was not released cleanly');
174+
expect(rendered).toContain('agent-device device release --stale');
175+
} finally {
176+
fs.rmSync(stateDir, { recursive: true, force: true });
177+
}
178+
});

‎src/cli/commands/daemon.ts‎

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,11 @@ function mergeShutdownReport(
4646
claimsReleased: report.claims.released,
4747
claimsOrphaned: report.claims.orphaned,
4848
claimsSuperseded: report.claims.superseded,
49-
warnings: [...stopped.warnings, ...supersededClaimWarnings(report.claims.superseded)],
49+
warnings: [
50+
...stopped.warnings,
51+
...supersededClaimWarnings(report.claims.superseded),
52+
...orphanedClaimWarnings(report.claims.orphaned),
53+
],
5054
}
5155
: stopped;
5256
}
@@ -71,6 +75,17 @@ function supersededClaimWarnings(superseded: DaemonStopResult['claimsSuperseded'
7175
];
7276
}
7377

78+
/** An orphaned claim keeps holding its device after the daemon is gone, and
79+
* only `device release --stale` or the next open settles it — say so instead
80+
* of leaving the block discoverable through --json alone. */
81+
function orphanedClaimWarnings(orphaned: DaemonStopResult['claimsOrphaned']): string[] {
82+
if (orphaned.length === 0) return [];
83+
const devices = orphaned.map((claim) => claim.deviceId).join(', ');
84+
return [
85+
`Ownership of ${devices} was not released cleanly; the claim now blocks other owners until it is settled. Inspect with: agent-device device status --stale, then release with: agent-device device release --stale.`,
86+
];
87+
}
88+
7489
function renderDaemonStop(
7590
result: Pick<DaemonStopResult, 'stopped' | 'mode' | 'warnings'> & {
7691
clean: boolean;

0 commit comments

Comments
 (0)