Skip to content

Commit 4db1b7d

Browse files
authored
test(coverage): split gap rows into owed work and accepted limitations (#3717)
* test(coverage): split gap rows into owed work and accepted limitations Every coverage-gap row is now one of two kinds: a known-gap row is owed work and must carry a tracking issue that owns it; a new accepted-gap row states a non-empty reason why no evidence lane will cover it and has no tracking issue. Enforce both in a shared manifest gate. Linux (9 rows) and web (1 row) carried rows pointing at closed issues #1915 and #1900; each states its permanent limitation instead, so the closed constants are removed. macOS moves doctor, shutdown, events, test, trace, and react-native to accepted limitations per the #1916 decision, keeping the scheduled interaction rows on the open umbrella. tvOS keeps all 38 rows as owed work on #1914. * test(coverage): keep macOS test-suite evidence as owed work and audit accepted reasons The CI macOS replay lane runs agent-device test with --report-junit and gates on its exit status, so the test row is closeable with real lane evidence and stays known-gap on #1916. Each accepted-gap reason now states the absence it claims against the repository's lanes, and the gate's docstring encodes that rule.
1 parent 59fdf00 commit 4db1b7d

10 files changed

Lines changed: 188 additions & 99 deletions

‎test/integration/command-coverage/declarations.ts‎

Lines changed: 62 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,10 @@ const C = PUBLIC_COMMANDS;
4242
* load time (`projectCoverage`), so adding a public command is one edit here rather than six
4343
* parallel edits in six manifests.
4444
*
45+
* A gap row is exactly one of two kinds: a `known-gap` row is owed work and its builder pins the
46+
* platform's open tracking issue; an `accepted-gap` row states a `reason` for why no evidence lane
47+
* will cover it and has no tracking issue. `assertCoverageGapRowsAreOwned` enforces both.
48+
*
4549
* `satisfies Record<PublicCommand, CommandCoverageDeclaration>` is the exhaustiveness check:
4650
* a command added to the catalog without a row here fails type-checking, and each platform's
4751
* coverage smoke test fails the same omission at runtime.
@@ -68,8 +72,9 @@ const COMMAND_COVERAGE_DECLARATIONS = {
6872
'artifacts lists a daemon-tracked artifact produced during a web session',
6973
'daemon artifact listing round-trips a tracked artifact for a web-backed session',
7074
),
71-
linux: linux.gap(
72-
'No Linux live command creates a downloadable daemon artifact for inventory yet',
75+
linux: linux.accepted(
76+
'No Linux live command creates a downloadable daemon artifact for inventory',
77+
'The Linux lanes run the local CLI, where command outputs are caller-owned local paths; request finalization inventories a response artifact only when the request carries a client artifact path (remote transport) or it is the sparse-snapshot fallback screenshot, which no Linux lane observes, so a Linux lane cannot produce a downloadable inventory entry without a local-transport product decision.',
7378
),
7479
},
7580
[C.devices]: {
@@ -135,7 +140,10 @@ const COMMAND_COVERAGE_DECLARATIONS = {
135140
'smoke:inventory-install',
136141
'doctor discovers the installed fixture app',
137142
),
138-
macos: macos.gap('No command-specific macOS doctor evidence exists yet'),
143+
macos: macos.accepted(
144+
'No command-specific macOS doctor evidence exists',
145+
'The doctor checks are daemon-owned host diagnostics — version, state dir, inventory probes, Metro — pinned platform-neutrally by the daemon doctor tests, and no macOS lane invokes the command: neither the System Settings replay nor the desktop and recording provider scenarios carry a doctor step.',
146+
),
139147
tvos: tvos.gap('No tvOS-specific doctor command evidence exists yet'),
140148
web: web.contract(
141149
'packages/platform-web/src/__tests__/doctor.test.ts',
@@ -204,14 +212,20 @@ const COMMAND_COVERAGE_DECLARATIONS = {
204212
'full:device-lifecycle',
205213
'shutdown succeeds and inventory reports the selected simulator stopped',
206214
),
207-
macos: macos.gap('No command-specific macOS shutdown evidence exists yet'),
215+
macos: macos.accepted(
216+
'No command-specific macOS shutdown evidence exists',
217+
'The Apple owner declares shutdownTarget unavailable on the macOS leaf (unsupported-platform-leaf): agent-device never powers off the host Mac it runs on, so the command refuses at admission and no lane will observe a successful macOS shutdown.',
218+
),
208219
tvos: tvos.gap('No tvOS-specific shutdown command evidence exists yet'),
209220
web: web.contract(
210221
'packages/platform-web/src/runtime.test.ts',
211222
'boot and shutdown report the runtime-owned unavailable readiness fact',
212223
'the web runtime fact rejects device shutdown',
213224
),
214-
linux: linux.gap('No Linux-specific shutdown command evidence exists yet'),
225+
linux: linux.accepted(
226+
'No Linux-specific shutdown command evidence exists',
227+
'The Linux owner declares shutdownTarget unavailable (unsupported-platform-leaf): the Linux desktop is the host machine itself and agent-device never powers off the machine it runs on, so the command refuses at admission and no lane will observe a successful shutdown.',
228+
),
215229
},
216230
[C.appState]: {
217231
androidEmulator: androidEmulator.live(
@@ -280,7 +294,10 @@ const COMMAND_COVERAGE_DECLARATIONS = {
280294
'logs reports the runtime-owned unavailable app-log facts',
281295
'the web runtime fact rejects native app-log commands',
282296
),
283-
linux: linux.gap('No Linux-specific app-log command evidence exists yet'),
297+
linux: linux.accepted(
298+
'No Linux-specific app-log command evidence exists',
299+
'The Linux runtime declares the whole app-log family unavailable (unsupported-platform-leaf) by owner decision: a Linux desktop session has no runtime-owned app log stream, so the command refuses at admission and no lane will observe a Linux app-log stream.',
300+
),
284301
},
285302
[C.events]: {
286303
androidEmulator: androidEmulator.live(
@@ -291,7 +308,10 @@ const COMMAND_COVERAGE_DECLARATIONS = {
291308
'full:observability-artifacts',
292309
'timeline contains commands from this session',
293310
),
294-
macos: macos.gap('No command-specific macOS event timeline evidence exists yet'),
311+
macos: macos.accepted(
312+
'No command-specific macOS event timeline evidence exists',
313+
'The events timeline is daemon-owned session bookkeeping, pinned by request-router-events.test.ts for a daemon-backed web session, and no macOS lane invokes the command: neither the System Settings replay nor the desktop and recording provider scenarios carry an events step.',
314+
),
295315
tvos: tvos.gap('No tvOS-specific session-event command evidence exists yet'),
296316
web: web.contract(
297317
'packages/daemon/src/server/__tests__/request-router-events.test.ts',
@@ -388,8 +408,9 @@ const COMMAND_COVERAGE_DECLARATIONS = {
388408
),
389409
macos: macos.gap('No command-specific macOS test-suite evidence exists yet'),
390410
tvos: tvos.gap('No tvOS-specific test-suite command evidence exists yet'),
391-
web: web.gap(
392-
"Web test-suite execution has no executable web evidence: ReplayTestPlatform = Exclude<PlatformSelector, 'web'> structurally excludes web from the declared-platform filter, so `test --platform web` can never select a script (proven by a regression test in session-command-replay.test.ts) — that is evidence of what the command cannot do, not that it works on web",
411+
web: web.accepted(
412+
'No command-specific web test-suite evidence exists',
413+
"ReplayTestPlatform = Exclude<PlatformSelector, 'web'> structurally excludes web from the declared-platform filter, so `test --platform web` can never select a script (proven by a regression test in session-command-replay.test.ts); there is no successful web test-suite execution for any lane to observe, and adding web replay-test support is a product decision, not scheduled work.",
393414
),
394415
linux: linux.commandEvidenceLive(
395416
'the command-evidence lane runs a dedicated Linux script as a test suite',
@@ -474,7 +495,10 @@ const COMMAND_COVERAGE_DECLARATIONS = {
474495
'install and reinstall share the runtime-owned unavailable deploy fact',
475496
'the web runtime fact rejects native app installation',
476497
),
477-
linux: linux.gap('No Linux-specific application installation command evidence exists yet'),
498+
linux: linux.accepted(
499+
'No Linux-specific application installation command evidence exists',
500+
'The Linux owner declares the app-deployment family unavailable (unsupported-platform-leaf): a Linux desktop app is launched by name through the desktop interactor and there is no installable package model, so the command refuses at admission and no lane will observe a Linux install.',
501+
),
478502
},
479503
[C.reinstall]: {
480504
androidEmulator: androidEmulator.contract(
@@ -500,7 +524,10 @@ const COMMAND_COVERAGE_DECLARATIONS = {
500524
'install and reinstall share the runtime-owned unavailable deploy fact',
501525
'the web runtime fact rejects native app reinstallation',
502526
),
503-
linux: linux.gap('No Linux-specific application reinstallation command evidence exists yet'),
527+
linux: linux.accepted(
528+
'No Linux-specific application reinstallation command evidence exists',
529+
'Reinstall reuses the app-deployment family the Linux runtime declares unavailable (unsupported-platform-leaf): there is no installable package model for the desktop interactor to reinstall, so the command refuses at admission and no lane will observe a Linux reinstall.',
530+
),
504531
},
505532
[C.push]: {
506533
androidEmulator: androidEmulator.live(
@@ -527,7 +554,10 @@ const COMMAND_COVERAGE_DECLARATIONS = {
527554
'push reports the runtime-owned unavailable readiness and push facts',
528555
'the web runtime fact rejects native push notification delivery',
529556
),
530-
linux: linux.gap('No Linux-specific push delivery command evidence exists yet'),
557+
linux: linux.accepted(
558+
'No Linux-specific push delivery command evidence exists',
559+
'The Linux runtime declares push delivery unavailable (unsupported-platform-leaf): the desktop interactor has no OS push channel to deliver through, so the command refuses at admission and no lane will observe a Linux push delivery.',
560+
),
531561
},
532562
[C.triggerAppEvent]: {
533563
androidEmulator: androidEmulator.contract(
@@ -763,7 +793,10 @@ const COMMAND_COVERAGE_DECLARATIONS = {
763793
'full:observability-artifacts',
764794
'Release fixture returns typed detected=false and dismissed=false overlay state',
765795
),
766-
macos: macos.gap('No command-specific macOS React Native inspection evidence exists yet'),
796+
macos: macos.accepted(
797+
'No command-specific macOS React Native inspection evidence exists',
798+
"react-native dismiss-overlay only answers meaningfully on a surface that carries a React Native overlay, the repository's React Native overlay evidence runs on iOS simulators and Android emulators, and no macOS lane invokes the command: neither the System Settings replay nor the desktop and recording provider scenarios carry a react-native step.",
799+
),
767800
tvos: tvos.gap('No tvOS-specific React Native inspection command evidence exists yet'),
768801
// R61: no owner fact refuses this command on a browser — its whole device work is one bound
769802
// `tapPoint` the web target admits — so it now runs and reports that no overlay is present.
@@ -800,7 +833,10 @@ const COMMAND_COVERAGE_DECLARATIONS = {
800833
'start web recording',
801834
'web recording starts and stops through the scoped provider',
802835
),
803-
linux: linux.gap('No Linux-specific recording command evidence exists yet'),
836+
linux: linux.accepted(
837+
'No Linux-specific recording command evidence exists',
838+
'The Linux runtime declares the screen-recording family unavailable (unsupported-platform-leaf): there is no Linux capture pipeline, so the command refuses at admission and no lane will produce a playable Linux recording.',
839+
),
804840
},
805841
[C.trace]: {
806842
androidEmulator: androidEmulator.live(
@@ -811,14 +847,20 @@ const COMMAND_COVERAGE_DECLARATIONS = {
811847
'full:observability-artifacts',
812848
'typed start/stop lifecycle retains the requested path and captures non-empty diagnostics',
813849
),
814-
macos: macos.gap('No command-specific macOS trace evidence exists yet'),
850+
macos: macos.accepted(
851+
'No command-specific macOS trace evidence exists',
852+
'trace never binds a device runtime: handleTraceCommand carries no bindDevice or inspectFacts call, its start/stop bookkeeping is pinned by the daemon trace-runtime tests, and no macOS lane invokes the command: neither the System Settings replay nor the desktop and recording provider scenarios carry a trace step.',
853+
),
815854
tvos: tvos.gap('No tvOS-specific trace command evidence exists yet'),
816855
web: web.contract(
817856
'packages/daemon/src/sessions/handlers/__tests__/trace-runtime.test.ts',
818857
'starts and stops one trace through the session-owned trace slot on a web session',
819858
'session-scoped trace start/stop bookkeeping works the same for a web session as any other platform',
820859
),
821-
linux: linux.gap('No Linux-specific trace command evidence exists yet'),
860+
linux: linux.accepted(
861+
'No Linux-specific trace command evidence exists',
862+
'trace never binds a device runtime: handleTraceCommand carries no bindDevice or inspectFacts call and its start/stop bookkeeping is pinned by the daemon trace-runtime tests, and no Linux lane invokes the command: neither the replay script nor the command-evidence lane carries a trace step.',
863+
),
822864
},
823865
[C.find]: {
824866
androidEmulator: androidEmulator.live(
@@ -1386,7 +1428,10 @@ const COMMAND_COVERAGE_DECLARATIONS = {
13861428
'install-from-source reports the runtime-owned unavailable materialize and deploy facts',
13871429
'the web runtime fact rejects source-based app installation',
13881430
),
1389-
linux: linux.gap('No Linux-specific source-install command evidence exists yet'),
1431+
linux: linux.accepted(
1432+
'No Linux-specific source-install command evidence exists',
1433+
'Source installation runs through the app-deployment family the Linux runtime declares unavailable (unsupported-platform-leaf): the desktop has no materialized-app install route to source from, so the command refuses at admission and no lane will observe a Linux source install.',
1434+
),
13901435
},
13911436
} satisfies Record<PublicCommand, CommandCoverageDeclaration>;
13921437

‎test/integration/command-coverage/entries.ts‎

Lines changed: 20 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -7,11 +7,9 @@ import {
77
} from '../macos-e2e/live-scenarios.ts';
88
import {
99
LINUX_COMMAND_EVIDENCE,
10-
LINUX_COVERAGE_GAP_ISSUE,
1110
LINUX_REPLAY_EVIDENCE,
1211
MACOS_COVERAGE_GAP_ISSUE,
1312
TVOS_COVERAGE_GAP_ISSUE,
14-
WEB_COVERAGE_GAP_ISSUE,
1513
WEB_SMOKE_EVIDENCE,
1614
type RepositoryEvidence,
1715
} from './evidence.ts';
@@ -58,6 +56,11 @@ export type MacOsPlatformCoverageEntry =
5856
assertion: string;
5957
level: 'known-gap';
6058
trackingIssue: number;
59+
}
60+
| {
61+
assertion: string;
62+
level: 'accepted-gap';
63+
reason: string;
6164
};
6265

6366
export type TvOsPlatformCoverageEntry =
@@ -91,8 +94,8 @@ export type WebPlatformCoverageEntry =
9194
}
9295
| {
9396
assertion: string;
94-
level: 'known-gap';
95-
trackingIssue: number;
97+
level: 'accepted-gap';
98+
reason: string;
9699
};
97100

98101
export type LinuxPlatformCoverageEntry =
@@ -103,8 +106,8 @@ export type LinuxPlatformCoverageEntry =
103106
}
104107
| {
105108
assertion: string;
106-
level: 'known-gap';
107-
trackingIssue: number;
109+
level: 'accepted-gap';
110+
reason: string;
108111
};
109112

110113
/** One command's coverage judgment on every platform that classifies the public catalog. */
@@ -170,6 +173,11 @@ export const macos = {
170173
level: 'known-gap',
171174
trackingIssue: MACOS_COVERAGE_GAP_ISSUE,
172175
}),
176+
accepted: (assertion: string, reason: string): MacOsPlatformCoverageEntry => ({
177+
assertion,
178+
level: 'accepted-gap',
179+
reason,
180+
}),
173181
};
174182

175183
export const tvos = {
@@ -207,10 +215,10 @@ export const web = {
207215
level: 'command-contract',
208216
owner: { path, test },
209217
}),
210-
gap: (assertion: string): WebPlatformCoverageEntry => ({
218+
accepted: (assertion: string, reason: string): WebPlatformCoverageEntry => ({
211219
assertion,
212-
level: 'known-gap',
213-
trackingIssue: WEB_COVERAGE_GAP_ISSUE,
220+
level: 'accepted-gap',
221+
reason,
214222
}),
215223
};
216224

@@ -230,10 +238,10 @@ export const linux = {
230238
level: 'command-contract',
231239
owner: { path, test },
232240
}),
233-
gap: (assertion: string): LinuxPlatformCoverageEntry => ({
241+
accepted: (assertion: string, reason: string): LinuxPlatformCoverageEntry => ({
234242
assertion,
235-
level: 'known-gap',
236-
trackingIssue: LINUX_COVERAGE_GAP_ISSUE,
243+
level: 'accepted-gap',
244+
reason,
237245
}),
238246
};
239247

‎test/integration/command-coverage/evidence.ts‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -12,11 +12,10 @@ export type RepositoryEvidence = {
1212

1313
const C = PUBLIC_COMMANDS;
1414

15-
// Tracking issues for the known-gap rows each platform still carries.
15+
// The tracking issues that own the `known-gap` rows each platform still carries. An
16+
// accepted-gap row states a `reason` instead of citing one of these.
1617
export const MACOS_COVERAGE_GAP_ISSUE = 1916;
1718
export const TVOS_COVERAGE_GAP_ISSUE = 1914;
18-
export const WEB_COVERAGE_GAP_ISSUE = 1900;
19-
export const LINUX_COVERAGE_GAP_ISSUE = 1915;
2019

2120
// Android contract evidence that only the coverage declarations cite. Evidence shared with a
2221
// production test module keeps living next to that module and is imported by declarations.ts.

‎test/integration/linux-e2e/coverage.ts‎

Lines changed: 4 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -3,21 +3,17 @@ import type { PublicCommand } from '../command-coverage/entries.ts';
33
import { LINUX_COMMAND_EVIDENCE, LINUX_REPLAY_EVIDENCE } from '../command-coverage/evidence.ts';
44
import { buildCoverageClassificationSummary } from '../support/coverage-classification.ts';
55

6-
export {
7-
LINUX_COMMAND_EVIDENCE,
8-
LINUX_COVERAGE_GAP_ISSUE,
9-
LINUX_REPLAY_EVIDENCE,
10-
} from '../command-coverage/evidence.ts';
6+
export { LINUX_COMMAND_EVIDENCE, LINUX_REPLAY_EVIDENCE } from '../command-coverage/evidence.ts';
117

128
/**
139
* One primary, observable owner for every public command on the Linux desktop.
1410
*
1511
* Live rows cite either the existing Linux replay or the separate command-evidence
1612
* lane. The existing replay scope stays unchanged. Contract rows cite the existing
17-
* provider scenario or dedicated Linux unit/runtime evidence; they do not turn
13+
* provider scenario or dedicated Linux runtime-fact evidence; they do not turn
1814
* mocked provider calls into live desktop claims. Capability denials are derived
19-
* from the owning command-descriptor matrix. Known gaps are explicit follow-up
20-
* work, not an implicit claim that a generic command works.
15+
* from the owning command-descriptor matrix. Every remaining gap is an accepted
16+
* limitation naming why no lane will cover it; Linux carries no open tracking issue.
2117
*
2218
* Projected from the command-owned declaration table; the rows are authored there.
2319
*/

‎test/integration/smoke-linux-coverage.test.ts‎

Lines changed: 21 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -5,9 +5,11 @@ import test from 'node:test';
55

66
import { parseReplayScriptDetailed } from '@agent-device/ad-script';
77
import { PUBLIC_COMMANDS } from '@agent-device/command-registry/catalog';
8-
import { assertCoverageClassificationSummaryWiredToManifest } from './support/coverage-classification.ts';
98
import {
10-
LINUX_COVERAGE_GAP_ISSUE,
9+
assertCoverageClassificationSummaryWiredToManifest,
10+
assertCoverageGapRowsAreOwned,
11+
} from './support/coverage-classification.ts';
12+
import {
1113
LINUX_COMMAND_EVIDENCE,
1214
LINUX_PLATFORM_COVERAGE,
1315
LINUX_PLATFORM_COVERAGE_CLASSIFICATION_SUMMARY,
@@ -24,6 +26,7 @@ const publicCommands = Object.values(PUBLIC_COMMANDS).sort();
2426

2527
test('Linux coverage exhaustively classifies the public catalog', () => {
2628
assert.deepEqual(Object.keys(LINUX_PLATFORM_COVERAGE).sort(), publicCommands);
29+
assertCoverageGapRowsAreOwned('Linux', LINUX_PLATFORM_COVERAGE);
2730

2831
for (const command of publicCommands) {
2932
const entry = LINUX_PLATFORM_COVERAGE[command];
@@ -32,13 +35,6 @@ test('Linux coverage exhaustively classifies the public catalog', () => {
3235
assert.ok(entry.owner.path.trim().length > 0, `${command} needs an evidence path`);
3336
assert.ok(entry.owner.test.trim().length > 0, `${command} needs named evidence`);
3437
}
35-
if (entry.level === 'known-gap') {
36-
assert.equal(
37-
entry.trackingIssue,
38-
LINUX_COVERAGE_GAP_ISSUE,
39-
`${command} has the wrong Linux gap issue`,
40-
);
41-
}
4238
}
4339
});
4440

@@ -117,11 +113,20 @@ test('Linux contract claims name existing executable evidence', () => {
117113
}
118114
});
119115

120-
test('Linux known gaps use one grouped tracking issue', () => {
121-
const gapIssues = new Set(
122-
Object.values(LINUX_PLATFORM_COVERAGE)
123-
.filter((entry) => entry.level === 'known-gap')
124-
.map((entry) => entry.trackingIssue),
125-
);
126-
assert.deepEqual([...gapIssues], [LINUX_COVERAGE_GAP_ISSUE]);
116+
test('Linux accepted gaps name the permanent Linux limitation behind each row', () => {
117+
const acceptedCommands = Object.entries(LINUX_PLATFORM_COVERAGE)
118+
.filter(([, entry]) => entry.level === 'accepted-gap')
119+
.map(([command]) => command)
120+
.sort();
121+
assert.deepEqual(acceptedCommands, [
122+
'artifacts',
123+
'install',
124+
'install-from-source',
125+
'logs',
126+
'push',
127+
'record',
128+
'reinstall',
129+
'shutdown',
130+
'trace',
131+
]);
127132
});

0 commit comments

Comments
 (0)