Skip to content

Commit 4a3ccf6

Browse files
thymikeeclaude
andauthored
test: close web platform command-coverage gaps from #1426 (#2011)
* test: close web platform command-coverage gaps from #1426 Closes #1900. The web coverage manifest carried 15 "known-gap" rows for commands with no web-specific evidence — some had never been checked against the web runtime, some just lacked a named test. Added dedicated tests and reclassified each row to command-contract: - boot, shutdown, install, reinstall, install-from-source, push, logs: new tests in packages/platform-web/src/runtime.test.ts prove the web runtime deliberately and permanently denies these operations (unsupported-platform-leaf), the same pattern already used for longPress/back/home/orientation/tvRemote/keyboard. - diff, press: new tests prove these commands share their exact runtime-execution plan with the already-live snapshot/click commands (snapshotRuntimePlanUses, pressRuntimeUses === clickRuntimeUses), so the admitted operation backing snapshot/click also backs them. - artifacts, events, batch, trace, replay: these commands have zero platform branching in their handlers; new tests dispatch each through its real production handler against a web-typed session/device to prove the existing generic code path works unchanged for web. - test: readReplayScriptMetadata deliberately drops `context platform=web` as a declarable value, so a scripted suite can never filter by --platform web. With no filter, discoverReplayTestEntries runs every discovered script unconditionally, so a new test proves an unfiltered `test` run executes correctly against a session already bound to a web device. Updates the manifest's classification-count gate to {capabilityDenial: 7, contract: 35, gap: 0, live: 12, total: 54} and replaces the now-vacuous "known gaps share one tracking issue" test with a planted-red proof that zero known-gap rows remain. Removes the now-dead WEB_COVERAGE_GAP_ISSUE constant and gap() helper. Live web-smoke scope is unchanged — no new commands were added to test/integration/smoke-web-platform.test.ts. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01SK6kvnNth6H7VxbbANi2xC * test: harden web coverage evidence from adversarial review An independent adversarial review of the previous commit found the `test` row's evidence was causally inert: the new test bound a web device to the session, but test-suite discovery (session-test-suite-command.ts) only ever reads flags.platform, never the session's device, so the test passed identically for any platform. Replace it with a test that proves the real, deterministic, web-specific behavior instead: `test --platform web` always reports zero matching scripts, because `readReplayScriptMetadata` drops `context platform=web` as an unsupported declared value and `ReplayTestPlatform` structurally excludes 'web' — so no `.ad` script, typed or untyped, can ever match that filter. Update the manifest row's assertion to state that limitation honestly instead of implying the command runs on web. Also: add a test pinning `press`'s runtime-execution plan to `click`'s (pressRuntimeUses === clickRuntimeUses, both descriptors reuse clickRuntimeUses) — the manifest's `press` row rested on that equality with nothing in the test suite that would catch it silently drifting. Fix a doc comment claiming appLogRuntimePlanUses spans all five app-log facts (it requires three; the other two are asserted as a reasonable superset, not because the plan needs them). Rework the manifest's top-of-file doc comment, which still described every contract row as "web-specific unit/provider evidence" after the previous commit added rows that instead prove a platform-agnostic code path or a structural limitation — state the three evidence shapes explicitly instead of retaining a sentence the new rows contradict. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01SK6kvnNth6H7VxbbANi2xC * test: keep test's web row a known-gap per review @thymikee is right: the test row's evidence proves the opposite of what command-contract is supposed to certify. ReplayTestPlatform excludes web from the declared-platform filter entirely, so `test --platform web` can never select a script — that's real, tested, command-specific behavior, but it documents a limitation, not executable evidence that the command works on web. Reclassifying it as command-contract on the strength of that test would misrepresent what was actually shown. Reverts the test row to known-gap, restores WEB_COVERAGE_GAP_ISSUE and gap(), and updates the classification counts to {contract: 34, gap: 1} (14 of 15 gaps closed). #1900 stays open for this one row until a separate decision adds web replay-test support or an explicit denial. The regression test proving the exclusion (session-command-replay.test.ts) stays in place; it's just no longer cited as coverage evidence. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01SK6kvnNth6H7VxbbANi2xC --------- Co-authored-by: Claude <[email protected]>
1 parent 021fe2a commit 4a3ccf6

9 files changed

Lines changed: 483 additions & 23 deletions

File tree

‎packages/platform-web/src/runtime.test.ts‎

Lines changed: 118 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -460,3 +460,121 @@ test('refuses web scrolling on a non-browser web cell', async () => {
460460
expect(binding.facts.operations.scrollDirection.available).toBe(false);
461461
expect(binding.operations.scrollDirection).toBeUndefined();
462462
});
463+
464+
// #1900: `boot`/`bootTargetHeadless` and `shutdown` have no web-specific evidence beyond this
465+
// runtime-owned fact — a managed browser session has no OS-level boot or shutdown to drive.
466+
test('boot and shutdown report the runtime-owned unavailable readiness fact', async () => {
467+
const binding = await createWebPlatformRuntime(host({ mode: 'transport-composed' })).bind({
468+
device,
469+
intent: { kind: 'ordinary' },
470+
scope: scope(),
471+
});
472+
for (const operation of ['bootTarget', 'bootTargetHeadless', 'shutdownTarget'] as const) {
473+
expect(binding.facts.operations[operation]).toEqual({
474+
available: false,
475+
reason: 'unsupported-platform-leaf',
476+
});
477+
expect(binding.operations[operation]).toBeUndefined();
478+
}
479+
});
480+
481+
// #1900: `install` and `reinstall` both resolve to `deployAppUse`, so one shared fact covers
482+
// both — a managed browser session has no installable native app package.
483+
test('install and reinstall share the runtime-owned unavailable deploy fact', async () => {
484+
const binding = await createWebPlatformRuntime(host({ mode: 'transport-composed' })).bind({
485+
device,
486+
intent: { kind: 'ordinary' },
487+
scope: scope(),
488+
});
489+
expect(binding.facts.operations.deployApp).toEqual({
490+
available: false,
491+
reason: 'unsupported-platform-leaf',
492+
});
493+
expect(binding.operations.deployApp).toBeUndefined();
494+
});
495+
496+
// #1900: `install-from-source` resolves to `readyMaterializeAndDeployAppUse`, requiring all three
497+
// of these facts; none of them exist for a browser target.
498+
test('install-from-source reports the runtime-owned unavailable materialize and deploy facts', async () => {
499+
const binding = await createWebPlatformRuntime(host({ mode: 'transport-composed' })).bind({
500+
device,
501+
intent: { kind: 'ordinary' },
502+
scope: scope(),
503+
});
504+
for (const operation of [
505+
'ensureReady',
506+
'materializeAppSource',
507+
'deployMaterializedApp',
508+
] as const) {
509+
expect(binding.facts.operations[operation]).toEqual({
510+
available: false,
511+
reason: 'unsupported-platform-leaf',
512+
});
513+
expect(binding.operations[operation]).toBeUndefined();
514+
}
515+
});
516+
517+
// #1900: `push` resolves to `readySendPushNotificationUse`, requiring both facts below; a browser
518+
// session has no native push channel to deliver through.
519+
test('push reports the runtime-owned unavailable readiness and push facts', async () => {
520+
const binding = await createWebPlatformRuntime(host({ mode: 'transport-composed' })).bind({
521+
device,
522+
intent: { kind: 'ordinary' },
523+
scope: scope(),
524+
});
525+
for (const operation of ['ensureReady', 'sendPushNotification'] as const) {
526+
expect(binding.facts.operations[operation]).toEqual({
527+
available: false,
528+
reason: 'unsupported-platform-leaf',
529+
});
530+
expect(binding.operations[operation]).toBeUndefined();
531+
}
532+
});
533+
534+
// #1900: `logs` resolves to `appLogRuntimePlanUses` (three of these five facts: appLogInspect,
535+
// appLogDoctor, appLogStart); the other two, appLogReattach/appLogCleanup, are asserted here too
536+
// since they answer the same "no native app-log channel" question a web target has no route for.
537+
test('logs reports the runtime-owned unavailable app-log facts', async () => {
538+
const binding = await createWebPlatformRuntime(host({ mode: 'transport-composed' })).bind({
539+
device,
540+
intent: { kind: 'ordinary' },
541+
scope: scope(),
542+
});
543+
for (const operation of [
544+
'appLogInspect',
545+
'appLogDoctor',
546+
'appLogStart',
547+
'appLogReattach',
548+
'appLogCleanup',
549+
] as const) {
550+
expect(binding.facts.operations[operation]).toEqual({
551+
available: false,
552+
reason: 'unsupported-platform-leaf',
553+
});
554+
expect(binding.operations[operation]).toBeUndefined();
555+
}
556+
});
557+
558+
// #1900: `diff` resolves to the exact same `snapshotRuntimePlanUses` as the live `snapshot`
559+
// command, so the admitted fact backing `snapshot` also backs `diff`.
560+
test('diff shares the admitted captureSnapshot fact that live snapshot and diff both require', async () => {
561+
const binding = await createWebPlatformRuntime(host({ mode: 'transport-composed' })).bind({
562+
device,
563+
intent: { kind: 'ordinary' },
564+
scope: scope(),
565+
});
566+
expect(binding.facts.operations.captureSnapshot).toEqual({ available: true });
567+
expect(binding.operations.captureSnapshot).toBeTypeOf('function');
568+
});
569+
570+
// #1900: `pressRuntimeUses` is literally `clickRuntimeUses` (`platform-runtime-operations.ts`),
571+
// so the admitted fact backing the live `click` command also backs `press`.
572+
test('press shares the admitted tapPoint fact that live click and press both require', async () => {
573+
const binding = await createWebPlatformRuntime(host({ mode: 'transport-composed' })).bind({
574+
device,
575+
intent: { kind: 'ordinary' },
576+
scope: scope(),
577+
});
578+
expect(binding.facts.operations.tapPoint).toEqual({ available: true });
579+
expect(binding.operations.tapPoint).toBeTypeOf('function');
580+
});
Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
import {
2+
clickRuntimeUses,
3+
pressRuntimeUses,
4+
} from '@agent-device/contracts/platform-runtime-operations';
5+
import { expect, test } from 'vitest';
6+
import { commandDescriptors } from '../registry.ts';
7+
8+
// #1900: the web coverage manifest's `press` row claims press shares click's admitted web
9+
// operation ("press shares the admitted tapPoint fact that live click and press both require").
10+
// That claim rests on `pressRuntimeUses` and the `press` descriptor literally being click's, not
11+
// merely looking similar — pin both here so a future divergence fails this test, not silently.
12+
test('press descriptor reuses the complete click plan uses with no legacy projection', () => {
13+
const click = commandDescriptors.find(({ name }) => name === 'click');
14+
const press = commandDescriptors.find(({ name }) => name === 'press');
15+
16+
expect(click).not.toHaveProperty('capability');
17+
expect(click).not.toHaveProperty('dispatch');
18+
expect(press).not.toHaveProperty('capability');
19+
expect(press).not.toHaveProperty('dispatch');
20+
expect(click?.platformExecution).toEqual({ kind: 'device-runtime', uses: clickRuntimeUses });
21+
expect(press?.platformExecution).toEqual({ kind: 'device-runtime', uses: clickRuntimeUses });
22+
expect(pressRuntimeUses).toBe(clickRuntimeUses);
23+
});
Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
import { createTestDeviceInventoryGateways } from '../../__tests__/test-utils/device-inventory-gateways.ts';
2+
import { test, expect } from 'vitest';
3+
import fs from 'node:fs';
4+
import os from 'node:os';
5+
import path from 'node:path';
6+
import { createRequestHandler } from './test-device-runtime-gateway.ts';
7+
import { LeaseRegistry } from '../lease-registry.ts';
8+
import type { DaemonArtifactInventoryEntry } from '@agent-device/contracts/observability';
9+
import { makeSessionStore } from '../../__tests__/test-utils/store-factory.ts';
10+
import { makeSession } from '../../__tests__/test-utils/session-factories.ts';
11+
import { WEB_DESKTOP_DEVICE } from '../../__tests__/test-utils/device-fixtures.ts';
12+
import { mkdtempForTestSync } from '../../__tests__/test-utils/tmp-dir.ts';
13+
import { cleanupDownloadableArtifact, trackDownloadableArtifact } from '../artifact-tracking.ts';
14+
15+
// #1900: `artifacts` (`handlers/lease.ts`) lists daemon-tracked artifacts by tenant scope with no
16+
// device or platform involvement at all — `listArtifactsForRequest` never reads `device.platform`.
17+
// This proves the command actually resolves and lists an artifact end to end for a session whose
18+
// device is web, the same round trip `request-router-events.test.ts` already proves for `events`.
19+
test('artifacts lists a daemon-tracked artifact produced during a web session', async () => {
20+
const sessionStore = makeSessionStore('agent-device-router-artifacts-web-');
21+
sessionStore.set('web-session', makeSession('web-session', { device: WEB_DESKTOP_DEVICE }));
22+
const artifactDir = mkdtempForTestSync('agent-device-router-artifacts-web-file-');
23+
const artifactPath = path.join(artifactDir, 'web-smoke.png');
24+
fs.writeFileSync(artifactPath, 'fixture-bytes');
25+
const artifactId = trackDownloadableArtifact({
26+
artifactPath,
27+
artifactType: 'screenshot',
28+
fileName: 'web-smoke.png',
29+
});
30+
31+
try {
32+
const handler = createRequestHandler({
33+
logPath: path.join(os.tmpdir(), 'daemon.log'),
34+
token: 'test-token',
35+
sessionStore,
36+
leaseRegistry: new LeaseRegistry(),
37+
deviceInventoryGateways: createTestDeviceInventoryGateways(),
38+
trackDownloadableArtifact: () => 'unused-artifact-id',
39+
});
40+
41+
const response = await handler({
42+
token: 'test-token',
43+
session: 'web-session',
44+
command: 'artifacts',
45+
positionals: [],
46+
flags: {},
47+
meta: { requestId: 'req-artifacts-web' },
48+
});
49+
50+
expect(response.ok).toBe(true);
51+
if (!response.ok) return;
52+
expect(response.data?.source).toBe('daemon');
53+
expect(response.data?.status).toBe('ready');
54+
const artifacts = response.data?.artifacts as unknown as DaemonArtifactInventoryEntry[];
55+
expect(
56+
artifacts.some(
57+
(artifact) => artifact.id === artifactId && artifact.filename === 'web-smoke.png',
58+
),
59+
).toBe(true);
60+
} finally {
61+
cleanupDownloadableArtifact(artifactId);
62+
}
63+
});

‎src/daemon/__tests__/request-router-events.test.ts‎

Lines changed: 43 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,8 @@ import path from 'node:path';
66
import { createRequestHandler } from './test-device-runtime-gateway.ts';
77
import { LeaseRegistry } from '../lease-registry.ts';
88
import { makeSessionStore } from '../../__tests__/test-utils/store-factory.ts';
9-
import { makeIosSession } from '../../__tests__/test-utils/session-factories.ts';
9+
import { makeIosSession, makeSession } from '../../__tests__/test-utils/session-factories.ts';
10+
import { WEB_DESKTOP_DEVICE } from '../../__tests__/test-utils/device-fixtures.ts';
1011

1112
test('events reads the daemon-owned session timeline without appending poll noise', async () => {
1213
const sessionStore = makeSessionStore('agent-device-router-events-');
@@ -173,6 +174,47 @@ test('events flushes pending event writes before reading', async () => {
173174
]);
174175
});
175176

177+
// #1900: `events` reads/flushes the session-owned timeline (`session-observability.ts`) with no
178+
// device-runtime binding at all, so a web-backed session exercises the exact same code path as
179+
// every other platform.
180+
test('events reads the daemon-owned session timeline for a web-backed session', async () => {
181+
const sessionStore = makeSessionStore('agent-device-router-events-web-');
182+
sessionStore.set('web-session', makeSession('web-session', { device: WEB_DESKTOP_DEVICE }));
183+
sessionStore.recordEvent('web-session', {
184+
kind: 'action.recorded',
185+
command: 'click',
186+
summary: 'Tapped Submit order',
187+
});
188+
189+
const handler = createRequestHandler({
190+
logPath: path.join(os.tmpdir(), 'daemon.log'),
191+
token: 'test-token',
192+
sessionStore,
193+
leaseRegistry: new LeaseRegistry(),
194+
deviceInventoryGateways: createTestDeviceInventoryGateways(),
195+
trackDownloadableArtifact: () => 'artifact-id',
196+
});
197+
198+
const response = await handler({
199+
token: 'test-token',
200+
session: 'web-session',
201+
command: 'events',
202+
positionals: ['10'],
203+
flags: {},
204+
meta: { requestId: 'req-events-web' },
205+
});
206+
207+
expect(response.ok).toBe(true);
208+
if (!response.ok) return;
209+
expect(response.data?.events).toEqual([
210+
expect.objectContaining({
211+
kind: 'action.recorded',
212+
command: 'click',
213+
summary: 'Tapped Submit order',
214+
}),
215+
]);
216+
});
217+
176218
test('request timeline records thrown request failures after scope creation', async () => {
177219
const sessionStore = makeSessionStore('agent-device-router-events-throws-');
178220
sessionStore.set('events-session', makeIosSession('events-session'));

‎src/daemon/handlers/__tests__/session-command-replay.test.ts‎

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -158,3 +158,79 @@ test('replay inherits parent device selectors for each invoked step', async () =
158158
expect(invoked[0]?.flags?.device).toBe('thymikee-iphone');
159159
expect(invoked[0]?.flags?.udid).toBe('00008150-001849640CF8401C');
160160
});
161+
162+
// #1900: `replay` routes through `handleSessionReplayCommandGroup` -> `session-replay.ts`, which
163+
// re-invokes each recorded step with no `platform === 'web'` branch anywhere in that path.
164+
test('replay inherits the parent web platform selector for each invoked step', async () => {
165+
const sessionStore = makeSessionStore();
166+
const replayRoot = mkdtempForTestSync('agent-device-replay-web-selectors-');
167+
const replayPath = path.join(replayRoot, 'web-selectors.ad');
168+
fs.writeFileSync(replayPath, 'open "http://127.0.0.1/"\n');
169+
170+
const invoked: DaemonRequest[] = [];
171+
const response = await handleSessionCommands({
172+
req: {
173+
token: 't',
174+
session: 'default',
175+
command: 'replay',
176+
positionals: [replayPath],
177+
flags: {
178+
platform: 'web',
179+
replayScriptSource: replayScriptSourceBundleFor(replayPath),
180+
},
181+
},
182+
sessionName: 'default',
183+
logPath: path.join(os.tmpdir(), 'daemon.log'),
184+
sessionStore,
185+
invoke: async (req) => {
186+
invoked.push(req);
187+
return { ok: true, data: {} };
188+
},
189+
});
190+
191+
expect(response).toBeTruthy();
192+
expect(response?.ok).toBe(true);
193+
expect(invoked.length).toBe(1);
194+
expect(invoked[0]?.flags?.platform).toBe('web');
195+
});
196+
197+
// #1900: `test`'s script discovery filters by each script's own `context platform=` declaration
198+
// (`session-test-discovery.ts`'s `matchesPlatformFilter`), but `readReplayScriptMetadata`
199+
// deliberately drops `web` as a declarable value (`ad-script`'s own
200+
// `readReplayScriptMetadata drops unsupported web platform` test) and
201+
// `ReplayTestPlatform = Exclude<PlatformSelector, 'web'>` (`session-test-types.ts`) excludes it
202+
// structurally. So no `.ad` script -- typed or untyped -- can ever declare or match `web`, and
203+
// `discoverReplayTestEntries` throws "No replay tests matched" for every source once a
204+
// `--platform web` filter is active (an untyped source is skipped as unfiltered-out, and no typed
205+
// source can ever carry the value the filter is looking for). That is the real, deterministic,
206+
// command-specific web behavior of `test`: not "it runs on web" but "its declared-platform filter
207+
// can never select web", proven here rather than merely asserted.
208+
test('test --platform web reports no matching scripts, typed or untyped, because ReplayTestPlatform excludes web', async () => {
209+
const sessionStore = makeSessionStore();
210+
const root = mkdtempForTestSync('agent-device-test-suite-web-excluded-');
211+
fs.writeFileSync(path.join(root, '01-untyped.ad'), 'open "http://127.0.0.1/"\n');
212+
fs.writeFileSync(path.join(root, '02-android.ad'), 'context platform=android\nopen "Demo"\n');
213+
214+
const response = await handleSessionCommands({
215+
req: {
216+
token: 't',
217+
session: 'default',
218+
command: 'test',
219+
positionals: [root],
220+
flags: { platform: 'web' },
221+
meta: { cwd: root, requestId: 'suite-web-excluded' },
222+
},
223+
sessionName: 'default',
224+
logPath: path.join(os.tmpdir(), 'daemon.log'),
225+
sessionStore,
226+
invoke: async () => {
227+
throw new Error('test must not invoke any step when --platform web matches nothing');
228+
},
229+
});
230+
231+
expect(response?.ok).toBe(false);
232+
if (response && !response.ok) {
233+
expect(response.error.code).toBe('INVALID_ARGS');
234+
expect(response.error.message).toBe('No replay tests matched for --platform web.');
235+
}
236+
});

‎src/daemon/handlers/__tests__/session-devices-batch-runtime.test.ts‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -258,6 +258,37 @@ test('batch step flags override parent selector flags', async () => {
258258
expect(response?.ok).toBe(true);
259259
});
260260

261+
// #1900: `batch` (`session-batch.ts` -> `runBatch`) just re-invokes each step through the normal
262+
// `DaemonInvokeFn` with no platform branching of its own, so a web platform selector threads
263+
// through the same way any other platform selector does.
264+
test('batch step forwards the parent web platform selector to each invoked step', async () => {
265+
const sessionStore = makeSessionStore();
266+
const response = await handleSessionCommands({
267+
req: {
268+
token: 't',
269+
session: 'default',
270+
command: 'batch',
271+
positionals: [],
272+
flags: {
273+
platform: 'web',
274+
batchSteps: [
275+
{ command: 'open', positionals: ['http://127.0.0.1/'] },
276+
{ command: 'screenshot', positionals: ['/tmp/web-batch.png'] },
277+
],
278+
},
279+
},
280+
sessionName: 'default',
281+
logPath: path.join(os.tmpdir(), 'daemon.log'),
282+
sessionStore,
283+
invoke: async (stepReq) => {
284+
expect(stepReq.flags?.platform).toBe('web');
285+
return { ok: true, data: {} };
286+
},
287+
});
288+
expect(response).toBeTruthy();
289+
expect(response?.ok).toBe(true);
290+
});
291+
261292
test('batch step forwards typed runtime payload', async () => {
262293
const sessionStore = makeSessionStore();
263294
const seenRuntimes: Array<DaemonRequest['runtime']> = [];

0 commit comments

Comments
 (0)