Skip to content

Commit b10c8cf

Browse files
authored
fix: warn agents that armed-repair diagnostic reads are recorded too (#1271 stage 1) (#1287)
During an armed `--save-script` record-and-heal repair, read-only diagnostics an agent runs to locate the corrected target (snapshot -i, get attrs, find, is) are recorded into the healed script by default, alongside the corrective press. The wave-3 E3 repair-economics experiment measured 0/4 trials producing a clean healed script hands-off, and one recorded `get attrs` caused a second, self-inflicted identity-mismatch divergence on fresh replay. This is stage 1 of the maintainer's two-stage triage on #1271: safe interim guidance only, no recording-behavior change. Stage 2 (defaulting read-only commands out of the repair transaction) stays gated on an ADR-0012 amendment. - divergence.ts: buildRepairHintGuidance appends a diagnostics --no-record clause to every repairHint's text guidance, gated on resume.repairSessionHeld === true (decision 6, R7 C1's armed-repair signal) so it never renders on a plain, non-repair divergence. - cli-help.ts: the "Agent-supervised repair (heal-by-doing)" section in `help workflow` now says the same thing. - Unit coverage: divergence.test.ts asserts the clause is present iff repairSessionHeld is true, across record-and-heal/state-repair/caution/ manual. - SkillGym regression: agent-device-smoke-suite.ts adds record-and-heal-diagnostics-no-record, verified 3/3 against claude-haiku and codex-mini live runners.
1 parent 042f1c3 commit b10c8cf

4 files changed

Lines changed: 284 additions & 1 deletion

File tree

‎src/cli/parser/cli-help.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -323,7 +323,7 @@ Validation and evidence:
323323
agent-device screenshot
324324
agent-device press 124 817
325325
agent-device snapshot -i
326-
Startup/CPU/memory/frame first pass: perf metrics --json (bare perf and metrics are aliases). Focused frame/jank health: perf frames --json. Memory-only sample: perf memory sample --json returns compact JSON with bounded top offenders. Heap/memgraph artifact escalation: perf memory snapshot --out heap.artifact; use --kind android-hprof on Android or --kind memgraph on supported Apple simulator/macOS app sessions. Android native profiling: perf cpu profile start|stop|report --kind simpleperf --out <path>; Android native traces: perf trace start|stop --kind perfetto --out <path>. Artifact collectors return compact state/path/size metadata only; raw heap/profile/trace files stay on disk. Treat native perf output as the agent evidence: for example, a Perfetto stop can return state=stopped, outPath=/tmp/app.perfetto-trace, sizeBytes=5392410, and method=adb-shell-perfetto while the 5.3 MB raw trace stays in the artifact. This is better than raw dumps for agents because it is stable, bounded, and keeps large artifacts out of context. heapprofd is deferred until Perfetto plumbing is available. Replay divergence and resume: a failing replay/test step returns REPLAY_DIVERGENCE with a bounded report (screen digest, ranked selector suggestions, resume). Repair app state, then resume with replay --from <n> --plan-digest <sha256> (both from the report's resume field) to continue from the failed step without re-running earlier ones; resume never re-executes skipped steps, so app state is the caller's responsibility, and it is rejected with INVALID_ARGS when the plan digest is stale, --from is out of range, or the skipped range/target touches runtime control flow. The plan digest binds the script, its includes, the effective --platform/--target, and per-action runtime/identity. Native .ad interpolation is late-bound after planning, so changing only its values keeps the digest; Maestro environment substitution occurs during compatibility parsing and can change action inputs, includes, or control expansion, so it can change the digest. --from is replay-only; test rejects it. --update/-u no longer rewrites the script (ADR 0012) — it is a no-op kept for compatibility; every divergence already carries the same ranked suggestions. Agent-supervised repair (heal-by-doing): arm replay <file>.ad --save-script[=<out>] before step 1 (armed once; --from continuations do not need it again). Every divergence carries a repairHint: record-and-heal means press the correct control via a blessed @ref from the divergence's screen.refs, recorded (no --no-record), then continue with replay --from <n+1> --plan-digest <sha256>; state-repair means the script is correct but app state is not, so fix state with --no-record actions, then replay --from <n> --plan-digest <sha256> to re-run the unchanged step; caution means something already matches the recorded selector, so a blind re-press may repeat the mistake; manual means no safe automated repair could be proven. Ending the repair with close --save-script[=<out>] writes only the steps recorded since the arming replay as <out>, defaulting to the <file> sibling <stem>.healed.ad — review its diff before promoting it over the original.
326+
Startup/CPU/memory/frame first pass: perf metrics --json (bare perf and metrics are aliases). Focused frame/jank health: perf frames --json. Memory-only sample: perf memory sample --json returns compact JSON with bounded top offenders. Heap/memgraph artifact escalation: perf memory snapshot --out heap.artifact; use --kind android-hprof on Android or --kind memgraph on supported Apple simulator/macOS app sessions. Android native profiling: perf cpu profile start|stop|report --kind simpleperf --out <path>; Android native traces: perf trace start|stop --kind perfetto --out <path>. Artifact collectors return compact state/path/size metadata only; raw heap/profile/trace files stay on disk. Treat native perf output as the agent evidence: for example, a Perfetto stop can return state=stopped, outPath=/tmp/app.perfetto-trace, sizeBytes=5392410, and method=adb-shell-perfetto while the 5.3 MB raw trace stays in the artifact. This is better than raw dumps for agents because it is stable, bounded, and keeps large artifacts out of context. heapprofd is deferred until Perfetto plumbing is available. Replay divergence and resume: a failing replay/test step returns REPLAY_DIVERGENCE with a bounded report (screen digest, ranked selector suggestions, resume). Repair app state, then resume with replay --from <n> --plan-digest <sha256> (both from the report's resume field) to continue from the failed step without re-running earlier ones; resume never re-executes skipped steps, so app state is the caller's responsibility, and it is rejected with INVALID_ARGS when the plan digest is stale, --from is out of range, or the skipped range/target touches runtime control flow. The plan digest binds the script, its includes, the effective --platform/--target, and per-action runtime/identity. Native .ad interpolation is late-bound after planning, so changing only its values keeps the digest; Maestro environment substitution occurs during compatibility parsing and can change action inputs, includes, or control expansion, so it can change the digest. --from is replay-only; test rejects it. --update/-u no longer rewrites the script (ADR 0012) — it is a no-op kept for compatibility; every divergence already carries the same ranked suggestions. Agent-supervised repair (heal-by-doing): arm replay <file>.ad --save-script[=<out>] before step 1 (armed once; --from continuations do not need it again). Every divergence carries a repairHint: record-and-heal means press the correct control via a blessed @ref from the divergence's screen.refs, recorded (no --no-record), then continue with replay --from <n+1> --plan-digest <sha256>; state-repair means the script is correct but app state is not, so fix state with --no-record actions, then replay --from <n> --plan-digest <sha256> to re-run the unchanged step; caution means something already matches the recorded selector, so a blind re-press may repeat the mistake; manual means no safe automated repair could be proven. While armed, read-only inspection used to locate the repair target (snapshot -i, get attrs, find, is) is recorded too — pass --no-record on those commands, or they land in the healed script alongside the corrective action. Ending the repair with close --save-script[=<out>] writes only the steps recorded since the arming replay as <out>, defaulting to the <file> sibling <stem>.healed.ad — review its diff before promoting it over the original.
327327
Recording: record start/stop. The default scope is app and expects an active session created by open <app>; this keeps app proof videos tied to the intended app session. Use record start --scope device/system to explicitly request whole-screen capture where the selected backend supports it, such as recordings that intentionally span multiple apps, home screen, settings, or app transitions. Use --max-size to cap the longest edge and --quality medium|high to choose output quality across Android and Apple targets. By default, stop burns touch overlays into the video; use record start --hide-touches for the fastest raw recording. Android record start publishes a durable device manifest. Android adb screenrecord has a 180s platform limit, so longer Android recordings are returned as multiple MP4 chunks while the daemon stays alive; after daemon restart, record stop recovers only manifest-owned chunks and warns when gesture overlays are unavailable. For gesture-heavy iOS simulator proof videos, prefer --hide-touches because overlay timing depends on a stable runner session while gestures are executing. Tracing: trace start ./trace.log, trace stop ./trace.log. Paths are positional.
328328
Stable known flow: batch ./steps.json, not workflow batch.
329329
Inline batch JSON example:

‎src/replay/__tests__/divergence.test.ts‎

Lines changed: 204 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -679,3 +679,207 @@ test('formatReplayDivergenceReport renders neither caution command when resume i
679679
assert.doesNotMatch(report!, /replay --from/);
680680
assert.match(report!, /cannot currently be resumed automatically/);
681681
});
682+
683+
// --- #1271 stage 1 (safe interim; default exclusion is stage 2, gated on an
684+
// ADR-0012 amendment): read-only diagnostics an agent runs mid-repair to
685+
// LOCATE the target (snapshot -i, get attrs, find, is) are recorded into the
686+
// healed script by default. The repairHint guidance must teach this — but
687+
// ONLY when `resume.repairSessionHeld === true` (decision 6, R7 C1's signal
688+
// that this divergence came from a repair-armed `--save-script` replay).
689+
// A plain non-repair divergence never carries `repairSessionHeld` and must
690+
// never render the clause — it would be pure noise. ---
691+
692+
const REPAIR_DIAGNOSTICS_CLAUSE_PATTERN =
693+
/Read-only inspection while armed \(snapshot -i, get attrs, find, is\) is recorded too — use --no-record on those commands or they will land in the healed script\./;
694+
695+
test('formatReplayDivergenceReport appends the diagnostics --no-record clause for record-and-heal when repairSessionHeld is true', async () => {
696+
const { formatReplayDivergenceReport } = await import('../divergence.ts');
697+
const report = formatReplayDivergenceReport({
698+
divergence: {
699+
version: 1,
700+
kind: 'action-failure',
701+
step: { index: 2, source: { path: '/tmp/flow.ad', line: 3 } },
702+
action: 'click id="article"',
703+
cause: { code: 'COMMAND_FAILED', message: 'not hittable' },
704+
screen: { state: 'available', refsGeneration: 1, refs: [{ ref: 'e1', role: 'button' }] },
705+
suggestions: [],
706+
suggestionCount: 0,
707+
resume: { allowed: true, from: 3, planDigest: 'deadbeef', repairSessionHeld: true },
708+
repairHint: 'record-and-heal',
709+
},
710+
});
711+
assert.ok(report);
712+
assert.match(report!, REPAIR_DIAGNOSTICS_CLAUSE_PATTERN);
713+
// The existing resume guidance must still render — the new clause is
714+
// additive, not a replacement.
715+
assert.match(report!, /then replay --from 3 --plan-digest deadbeef\./);
716+
});
717+
718+
test('formatReplayDivergenceReport OMITS the diagnostics clause for record-and-heal when repairSessionHeld is absent (plain, non-repair divergence)', async () => {
719+
const { formatReplayDivergenceReport } = await import('../divergence.ts');
720+
const report = formatReplayDivergenceReport({
721+
divergence: {
722+
version: 1,
723+
kind: 'action-failure',
724+
step: { index: 2, source: { path: '/tmp/flow.ad', line: 3 } },
725+
action: 'click id="article"',
726+
cause: { code: 'COMMAND_FAILED', message: 'not hittable' },
727+
screen: { state: 'available', refsGeneration: 1, refs: [{ ref: 'e1', role: 'button' }] },
728+
suggestions: [],
729+
suggestionCount: 0,
730+
// No `repairSessionHeld` — this divergence is not from a repair-armed
731+
// replay, so the diagnostics clause would be pure noise.
732+
resume: { allowed: true, from: 3, planDigest: 'deadbeef' },
733+
repairHint: 'record-and-heal',
734+
},
735+
});
736+
assert.ok(report);
737+
assert.doesNotMatch(report!, /Read-only inspection/);
738+
assert.doesNotMatch(report!, /--no-record on those commands/);
739+
});
740+
741+
test('formatReplayDivergenceReport appends the diagnostics clause for state-repair when armed, distinct from the existing app-state --no-record clause', async () => {
742+
const { formatReplayDivergenceReport } = await import('../divergence.ts');
743+
const report = formatReplayDivergenceReport({
744+
divergence: {
745+
version: 1,
746+
kind: 'action-failure',
747+
step: { index: 4, source: { path: '/tmp/flow.ad', line: 8 } },
748+
action: 'click label="Continue"',
749+
cause: { code: 'COMMAND_FAILED', message: 'not hittable' },
750+
screen: { state: 'available', refsGeneration: 1, refs: [{ ref: 'e1', role: 'button' }] },
751+
suggestions: [],
752+
suggestionCount: 0,
753+
resume: { allowed: true, from: 4, planDigest: 'cafef00d', repairSessionHeld: true },
754+
repairHint: 'state-repair',
755+
},
756+
});
757+
assert.ok(report);
758+
// The app-state-fix clause (about correcting APP STATE) still renders...
759+
assert.match(report!, /fix app state with --no-record actions, then replay --from 4/);
760+
// ...and the diagnostics clause (about inspection READS) is additive, not
761+
// a replacement for it — both can be true for the same divergence.
762+
assert.match(report!, REPAIR_DIAGNOSTICS_CLAUSE_PATTERN);
763+
});
764+
765+
test('formatReplayDivergenceReport OMITS the diagnostics clause for state-repair when repairSessionHeld is absent', async () => {
766+
const { formatReplayDivergenceReport } = await import('../divergence.ts');
767+
const report = formatReplayDivergenceReport({
768+
divergence: {
769+
version: 1,
770+
kind: 'action-failure',
771+
step: { index: 4, source: { path: '/tmp/flow.ad', line: 8 } },
772+
action: 'click label="Continue"',
773+
cause: { code: 'COMMAND_FAILED', message: 'not hittable' },
774+
screen: { state: 'available', refsGeneration: 1, refs: [{ ref: 'e1', role: 'button' }] },
775+
suggestions: [],
776+
suggestionCount: 0,
777+
resume: { allowed: true, from: 4, planDigest: 'cafef00d' },
778+
repairHint: 'state-repair',
779+
},
780+
});
781+
assert.ok(report);
782+
assert.match(report!, /fix app state with --no-record actions, then replay --from 4/);
783+
assert.doesNotMatch(report!, /Read-only inspection/);
784+
});
785+
786+
test('formatReplayDivergenceReport appends the diagnostics clause for caution when armed, alongside the dual-path resume commands', async () => {
787+
const { formatReplayDivergenceReport } = await import('../divergence.ts');
788+
const report = formatReplayDivergenceReport({
789+
divergence: {
790+
version: 1,
791+
kind: 'identity-mismatch',
792+
step: { index: 2, source: { path: '/tmp/flow.ad', line: 3 } },
793+
action: 'click label="Save"',
794+
cause: { code: 'IDENTITY_MISMATCH', message: 'resolved a different element' },
795+
screen: { state: 'available', refsGeneration: 1, refs: [{ ref: 'e1', role: 'button' }] },
796+
suggestions: [],
797+
suggestionCount: 0,
798+
resume: {
799+
allowed: true,
800+
from: 2,
801+
planDigest: 'deadbeef',
802+
alternateFrom: 3,
803+
repairSessionHeld: true,
804+
},
805+
repairHint: 'caution',
806+
},
807+
});
808+
assert.ok(report);
809+
assert.match(
810+
report!,
811+
/if you performed the step's intent as a recorded action: replay --from 3 --plan-digest deadbeef\./,
812+
);
813+
assert.match(report!, REPAIR_DIAGNOSTICS_CLAUSE_PATTERN);
814+
});
815+
816+
test('formatReplayDivergenceReport OMITS the diagnostics clause for caution when repairSessionHeld is absent', async () => {
817+
const { formatReplayDivergenceReport } = await import('../divergence.ts');
818+
const report = formatReplayDivergenceReport({
819+
divergence: {
820+
version: 1,
821+
kind: 'identity-mismatch',
822+
step: { index: 2, source: { path: '/tmp/flow.ad', line: 3 } },
823+
action: 'click label="Save"',
824+
cause: { code: 'IDENTITY_MISMATCH', message: 'resolved a different element' },
825+
screen: { state: 'available', refsGeneration: 1, refs: [{ ref: 'e1', role: 'button' }] },
826+
suggestions: [],
827+
suggestionCount: 0,
828+
resume: { allowed: true, from: 2, planDigest: 'deadbeef', alternateFrom: 3 },
829+
repairHint: 'caution',
830+
},
831+
});
832+
assert.ok(report);
833+
assert.doesNotMatch(report!, /Read-only inspection/);
834+
});
835+
836+
test('formatReplayDivergenceReport appends the diagnostics clause for manual when armed, even when resume is NOT allowed', async () => {
837+
const { formatReplayDivergenceReport } = await import('../divergence.ts');
838+
// `repairSessionHeld` reports the daemon KEPT THE SESSION LIVE, independent
839+
// of `resume.allowed` (plan-resumability) — the diagnostics clause must
840+
// still render here: the agent may still inspect the (held) session while
841+
// deciding on a manual repair, and any such read is still recorded.
842+
const report = formatReplayDivergenceReport({
843+
divergence: {
844+
version: 1,
845+
kind: 'action-failure',
846+
step: { index: 5, source: { path: '/tmp/flow.ad', line: 9 } },
847+
action: 'click label="Confirm"',
848+
cause: { code: 'COMMAND_FAILED', message: 'not hittable' },
849+
screen: { state: 'available', refsGeneration: 1, refs: [{ ref: 'e1', role: 'button' }] },
850+
suggestions: [],
851+
suggestionCount: 0,
852+
resume: {
853+
allowed: false,
854+
from: 5,
855+
planDigest: 'cafef00d',
856+
reason: 'step 4 is inside runtime control flow (retry); skipping it cannot be proven safe.',
857+
repairSessionHeld: true,
858+
},
859+
repairHint: 'manual',
860+
},
861+
});
862+
assert.ok(report);
863+
assert.match(report!, /cannot currently be resumed automatically/);
864+
assert.match(report!, REPAIR_DIAGNOSTICS_CLAUSE_PATTERN);
865+
});
866+
867+
test('formatReplayDivergenceReport OMITS the diagnostics clause for manual when repairSessionHeld is absent', async () => {
868+
const { formatReplayDivergenceReport } = await import('../divergence.ts');
869+
const report = formatReplayDivergenceReport({
870+
divergence: {
871+
version: 1,
872+
kind: 'action-failure',
873+
step: { index: 5, source: { path: '/tmp/flow.ad', line: 9 } },
874+
action: 'click label="Confirm"',
875+
cause: { code: 'COMMAND_FAILED', message: 'not hittable' },
876+
screen: { state: 'available', refsGeneration: 1, refs: [{ ref: 'e1', role: 'button' }] },
877+
suggestions: [],
878+
suggestionCount: 0,
879+
resume: { allowed: true, from: 5, planDigest: 'cafef00d', alternateFrom: 6 },
880+
repairHint: 'manual',
881+
},
882+
});
883+
assert.ok(report);
884+
assert.doesNotMatch(report!, /Read-only inspection/);
885+
});

0 commit comments

Comments
 (0)