Repository navigation
Closed
Description
Context
PR #1217 (typed direct Maestro engine, ADR 0015) ships with a small parser-regression fixture (scripts/maestro-conformance*, now running in the vitest unit project). That harness is deliberately narrow: 5 hand-authored flows compared at parsed-IR level against a hand-typed transcription of Maestro 2.5.1's command model. It proves parser self-consistency, not conformance — during #1217 development, four bug classes (percent rounding/rejection, target-swipe direction default, retry-cap semantics, settle-loop ordering) all slipped past it and were found by multi-hour live suite runs instead.
ADR 0015 commits to a real three-layer oracle as follow-up. This issue tracks it.
The three layers
- Parser normalization, generated from upstream. Drive the pinned Maestro JVM parser (
maestro-orchestraat tag v2.5.1 / a4c7c95f) over a corpus — starting with upstream's owne2e/demo_app/.maestro/*.yamlandmaestro-testresource flows — and emit normalized fixtures. Classify every flow: parses-identically / explicitly-rejected / MISMATCH. The mismatch list is the parity backlog (this would have caughtoptional,pasteText,assertTruemechanically). - Semantic vectors, generated by a JVM harness. A small Kotlin harness that calls the actual upstream functions (relative-point conversion,
ElementCoordinateUtil, swipe-endpoint math, retry clamping, settle constants) against the pinned checkout and dumps JSON vectors. Hand-derived vectors are explicitly out — hand transcription is the failure mode this replaces. - App-observable differential scenarios. A small set of device flows run through both real Maestro and agent-device, comparing outcomes (settle behavior, tap retries, warned-vs-failed classification for
optional). Opt-in/scheduled, not unit CI.
Wiring
- Layers 1–2 regenerate only on explicit upstream-pin updates; the checked-in generated fixtures verify deterministically in normal CI (same pattern as the layering-guard
node --testjob). No Java in normal CI. - Layer 3 runs scheduled (like perf-nightly), not per-PR.
- Verify the pinned upstream artifact SHA-256s at regeneration time (the current harness parses but never checks them).
Acceptance
- The four refactor: replace Maestro compat with typed direct engine #1217 bug classes above each have a failing-before/passing-after fixture.
- Every command in
support-matrix.tsis either fixture-backed or explicitly listed as unverified. - Documented intentional deviations (hierarchy-vs-screenshot animation wait, horizontal-swipe presets, condition-poll-as-stabilization, ambiguity strictness under
optional) are encoded as expected-divergence entries, not silent mismatches.
Metadata
Metadata
Assignees
Labels
No labels
Activity
feat: declare layer-3 divergences and schedule the differential
chore(ci): remove the temporary branch push trigger
test: replace the hand-typed Maestro fixture with a generated conform…