Repository navigation
Preserve recording overlay coordinate references before the first snapshot #4448
Description
Current execution status — 2026-11-13
Open; adopted fourth-wave scope. Renderer-refusal source is reviewed, but rendered-frame execution and the native producer/canvas basis remain incomplete. Planning-viewport projection is rejected; the positive no-snapshot route still needs proof.
Latest detailed Current state.
Related: #4447/#4428 and audit umbrella #4368. Adopted into the authorized fourth-wave continuation through the existing issue; no duplicate implementation.
Observed trigger
On candidate 2640216c2eff655a3374130f4451b8e383096736, owned iPhone18ProMax/iOS27.0 (1C2C9FD4-B720-4160-BAEB-7D4724D7B5F3), a fresh session ran open Settings without a foreground/snapshot capture, then record start ... --scope device, swipe 220 700 220 300 --count 3 --pause-ms 500, and record stop. The resulting1320x2868/30fps video contains three timed swipe markers. The telemetry has x=220 and y=700→300, but no referenceWidth/referenceHeight; rendered markers appear at x≈220video pixels instead of scaling point-space input to the3x canvas. Settings visibly scrolls. Cleanup completed.
The timing candidate changes no coordinate-reference or Swift-scaling policy. Source on the named base also has the same snapshot-only fallback and Swift1.0 scaling for absent dimensions; no separate baseline live run is claimed.
Existing owner and prevention contract
packages/daemon/src/core/recording-gestures.ts:resolveEventReferenceFrame uses explicit result dimensions or getSnapshotReferenceFrame(session.snapshot). If both are absent, the event lacks a coordinate reference. apple/runner/AgentDeviceRunner/RecordingScripts/recording-overlay.swift:overlayPoint/scaledAxis treats missing reference size as1.0, silently interpreting point coordinates as video pixels.
Use the existing admitted touch/viewport coordinate owner to construct unambiguous recording telemetry for valid commands with no prior snapshot. Define coordinate-space/reference dimensions in the owning type and construction boundary. Do not infer the space from a magic3x ratio or invent a second device-size cache. For truly unknown coordinates, choose an explicit truthful refusal/omission policy rather than a plausible wrong overlay. Preserve the documented no-snapshot command flow; do not require an unrelated screenshot merely to repair hidden recording state.
Completion evidence
Exercise actual action→recording construction before any snapshot, at point/pixel scaling factors including1x/3x, and verify burned-in marker position from rendered frames against native touch coordinates. Include existing snapshot-backed and explicit-reference controls, rotation/scoped target behavior as applicable, and independent app/video evidence. Old producer or a missing-reference mutant must fail. Require current overlap check, independent review, full format, exact-head validation and applicable CI. Keep #4050 and budgets; no blanket test deletion, fallback framework, merge/release/tags.
This issue needs source/compatibility triage before implementation. The separate repeated-timing regression remains #4428; this report does not claim a new timing regression or a ready fix.
Activity
apex-sentinel-staging commentedon Nov 11, 2026
Apex Sentinel · Issue review
Suggested label:
bugThe issue reports incorrect existing behavior where recording telemetry lacks reference dimensions before the first snapshot and the Swift overlay scales missing references as 1.0, causing point-space coordinates to be rendered as video pixels. It includes a concrete repro command sequence, version/commit, and source owners, making it actionable.
Required sections
thymikee commentedon Nov 12, 2026
Decision: fourth-wave adoption: This live-discovered issue is now one of the next ten authorized scopes under #4368 (comment). Reuse the existing coordinate construction owner and this issue. Source/compatibility triage and an explicit unknown-reference policy precede implementation. Preserve no-snapshot commands; do not infer a magic display scale or add another viewport cache. The required regression and rendered live evidence remain part of completion.
thymikee commentedon Nov 13, 2026
Decision: recording coordinate basis and unknown-marker policy:
Adopt the following finite policy for #4448 at baseline 80075b6 after the source trace and independent review. A gesture planning viewport is a movement constraint, not proof of the recording canvas. Do not derive a recording basis from its width/height or x+width/y+height; keyboard clipping, window placement and scoped recording invalidate that shortcut.
A marker reference must come from the existing platform producer that actually knows the executed coordinates' space. Keep the planning rect distinct. An additive same-operation basis must name origin, units/extent and orientation/scope sufficiently to establish the transform into the captured canvas. Propagate it through the existing gesture runtime/result and recording construction; do not guess from display scale, take an unrelated snapshot, create another cache, or remove the Apple adapter's intentional coordinate stripping. Where that same-space/canvas relationship is not proven, do not attach a plausible reference. Existing explicit/snapshot-backed paths require their existing validity guarantees; an old snapshot is not authority merely because it exists.
When an event lacks a finite, positive paired reference, omit only its rendered marker/layers. Preserve action success, the video and the telemetry event. Remove Swift's missing-reference scale=1 fallback on all marker paths. Do not make optional released telemetry fields required or silently change public command responses. This omission policy is intentional observable behavior for unknown geometry.
The first source slice implements that renderer refusal and authors its direct proof cases. Full #4448 completion still requires a producer-owned reference on the supported no-snapshot flow, exact producer/wire tracing, scope/rotation and keyboard controls, request-path regression/mutation proof and actual native-touch versus rendered-frame evidence at 1x/3x. A renderer-only patch is incomplete. The fabricated 1000x1000 scroll basis remains a separate recorded umbrella residual; this fix does not validate or expand it. No coding against unproven viewport projection is authorized. Keep #4447 repetition timing, #4050 fallback and unrelated PRs intact.
thymikee commentedon Nov 13, 2026
Current state: This is an adopted fourth-wave scope with one canonical implementation record. The renderer-refusal component is local at
ac2d6982244625406fd4d198e38376c8a486f917(renderer sourceeb2f4bd8ae9f2103e0de5c497b796347d4154998plus documentation correction). Independent source review found no P1/P2 in that bounded component. Rendered-frame cases are authored but remain unexecuted; no exact-gate or native coordinate proof is claimed.The producer/canvas component is still incomplete. The latest source-grounded plan identifies native display geometry, but the daemon-to-recording propagation and its relationship to the captured canvas still need proof. A separately named basis must survive the planned-gesture response boundary before the daemon can consume it; adding a field before a fixed whitelist and assuming it survives is insufficient.
Next: measure the native producer basis against an actual captured canvas in the relevant orientations, complete the private basis propagation and exercise the positive no-snapshot route plus unknown-basis omission. Then run rendered-pixel, request-path, isolated regression, exact-head and live evidence. Planning-viewport projection remains rejected. Missing basis omits the marker while retaining the action, telemetry and video. The fabricated 1000×1000 scroll reference remains a separate residual.
No published PR, readiness or whole-issue completion. No new cache, guessed scale or weakened visual assertion.