Skip to content

fix(apple): use typed screenshot timeouts and observed shutdown recovery - #4544

Draft
thymikee wants to merge 7 commits into
mainfrom
t3/audit-4442-test-gates
Draft

thymikee wants to merge 7 commits into
mainfrom
t3/audit-4442-test-gates

Conversation

@thymikee

@thymikee thymikee commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

Summary

Fixes #4442. Replace screenshot-error prose matching with a timeout reason stamped by the local simctl adapter. A fresh JSON state read, capped at 5 seconds, permits one boot/re-capture only for literal Shutdown.

Changes 15 files. Scope expansion: isolate process-identity test fixtures from live Linux /proc, fixing the matching unchanged-base Coverage failure without changing assertions or production code.

Custom command providers lose prose-derived code-60 retries. Eligible capture failures may add a probe. Existing boot-memo and capture/boot signal limitations remain.

Validation

Base d78be6c88715ad80fc658497f23f91c86bf9489b; tested head 50034fe3afeaba5893090fdc4f060586dda5fb9f.

  • Pinned pnpm 11.17.0 exact affected gate passed: exactCommit=true, checkoutChanged=false, failOpen=false. Eight checks passed; unit 19,824 passed/10 skipped, related 377 passed. Full-unit fallback retained. Provider integration deduplicated; four coverage/replay checks skipped.
  • Independent small-delta source review: no findings, with unread policy/evidence inputs disclosed. Screenshot regression evidence retains original SHA attribution.
  • Simulated Linux fixture boundary: 21 passed; removing the isolation caused the exact PID-202 assertion failure (1 failed/20 passed); restored control 21 passed. Dirty-tree model evidence; real Linux CI remains required. Earlier 855a Coverage failure stays failed.
  • Native blocker: candidate-bound ordinary screenshot passed; Shutdown recovery and real code-60 header unproven. Draft remains blocked. External feat(screenshot): --native-density keeps the display's own pixels #4111 overlaps these paths. Historical 600d timeout cause remains unknown.

…oot on observed state

The screenshot retry, the runner fallback and the skipped-boot-check recovery each decided
from the tool's prose. `screenshot.ts` matched "timeout waiting for screen surfaces", a
POSIX-domain string pair, and six spellings of a shutdown state, so any CoreSimulator wording
change could silently retire all three decisions at once.

`tool-provider.ts` is the one adapter that runs `xcrun simctl` itself, so it is the one place
allowed to read simctl's error protocol. It takes the top-level header's `domain=
NSPOSIXErrorDomain, code=60` and stamps `details.reason`; a nested `Underlying error` line,
another domain or code, stdout, and a caller-supplied command or simctl callback stay
unattributed. The reason is stamped, never host-kit's `timeoutMs`, which is that layer's own
evidence that it killed a child at its deadline.

Recovery now asks the tool what state the simulator is in. `simulator.ts` owns the one
boot-state read and takes the caller's ceiling, so the probe runs under a 5s screenshot-owned
budget instead of the 60s inventory budget. Only the literal `Shutdown` authorizes a boot:
`Booted`, `Booting`, `Creating` and `Shutting Down` describe a simulator that is already on its
way, and a listing that failed, omits the simulator, or names no state proves nothing. A
canceled request is terminal and never reaches the probe; a probe that is itself canceled
travels on; every other probe failure answers `false` and leaves the capture's own error,
details and hint, as the reported failure. The boot memo policy is unchanged, so a simulator
turned off outside this process is still not seen until the memo expires.

Refs #4442
@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Size Report

Metric Base Current Diff
Installed (including dependencies) 6.28 MB 6.28 MB +514 B
Package (unpacked) 6.28 MB 6.28 MB +514 B
Package (download) 1.90 MB 1.90 MB +291 B

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 21.9 ms 23.2 ms +1.3 ms
CLI --help 43.0 ms 43.1 ms +0.1 ms

@thymikee

Copy link
Copy Markdown
Member Author

Coverage is base-blocked at 855aa4a, before coverage thresholds: host-process-liveness.test.ts:182 fails in the fleet identity case. Expected fabricated PID 202 to have the fixture's start time and zombie:true; Linux returned a real /proc/202 identity and zombie:false.

The same test, location and assertion already fail in unchanged-main CI at d78be6c8, with the same Node 24.13 and coverage command. The source and test are byte-identical between that base and this head. This PR's failed job is retained as failed.

The fixture mocks ps but leaves /proc live, so a real process can replace its synthetic facts. I accept ownership of a bounded fixture-isolation repair, retaining all assertions and the dedicated Linux /proc tests. The patch is prepared but unrun; #4426 currently owns local execution. It will need focused proof and a new exact-head gate before a push, followed by Linux CI. No CI retry, test deletion or timeout change was made. Native recovery and real code-60 evidence remain separate draft blockers.

@thymikee

Copy link
Copy Markdown
Member Author

Pushed the fixture correction as 50034fe. All existing identity, zombie, command, ordering and call-count assertions remain; the dedicated Linux /proc tests retain their own overrides.

Regression proof in an isolated simulated Linux process/filesystem boundary, with the original assertions unchanged: corrected fixture 21 passed; removing only the seven-line isolation 1 failed / 20 passed, at the exact PID-202 start-time/zombie assertion from the failed Coverage jobs; restored fixture 21 passed. The harness was removed and source hashes restored. This models the boundary; it is not a Linux-host run.

One pinned pnpm 11.17.0 exact affected gate passed at 5003: exactCommit=true, checkoutChanged=false, failOpen=false. Eight checks passed, including 19,824 unit tests / 10 skipped and 377 related tests. Provider integration was deduplicated; coverage and replay checks were skipped, not passed. Independent review found no small-delta source findings, with unread policy/evidence inputs disclosed.

Real Linux CI is now running on the pushed fix. The original 855a Coverage job remains failed; this was a source correction, not a retry of unchanged CI. The PR stays draft: native Shutdown recovery and the real code-60 header remain unproven.

@thymikee

Copy link
Copy Markdown
Member Author

Thanks for the typed timeout work. At 855aa4a I found one likely regression in the new retry rule, and the live evidence the PR body says is missing is still missing.

SIMCTL_ERROR_HEADER only matches at the start of stderr, and a test fixes "header after a diagnostic preamble" as not recognized. The failure this retry exists for (#167, still visible in main's screenshot-simulator-capture.test.ts) has simctl print Detected file type from extension: PNG on stderr before the An error was encountered processing the command (domain=NSPOSIXErrorDomain, code=60) header. If that ordering is real, no reason is stamped, shouldRetryIosSimulatorScreenshot returns false, and the capture fails on the first attempt with no runner fallback. Before this PR the same failure retried and then fell back to the runner. The PR moved the PNG line to stdout in the existing fixtures, so no test covers the stderr ordering. Could the producer recognize the top-level header as an unindented line anywhere in stderr (a multiline anchor), while still excluding indented Underlying error lines? Please add a protocol test with the preamble ordering on stderr. I could not confirm real simctl output, so if it writes that line to stdout this does not apply.

This change rewires native screenshot retry, runner fallback and skipped-boot recovery on the local simulator route, and every test feeds hand-written stderr. If the typed reason never fires on real hardware, the retries disappear and the tests still pass. Please run once on an owned simulator. First, capture with skipIosSimulatorBootCheck against a simulator that was shut down after its boot memo was written, and show a probe returning Shutdown, one ensureBooted, one re-capture and a written PNG. Second, attach the raw stderr of a real xcrun simctl io <udid> screenshot failure (ideally code=60) with the stamped details.reason, or the ios_simulator_screenshot retry diagnostics with attempt greater than 1. That also settles the header position question above.

Not blocking, take or leave: shouldRetryIosSimulatorScreenshot is also the runner fallback predicate and runs on errors from captureAndNormalize, so any exec timeout in density normalization (such as the scale probe) now selects the runner fallback; scoping it to the capture step would keep the old behavior. The doc blocks in tool-provider.ts could each shrink to one line, and the runLocalSimctlCommand link names SIMCTL_SCREENSHOT_TIMEOUT_CODE where it means SIMCTL_SCREENSHOT_TIMEOUT_REASON. The four new test files add about 1,000 lines, so folding the retry-policy cases into the protocol suite with one negative table would keep it smaller.

The Coverage failure looks unrelated: the diff only touches packages/platform-apple, and the failing test named in the PR is a host-kit Linux /proc fixture. I could not see the failing assertion in the log, so that is not confirmed. No conflicts. Before merge, the header match needs to handle a stderr preamble, the simulator evidence above needs to be attached, and Coverage needs to pass once the host-kit fixture is repaired on main.

This review covers 855aa4a. The newer head 50034fe only adds a host-kit liveness test fixture, which does not touch the findings above and may address the Coverage failure.

@thymikee

Copy link
Copy Markdown
Member Author

I own #4544 and am addressing this review.

The preamble finding is accepted. Local commit cec1726bb03383a3c678d8ea7313713280a5412e recognizes the first unindented protocol header anywhere on stderr. The new positive cases put Detected file type from extension: PNG before the timeout header, with LF and CRLF. Indented/nested/stdout-only headers remain negative; an earlier unrelated top-level error is not skipped to select a later timeout. The comment blocks and incorrect reason link are also corrected. Regression discrimination and the new exact gate are assigned; this commit is not pushed yet.

The native requests remain open. The earlier ordinary Settings screenshot did not prove recovery or a real timeout header. I am checking a deterministic supported T3-bound route before another owned-simulator attempt, including whether a running batch can retain the session across T3 shutdown without admission booting first. I will attach actual command/probe/capture evidence and raw adapter stderr if reachable, and will not substitute scripted stderr for it.

The separate Linux process-fixture correction is already pushed at 50034fe3, with its passing exact gate and isolated regression evidence in the previous comment. The size-report edit is informational. The PR remains a blocked draft, with no merge planned.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Use typed Apple screenshot recovery reasons

1 participant