Skip to content

fix(serve-sim): recover capture after framebuffer loss - #207

Merged
gwdp merged 1 commit into
mainfrom
gwdp/capture-self-heal
Oct 8, 2026
Merged

gwdp merged 1 commit into
mainfrom
gwdp/capture-self-heal

Conversation

@gwdp

@gwdp gwdp commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

What

Recover capture after framebuffer loss, with retries, loss logs, and capture stats. Read live surfaces once per second independently of callback-driven picks; callbacks use their supplied surfaces immediately. Clear stale seeds and canvas geometry when surfaces change.

Why

A preview stopped producing frames. The customer's exact trigger remains unknown. Rebooting a simulator exposed the missing recovery path: wiring was retried only before the first frame. Independent live-read timing also avoids repeated private API calls during compositor swaps.

Authorship and LLM use

  • LLM involvement: Claude generated the initial code/tests. Codex revised code, tests, LLP documentation, and this description.
  • Process and decisions: Gabe directed the investigation and approved the fixes, including szdziedzic's request to rate-limit live reads.
  • Iterations: Codex and subagent reviews led to surface/seed invalidation, masked fallback, correct recovery timing, resize notifications, and pixel regressions. The human review then prompted independent live-read timing and callback precedence. Three Codex subagents reviewed that follow-up without remaining findings.
  • Human involvement: Gabe reviewed f5f10e81 and approved this follow-up's direction. Codex ran the checks below.
  • Review status: Human review of the follow-up diff is pending. Gabe owns maintenance.

Evidence

Codex checked 4823a7eb, one commit on main dcc46a59:

  • Serve-sim lint and typecheck passed. bun run check still fails on knip findings unchanged from the pre-follow-up baseline.
  • Root build/typecheck: n/a; confined to serve-sim. Serve-sim bun run build passed.
  • Isolated bun run test: 2,097 pass, 158 skip, 0 fail; Bun 1.3.14.
  • Owned iPhone 17 Pro, iOS 26.4 / Xcode 26.4: reboot recovery plus tap, drag, scroll, 3 pass. Owned iPhone Duo, iOS 27.1 / Xcode 27.1 beta: reboot and same-process capture/HID recovery, 4 pass. These are local Simulator runs.
  • Failing tests first: original reboot regression failed on trunk; new cadence/callback regressions failed before the fixes.
  • Native checks:
swift test --filter 'StreamingPolicyTests|SimNativeTests'
# 174 XCTest + 68 Swift Testing cases passed; 0 failures.
  • Hosted EAS web preview: selected the exact-head artifact with --package-version on iPhone Duo / iOS 27.1. Fresh WebRTC pixels and UIKit input passed; static-screen and eight fold switches across 2D and fixed-panel 3D feeds reported zero surface losses or rewires. Managed reboot recovered video/input; that path restarts capture, so direct capture self-heal remains covered by the local and CI regressions.

bun run llp:check: 10 docs, 20 refs, 0 errors. Commands, device IDs, timing samples, hosted evidence, and CI diagnosis. Exact-head EAS tests and check passed. Android gRPC duration CI failed before recording validation because Hub's selected port was already in use; its test/CLI paths are unchanged by this PR.

Rollback

Plain revert.

— Codex

@changeset-bot

changeset-bot Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 4823a7e

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
@expo/serve-sim Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@gwdp
gwdp force-pushed the gwdp/capture-self-heal branch from f65bf60 to 7bd5409 Compare October 1, 2026 07:37
@gwdp

gwdp commented Oct 1, 2026

Copy link
Copy Markdown
Contributor Author

Shortened the changeset and amended the existing commit to replace the Claude co-author trailer with Codex, at Gabe’s request.

— Codex

@gwdp
gwdp force-pushed the gwdp/capture-self-heal branch from 7bd5409 to 07623e7 Compare October 1, 2026 07:53
@gwdp

gwdp commented Oct 1, 2026

Copy link
Copy Markdown
Contributor Author

Codex contribution on 07623e7: removed or shortened redundant implementation and test comments. Executable lines are unchanged; the existing commit was amended at Gabe’s request.

— Codex

@gwdp

This comment has been minimized.

@devin-ai-integration

This comment has been minimized.

devin-ai-integration[bot]

This comment was marked as resolved.

@gwdp
gwdp force-pushed the gwdp/capture-self-heal branch from 07623e7 to 98c04b2 Compare October 1, 2026 08:13
@gwdp

This comment has been minimized.

@devin-ai-integration

This comment has been minimized.

@gwdp
gwdp requested a review from szdziedzic October 1, 2026 16:34
@gwdp
gwdp marked this pull request as ready for review October 1, 2026 16:34
@gwdp
gwdp force-pushed the gwdp/capture-self-heal branch 2 times, most recently from 14797e9 to 9654892 Compare October 2, 2026 20:02
@linear-code

linear-code Bot commented Oct 3, 2026

Copy link
Copy Markdown

ENG-27793

@gwdp

gwdp commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor Author

Codex follow-up in 8836374a (now 6fcbc4d7 after rebasing onto main b2f52471): prevent slow healthy capture from triggering a false re-wire, notify screen observers when recovery changes dimensions, and declare sips as a system binary for knip.

Both new native regressions failed on 9654892f before the fixes. Three Codex subagents rereviewed correctness, readability/design, and tests/contracts with no remaining substantive findings. Gabe directed and authorized this follow-up; human review of these new changes is pending.

Codex ran:

  • swift test --filter 'StreamingPolicyTests|SimNativeTests': 164 XCTest and 68 Swift Testing cases passed.
  • bun run build and bun run typecheck: passed. Final isolated bun run test with pinned Bun 1.3.14: 2,097 pass, 158 skip, 0 fail.
  • capture-reboot-recovery.e2e.test.ts: 2 pass on the review-created iPhone 17 Pro, iOS 26.4, Xcode 26.4, UDID 3EB3D2AC-2FC7-404E-860D-55482C2D528D.
  • Before rebasing, that suite plus duo-reboot-recovery.e2e.test.ts on 8836374a: 4 pass on the review-created iPhone Duo, iOS 27.1, Xcode 27.1 beta, UDID 157DC551-C1D1-42E9-95E6-4D35C84B0538. Verified fresh delivered pixels after direct reboot and capture/HID recovery through preview and network-capture reboots.

Build, typecheck, isolated tests, native tests, and the iPhone reboot suite were rerun after rebasing. git range-diff confirms both commit patches are unchanged; independent integration review found no regressions. bun run llp:check passed with 0 errors. Duo was not rerun after this rebase.

Device runs used the rebuilt local CLI and the pinned-device scripts/test/e2e.sh wrapper. Local lint/knip still have baseline failures; the new E2E's sips warning is resolved. An initial Bun 1.3.11 run hit proxy-test timeouts that also reproduced on base; the final pinned-version run passed. CI is rerunning.

Gabe-approved LLP follow-up in c690ec50: documented capture recovery and diagnostics in LLP 0001 and added one policy @ref. No executable code changed. A Codex subagent reviewed the applied diff with no findings; bun run llp:check passed (10 docs, 19 refs, 0 errors), and git diff --check passed. The local bun run check still reports the same 83 UI lint errors.

At Gabe’s request, the three PR commits are now one commit, f5f10e81, with the same parent base and an identical file tree to c690ec50. An independent Codex subagent confirmed preservation. bun run llp:check passed again (10 docs, 19 refs, 0 errors). Runtime suites were not repeated for this history-only change. The fresh-checkout bun run check could not start lint because oxlint was not installed there; earlier local check results above remain unchanged.

2026-10-08 reviewer follow-up

Codex implemented szdziedzic's live-read rate-limit request, added regressions, and updated LLP 0001. Gabe approved this direction after reviewing the earlier f5f10e81 diff; human review of this follow-up is pending. Three Codex subagents reviewed correctness, design/readability, and tests/contracts with no remaining findings. A final integration review found no interaction with main's recorder keyframe change.

Published head: 4823a7eb18b5c39550a754e498bc18200371c8f4, one commit on main dcc46a59055185eb5866b736e0f3560de02568aa. The final rebase preserved the capture patch; only LLP revision-date context needed resolution, and main's keyframe documentation remains intact.

Live reads now have an independent one-second cadence. Surface-change callbacks use their supplied cached surface immediately, even when a live read is overdue. Callback bursts cannot reset the deadline or repeatedly query the getter; wiring a new descriptor set performs one bootstrap live read. Cached nil between live observations cannot advance the recovery timer. The comment now describes these paths directly.

Before the production fix, getter-count and pixel regressions failed on the rebased pre-follow-up source: callback bursts made 26 reads instead of 1, and nil-cache bursts made 51 instead of 1. A separate overdue-callback regression then reproduced stale/dropped replacement pixels before that boundary was fixed. The final tests use a controlled clock for deadlines and check pixels, equal seeds, getter counts, and recovery counts.

Codex ran on the published head, with Bun 1.3.14:

  • Leaf bun run build: passed. Each device run started a fresh process with the rebuilt local CLI.
  • Parent bun run check: lint 0 warnings/errors and typecheck passed; knip exited 1. Its findings match the pre-follow-up rebased PR baseline: unused oxlint dependency, unlisted oxlint/tsc/knip binaries, unused captureReapersForTest export, and the sips configuration hint. No knip cleanup was added to this follow-up.
  • Parent bun run test: 2,097 pass, 158 skip, 0 fail.
  • Leaf swift test --filter 'StreamingPolicyTests|SimNativeTests': 174 XCTest + 68 Swift Testing cases passed, 0 failures.
  • Root bun run llp:check: 10 docs, 20 refs, 0 errors; git diff --check passed. LLP author references remain Gabe Debes.

Local device validation used only simulators created for this review:

# From packages/serve-sim, with pinned Bun 1.3.14 on PATH:
SERVE_SIM_TEST_UDID=07564F82-291A-496C-9B4A-40094C544482 \
bash packages/serve-sim/scripts/test/e2e.sh \
  packages/serve-sim/src/__tests__/capture-reboot-recovery.e2e.test.ts \
  packages/serve-sim/src/__tests__/hardware-keyboard-shift.e2e.test.ts \
  -t 'preconditions for capture recovery|captures again after the device reboots|tap, drag, and scroll still reach UIKit'
# iPhone 17 Pro, iOS 26.4 / Xcode 26.4: 3 pass, 10 filtered out, 0 fail.

DEVELOPER_DIR=/Applications/Xcode-27.1.0-Beta.app/Contents/Developer \
SERVE_SIM_TEST_UDID=BB26150B-17C3-4641-BC64-CF7454D8C4EF \
SERVE_SIM_DUO_REBOOT_E2E=1 \
bash packages/serve-sim/scripts/test/e2e.sh \
  packages/serve-sim/src/__tests__/capture-reboot-recovery.e2e.test.ts \
  packages/serve-sim/src/__tests__/duo-reboot-recovery.e2e.test.ts
# iPhone Duo, iOS 27.1 / Xcode 27.1 beta: 4 pass, 0 fail.

Those runs check fresh delivered pixels after reboot; Duo also checks capture and HID recovery in the same server PID after preview and network-capture reboots. The unaffected iPhone input test observes tap, drag, and scroll reaching UIKit. These are local Simulator checks.

Duo timing sample: four fold/unfold switches per build on the same owned device, with ~3 seconds per interval and an MJPEG viewer. Across the switch intervals, /webrtc/stats delta pickSumMs was 48.866 before versus 26.542 after (828 versus 822 picks; mean 0.059 versus 0.032 ms). Cumulative pickMaxMs, including startup, was 5.590 versus 3.131 ms. Both runs selected 1398×2034 / 2008×2854 panels and reported zero rewires. This is a small local sample, not a general performance benchmark. It ran before the final main rebase; the capture patch is unchanged by that rebase.

Previous-head CI diagnosis: the Android byte-cap failure also occurred on main and is left separate. EAS passed the reboot test on both attempts but timed out starting mitmproxy. This rebase includes the already-merged mitmproxy quarantine fix. Exact-head CI results are recorded below.

2026-10-08 hosted EAS validation

Codex downloaded the successful EAS workflow's artifact for exact head 4823a7eb18b5c39550a754e498bc18200371c8f4 and ran it in a real, isolated EAS web-preview-only session. Package: @expo/[email protected]; artifact 01a11cef-7a48-7e44-8b31-90ebc48b020d; SHA-256 035b6cb296b6765813929cb2e11f045f3fe451f4d8f42296a738408b3985ee3b. Source identity is established by workflow/artifact metadata and the selected URL; the remote API does not expose the running binary's checksum.

eas simulator:start --platform ios --type web-preview-only \
  --non-interactive --json --device 'iPhone Duo' \
  --name 'PR207 framebuffer recovery validation 20261008' \
  --tag pr207-validation --max-duration-minutes 30 \
  --package-version <signed exact-head artifact URL>

EAS CLI 24.12.0; iPhone Duo / iOS 27.1; remote UDID 74195B53-4DBC-4EAA-9991-F98E5BB86BA8. Session record. No loopback cloud adapters were used in this run.

  • A small native UIKit fixture changed a sequence/pixel marker every 0.5 seconds. Actual decoded WebRTC frames showed the marker; AX labels confirmed a tap and drag reaching UIKit.
  • Four open/closed switches in 2D kept the same connected WebRTC peer sending frames. The selected panels were 2007×2853 and 1398×2034. Four further switches in 3D kept both fixed-panel peers connected and sending frames.
  • With the fixture frozen, a 69-second stats interval added 315 idle frames, with zero additional losses, lost milliseconds, or rewires. All fold samples also reported zero losses and rewires.
  • Supported capture.reboot completed with exit 0. The server stayed at PID 2096; capture counters reset as expected. Fresh video returned, and the relaunched fixture had a new run marker D41201B4, advancing sequence, and TAPS 1 / DRAGS 1.

Boundary: this hosted reboot command closes DeviceSession before boot and creates new capture. It does not reproduce a missing surface under the original FrameCapture actor. Direct shutdown/boot recovery and the live-read/callback cadence regressions remain established by the local and CI tests above; positive hosted loss/rewire counters were not exercised. No hosted getter-rate measurement was available. The original customer's roughly 70-minute failure was not reproduced.

Codex stopped this session explicitly after validation. No PR source files changed during the hosted run.

Exact-head CI: EAS serve-sim tests, check, Android gRPC recording, and both scrcpy recording jobs passed. The gRPC duration job exited before recording verification with Port 42953 is already in use — pick another with --port. The test/Hub CLI paths are unchanged by this PR, and the matching jobs passed on this PR's parent and newer main. The log does not identify the process using that port; no CI retry or unrelated code change was made.

— Codex

@gwdp
gwdp force-pushed the gwdp/capture-self-heal branch 3 times, most recently from c690ec5 to f5f10e8 Compare October 7, 2026 20:12
Retry display wiring once a second before the first frame and when a
surface stays missing after capture has started. Read each descriptor's
live unmasked or masked surface on revalidation, and invalidate stale
seeds and canvas geometry when the surface changes.

Sample recovery timing before capture work so a slow healthy poll does
not trigger a rewire. Publish resize notifications from successful
capture rather than preassigning dimensions during wiring. Add loss logs
and surfaceLosses, surfaceLostMs, and rewires to capture stats.

Cover policy decisions, real IOSurface swaps, masked-only frames, failed
rewires, slow capture, resized recovery, and fresh pixels after reboot.
Declare sips for knip and document recovery in LLP 0001 with one @ref.

Co-Authored-By: Codex <[email protected]>
@gwdp
gwdp force-pushed the gwdp/capture-self-heal branch from f5f10e8 to 4823a7e Compare October 8, 2026 19:11
@gwdp
gwdp merged commit 19801ba into main Oct 8, 2026
6 of 7 checks passed
@gwdp
gwdp deleted the gwdp/capture-self-heal branch October 8, 2026 20:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants