|
| 1 | +# iOS XCTest fixture reuse |
| 2 | + |
| 3 | +## Scope |
| 4 | + |
| 5 | +The base is `b5f47e7542e274fc84a193c4e111e6c73ac3b1f3`. |
| 6 | +The test-source commit is `cefec4dd1ffe62537991f124ac8d26e94fa5505a`. |
| 7 | +The change groups ten native methods into three methods. |
| 8 | +It changes no production runner code, deadline, retry, or workflow selection policy. |
| 9 | + |
| 10 | +| Retained method | Preserved contracts | |
| 11 | +| --- | --- | |
| 12 | +| `testAlertActivationConfirmsBeforeDeadlineWithoutHandlingTheNotificationBanner` | A real notification banner exists. Activation consults no interruption monitor. The response succeeds before its deadline. The original action occurs once. The replacement action never occurs. | |
| 13 | +| `testSkipAppActivationPreflightRequiresCachedForegroundCoordinateOnlyTap` | Missing cached targets refuse tap and scroll. A foreground cached coordinate tap bypasses activation. Selector tap, drag, scroll and mixed or coordinate gesture sequences retain activation. A changed bundle refuses the bypass. A terminated cached target refuses the bypass. | |
| 14 | +| `testRunnerWindowResolutionReadsOnlyThroughTheFirstUsableWindow` | Two real windows exist before the frame probe is installed. The first usable window costs one frame read. An empty first window costs two reads. All empty windows produce no selected window and a zero app-frame fallback after three reads. The probe resets between rows. | |
| 15 | + |
| 16 | +The activation predicate runs its destructive termination input last. |
| 17 | +Its former “background” method used `terminate()`. |
| 18 | +The retained method names that state accurately. |
| 19 | +The retained banner test also observes the interruption opt-out during native dispatch. |
| 20 | +It uses the existing synthesis stub for a nonperforming probe. |
| 21 | +It restores the original implementation before the real activation. |
| 22 | +The process-wide frame implementation is restored with `defer`. |
| 23 | +Each method terminates its fixture and clears the state that it owns. |
| 24 | + |
| 25 | +## Selection |
| 26 | + |
| 27 | +Source-derived PR selection changes from 125 to 118 methods. |
| 28 | +All 115 unaffected identifiers remain. |
| 29 | +The ten superseded identifiers are absent. |
| 30 | +All three retained identifiers are present. |
| 31 | +No declaration becomes unreachable. |
| 32 | +The production `RunnerTests/testCommand` entrypoint remains excluded. |
| 33 | +The macOS host selection remains 338 methods. |
| 34 | + |
| 35 | +## Local measurement |
| 36 | + |
| 37 | +The runs use Xcode 26.2, iOS 26.2 and a dedicated iPhone 17 Pro simulator. |
| 38 | +The simulator UDID is `DB573A6D-FA35-4516-B5BB-478D8EFC8C5F`. |
| 39 | +Dependencies use pnpm 11.17.0 with the frozen lockfile. |
| 40 | +Each checkout is installed and built separately. |
| 41 | +Native builds include `AGENT_DEVICE_XCUITEST_INCLUDE_UNIT_TESTS=1`. |
| 42 | +All test invocations use `test-without-building` and disable parallel testing. |
| 43 | +The base and candidate run in sequence on the same simulator. |
| 44 | +No broad gate runs during the native experiments. |
| 45 | + |
| 46 | +| Sample | Executed / passed / failed / skipped | Sum of method durations | |
| 47 | +| --- | --- | ---: | |
| 48 | +| First base run | 10 / 10 / 0 / 0 | 58.044 s | |
| 49 | +| Warm base run | 10 / 10 / 0 / 0 | 38.213 s | |
| 50 | +| Warm candidate run with the strengthened interruption oracle | 3 / 3 / 0 / 0 | 18.523 s | |
| 51 | + |
| 52 | +The warm comparison reduces method time by 19.690 s, or 51.5%. |
| 53 | + |
| 54 | +| Contract group | Warm base | Warm candidate | |
| 55 | +| --- | ---: | ---: | |
| 56 | +| Banner activation | 13.813 s | 11.450 s | |
| 57 | +| Window resolution | 12.425 s | 4.089 s | |
| 58 | +| Activation preflight | 11.975 s | 2.984 s | |
| 59 | + |
| 60 | +Observed fixture launches change from nine to three. |
| 61 | +The first base run includes fresh simulator and notification setup. |
| 62 | +Use the warm comparison for the headline result. |
| 63 | +These are local focused measurements. |
| 64 | +They do not predict the complete hosted Apple job duration. |
| 65 | +Build, simulator boot and Xcode invocation overhead are outside the method sum. |
| 66 | + |
| 67 | +The command shape is: |
| 68 | + |
| 69 | +```sh |
| 70 | +DEVELOPER_DIR=/Applications/Xcode-26.2.0.app/Contents/Developer \ |
| 71 | +xcodebuild test-without-building \ |
| 72 | + -xctestrun "$XCTESTRUN_PATH" \ |
| 73 | + -destination "platform=iOS Simulator,id=$TEST_SIMULATOR_UDID" \ |
| 74 | + -parallel-testing-enabled NO \ |
| 75 | + -resultBundlePath "$RESULT_BUNDLE" \ |
| 76 | + -only-testing:AgentDeviceRunnerUITests/RunnerTests/testAlertActivationConfirmsBeforeDeadlineWithoutHandlingTheNotificationBanner \ |
| 77 | + -only-testing:AgentDeviceRunnerUITests/RunnerTests/testSkipAppActivationPreflightRequiresCachedForegroundCoordinateOnlyTap \ |
| 78 | + -only-testing:AgentDeviceRunnerUITests/RunnerTests/testRunnerWindowResolutionReadsOnlyThroughTheFirstUsableWindow |
| 79 | +``` |
| 80 | + |
| 81 | +The base uses the ten original identifiers from the diff. |
| 82 | +Each log is checked for the exact expected identifier set. |
| 83 | +Count equality alone does not establish discovery. |
| 84 | + |
| 85 | +## Interruption oracle |
| 86 | + |
| 87 | +At the first test-source commit, `cb5360a045c1f8b4061bc01750e5bc6c22d264ed`, |
| 88 | +a mutation that enabled interruption handling passed. |
| 89 | +The empty callback list did not establish that the opt-out flag was set. |
| 90 | +That survivor was not accepted as detection evidence. |
| 91 | +The current test observes the flag inside the existing native synthesis stub. |
| 92 | +It still requires the real banner, monitor, successful activation, deadline and action counts. |
| 93 | +The current timing includes that additional probe. |
| 94 | + |
| 95 | +An initial deliberately failing run started a simulator diagnostic collection with a 600-second timeout. |
| 96 | +That collection was interrupted after the assertion failures were recorded. |
| 97 | +That interrupted run is excluded from mutation completion counts. |
| 98 | +Subsequent mutation invocations add `-collect-test-diagnostics never`. |
| 99 | +They retain test logs, result bundles, exact identifier checks and failure exit status. |
| 100 | +CI diagnostics remain unchanged. |
| 101 | + |
| 102 | +## Mutation proof |
| 103 | + |
| 104 | +The disposable checkout starts at the test-source commit named above. |
| 105 | +The superseded methods are absent from that checkout. |
| 106 | +The unmodified retained suite passes all three selected methods. |
| 107 | +Each production mutation is then built and tested separately. |
| 108 | +Each mutant runs only its relevant retained method. |
| 109 | +All nine builds succeed. |
| 110 | +Each test run reports zero passed, one failed and zero skipped methods, with exit 65. |
| 111 | +The source edit is restored before the next mutation. |
| 112 | + |
| 113 | +| Production mutation | Intended retained assertion | |
| 114 | +| --- | --- | |
| 115 | +| Return no activation outcome for alert accept | Native dispatch observation is absent. The real response fails. The original action count remains zero. | |
| 116 | +| Set `doesNotHandleUIInterruptions` to false | The native dispatch probe observes false instead of true. | |
| 117 | +| Read past all windows and return only the app-frame fallback | First usable-window selection and one-read witness fail. | |
| 118 | +| Accept an empty window frame | Empty-first and all-empty controls fail their read count, frame or window-presence assertions. | |
| 119 | +| Reject every command after the cached-target guard | The foreground cached coordinate-tap positive fails. | |
| 120 | +| Accept every command after the cached-target guard | Drag, scroll and gesture-sequence negatives fail. | |
| 121 | +| Remove the cached-target guard | Missing, changed-bundle and terminated-target negatives fail. | |
| 122 | +| Accept every requested bundle after the foreground check | The changed-bundle negative fails. | |
| 123 | +| Remove the foreground-state check | The terminated cached-target negative fails. | |
| 124 | + |
| 125 | +The implementation checkout contains none of these production edits. |
| 126 | +The disposable checkout is clean after the experiments. |
| 127 | +The raw logs, mutation patches and result bundles are retained locally. |
| 128 | +This report records completed runs, not predicted detection. |
| 129 | + |
| 130 | +## Publication obligations |
| 131 | + |
| 132 | +The local measurements do not establish a hosted speedup. |
| 133 | +Apple smoke, Integration Tests and Coverage remain CI obligations at the PR head. |
| 134 | +The source-derived selection check and packaged-runner check remain required local gates. |
0 commit comments