Skip to content

Commit d3a651c

Browse files
committed
refactor(ios): run the input-preparation regression in the shared test process
The process-scoped preparation flag forced its regression into its own xcodebuild invocation on the PR and nightly lanes, with an xcresult merge and a multi-invocation selection parser to keep the executed-count guarantee. A test-only reset on RunnerSynthesizedGesture makes the test self-contained instead, so it runs like every other runner unit test and the workflows, selection script and its tests return to main. Also drops the iOS-only guard (every synthesized route is already `#if os(iOS)` in Swift) and the lock (commands run on a serial queue), and names the preparation consistently in the log marker and test file.
1 parent fa0e7a0 commit d3a651c

7 files changed

Lines changed: 94 additions & 152 deletions

File tree

‎.github/workflows/ios.yml‎

Lines changed: 1 addition & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -149,16 +149,11 @@ jobs:
149149
run: |
150150
XCTESTRUN_PATH="$(find "$AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH/Build/Products" -maxdepth 1 -name '*.xctestrun' -print -quit)"
151151
test -n "$XCTESTRUN_PATH"
152-
# The preparation flag is process-scoped; isolate this ordering test
153-
# from other tests that may already have synthesized input.
154-
xcodebuild test-without-building \
155-
-xctestrun "$XCTESTRUN_PATH" \
156-
-destination "platform=iOS Simulator,id=${{ steps.ios-simulator.outputs.simulator-udid }}" \
157-
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testSynthesizedInputPreparationDoesNotDeliverContactsAndOrdersMixedRoutes
158152
xcodebuild test-without-building \
159153
-xctestrun "$XCTESTRUN_PATH" \
160154
-destination "platform=iOS Simulator,id=${{ steps.ios-simulator.outputs.simulator-udid }}" \
161155
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testSinglePointerFlingFallsBackToXCTestCoordinateDragWhenPrivateSynthesisFails \
156+
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testSynthesizedInputPreparationDoesNotDeliverContactsAndOrdersMixedRoutes \
162157
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testAlertDispatchResolvesItsOwnModalWithoutCoordinateTapRoutingProbe \
163158
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testAlertResolutionCannotBypassRequestedDeadline \
164159
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testTypeWithoutResolvedInputReturnsTypedFailureBeforeDispatchingText \

‎.github/workflows/xctest-nightly.yml‎

Lines changed: 6 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,9 @@ name: XCTest Nightly
22

33
# The full iOS runner XCTest suite (#1781 A7). The PR lane (ios.yml) names a hand-written
44
# subset of the target's methods in an `-only-testing:` list; everything outside that list ran
5-
# nowhere at all. This lane runs the test plan whole, except `testCommand` (the runner's
6-
# server entry point). The process-scoped input-preparation regression runs in isolation;
7-
# its result is merged with the remaining suite before checking the executed count.
5+
# nowhere at all. This lane drops the filter and runs the test plan whole, skipping only
6+
# `testCommand` — which is the runner's server entry point rather than a test (see the step
7+
# below).
88
#
99
# No count is quoted here on purpose. `pnpm check:xctest-selection` prints the live split
1010
# (declared / PR-selected / skipped / nightly-only) and is the only place those numbers are
@@ -117,21 +117,8 @@ jobs:
117117
runtime-version: ${{ env.IOS_RUNTIME_VERSION }}
118118
preferred-device-name: iPhone 17 Pro
119119

120-
# The preparation regression needs a fresh process before any other synthesized
121-
# input. Run it separately, then merge both bundles for the source-derived count.
122-
- name: Run isolated synthesized-input preparation regression
123-
run: |
124-
set -euo pipefail
125-
XCTESTRUN_PATH="$(find "$AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH/Build/Products" -maxdepth 1 -name '*.xctestrun' -print -quit)"
126-
test -n "$XCTESTRUN_PATH"
127-
mkdir -p "$(dirname "$RESULT_BUNDLE_PATH")"
128-
xcodebuild test-without-building \
129-
-xctestrun "$XCTESTRUN_PATH" \
130-
-destination "platform=iOS Simulator,id=${{ steps.ios-simulator.outputs.simulator-udid }}" \
131-
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testSynthesizedInputPreparationDoesNotDeliverContactsAndOrdersMixedRoutes \
132-
-resultBundlePath "${RESULT_BUNDLE_PATH%.xcresult}-isolated.xcresult"
133-
134-
# The remaining suite follows the xctestrun plan, excluding the isolated test.
120+
# Same command as ios.yml's targeted step minus every `-only-testing:` flag, so the
121+
# xctestrun's own test plan decides what runs — with one exception.
135122
#
136123
# `RunnerTests/testCommand` is not a test. It is the runner's server entry point: it
137124
# opens an NWListener and blocks in `XCTWaiter.wait(timeout: 24 * 60 * 60)` until a
@@ -156,16 +143,7 @@ jobs:
156143
-xctestrun "$XCTESTRUN_PATH" \
157144
-destination "platform=iOS Simulator,id=${{ steps.ios-simulator.outputs.simulator-udid }}" \
158145
-skip-testing:AgentDeviceRunnerUITests/RunnerTests/testCommand \
159-
-skip-testing:AgentDeviceRunnerUITests/RunnerTests/testSynthesizedInputPreparationDoesNotDeliverContactsAndOrdersMixedRoutes \
160-
-resultBundlePath "${RESULT_BUNDLE_PATH%.xcresult}-remaining.xcresult"
161-
162-
- name: Merge isolated and remaining XCTest results
163-
if: always()
164-
run: |
165-
set -euo pipefail
166-
xcrun xcresulttool merge --output-path "$RESULT_BUNDLE_PATH" \
167-
"${RESULT_BUNDLE_PATH%.xcresult}-isolated.xcresult" \
168-
"${RESULT_BUNDLE_PATH%.xcresult}-remaining.xcresult"
146+
-resultBundlePath "$RESULT_BUNDLE_PATH"
169147
170148
# Best-effort and never the job's verdict on its own; the step below is what asserts.
171149
# `--compact` first because a red night's summary is the large one, and the job summary

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerSynthesizedGesture.h‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,10 @@ FOUNDATION_EXPORT double RunnerControlledScrollProgress(double t);
3838
+ (NSString * _Nullable)synthesizeGestureWithApplication:(id)application
3939
pointerSamples:(NSArray<NSArray<NSDictionary<NSString *, NSNumber *> *> *> *)pointerSamples;
4040

41+
// Forgets that this process already synthesized its empty preparation record, so a
42+
// test can observe the preparation again.
43+
+ (void)resetSynthesizedInputPreparation;
44+
4145
// UIInterfaceOrientation of the app (1 portrait, 2 upsideDown, 3 landscapeRight,
4246
// 4 landscapeLeft), or 0 if unreadable.
4347
+ (NSInteger)interfaceOrientationForApplication:(id)application;

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerSynthesizedGesture.m‎

Lines changed: 65 additions & 47 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,6 @@
22
#import "RunnerXCTestEventBridge.h"
33

44
#import <CoreGraphics/CoreGraphics.h>
5-
#import <TargetConditionals.h>
65
#import <objc/message.h>
76

87
static NSString *const RunnerGestureSynthesisSurface = @"event";
@@ -46,6 +45,17 @@ typedef id (*RunnerDragPointerPathFactory)(
4645
const RunnerGestureEventBridge *bridge,
4746
id record
4847
);
48+
static id _Nullable RunnerAllocateEventRecord(
49+
const RunnerGestureEventBridge *bridge,
50+
NSString *recordName,
51+
NSInteger interfaceOrientation,
52+
NSInteger targetProcessID
53+
);
54+
static void RunnerPrepareSynthesizedInput(
55+
const RunnerGestureEventBridge *bridge,
56+
NSInteger interfaceOrientation,
57+
NSInteger targetProcessID
58+
);
4959
static id RunnerSwipePointerPath(
5060
const RunnerGestureEventBridge *bridge,
5161
CGPoint start,
@@ -87,6 +97,8 @@ static id RunnerSampledDragPointerPath(
8797
static const NSTimeInterval RunnerSwipeMovementDurationSeconds = 0.1;
8898
static const double RunnerDragSampleIntervalMs = 16.0;
8999
static const NSInteger RunnerControlledScrollMaxFrameCount = 30;
100+
// Set once the empty preparation record has synthesized in this process.
101+
static BOOL RunnerSynthesizedInputPrepared = NO;
90102
static id RunnerTapPointerPath(
91103
const RunnerGestureEventBridge *bridge,
92104
CGPoint point
@@ -215,6 +227,10 @@ + (NSString * _Nullable)synthesizeGestureWithApplication:(id)application
215227
}
216228
}
217229

230+
+ (void)resetSynthesizedInputPreparation {
231+
RunnerSynthesizedInputPrepared = NO;
232+
}
233+
218234
+ (NSInteger)interfaceOrientationForApplication:(id)application {
219235
SEL selector = NSSelectorFromString(@"interfaceOrientation");
220236
if (![application respondsToSelector:selector]) {
@@ -310,21 +326,6 @@ + (NSInteger)interfaceOrientationForApplication:(id)application {
310326
return nil;
311327
}
312328

313-
static id RunnerAllocateGestureRecord(
314-
const RunnerGestureEventBridge *bridge,
315-
NSString *name,
316-
NSInteger orientation,
317-
NSInteger processID
318-
) {
319-
id record = ((RunnerMsgSendInitRecord)objc_msgSend)(
320-
[bridge->core.recordClass alloc], bridge->initRecordSelector, name, orientation
321-
);
322-
if (record != nil) {
323-
((RunnerMsgSendSetInteger)objc_msgSend)(record, bridge->core.setTargetProcessIDSelector, processID);
324-
}
325-
return record;
326-
}
327-
328329
static NSString * _Nullable RunnerCreateEventRecord(
329330
id application,
330331
NSString *recordName,
@@ -342,41 +343,58 @@ static id RunnerAllocateGestureRecord(
342343
return @"private XCTest event synthesis unavailable: could not resolve target process ID";
343344
}
344345

345-
#if TARGET_OS_IOS
346-
// Prepare at the shared record boundary, before any route constructs timed
347-
// pointer paths. An empty record carries no contact (including no status-bar
348-
// tap), and reuses the caller's resolved orientation/PID without AX or images.
349-
static BOOL didPrepare = NO;
350-
@synchronized ([RunnerSynthesizedGesture class]) {
351-
if (!didPrepare) {
352-
NSTimeInterval startedAt = NSProcessInfo.processInfo.systemUptime;
353-
NSString *preparationError = nil;
354-
@try {
355-
id preparation = RunnerAllocateGestureRecord(
356-
bridge, @"agent-device-input-preparation", interfaceOrientation, targetProcessID
357-
);
358-
preparationError = preparation == nil
359-
? @"could not create preparation record"
360-
: RunnerSynthesizeEventRecord(bridge, preparation);
361-
didPrepare = preparationError == nil;
362-
} @catch (NSException *exception) {
363-
preparationError = RunnerFormatXCTestException(exception, @"input preparation failed");
364-
}
365-
NSLog(
366-
@"AGENT_DEVICE_RUNNER_SYNTHESIZED_INPUT_WARMUP outcome=%@ elapsedMs=%.0f detail=%@",
367-
didPrepare ? @"performed" : @"unsupported",
368-
(NSProcessInfo.processInfo.systemUptime - startedAt) * 1000,
369-
preparationError ?: @""
370-
);
371-
}
346+
if (!RunnerSynthesizedInputPrepared) {
347+
RunnerPrepareSynthesizedInput(bridge, interfaceOrientation, targetProcessID);
372348
}
373-
#endif
374-
// Failed preparation is best-effort and remains eligible on the next request.
375-
// The requested record is always fresh and retains its original timing.
376-
*record = RunnerAllocateGestureRecord(bridge, recordName, interfaceOrientation, targetProcessID);
349+
*record = RunnerAllocateEventRecord(bridge, recordName, interfaceOrientation, targetProcessID);
377350
return *record == nil ? @"private XCTest event synthesis failed: could not create event record" : nil;
378351
}
379352

353+
static id _Nullable RunnerAllocateEventRecord(
354+
const RunnerGestureEventBridge *bridge,
355+
NSString *recordName,
356+
NSInteger interfaceOrientation,
357+
NSInteger targetProcessID
358+
) {
359+
id record = ((RunnerMsgSendInitRecord)objc_msgSend)(
360+
[bridge->core.recordClass alloc],
361+
bridge->initRecordSelector,
362+
recordName,
363+
interfaceOrientation
364+
);
365+
if (record != nil) {
366+
((RunnerMsgSendSetInteger)objc_msgSend)(record, bridge->core.setTargetProcessIDSelector, targetProcessID);
367+
}
368+
return record;
369+
}
370+
371+
// Pays XCTest's one-time synthesis setup with a record that carries no pointer path, so no
372+
// contact reaches the target, before the first timed record. A failure is logged and retried
373+
// on the next record; the requested record is created afterwards either way.
374+
static void RunnerPrepareSynthesizedInput(
375+
const RunnerGestureEventBridge *bridge,
376+
NSInteger interfaceOrientation,
377+
NSInteger targetProcessID
378+
) {
379+
NSTimeInterval startedAt = NSProcessInfo.processInfo.systemUptime;
380+
NSString *error = nil;
381+
@try {
382+
id record = RunnerAllocateEventRecord(
383+
bridge, @"agent-device-input-preparation", interfaceOrientation, targetProcessID
384+
);
385+
error = record == nil ? @"could not create event record" : RunnerSynthesizeEventRecord(bridge, record);
386+
} @catch (NSException *exception) {
387+
error = RunnerFormatXCTestException(exception, @"input preparation failed");
388+
}
389+
RunnerSynthesizedInputPrepared = error == nil;
390+
NSLog(
391+
@"AGENT_DEVICE_RUNNER_SYNTHESIZED_INPUT_PREPARATION outcome=%@ elapsedMs=%.0f detail=%@",
392+
error == nil ? @"performed" : @"failed",
393+
(NSProcessInfo.processInfo.systemUptime - startedAt) * 1000,
394+
error ?: @""
395+
);
396+
}
397+
380398
static NSString * _Nullable RunnerSynthesizeEventRecord(
381399
const RunnerGestureEventBridge *bridge,
382400
id record

apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/UnitTests/RunnerTests+SynthesizedInputWarmupTests.swift renamed to apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/UnitTests/RunnerTests+SynthesizedInputPreparationTests.swift

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@ private final class SynthesizedInputRecordSpy: NSObject {
2727
)
2828
if SynthesizedInputSpy.rejectNext {
2929
SynthesizedInputSpy.rejectNext = false
30-
error?.pointee = NSError(domain: "WarmupTest", code: 1)
30+
error?.pointee = NSError(domain: "SynthesizedInputPreparationTest", code: 1)
3131
return false
3232
}
3333
return true
@@ -47,21 +47,23 @@ extension RunnerTests {
4747
originals.append((method, method_getImplementation(method)))
4848
method_setImplementation(method, method_getImplementation(spy))
4949
}
50+
RunnerSynthesizedGesture.resetSynthesizedInputPreparation()
5051
defer {
5152
for (method, implementation) in originals { method_setImplementation(method, implementation) }
53+
RunnerSynthesizedGesture.resetSynthesizedInputPreparation()
5254
SynthesizedInputSpy.paths = [:]
5355
SynthesizedInputSpy.submittedPathCounts = []
5456
SynthesizedInputSpy.rejectNext = true
5557
}
5658
let target = SynthesizedInputTarget()
57-
// Exercise the actual bridge entry points used by gesture/sequence, scroll,
58-
// synthesized drag and coordinate tap. No real event reaches the simulator.
59+
// Every bridge entry point gesture/sequence, scroll, synthesized drag, swipe and
60+
// coordinate tap use. The spy keeps any record from reaching the simulator.
5961
let samples: [[[String: NSNumber]]] = [[
6062
["x": 10, "y": 10, "offsetMs": 0],
6163
["x": 20, "y": 20, "offsetMs": 100],
6264
]]
63-
// A failed preparation must leave the real request available and permit the
64-
// next route to prepare again. Only a successful empty synthesis consumes it.
65+
// The first preparation is rejected: the request still goes out, and the next route
66+
// prepares again. Only a successful empty record retires the preparation.
6567
XCTAssertNil(RunnerSynthesizedGesture.synthesizeGesture(withApplication: target, pointerSamples: samples))
6668
XCTAssertNil(RunnerSynthesizedGesture.synthesizeControlledScroll(withApplication: target, x: 10, y: 10, x2: 20, y2: 20, durationMs: 100))
6769
XCTAssertNil(RunnerSynthesizedGesture.synthesizeContinuousDrag(withApplication: target, x: 10, y: 10, x2: 20, y2: 20, durationMs: 100))

‎scripts/__tests__/xctest-selection.test.ts‎

Lines changed: 2 additions & 43 deletions
Original file line numberDiff line numberDiff line change
@@ -84,20 +84,13 @@ describe('the real tree', () => {
8484
test('the whole-bundle lanes skip the runner server entry point, which is not a test', () => {
8585
// The whole reason -skip-testing: exists in this repo. `testCommand` opens an
8686
// NWListener and waits 24 hours; an unfiltered run would hang the lane to its timeout.
87-
const report = loadReport(repoRoot);
88-
const skipped = report.flagged.filter((entry) => entry.flag === 'skip-testing');
87+
const skipped = loadReport(repoRoot).flagged.filter((entry) => entry.flag === 'skip-testing');
8988
for (const entry of LANES.filter((entry) => entry.selection === 'whole')) {
9089
expect(
9190
skipped.filter((flag) => flag.workflow === entry.workflow).map((flag) => flag.identifier),
9291
).toContain(ENTRY_POINT);
9392
}
94-
for (const entry of skipped) {
95-
if (entry.identifier === ENTRY_POINT) continue;
96-
// A real test may be excluded from the shared process only if another invocation
97-
// on the same lane still reaches it. Removing the isolated run must fail this.
98-
const current = LANES.find((lane) => lane.workflow === entry.workflow)!;
99-
expect(report.reach[current.id].has(entry.identifier)).toBe(true);
100-
}
93+
for (const entry of skipped) expect(entry.identifier).toBe(ENTRY_POINT);
10194
});
10295

10396
test('the simulator-only tests are exactly the ones the macOS build compiles out', () => {
@@ -303,40 +296,6 @@ describe('a planted guard', () => {
303296
});
304297
});
305298

306-
describe('isolated invocations', () => {
307-
const isolated = `${TARGET}/RunnerTests/testIsolated`;
308-
const swift = source(`${ENTRY_SOURCE}extension RunnerTests {\n func testIsolated() {}\n}\n`);
309-
const shared = `xcodebuild test-without-building -skip-testing:${ENTRY_POINT} -skip-testing:${isolated}`;
310-
311-
test('a separate selected invocation restores a test skipped by the shared process', () => {
312-
const report = buildReport(TARGET, swift, [
313-
...laneWorkflows().filter((entry) => entry.workflow !== NIGHTLY_WORKFLOW_FILE),
314-
{
315-
workflow: NIGHTLY_WORKFLOW_FILE,
316-
text: `xcodebuild test-without-building -only-testing:${isolated}\n${shared}`,
317-
},
318-
]);
319-
expect([...report.reach.nightly]).toEqual([isolated]);
320-
expect(report.entryPointReachedBy).toEqual([]);
321-
});
322-
323-
test('removing the isolated invocation leaves that test unreachable on nightly', () => {
324-
const report = buildReport(TARGET, swift, [
325-
...laneWorkflows().filter((entry) => entry.workflow !== NIGHTLY_WORKFLOW_FILE),
326-
{ workflow: NIGHTLY_WORKFLOW_FILE, text: shared },
327-
]);
328-
expect(report.reach.nightly.has(isolated)).toBe(false);
329-
});
330-
331-
test('a skip still wins over an only flag in the same invocation', () => {
332-
const report = buildReport(TARGET, swift, [
333-
...laneWorkflows().filter((entry) => entry.workflow !== NIGHTLY_WORKFLOW_FILE),
334-
{ workflow: NIGHTLY_WORKFLOW_FILE, text: `${shared} -only-testing:${isolated}` },
335-
]);
336-
expect(report.reach.nightly.has(isolated)).toBe(false);
337-
});
338-
});
339-
340299
describe('the workflow scan', () => {
341300
test('reads both flags on the multi-line xcodebuild invocation', () => {
342301
expect(

‎scripts/check-xctest-selection.ts‎

Lines changed: 9 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -188,26 +188,19 @@ function identifiers(
188188
);
189189
}
190190

191-
/** Union the invocations: a skip in one process must not erase an isolated run. */
191+
/** What one lane reaches: its platform's compiled set, narrowed by its flags. */
192192
function laneReach(
193193
entry: Lane,
194194
declaredTests: readonly DeclaredTest[],
195-
workflowText: string,
195+
flagged: readonly FlaggedTest[],
196196
): Set<string> {
197-
const parts = workflowText.split(/^\s*(?:run:\s*)?xcodebuild\s+test(?:-without-building)?\b/m);
198-
const invocations = parts.length > 1 ? parts.slice(1) : parts;
199-
const compiled = declaredTests
200-
.filter((test) => test.platforms.includes(entry.platform))
201-
.map((test) => test.identifier);
197+
const only = identifiers(flagged, entry.workflow, 'only-testing');
198+
const skipped = identifiers(flagged, entry.workflow, 'skip-testing');
202199
return new Set(
203-
invocations.flatMap((text) => {
204-
const flagged = parseFlaggedTests(entry.workflow, text);
205-
const only = identifiers(flagged, entry.workflow, 'only-testing');
206-
const skipped = identifiers(flagged, entry.workflow, 'skip-testing');
207-
// An explicit list narrows even a whole-bundle lane's isolated invocation.
208-
const whole = entry.selection === 'whole' && only.size === 0;
209-
return compiled.filter((id) => (whole || only.has(id)) && !skipped.has(id));
210-
}),
200+
declaredTests
201+
.filter((test) => test.platforms.includes(entry.platform))
202+
.map((test) => test.identifier)
203+
.filter((id) => (entry.selection === 'whole' || only.has(id)) && !skipped.has(id)),
211204
);
212205
}
213206

@@ -227,14 +220,7 @@ export function buildReport(
227220
// and cannot speak for anything else a workflow might select.
228221
const owned = flagged.filter((entry) => entry.identifier.startsWith(`${target}/`));
229222
const reach = Object.fromEntries(
230-
LANES.map((entry) => [
231-
entry.id,
232-
laneReach(
233-
entry,
234-
declaredTests,
235-
workflows.find((workflow) => workflow.workflow === entry.workflow)?.text ?? '',
236-
),
237-
]),
223+
LANES.map((entry) => [entry.id, laneReach(entry, declaredTests, flagged)]),
238224
) as Record<LaneId, ReadonlySet<string>>;
239225
const entryPoint = `${target}/${ENTRY_POINT_METHOD}`;
240226
const reachedAnywhere = new Set(LANES.flatMap((entry) => [...reach[entry.id]]));

0 commit comments

Comments
 (0)