Skip to content

Commit dbd5518

Browse files
committed
fix(ios-runner): classify a build failure only from what xcodebuild published
The startup rules read the whole `details` bag as JSON, and `execFailureDetails` puts `cmd`/`args` in that bag. A caller who pinned a profile therefore handed the profile rule its trigger: an unrelated Swift compile error classified as `signing_provisioning_profile_missing` and lost the cache-recovery hint. The same bag holds the `reason` and `hint` this classifier publishes, so a re-wrapped failure would match its own verdict. `runnerToolText` now builds the one haystack a startup rule may read — our message plus the tool's stdout and stderr — and the argv case has a fixture that proves a pinned profile is not evidence (#2680). Reading only `details` also dropped the message on the floor, so a failure the exec layer raised as a plain `Error` (which the catch wraps with `String(err)`) became unclassified while the same sentence in `stdout` classified. The message is in the haystack now, and a `message-only` fixture keeps both carriers at parity. `signing_style_conflict` is removed rather than kept as a guess: the hint named a `CODE_SIGN_STYLE` env lever that does not exist and claimed a cause nothing captured. The conflicting-settings line keeps a row so the profile row below cannot answer it with missing-profile advice, but it publishes `build_failed_unclassified` until a capture shows which setting disagrees and which lever clears it. Versioned help and the installation docs no longer list the withdrawn reason, and `.device-evidence/CHECKLIST.md` gained the capture that would let a follow-up claim it. Provenance got honest vocabulary. `OBSERVED_ON` stamped invented sentences as observed on Xcode 26.2, and one invocation was recorded as the producer of seven configurations that cannot coexist; nothing here was captured, so entries now say `shipped-sniff-trigger` (the matched substrings shipped before #2680, sentence reconstructed) or `invented-shape`, carry `xcodeVersion: 'unobserved'`, and omit `command` until a run records one.
1 parent f620d48 commit dbd5518

6 files changed

Lines changed: 308 additions & 146 deletions

File tree

‎.device-evidence/CHECKLIST.md‎

Lines changed: 26 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,9 @@ Expected: exit non-zero, one error object with
3535
`hint`, `logPath` and `diagnosticId` are top-level, never inside `details`. Also record the Xcode
3636
version (`xcodebuild -version`) so the `signing_no_development_team` fixture in
3737
`packages/platform-apple/src/runner/__tests__/runner-startup-failure-fixtures.ts` can move from
38-
`inherited-sniff-trigger` to `captured`.
38+
`shipped-sniff-trigger` to `captured` and its `xcodeVersion` from `unobserved` to that version.
39+
Paste the whole error so the fixture's `output` can become the capture and its `command` can be
40+
recorded.
3941

4042
### 2. A bundle identifier somebody else already owns -> `bundle_identifier_already_registered`
4143

@@ -61,14 +63,32 @@ version so the matching fixture's `provenance` can be upgraded:
6163
# devtools_security_developer_mode_disabled (macOS admin state, no device work)
6264
DevToolsSecurity -status
6365

64-
# signing_provisioning_profile_missing / signing_style_conflict: point the runner build at a
65-
# profile that is not installed, then read the reason off the same prepare command as above.
66+
# signing_provisioning_profile_missing: point the runner build at a profile that is not installed,
67+
# then read the reason off the same prepare command as above.
6668
env AGENT_DEVICE_IOS_TEAM_ID="<your team id>" \
6769
AGENT_DEVICE_IOS_PROVISIONING_PROFILE="no-such-profile-installed" \
6870
node --experimental-strip-types src/bin.ts --json \
6971
prepare ios-runner --platform ios --device "<physical iPhone name>"
7072
```
7173

72-
Expected: `details.reason` is `signing_provisioning_profile_missing`, or `signing_style_conflict`
73-
when xcodebuild reports conflicting provisioning settings. If neither reason appears, say which
74-
reason did and treat the fixture as unconfirmed rather than editing the rule to fit.
74+
Expected: `details.reason` is `signing_provisioning_profile_missing`. If a different reason appears,
75+
say which one did and treat the fixture as unconfirmed rather than editing the rule to fit.
76+
77+
### 4. The line that claims no reason yet -> `build_failed_unclassified`
78+
79+
`xcodebuild` reports a settings mismatch with a line that names a profile ("has conflicting
80+
provisioning settings"). #2680 deliberately publishes `build_failed_unclassified` for it, because no
81+
capture has proved which lever clears it. To reach it, pin a profile while leaving automatic signing
82+
on:
83+
84+
```sh
85+
env AGENT_DEVICE_IOS_TEAM_ID="<your team id>" \
86+
AGENT_DEVICE_IOS_PROVISIONING_PROFILE="match-development" \
87+
node --experimental-strip-types src/bin.ts --json \
88+
prepare ios-runner --platform ios --device "<physical iPhone name>"
89+
```
90+
91+
Expected: either `signing_provisioning_profile_missing` (xcodebuild complained about the profile
92+
first) or `build_failed_unclassified`. Paste the error and the `xcodebuild -version` either way: a
93+
capture of the conflicting-settings line is what would let a follow-up name the cause, and the
94+
capture must show which build setting disagrees before any hint naming a lever is written.

‎packages/platform-apple/src/runner/__tests__/runner-startup-failure-fixtures.ts‎

Lines changed: 110 additions & 65 deletions
Original file line numberDiff line numberDiff line change
@@ -4,121 +4,157 @@ import type { RunnerStartupFailureReason } from '../runner-contract.ts';
44
/**
55
* Recorded startup failures for {@link classifyRunnerStartupFailure} (#2680).
66
*
7-
* Each entry carries the tool output exactly as it reaches the host, the command that produced it,
8-
* the Xcode that produced it, and how it got here. `provenance` is what says whether a line was
9-
* observed or transcribed: an entry stays `inherited-sniff-trigger` until the matching command in
10-
* `.device-evidence/CHECKLIST.md` is run against real hardware, at which point `output` is replaced
11-
* with the capture and `provenance` becomes `captured`. Nothing in the classifier reads these
12-
* fields — they exist so a reason can be traced to an observation instead to a guess.
7+
* Provenance is the point of this file, so it is stated per entry and never as a blanket claim:
8+
*
9+
* - `captured` — `output` was pasted from a run, and `command` plus `xcodeVersion` (from
10+
* `xcodebuild -version`) were recorded with it by `.device-evidence/CHECKLIST.md`.
11+
* - `shipped-sniff-trigger` — the substrings a rule matches are the ones shipped in
12+
* `resolveSigningFailureHint` before #2680, which is evidence xcodebuild can emit them. The
13+
* sentence around them is ours, so `command` and `xcodeVersion` stay unrecorded.
14+
* - `invented-shape` — no shipped trigger and no capture. The entry exists to exercise a rule and
15+
* makes no claim about wording xcodebuild prints.
16+
*
17+
* Until Phase B captures the real runs, every entry is `unobserved` for `xcodeVersion` and carries
18+
* no `command`: an invocation we did not run is not provenance. Nothing in the classifier reads
19+
* these fields; they exist so a reason can be traced to an observation instead of to a guess.
1320
*/
1421

1522
export type RunnerStartupFailureSite = 'build-for-testing' | 'host-dev-tools-security';
1623

24+
/**
25+
* Whether the text reaches the build catch inside the exec error's `details` (`exec-details`, which
26+
* is how a non-zero `xcodebuild` arrives) or only in the thrown message (`message-only`, which is
27+
* how anything the exec layer raised as a plain `Error` arrives after the catch wraps `String(err)`).
28+
*/
29+
export type RunnerStartupFailureCarrier = 'exec-details' | 'message-only';
30+
31+
const UNOBSERVED = 'unobserved';
32+
1733
export type RunnerStartupFailureFixture = Readonly<{
34+
/** Stable name for a focused test or a review comment. */
35+
id: string;
1836
/** The reason this output must reach the caller with. */
1937
reason: RunnerStartupFailureReason;
2038
/** Which throw site receives this output. */
2139
site: RunnerStartupFailureSite;
22-
/** The invocation that produced {@link RunnerStartupFailureFixture.output}. */
23-
command: string;
24-
/** `xcodebuild -version` of the machine that produced it. */
40+
carrier?: RunnerStartupFailureCarrier;
41+
/** The invocation that produced {@link RunnerStartupFailureFixture.output}, once one is recorded. */
42+
command?: string;
43+
/** `xcodebuild -version` recorded from that run, or `unobserved`. */
2544
xcodeVersion: string;
26-
provenance: 'captured' | 'inherited-sniff-trigger' | 'tool-error-shape';
27-
/** The tool's own stdout/stderr, kept on one shape so JSON detail matching sees it as the host does. */
45+
provenance: 'captured' | 'shipped-sniff-trigger' | 'invented-shape';
46+
/** The tool's own stdout/stderr. */
2847
output: string;
29-
/** What the pending capture still has to show. */
48+
/** The argv the exec reported, which is never evidence of a cause (#2680). */
49+
args?: readonly string[];
50+
/** What the pending capture still has to show, and how to reach it. */
3051
note?: string;
3152
}>;
3253

33-
/**
34-
* The `xcodebuild` invocation `buildRunnerXctestrun` issues for a physical iOS device with Automatic
35-
* Signing and no profile pinned, which is the configuration every signing reason below is about.
36-
*/
37-
const BUILD_FOR_TESTING_COMMAND =
38-
'xcodebuild build-for-testing -project apple/runner/AgentDeviceRunner/AgentDeviceRunner.xcodeproj ' +
39-
'-scheme AgentDeviceRunner -parallel-testing-enabled NO -destination generic/platform=iOS ' +
40-
'-derivedDataPath <derived> -allowProvisioningUpdates CODE_SIGN_STYLE=Automatic ' +
41-
'DEVELOPMENT_TEAM=<AGENT_DEVICE_IOS_TEAM_ID>';
42-
43-
const OBSERVED_ON = 'Xcode 26.2 (Build 17C52)';
44-
4554
export const RUNNER_STARTUP_FAILURE_FIXTURES: readonly RunnerStartupFailureFixture[] = [
4655
{
56+
id: 'bundle-id-registration-failed',
4757
reason: 'bundle_identifier_already_registered',
4858
site: 'build-for-testing',
49-
command: BUILD_FOR_TESTING_COMMAND,
50-
xcodeVersion: OBSERVED_ON,
51-
provenance: 'inherited-sniff-trigger',
59+
xcodeVersion: UNOBSERVED,
60+
provenance: 'shipped-sniff-trigger',
5261
output:
53-
"error: Failed registering bundle identifier \"com.yourname.agentdevice.runner\" with the developer portal: An App ID with Identifier 'com.yourname.agentdevice.runner' is not available. Please enter a different string. (in target 'AgentDeviceRunner' from project 'AgentDeviceRunner')\n** TEST BUILD FAILED **\n",
54-
note: 'Capture with AGENT_DEVICE_IOS_BUNDLE_ID set to an identifier already registered by another team.',
62+
"error: Failed registering bundle identifier \"com.yourname.agentdevice.runner\" with the developer portal (in target 'AgentDeviceRunner' from project 'AgentDeviceRunner')\n** TEST BUILD FAILED **\n",
63+
note: 'Capture with AGENT_DEVICE_IOS_BUNDLE_ID set to an identifier already registered by another team, and record the `xcodebuild -version` of the machine.',
5564
},
5665
{
66+
id: 'app-id-not-available',
5767
reason: 'bundle_identifier_already_registered',
5868
site: 'build-for-testing',
59-
command: BUILD_FOR_TESTING_COMMAND,
60-
xcodeVersion: OBSERVED_ON,
61-
provenance: 'tool-error-shape',
69+
xcodeVersion: UNOBSERVED,
70+
provenance: 'shipped-sniff-trigger',
6271
output:
63-
"error: App Identifier 'com.yourname.agentdevice.runner' is not available. Choose a different App Identifier, or register it in your Apple Developer account before building. (in target 'AgentDeviceRunnerUITests' from project 'AgentDeviceRunner')\n** TEST BUILD FAILED **\n",
64-
note: 'The second shape of the same cause: no "failed registering" line, so only the two-part match can name it.',
72+
"error: App Identifier 'com.yourname.agentdevice.runner' is not available (in target 'AgentDeviceRunner' from project 'AgentDeviceRunner')\n** TEST BUILD FAILED **\n",
73+
note: 'The second shape of the same cause: no "failed registering" line, so only the two-part "app identifier" + "not available" trigger can name it. Trimmed to the shipped trigger; the real sentence is still unrecorded.',
6574
},
6675
{
76+
id: 'requires-development-team',
6777
reason: 'signing_no_development_team',
6878
site: 'build-for-testing',
69-
command: BUILD_FOR_TESTING_COMMAND,
70-
xcodeVersion: OBSERVED_ON,
71-
provenance: 'inherited-sniff-trigger',
79+
xcodeVersion: UNOBSERVED,
80+
provenance: 'shipped-sniff-trigger',
7281
output:
73-
"error: Signing for \"AgentDeviceRunner\" requires a development team. Select a development team in the Signing & Capabilities editor. (in target 'AgentDeviceRunner' from project 'AgentDeviceRunner')\n** TEST BUILD FAILED **\n",
82+
"error: Signing for \"AgentDeviceRunner\" requires a development team (in target 'AgentDeviceRunner' from project 'AgentDeviceRunner')\n** TEST BUILD FAILED **\n",
7483
note: 'Capture with AGENT_DEVICE_IOS_TEAM_ID unset on a signed-in-but-team-less account.',
7584
},
7685
{
86+
id: 'requires-development-team-message-only',
87+
reason: 'signing_no_development_team',
88+
site: 'build-for-testing',
89+
carrier: 'message-only',
90+
xcodeVersion: UNOBSERVED,
91+
provenance: 'shipped-sniff-trigger',
92+
output:
93+
"error: Signing for \"AgentDeviceRunner\" requires a development team (in target 'AgentDeviceRunner' from project 'AgentDeviceRunner')",
94+
note: 'Same text arriving in the thrown message instead of the exec details: the catch wraps a non-AppError with String(err), and the rule still has to see it.',
95+
},
96+
{
97+
id: 'no-profiles-for-bundle-id',
7798
reason: 'signing_provisioning_profile_missing',
7899
site: 'build-for-testing',
79-
command: BUILD_FOR_TESTING_COMMAND,
80-
xcodeVersion: OBSERVED_ON,
81-
provenance: 'inherited-sniff-trigger',
100+
xcodeVersion: UNOBSERVED,
101+
provenance: 'shipped-sniff-trigger',
82102
output:
83-
"error: No profiles for 'com.yourname.agentdevice.runner' were found: Xcode couldn't find any iOS App Development provisioning profiles matching 'com.yourname.agentdevice.runner'. Automatic signing is disabled and unable to generate a profile. To enable automatic signing, pass -allowProvisioningUpdates to xcodebuild. (in target 'AgentDeviceRunner' from project 'AgentDeviceRunner')\n** TEST BUILD FAILED **\n",
103+
"error: No profiles for 'com.yourname.agentdevice.runner' were found (in target 'AgentDeviceRunner' from project 'AgentDeviceRunner')\n** TEST BUILD FAILED **\n",
84104
note: 'Capture with AGENT_DEVICE_IOS_PROVISIONING_PROFILE naming a profile that is not installed.',
85105
},
86106
{
87-
reason: 'signing_style_conflict',
107+
id: 'conflicting-provisioning-settings',
108+
reason: 'build_failed_unclassified',
88109
site: 'build-for-testing',
89-
command: BUILD_FOR_TESTING_COMMAND,
90-
xcodeVersion: OBSERVED_ON,
91-
provenance: 'tool-error-shape',
110+
xcodeVersion: UNOBSERVED,
111+
provenance: 'invented-shape',
92112
output:
93-
'error: "AgentDeviceRunner" has conflicting provisioning settings. AgentDeviceRunner is automatically signed, but provisioning profile "match-development-com-yourname-agentdevice-runner" has been manually specified. Set the provisioning profile value to "Automatic" in the build settings editor, or switch to manual signing in the Signing & Capabilities editor. (in target \'AgentDeviceRunner\' from project \'AgentDeviceRunner\')\n** TEST BUILD FAILED **\n',
94-
note: 'New reason: capture the conflicting-settings line before claiming this wording on hardware.',
113+
"error: \"AgentDeviceRunner\" has conflicting provisioning settings (in target 'AgentDeviceRunner' from project 'AgentDeviceRunner')\n** TEST BUILD FAILED **\n",
114+
note: 'Names a profile while saying the settings disagree, so the profile row must not win. No reason is claimed until a capture proves which lever clears it.',
95115
},
96116
{
117+
id: 'code-signing-required',
97118
reason: 'signing_unspecified',
98119
site: 'build-for-testing',
99-
command: BUILD_FOR_TESTING_COMMAND,
100-
xcodeVersion: OBSERVED_ON,
101-
provenance: 'inherited-sniff-trigger',
120+
xcodeVersion: UNOBSERVED,
121+
provenance: 'shipped-sniff-trigger',
102122
output:
103-
"error: Code signing is required for product type 'Application' in SDK 'iOS 26.2' (in target 'AgentDeviceRunner' from project 'AgentDeviceRunner')\n** TEST BUILD FAILED **\n",
123+
"error: Code signing is required for product type 'Application' (in target 'AgentDeviceRunner' from project 'AgentDeviceRunner')\n** TEST BUILD FAILED **\n",
104124
note: 'Signing is named and nothing above it is: the reason stays unspecified on purpose.',
105125
},
106126
{
127+
id: 'compile-error',
107128
reason: 'build_failed_unclassified',
108129
site: 'build-for-testing',
109-
command: BUILD_FOR_TESTING_COMMAND,
110-
xcodeVersion: OBSERVED_ON,
111-
provenance: 'tool-error-shape',
130+
xcodeVersion: UNOBSERVED,
131+
provenance: 'invented-shape',
112132
output:
113133
"error: cannot find 'AgentDeviceRunnerCommand' in scope (in target 'AgentDeviceRunnerUITests' from project 'AgentDeviceRunner')\n** TEST BUILD FAILED **\n",
114134
note: 'Any build failure that names no signing fact must keep the cache-recovery hint.',
115135
},
116136
{
137+
id: 'argv-names-a-provisioning-profile',
138+
reason: 'build_failed_unclassified',
139+
site: 'build-for-testing',
140+
xcodeVersion: UNOBSERVED,
141+
provenance: 'invented-shape',
142+
output:
143+
"error: cannot find 'AgentDeviceRunnerCommand' in scope (in target 'AgentDeviceRunnerUITests' from project 'AgentDeviceRunner')\n** TEST BUILD FAILED **\n",
144+
args: [
145+
'build-for-testing',
146+
'PROVISIONING_PROFILE_SPECIFIER=match-development',
147+
'Provisioning Profile: match-development',
148+
],
149+
note: 'The argv we were asked to run is not xcodebuild evidence: a caller who pinned a profile still gets cache-recovery advice for a compile error (#2680).',
150+
},
151+
{
152+
id: 'devtools-security-disabled',
117153
reason: 'devtools_security_developer_mode_disabled',
118154
site: 'host-dev-tools-security',
119155
command: 'DevToolsSecurity -status',
120-
xcodeVersion: OBSERVED_ON,
121-
provenance: 'inherited-sniff-trigger',
156+
xcodeVersion: UNOBSERVED,
157+
provenance: 'shipped-sniff-trigger',
122158
output: 'Developer mode is currently disabled for development tools.\n',
123159
note: "Host-side refusal. It says nothing about the device's Developer Mode toggle (#2683 reads that).",
124160
},
@@ -128,21 +164,30 @@ export function buildForTestingFixtures(): RunnerStartupFailureFixture[] {
128164
return RUNNER_STARTUP_FAILURE_FIXTURES.filter((fixture) => fixture.site === 'build-for-testing');
129165
}
130166

167+
export function buildFixtureById(id: string): RunnerStartupFailureFixture {
168+
const fixture = RUNNER_STARTUP_FAILURE_FIXTURES.find((candidate) => candidate.id === id);
169+
if (!fixture) throw new Error(`no startup failure fixture records ${id}`);
170+
return fixture;
171+
}
172+
131173
/**
132-
* The error the exec layer hands the build-failure catch when `xcodebuild` exits non-zero: a
133-
* COMMAND_FAILED whose message is the exec's own and whose tool output sits in `details`, which is
134-
* exactly why the rules below read details text and not only the message.
174+
* What the exec layer hands the build-failure catch: for `exec-details` a COMMAND_FAILED carrying
175+
* the tool's output and the argv in `details` (`execFailureDetails` shape), and for `message-only`
176+
* the plain `Error` the catch turns into `new AppError('COMMAND_FAILED', String(error))`.
135177
*/
136-
export function buildForTestingExecError(
137-
fixture: Pick<RunnerStartupFailureFixture, 'output'>,
178+
export function buildForTestingExecFailure(
179+
fixture: RunnerStartupFailureFixture,
138180
exitCode = 65,
139-
): AppError {
140-
return new AppError('COMMAND_FAILED', 'xcodebuild exited with code 65', {
181+
): unknown {
182+
if ((fixture.carrier ?? 'exec-details') === 'message-only') {
183+
return new Error(`xcodebuild exited with code ${exitCode}: ${fixture.output}`);
184+
}
185+
return new AppError('COMMAND_FAILED', `xcodebuild exited with code ${exitCode}`, {
141186
stdout: fixture.output,
142187
stderr: '',
143188
exitCode,
144189
processExitError: true,
145190
cmd: 'xcodebuild',
146-
args: ['build-for-testing'],
191+
args: fixture.args ?? ['build-for-testing'],
147192
});
148193
}

0 commit comments

Comments
 (0)