Skip to content

Commit fc58eec

Browse files
luisleo526claude
andcommitted
ci: re-run aggregate gates that failed only because of runner loss [skip ci]
The CI post-merge re-run controller now also re-runs a failed aggregate gate job when its only failed step is a "Require every ..." gate exit and every failed job it needs, followed transitively through the workflow's needs graph (including called workflows), is a runner-loss job or another such gate. A gate job that itself lost its runner counts as runner loss. It still refuses the whole run on any job's own failure, a gate without failure annotations, an unmappable job or need, a cycle, or any lookup error, and it still requires at least one real runner-loss job. Called workflows are read as data at the run's head commit; nothing is executed. Tests: scripts/test_ci_rerun_preempted.py, with fixtures trimmed from three captured runs. Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent 12148f9 commit fc58eec

8 files changed

Lines changed: 3200 additions & 30 deletions

File tree

‎scripts/ci/rerun_preempted.py‎

Lines changed: 476 additions & 26 deletions
Large diffs are not rendered by default.

‎scripts/fixtures/ci_rerun_preempted/README.md‎

Lines changed: 126 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# Captured runner-loss evidence
22

3-
Trimmed directly from laneboss's `lb1012/fixtures/` captures, without copying
4-
the peer backend's tests. `captured.json` retains the original source filenames
3+
Trimmed directly from the original runner-loss captures, without copying
4+
any other controller's tests. `captured.json` retains the original source filenames
55
as keys. Run objects retain the gate's fields and repository identity; job
66
pages retain only sanitizers-shard (4/4), job 125760264898, or the ordinary
77
preflight failure, job 125518846442. Step names/conclusions and failure
@@ -21,3 +21,127 @@ mock of gh's real escape-sequence refusal exercises the successful transport
2121
path. No full logs, credentials or response bodies are printed by the tests.
2222
Run on the designated test host with
2323
`python3 scripts/test_ci_rerun_preempted.py`; no network or gh is used.
24+
25+
The aggregate-gate change adds these trims of the supplied capture directory:
26+
27+
| File | Repository | Run | Head prefix |
28+
| --- | --- | --- | --- |
29+
| `e1.json` | engine | 41712833348 | 59f98ba8 |
30+
| `e2.json` | engine | 41755437507 | cf68f156 |
31+
| `hpo.json` | hpo | 41749596952 | bef4c2bf |
32+
33+
Each JSON object retains the original filenames as keys: `run.json`,
34+
`jobs-attempt1.json`, `workflow-at-head.yml`, `failed-ids.txt`, and each failed
35+
job's `annotations-<job>.json` and `job-<job>.log`. Run identity and gate fields
36+
are retained. All job IDs, display names, statuses and conclusions are kept;
37+
failed jobs also keep labels, check identity and every step name/conclusion.
38+
Only failure annotations are kept, with messages unchanged. Workflow data
39+
retains all job IDs and their exact names, needs, reusable calls and strategies;
40+
unneeded scripts, settings and comments are omitted. Long JSON string tokens
41+
use Unicode escapes solely for the commit scan; decoded strings are unchanged.
42+
43+
The stored `e2` run snapshot was captured while attempt 2 was pending;
44+
`hpo` records successful attempt 2. `CaptureAPI` replays failed attempt 1
45+
with the captured attempt-1 jobs. The stored snapshots are not changed.
46+
Amendment 1 supplies real logs for all 11 failed `e1` jobs and all three
47+
failed `hpo` jobs. The trims retain every captured `##[error]` and runner-loss
48+
line, including the display columns added by `gh run view --job --log`.
49+
`api_logs` removes only the matching job-name and step-name columns to model
50+
the controller's raw log API; timestamped diagnostic text stays unchanged.
51+
The `e1` trim contains 16 diagnostic lines, including all five shutdowns.
52+
The supplied HPO logs contain no such diagnostic lines, so their trimmed
53+
strings are empty. Their failure annotations remain unchanged and prove the
54+
memory gate's own failure; missing native loss proof cannot excuse that job.
55+
Only exact `job-<id>.log` files were read; `.log.new` leftovers are ignored.
56+
57+
The four `e2` runner-loss logs come from the supplied 12:28 recapture and
58+
retain nine diagnostic lines, including four shutdowns. Attempt-1 logs for
59+
gates 126342953893, 126346363870 and 126359176509 are unavailable after the
60+
attempt-2 rerun: GitHub now serves the later attempt's logs. Their log fields
61+
contain exactly `attempt-1 log unavailable after the attempt-2 re-run`.
62+
The previous three unsupported log lines and API recovery claim are removed.
63+
The gates' available attempt-1 evidence is the exit-code-1 annotations and
64+
job steps/conclusions captured at 11:05, before the rerun. That evidence is
65+
unchanged. Amendment 6 maps each unavailable marker to `LogUnavailable`,
66+
the exception used for missing or expired log API responses (404/410).
67+
It is never returned as successful log text. The old e2 controller now
68+
refuses with `lookup_or_request_failed` when fetching its first gate log;
69+
the historical independent-failure blocker cannot be replayed without that
70+
attempt's log. The new controller inspects available logs and falls back to
71+
the captured steps and annotations for these three unavailable gate logs.
72+
Both engine cases assert the old refusal and the new rerun with exact
73+
dependent IDs and POST behavior; no pending-log skips remain. The synthetic
74+
controls are separate and clearly labelled.
75+
76+
Each fixture with logs also has a `capture` section containing its original
77+
trimmed capture lines, including their line endings. The e1/e2/HPO sections
78+
are populated directly from the supplied capture files, independently of
79+
the replay log fields. A null entry explicitly records an unavailable gate
80+
log; an empty list means an available capture has no retained diagnostics.
81+
The legacy `captured.json` references the two unchanged checked-in log
82+
assets with `log_files` and keeps their original trimmed rows in `capture`.
83+
`CaptureTests` loads every JSON fixture and every referenced log asset. It
84+
requires replay logs to equal their own capture rows verbatim, or to carry
85+
the exact unavailable marker with a null capture and a matching gate job.
86+
Corruption controls reject inserted log text and replacements of the three
87+
unavailable e2 markers. No raw logs are printed on a provenance failure.
88+
89+
`BeforeTests` loads the unchanged controller with `git show` at `cf68f156`
90+
and asserts e1's original blocker and e2's unavailable-log refusal. It does
91+
not call gh or the network; the base commit must exist on the test host.
92+
`AggregateTests` checks
93+
decisions, dependent IDs, blockers, API reads and the exact failed-jobs POST.
94+
It also loads the full `e2` workflow from its captured Git commit to check
95+
the untrimmed parser, matrix mappings and needs without requiring job logs.
96+
The default suite audits provenance, then runs the red witnesses
97+
before the existing tests and the new behavior tests.
98+
Original `ControllerTests` methods remain unchanged. Tests ran on a separate
99+
test host.
100+
101+
Amendment 2 adds `reuse.json`, with the called workflow paths as keys.
102+
It trims the native-live and corpus-parity workflows read with `git show`
103+
at the captured engine heads, `59f98ba8` and `cf68f156`; both files are
104+
identical between those commits. Every job ID, name, needs list and matrix
105+
is retained. Scripts and unrelated settings are omitted. That amendment
106+
left the three run fixtures and their diagnostic evidence unchanged.
107+
108+
The controller reads those same-repository called files through the contents
109+
API at the run's head SHA, once per file. Child jobs use exact names and their
110+
own workflow needs; caller matrices keep separate child scopes. Caller needs
111+
remain on the caller, and a need on a caller includes its failed child jobs.
112+
Unreadable or invalid called files, call cycles, unknown children, and remote
113+
or non-workflow references refuse with `aggregate_mapping_unavailable`.
114+
The untrimmed workflow test now reads both callees at the captured commit too.
115+
116+
Gate recognition now requires exactly one failed step whose name starts with
117+
`Require every`, together with the existing diagnostic/admission checks.
118+
Amendment 6 restores log inspection for every gate. Runner-loss diagnosis
119+
runs first over all available error evidence: a gate with shutdown evidence
120+
and compatible exit-130 fallout is runner loss, not a dependent gate. An
121+
independent diagnostic in either its log or its failure annotations still
122+
refuses the whole run, even alongside runner-loss evidence. Only ordinary
123+
`Process completed with exit code N.` diagnostics may be excused as gate
124+
exits, and an exit code alone never proves runner loss.
125+
Only a missing or expired gate log permits annotation fallback; permission,
126+
rate-limit, transport and other request failures still refuse. Dependent
127+
gates still require positive failure annotations, and unreadable annotations
128+
refuse. Shards still require log inspection; their annotations remain
129+
supplemental. Tests cover log-only and annotation-only assertion faults,
130+
runner loss in the gate itself, missing-log transport and mandatory shard
131+
logs. The assertion faults reuse the existing synthetic control; none are
132+
written into captured fixtures. No captured evidence was changed.
133+
Amendment 3 resolves the chain rule: every failed workflow need must be a
134+
runner-loss job or a proved dependent gate. A topological pass reaches the
135+
fixed point on the workflow's needs DAG; cyclic or unmappable needs refuse
136+
with `aggregate_mapping_unavailable`. A script reference to needs or an
137+
always condition proves nothing. Any own failure refuses the whole run,
138+
and at least one actual runner-loss job is still required.
139+
140+
The original live-run assertions include e1's final build job, 126229967614,
141+
and e2's final job, 126359176509, in the dependent lists. New controls cover
142+
reverse job order, an inner gate with a second failed step or an ordinary
143+
failed step, chains without runner loss, and chains through reusable calls.
144+
The inline-own-failure and inner-shard controls remain, along with checks
145+
for caller and child needs, matrix isolation, exact names, called-file
146+
lookup failures, and cycles inside called workflows.
147+
The gate-chain cases remain intact after the e2 refresh; no tests ran here.

‎scripts/fixtures/ci_rerun_preempted/captured.json‎

Lines changed: 19 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -140,5 +140,23 @@
140140
"annotation_level": "failure",
141141
"message": "Process completed with exit code 1."
142142
}
143-
]
143+
],
144+
"log_files": [
145+
"job-125518846442.log",
146+
"job-125760264898.log"
147+
],
148+
"capture": {
149+
"job-125518846442.log": [
150+
"2026-11-02T20:14:41.3063508Z ##[error]Process completed with exit code 1.\n"
151+
],
152+
"job-125760264898.log": [
153+
"2026-11-03T06:17:42.1290045Z \u001b[36;1mpython3 -m pip install --disable-pip-version-check --no-input \\\u001b[0m\n",
154+
"2026-11-03T06:17:42.1290763Z \u001b[36;1m --require-hashes --only-binary ':all:' -r scripts/requirements-preflight.txt\u001b[0m\n",
155+
"2026-11-03T06:17:44.4077827Z \u001b[36;1msudo apt-get update\u001b[0m\n",
156+
"2026-11-03T06:17:44.4078408Z \u001b[36;1msudo DEBIAN_FRONTEND=noninteractive apt-get install -y libeigen3-dev tzdata-legacy ccache\u001b[0m\n",
157+
"2026-11-03T06:17:50.0756396Z \u001b[36;1mpython3 scripts/ci_verify.py sanitizers --build-dir build-asan --jobs \"$(getconf _NPROCESSORS_ONLN)\" --ccache --ctest-shard 4/4\u001b[0m\n",
158+
"2026-11-03T06:29:00.0326985Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n",
159+
"2026-11-03T06:29:01.0112153Z ##[error]The operation was canceled.\n"
160+
]
161+
}
144162
}

0 commit comments

Comments
 (0)