11# Captured runner-loss evidence
22
3- Trimmed directly from laneboss's ` lb1012/fixtures/ ` captures, without copying
4- the peer backend 's tests. ` captured.json ` retains the original source filenames
3+ Trimmed directly from the original runner-loss captures, without copying
4+ any other controller 's tests. ` captured.json ` retains the original source filenames
55as keys. Run objects retain the gate's fields and repository identity; job
66pages retain only sanitizers-shard (4/4), job 125760264898, or the ordinary
77preflight failure, job 125518846442. Step names/conclusions and failure
@@ -21,3 +21,127 @@ mock of gh's real escape-sequence refusal exercises the successful transport
2121path. No full logs, credentials or response bodies are printed by the tests.
2222Run on the designated test host with
2323` python3 scripts/test_ci_rerun_preempted.py ` ; no network or gh is used.
24+
25+ The aggregate-gate change adds these trims of the supplied capture directory:
26+
27+ | File | Repository | Run | Head prefix |
28+ | --- | --- | --- | --- |
29+ | ` e1.json ` | engine | 41712833348 | 59f98ba8 |
30+ | ` e2.json ` | engine | 41755437507 | cf68f156 |
31+ | ` hpo.json ` | hpo | 41749596952 | bef4c2bf |
32+
33+ Each JSON object retains the original filenames as keys: ` run.json ` ,
34+ ` jobs-attempt1.json ` , ` workflow-at-head.yml ` , ` failed-ids.txt ` , and each failed
35+ job's ` annotations-<job>.json ` and ` job-<job>.log ` . Run identity and gate fields
36+ are retained. All job IDs, display names, statuses and conclusions are kept;
37+ failed jobs also keep labels, check identity and every step name/conclusion.
38+ Only failure annotations are kept, with messages unchanged. Workflow data
39+ retains all job IDs and their exact names, needs, reusable calls and strategies;
40+ unneeded scripts, settings and comments are omitted. Long JSON string tokens
41+ use Unicode escapes solely for the commit scan; decoded strings are unchanged.
42+
43+ The stored ` e2 ` run snapshot was captured while attempt 2 was pending;
44+ ` hpo ` records successful attempt 2. ` CaptureAPI ` replays failed attempt 1
45+ with the captured attempt-1 jobs. The stored snapshots are not changed.
46+ Amendment 1 supplies real logs for all 11 failed ` e1 ` jobs and all three
47+ failed ` hpo ` jobs. The trims retain every captured ` ##[error] ` and runner-loss
48+ line, including the display columns added by ` gh run view --job --log ` .
49+ ` api_logs ` removes only the matching job-name and step-name columns to model
50+ the controller's raw log API; timestamped diagnostic text stays unchanged.
51+ The ` e1 ` trim contains 16 diagnostic lines, including all five shutdowns.
52+ The supplied HPO logs contain no such diagnostic lines, so their trimmed
53+ strings are empty. Their failure annotations remain unchanged and prove the
54+ memory gate's own failure; missing native loss proof cannot excuse that job.
55+ Only exact ` job-<id>.log ` files were read; ` .log.new ` leftovers are ignored.
56+
57+ The four ` e2 ` runner-loss logs come from the supplied 12:28 recapture and
58+ retain nine diagnostic lines, including four shutdowns. Attempt-1 logs for
59+ gates 126342953893, 126346363870 and 126359176509 are unavailable after the
60+ attempt-2 rerun: GitHub now serves the later attempt's logs. Their log fields
61+ contain exactly ` attempt-1 log unavailable after the attempt-2 re-run ` .
62+ The previous three unsupported log lines and API recovery claim are removed.
63+ The gates' available attempt-1 evidence is the exit-code-1 annotations and
64+ job steps/conclusions captured at 11:05, before the rerun. That evidence is
65+ unchanged. Amendment 6 maps each unavailable marker to ` LogUnavailable ` ,
66+ the exception used for missing or expired log API responses (404/410).
67+ It is never returned as successful log text. The old e2 controller now
68+ refuses with ` lookup_or_request_failed ` when fetching its first gate log;
69+ the historical independent-failure blocker cannot be replayed without that
70+ attempt's log. The new controller inspects available logs and falls back to
71+ the captured steps and annotations for these three unavailable gate logs.
72+ Both engine cases assert the old refusal and the new rerun with exact
73+ dependent IDs and POST behavior; no pending-log skips remain. The synthetic
74+ controls are separate and clearly labelled.
75+
76+ Each fixture with logs also has a ` capture ` section containing its original
77+ trimmed capture lines, including their line endings. The e1/e2/HPO sections
78+ are populated directly from the supplied capture files, independently of
79+ the replay log fields. A null entry explicitly records an unavailable gate
80+ log; an empty list means an available capture has no retained diagnostics.
81+ The legacy ` captured.json ` references the two unchanged checked-in log
82+ assets with ` log_files ` and keeps their original trimmed rows in ` capture ` .
83+ ` CaptureTests ` loads every JSON fixture and every referenced log asset. It
84+ requires replay logs to equal their own capture rows verbatim, or to carry
85+ the exact unavailable marker with a null capture and a matching gate job.
86+ Corruption controls reject inserted log text and replacements of the three
87+ unavailable e2 markers. No raw logs are printed on a provenance failure.
88+
89+ ` BeforeTests ` loads the unchanged controller with ` git show ` at ` cf68f156 `
90+ and asserts e1's original blocker and e2's unavailable-log refusal. It does
91+ not call gh or the network; the base commit must exist on the test host.
92+ ` AggregateTests ` checks
93+ decisions, dependent IDs, blockers, API reads and the exact failed-jobs POST.
94+ It also loads the full ` e2 ` workflow from its captured Git commit to check
95+ the untrimmed parser, matrix mappings and needs without requiring job logs.
96+ The default suite audits provenance, then runs the red witnesses
97+ before the existing tests and the new behavior tests.
98+ Original ` ControllerTests ` methods remain unchanged. Tests ran on a separate
99+ test host.
100+
101+ Amendment 2 adds ` reuse.json ` , with the called workflow paths as keys.
102+ It trims the native-live and corpus-parity workflows read with ` git show `
103+ at the captured engine heads, ` 59f98ba8 ` and ` cf68f156 ` ; both files are
104+ identical between those commits. Every job ID, name, needs list and matrix
105+ is retained. Scripts and unrelated settings are omitted. That amendment
106+ left the three run fixtures and their diagnostic evidence unchanged.
107+
108+ The controller reads those same-repository called files through the contents
109+ API at the run's head SHA, once per file. Child jobs use exact names and their
110+ own workflow needs; caller matrices keep separate child scopes. Caller needs
111+ remain on the caller, and a need on a caller includes its failed child jobs.
112+ Unreadable or invalid called files, call cycles, unknown children, and remote
113+ or non-workflow references refuse with ` aggregate_mapping_unavailable ` .
114+ The untrimmed workflow test now reads both callees at the captured commit too.
115+
116+ Gate recognition now requires exactly one failed step whose name starts with
117+ ` Require every ` , together with the existing diagnostic/admission checks.
118+ Amendment 6 restores log inspection for every gate. Runner-loss diagnosis
119+ runs first over all available error evidence: a gate with shutdown evidence
120+ and compatible exit-130 fallout is runner loss, not a dependent gate. An
121+ independent diagnostic in either its log or its failure annotations still
122+ refuses the whole run, even alongside runner-loss evidence. Only ordinary
123+ ` Process completed with exit code N. ` diagnostics may be excused as gate
124+ exits, and an exit code alone never proves runner loss.
125+ Only a missing or expired gate log permits annotation fallback; permission,
126+ rate-limit, transport and other request failures still refuse. Dependent
127+ gates still require positive failure annotations, and unreadable annotations
128+ refuse. Shards still require log inspection; their annotations remain
129+ supplemental. Tests cover log-only and annotation-only assertion faults,
130+ runner loss in the gate itself, missing-log transport and mandatory shard
131+ logs. The assertion faults reuse the existing synthetic control; none are
132+ written into captured fixtures. No captured evidence was changed.
133+ Amendment 3 resolves the chain rule: every failed workflow need must be a
134+ runner-loss job or a proved dependent gate. A topological pass reaches the
135+ fixed point on the workflow's needs DAG; cyclic or unmappable needs refuse
136+ with ` aggregate_mapping_unavailable ` . A script reference to needs or an
137+ always condition proves nothing. Any own failure refuses the whole run,
138+ and at least one actual runner-loss job is still required.
139+
140+ The original live-run assertions include e1's final build job, 126229967614,
141+ and e2's final job, 126359176509, in the dependent lists. New controls cover
142+ reverse job order, an inner gate with a second failed step or an ordinary
143+ failed step, chains without runner loss, and chains through reusable calls.
144+ The inline-own-failure and inner-shard controls remain, along with checks
145+ for caller and child needs, matrix isolation, exact names, called-file
146+ lookup failures, and cycles inside called workflows.
147+ The gate-chain cases remain intact after the e2 refresh; no tests ran here.
0 commit comments