Skip to content

Commit 3ea5a0d

Browse files
luisleo526claude
andauthored
ci: split the long test rows, shard the full CI legs and run them on the larger hosted runners (#544)
CI speed: split the long test rows, shard the full CI legs and run them on the larger hosted runners (owner's 15-min CI order). Measured CI wall (diag run, same tree): Release 73.9 min -> shards 11.4-17.5; Debug 41.5 -> 10.8; kernel 57.5 -> 14.1-14.7; sanitizers 51.9 -> 8.1-19.6; live-tsan 64.0 -> ~12. live-sanitizers (91.5) still over 15 min: CI follow-up. Merge gate: pineforge/verify (4 clean-tree CLOUD jobs on tree 80c14b2) and pineforge/parity (sweep exp-ci544-gate-sweep-20261027, 7989 probes, cov lost 0, band1 +0/-0, band2 +0/-0, hard reg 0, leavers 0). Paired workflow PR pineforge-workflow#241 (lab remote shard floors) merges right after. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent 30e9f9a commit 3ea5a0d

31 files changed

Lines changed: 4615 additions & 630 deletions

‎.github/actionlint.yaml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,3 +9,6 @@ self-hosted-runner:
99
- pf-gcp-main
1010
- pf-ci-x64
1111
- pf-ci-x64-16
12+
- pf-linux-x64-16
13+
- pf-linux-x64-32
14+
- pf-linux-x64-64

‎.github/workflows/ci-post.yml‎

Lines changed: 227 additions & 43 deletions
Large diffs are not rendered by default.

‎.github/workflows/native-live.yml‎

Lines changed: 158 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -101,27 +101,18 @@ jobs:
101101
native-live:
102102
name: native-live (${{ matrix.profile }})
103103
needs: [authz]
104-
# All three post-merge profiles use the same fixed fleet.
105-
runs-on: ["self-hosted","pf-gcp-post"]
104+
# Each instrumented tree builds once on a dedicated 64-core hosted runner.
105+
runs-on: pf-linux-x64-64
106106
strategy:
107107
fail-fast: false
108108
matrix:
109109
include:
110-
- profile: native
111-
build_dir: build-live
112110
- profile: live-sanitizers
113111
build_dir: build-live-sanitizers
114112
- profile: live-tsan
115113
build_dir: build-live-tsan
116-
# 360 minutes is GitHub's job maximum. It is deliberately the ONLY time
117-
# limit here (owner rule, 2026-10-12): the old 75/45/45-minute budgets and
118-
# the verifier's 2700/2820 and 900/1200-second row and stage deadlines
119-
# cut off runs whose own rows were merely slow -- main's 2820s stage
120-
# deadline refused this very row set's 3600s TIMEOUT property -- so no
121-
# stage, step or row carries a bound of ours any more. Only a hang reaches
122-
# the platform maximum. No row inventory changes.
114+
# Keep every runner row and let the platform supply its own job maximum.
123115
env:
124-
PF_CI_LABELS: '["self-hosted","pf-ci","pf-ci-x64-16"]'
125116
CCACHE_DIR: ${{ github.workspace }}/.ccache/${{ matrix.profile }}
126117
steps:
127118
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
@@ -172,7 +163,7 @@ jobs:
172163
173164
- name: Verify (${{ matrix.profile }})
174165
id: verify
175-
run: python3 scripts/ci_verify.py ${{ matrix.profile }} --build-dir ${{ matrix.build_dir }} --jobs "$(getconf _NPROCESSORS_ONLN)" --generator Ninja --curl-dir "${{ steps.curl-deps.outputs.curl-dir }}" --ccache ${{ matrix.profile == 'native' && '--require-websocket' || '' }} ${{ matrix.profile == 'native' && inputs.exclude_slow && '--exclude-label slow' || '' }}
166+
run: python3 scripts/ci_verify.py ${{ matrix.profile }} --build-dir ${{ matrix.build_dir }} --jobs "$(getconf _NPROCESSORS_ONLN)" --generator Ninja --curl-dir "${{ steps.curl-deps.outputs.curl-dir }}" --ccache
176167

177168
- name: Stage and summarize diagnostics
178169
if: always()
@@ -205,12 +196,158 @@ jobs:
205196
path: ${{ env.CCACHE_DIR }}
206197
key: ${{ steps.ccache-restore.outputs.cache-primary-key }}
207198

199+
native-live-shards:
200+
name: native-live-shard (${{ matrix.profile }}, ${{ matrix.shard }}/4)
201+
needs: [authz]
202+
runs-on: pf-linux-x64-16
203+
strategy:
204+
fail-fast: false
205+
matrix:
206+
shard: [1, 2, 3, 4]
207+
include:
208+
- profile: native
209+
build_dir: build-live
210+
outputs:
211+
shard_1: ${{ steps.shard-proof.outputs.shard_1 }}
212+
shard_2: ${{ steps.shard-proof.outputs.shard_2 }}
213+
shard_3: ${{ steps.shard-proof.outputs.shard_3 }}
214+
shard_4: ${{ steps.shard-proof.outputs.shard_4 }}
215+
env:
216+
CCACHE_DIR: ${{ github.workspace }}/.ccache/${{ matrix.profile }}
217+
steps:
218+
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
219+
with:
220+
persist-credentials: false
221+
fetch-depth: 0
222+
223+
- name: Set up Python
224+
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
225+
with:
226+
python-version: '3.12'
227+
228+
- name: Install the CTest pytest runner
229+
run: |
230+
python3 -m pip install --disable-pip-version-check --no-input \
231+
--require-hashes --only-binary ':all:' -r scripts/requirements-preflight.txt
232+
233+
- name: Install native build dependencies
234+
run: |
235+
sudo apt-get update
236+
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y cmake ninja-build libeigen3-dev tzdata-legacy libsqlite3-dev libssl-dev zlib1g-dev ccache git-lfs strace
237+
238+
# Each shard has its own save key; four cold builds cannot reserve one key.
239+
- name: Restore ccache
240+
id: ccache-restore
241+
if: ${{ !inputs.cold_cache }}
242+
uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
243+
with:
244+
path: ${{ env.CCACHE_DIR }}
245+
key: ccache-${{ runner.os }}-${{ runner.arch }}-live-v2-${{ matrix.profile }}-shard-${{ matrix.shard }}-${{ github.sha }}
246+
restore-keys: |
247+
ccache-${{ runner.os }}-${{ runner.arch }}-live-v2-${{ matrix.profile }}-shard-${{ matrix.shard }}-
248+
249+
- name: Set up pinned WebSocket-enabled libcurl
250+
id: curl-deps
251+
uses: ./.github/actions/setup-live-curl
252+
with:
253+
cache-namespace: ${{ matrix.profile }}
254+
cold-cache: ${{ inputs.cold_cache }}
255+
256+
- name: Record compiler cache before verification
257+
run: |
258+
mkdir -p "${{ matrix.build_dir }}"
259+
ccache --show-stats > "${{ matrix.build_dir }}/ccache-before.log"
260+
ccache --zero-stats
261+
262+
- name: Verify (${{ matrix.profile }}, ${{ matrix.shard }}/4)
263+
id: verify
264+
run: python3 scripts/ci_verify.py ${{ matrix.profile }} --build-dir ${{ matrix.build_dir }} --jobs "$(getconf _NPROCESSORS_ONLN)" --generator Ninja --curl-dir "${{ steps.curl-deps.outputs.curl-dir }}" --ccache --require-websocket ${{ matrix.profile == 'native' && inputs.exclude_slow && '--exclude-label slow' || '' }} --ctest-shard ${{ matrix.shard }}/4
265+
266+
- name: Stage and summarize diagnostics
267+
if: always()
268+
env:
269+
CURL_CACHE_HIT: ${{ steps.curl-deps.outputs.cache-hit }}
270+
CCACHE_CACHE_HIT: ${{ steps.ccache-restore.outputs.cache-hit }}
271+
CCACHE_RESTORE_KEY: ${{ steps.ccache-restore.outputs.cache-matched-key }}
272+
COLD_CACHE: ${{ inputs.cold_cache }}
273+
run: |
274+
python3 scripts/collect_ci_diagnostics.py --build-dir ${{ matrix.build_dir }} --profile ${{ matrix.profile }} --dependency-dir build-native-deps
275+
printf 'cold_cache=%s\ncompiler_exact_hit=%s\ncompiler_restore_key=%s\n' "$COLD_CACHE" "$CCACHE_CACHE_HIT" "$CCACHE_RESTORE_KEY" > ci-diagnostics/cache-info.txt
276+
if [[ -f "${{ matrix.build_dir }}/ccache-before.log" ]]; then
277+
cp "${{ matrix.build_dir }}/ccache-before.log" ci-diagnostics/ccache-before.log
278+
fi
279+
280+
- name: Retain CI diagnostics
281+
if: always()
282+
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
283+
with:
284+
name: ci-diagnostics-native-live-${{ matrix.profile }}-${{ matrix.shard }}of4
285+
path: ci-diagnostics/
286+
if-no-files-found: warn
287+
288+
- name: Save ccache after verification
289+
if: ${{ !cancelled() && !inputs.cold_cache && steps.verify.outcome != 'skipped' && steps.ccache-restore.outputs.cache-hit != 'true' }}
290+
uses: actions/cache/save@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
291+
with:
292+
path: ${{ env.CCACHE_DIR }}
293+
key: ${{ steps.ccache-restore.outputs.cache-primary-key }}
294+
295+
- name: Record native shard proof
296+
id: shard-proof
297+
shell: python
298+
env:
299+
SHARD_INDEX: ${{ matrix.shard }}
300+
run: |
301+
import json
302+
import os
303+
from pathlib import Path
304+
source = Path("build-live/ci-summary.json")
305+
print(f"Reading native shard proof: {source}", flush=True)
306+
summary = json.loads(source.read_text())
307+
index = int(os.environ["SHARD_INDEX"])
308+
if (summary.get("profile") != "native" or summary.get("status") != "passed"
309+
or type(summary.get("exitCode")) is not int or summary["exitCode"] != 0
310+
or summary.get("failures") != []
311+
or summary.get("ctestShard") != {"index": index, "count": 4}):
312+
raise SystemExit("native shard summary did not pass")
313+
with Path(os.environ["GITHUB_OUTPUT"]).open("a") as output:
314+
output.write(f"shard_{index}=success\n")
315+
316+
native-live-native:
317+
name: native-live (native)
318+
needs: [authz, native-live-shards]
319+
if: always()
320+
runs-on: ubuntu-24.04
321+
steps:
322+
- name: Refuse an unauthorised run before anything else
323+
if: ${{ needs.authz.result != 'success' }}
324+
run: |
325+
echo "::error::authz did not pass; nothing of this run's code is checked out or run"
326+
exit 1
327+
- name: Require every native shard to have passed
328+
env:
329+
AUTHZ_RESULT: ${{ needs.authz.result }}
330+
SHARDS_RESULT: ${{ needs.native-live-shards.result }}
331+
SHARD_1_RESULT: ${{ needs.native-live-shards.outputs.shard_1 }}
332+
SHARD_2_RESULT: ${{ needs.native-live-shards.outputs.shard_2 }}
333+
SHARD_3_RESULT: ${{ needs.native-live-shards.outputs.shard_3 }}
334+
SHARD_4_RESULT: ${{ needs.native-live-shards.outputs.shard_4 }}
335+
run: |
336+
printf 'native shards: %s\n' "$SHARDS_RESULT"
337+
if test "$SHARDS_RESULT" = cancelled; then
338+
printf 'HANG (re-run): a native shard job was cancelled\n'
339+
fi
340+
test "$AUTHZ_RESULT" = success
341+
test "$SHARDS_RESULT" = success
342+
test "$SHARD_1_RESULT" = success
343+
test "$SHARD_2_RESULT" = success
344+
test "$SHARD_3_RESULT" = success
345+
test "$SHARD_4_RESULT" = success
346+
208347
native-live-gate:
209-
needs: [authz, native-live]
348+
needs: [authz, native-live, native-live-native]
210349
if: always()
211-
runs-on: ["self-hosted","pf-gcp-post"]
212-
env:
213-
PF_CI_LABELS: '["self-hosted","pf-ci","pf-ci-x64"]'
350+
runs-on: ubuntu-24.04
214351
steps:
215352
- name: Refuse an unauthorised run before anything else
216353
if: ${{ needs.authz.result != 'success' }}
@@ -221,10 +358,13 @@ jobs:
221358
env:
222359
AUTHZ_RESULT: ${{ needs.authz.result }}
223360
PROFILES_RESULT: ${{ needs.native-live.result }}
361+
NATIVE_RESULT: ${{ needs.native-live-native.result }}
224362
run: |
225363
printf 'native profiles: %s\n' "$PROFILES_RESULT"
226-
if test "$PROFILES_RESULT" = cancelled; then
364+
printf 'native shard aggregate: %s\n' "$NATIVE_RESULT"
365+
if test "$PROFILES_RESULT" = cancelled || test "$NATIVE_RESULT" = cancelled; then
227366
printf 'HANG (re-run): a native profile job was cancelled, which is how the platform stops a job at its 360-minute maximum\n'
228367
fi
229368
test "$AUTHZ_RESULT" = success
230369
test "$PROFILES_RESULT" = success
370+
test "$NATIVE_RESULT" = success

‎CONTRIBUTING.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -160,7 +160,7 @@ What each gate refuses:
160160

161161
`ci_verify.py` counts the CTest rows that actually **ran** and fails below a
162162
floor. The floors' numbers live in one place, the
163-
`PRIVATE_FIXTURE_PROFILE_FLOORS` ci_verify.py:802 table: per profile, `full`
163+
`PRIVATE_FIXTURE_PROFILE_FLOORS` ci_verify.py:804 table: per profile, `full`
164164
is its overlay-present registered population and `pinned` is how many rows of
165165
`tests/private_fixture_rows.txt` that registry registers, so with the private
166166
fixtures absent the floor is `full` minus `pinned`; every other profile

0 commit comments

Comments
 (0)