Skip to content

Commit 065ef1d

Browse files
luisleo526claude
andauthored
Split native CI profiles and make cold verification reproducible (#354)
* Split native CI profiles and retain bounded cold-run diagnostics * Document independent native proofs and preserve source anchors docs/ci.md replaces two superseded descriptions of the old shared native-live job: - dbd17b3 (#320): the sentence "native-live.yml runs both profiles alongside native, reusing the same checksum-pinned WebSocket-enabled curl" is replaced by the three independent matrix cells description. - INT26 (d093560): the old `native-live` timeout-table row (the larger runner's full set "about 25 min", "The verification hosts' release profile, the nearest they run, took 11.2-18.4 min" and "A fork's pull request runs the set without that row: 3.4 min on the standard runner") is replaced by the per-cell native, live-sanitizers and live-tsan rows measured on main runs 37398893123, 37461653250 and 37478400505. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Bound native rows by the slowest measured host and keep failed-row counts A cold full-row native profile of d5ffd30 on a c5.4xlarge Spot host ran 941/941 rows: 940 passed and test_ci_verify was cut off by the 2100 s default deadline (Timeout 2100.07 s). The same script then completed in 2177.7 s (162 tests OK) in that host's preflight. GitHub measured 984-1695 s. Raise the native default per-test deadline to 2700 s (24% over the slowest measured host) inside a 2820 s CTest stage. The measured cold pre-CTest time was 12.3 min, so a worst-case stage still fits the unchanged 65-min verification step and 75-min job. Live profiles keep 900/1200 s. That failed run's ci-summary recorded no executed/skipped counts. CTest prints a listed row's labels after its status ("2 - test_ci_verify (Timeout) slow"), and CTEST_LISTED_ROW required the line to end at the status, so the row list was unreadable. Accept trailing labels. A new real-CTest test pins labelled failed, skipped and not-run rows. docs/ci.md rewords the `native-live (native)` timeout-table row written by the docs commit "Document independent native proofs and preserve source anchors". Its sentence "The longest measured test_ci_verify row was 1695 s; its new 2100-s default deadline gives 24% margin within a 2400-s CTest stage" now carries the Spot measurement and the new bounds. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Keep real CTest deadline test compatible with optional JUnit Use the existing CTest capability detector and condition only XML presence; retain the real timeout, witness, row-count, elapsed and log checks. Rewrite the driver-default and blanket two-times timeout sentences introduced by d093560 (INT26, #290). Document the workflow's 47-minute native and 20-minute live overrides and the table's measured-margin policy without changing historical timings. * Accept older CTest failed-list rendering without losing label coverage Allow the real labelled fixture to render an undecorated failed-list row. Verify slow/live labels for every fixture row through genuine CTest JSON inventory, retain the synthetic label-suffixed parser witness and all row accounting, and require the real failure exit code. Leave JUnit capability handling and production code unchanged. --------- Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
1 parent 49bb202 commit 065ef1d

8 files changed

Lines changed: 673 additions & 85 deletions

File tree

‎.github/actions/setup-live-curl/action.yml‎

Lines changed: 19 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,12 @@
11
name: Set up native runner libcurl
22
description: Cache and build the checksum-pinned WebSocket-enabled libcurl
3+
inputs:
4+
cache-namespace:
5+
description: Isolate dependency caches for each verification profile
6+
default: shared
7+
cold-cache:
8+
description: Bypass dependency cache restore and save
9+
default: 'false'
310
outputs:
411
curl-dir:
512
description: CMake package directory for ci_verify.py --curl-dir
@@ -26,12 +33,22 @@ runs:
2633
echo "curl-dir=${GITHUB_WORKSPACE}/build-native-deps/curl-install/lib/cmake/CURL" >> "$GITHUB_OUTPUT"
2734
- name: Restore pinned WebSocket-enabled libcurl
2835
id: curl-cache
29-
uses: actions/cache@v4
36+
if: ${{ inputs.cold-cache != 'true' }}
37+
uses: actions/cache/restore@v4
3038
with:
3139
path: build-native-deps/curl-install
32-
key: curl-${{ steps.dependencies.outputs.version }}-${{ steps.dependencies.outputs.sha256 }}-${{ steps.dependencies.outputs.identity }}-${{ hashFiles('.github/actions/setup-live-curl/action.yml', 'scripts/build_live_curl.sh') }}-${{ runner.os }}-${{ runner.arch }}
40+
key: curl-${{ inputs.cache-namespace }}-${{ steps.dependencies.outputs.version }}-${{ steps.dependencies.outputs.sha256 }}-${{ steps.dependencies.outputs.identity }}-${{ hashFiles('.github/actions/setup-live-curl/action.yml', 'scripts/build_live_curl.sh') }}-${{ runner.os }}-${{ runner.arch }}
3341
- name: Build pinned WebSocket-enabled libcurl
42+
id: curl-build
3443
shell: bash
3544
env:
3645
CURL_CACHE_HIT: ${{ steps.curl-cache.outputs.cache-hit }}
3746
run: bash scripts/build_live_curl.sh build-native-deps 4
47+
# Save as soon as the dependency build succeeds. Later test failures and
48+
# job timeouts must not prevent a valid pinned installation from surviving.
49+
- name: Save pinned WebSocket-enabled libcurl
50+
if: ${{ inputs.cold-cache != 'true' && steps.curl-build.outcome == 'success' && steps.curl-cache.outputs.cache-hit != 'true' }}
51+
uses: actions/cache/save@v4
52+
with:
53+
path: build-native-deps/curl-install
54+
key: ${{ steps.curl-cache.outputs.cache-primary-key }}

‎.github/workflows/ci.yml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,11 @@ on:
66
pull_request:
77
branches: [main]
88
workflow_dispatch:
9+
inputs:
10+
cold_native_cache:
11+
description: Run full native profiles without compiler or libcurl caches
12+
type: boolean
13+
default: false
914

1015
permissions:
1116
contents: read
@@ -332,6 +337,7 @@ jobs:
332337
uses: ./.github/workflows/native-live.yml
333338
with:
334339
exclude_slow: ${{ github.event_name == 'pull_request' }}
340+
cold_cache: ${{ inputs.cold_native_cache || false }}
335341

336342
# Advisory TradingView parity subset on every CI run of a change that is not
337343
# docs-only. The maintainers post the required pineforge/parity status from

‎.github/workflows/native-live.yml‎

Lines changed: 97 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -7,100 +7,155 @@ on:
77
description: Exclude measured slow CTest rows in the caller's PR run
88
type: boolean
99
default: false
10+
cold_cache:
11+
description: Bypass compiler and libcurl cache restore and save for every profile
12+
type: boolean
13+
default: false
1014
workflow_dispatch:
15+
inputs:
16+
cold_cache:
17+
description: Prove the full row set without compiler or libcurl caches
18+
type: boolean
19+
default: false
1120

1221
permissions:
1322
contents: read
1423

15-
# The caller owns PR cancellation. Keep a distinct group: github.workflow in
16-
# a reusable workflow names the caller, so an unprefixed group would collide
17-
# with CI and could cancel it. Main/manual proof groups stay unique per run.
24+
# The caller owns PR cancellation. A reusable workflow's github.workflow is
25+
# its caller, so the prefix avoids cancelling CI. Main/manual runs are unique.
1826
concurrency:
1927
group: native-live-${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }}
2028
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
2129

2230
env:
23-
CCACHE_DIR: ${{ github.workspace }}/.ccache
2431
CCACHE_COMPILERCHECK: content
2532
CCACHE_COMPRESS: "1"
2633
CCACHE_MAXSIZE: 500M
27-
# The run-failure source guards in every ci_verify profile read the newest
28-
# release tag, which this checkout fetches (fetch-depth: 0): a missing tag
29-
# fails them rather than taking their tag-less offline path. A called
30-
# workflow does not inherit CI's env, so it says so itself.
34+
# A reusable workflow does not inherit the caller's environment. The guards
35+
# require the release tags fetched by every profile's full checkout.
3136
PINEFORGE_REQUIRE_RELEASE_TAGS: "1"
3237

33-
# The organization's 16-core larger runner for a push, the schedule, a
34-
# dispatch and a pull request from a branch of this repository; every other
35-
# event, a fork's pull request above all, runs on the free standard runner. In
36-
# a workflow_call the github context is the caller's, so a fork's pull request
37-
# through CI lands there too. scripts/ci_preflight.py pins the runner and limit.
3838
jobs:
3939
native-live:
40+
name: native-live (${{ matrix.profile }})
41+
# Same trusted-event 16-core policy as CI; forks use the standard runner.
4042
runs-on: ${{ (github.event_name == 'push' || github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository)) && 'pf-linux-x64-16' || 'ubuntu-24.04' }}
41-
# The full set waits on the test_ci_verify CTest row, 1034 s on the
42-
# standard runner at 0d76a099, which more cores do not shorten.
43-
timeout-minutes: 60
43+
timeout-minutes: ${{ matrix.job_minutes }}
44+
strategy:
45+
fail-fast: false
46+
matrix:
47+
include:
48+
- profile: native
49+
build_dir: build-live
50+
job_minutes: 75
51+
verify_minutes: 65
52+
test_timeout: 2700
53+
ctest_timeout: 2820
54+
- profile: live-sanitizers
55+
build_dir: build-live-sanitizers
56+
job_minutes: 45
57+
verify_minutes: 35
58+
test_timeout: 900
59+
ctest_timeout: 1200
60+
- profile: live-tsan
61+
build_dir: build-live-tsan
62+
job_minutes: 45
63+
verify_minutes: 35
64+
test_timeout: 900
65+
ctest_timeout: 1200
66+
# Main runs 37398893123/37461653250/37478400505 took 35.4–40.7m
67+
# native + 15.1–16.3m ASan before TSan exhausted the shared hour, despite
68+
# compiler and curl cache hits. Independent 75/45/45m budgets allow cold
69+
# builds and leave ten minutes outside verification for setup/collection.
70+
# test_ci_verify took 1695s on GitHub and 2178s on a c5.4xlarge; 2700s
71+
# gives that host 24% margin, inside a 2820s stage. Live rows allow up to
72+
# 900s; keep those limits and a 1200s stage. No row inventory changes.
73+
env:
74+
CCACHE_DIR: ${{ github.workspace }}/.ccache/${{ matrix.profile }}
4475
steps:
4576
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
4677
with:
4778
persist-credentials: false
48-
fetch-depth: 0 # ci_verify runs the benchmark provenance CTest row
79+
fetch-depth: 0
4980

5081
- name: Set up Python
5182
uses: actions/setup-python@v5
5283
with:
5384
python-version: '3.12'
5485

55-
# tzdata-legacy carries the backward links test_native_calendar resolves
56-
# (US/Eastern, Japan, GB); noble split them out of tzdata, and this job
57-
# runs the calendar row too.
5886
- name: Install native build dependencies
5987
run: |
6088
sudo apt-get update
6189
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y cmake ninja-build libeigen3-dev tzdata-legacy libsqlite3-dev libssl-dev zlib1g-dev ccache
6290
6391
- name: Restore ccache
64-
uses: actions/cache@v4
92+
id: ccache-restore
93+
if: ${{ !inputs.cold_cache }}
94+
uses: actions/cache/restore@v4
6595
with:
6696
path: ${{ env.CCACHE_DIR }}
67-
key: ccache-${{ runner.os }}-${{ runner.arch }}-native-${{ github.sha }}
97+
key: ccache-${{ runner.os }}-${{ runner.arch }}-live-v2-${{ matrix.profile }}-${{ github.sha }}
6898
restore-keys: |
69-
ccache-${{ runner.os }}-${{ runner.arch }}-native-
99+
ccache-${{ runner.os }}-${{ runner.arch }}-live-v2-${{ matrix.profile }}-
70100
71101
- name: Set up pinned WebSocket-enabled libcurl
72102
id: curl-deps
73103
uses: ./.github/actions/setup-live-curl
104+
with:
105+
cache-namespace: ${{ matrix.profile }}
106+
cold-cache: ${{ inputs.cold_cache }}
74107

75-
- name: Verify native live
76-
run: python3 scripts/ci_verify.py native --build-dir build-live --jobs "$(getconf _NPROCESSORS_ONLN)" --generator Ninja --curl-dir "${{ steps.curl-deps.outputs.curl-dir }}" --ccache --require-websocket ${{ inputs.exclude_slow && '--exclude-label slow' || '' }}
108+
- name: Record compiler cache before verification
109+
run: |
110+
mkdir -p "${{ matrix.build_dir }}"
111+
ccache --show-stats > "${{ matrix.build_dir }}/ccache-before.log"
112+
ccache --zero-stats
77113
78-
- name: Verify every live runner target with ASan and UBSan
79-
if: ${{ !cancelled() }}
80-
run: python3 scripts/ci_verify.py live-sanitizers --build-dir build-live-sanitizers --jobs "$(getconf _NPROCESSORS_ONLN)" --generator Ninja --curl-dir "${{ steps.curl-deps.outputs.curl-dir }}" --ccache
114+
- name: Verify (${{ matrix.profile }})
115+
id: verify
116+
timeout-minutes: ${{ matrix.verify_minutes }}
117+
run: python3 scripts/ci_verify.py ${{ matrix.profile }} --build-dir ${{ matrix.build_dir }} --jobs "$(getconf _NPROCESSORS_ONLN)" --generator Ninja --curl-dir "${{ steps.curl-deps.outputs.curl-dir }}" --ccache --test-timeout ${{ matrix.test_timeout }} --ctest-timeout ${{ matrix.ctest_timeout }} ${{ matrix.profile == 'native' && '--require-websocket' || '' }} ${{ matrix.profile == 'native' && inputs.exclude_slow && '--exclude-label slow' || '' }}
81118

82119
- name: Stage and summarize diagnostics
83120
if: always()
84121
env:
85122
CURL_CACHE_HIT: ${{ steps.curl-deps.outputs.cache-hit }}
86-
run: python3 scripts/collect_ci_diagnostics.py --build-dir build-live --profile native --dependency-dir build-native-deps
87-
88-
- name: Verify every live runner target with ThreadSanitizer
89-
if: ${{ !cancelled() }}
90-
run: python3 scripts/ci_verify.py live-tsan --build-dir build-live-tsan --jobs "$(getconf _NPROCESSORS_ONLN)" --generator Ninja --curl-dir "${{ steps.curl-deps.outputs.curl-dir }}" --ccache
91-
92-
- name: Stage thread sanitizer diagnostics
93-
if: always()
94-
run: python3 scripts/collect_ci_diagnostics.py --build-dir build-live-tsan --profile live-tsan --output ci-diagnostics/live-tsan --dependency-dir build-native-deps
95-
96-
- name: Stage live sanitizer diagnostics
97-
if: always()
98-
run: python3 scripts/collect_ci_diagnostics.py --build-dir build-live-sanitizers --profile live-sanitizers --output ci-diagnostics/live-sanitizers --dependency-dir build-native-deps
123+
CCACHE_CACHE_HIT: ${{ steps.ccache-restore.outputs.cache-hit }}
124+
CCACHE_RESTORE_KEY: ${{ steps.ccache-restore.outputs.cache-matched-key }}
125+
COLD_CACHE: ${{ inputs.cold_cache }}
126+
run: |
127+
python3 scripts/collect_ci_diagnostics.py --build-dir ${{ matrix.build_dir }} --profile ${{ matrix.profile }} --dependency-dir build-native-deps
128+
printf 'cold_cache=%s\ncompiler_exact_hit=%s\ncompiler_restore_key=%s\n' "$COLD_CACHE" "$CCACHE_CACHE_HIT" "$CCACHE_RESTORE_KEY" > ci-diagnostics/cache-info.txt
129+
if [[ -f "${{ matrix.build_dir }}/ccache-before.log" ]]; then
130+
cp "${{ matrix.build_dir }}/ccache-before.log" ci-diagnostics/ccache-before.log
131+
fi
99132
100133
- name: Retain CI diagnostics
101134
if: always()
102135
uses: actions/upload-artifact@v4
103136
with:
104-
name: ci-diagnostics-native-live
137+
name: ci-diagnostics-native-live-${{ matrix.profile }}
105138
path: ci-diagnostics/
106139
if-no-files-found: warn
140+
141+
# A failing test does not invalidate compiler objects. Save explicitly
142+
# after diagnostics, even on normal failure; cancellation is no cache plan.
143+
- name: Save ccache after verification
144+
if: ${{ !cancelled() && !inputs.cold_cache && steps.verify.outcome != 'skipped' && steps.ccache-restore.outputs.cache-hit != 'true' }}
145+
uses: actions/cache/save@v4
146+
with:
147+
path: ${{ env.CCACHE_DIR }}
148+
key: ${{ steps.ccache-restore.outputs.cache-primary-key }}
149+
150+
native-live-gate:
151+
needs: native-live
152+
if: always()
153+
runs-on: ubuntu-24.04
154+
timeout-minutes: 5
155+
steps:
156+
- name: Require every native profile to have passed
157+
env:
158+
PROFILES_RESULT: ${{ needs.native-live.result }}
159+
run: |
160+
printf 'native profiles: %s\n' "$PROFILES_RESULT"
161+
test "$PROFILES_RESULT" = success

0 commit comments

Comments
 (0)