Skip to content

Commit eae558b

Browse files
committed
ci: preserve current routing and bootstrap contracts after split
1 parent 98c956f commit eae558b

6 files changed

Lines changed: 59 additions & 47 deletions

File tree

‎.github/workflows/README.md‎

Lines changed: 23 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -143,27 +143,27 @@ of the Linux CI-profile Cargo cache, and only saves it on pushes to `main`.
143143

144144
## What runs when
145145

146-
| Job in `ci.yml` | Triggered by | Routing rule |
147-
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------- |
148-
| `preflight` | every PR / merge group / push / schedule / dispatch / label | none (always runs) |
149-
| `changes` | every PR / merge group / push / schedule / dispatch / label | runs `dev/ci/compute-changes.py` |
150-
| `pr_build_linux` | PR, merge group or push to main, paths matched; on push only the cache-writing jobs, via `build_linux_full`; the test matrix's non-default Spark profiles only in the nightly run **or** with `run-all-spark-profiles`, via `build_linux_all_profiles` | `dev/ci/compute-changes.py` |
151-
| `pr_build_macos` | merge group, **or** PR with `run-macos-tests`; on push to main only `build-native`, to keep main's macOS cargo cache warm, via `build_macos_full` | `dev/ci/compute-changes.py` |
152-
| `pr_benchmark_check` | merge group, **or** PR with `run-benchmark-check` | benchmark sources only |
153-
| `delta_build_gate` | merge group, **or** PR with `run-delta-build-gate` | main sources, poms, `contrib/delta` |
154-
| `pyarrow_udf_test` | merge group, **or** PR with `run-pyarrow-udf-tests` | map-in-batch and Python runner code |
155-
| `docs` | push to main, paths matched, **or** dispatch on `main` | `.asf.yaml`, `docs/**`, `docs.yaml` |
156-
| `spark_3_5` | nightly, **or** PR with `run-spark-3.5-tests` | Spark 3.5 sources |
157-
| `spark_4_1` | merge group, **or** PR with `run-spark-4.1-tests`; the `sql_hive` shards alone with `run-spark-4.1-hive-tests` | Spark 4.1 sources |
158-
| `spark_3_4` | PR with `run-spark-3.4-tests`, or dispatch | Spark 3.4 sources |
159-
| `spark_4_0` | nightly, **or** PR with `run-spark-4.0-tests` | Spark 4.0 sources |
160-
| `spark_4_2` | nightly, **or** PR with `run-spark-4.2-tests` | Spark 4.2 sources |
161-
| `iceberg_1_11` | merge group, **or** PR with `run-iceberg-tests` | Iceberg sources |
162-
| `iceberg_1_8` | nightly, **or** PR with `run-iceberg-tests` | Iceberg sources |
163-
| `iceberg_1_9` | nightly, **or** PR with `run-iceberg-tests` | Iceberg sources |
164-
| `iceberg_1_10` | nightly, **or** PR with `run-iceberg-tests` | Iceberg sources |
165-
| `required_checks` | always, after every job above except `docs` | none (always runs) |
166-
| `nightly_report` | schedule only, after `required_checks`, when it is not green | none |
146+
| Job in `ci.yml` | Triggered by | Routing rule |
147+
| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------- |
148+
| `preflight` | every PR / merge group / push / schedule / dispatch / label | none (always runs) |
149+
| `changes` | every PR / merge group / push / schedule / dispatch / label | runs `dev/ci/compute-changes.py` |
150+
| `pr_build_linux` | PR, merge group or push to main, paths matched; on push only the cache-writing jobs, via `build_linux_full`; the test matrix's non-default Spark profiles only in the nightly run **or** with `run-all-spark-profiles`, via `build_linux_all_profiles` | `dev/ci/compute-changes.py` |
151+
| `pr_build_macos` | merge group, **or** PR with `run-macos-tests`; on push to main only `build-native`, to keep main's macOS cargo cache warm, via `build_macos_full` | `dev/ci/compute-changes.py` |
152+
| `pr_benchmark_check` | merge group, **or** PR with `run-benchmark-check` | benchmark sources only |
153+
| `delta_build_gate` | merge group, **or** PR with `run-delta-build-gate` | main sources, poms, `contrib/delta` |
154+
| `pyarrow_udf_test` | merge group, **or** PR with `run-pyarrow-udf-tests` | map-in-batch and Python runner code |
155+
| `docs` | push to main, paths matched, **or** dispatch on `main` | `.asf.yaml`, `docs/**`, `docs.yaml` |
156+
| `spark_3_5` | nightly, **or** PR with `run-spark-3.5-tests` | Spark 3.5 sources |
157+
| `spark_4_1` | merge group, **or** PR with `run-spark-4.1-tests`; the `sql_hive` shards alone with `run-spark-4.1-hive-tests` | Spark 4.1 sources |
158+
| `spark_3_4` | PR with `run-spark-3.4-tests`, or dispatch | Spark 3.4 sources |
159+
| `spark_4_0` | nightly, **or** PR with `run-spark-4.0-tests` | Spark 4.0 sources |
160+
| `spark_4_2` | nightly, **or** PR with `run-spark-4.2-tests` | Spark 4.2 sources |
161+
| `iceberg_1_11` | merge group, **or** PR with `run-iceberg-tests` | Iceberg sources |
162+
| `iceberg_1_8` | nightly, **or** PR with `run-iceberg-tests` | Iceberg sources |
163+
| `iceberg_1_9` | nightly, **or** PR with `run-iceberg-tests` | Iceberg sources |
164+
| `iceberg_1_10` | nightly, **or** PR with `run-iceberg-tests` | Iceberg sources |
165+
| `required_checks` | always, after every job above except `docs` | none (always runs) |
166+
| `nightly_report` | schedule only, after `required_checks`, when it is not green | none |
167167
| `build_linux_native` | any selected Linux/Spark/Iceberg consumer, including nightly and profile-label runs | `dev/ci/compute-changes.py` |
168168
| `pr_build_linux_checks` | same selection as `pr_build_linux`; cache-refresh and nightly inputs skip unnecessary checks | `dev/ci/compute-changes.py` |
169169

@@ -298,8 +298,8 @@ umbrella doesn't watch, or operate independently of the rest of CI:
298298
| `docs.yaml` | `docs` |
299299
| `spark_sql_test_reusable.yml` | `spark_3_4`, `spark_3_5`, `spark_4_0`, `spark_4_1`, `spark_4_2` |
300300
| `iceberg_spark_test_reusable.yml` | `iceberg_1_8`, `iceberg_1_9`, `iceberg_1_10`, `iceberg_1_11` |
301-
| `build_linux_native.yml` | `build_linux_native` |
302-
| `pr_build_linux_checks.yml` | `pr_build_linux_checks` |
301+
| `build_linux_native.yml` | `build_linux_native` |
302+
| `pr_build_linux_checks.yml` | `pr_build_linux_checks` |
303303

304304
## Changing what runs when
305305

‎.github/workflows/spark_sql_test_reusable.yml‎

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -210,9 +210,6 @@ jobs:
210210
# Restore mtimes so Zinc skips compilation on the test JVM.
211211
tar -xzpf apache-spark.tar.gz
212212
rm -f apache-spark.tar.gz
213-
- name: Bootstrap Maven
214-
uses: ./.github/actions/maven-bootstrap
215-
216213
- name: Setup Spark
217214
uses: ./.github/actions/setup-spark-builder
218215
with:

‎dev/ci/check-ci-config.py‎

Lines changed: 7 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1599,20 +1599,6 @@ def permission_blocks(lines):
15991599
return blocks
16001600

16011601

1602-
def caller_passes_cache_refresh(ci_lines, workflow_name):
1603-
"""Does the ci.yml job that `uses:` this workflow set `cache-refresh-only:`?"""
1604-
job_start = None
1605-
for index, line in enumerate(ci_lines):
1606-
if JOB_KEY.match(line):
1607-
job_start = index
1608-
if job_start is not None and f"uses: ./.github/workflows/{workflow_name}" in line:
1609-
end = index + 1
1610-
while end < len(ci_lines) and not JOB_KEY.match(ci_lines[end]):
1611-
end += 1
1612-
return any(CACHE_REFRESH_INPUT.match(entry) for entry in ci_lines[job_start:end])
1613-
return False
1614-
1615-
16161602
def linux_scope_failures(workflows, allowed, guard, mode):
16171603
"""Require each workflow's retained jobs and guard every other job in that mode."""
16181604
failures = []
@@ -1644,6 +1630,13 @@ def cache_refresh_failures(workflows):
16441630
workflows, CACHE_REFRESH_JOBS, CACHE_REFRESH_GUARD, "cache-refresh-only")
16451631
failures.extend(linux_mode_input_failures(
16461632
workflows, "cache-refresh-only", CACHE_REFRESH_EXPRESSION))
1633+
ci = (workflows / "ci.yml").read_text(encoding="utf-8")
1634+
jobs = block_mapping(block_mapping(ci, 0).get("jobs", ("", ""))[1], 2)
1635+
macos = block_mapping(jobs.get("pr_build_macos", ("", ""))[1], 4)
1636+
inputs = block_mapping(macos.get("with", ("", ""))[1], 6)
1637+
if scalar(inputs.get("cache-refresh-only", ("", ""))[0]) != (
1638+
"${{ needs.changes.outputs.build_macos_full != 'true' }}"):
1639+
failures.append("ci.yml: pr_build_macos must pass its cache-refresh-only mode")
16471640
return failures
16481641

16491642

‎dev/ci/compute-changes.py‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -503,9 +503,9 @@
503503
# On push that is the *only* thing it is for. The queue already tested the
504504
# exact tree that landed, so re-running the lints and the linux-test
505505
# matrix there tests nothing, and they are 514 of the 587 runner-minutes a
506-
# push run costs. The cache writers now span build_linux_native.yml,
507-
# pr_build_linux_checks.yml (independent checks), and pr_build_linux.yml (Maven
508-
# and TPC-H/TPC-DS), so push must select all three workflows. The split
506+
# push run costs. The Linux cache writers span build_linux_native.yml
507+
# and pr_build_linux.yml (Maven and TPC-H/TPC-DS). The independent checks
508+
# workflow shares this route but skips every job on push. The split
509509
# below keeps those writers on push and moves everything else behind
510510
# `build_linux_full`.
511511
"build_linux": ["pr", "queue", "push"],

‎dev/ci/test-ci-config.py‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -123,6 +123,11 @@ def test_existing_checks_cannot_lose_cache_or_nightly_guards(self):
123123
self.assertTrue(any(f"job `{job_id}` has no" in item for item in failures), failures)
124124
path.write_text(original, encoding="utf-8")
125125

126+
def test_macos_caller_keeps_cache_refresh_mode(self):
127+
self.replace("ci.yml", " cache-refresh-only: ${{ needs.changes.outputs.build_macos_full != 'true' }}\n", "")
128+
failures = CHECK.cache_refresh_failures(self.workflows)
129+
self.assertTrue(any("pr_build_macos must pass" in item for item in failures), failures)
130+
126131
def test_linux_callers_each_require_the_correct_mode_inputs(self):
127132
"""One intact caller must not hide missing or miswired inputs on its sibling."""
128133
path = self.workflows / "ci.yml"

0 commit comments

Comments
 (0)