Skip to content

Commit 347fb07

Browse files
committed
ci: narrow native cache inputs and test routing
1 parent e3b9e85 commit 347fb07

5 files changed

Lines changed: 74 additions & 36 deletions

File tree

‎.github/actions/build-native-ci/action.yaml‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -20,13 +20,15 @@ description: 'Reuse an exact-input native library, otherwise build it with the C
2020
runs:
2121
using: composite
2222
steps:
23+
- name: Pin native build flags
24+
shell: bash
25+
run: echo 'RUSTFLAGS=-Ctarget-cpu=x86-64-v3 -Clink-arg=-fuse-ld=bfd' >> "$GITHUB_ENV"
26+
2327
# Call after checkout and setup-builder. Compute once, before Cargo writes
2428
# generated Rust files, and use the same keys for both restore and save.
2529
- name: Fingerprint native build inputs
2630
id: key
2731
shell: bash
28-
env:
29-
RUSTFLAGS: '-Ctarget-cpu=x86-64-v3 -Clink-arg=-fuse-ld=bfd'
3032
run: python3 dev/ci/native-cache-key.py --profile ci --github-output "$GITHUB_OUTPUT"
3133

3234
- name: Restore native library cache
@@ -54,8 +56,6 @@ runs:
5456
- name: Build native library (CI profile)
5557
if: steps.binary-cache.outputs.cache-hit != 'true' || (github.event_name == 'push' && github.ref == 'refs/heads/main')
5658
shell: bash
57-
env:
58-
RUSTFLAGS: '-Ctarget-cpu=x86-64-v3 -Clink-arg=-fuse-ld=bfd'
5959
run: |
6060
cd native
6161
# A target-directory cache is only an incremental build aid. Cargo

‎.github/workflows/README.md‎

Lines changed: 21 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -350,18 +350,30 @@ downstream tests use the same paths in either case.
350350
`dev/ci/native-cache-key.py` snapshots tracked native/protobuf/dependency files,
351351
Cargo configuration and the native build recipes before Cargo generates source
352352
files. The key also includes Rust versions, installed system package
353-
versions, architecture, JDK release/path and compiler flags. The helper targets
354-
our official Rust container and `setup-builder`, not arbitrary local toolchains.
355-
Spark-only edits, documentation, unrelated workflows and generated files preserve
356-
the key; native/protobuf changes invalidate it. Optional contrib crates contribute
353+
versions, architecture, JDK release/path and the build environment: Cargo/Rust
354+
settings, C/C++ compiler and flag overrides (including target-specific variants),
355+
and the HDFS library overrides used by the default dependencies. The helper targets
356+
our official Rust container and `setup-builder`. Adding external tools or files
357+
requires updating this contract; recording an override's path does not identify
358+
arbitrary contents stored there.
359+
360+
The shared build and setup actions are fingerprinted; the four caller workflows
361+
are not. Their selected Rust/JDK versions and build environment are observed
362+
directly, so editing a test matrix or shard does not force a native rebuild.
363+
Spark-only edits, documentation and generated files also preserve the key;
364+
native/protobuf changes invalidate it. Optional contrib crates contribute
357365
their manifests, which Cargo resolves even with their features disabled, but not
358366
their Rust sources or standalone lockfiles. Benchmarks enter the debug cache key
359367
but not the library key. The input lists and glob matcher are shared with main's
360368
cache routing in `compute-changes.py`. The shared action uses portable `x86-64-v3`
361369
code generation.
362370

363371
The incremental cache contains the effective `CARGO_HOME` registry/git directories
364-
and `native/target`. Its dependency prefix permits reuse after source changes
372+
and `native/target`. In the Rust container, correcting `~/.cargo` to
373+
`/usr/local/cargo` adds the registry and Git checkouts that the old entry did not
374+
contain. The incremental entry therefore grows alongside the addition of the
375+
separate finished-library entry.
376+
Its dependency prefix permits reuse after source changes
365377
within the same build environment, but every restore still invokes Cargo.
366378
Environment changes also invalidate this fallback: native dependencies compile C
367379
against JNI headers and cache build-script outputs that Cargo does not fully
@@ -376,8 +388,10 @@ also trigger main's cache warmer. GitHub Actions handles cache storage and
376388
restoration.
377389

378390
Preflight tests key invalidation, generated-file stability, container checkout
379-
ownership, and that every binary-key input triggers main's cache warmer. After
380-
main populates the new namespace, verify a hosted library hit
391+
ownership, and that every binary-key input triggers main's cache warmer. On the
392+
first main push that populates these namespaces, report the compressed cache
393+
sizes in bytes for both the finished library and the incremental Cargo entry,
394+
using the cache-save logs or Actions cache API. Then verify a hosted library hit
381395
by checking that `Restore native library cache` reports an exact hit and the
382396
Cargo steps skip, while the normal artifact upload and downstream tests pass.
383397

‎dev/ci/compute-changes.py‎

Lines changed: 3 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -38,8 +38,8 @@
3838
import sys
3939
from pathlib import Path
4040

41-
# Shared cache recipes affect every native producer. Tests exercise the recipes
42-
# but do not affect the resulting library, so they are routed separately below.
41+
# Shared cache recipes affect every native producer. Their tests run in
42+
# Preflight and retain the existing dev/ci/** routes, without adding consumers.
4343
NATIVE_CACHE_RECIPES = (
4444
".github/actions/build-native-ci/**",
4545
"dev/ci/native-cache-key.py", "dev/ci/compute-changes.py",
@@ -51,8 +51,6 @@
5151
NATIVE_BUILD_INPUTS = (
5252
"native/**", "contrib/*/native/Cargo.toml", ".cargo/**",
5353
".github/actions/setup-builder/**", *NATIVE_CACHE_RECIPES,
54-
".github/workflows/pr_build_linux.yml", ".github/workflows/spark_sql_test_reusable.yml",
55-
".github/workflows/iceberg_spark_test_reusable.yml", ".github/workflows/spark_sql_writer_tests.yml",
5654
"rust-toolchain", "rust-toolchain.toml", "!**.md",
5755
)
5856
NATIVE_LIBRARY_INPUTS = (*NATIVE_BUILD_INPUTS, "!**/benches/**")
@@ -413,10 +411,7 @@
413411
"build_linux", "spark_3_4", "spark_3_5", "spark_4_0", "spark_4_1",
414412
"iceberg_1_8", "iceberg_1_9", "iceberg_1_10", "iceberg_1_11",
415413
):
416-
FILTERS[_native_consumer].extend([
417-
*NATIVE_CACHE_RECIPES,
418-
"dev/ci/test-native-cache-key.py",
419-
])
414+
FILTERS[_native_consumer].extend(NATIVE_CACHE_RECIPES)
420415

421416
FILTERS["spark_4_1_hive"] = FILTERS["spark_4_1"]
422417
FILTERS["build_linux_full"] = FILTERS["build_linux"]

‎dev/ci/native-cache-key.py‎

Lines changed: 16 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -25,24 +25,24 @@
2525

2626
import argparse
2727
import hashlib
28+
import importlib.util
2829
import json
2930
import os
3031
from pathlib import Path
31-
import runpy
3232
import subprocess
3333

3434

3535
# Share both the input patterns and their glob semantics with main's warmer.
36-
CHANGES = runpy.run_path(str(Path(__file__).with_name("compute-changes.py")))
36+
SPEC = importlib.util.spec_from_file_location("compute_changes", Path(__file__).with_name("compute-changes.py"))
37+
CHANGES = importlib.util.module_from_spec(SPEC)
38+
SPEC.loader.exec_module(CHANGES)
3739

3840

3941
def digest(value):
40-
"""Return a stable SHA-256 for JSON-compatible build inputs."""
4142
return hashlib.sha256(json.dumps(value, sort_keys=True).encode()).hexdigest()
4243

4344

4445
def command(args, cwd):
45-
"""Read command stdout in cwd; missing tools or unsuccessful commands fail CI."""
4646
return subprocess.check_output(args, cwd=cwd, text=True).strip()
4747

4848

@@ -54,15 +54,15 @@ def source_inputs(root, profile="ci"):
5454
builds omit benchmarks; debug checks compile them. Trust only this checkout
5555
for the Git read: container steps can run as a different owner than checkout.
5656
"""
57-
patterns = CHANGES["NATIVE_LIBRARY_INPUTS" if profile == "ci" else "NATIVE_BUILD_INPUTS"]
57+
patterns = CHANGES.NATIVE_LIBRARY_INPUTS if profile == "ci" else CHANGES.NATIVE_BUILD_INPUTS
5858
inventory = command(["git", "-c", f"safe.directory={root}",
5959
"ls-files", "--stage", "-z"], root)
6060
sources = {}
6161
for record in inventory.split("\0"):
6262
if not record:
6363
continue
6464
metadata, name = record.split("\t", 1)
65-
if CHANGES["matches"](patterns, [name]):
65+
if CHANGES.matches(patterns, [name]):
6666
sources[name] = [metadata.split()[0],
6767
hashlib.sha256((root / name).read_bytes()).hexdigest()]
6868
dependencies = {name: value for name, value in sources.items()
@@ -75,9 +75,10 @@ def environment_inputs(root, env):
7575
7676
Rust's versions include the compiler commit; dpkg identifies the installed
7777
C/C++/protobuf tools and system libraries. The JDK release file identifies
78-
the vendor/build supplying JNI headers and libjvm. Paths and RUSTFLAGS are
79-
included because linking can embed them. Native producer workflows/actions
80-
in the source map cover how these tools are configured and invoked.
78+
the vendor/build supplying JNI headers and libjvm. Record build overrides,
79+
including target-qualified cc variables and HDFS linking options, without
80+
including unrelated per-run GitHub variables. The shared setup/build actions
81+
are hashed separately; caller test configuration does not affect the library.
8182
"""
8283
java_home = Path(env["JAVA_HOME"])
8384
return {
@@ -91,7 +92,12 @@ def environment_inputs(root, env):
9192
"java_home": str(java_home),
9293
"java_release": (java_home / "release").read_text(),
9394
"cargo_home": env.get("CARGO_HOME", str(Path.home() / ".cargo")),
94-
"rustflags": env["RUSTFLAGS"],
95+
"env": {name: value for name, value in env.items()
96+
if name.startswith(("CARGO_", "RUST", "HOST_", "TARGET_", "HDFS_"))
97+
or name.split("_", 1)[0] in {"CC", "CXX", "CFLAGS", "CXXFLAGS", "CXXSTDLIB",
98+
"LDFLAGS", "AR", "ARFLAGS", "RANLIB", "RANLIBFLAGS", "PROTOC"}
99+
or name in {"JAVA_HOME", "PATH", "HADOOP_HOME", "DOCS_RS",
100+
"CRATE_CC_NO_DEFAULTS", "CROSS_COMPILE"}},
95101
}
96102

97103

‎dev/ci/test-native-cache-key.py‎

Lines changed: 30 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,6 @@
2222
import io
2323
import os
2424
from pathlib import Path
25-
import runpy
2625
import subprocess
2726
import sys
2827
import tempfile
@@ -48,8 +47,13 @@ def setUp(self):
4847
"native/lib.rs": "fn native() {}\n", "native/proto/expr.proto": "message Expr {}\n",
4948
"spark/Plan.scala": "object Plan {}\n", "README.md": "Comet\n",
5049
".github/workflows/README.md": "CI documentation\n",
50+
".github/workflows/pr_build_linux.yml": "jobs: {}\n",
5151
".github/workflows/spark_sql_test_reusable.yml": "jobs: {}\n",
52+
".github/workflows/iceberg_spark_test_reusable.yml": "jobs: {}\n",
53+
".github/workflows/spark_sql_writer_tests.yml": "jobs: {}\n",
5254
".github/workflows/check_pr_title.yml": "jobs: {}\n",
55+
".github/actions/build-native-ci/action.yaml": "runs: {}\n",
56+
".github/actions/setup-builder/action.yaml": "runs: {}\n",
5357
"dev/ci/compute-changes.py": "# shared native input rules\n",
5458
"contrib/delta/native/Cargo.toml": '[package]\nname = "delta"\n',
5559
"contrib/delta/native/src/lib.rs": "fn delta() {}\n",
@@ -84,7 +88,7 @@ def test_source_and_dependency_changes_invalidate_the_right_keys(self):
8488
before = self.keys()
8589
for name in ("native/lib.rs", "native/proto/expr.proto", "native/Cargo.toml", "native/Cargo.lock",
8690
"contrib/delta/native/Cargo.toml", "dev/ci/compute-changes.py",
87-
".github/workflows/spark_sql_test_reusable.yml"):
91+
".github/actions/build-native-ci/action.yaml", ".github/actions/setup-builder/action.yaml"):
8892
with self.subTest(name=name):
8993
self.write(name, self.inputs[name] + "changed\n")
9094
after = self.keys()
@@ -100,30 +104,39 @@ def test_generated_files_and_unrelated_jvm_edits_preserve_keys(self):
100104
"""Generated files and non-build edits preserve reuse; debug still tracks benchmarks."""
101105
before = self.keys()
102106
debug = self.keys("debug")
107+
for name in self.inputs:
108+
if name.startswith(".github/workflows/"):
109+
self.write(name, "unrelated test configuration\n")
110+
self.env["GITHUB_RUN_ID"] = "12345"
111+
self.assertEqual(before, self.keys())
112+
self.assertEqual(debug, self.keys("debug"))
103113
self.write("native/proto/src/generated/expr.rs", "generated Rust")
104114
self.write("native/target/ci/libcomet.so", "compiled library")
105115
self.write("spark/Plan.scala", "object NewPlan {}")
106116
self.write("README.md", "updated docs")
107-
self.write(".github/workflows/README.md", "updated CI docs")
108-
self.write(".github/workflows/check_pr_title.yml", "jobs: {changed: {}}")
109117
self.write("contrib/delta/native/src/lib.rs", "fn changed_delta() {}")
110118
self.write("contrib/delta/native/Cargo.lock", "version = 3\n")
111119
self.write("contrib/a/b/native/Cargo.toml", '[package]\nname = "changed_nested"\n')
112120
self.write("native/core/benches/perf.rs", "fn changed_benchmark() {}")
113121
self.assertEqual(before, self.keys())
114122
self.assertNotEqual(debug["source-key"], self.keys("debug")["source-key"])
115123

116-
def test_every_binary_key_input_has_a_main_cache_warmer(self):
117-
"""Check real tracked inputs against push routing, plus inputs absent from today's tree."""
124+
def test_native_input_routing(self):
125+
"""Library inputs warm main; helper tests retain Linux coverage without extra consumers."""
118126
project = Path(__file__).resolve().parents[2]
119-
route = runpy.run_path(str(project / "dev/ci/compute-changes.py"))["compute"]
127+
route = CACHE.CHANGES.compute
120128
_, sources = CACHE.source_inputs(project)
121129
for name in [*sources, ".cargo/config.toml", "rust-toolchain", "contrib/new/native/Cargo.toml"]:
122130
self.assertTrue(route([name], {"name": "push"})["build_linux"], name)
123131
for name in ("contrib/delta/native/src/lib.rs", "contrib/delta/native/Cargo.lock",
124132
"contrib/a/b/native/Cargo.toml", "contrib/a/b/native/x.rs"):
125133
self.assertFalse(route([name], {"name": "push"})["build_linux"], name)
126134
self.assertFalse(route(["contrib/new/native/Cargo.toml"], {"name": "pull_request"})["build_linux"])
135+
for event in ("merge_group", "schedule"):
136+
routed = route(["dev/ci/test-native-cache-key.py"], {"name": event})
137+
self.assertTrue(routed["build_linux" if event == "merge_group" else "build_linux_all_profiles"])
138+
self.assertFalse(any(selected for name, selected in routed.items()
139+
if name.startswith(("spark_", "iceberg_"))))
127140

128141
def test_tools_jdk_flags_and_tracked_build_configuration_invalidate(self):
129142
"""Observed tool/package versions, Java metadata, flags and tracked configs enter keys."""
@@ -142,6 +155,16 @@ def test_tools_jdk_flags_and_tracked_build_configuration_invalidate(self):
142155
self.env["RUSTFLAGS"] += " -Copt-level=1"
143156
self.assertNotEqual(before["binary-key"], self.keys()["binary-key"])
144157
self.env["RUSTFLAGS"] = "-Ctarget-cpu=x86-64-v3 -Clink-arg=-fuse-ld=bfd"
158+
for name in ("CC", "CXX", "CFLAGS", "LDFLAGS", "AR", "PROTOC", "PROTOC_INCLUDE",
159+
"RUSTC_WRAPPER", "CARGO_BUILD_TARGET", "CARGO_PROFILE_CI_OPT_LEVEL",
160+
"CC_x86_64_unknown_linux_gnu", "HOST_CC", "TARGET_CFLAGS",
161+
"HDFS_LIB_DIR", "HADOOP_HOME", "HDFS_STATIC", "DOCS_RS", "PATH"):
162+
with self.subTest(environment=name):
163+
self.env[name] = "build override"
164+
after = self.keys()
165+
for key in ("binary-key", "source-key", "restore-prefix"):
166+
self.assertNotEqual(before[key], after[key])
167+
del self.env[name]
145168
self.write(".cargo/config.toml", "[build]\nincremental = false\n")
146169
subprocess.run(["git", "add", ".cargo/config.toml"], cwd=self.root, check=True)
147170
self.assertNotEqual(before["binary-key"], self.keys()["binary-key"])

0 commit comments

Comments
 (0)