Repository navigation
Investigate possibility to run heavy CI checks on local #5966
Description
What is the problem the feature request solves?
Followup on #5838
Comet CI is heavyweight, so it would be nice to pass tests locally including Spark tests and also make agents to run local checks before the merge. if done this would dramatically lower chances on broken CI and not spending resources for the rerun
Describe the potential solution
No response
Additional context
No response
Activity
ErikBPF commentedon Sep 22, 2026
Two contributions, both measured in a digest-pinned container.
Pre-commit (opt-in, ~1-5 s): the seven
dev/cichecks (<0.3 s each, offline),cargo fmt --all -- --check, RAT/prettier when present, scalafix syntactic only. Installed bydev/install-hooks.sh.Pre-push (opt-in, aggressive — 5-15 min warm): runs the deterministic CI layers in the pinned image, selecting by changed paths — config checks;
cargo check --locked --profile ci+cargo test -p datafusion-comet;./mvnw -B -Pspark-4.1 -pl spark -Dsuites=<selected> test, or the mergeddev/local-ci.sh. Each prevented red push saves the ~175-min PR-tier run.Measured (28 vCPU / 125 GiB host, image
comet-ci-gate:2026-09-19): config <0.3 s each;cargo check6m32s;cargo test9.1 s run; JVM compile 2m26s; one suite 5m13s; full CI suite set in four JVMs 51m54s sequential (~20 min parallel; each JVM capped at 7 vCPU / 12 GB). A 64 GB / 28 vCPU container was used for the resource-constraint check.Container traps found while doing this: glibc must match between the native build and the test JVM (else
GLIBC_2.38 not foundand a silent fallback);--pids-limit=-1(container default 2048 causespthread_create EAGAIN); cleanjacoco.exec; mount.git; mount cargo/sbt caches; setSPARK_HOME. Four separate JVMs are the faithful CI shape — the combined all-suite single JVM aborts nondeterministically (#6096).Recommendation: pre-commit on by default; pre-push opt-in and documented.