test: contrib Delta own-suite regression harness [Delta contrib split, part 7] - #11

…, part 7]

Adds the harness that runs Delta Lake's OWN test suites against a Comet build,
proving the contrib reads stay correct across the full upstream Delta test
corpus (not just Comet's contrib suites from part 6).

What this adds (test/CI tooling only -- no production, native, or main code):
- contrib/delta/dev/diffs/{3.3.2,4.0.0,4.1.0}.diff: per-Delta-version patches
  that wire a locally-published Comet into Delta's sbt build and enable Comet on
  Delta's own test session. Verified to `git apply --check` cleanly against the
  upstream v3.3.2 / v4.0.0 / v4.1.0 source tags.
- contrib/delta/dev/run-regression.sh: clones the target Delta tag, applies the
  diff, builds+installs Comet for the matching Spark profile (FAST=1 skips
  spotless/RAT/javadoc and uses the release dylib), and runs the smoke / lite /
  full sweep. Derives cometVersion from the comet pom so the diff's injected
  version stays in lockstep with the checkout.
- contrib/delta/dev/run-test.sh: local helper to rerun a single Delta suite (or
  a -z name filter) against an already-prepared regression checkout.
- .github/workflows/delta_regression_test.yml: smoke job per (Spark, Delta) pair
  on PRs, plus a fuller filtered sweep on push-to-main / dispatch.

No -Dspark.version=4.1.1 pin is needed here (unlike part 6's contrib unit tests):
the regression run executes Delta's OWN test suite, where Delta's build selects
its own compatible Spark version (Delta 4.1.0 -> Spark 4.1.0 via its
CrossSparkVersions), so delta-spark 4.1.0 never runs on the pom's 4.1.2 and the
removed-IgnoreCachedData skew cannot occur. The Comet install compiles against
4.1.2 but references no removed API.

Robustness/portability hardening (review-driven, improving on the integration branch):
- run-regression.sh: `|| true` on the cometVersion auto-derive so a future regex
  miss reaches the documented WARNING+hardcoded-fallback instead of aborting under
  `set -o pipefail`; graceful re-clone when a reused DELTA_WORKDIR lacks the
  requested tag (was a hard `git checkout` abort); removed the dead `2.4.0` case
  (no 2.4.0.diff exists); documented the COMET_PUBLISH_DIR/diff-path coupling.
- run-test.sh: replaced a hardcoded personal JAVA_HOME fallback with the macOS
  java_home helper, falling back to `java` on PATH (no developer-specific path).
- delta_regression_test.yml: scoped the PR trigger to Delta-affecting paths
  (contrib, native, the SPI/rule/serde wiring, poms) so unrelated PRs skip the
  expensive build+clone+smoke; corrected the misleading "matching smoke cell"
  comment (GitHub `needs` is job-level) and made the smoke-on-PR / full-on-merge
  cost tradeoff explicit.

Verification: all three diffs apply cleanly to the real upstream Delta tags; both
scripts pass `bash -n`; the workflow YAML parses; run-regression.sh carries the
pipefail-safe cometVersion auto-derive. The diffs are byte-identical to the
integration branch, where the sweeps were actually executed. A local full/smoke
run is deferred (the full Delta-3.3.2 sweep is ~29h and the gated heavyweight per
the plan; this host is disk-constrained) and runs via the workflow.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01BtErWgRQKCDRAg8Mk6qR4G

Post-review fix (found running the 4.1 full regression locally): the SBT_REPO_OVERRIDE
array expansion used the bare `"${SBT_REPO_OVERRIDE[@]}"`, which is an empty array on
the 4.x cells -- and macOS's stock bash 3.2 treats `"${empty[@]}"` as an unbound variable
under `set -u` (bash >= 4.4 / Linux CI tolerates it, so CI never hit it). Switched the four
call sites to the portable `${SBT_REPO_OVERRIDE[@]+"${SBT_REPO_OVERRIDE[@]}"}` alternation.

…n-suite scan finders [themeB, folded into A.6b]