Skip to content

Commit 65d16df

Browse files
andygroveandygrove
committed
test: run the dynamic Hive UDF suite, document the remaining CI gates
Drop the `IgnoreCometSuite` mixin from `HiveUDFDynamicLoadSuite`. The justification carried by the 4.1 patch is that `hive-test-udfs.jar` is stripped from the release source tag under the ASF binary-artifact policy, so the suite cannot run. Spark 4.2 added `TestHiveUdfsJar`, which builds that jar at test runtime from sources under `src/test/resources/hive-test-udfs/`, so that reason no longer holds and the 4.2 patch should not carry the exclusion. The replacement rationale cited #4098, a closed umbrella for unrelated Spark 4.1 failures which states that Spark SQL tests should not reference it. Also record the two CI registration sites this PR needed but the checklist omitted: the `workflow_dispatch` forced-output list and the `preflight` label allowlist. Both are hard gates that silently skip the whole job.
1 parent e0c7963 commit 65d16df

2 files changed

Lines changed: 14 additions & 29 deletions

File tree

‎dev/diffs/4.2.0.diff‎

Lines changed: 0 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -4222,32 +4222,6 @@ index 1a02a68a069..ee104c6e193 100644
42224222
case h: HiveTableScanExec => h.partitionPruningPred.collect {
42234223
case d: DynamicPruningExpression => d.child
42244224
}
4225-
diff --git a/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveUDFDynamicLoadSuite.scala b/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveUDFDynamicLoadSuite.scala
4226-
index 362353ed0b2..9e0fd8f8545 100644
4227-
--- a/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveUDFDynamicLoadSuite.scala
4228-
+++ b/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveUDFDynamicLoadSuite.scala
4229-
@@ -17,7 +17,7 @@
4230-
4231-
package org.apache.spark.sql.hive
4232-
4233-
-import org.apache.spark.sql.{QueryTest, Row}
4234-
+import org.apache.spark.sql.{IgnoreCometSuite, QueryTest, Row}
4235-
import org.apache.spark.sql.catalyst.expressions.{AttributeReference, Expression}
4236-
import org.apache.spark.sql.hive.HiveShim.HiveFunctionWrapper
4237-
import org.apache.spark.sql.hive.test.{TestHiveSingleton, TestHiveUdfsJar}
4238-
@@ -25,7 +25,11 @@ import org.apache.spark.sql.types.{IntegerType, StringType}
4239-
import org.apache.spark.util.ArrayImplicits._
4240-
import org.apache.spark.util.Utils
4241-
4242-
-class HiveUDFDynamicLoadSuite extends QueryTest with TestHiveSingleton {
4243-
+// Comet: mix in IgnoreCometSuite so these tests are reported as ignored when Comet is
4244-
+// enabled (ENABLE_COMET=true). These Hive UDF tests are not Comet-specific and have not
4245-
+// been validated against Comet; see https://github.com/apache/datafusion-comet/issues/4098.
4246-
+class HiveUDFDynamicLoadSuite extends QueryTest with TestHiveSingleton
4247-
+ with IgnoreCometSuite {
4248-
4249-
case class UDFTestInformation(
4250-
identifier: String,
42514225
diff --git a/sql/hive/src/test/scala/org/apache/spark/sql/hive/InsertSuite.scala b/sql/hive/src/test/scala/org/apache/spark/sql/hive/InsertSuite.scala
42524226
index bd2b256e492..9a3f5b9e160 100644
42534227
--- a/sql/hive/src/test/scala/org/apache/spark/sql/hive/InsertSuite.scala

‎docs/source/contributor-guide/adding_a_new_spark_version.md‎

Lines changed: 14 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -264,9 +264,20 @@ version:
264264
Add a `spark_X_Y` job to `ci.yml` passing `spark-short`, `spark-full`, and
265265
`java`, using the closest existing job as a template. A brand-new version
266266
should be gated behind a `run-spark-X.Y-tests` label so it runs on pushes to
267-
main but stays off the default PR path. Also add a matching `spark_X_Y`
268-
entry to `FILTERS` in `dev/ci/compute-changes.py` and expose it as an output
269-
of the `changes` job, otherwise the new job never fires.
267+
main but stays off the default PR path.
268+
269+
Four more registrations are needed, and the job is silently skipped if any is
270+
missed. In `dev/ci/compute-changes.py`, add a matching `spark_X_Y` entry to
271+
`FILTERS`. In `ci.yml`:
272+
273+
- expose `spark_X_Y` as an output of the `changes` job, otherwise the `if:`
274+
gate reads an empty string on every event;
275+
- add `spark_X_Y` to the `workflow_dispatch` key list in that job's compute
276+
step, otherwise a manual run leaves the output unset even though the job's
277+
own `if:` accepts `workflow_dispatch`;
278+
- add `run-spark-X.Y-tests` to the label allowlist in the `preflight` job's
279+
`if:`, otherwise the whole workflow is skipped for the `labeled` event and
280+
the label never triggers anything.
270281

271282
Before merging, run `make format`, run clippy
272283
(`cd native && cargo clippy --all-targets --workspace -- -D warnings`), and

0 commit comments

Comments
 (0)