Skip to content

SQLQueryTestSuite.ignoreList entries skip the Spark baseline, not just Comet #5570

Closed

Description

@andygrove

SQLQueryTestSuite.ignoreList in the Spark test diffs carries two entries with no linked tracking issue:

// Comet: ORDER BY column has ties; row order is non-deterministic when
// running with high parallelism. Tracked for restoration once Comet
// produces stable ordering for these queries.
"replacing-missing-expression-with-alias.sql",
"in-set-operations.sql",

Present in dev/diffs/4.1.3.diff and dev/diffs/4.2.0.diff.

Why this is worse than it looks

ignoreList is consulted in createScalaTestCase before any Comet-specific or test-type dispatch:

protected def createScalaTestCase(testCase: TestCase): Unit = {
  if (ignoreList.exists(t =>
    testCase.name.toLowerCase(Locale.ROOT).contains(t.toLowerCase(Locale.ROOT)))) {
    ignore(testCase.name) { /* Do nothing */ }
  } else testCase match {

so it registers a ScalaTest ignore unconditionally. The two files are therefore skipped in the ENABLE_COMET=false baseline as well, and in both the regular and analyzer variants: roughly 47 result queries plus three setup statements never run at all, in any mode. Nothing is being verified for them, not even Spark's own behaviour.

The same problem applied to thetasketch.sql / tuplesketch.sql; those were moved to the input-file --SET spark.comet.enabled = false header in #4950, which keeps every query running and compared against the golden file.

What to do

  1. Investigate the actual non-determinism. The stated cause is ORDER BY ties under parallelism; confirm whether it still reproduces on main, and on which queries.
  2. If it does, replace the global ignoreList entries with the --SET spark.comet.enabled = false header used by the other opted-out files, so the Spark baseline and the golden comparison are preserved. Scope it to the affected queries if that is possible without regenerating goldens.
  3. If it does not reproduce, remove the entries.

Surfaced in review of #4950 by @sunchao. Related: #5569 (stale exclusions whose tracking issues are already closed).

Activity

  1. added
    enhancementNew feature or request
    testTesting related
    priority:mediumFunctional bugs, performance regressions, broken features
    area:ciCI/CD, GitHub Actions, build tooling
    on Aug 31, 2026
  2. rich7420 commented on Sep 13, 2026

    @rich7420
    Contributor

    take

  3. added a commit that references this issue on Sep 22, 2026
    f2d038f
  4. added a commit that references this issue on Sep 29, 2026
    8c3eb7b
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:ciCI/CD, GitHub Actions, build toolingenhancementNew feature or requestpriority:mediumFunctional bugs, performance regressions, broken featurestestTesting related

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions