Repository navigation
Conversation
When no branch value, and no WHEN after the first, can fail, evaluate them for the whole batch and take each row's value from its branch with typed merges, instead of filtering the batch for each branch and merging the partial results through MutableArrayData. A branch that can fail still goes to DataFusion's CaseExpr, which evaluates it only for the rows that choose it. The planner also stops wrapping each branch in a cast to the type it already has. Closes apache#3025.
…branch Take the branches, Display, fmt_sql, Hash and PartialEq from the wrapped CaseExpr instead of keeping a second copy, and keep the result type with the eager decision instead of working it out again for every batch. When one array branch is the only one with valid values, as in `CASE WHEN p THEN col END` and Comet's divisor guard, keep its buffers and only clear validity, as nullif does. Those shapes get 1.4x faster. The rules for which casts and float normalizations cannot fail now sit on Cast and NormalizeNaNAndZero, next to the code they describe.
rich7420
added
run-spark-4.1-tests
github-actions
Bot
added
enhancement
This was referenced Sep 30, 2026
This was referenced Oct 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Historical integration evidence, status updated 2026-10-07. The measurements and completed CI below apply only to
4528a2a67against main62ed9d16d. Upstream #6350, #6396, and #6397 have all merged. This candidate is no longer waiting on those prerequisites and does not validate the later published feature refreshc34480068. Current delivery and final-head verification belong to upstream #6180.Current delivery plan.
Historical scope and commit-specific validation (retained unchanged)
Which issue does this PR close?
Related to apache#6093 and apache#6180. Fork integration verification only; this PR does not close an upstream issue.
Rationale for this change
The AtLeastNNonNulls query workload exposed the cost of conditional expressions and filtering unused output columns. Shared CASE/IF evaluation has now merged upstream in apache#6350. This candidate verifies the feature and the remaining projection prerequisite against actual main before updating upstream apache#6180.
What changes are included in this PR?
The candidate is
4528a2a67cff87c06179337ce4e1b35de2355839, with pinned actual-main base62ed9d16d547fed1ac25260a1910893f076ab638. This advances the previous published fork candidate through a normal merge, preserving its history.CASE WHENandIF(up to 12x faster) apache/datafusion-comet#6350, including its UTC timestamp reconciliation fix and regression. CASE/IF and float normalization source match actual main; no retired standalone IF fast path remains.256239ebe) and the independent IF semantic tests from test: cover IF branch selection and short-circuiting apache/datafusion-comet#6397. test: cover IF branch selection and short-circuiting apache/datafusion-comet#6397 is not an implementation prerequisite.Relative to actual main, the combined candidate changes 18 files. Relative to a prerequisite-only tree (main + apache#6396 + apache#6397's SQL tests), it changes 13 AtLeastNNonNulls files. Upstream apache#6180 remains Draft until apache#6396 merges; its final branch will be updated from actual main and independently verified.
How are these changes tested?
Local verification for this exact tree passed on Apple M3 Pro / JDK 21:
438cf2533d3f4af082aa3e8355a511063053deb60fef0167ab1a6672163c5817.-D warnings, cargo fmt, Spotless and RAT.na.drop(16), count and all 32 cleaned sums. Twelve rounds alternated mode order; medians exclude the first three. The second process reversed initial mode and NaN-rate order. Spark used local[1], release native code with target-cpu=native.The 5% NaN comparisons range from near parity (1.1%) to 11.2% lower median time. Variance is material; these are combined-candidate measurements for this workload, not a fixed or universal speedup. The original issue author's workload remains unverified.
The permanent
CometPredicateExpressionBenchmark atleastnnonnulls --focus-doublealso passed its answer/path checks. Native/fallback averages were 320/338 ms at 0% NaN and 325/354 ms at 5% NaN; the latter standard deviations were 11/37 ms.Earlier full fork CI passed at old head
7e5e2a73c, including all seven Spark 4.1 SQL shards selected by that older workflow. Current-head fork CI now passes at4528a2a67, including Required Checks, all Comet suites, Rust tests, strict Spark 3.5/JDK17 compilation, benchmark checks, TPC-H/TPC-DS and all nine Spark 4.1 SQL shards. The earlier same-head run cancelled during this push is superseded by that completed run. Other Spark runtime profiles, Iceberg and macOS suites were skipped by this run and are not covered by its verdict. Upstream apache#6180 still waits for apache#6396 to merge and requires verification after its final actual-main update.