Skip to content

[native_datafusion] [Spark SQL Tests] Dynamic Partition Pruning (DPP) not working correctly #3313

New issue
New issue
Closed
Closed
[native_datafusion] [Spark SQL Tests] Dynamic Partition Pruning (DPP) not working correctly#3313
#4092 (+2)
Assignees
andygrove

Description

@andygrove
andygrove
opened on Jan 29, 2026

Summary

3 Spark SQL tests fail due to DPP-related issues when native_datafusion scan is used in auto mode.

Failing Tests

  • DynamicPartitionPruningSuite: "static scan metrics" — scanOption.isDefined was false
  • ExplainSuite: "explain formatted - check presence of subquery in case of DPP" — expected 2 subqueries, found 0
  • Multiple TASK_WRITE_FAILED errors in DPP-related stages

Root Cause

nativeDataFusionScan already has a check for dynamic pruning filters (isDynamicPruningFilter), but these tests use DPP scenarios (e.g. static pruning, subqueries) that may bypass that check. The fallback logic may need to be broadened, or the plan structure produced by native_datafusion may not be compatible with DPP expectations.

Related

Discovered in CI for #3307 (enable native_datafusion in auto scan mode).

Activity

  1. peterxcli commented on Jan 30, 2026

    @peterxcli
    peterxcli
    on Jan 30, 2026
    Member

    take

  2. github-actions
    assigned
    peterxcli
    on Jan 30, 2026
  3. andygrove
    added 2 commits that reference this issue on Feb 4, 2026

    Add IgnoreCometNativeDataFusion annotations for failing native_datafu…

    e93986c

    skip additional native_datafusion test failures in Spark SQL tests

    347376f
  4. andygrove
    assigned
    andygrove
    and unassigned
    peterxcli
    on Mar 13, 2026
  5. andygrove
    added a commit that references this issue on Mar 13, 2026

    fix: detect all DPP forms in native_datafusion scan fallback

    eacc865
  6. andygrove
    closed this as completedin #3694on Mar 17, 2026
  7. andygrove
    added 3 commits that reference this issue on Mar 24, 2026

    chore: Enable spark SQL tests for issues now resolved in native_dataf…

    d7fd22e

    chore: re-enable Spark SQL tests for fixed issues in 3.5.8 diff

    96622cf

    fix: remove unnecessary IgnoreCometNativeDataFusion tags from 3.5.8 diff

    803ae92

  8. andygrove commented on Apr 23, 2026

    @andygrove
    andygrove
    on Apr 23, 2026
    MemberAuthor

    Reopening. When enabling native_datafusion in auto scan mode with Spark 4.0.1 and unignoring the IgnoreCometNativeDataFusion("…/3321") tags, this DPP test still fails:

    • DynamicPartitionPruningSuite: "Subquery reuse across the whole plan" (fails in V1SuiteAEOff; passes in V2SuiteAEOff, V2FilterSuiteAEOff; AEOn variants are skipped via DisableAdaptiveExecution). Failure: List(880, 787, 660) had size 3 instead of expected size 2 — whole-plan subquery reusing not working correctly.
  9. andygrove
    reopened this on Apr 23, 2026
  10. andygrove
    added a commit that references this issue on Apr 26, 2026

    fix: enable Spark 4 SQL tests previously ignored for issues #3313 and #…

    bf3cf9b
  11. andygrove
    closed this as completedin #4092on Apr 26, 2026
  12. andygrove
    added a commit that references this issue on May 3, 2026

    test: unignore Spark 4.1.1 tests gated on closed issues apache#4045, a…

    05a62a2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions