Skip to content

Update projection column by correct index in expression pushdown optimizers - #25431

Merged
pdet merged 3 commits into
duckdb:v2.0-cyanopterafrom
myrrc:myrrc/cast-optimizer-bug-big
Sep 18, 2026
Merged

pdet merged 3 commits into
duckdb:v2.0-cyanopterafrom
myrrc:myrrc/cast-optimizer-bug-big

Conversation

@myrrc

@myrrc myrrc commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

This change fixes a bug in cast/scalar fn pushdown optimizers: whenever column was part of a Projection wrapping a Get, column's index within the projection may not have been the same as column's index within a Get.

This change also fixes a bug in FinalizeScan where a thread without a scan.job (exhausted without scanning anything) got to run FinalizeScan which failed in MultiFileGetPartitionData after.

@myrrc myrrc linked an issue Sep 7, 2026 that may be closed by this pull request
1 of 2 tasks
@duckdb-draftbot
duckdb-draftbot marked this pull request as draft September 7, 2026 13:37
@myrrc
myrrc marked this pull request as ready for review September 7, 2026 13:38
@lnkuiper

lnkuiper commented Sep 9, 2026

Copy link
Copy Markdown
Member

Thanks for the PR! Two (codex-assisted) comments:

  1. Could we let the projection derive its types from its expressions? The index correction fixes the crash, but leaves four places manually updating projection->types in type_pushdown.cpp and scalar_fn_pushdown.cpp.

    The replacement visitors already update the expressions. I would remove those four writes and resolve types once at the end of PushdownOptimize in type_pushdown.hpp:

    if (any_pushed) {
    	Replace(analyses, projections).VisitOperator(*op);
    	op->ResolveOperatorTypes();
    }

    Then GetBinding only needs the scan analysis and scan column index. Keep the existing binding lookup and eligibility checks. The sum(strlen(s)) query in test/optimizer/pushdown/type_pushdown_projection_index.test aborts on the parent ("index 2 within vector of size 1") and returns 100 here.

  2. The data.job guard in multi_file_function.hpp can drop the final aggregate row. Vortex emits that row through FinalizeScan, including for empty inputs. With no jobs, neither exhaustion path calls it anymore.

    I think finalization needs its own output metadata. Keep local state for exhausted scans when reader global state exists, and allow finalization without a job.

@myrrc

myrrc commented Sep 15, 2026

Copy link
Copy Markdown
Contributor Author

Hi @lnkuiper. We can reason about (1) later but I don't want to introduce this change here to keep PR small and merge it faster. I've updated (2).

@duckdb-draftbot
duckdb-draftbot marked this pull request as draft September 15, 2026 12:56
@myrrc
myrrc marked this pull request as ready for review September 15, 2026 12:56
@pdet

pdet commented Sep 18, 2026

Copy link
Copy Markdown
Member

Thanks!

@pdet
pdet added this pull request to the merge queue Sep 18, 2026
Merged via the queue into duckdb:v2.0-cyanoptera with commit 63da6e5 Sep 18, 2026
101 checks passed
JelteF pushed a commit to JelteF/duckdb that referenced this pull request Sep 22, 2026
…mizers (duckdb#25431)

This change fixes a bug in cast/scalar fn pushdown optimizers: whenever
column was part of a Projection wrapping a Get, column's index within
the projection may not have been the same as column's index within a
Get.

This change also fixes a bug in FinalizeScan where a thread without a
`scan.job` (exhausted without scanning anything) got to run FinalizeScan
which failed in MultiFileGetPartitionData after.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Duckdb 2.0

3 participants