Repository navigation
Conversation
- Planner estimates can partition joins whose actual builds are small, or
collect builds that benefit from parallel hashing.
- Start with CollectLeft and choose the distribution from buffered build
bytes using StageBoundary, retaining the planner's build-side choice.
- Use the SF1-tuned 2.5 MiB total-build threshold.
- Start both input subtrees before awaiting the build so independent stages
retain execution overlap. Preserve the original join's dynamic filters.
- Enable the experiment in the normal TPC-DS command so the benchmark runner
exercises it without custom flags.
- Provide a default-planner opt-out and a byte-threshold override through
both CLI options and environment variables.
- Keep the normal five-iteration timing and result reporting paths.
- Local SF1 with 14 partitions reduced the sum of per-query minima from
6748.5 ms to 6511.0 ms (3.5%).
- Average of two reversed-order passes, five iterations each, none discarded.
- Measured with release-nonlto on an M4 Pro; larger scales remain untested.
- Keep this draft confined to benchmarks: buffer complete builds without
spilling and leave null-aware joins in CollectLeft mode.
Co-authored-by: OpenAI Codex <[email protected]>
Samyak2
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An experiment to prove the value of dynamically choosing CollectLeft/Partitioned join instead of plan-time (the idea is from apache#20847).
Warning: pretty much fully vibe-coded by Astra. I have not looked at the code.
Built on top of apache#25798 (to prove the value of such an API inside datafusion).
Results
On my laptop (MacBook Pro, M4 Pro, 14 cores). The time of TPC-DS SF1 went from 6748.5 ms to 6511.0 ms (~3% improvement).
Not a big improvement overall, but some queries are very interesting.