Optimize date_bin (2x faster)#10215
Merged
Dandandan merged 2 commits intoapache:mainfrom Apr 25, 2024
Merged
Conversation
As mentioned in the docs for `PrimaryArray::unary` it is faster to apply an infallible operation across both valid and invalid values, rather than branching at every value. 1) Make stride function infallible 2) Use `unary` method This gives this speedup on my machine: Before: 22.345 µs After: 10.558 µs So around 2x faster
7 tasks
alamb
approved these changes
Apr 25, 2024
Contributor
alamb
left a comment
alamb
left a comment
There was a problem hiding this comment.
This is beautiful @simonvandel -- thank you very much 😍 -- it makes a lot of sense to check the option once per batch rather than once per row.
I filed #10228 to track this improvement.
really nice 🥇
Dandandan
approved these changes
Apr 25, 2024
Contributor
|
Nice work, thank you @simonvandel |
ccciudatu
pushed a commit
to hstack/datafusion
that referenced
this pull request
Apr 26, 2024
* add date_bin benchmark * optimize date_bin As mentioned in the docs for `PrimaryArray::unary` it is faster to apply an infallible operation across both valid and invalid values, rather than branching at every value. 1) Make stride function infallible 2) Use `unary` method This gives this speedup on my machine: Before: 22.345 µs After: 10.558 µs So around 2x faster
8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #10228
Rationale for this change
date_bincould be faster.What changes are included in this PR?
As mentioned in the docs for
PrimaryArray::unaryit is faster to apply an infallible operation across both valid and invalid values, rather than branching at every value.unarymethodThis gives this speedup on my machine:
Before: 22.345 µs
After: 10.558 µs
So around 2x faster
Are these changes tested?
Yes, existing tests.
Are there any user-facing changes?
The
date_binfunction runs faster.