Repository navigation
parquet plain dict bench - #10591
Merged
Jefffrey merged 2 commits intoAug 11, 2026
Merged
parquet plain dict bench#10591
Conversation
Rich-T-kid
changed the title
Rich-T-kid
force-pushed
the
rich-T-kid/parquet-Plain-dict-bench
branch
from
August 7, 2026 21:55
906a25b to
f4ee01b
Compare
Jefffrey
reviewed
Aug 10, 2026
Jefffrey
left a comment
Jefffrey
left a comment
Contributor
There was a problem hiding this comment.
something i noticed was the other benches in this file seem to be at the page iterator or array reader level, whilst this new one is pretty much at the top level with the record batch reader api
is this something to be concerned about, or the behaviour we're trying to benchmark here can only be done via the top level api?
Contributor
Author
@Jefffrey The existing benchmarks measure the time to decode an already dictionary-encoded Parquet page into an Arrow dictionary array. This PR's benchmarks measure converting a plain-encoded (non-dictionary) string column into a dictionary array. I used the top-level API here because it's the less verbose, and mirrors how I'd expect DataFusion to actually use it. That said, it's likely possible to do something similar with the finer-grained APIs. should I update the PR? |
Jefffrey
added
the
development-process
MassivePizza
pushed a commit
to massive-com/arrow-rs
that referenced
this pull request
Aug 19, 2026
# Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. --> - works towards apache#10590 - works towards apache/datafusion#24111 # Rationale for this change see apache#10590 <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> # What changes are included in this PR? introduces benchmarks for reading string values into dictionary arrays <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> # Are these changes tested? n/a <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? If this PR claims a performance improvement, please include evidence such as benchmark results. --> # Are there any user-facing changes? no <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. If there are any breaking changes to public APIs, please call them out. -->
Jefffrey
added a commit
that referenced
this pull request
Aug 22, 2026
# Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. --> - Closes #10590 - builds on top of #10591 - works towards apache/datafusion#24111 # Rationale for this change - see #10590 <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> # What changes are included in this PR? Optimizes plain-encoded string/binary columns read into Dictionary Arrow arrays by eliminating the intermediate StringArray materialization. Previously, all values were decoded into a flat array first, then iterated again to build the dictionary. Now, hashes are computed upfront in a single batch pass and used to deduplicate values directly into the dictionary output buffers, reducing the work to a single pass over the data. The hash scratch buffer is also reused across batches, so allocation overhead amortizes as batch sizes grow. <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> # Are these changes tested? yes existing test cover this functionality. nothing logically has changed <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? If this PR claims a performance improvement, please include evidence such as benchmark results. --> # Are there any user-facing changes? ~~yes, `DictionaryBuffer::into_array()` now takes in scratch vector. this is a breaking change~~ EDIT: no breaking changes, `DictionaryBuffer` isn't part of public API --------- Co-authored-by: Jeffrey Vo <[email protected]>
Rich-T-kid
added a commit
to Rich-T-kid/arrow-rs
that referenced
this pull request
Aug 26, 2026
# Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. --> - Closes apache#10590 - builds on top of apache#10591 - works towards apache/datafusion#24111 # Rationale for this change - see apache#10590 <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> # What changes are included in this PR? Optimizes plain-encoded string/binary columns read into Dictionary Arrow arrays by eliminating the intermediate StringArray materialization. Previously, all values were decoded into a flat array first, then iterated again to build the dictionary. Now, hashes are computed upfront in a single batch pass and used to deduplicate values directly into the dictionary output buffers, reducing the work to a single pass over the data. The hash scratch buffer is also reused across batches, so allocation overhead amortizes as batch sizes grow. <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> # Are these changes tested? yes existing test cover this functionality. nothing logically has changed <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? If this PR claims a performance improvement, please include evidence such as benchmark results. --> # Are there any user-facing changes? ~~yes, `DictionaryBuffer::into_array()` now takes in scratch vector. this is a breaking change~~ EDIT: no breaking changes, `DictionaryBuffer` isn't part of public API --------- Co-authored-by: Jeffrey Vo <[email protected]>
Rich-T-kid
added a commit
to Rich-T-kid/arrow-rs
that referenced
this pull request
Aug 28, 2026
# Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. --> - Closes apache#10590 - builds on top of apache#10591 - works towards apache/datafusion#24111 # Rationale for this change - see apache#10590 <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> # What changes are included in this PR? Optimizes plain-encoded string/binary columns read into Dictionary Arrow arrays by eliminating the intermediate StringArray materialization. Previously, all values were decoded into a flat array first, then iterated again to build the dictionary. Now, hashes are computed upfront in a single batch pass and used to deduplicate values directly into the dictionary output buffers, reducing the work to a single pass over the data. The hash scratch buffer is also reused across batches, so allocation overhead amortizes as batch sizes grow. <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> # Are these changes tested? yes existing test cover this functionality. nothing logically has changed <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? If this PR claims a performance improvement, please include evidence such as benchmark results. --> # Are there any user-facing changes? ~~yes, `DictionaryBuffer::into_array()` now takes in scratch vector. this is a breaking change~~ EDIT: no breaking changes, `DictionaryBuffer` isn't part of public API --------- Co-authored-by: Jeffrey Vo <[email protected]>
Rich-T-kid
added a commit
to Rich-T-kid/arrow-rs
that referenced
this pull request
Sep 2, 2026
# Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. --> - Closes apache#10590 - builds on top of apache#10591 - works towards apache/datafusion#24111 # Rationale for this change - see apache#10590 <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> # What changes are included in this PR? Optimizes plain-encoded string/binary columns read into Dictionary Arrow arrays by eliminating the intermediate StringArray materialization. Previously, all values were decoded into a flat array first, then iterated again to build the dictionary. Now, hashes are computed upfront in a single batch pass and used to deduplicate values directly into the dictionary output buffers, reducing the work to a single pass over the data. The hash scratch buffer is also reused across batches, so allocation overhead amortizes as batch sizes grow. <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> # Are these changes tested? yes existing test cover this functionality. nothing logically has changed <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? If this PR claims a performance improvement, please include evidence such as benchmark results. --> # Are there any user-facing changes? ~~yes, `DictionaryBuffer::into_array()` now takes in scratch vector. this is a breaking change~~ EDIT: no breaking changes, `DictionaryBuffer` isn't part of public API --------- Co-authored-by: Jeffrey Vo <[email protected]>
Rich-T-kid
added a commit
to Rich-T-kid/arrow-rs
that referenced
this pull request
Sep 18, 2026
# Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. --> - Closes apache#10590 - builds on top of apache#10591 - works towards apache/datafusion#24111 # Rationale for this change - see apache#10590 <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> # What changes are included in this PR? Optimizes plain-encoded string/binary columns read into Dictionary Arrow arrays by eliminating the intermediate StringArray materialization. Previously, all values were decoded into a flat array first, then iterated again to build the dictionary. Now, hashes are computed upfront in a single batch pass and used to deduplicate values directly into the dictionary output buffers, reducing the work to a single pass over the data. The hash scratch buffer is also reused across batches, so allocation overhead amortizes as batch sizes grow. <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> # Are these changes tested? yes existing test cover this functionality. nothing logically has changed <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? If this PR claims a performance improvement, please include evidence such as benchmark results. --> # Are there any user-facing changes? ~~yes, `DictionaryBuffer::into_array()` now takes in scratch vector. this is a breaking change~~ EDIT: no breaking changes, `DictionaryBuffer` isn't part of public API --------- Co-authored-by: Jeffrey Vo <[email protected]>
Rich-T-kid
added a commit
to Rich-T-kid/arrow-rs
that referenced
this pull request
Sep 21, 2026
# Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. --> - Closes apache#10590 - builds on top of apache#10591 - works towards apache/datafusion#24111 # Rationale for this change - see apache#10590 <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> # What changes are included in this PR? Optimizes plain-encoded string/binary columns read into Dictionary Arrow arrays by eliminating the intermediate StringArray materialization. Previously, all values were decoded into a flat array first, then iterated again to build the dictionary. Now, hashes are computed upfront in a single batch pass and used to deduplicate values directly into the dictionary output buffers, reducing the work to a single pass over the data. The hash scratch buffer is also reused across batches, so allocation overhead amortizes as batch sizes grow. <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> # Are these changes tested? yes existing test cover this functionality. nothing logically has changed <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? If this PR claims a performance improvement, please include evidence such as benchmark results. --> # Are there any user-facing changes? ~~yes, `DictionaryBuffer::into_array()` now takes in scratch vector. this is a breaking change~~ EDIT: no breaking changes, `DictionaryBuffer` isn't part of public API --------- Co-authored-by: Jeffrey Vo <[email protected]>
Rich-T-kid
added a commit
to Rich-T-kid/arrow-rs
that referenced
this pull request
Sep 25, 2026
# Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. --> - Closes apache#10590 - builds on top of apache#10591 - works towards apache/datafusion#24111 # Rationale for this change - see apache#10590 <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> # What changes are included in this PR? Optimizes plain-encoded string/binary columns read into Dictionary Arrow arrays by eliminating the intermediate StringArray materialization. Previously, all values were decoded into a flat array first, then iterated again to build the dictionary. Now, hashes are computed upfront in a single batch pass and used to deduplicate values directly into the dictionary output buffers, reducing the work to a single pass over the data. The hash scratch buffer is also reused across batches, so allocation overhead amortizes as batch sizes grow. <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> # Are these changes tested? yes existing test cover this functionality. nothing logically has changed <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? If this PR claims a performance improvement, please include evidence such as benchmark results. --> # Are there any user-facing changes? ~~yes, `DictionaryBuffer::into_array()` now takes in scratch vector. this is a breaking change~~ EDIT: no breaking changes, `DictionaryBuffer` isn't part of public API --------- Co-authored-by: Jeffrey Vo <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
see #10590
What changes are included in this PR?
introduces benchmarks for reading string values into dictionary arrays
Are these changes tested?
n/a
Are there any user-facing changes?
no