Skip to content

speed up dict cast key width - #11502

Merged
Jefffrey merged 3 commits into
apache:mainfrom
Rich-T-kid:rich-T-kid/dict-cast-key-width
Oct 18, 2026
Merged

Jefffrey merged 3 commits into
apache:mainfrom
Rich-T-kid:rich-T-kid/dict-cast-key-width

Conversation

@Rich-T-kid

@Rich-T-kid Rich-T-kid commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

Dictionary(K1, V) → Dictionary(K2, V) casts (changing only the key width) currently take the general path in dictionary_to_dictionary_cast, which does three things that are unnecessary when the value type is unchanged:

  1. Calls cast_with_options on the values array — dispatch overhead even for identity casts, real work for near-identity casts.
  2. Rebuilds the dictionary's ArrayData graph via into_builder().child_data(...).build_unchecked().
  3. Wraps the result in DictionaryArray::<K2>::from(data), which runs an O(N_rows) bounds-check pass over every key.

If the source dict was valid and the key cast didn't overflow, the result is valid by construction

What changes are included in this PR?

A fast path in dictionary_to_dictionary_cast: when the source value type equals the target value type (and the source isn't a nested dict), delegate to cast_keys_only, which casts the keys array and reuses the values Arc via DictionaryArray::new_unchecked. Narrowing overflow remains a hard error regardless of cast_options.safe, same contract as the general path.

Are these changes tested?

Are there any significant user-facing changes?

no

@github-actions github-actions Bot added the arrow-kernels Crates that transform arrow data (e.g. compute, cast) label Oct 10, 2026
@Rich-T-kid Rich-T-kid changed the title Rich t kid/dict cast key width WIP dict cast key width Oct 10, 2026

@Rich-T-kid Rich-T-kid left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is self evident enough to not warrant benchmarks

@Rich-T-kid
Rich-T-kid marked this pull request as ready for review October 12, 2026 00:37
@Rich-T-kid Rich-T-kid changed the title WIP dict cast key width speed up dict cast key width Oct 12, 2026
@Rich-T-kid
Rich-T-kid force-pushed the rich-T-kid/dict-cast-key-width branch from 5f69b73 to 81ba7eb Compare October 16, 2026 16:00
Comment thread arrow-cast/src/cast/dictionary.rs Resolved
@Rich-T-kid
Rich-T-kid force-pushed the rich-T-kid/dict-cast-key-width branch from 81ba7eb to f94afdf Compare October 17, 2026 19:50
@Rich-T-kid
Rich-T-kid force-pushed the rich-T-kid/dict-cast-key-width branch from f94afdf to 76b8c9c Compare October 17, 2026 20:06
@Jefffrey
Jefffrey added this pull request to the merge queue Oct 18, 2026
@Jefffrey

Copy link
Copy Markdown
Contributor

thanks @Rich-T-kid

Merged via the queue into apache:main with commit 5effeb5 Oct 18, 2026
35 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

arrow-kernels Crates that transform arrow data (e.g. compute, cast) performance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants