Skip to content

fix(parquet): honor requested dictionary value type - #12126

Closed
Rich-T-kid wants to merge 1 commit into
apache:mainfrom
Rich-T-kid:rich-T-kid/parquet-dictionary-requested-value-type
Closed

Rich-T-kid wants to merge 1 commit into
apache:mainfrom
Rich-T-kid:rich-T-kid/parquet-dictionary-requested-value-type

Conversation

@Rich-T-kid

@Rich-T-kid Rich-T-kid commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

An unannotated Parquet BYTE_ARRAY is inferred as Arrow Binary, even when a supplied reader schema requests Dictionary<K, Utf8>. The dictionary reader currently chooses its value type from the Parquet annotation, so it produces Dictionary<K, Binary>. DataFusion must then discard the encoded representation and rebuild it row by row:

Parquet dictionary page
  values = [b"Chrome", b"Firefox"]
  keys   = [0, 0, 1, 0]
              |
              v
Dictionary<K, Binary>
              |
              | expand keys, validate/convert every row, hash every row
              v
["Chrome", "Chrome", "Firefox", "Chrome"]
              |
              | rebuild dictionary
              v
Dictionary<K, Utf8>

This matters for DataFusion's cost-based dictionary encoding (CBDE) work in apache/datafusion#24111. CBDE can only benefit from the Parquet dictionary when the scan preserves it in the requested logical string type:

Parquet RLE dictionary
        -> Arrow Dictionary<K, Utf8>
        -> DataFusion scan
        -> CBDE can keep the representation when it is beneficial

Without this reader support, the scan pays the expansion and rebuild cost before CBDE can use the dictionary.

What changes are included in this PR?

Decode dictionary pages using their physical Parquet value type, then convert only the distinct values to the value type requested by the supplied Arrow schema:

values: [b"Chrome", b"Firefox"] -> ["Chrome", "Firefox"]
keys:   [0, 0, 1, 0]            -> [0, 0, 1, 0]
                                          unchanged

The converted dictionary values are reused across batches, producing Dictionary<K, Utf8> directly without expanding the keys. Plain-encoded pages continue to use the existing dictionary-building fallback.

The same path handles compatible string/binary and view-type requests, including UTF-8 validation when a binary dictionary is requested as a string dictionary.

Are these changes tested?

Yes. Regression tests cover:

  • unannotated binary dictionaries requested as Dictionary<Int32, Utf8> across multiple batches
  • preserved keys and shared dictionary values
  • invalid UTF-8 input
  • annotated UTF-8 dictionaries requested as binary

DataFusion validation

Are there any significant user-facing changes?

Supplied dictionary schemas now produce columns with the requested compatible string or binary value type. There are no public API changes.

@boring-cyborg boring-cyborg Bot added the parquet Changes to the parquet crate label Nov 8, 2026
@Rich-T-kid
Rich-T-kid marked this pull request as draft November 8, 2026 18:12
@Rich-T-kid
Rich-T-kid force-pushed the rich-T-kid/parquet-dictionary-requested-value-type branch from ea86d97 to 80e9f66 Compare November 8, 2026 18:18
@Rich-T-kid
Rich-T-kid force-pushed the rich-T-kid/parquet-dictionary-requested-value-type branch from 80e9f66 to 83ae72c Compare November 8, 2026 18:24
@Rich-T-kid
Rich-T-kid marked this pull request as ready for review November 8, 2026 19:13
@Rich-T-kid
Rich-T-kid marked this pull request as draft November 8, 2026 19:18
@Rich-T-kid
Rich-T-kid marked this pull request as ready for review November 8, 2026 19:20
@Rich-T-kid
Rich-T-kid marked this pull request as draft November 8, 2026 19:34
@Rich-T-kid

Copy link
Copy Markdown
Contributor Author

Biggest query-level improvements in the dedicated runs:

  • TPC-DS: Q72 (1.11x), Q74 (1.06x), Q73 (1.05x)
  • ClickBench: Q39 (1.06x)
  • TPC-H: no faster queries in the primary comparison

@Rich-T-kid
Rich-T-kid marked this pull request as ready for review November 8, 2026 20:05
@Rich-T-kid

Copy link
Copy Markdown
Contributor Author

DataFusion E2E A/B: apache/datafusion#27249, comparing Arrow parent 62a3c43f with this PR at 83ae72c6, with RLE dictionary reads enabled on both sides.

  • TPC-H: 796.01 → 799.79 ms (+0.5%).
  • TPC-DS: 9.02 → 8.88 s (-1.6%).
  • ClickBench: 38.26 → 38.16 s (-0.3%).

Largest wins were TPC-DS Q72 (1573.84 → 1411.64 ms, -10.3%), TPC-DS Q74 (236.38 → 221.98 ms, -6.1%), and ClickBench Q39 (296.02 → 278.66 ms, -5.9%). The largest regression was TPC-DS Q28 (57.62 → 61.94 ms, +7.5%).

@Rich-T-kid Rich-T-kid closed this Nov 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

parquet Changes to the parquet crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Parquet reader returns Binary dictionary values when a supplied schema requests Utf8

1 participant