Repository navigation
feat(optimizer): preserve profitable Parquet dictionaries for aggregation - #26789
Draft
Rich-T-kid wants to merge 5 commits into
Draft
Rich-T-kid wants to merge 5 commits into
Rich-T-kid wants to merge 5 commits into
Conversation
github-actions
Bot
added
documentation
Rich-T-kid
force-pushed
the
rich-T-kid/wip-cost-based-parquet-dictionary
branch
from
October 27, 2026 04:30
53391c9 to
f63f7dd
Compare
Rich-T-kid
force-pushed
the
rich-T-kid/wip-cost-based-parquet-dictionary
branch
from
October 27, 2026 05:33
f63f7dd to
3a1dc8e
Compare
github-actions
Bot
removed
documentation
Rich-T-kid
changed the title
Contributor
Author
|
run benchmark clickbench_partitioned env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true" |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-T-kid/wip-cost-based-parquet-dictionary (3a1dc8e) to c18e5d9 (merge-base) diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"Results will be posted here when complete File an issue against this benchmark runner |
Contributor
Author
|
show benchmark queue |
|
Hi @Rich-T-kid, you asked to view the benchmark queue (#26789 (comment)).
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing rich-T-kid/wip-cost-based-parquet-dictionary (3a1dc8e) to c18e5d9 (merge-base) diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
Contributor
Author
|
run benchmark tpch tpcds wide_schema env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true" |
Contributor
Author
|
show benchmark queue |
|
Hi @Rich-T-kid, you asked to view the benchmark queue (#26789 (comment)).
File an issue against this benchmark runner |
Contributor
Author
Exhaustive TPC-H dictionary selection validationThis run is not comparing a different Arrow revision, the global RLE dictionary-read default, or unrelated PR code against The column-by-column TPC-H sweep selected only these paths:
Every other tested string column remains |
Contributor
Author
|
run benchmark tpch env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "f7f780a700375bf0f5d944cdd74a96062a7f3462"
env:
DATAFUSION_OPTIMIZER_ENABLE_DICTIONARY_PRESERVATION: "false"
changed:
ref: "f7f780a700375bf0f5d944cdd74a96062a7f3462"
env:
DATAFUSION_OPTIMIZER_ENABLE_DICTIONARY_PRESERVATION: "true"
resources:
cpu: "12"
memory: "65Gi"
arch: arm64 |
Contributor
Author
|
show benchmark queue |
|
Hi @Rich-T-kid, you asked to view the benchmark queue (#26789 (comment)).
File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing f7f780a (f7f780a) to f7f780a diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "f7f780a700375bf0f5d944cdd74a96062a7f3462"
env:
DATAFUSION_OPTIMIZER_ENABLE_DICTIONARY_PRESERVATION: "false"
changed:
ref: "f7f780a700375bf0f5d944cdd74a96062a7f3462"
env:
DATAFUSION_OPTIMIZER_ENABLE_DICTIONARY_PRESERVATION: "true"
resources:
cpu: "12"
memory: "65Gi"
arch: "arm64"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing f7f780a (f7f780a) to f7f780a diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "f7f780a700375bf0f5d944cdd74a96062a7f3462"
env:
DATAFUSION_OPTIMIZER_ENABLE_DICTIONARY_PRESERVATION: "false"
changed:
ref: "f7f780a700375bf0f5d944cdd74a96062a7f3462"
env:
DATAFUSION_OPTIMIZER_ENABLE_DICTIONARY_PRESERVATION: "true"
resources:
cpu: "12"
memory: "65Gi"
arch: "arm64"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
Contributor
Author
|
The first same-SHA TPC-H run showed the intended wins in Q1 and Q19, but Q16 moved in the wrong direction under the stricter zero-regression gate:
Commit c37a2aa removes the Q16-specific filtered grouping path while retaining the Q1 source-filtered grouping path and Q19 equality/IN filter-only path. It also adds negative coverage for the exact Q16 NOT LIKE shape at catalog selection and physical optimization. A fresh Q1-Q22 plan audit with RLE dictionary reads disabled now has only Q1 visibly changing. Q16 and the other 20 queries are byte-identical with the optimizer off; Q19 remains textually identical because its dictionary representation is internal to the scan/filter boundary. The next benchmark compares this exact SHA against itself with only DATAFUSION_OPTIMIZER_ENABLE_DICTIONARY_PRESERVATION toggled. |
Contributor
Author
|
run benchmark tpch env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "c37a2aada6b21adc699c3fd6d1ddb648c130ec84"
env:
DATAFUSION_OPTIMIZER_ENABLE_DICTIONARY_PRESERVATION: "false"
changed:
ref: "c37a2aada6b21adc699c3fd6d1ddb648c130ec84"
env:
DATAFUSION_OPTIMIZER_ENABLE_DICTIONARY_PRESERVATION: "true"
resources:
cpu: "12"
memory: "65Gi"
arch: arm64 |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing c37a2aa (c37a2aa) to c37a2aa diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "c37a2aada6b21adc699c3fd6d1ddb648c130ec84"
env:
DATAFUSION_OPTIMIZER_ENABLE_DICTIONARY_PRESERVATION: "false"
changed:
ref: "c37a2aada6b21adc699c3fd6d1ddb648c130ec84"
env:
DATAFUSION_OPTIMIZER_ENABLE_DICTIONARY_PRESERVATION: "true"
resources:
cpu: "12"
memory: "65Gi"
arch: "arm64"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing c37a2aa (c37a2aa) to c37a2aa diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "c37a2aada6b21adc699c3fd6d1ddb648c130ec84"
env:
DATAFUSION_OPTIMIZER_ENABLE_DICTIONARY_PRESERVATION: "false"
changed:
ref: "c37a2aada6b21adc699c3fd6d1ddb648c130ec84"
env:
DATAFUSION_OPTIMIZER_ENABLE_DICTIONARY_PRESERVATION: "true"
resources:
cpu: "12"
memory: "65Gi"
arch: "arm64"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
Contributor
Author
|
The c37a2aa same-SHA rerun passes the TPC-H timing gate with RLE dictionary reads disabled:
The bot reports 2 faster, 0 slower, 20 unchanged, and 0 failures. Total minimum time improves 722.40 -> 720.50 ms (-0.26%); total mean improves 737.70 -> 734.64 ms (-0.41%). The column search is exhaustive for this workload: 29 TPC-H string columns accounted for, all 23 read by Q1-Q22 tested across 52 query/column pairs. The last plausible Q19 compound was also rejected: adding p_brand or p_container to l_shipinstruct+l_shipmode regressed about 0.97% each, and adding both was unstable with essentially no pooled-median gain. Memory caveat: Q1 again increases its MemoryPool peak from 30.0 to 40.1 MiB (+10.1 MiB), although the suite maximum remains 167.1 MiB, peak RSS falls from 1.2 to 1.1 GiB, average RSS falls from 494.3 to 485.2 MiB, and neither side spills. The PR remains draft. |
Contributor
Author
|
This control isolates raw RLE dictionary reads after the Arrow dictionary-value conversion fix. Both sides use exact SHA |
Contributor
Author
|
run benchmark tpch tpcds clickbench_partitioned env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
DATAFUSION_EXECUTION_PARQUET_SCHEMA_FORCE_VIEW_TYPES: "true"
DATAFUSION_OPTIMIZER_ENABLE_DICTIONARY_PRESERVATION: "false"
baseline:
ref: "c37a2aada6b21adc699c3fd6d1ddb648c130ec84"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
changed:
ref: "c37a2aada6b21adc699c3fd6d1ddb648c130ec84"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
resources:
cpu: "12"
memory: "65Gi"
arch: arm64 |
Contributor
Author
|
show benchmark queue |
|
Hi @Rich-T-kid, you asked to view the benchmark queue (#26789 (comment)).
File an issue against this benchmark runner |
Contributor
Author
|
show benchmark queue |
|
Hi @Rich-T-kid, you asked to view the benchmark queue (#26789 (comment)).
File an issue against this benchmark runner |
Contributor
Author
|
show benchmark queue |
|
Hi @Rich-T-kid, you asked to view the benchmark queue (#26789 (comment)).
File an issue against this benchmark runner |
Contributor
Author
|
show benchmark queue |
|
Hi @Rich-T-kid, you asked to view the benchmark queue (#26789 (comment)).
File an issue against this benchmark runner |
Contributor
Author
|
show benchmark queue |
|
Hi @Rich-T-kid, you asked to view the benchmark queue (#26789 (comment)).
File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing c37a2aa (c37a2aa) to c37a2aa diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_SCHEMA_FORCE_VIEW_TYPES: "true"
DATAFUSION_OPTIMIZER_ENABLE_DICTIONARY_PRESERVATION: "false"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "c37a2aada6b21adc699c3fd6d1ddb648c130ec84"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
changed:
ref: "c37a2aada6b21adc699c3fd6d1ddb648c130ec84"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
resources:
cpu: "12"
memory: "65Gi"
arch: "arm64"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing c37a2aa (c37a2aa) to c37a2aa diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_SCHEMA_FORCE_VIEW_TYPES: "true"
DATAFUSION_OPTIMIZER_ENABLE_DICTIONARY_PRESERVATION: "false"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "c37a2aada6b21adc699c3fd6d1ddb648c130ec84"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
changed:
ref: "c37a2aada6b21adc699c3fd6d1ddb648c130ec84"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
resources:
cpu: "12"
memory: "65Gi"
arch: "arm64"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing c37a2aa (c37a2aa) to c37a2aa diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_SCHEMA_FORCE_VIEW_TYPES: "true"
DATAFUSION_OPTIMIZER_ENABLE_DICTIONARY_PRESERVATION: "false"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "c37a2aada6b21adc699c3fd6d1ddb648c130ec84"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
changed:
ref: "c37a2aada6b21adc699c3fd6d1ddb648c130ec84"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
resources:
cpu: "12"
memory: "65Gi"
arch: "arm64"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing c37a2aa (c37a2aa) to c37a2aa diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_SCHEMA_FORCE_VIEW_TYPES: "true"
DATAFUSION_OPTIMIZER_ENABLE_DICTIONARY_PRESERVATION: "false"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "c37a2aada6b21adc699c3fd6d1ddb648c130ec84"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
changed:
ref: "c37a2aada6b21adc699c3fd6d1ddb648c130ec84"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
resources:
cpu: "12"
memory: "65Gi"
arch: "arm64"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing c37a2aa (c37a2aa) to c37a2aa diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_SCHEMA_FORCE_VIEW_TYPES: "true"
DATAFUSION_OPTIMIZER_ENABLE_DICTIONARY_PRESERVATION: "false"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "c37a2aada6b21adc699c3fd6d1ddb648c130ec84"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
changed:
ref: "c37a2aada6b21adc699c3fd6d1ddb648c130ec84"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
resources:
cpu: "12"
memory: "65Gi"
arch: "arm64"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing c37a2aa (c37a2aa) to c37a2aa diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_SCHEMA_FORCE_VIEW_TYPES: "true"
DATAFUSION_OPTIMIZER_ENABLE_DICTIONARY_PRESERVATION: "false"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "c37a2aada6b21adc699c3fd6d1ddb648c130ec84"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
changed:
ref: "c37a2aada6b21adc699c3fd6d1ddb648c130ec84"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
resources:
cpu: "12"
memory: "65Gi"
arch: "arm64"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #24117.
This adds a default-off physical optimizer rule that preserves profitable Parquet string dictionaries through aggregation:
Before:
After, when the cost and safety checks pass:
The rule fails closed. It requires Parquet footer metadata proving that every data page is dictionary encoded, native
Utf8dictionary values, at least 16 decoded value bytes per row, a decoded-to-Parquet byte ratio of at least 4:1, and at least 8 MiB of estimated string work at the aggregate. It traces group-key payloads through projections and safe inner equijoins, and skips filtered, limited, ranged, schema-overridden, or otherwise unsupported scan paths. SQL-visible schemas do not change.Physical encoding metadata is collected only while the option is enabled and cached with the normal file statistics. This draft temporarily pins Arrow to apache/arrow-rs#12161, which lets the Parquet reader emit the requested
Dictionary<Int32, Utf8>directly instead of repairing the value type with a per-batch cast.Benchmarks
Release build, Arrow PR head
eff59bb9, flag off vs on. Full TPC-H, TPC-DS, and partitioned ClickBench screens used alternatingABBA + BAABprocesses and discarded the first iteration in each process. Every apparent regression above 2% was rerun as an isolated query with longer alternating runs.No isolated regression remained above 2%:
Validation
Dictionary<Int32, Utf8>is returned directly with that type.avro,integration-tests,extended_tests, formatting, typos, catalog-listing tests, and focused Parquet/optimizer tests pass.