Repository navigation
perf(arrow): coalesce dictionary keys without concat - #11743
Merged
Rich-T-kid merged 8 commits intoNov 6, 2026
Merged
Conversation
github-actions
Bot
added
the
arrow-kernels
Rich-T-kid
marked this pull request as draft
October 20, 2026 18:12
Rich-T-kid
force-pushed
the
rich-T-kid/coalescer-dictionary-filter
branch
2 times, most recently
from
October 20, 2026 20:44
cb67d1c to
ef2a69c
Compare
Contributor
Author
|
run benchmark coalesce_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing rich-T-kid/coalescer-dictionary-filter (ef2a69c) to d58a0fc (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: dict
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
Contributor
Author
|
run benchmark coalesce_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing rich-T-kid/coalescer-dictionary-filter (24e3433) to 01a9917 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: dict
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing rich-T-kid/coalescer-dictionary-filter (ef2a69c) to d58a0fc (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: dict
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-11mn6c6n-aed7f489-bblw Instance: c4a-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing rich-T-kid/coalescer-dictionary-filter (24e3433) to 01a9917 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: dict
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-11mn6c6n-356aa77f-8lfj Instance: c4a-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
Rich-T-kid
commented
Oct 21, 2026
Rich-T-kid
left a comment
Rich-T-kid
left a comment
Contributor
Author
There was a problem hiding this comment.
I don't love that this is is not type agnostic
Contributor
Author
|
run benchmark coalesce_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing rich-T-kid/coalescer-dictionary-filter (d7a63c2) to 01a9917 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: dict
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
Contributor
Author
|
run benchmark coalesce_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing rich-T-kid/coalescer-dictionary-filter (d7a63c2) to 01a9917 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: dict
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing rich-T-kid/coalescer-dictionary-filter (d7a63c2) to 01a9917 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: dict
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-vaa38lox-43786d54-p5k6 Instance: c4a-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing rich-T-kid/coalescer-dictionary-filter (d7a63c2) to 01a9917 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: dict
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-7c3qk7tq-fad8f8c4-49kp Instance: c4a-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
Contributor
Author
|
run benchmark coalesce_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing rich-T-kid/coalescer-dictionary-filter (11659ed) to df4dc3e (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: dict
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing rich-T-kid/coalescer-dictionary-filter (11659ed) to df4dc3e (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: dict
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-102okzyr-dc479920-2hqj Instance: c4a-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
Rich-T-kid
force-pushed
the
rich-T-kid/coalescer-dictionary-filter
branch
from
October 25, 2026 22:20
d4cca08 to
7f46592
Compare
github-actions
Bot
added
the
arrow-core
Contributor
Author
|
run benchmark coalesce_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing rich-T-kid/coalescer-dictionary-filter (7f46592) to e7c6d29 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: dict
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing rich-T-kid/coalescer-dictionary-filter (7f46592) to e7c6d29 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: dict
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-mswc4tih-e9f02941-sx7r Instance: c4a-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
Contributor
Author
|
run benchmark coalesce_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing rich-T-kid/coalescer-dictionary-filter (7f46592) to e7c6d29 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: 'take: single_dict, 8192, nulls: 0.1, selectivity: 0.001'
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing rich-T-kid/coalescer-dictionary-filter (7f46592) to e7c6d29 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: 'take: single_dict, 8192, nulls: 0.1, selectivity: 0.001'
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-qrtsme6q-e87259ad-854z Instance: c4a-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
Rich-T-kid
force-pushed
the
rich-T-kid/coalescer-dictionary-filter
branch
from
October 26, 2026 04:43
7f46592 to
23e5277
Compare
github-actions
Bot
removed
the
arrow-core
Rich-T-kid
commented
Nov 3, 2026
Rich-T-kid
left a comment
Rich-T-kid
left a comment
Contributor
Author
There was a problem hiding this comment.
since this PR has been through a few iterations thought id point this out
Jefffrey
reviewed
Nov 4, 2026
Contributor
Author
|
run benchmark coalesce_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing rich-T-kid/coalescer-dictionary-filter (d87b6e9) to b65bab9 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: dict
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
Contributor
Author
|
run benchmark coalesce_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing rich-T-kid/coalescer-dictionary-filter (d87b6e9) to b65bab9 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: dict
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
Contributor
Author
|
run benchmark coalesce_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing rich-T-kid/coalescer-dictionary-filter (d87b6e9) to b65bab9 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: dict
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing rich-T-kid/coalescer-dictionary-filter (d87b6e9) to b65bab9 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: dict
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-1qhnggp6-da0f0032-pb7s Instance: c4a-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing rich-T-kid/coalescer-dictionary-filter (d87b6e9) to b65bab9 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: dict
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-1qhnggp6-b1096d2e-t57n Instance: c4a-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing rich-T-kid/coalescer-dictionary-filter (d87b6e9) to b65bab9 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: dict
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-1qhnggp6-b1096d2e-vwz7 Instance: c4a-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
Jefffrey
reviewed
Nov 5, 2026
Jefffrey
left a comment
Jefffrey
left a comment
Contributor
There was a problem hiding this comment.
(identified by codex)
theres a regression for when merging dictionary values that dont point to the same arc, but the underlying array buffers are the same (i.e. via Arc::new(values.clone()) instead of an Arc::clone); see example test:
#[test]
fn coalesce_dictionary_with_shared_buffers_in_separate_arcs() {
let values = StringArray::from_iter_values((0..128).map(|i| format!("value-{i}")));
let make_batch = || {
let dictionary = DictionaryArray::<Int8Type>::new(
Int8Array::from_iter_values(0..=127),
// Each call creates a separate Arc, but shares the underlying buffers.
Arc::new(values.clone()),
);
RecordBatch::try_from_iter([("dict", Arc::new(dictionary) as ArrayRef)]).unwrap()
};
let first = make_batch();
let mut coalescer = BatchCoalescer::new(first.schema(), 256);
coalescer.push_batch(first).unwrap();
coalescer.push_batch(make_batch()).unwrap();
let output = coalescer.next_completed_batch().unwrap();
assert_eq!(output.num_rows(), 256);
let dictionary = output.column(0).as_dictionary::<Int8Type>();
let output_values = dictionary.values().as_string::<i32>();
for (row, key) in dictionary.keys_iter().enumerate() {
assert_eq!(output_values.value(key.unwrap()), values.value(row % 128));
}
}i guess its because currently this PR is checking based on arc equality, and not checking buffer equality of the underlying array the arc points to. it seems we have functionality for that, e.g. here
arrow-rs/arrow-select/src/dictionary.rs
Lines 150 to 165 in ea3c266
we could fix the check to also consider buffer equality if the arc equality fails, though i wonder if this case should be handled by a fallback anyway during merge 🤔
Contributor
Author
|
makes sense, added it in the latest commit. this may actually be useful for other places where we check the arc-ptr for equality. i guess an easier first solution is to look for places where do |
Rich-T-kid
added a commit
to Rich-T-kid/arrow-rs
that referenced
this pull request
Nov 6, 2026
# Which issue does this PR close? Follow-up to the pointer identity review on apache#11743. # Rationale for this change `Arc::ptr_eq` misses arrays wrapped in different `Arc`s even when they share the same backing buffers, such as `Arc::new(array.clone())`. # What changes are included in this PR? Add a crate-private `child_values_ptr_eq` helper in `arrow-select`. It checks `Arc` identity first, compares byte-array offsets, values, and null buffers directly, and falls back to `ArrayData::ptr_eq` for other types. The helper stays internal for use by apache#11743 without changing the existing dictionary merge heuristic or adding public API. # Are these changes tested? The tests cover same-`Arc` values, shallow clones sharing backing buffers, empty primitive arrays, and `NullArray`s without allocated storage. They reject logically equal arrays with independent buffers, different slices or lengths, different nested children, and different `StringView` payload buffers. Shared run-end encoded values are also recognized across different run boundaries. # Are there any significant user-facing changes? No.
Rich-T-kid
force-pushed
the
rich-T-kid/coalescer-dictionary-filter
branch
from
November 6, 2026 02:31
a684be7 to
afc7ef9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #11719.
Rationale for this change
Dictionary arrays currently use the generic coalescer path, which retains input arrays and calls
concatwhen a batch is emitted. Dictionary-heavy workloads therefore repeat value merging and hashing and keep more input memory alive than necessary.What changes are included in this PR?
This adds a dictionary-specific in-progress array for byte-array values. It retains zero-copy slices of the key buffers and stores each values
Arconce.Merging distinct values arrays
When every chunk uses the same values
Arc,finishreuses it directly. Otherwise,merge_dictionary_values_with_masksmarks only the referenced values, builds a local-to-output key mapping for each source, and callsinterleaveonce to write the final values array. This only gathers dictionary values; the stored keys are remapped directly into the final key buffer, with no filtered dictionary arrays orconcatin between.For example, start with three dictionary arrays:
Each call retains a multi-row key slice in its original local key space:
At flush, only the referenced values are appended and each local key is mapped to its output position:
The direct path keeps repeated values from different sources because the 12 referenced values fit the key type. If they did not fit, the fallback would deduplicate the shared values from C:
Are these changes tested?
The tests cover shared and distinct values arrays, alternating sources, sparse remapping, null keys, row order, direct values reuse, and key-type overflow fallback.
Two final-head hosted runs show large filter and take gains, smaller interleave gains, and peak memory dropping from 63 MiB to 23 MiB.
Are there any significant user-facing changes?
No.