Skip to content

Commit 1ffb342

Browse files
LiaCastanedaOmega359
authored andcommitted
bench: cover dictionary group keys with a shared values array (apache#25198)
## Which issue does this PR close? [Related to apache#24822. No issue closed.](apache#25185) ## Rationale for this change The benchmarks in this file build a fresh values array per batch and cap cardinality at the batch size. Neither shape matches a real plan: `take` and `filter` clone the values `Arc` and rewrite only the keys, so batches downstream of a repartition share one values array, and because it is never compacted its cardinality can far exceed the rows in a batch. Per-batch work that scales with dictionary cardinality is invisible under the current parameters. ## What changes are included in this PR? Adds `bench_shared_values_arc`: 32 batches of 8192 rows sharing one values array, at cardinalities 8192, 100k and 500k. ## What is the testing strategy for this PR? Benchmark-only. Ran the bench locally and `cargo clippy --all-targets -D warnings`. ## Are there any user-facing changes? No
1 parent 98c0534 commit 1ffb342

1 file changed

Lines changed: 73 additions & 1 deletion

File tree

‎datafusion/physical-plan/benches/dictionary_group_values.rs‎

Lines changed: 73 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -247,11 +247,83 @@ fn bench_take_n(c: &mut Criterion) {
247247
group.finish();
248248
}
249249

250+
/// Batches that share one dictionary values array, as produced downstream of
251+
/// a repartition or filter: `take`/`filter` clone the values `Arc` and rewrite
252+
/// only the keys, so the values array is never compacted and its cardinality
253+
/// reflects the whole upstream stream rather than a single batch.
254+
///
255+
/// The other benchmarks here always allocate a fresh values array per batch
256+
/// and cap cardinality at the batch size, so neither the `Arc` reuse nor the
257+
/// `cardinality >> rows` regime is covered by them.
258+
fn bench_shared_values_arc(c: &mut Criterion) {
259+
// More batches than `N_BATCHES`: the cost this exercises is paid once per
260+
// batch, so a longer run per values array is what a real partition looks
261+
// like (thousands of batches sharing one dictionary).
262+
const BATCHES: usize = 32;
263+
264+
let mut group = c.benchmark_group("dict_shared_values_arc");
265+
let size = SIZES[0];
266+
267+
for &cardinality in &[size, 32 * 1024, 64 * 1024, 100_000, 500_000] {
268+
// Seeded per cardinality so each configuration is reproducible on its
269+
// own and adding one does not change the data used by the others.
270+
let mut rng = StdRng::seed_from_u64(SEED);
271+
// One values array, shared by every batch.
272+
let strings: Vec<String> =
273+
(0..cardinality).map(|i| format!("v_{i:08}")).collect();
274+
let values: ArrayRef = Arc::new(StringArray::from(
275+
strings.iter().map(|s| Some(s.as_str())).collect::<Vec<_>>(),
276+
));
277+
278+
let batches: Vec<ArrayRef> = (0..BATCHES)
279+
.map(|_| {
280+
let keys: Vec<i32> = (0..size)
281+
.map(|_| rng.random_range(0..cardinality) as i32)
282+
.collect();
283+
Arc::new(DictionaryArray::<Int32Type>::new(
284+
PrimitiveArray::<Int32Type>::from(keys),
285+
Arc::clone(&values),
286+
)) as ArrayRef
287+
})
288+
.collect();
289+
290+
let schema = dict_schema();
291+
group.throughput(Throughput::Elements((size * BATCHES) as u64));
292+
group.bench_function(
293+
BenchmarkId::new(
294+
"shared_values_arc",
295+
format!("size_{size}_card_{cardinality}"),
296+
),
297+
|b| {
298+
b.iter_batched_ref(
299+
|| {
300+
(
301+
new_group_values(schema.clone(), &GroupOrdering::None)
302+
.unwrap(),
303+
Vec::<usize>::with_capacity(size),
304+
)
305+
},
306+
|(gv, groups)| {
307+
for arr in &batches {
308+
gv.intern(std::slice::from_ref(arr), groups).unwrap();
309+
black_box(&*groups);
310+
}
311+
black_box(gv.emit(EmitTo::All).unwrap());
312+
},
313+
BatchSize::SmallInput,
314+
);
315+
},
316+
);
317+
}
318+
group.finish();
319+
}
320+
250321
criterion_group!(
251322
benches,
252323
bench_intern_emit,
253324
bench_repeated_intern_emit,
254325
bench_scalar_append_equal,
255-
bench_take_n
326+
bench_take_n,
327+
bench_shared_values_arc
256328
);
257329
criterion_main!(benches);

0 commit comments

Comments
 (0)