`array_agg()` add tests and benchmarks by fred1268 · Pull Request #23740 · apache/datafusion

fred1268 · 2026-07-21T07:23:09Z

Which issue does this PR close?

Related to array_agg() performance #23715

Rationale for this change

Add some tests and benchmarks for array_agg() to have some measures before performance improvements.

What changes are included in this PR?

4 unit tests and 2 benchmarks

Are these changes tested?

Changes are only tests and benchmarks, so yes.

Are there any user-facing changes?

No user-facing changes

No breaking changes to public APIs

codecov-commenter · 2026-07-21T08:08:14Z

Codecov Report

❌ Patch coverage is 79.48718% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.69%. Comparing base (1e58928) to head (5a6f1ae).
⚠️ Report is 28 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/functions-aggregate/src/array_agg.rs	79.48%	4 Missing and 12 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #23740      +/-   ##
==========================================
+ Coverage   80.67%   80.69%   +0.01%     
==========================================
  Files        1086     1089       +3     
  Lines      366800   368502    +1702     
  Branches   366800   368502    +1702     
==========================================
+ Hits       295912   297345    +1433     
- Misses      53260    53430     +170     
- Partials    17628    17727      +99

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

fred1268 · 2026-07-21T08:14:26Z

run benchmarks array_agg

adriangbot · 2026-07-21T08:14:27Z

Hi @fred1268, thanks for the request (#23740 (comment)). Only whitelisted users can trigger benchmarks. Allowed users: Dandandan, Fokko, Jefffrey, Omega359, Rachelint, adriangb, alamb, asubiotto, brunal, buraksenn, cetra3, codephage2020, coderfender, comphead, erenavsarogullari, etseidl, friendlymatthew, gabotechs, geoffreyclaude, grtlr, haohuaijin, jonathanc-n, kevinjqliu, klion26, kosiew, kumarUjjawal, kunalsinghdadhwal, liamzwbao, mbutrovich, mkleen, mzabaluev, neilconway, rluvaton, sdf-jkl, timsaucer, xudong963, zhuqi-lucas.

File an issue against this benchmark runner

gabotechs · 2026-07-21T08:16:13Z

run benchmarks array_agg

adriangbot · 2026-07-21T08:19:02Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c5031676791-1198-8m6bb 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing bench/array_aggregate (5a6f1ae) to 1e58928 (merge-base) diff using: array_agg
Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-07-21T08:26:33Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                                                                              HEAD                                   bench_array_aggregate
-----                                                                              ----                                   ---------------------
array_agg i64 merge_batch 30% nulls, 0% of nulls point to a zero length array      1.01    502.6±5.86µs        ? ?/sec    1.00    496.8±5.55µs        ? ?/sec
array_agg i64 merge_batch 30% nulls, 100% of nulls point to a zero length array    1.00      4.6±0.01µs        ? ?/sec    1.00      4.6±0.00µs        ? ?/sec
array_agg i64 merge_batch 30% nulls, 50% of nulls point to a zero length array     1.10   549.1±48.57µs        ? ?/sec    1.00    500.0±6.00µs        ? ?/sec
array_agg i64 merge_batch 30% nulls, 90% of nulls point to a zero length array     1.01    506.4±9.94µs        ? ?/sec    1.00    502.4±5.74µs        ? ?/sec
array_agg i64 merge_batch 30% nulls, 99% of nulls point to a zero length array     1.00    503.4±5.68µs        ? ?/sec    1.00    502.4±5.75µs        ? ?/sec
array_agg i64 merge_batch 70% nulls, 0% of nulls point to a zero length array      1.01    216.4±2.51µs        ? ?/sec    1.00    214.7±2.54µs        ? ?/sec
array_agg i64 merge_batch 70% nulls, 100% of nulls point to a zero length array    1.00      4.4±0.00µs        ? ?/sec    1.00      4.4±0.01µs        ? ?/sec
array_agg i64 merge_batch 70% nulls, 50% of nulls point to a zero length array     1.01    216.7±2.60µs        ? ?/sec    1.00    214.9±2.54µs        ? ?/sec
array_agg i64 merge_batch 70% nulls, 90% of nulls point to a zero length array     1.08    234.8±2.42µs        ? ?/sec    1.00    216.6±2.48µs        ? ?/sec
array_agg i64 merge_batch 70% nulls, 99% of nulls point to a zero length array     1.00    215.9±2.90µs        ? ?/sec    1.00    216.0±2.60µs        ? ?/sec
array_agg i64 merge_batch all nulls, 100% of nulls point to a zero length array    1.09     79.7±5.14ns        ? ?/sec    1.00     72.9±2.62ns        ? ?/sec
array_agg i64 merge_batch all nulls, 90% of nulls point to a zero length array     1.00     73.0±3.03ns        ? ?/sec    1.00     72.9±2.59ns        ? ?/sec
array_agg i64 merge_batch no nulls                                                 1.01     82.0±2.06ns        ? ?/sec    1.00     81.6±1.47ns        ? ?/sec
distinct_array_agg utf8 high cardinality (~7800 distinct, 5% db names)                                                    1.00   1126.9±2.73µs        ? ?/sec
distinct_array_agg utf8 low cardinality (~25 distinct)                                                                    1.00    578.0±1.73µs        ? ?/sec

Resource Usage

array_agg — base (merge-base)

Metric	Value
Wall time	180.0s
Peak memory	30.4 MiB
Avg memory	17.9 MiB
CPU user	155.8s
CPU sys	0.1s
Peak spill	0 B

array_agg — branch

Metric	Value
Wall time	260.1s
Peak memory	32.6 MiB
Avg memory	15.2 MiB
CPU user	180.9s
CPU sys	0.2s
Peak spill	0 B

File an issue against this benchmark runner

gabotechs

👍 No comments, sounds reasonable. Thanks @fred1268!

## Which issue does this PR close? - Related to apache#23715 ## Rationale for this change Add some tests and benchmarks for `array_agg()` to have some measures before performance improvements. ## What changes are included in this PR? 4 unit tests and 2 benchmarks ## Are these changes tested? Changes are only tests and benchmarks, so yes. ## Are there any user-facing changes? No user-facing changes No breaking changes to public APIs

fred1268 added 3 commits July 21, 2026 09:15

add tests for array_agg()

9b3a5c6

add benches for array_agg()

0c7fe42

cargo fmt

5a6f1ae

github-actions Bot added the functions Changes to functions implementation label Jul 21, 2026

gabotechs approved these changes Jul 21, 2026

View reviewed changes

gabotechs added this pull request to the merge queue Jul 21, 2026

Merged via the queue into apache:main with commit 1fcdef2 Jul 21, 2026
37 checks passed

nuno-faria added the v55.0.0 label Oct 3, 2026

Navigation Menu

Repository navigation

`array_agg()` add tests and benchmarks - #23740

`array_agg()` add tests and benchmarks#23740
gabotechs merged 3 commits into
apache:mainfrom
fred1268:bench/array_aggregate

fred1268 commented Jul 21, 2026

Uh oh!

codecov-commenter commented Jul 21, 2026

Uh oh!

fred1268 commented Jul 21, 2026

Uh oh!

adriangbot commented Jul 21, 2026

Uh oh!

gabotechs commented Jul 21, 2026

Uh oh!

adriangbot commented Jul 21, 2026

Uh oh!

adriangbot commented Jul 21, 2026

Uh oh!

gabotechs left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

fred1268 commented Jul 21, 2026

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

codecov-commenter commented Jul 21, 2026

Codecov Report

Uh oh!

fred1268 commented Jul 21, 2026

Uh oh!

adriangbot commented Jul 21, 2026

Uh oh!

gabotechs commented Jul 21, 2026

Uh oh!

adriangbot commented Jul 21, 2026

Uh oh!

adriangbot commented Jul 21, 2026

Uh oh!

gabotechs left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants