feat(aggregate): add ExtensionStoragePushdown to push an aggregate into an extension type's storage by joseph-isaacs · Pull Request #10566 · vortex-data/vortex

joseph-isaacs · 2026-10-16T15:57:19Z

Summary

Aggregates that support an extension dtype currently do so by hand: each one matches Canonical::Extension and recurses on storage_array() (min_max, is_constant, is_sorted, uncompressed_size_in_bytes, the zoned bloom filter). Aggregates that don't, such as IsSorted, reject extension dtypes outright, even where running them on the storage is correct (e.g. timestamps, which compare like their i64 storage).

This adds an opt-in pushdown: declaring that one aggregate may run on one extension type's storage is a single registration, with no per-aggregate code:

session.aggregate_fns().register(ExtensionStoragePushdown::new(IsSorted, Timestamp));

Changes

New ExtensionStoragePushdown<A: AggregateFnVTable, E: ExtVTable> in vortex-array/src/aggregate_fn/storage_pushdown.rs, an AggregateFnVTable of its own:
- id <A>[<E>] (e.g. vortex.is_sorted[vortex.timestamp]), built once in new; it never replaces A. Options serialize and deserialize through A, so a registered pair round-trips by id.
- accepts only extension dtypes of type E; any other input makes return_dtype/partial_dtype return None.
- the partial is A::Partial, and every vtable call forwards to A with the storage dtype borrowed from the extension dtype, so partial handling allocates nothing (perf(aggregate): hold the storage partial directly in ExtensionAggregate #10612).
- unwraps each batch to its storage array and accumulates it through a per-batch Accumulator<A>, so kernels and cached stats registered for A on storage encodings still fire. Constant extension batches become constant storage arrays.
- partials and results are those of A over the storage dtype, with no lifting back to the extension.
The doc comment says a registration is an unchecked claim, and is only correct when two things hold:
- E's domain supports the aggregation.
- The storage computes it soundly. Each operation the aggregate uses must give the same answer on the storage: comparison, equality, addition, NaN detection.
It uses Date/Time/Timestamp as the example (comparison yes, sum no), and a doctest registers and runs ExtensionStoragePushdown<IsSorted, Timestamp>.
The existing hand-written storage delegations are left unchanged: they handle every extension type rather than one specific E.

Tests:

pushes_down_into_storage registers ExtensionStoragePushdown<MinMax, Timestamp>, finds it by id, and checks that the result matches MinMax over the storage.
combines_flushed_partials checks that merging two flushed partials matches accumulating both batches.
rejects_other_extension_types checks that a Date input is rejected.
A doctest shows registering and running a pair.

API Changes

Adds the public vortex_array::aggregate_fn::ExtensionStoragePushdown. Nothing existing changes.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Hp7ZGEG8mrip6qkAtK4iWD

…ion storage Add a StorageLifter trait and an ExtensionDelegate<L> adapter that implements AggregateFnVTable for an extension dtype by running the lifter's storage aggregate over the extension's storage array and lifting the final result back. Route the existing hand-written storage delegations through it, with one lifter per aggregate: min_max, is_constant, is_sorted, uncompressed_size_in_bytes and the zoned bloom filter. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Hp7ZGEG8mrip6qkAtK4iWD Signed-off-by: Claude <[email protected]>

…A, E> Replace the StorageLifter trait and ExtensionDelegate with a single ExtensionAggregate<A, E> adapter, which defines aggregate A over extension type E by running A over E's storage. It is an aggregate of its own, with id `<A>[<E>]`, so supporting a new pair is one session registration: session.aggregate_fns().register(ExtensionAggregate::new(Sum, Timestamp)); Restore the original storage delegation in min_max, is_constant, is_sorted, uncompressed_size_in_bytes and the zoned bloom filter. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Hp7ZGEG8mrip6qkAtK4iWD Signed-off-by: Claude <[email protected]>

codspeed · 2026-10-16T16:06:47Z

Merging this PR will regress 2 benchmarks

⚠️

Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️

1 benchmark measured no execution time

Nothing ran under measurement, usually because the compiler removed the code under test. This result is not comparable, so it counts as unchanged.

Preventing compiler optimizations

bench_compare_primitive[(10000, 2)]

⚠️

20 benchmarks spent significant time in system calls

System calls cannot be consistently instrumented, so they are not included in the measure, which understates the real cost. Please switch to the Walltime instrument to accurately measure system calls.

Measurement and system calls

test_scan_scalar_at[int]

test_scan_filter[int_str_float]

test_repeated_scan_scalar_at[int_str_float]

test_repeated_scan_scalar_at[int_float]

test_scan_filter[int_str]

test_scan_filter[int_float]

test_scan[int_str_float]

multipolygon_mixed_validity

random_i128[0.01]

take_varbin

…and 10 more

ℹ️ 40 benchmarks have no flame graph

The flamegraph could not be generated because we failed to resolve the symbols in the profiling trace. This happens when we fail to harvest information about language frames.

test_scan_filter[int]

test_repeated_scan_scalar_at[int_str]

test_repeated_scan[int]

test_scan_scalar_at[int]

test_repeated_scan_scalar_at[int]

test_scan[int_float]

test_scan_filter[int_str_float]

test_repeated_scan_scalar_at[int_str_float]

test_repeated_scan_scalar_at[int_float]

test_scan_filter_to_arrow_array[int_str_float]

…and 30 more

⚠️

Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 4 improved benchmarks
❌ 2 regressed benchmarks
✅ 2402 untouched benchmarks
⏩ 12 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	`BASE`	`HEAD`	Efficiency
❌	WallTime	`bitpack_blocked_compress_avx2`	6.7 µs	7.6 µs	-11.34%
❌	Simulation	`take[routing/grouped_chunks/primitive/nonnull/chunks=32/indices=1000]`	385 µs	429.4 µs	-10.35%
⚡	Simulation	`chunked_opt_bool_canonical_into[(1000, 10)]`	65.4 µs	55.1 µs	+18.6%
⚡ ℹ️	Simulation	`test_scan_filter[int]`	3.6 ms	3 ms	+18.36%
⚡ ℹ️	Simulation	`test_repeated_scan_scalar_at[int_str]`	481.3 µs	408.6 µs	+17.79%
⚡	WallTime	`threshold_avx2[80]`	3 µs	2.7 µs	+10.57%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing ji/stoic-pascal-p9i2a9 (597e2ae) with develop (61ff28d)}

12 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

Signed-off-by: Joe Isaacs <[email protected]>

joseph-isaacs · 2026-10-19T13:27:17Z

CodSpeed Performance Analysis is red on 3db29a5, but I don't think either regression is this PR's:

chunked_opt_bool_canonical_into[(1000, 10)] (-15%) builds canonical bool arrays from chunks.
test_scan_filter[int] (-11%) is a Python scan with a filter on an int column.

This PR only adds ExtensionAggregate (vortex-array/src/aggregate_fn/extension.rs) and its module line. Nothing existing calls it or registers it, so neither benchmark can reach the new code. The previous CodSpeed run on 241723d also flagged 4 regressions, and none of them are among these 2. Most of the flagged benchmarks there were Python scans with flame graphs unavailable. CodSpeed's own report warns about walltime runs on hosted runners and about comparisons across different runtime environments.

No fix is needed in this PR. The regressions can be acknowledged on CodSpeed.

Generated by Claude Code

robert3005 · 2026-10-19T13:46:31Z

made a small adjustment in #10612

…ate (#10612) Co-authored-by: Claude <[email protected]> Signed-off-by: Claude <[email protected]>

…aggregate Document, per aggregate, the property the storage must share with the extension type before a pair is registered, with examples that fit and that do not, and add a doctest that registers and runs ExtensionAggregate<IsSorted, Timestamp>. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Hp7ZGEG8mrip6qkAtK4iWD Signed-off-by: Claude <[email protected]>

github-actions · 2026-10-19T16:34:57Z

Polar Signals Profiling Results

Latest Run

Status	Commit	Job	Attempt	Link
🟢 Done	`3ce4231`	compress-bench	1	Explore Profiling Data

Powered by Polar Signals Cloud

The adapter is an opt-in pushdown of one aggregate into one extension type's storage, not an aggregate over any extension type, so name it for that and move it to aggregate_fn/storage_pushdown.rs. Expand its docs: registering a pair is an unchecked claim about the extension type; a pushdown is correct only when every operation the aggregate uses (comparison, equality, addition, NaN detection) derives from the same operation on the storage; Timestamp supports it for comparison but not addition. Switch the tests and examples from Sum to MinMax, which is correct for Timestamp. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Hp7ZGEG8mrip6qkAtK4iWD Signed-off-by: Claude <[email protected]>

github-actions · 2026-10-19T16:44:47Z

Benchmarks: Compression 📖

Commits: PR 3ce42311 vs base 18aebeaa

vortex / vortex-file-compressed / ns (0.988x ➖, 1↑ 0↓)

name	ns (PR / base / %diff)
compress time/Arade	773402437 / 787435434 / -1.8%
compress time/Bimbo	3567153364 / 3692508802 / -3.4%
compress time/CMSprovider	2538776849 / 2545029989 / -0.2%
compress time/Euro2016	268753168 / 267512769 / +0.5%
compress time/Food	321271426 / 317789581 / +1.1%
compress time/HashTags	461956561 / 463547621 / -0.3%
compress time/TPC-H l_comment canonical	606476713 / 600311849 / +1.0%
compress time/TPC-H l_comment chunked	606847827 / 603742733 / +0.5%
compress time/taxi	389901087 / 387878829 / +0.5%
compress time/wide table cols=100 chunks=1 rows=1000	7135277 / 7082137 / +0.8%
compress time/wide table cols=100 chunks=50 rows=1000	7201612 / 7133163 / +1.0%
compress time/wide table cols=1000 chunks=1 rows=1000	70068105 / 71662532 / -2.2%
compress time/wide table cols=1000 chunks=50 rows=1000	71187164 / 70054687 / +1.6%
compress time/wide table cols=10000 chunks=1 rows=1000	764614956 / 769400585 / -0.6%
compress time/wide table cols=10000 chunks=50 rows=1000	767616449 / 778349156 / -1.4%
decompress time/Arade	8678495 / 9053434 / -4.1%
decompress time/Bimbo	35562142 / 37381017 / -4.9%
decompress time/CMSprovider	26175851 / 27149413 / -3.6%
decompress time/Euro2016	9331847 / 9746141 / -4.3%
decompress time/Food	2475820 / 2908948 / -14.9% 🟢
decompress time/HashTags	43108330 / 43177640 / -0.2%
decompress time/TPC-H l_comment canonical	10543356 / 11031972 / -4.4%
decompress time/TPC-H l_comment chunked	10538043 / 10909365 / -3.4%
decompress time/taxi	5284879 / 5181307 / +2.0%
decompress time/wide table cols=100 chunks=1 rows=1000	1307445 / 1345212 / -2.8%
decompress time/wide table cols=100 chunks=50 rows=1000	1309478 / 1348698 / -2.9%
decompress time/wide table cols=1000 chunks=1 rows=1000	12823055 / 12460271 / +2.9%
decompress time/wide table cols=1000 chunks=50 rows=1000	12826341 / 12409842 / +3.4%
decompress time/wide table cols=10000 chunks=1 rows=1000	136588951 / 133669276 / +2.2%
decompress time/wide table cols=10000 chunks=50 rows=1000	137165289 / 133497275 / +2.7%

vortex / vortex-file-compressed / bytes (1.000x ➖, 0↑ 0↓)

name	bytes (PR / base / %diff)
vortex-file-compressed size/Arade	142213756 / 142213756 / +0.0%
vortex-file-compressed size/Bimbo	458581212 / 458581212 / +0.0%
vortex-file-compressed size/CMSprovider	420794836 / 420794836 / +0.0%
vortex-file-compressed size/Euro2016	164125572 / 164125572 / +0.0%
vortex-file-compressed size/Food	38950752 / 38950752 / +0.0%
vortex-file-compressed size/HashTags	196170516 / 196170516 / +0.0%
vortex-file-compressed size/TPC-H l_comment canonical	178688232 / 178688232 / +0.0%
vortex-file-compressed size/TPC-H l_comment chunked	178688232 / 178688232 / +0.0%
vortex-file-compressed size/taxi	49998092 / 49998092 / +0.0%
vortex-file-compressed size/wide table cols=100 chunks=1 rows=1000	934080 / 934080 / +0.0%
vortex-file-compressed size/wide table cols=100 chunks=50 rows=1000	934080 / 934080 / +0.0%
vortex-file-compressed size/wide table cols=1000 chunks=1 rows=1000	9325680 / 9325680 / +0.0%
vortex-file-compressed size/wide table cols=1000 chunks=50 rows=1000	9325680 / 9325680 / +0.0%
vortex-file-compressed size/wide table cols=10000 chunks=1 rows=1000	93277680 / 93277680 / +0.0%
vortex-file-compressed size/wide table cols=10000 chunks=50 rows=1000	93277680 / 93277680 / +0.0%

vortex / vortex-file-compressed / ratio (0.992x ➖, 1↑ 0↓)

name	ratio (PR / base / %diff)
vortex:parquet-zstd ratio compress time/Arade	0.343246814 / 0.350073615 / -2.0%
vortex:parquet-zstd ratio compress time/Bimbo	0.302295861 / 0.311439563 / -2.9%
vortex:parquet-zstd ratio compress time/CMSprovider	0.396176701 / 0.398612965 / -0.6%
vortex:parquet-zstd ratio compress time/Euro2016	0.248551222 / 0.2472584 / +0.5%
vortex:parquet-zstd ratio compress time/Food	0.433691486 / 0.429202695 / +1.0%
vortex:parquet-zstd ratio compress time/HashTags	0.252266432 / 0.25390901 / -0.6%
vortex:parquet-zstd ratio compress time/TPC-H l_comment canonical	0.230802677 / 0.228396024 / +1.1%
vortex:parquet-zstd ratio compress time/TPC-H l_comment chunked	0.230193524 / 0.229279183 / +0.4%
vortex:parquet-zstd ratio compress time/taxi	0.356565369 / 0.356144023 / +0.1%
vortex:parquet-zstd ratio compress time/wide table cols=100 chunks=1 rows=1000	1.80278272 / 1.79102265 / +0.7%
vortex:parquet-zstd ratio compress time/wide table cols=100 chunks=50 rows=1000	1.82740056 / 1.76216895 / +3.7%
vortex:parquet-zstd ratio compress time/wide table cols=1000 chunks=1 rows=1000	1.56844507 / 1.61095158 / -2.6%
vortex:parquet-zstd ratio compress time/wide table cols=1000 chunks=50 rows=1000	1.59965213 / 1.58194927 / +1.1%
vortex:parquet-zstd ratio compress time/wide table cols=10000 chunks=1 rows=1000	1.66440804 / 1.67738835 / -0.8%
vortex:parquet-zstd ratio compress time/wide table cols=10000 chunks=50 rows=1000	1.65343038 / 1.72058246 / -3.9%
vortex:parquet-zstd ratio decompress time/Arade	0.0155915147 / 0.0162564564 / -4.1%
vortex:parquet-zstd ratio decompress time/Bimbo	0.0222213856 / 0.0233227159 / -4.7%
vortex:parquet-zstd ratio decompress time/CMSprovider	0.0192159569 / 0.0199083036 / -3.5%
vortex:parquet-zstd ratio decompress time/Euro2016	0.0281051581 / 0.0293179881 / -4.1%
vortex:parquet-zstd ratio decompress time/Food	0.014603112 / 0.017115439 / -14.7% 🟢
vortex:parquet-zstd ratio decompress time/HashTags	0.0833275149 / 0.0835751162 / -0.3%
vortex:parquet-zstd ratio decompress time/TPC-H l_comment canonical	0.0211894009 / 0.0221928205 / -4.5%
vortex:parquet-zstd ratio decompress time/TPC-H l_comment chunked	0.0211340315 / 0.0219569447 / -3.7%
vortex:parquet-zstd ratio decompress time/taxi	0.024334222 / 0.0238150044 / +2.2%
vortex:parquet-zstd ratio decompress time/wide table cols=100 chunks=1 rows=1000	0.642088498 / 0.653945343 / -1.8%
vortex:parquet-zstd ratio decompress time/wide table cols=100 chunks=50 rows=1000	0.647013018 / 0.637994781 / +1.4%
vortex:parquet-zstd ratio decompress time/wide table cols=1000 chunks=1 rows=1000	0.547926437 / 0.533848905 / +2.6%
vortex:parquet-zstd ratio decompress time/wide table cols=1000 chunks=50 rows=1000	0.555159658 / 0.539979908 / +2.8%
vortex:parquet-zstd ratio decompress time/wide table cols=10000 chunks=1 rows=1000	0.559573797 / 0.552586204 / +1.3%
vortex:parquet-zstd ratio decompress time/wide table cols=10000 chunks=50 rows=1000	0.555542798 / 0.553478274 / +0.4%
vortex:parquet-zstd size/Arade	0.551185597 / 0.551185597 / -0.0%
vortex:parquet-zstd size/Bimbo	1.19261532 / 1.19261532 / +0.0%
vortex:parquet-zstd size/CMSprovider	1.11649863 / 1.11649863 / -0.0%
vortex:parquet-zstd size/Euro2016	1.33458157 / 1.33458157 / -0.0%
vortex:parquet-zstd size/Food	1.09107276 / 1.09107276 / -0.0%
vortex:parquet-zstd size/HashTags	1.46946859 / 1.46946859 / -0.0%
vortex:parquet-zstd size/TPC-H l_comment canonical	1.12837977 / 1.12837977 / -0.0%
vortex:parquet-zstd size/TPC-H l_comment chunked	1.12837977 / 1.12837977 / -0.0%
vortex:parquet-zstd size/taxi	0.904392267 / 0.904392267 / -0.0%
vortex:parquet-zstd size/wide table cols=100 chunks=1 rows=1000	1.0017975 / 1.0017975 / -0.0%
vortex:parquet-zstd size/wide table cols=100 chunks=50 rows=1000	1.0017975 / 1.0017975 / -0.0%
vortex:parquet-zstd size/wide table cols=1000 chunks=1 rows=1000	1.00017975 / 1.00017975 / -0.0%
vortex:parquet-zstd size/wide table cols=1000 chunks=50 rows=1000	1.00017975 / 1.00017975 / -0.0%
vortex:parquet-zstd size/wide table cols=10000 chunks=1 rows=1000	1.00040408 / 1.00040408 / -0.0%
vortex:parquet-zstd size/wide table cols=10000 chunks=50 rows=1000	1.00040408 / 1.00040408 / -0.0%

vortex / parquet / ns (1.000x ➖, 0↑ 0↓)

name	ns (PR / base / %diff)
parquet_rs-zstd compress time/Arade	2253196260 / 2249342423 / +0.2%
parquet_rs-zstd compress time/Bimbo	11800205755 / 11856261171 / -0.5%
parquet_rs-zstd compress time/CMSprovider	6408193222 / 6384714530 / +0.4%
parquet_rs-zstd compress time/Euro2016	1081278805 / 1081915796 / -0.1%
parquet_rs-zstd compress time/Food	740783337 / 740418419 / +0.0%
parquet_rs-zstd compress time/HashTags	1831224855 / 1825644635 / +0.3%
parquet_rs-zstd compress time/TPC-H l_comment canonical	2627684918 / 2628381341 / -0.0%
parquet_rs-zstd compress time/TPC-H l_comment chunked	2636250649 / 2633220887 / +0.1%
parquet_rs-zstd compress time/taxi	1093491182 / 1089106665 / +0.4%
parquet_rs-zstd compress time/wide table cols=100 chunks=1 rows=1000	3957924 / 3954242 / +0.1%
parquet_rs-zstd compress time/wide table cols=100 chunks=50 rows=1000	3940905 / 4047945 / -2.6%
parquet_rs-zstd compress time/wide table cols=1000 chunks=1 rows=1000	44673611 / 44484597 / +0.4%
parquet_rs-zstd compress time/wide table cols=1000 chunks=50 rows=1000	44501653 / 44283776 / +0.5%
parquet_rs-zstd compress time/wide table cols=10000 chunks=1 rows=1000	459391531 / 458689596 / +0.2%
parquet_rs-zstd compress time/wide table cols=10000 chunks=50 rows=1000	464256893 / 452375388 / +2.6%
parquet_rs-zstd decompress time/Arade	556616541 / 556913129 / -0.1%
parquet_rs-zstd decompress time/Bimbo	1600356643 / 1602772898 / -0.2%
parquet_rs-zstd decompress time/CMSprovider	1362193473 / 1363723072 / -0.1%
parquet_rs-zstd decompress time/Euro2016	332033251 / 332428711 / -0.1%
parquet_rs-zstd decompress time/Food	169540575 / 169960467 / -0.2%
parquet_rs-zstd decompress time/HashTags	517336081 / 516632725 / +0.1%
parquet_rs-zstd decompress time/TPC-H l_comment canonical	497576881 / 497096436 / +0.1%
parquet_rs-zstd decompress time/TPC-H l_comment chunked	498629096 / 496852597 / +0.4%
parquet_rs-zstd decompress time/taxi	217178877 / 217564814 / -0.2%
parquet_rs-zstd decompress time/wide table cols=100 chunks=1 rows=1000	2036238 / 2057071 / -1.0%
parquet_rs-zstd decompress time/wide table cols=100 chunks=50 rows=1000	2023882 / 2113964 / -4.3%
parquet_rs-zstd decompress time/wide table cols=1000 chunks=1 rows=1000	23402877 / 23340445 / +0.3%
parquet_rs-zstd decompress time/wide table cols=1000 chunks=50 rows=1000	23103878 / 22982044 / +0.5%
parquet_rs-zstd decompress time/wide table cols=10000 chunks=1 rows=1000	244094616 / 241897599 / +0.9%
parquet_rs-zstd decompress time/wide table cols=10000 chunks=50 rows=1000	246903190 / 241196956 / +2.4%

vortex / parquet / bytes (1.000x ➖, 0↑ 0↓)

name	bytes (PR / base / %diff)
parquet size/Arade	258014282 / 258014282 / +0.0%
parquet size/Bimbo	384517292 / 384517292 / +0.0%
parquet size/CMSprovider	376887911 / 376887911 / +0.0%
parquet size/Euro2016	122979049 / 122979049 / +0.0%
parquet size/Food	35699500 / 35699500 / +0.0%
parquet size/HashTags	133497590 / 133497590 / +0.0%
parquet size/TPC-H l_comment canonical	158358238 / 158358238 / +0.0%
parquet size/TPC-H l_comment chunked	158358238 / 158358238 / +0.0%
parquet size/taxi	55283635 / 55283635 / +0.0%
parquet size/wide table cols=100 chunks=1 rows=1000	932404 / 932404 / +0.0%
parquet size/wide table cols=100 chunks=50 rows=1000	932404 / 932404 / +0.0%
parquet size/wide table cols=1000 chunks=1 rows=1000	9324004 / 9324004 / +0.0%
parquet size/wide table cols=1000 chunks=50 rows=1000	9324004 / 9324004 / +0.0%
parquet size/wide table cols=10000 chunks=1 rows=1000	93240004 / 93240004 / +0.0%
parquet size/wide table cols=10000 chunks=50 rows=1000	93240004 / 93240004 / +0.0%

vortex / arrow-ipc / ns (1.021x ➖, 0↑ 5↓)

name	ns (PR / base / %diff)
arrow-ipc compress time/Arade	98831091 / 100526703 / -1.7%
arrow-ipc compress time/Bimbo	614349965 / 632004766 / -2.8%
arrow-ipc compress time/CMSprovider	383847964 / 388818942 / -1.3%
arrow-ipc compress time/Euro2016	38387401 / 38778483 / -1.0%
arrow-ipc compress time/Food	33941136 / 34273067 / -1.0%
arrow-ipc compress time/HashTags	78620189 / 79055014 / -0.6%
arrow-ipc compress time/TPC-H l_comment canonical	950461125 / 952545520 / -0.2%
arrow-ipc compress time/TPC-H l_comment chunked	960779625 / 958239676 / +0.3%
arrow-ipc compress time/taxi	74379124 / 75598596 / -1.6%
arrow-ipc compress time/wide table cols=100 chunks=1 rows=1000	205785 / 213106 / -3.4%
arrow-ipc compress time/wide table cols=100 chunks=50 rows=1000	198130 / 203702 / -2.7%
arrow-ipc compress time/wide table cols=1000 chunks=1 rows=1000	2240016 / 2273386 / -1.5%
arrow-ipc compress time/wide table cols=1000 chunks=50 rows=1000	2226534 / 2276210 / -2.2%
arrow-ipc compress time/wide table cols=10000 chunks=1 rows=1000	25579681 / 25869661 / -1.1%
arrow-ipc compress time/wide table cols=10000 chunks=50 rows=1000	25705054 / 25846631 / -0.5%
arrow-ipc decompress time/Arade	181396212 / 142125027 / +27.6% 🔴
arrow-ipc decompress time/Bimbo	404436641 / 432178101 / -6.4%
arrow-ipc decompress time/CMSprovider	581938688 / 476421809 / +22.1% 🔴
arrow-ipc decompress time/Euro2016	74665083 / 64155739 / +16.4% 🔴
arrow-ipc decompress time/Food	58408258 / 49190058 / +18.7% 🔴
arrow-ipc decompress time/HashTags	142968386 / 122222178 / +17.0% 🔴
arrow-ipc decompress time/TPC-H l_comment canonical	620297909 / 641278112 / -3.3%
arrow-ipc decompress time/TPC-H l_comment chunked	627920982 / 641921911 / -2.2%
arrow-ipc decompress time/taxi	48439778 / 45692253 / +6.0%
arrow-ipc decompress time/wide table cols=100 chunks=1 rows=1000	317193 / 320811 / -1.1%
arrow-ipc decompress time/wide table cols=100 chunks=50 rows=1000	319013 / 320196 / -0.4%
arrow-ipc decompress time/wide table cols=1000 chunks=1 rows=1000	3629887 / 3558015 / +2.0%
arrow-ipc decompress time/wide table cols=1000 chunks=50 rows=1000	3637852 / 3676421 / -1.0%
arrow-ipc decompress time/wide table cols=10000 chunks=1 rows=1000	36792598 / 36985971 / -0.5%
arrow-ipc decompress time/wide table cols=10000 chunks=50 rows=1000	36870777 / 36763188 / +0.3%

vortex / arrow-ipc / bytes (1.000x ➖, 0↑ 0↓)

name	bytes (PR / base / %diff)
arrow-ipc size/Arade	691765162 / 691765162 / +0.0%
arrow-ipc size/Bimbo	3279590114 / 3279590114 / +0.0%
arrow-ipc size/CMSprovider	2422259682 / 2422259682 / +0.0%
arrow-ipc size/Euro2016	360582666 / 360582666 / +0.0%
arrow-ipc size/Food	255782338 / 255782338 / +0.0%
arrow-ipc size/HashTags	704785810 / 704785810 / +0.0%
arrow-ipc size/TPC-H l_comment canonical	7317023586 / 7317023586 / +0.0%
arrow-ipc size/TPC-H l_comment chunked	7317023586 / 7317023586 / +0.0%
arrow-ipc size/taxi	482329690 / 482329690 / +0.0%
arrow-ipc size/wide table cols=100 chunks=1 rows=1000	1257290 / 1257290 / +0.0%
arrow-ipc size/wide table cols=100 chunks=50 rows=1000	1257290 / 1257290 / +0.0%
arrow-ipc size/wide table cols=1000 chunks=1 rows=1000	12568506 / 12568506 / +0.0%
arrow-ipc size/wide table cols=1000 chunks=50 rows=1000	12568506 / 12568506 / +0.0%
arrow-ipc size/wide table cols=10000 chunks=1 rows=1000	125752506 / 125752506 / +0.0%
arrow-ipc size/wide table cols=10000 chunks=50 rows=1000	125752506 / 125752506 / +0.0%

…d none count, null_count, all_null, all_non_null and uncompressed_size_in_bytes already accept any extension dtype, so say they need no pushdown instead of listing them as always-correct pushdowns. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Hp7ZGEG8mrip6qkAtK4iWD Signed-off-by: Claude <[email protected]>

…wn, tighten docs Name the extension in the type, and cut its docs down to the two conditions a registration must meet: the extension type's domain supports the aggregation, and the storage computes it soundly, with the operation each aggregate needs. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Hp7ZGEG8mrip6qkAtK4iWD Signed-off-by: Claude <[email protected]>

connortsui20

suggested doc change

Co-authored-by: connortsui20 <[email protected]> Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Hp7ZGEG8mrip6qkAtK4iWD Signed-off-by: Claude <[email protected]>

X10 records that ExtensionStoragePushdown (#10566, #10616) is a temporary bridge that phase 2 replaces, and part 3 explains how it differs from the design. X11 records that derives are a goal: authors opt in to common behavior with little code, and the Derives sketch is not a decision. XQ7 asks which mechanism gives that opt-in. The extension plan now adds ExtensionArray::new_unchecked in phase 1, because the rules of phases 2 and 3 build their results with it. Phase 4 splits into the lift rule (4a), which waits for nothing and is the extension case of #10748, and statistics with DateTimeParts (4b). Phase 2 starts with a design pass for XQ2, XQ5 and XQ7, and its typing covers lambda bodies. The type graph gets the missing N2 to N4 edge, and the reason for N2 to N3 now follows the design notes. The design notes rename their derive rules to DR1 to DR4 and their nullability steps to NU1 and NU2, so that they no longer collide with D1 to D4 and N1 and N2. Signed-off-by: Connor Tsui <[email protected]>

No commit since 5fc5cb8 fixed a finding, so U1 to U8 are all still needed. Part 4 now gives line numbers at 1d09649. U5 also covers Time, whose Span setters panic like those of Timestamp. U2 fixes the DataFusion literal conversion of EXT-B1 in the same change, and checks the dynamic filter path. U6 makes the claim in the unpack_native docs true. EXT-A1 records #10566 and #10616, which add composite aggregates on storage for the temporal types and Uuid, while plain is_sorted and min_max keep their old answers. #10246 closed without a merge. The Duration and Interval PRs touch the code of U1, U2 and phase 3. Signed-off-by: Connor Tsui <[email protected]>

claude added 2 commits October 16, 2026 10:11

joseph-isaacs marked this pull request as ready for review October 16, 2026 16:00

joseph-isaacs added the changelog/feature A new feature label Oct 19, 2026

fix(aggregate): resolve extension adapter lint failures

3db29a5

Signed-off-by: Joe Isaacs <[email protected]>

perf(aggregate): hold the storage partial directly in ExtensionAggreg…

3ce4231

…ate (#10612) Co-authored-by: Claude <[email protected]> Signed-off-by: Claude <[email protected]>

joseph-isaacs added the action/bench-compress Run only the compression benchmark on this PR label Oct 19, 2026

github-actions Bot removed the action/bench-compress Run only the compression benchmark on this PR label Oct 19, 2026

joseph-isaacs mentioned this pull request Oct 19, 2026

feat(aggregate): register is_sorted and min_max for temporal and UUID extension types #10616

Merged