Repository navigation
Preserve aggregate facts from writers and array producers - #10269
connortsui20 wants to merge 5 commits into
Conversation
Signed-off-by: Connor Tsui <[email protected]>
Signed-off-by: Connor Tsui <[email protected]>
Signed-off-by: Connor Tsui <[email protected]>
Signed-off-by: Connor Tsui <[email protected]>
Signed-off-by: Connor Tsui <[email protected]>
Merging this PR will degrade performance by 19.9%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | Simulation | bench_compare_sliced_dict_primitive[(2500, 10000)] |
28.7 µs | 84 µs | -65.78% |
| ❌ | Simulation | runend_compress_u32 |
408.6 µs | 506.5 µs | -19.33% |
| ❌ | Simulation | compress[(10000, 4)] |
500.8 µs | 577.3 µs | -13.25% |
| ❌ | Simulation | compress[(4000, 4)] |
201.4 µs | 231.6 µs | -13.03% |
| ❌ | Simulation | compress[(10000, 16)] |
249.3 µs | 282.8 µs | -11.85% |
| ❌ | Simulation | compress_fsst[(200, 16, 8)] |
162.3 µs | 182 µs | -10.84% |
| ❌ | WallTime | dict_canonicalize_gt_u8_neon[1000000] |
489.6 µs | 547.5 µs | -10.58% |
| ⚡ | Simulation | compress_fsst[(200, 4, 8)] |
147.9 µs | 127.7 µs | +15.83% |
| Simulation | bench_compare_sliced_dict_primitive[(3333, 10000)] |
77.7 µs | < 1 ns | N/A |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing ct/stats-aggregate-producers (e41dda1) with ct/stats-sequence-boundaries (f46a4e1)
Footnotes
-
503 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
connortsui20
changed the title
Tracking Issue: #10177
Summary
Some statistics are already known when an array is constructed or a writer computes a chunk summary. Preserving those facts avoids rescanning data merely because the metadata API changed.
Writers must retain real accumulator states when combining chunks because finalized answers can lose information such as sum overflow. Producers can publish facts derived from their construction, while checks that make an unchecked cast safe must still inspect validity data.
Changes
For review, follow writer computation first, then typed construction and the index builder. The Sequence and VarBinView changes cover the edge cases those metadata paths expose.
API Changes
Adds an optional encoding hook for initial aggregate results and an ascending index builder. The hook describes metadata derived from construction and does not establish unsafe preconditions.
Stack