Repository navigation
Accumulate file summaries with aggregate states - #10252
connortsui20 wants to merge 2 commits into
Conversation
Merging this PR will regress 1 benchmark
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | Simulation | compress_fsst[(200, 16, 8)] |
162.3 µs | 182.5 µs | -11.07% |
| ⚡ | WallTime | words_gather_dispatch_avx512[1048576] |
14.8 µs | 13.1 µs | +12.78% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing ct/stats-file-aggregates (1169c46) with ct/stats-footer-results (5b899f1)
Footnotes
-
503 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
Signed-off-by: Connor Tsui <[email protected]>
Signed-off-by: Connor Tsui <[email protected]>
653eeb0 to
5b899f1
Compare
00b0dad to
1169c46
Compare
Superseded by #10265, which groups this change with the related statistics work. The original commits and branch are preserved.
Original PR description
Tracking Issue: #10177
Summary
File summaries currently pass each chunk through the fixed-stat API before combining it. That loses information needed for aggregation, including Sum's overflow state and the boundary values needed for sortedness. Retaining the actual accumulator states lets the writer combine chunks correctly and finalize each file summary once.
Changes
Select file summaries by aggregate function and options, share Min/Max computation, and truncate string and binary bounds after finalization. The existing footer format remains in use, so unsupported selections are rejected before writing bytes. Chunk hints remain cached on arrays, and floating Sum retains the existing chunk grouping and NaN policy.
For review, start with
FieldAccumulator, then follow selection validation and the writer hookup. An overflowing Sum is now an exact null result, distinct from missing metadata. Empty chunks contribute counts and size without introducing extrema or ordering boundaries.API Changes
with_file_statisticsaccepts aggregate functions instead of fixedStatidentifiers. Explicit sortedness and constantness selections now compute their summaries across chunk boundaries.Stack
Migration stack #10229, in review order: #10226 → #10227 → #10232 → #10235 → #10236 → #10252 → #10253 → #10257 → #10258 → #10259 → #10260 → #10261. Independent fixes: file Sum overflow #10228 and empty Constant aggregate states #10254.