Repository navigation
Migrate file footer summaries to aggregate results - #10236
connortsui20 wants to merge 1 commit into
Conversation
Merging this PR will improve performance by 12.73%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ⚡ | Simulation | compress_fsst[(200, 16, 8)] |
182.9 µs | 162.3 µs | +12.73% |
Tip
Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.
Comparing ct/stats-footer-results (5b899f1) with ct/stats-array-cache (1643e97)
Footnotes
-
503 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
6f0aa7d to
653eeb0
Compare
Signed-off-by: Connor Tsui <[email protected]>
653eeb0 to
5b899f1
Compare
Superseded by #10265, which groups this change with the related statistics work. The original commits and branch are preserved.
Original PR description
Tracking Issue: #10177
Summary
File summaries are finalized answers, but their current
StatsSetrepresentation ties them to the legacy fixed-stat API and cannot represent known nulls. Moving the footer owner to aggregate results separates persisted metadata from the accumulator states used to compute it.Changes
Convert historical footer fields directly into aggregate-keyed results and update their file, DataFusion, DuckDB, and JNI consumers together. A compatibility codec retains the existing wire fields, historical scalar types, precision, and missing values. The writer converts its existing statistics output once when constructing the footer.
For review, start with the compatibility codec and
FileStatistics, then follow the consumer changes. Writer selection and streaming accumulation migrate in the next layer.API Changes
FileStatisticsconstructors and field accessors now useAggregateResults.results()replacesstats_sets(), and lookups use aggregate functions including their options.Stack
Migration stack #10229, in review order: #10226 → #10227 → #10232 → #10235 → #10236 → #10252 → #10253 → #10257 → #10258 → #10259 → #10260 → #10261. Independent fixes: file Sum overflow #10228 and empty Constant aggregate states #10254.