Repository navigation
Conversation
Merging this PR will regress 2 benchmarks
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | Simulation | density_sweep_dense_runs[0.001] |
29.6 µs | 47.6 µs | -37.82% |
| ❌ | WallTime | bitpack_blocked_compress_avx512 |
4.8 µs | 5.6 µs | -14.46% |
| ⚡ |
Simulation | decode_primitives[i64, (1000, 32)] |
38.7 µs | 25 µs | +54.56% |
| ⚡ | WallTime | threshold_avx512[8] |
2 µs | 1.8 µs | +11.59% |
| ⚡ | WallTime | words_gather_dispatch_neon[65536] |
2.3 µs | 2 µs | +11.18% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing joacoc:sorted-file-stats (cb371d9) with develop (58d0490)
This commit adds the code to add into the footer the `IsSorted` and `IsStrictSorted` stats. The compute of these stats happens by re-using each chunk sortedness partial and merging them to calculate the final file stat. This means that the final sortedness has the same properties that a chunk sortedness has (e.g. supports only ASC order). Extension types are not included, so sorting timestamps should be revisited after their implementation. Signed-off-by: Joaquin Colacci <[email protected]>
Signed-off-by: Joaquin Colacci <[email protected]>
cb371d9 to
667d12d
Compare
|
I'll probably close this PR in favor of #10661, probably we will not need this one. |
U2, U3, U4, U6, U7 and U8 merged on 2026-11-02 in #11012, #11014 and #11016, and the DataFusion literal fix #11067 merged right after U2. So order constraint 1 holds, and phase 1 starts from develop. The graph shows these nodes as merged, and parts 2 and 4 name the PR that fixed each finding. #10953 now keeps i32 days and widens only the days part to i64 when a value does not fit. It has no review, and it conflicts with the DateTimeParts compare kernel that U2 changed. #11056 still waits on its request for changes. #10661 replaces the closed #10482 for file sort statistics. In the type roadmap, only the N1 sentence changes. Signed-off-by: Connor Tsui <[email protected]>
Summary
This PR calculates and writes a column's
IsSortedandIsStrictSortedstats to the file footer by reusing each chunk's sortedness partials and merging them to get the final file-level sortedness stat.The final sortedness stat has the same properties that a chunk sortedness stat has (e.g. supports only ASC order), this also means that extension types are not supported, so sortedness for timestamps (if they are an extension, I haven't checked) should be revisited.
This PR only records the
is_sorted/is_strict_sortedstats. I'll work on the per-file sort key in the next PR.Changes
Only affects how
IsSortedandIsStrictSortedare computed inside thevortex-layoutcrate. All the other stats follow the same path as before.The biggest difference compared to other stats is that, when merging sortedness, you need the partials to compare the min/max values from each chunk to make sure they're also sorted. For example,
[[1, 2], [1, 4]]is sorted at the chunk level, but not at the column/file level. This is not possible to check if we just collect the chunk sortedness output (true/false).