Repository navigation
refactor(array): give canonical decimals one integer child - #10230
connortsui20 wants to merge 19 commits into
Conversation
bfbb921 to
c55281d
Compare
Merging this PR will regress 3 benchmarks
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | Simulation | random_i128[0.01] |
9.6 µs | 23.1 µs | -58.38% |
| ❌ | Simulation | patterns_i128[Contiguous] |
20.2 µs | 30.2 µs | -33.07% |
| ❌ | Simulation | in_place_non_nullable[16384] |
37.9 µs | 49.3 µs | -23.06% |
| ⚡ | Simulation | random_i128[0.8] |
177.3 µs | 22.5 µs | ×7.9 |
| ⚡ | Simulation | random_i128[0.95] |
161.9 µs | 22.6 µs | ×7.2 |
| ⚡ | Simulation | random_i256[0.95] |
127 µs | 22.6 µs | ×5.6 |
| ⚡ | Simulation | random_i256[0.8] |
125.8 µs | 22.6 µs | ×5.6 |
| ⚡ | Simulation | random_i256[0.5] |
122.1 µs | 22.5 µs | ×5.4 |
| ⚡ | Simulation | patterns_i128[Random] |
119.2 µs | 22.5 µs | ×5.3 |
| ⚡ | Simulation | random_i128[0.5] |
119.2 µs | 22.6 µs | ×5.3 |
| ⚡ | Simulation | patterns_i128[Runs] |
119.8 µs | 23.6 µs | ×5.1 |
| ⚡ | Simulation | cached_indices_i128[0.1] |
41.2 µs | 21.1 µs | +95.8% |
| ⚡ | Simulation | random_i256[0.01] |
39.2 µs | 23.2 µs | +68.74% |
| ⚡ | Simulation | decompress[u32, (1000, 256)] |
54.9 µs | 32.9 µs | +66.97% |
| ⚡ | Simulation | cached_indices_i128[0.01] |
46.5 µs | 30.8 µs | +50.64% |
| ⚡ | Simulation | cached_indices_i32[0.1] |
102.6 µs | 78.5 µs | +30.68% |
| ⚡ | Simulation | take_map[(0.1, 1.0)] |
289.3 µs | 221.7 µs | +30.52% |
| ⚡ | Simulation | in_place_nullable[16384] |
32.7 µs | 26.3 µs | +24.53% |
| ⚡ | Simulation | random_i64[0.8] |
167 µs | 136.9 µs | +21.94% |
| ⚡ | Simulation | random_i64[0.95] |
174.4 µs | 143.9 µs | +21.21% |
| ... | ... | ... | ... | ... | ... |
ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing ct/decimal-integer-child (173be79) with ct/narrow-compression (71f9458)
Footnotes
-
476 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
-
1 benchmark was run, but is now archived. If it was deleted in another branch, consider rebasing to remove it from the report. Instead if it was added back, click here to restore it. ↩
54c5826 to
cce61ed
Compare
cce61ed to
e29d26c
Compare
e29d26c to
7571adf
Compare
c21b7df to
c0d97fe
Compare
c0d97fe to
4931e50
Compare
4931e50 to
76afa79
Compare
76afa79 to
d936b02
Compare
d936b02 to
55b8621
Compare
Decimal precision selects the logical integer width. Narrow keeps smaller stored values and encoded children through canonicalization, with native buffer consumers decoding only when needed. Signed-off-by: Connor Tsui <[email protected]>
Signed-off-by: Connor Tsui <[email protected]>
Signed-off-by: Connor Tsui <[email protected]>
Signed-off-by: Connor Tsui <[email protected]>
Signed-off-by: Connor Tsui <[email protected]>
Signed-off-by: Connor Tsui <[email protected]>
Signed-off-by: Connor Tsui <[email protected]>
Signed-off-by: Connor Tsui <[email protected]>
Signed-off-by: Connor Tsui <[email protected]>
Signed-off-by: Connor Tsui <[email protected]>
Signed-off-by: Connor Tsui <[email protected]>
Signed-off-by: Connor Tsui <[email protected]>
Signed-off-by: Connor Tsui <[email protected]>
Signed-off-by: Connor Tsui <[email protected]>
Signed-off-by: Connor Tsui <[email protected]>
Signed-off-by: Connor Tsui <[email protected]>
Signed-off-by: Connor Tsui <[email protected]>
Signed-off-by: Connor Tsui <[email protected]>
b82e21b to
1b342f3
Compare
Signed-off-by: Connor Tsui <[email protected]>
|
talked offline, we are not going to have integer children in decimal, instead we will just make sure that the single buffer of decimal is a lot more sound (i.e. only one canonical form per decimal type) |
## Summary - Closes #10750 - Replaces #10746 The docs on `Validity::cast_nullability` read as if every `CastReduce` rule that removes nullability must call `Validity::trivially_cast_nullability` and pair with a `CastKernel`. Most push-down rules (FoR, ALP, ALP-RD, ZigZag, RLE, RunEnd, DateTimeParts, DecimalByteParts, Chunked, Sparse, Extension) instead cast the child that holds their validity, and the cast of that child already fails on nulls, so no result is wrong. Guarding each rule, as #10746 does for DecimalByteParts, adds a kernel per encoding and delays the rewrite until execution without changing any result. The guard in #10746 comes from the closed #10230, where a canonical decimal held a lazy child that a cast to canonical did not execute, by way of the FX2 notes in #10633. `develop` keeps one buffer per decimal, so that reason no longer applies. ## Changes `CastReduce` now owns the contract and documents both forms of rule. A rule that builds a new `Validity` keeps the guard and the kernel, and a rule that pushes the cast into its children needs neither when the nulls of those children are exactly the null rows of the array. `Dict` is the documented exception, because its values can hold nulls that no code references. No new tests are needed: every push-down encoding listed above already runs `test_cast_conformance` on a partially valid array, and that suite asserts that the cast to non-nullable fails. --- _Generated by [Claude Code](https://claude.ai/code/session_01V1YHcosAAKufkyKPorUC2n)_
Stacked on #10333. This experimental draft remains at the top of the stack.
A decimal with precision 76 can contain values that fit in 32 bits. Giving its canonical representation a plain 256-bit integer buffer would expand the value buffer eightfold before any computation.
Decimal precision now determines the logical dtype of one signed integer child. Narrow retains smaller stored values and encoded children during canonicalization, so selection, comparison, and supported aggregates can work through the child. Consumers that need native buffers explicitly materialize values at the stored width.
The experiment adds signed 128-bit and 256-bit integer extensions and retains the existing Decimal wire representation.