Skip to content

Commit e31250d

Browse files
committed
Keep the DecimalByteParts benchmarks and record the GPU legacy read in #11284
Signed-off-by: Connor Tsui <[email protected]>
1 parent df39560 commit e31250d

4 files changed

Lines changed: 8 additions & 8 deletions

File tree

‎research/decimal/02-design.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -339,7 +339,8 @@ MSP on the device fails today. [#11284](https://github.com/vortex-data/vortex/is
339339
device `Widen` kernel that sign-extends on the device, for integer and decimal `Widen`s (decided,
340340
D16). The new decoding depends on it, because every narrow DBP read on the GPU then has a `Widen`
341341
MSP, and it adds no stopgap of its own. The GPU still rejects a legacy MSP that is wider than its
342-
logical dtype.
342+
logical dtype. #11284 also covers reading legacy `vortex.decimal` nodes on the GPU, which CD6
343+
leaves as an error (D40).
343344

344345
The compressor has one DBP scheme (decided, D17):
345346

‎research/decimal/04-implementation.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -79,7 +79,7 @@ planning review on 2026-10-23.
7979
| CD4p | Decimal `Interleave` execution, so that CD4 can test its fallbacks | None | Medium |
8080
| CD4 | Decimal `Widen` in memory, `WidenArray::from_stored_decimal` (D40), the decimal `Sum` and `MinMax` kernels (D36), and the serializer lowering (D26) | WD1, WD3, WD4, CD1, CD3, CD3b, CD4p | Large |
8181
| CD4b | Export decimal `Widen` arrays to Arrow with one copy | CD4 | Small |
82-
| LT3 | Sum, min, and max on the DecimalByteParts MSP, and the reshaped DecimalByteParts benchmarks (D19, D42) | WD3, CD1 | Medium |
82+
| LT3 | Sum, min, and max on the DecimalByteParts MSP (D19). No benchmark changes (D42) | WD3, CD1 | Medium |
8383
| CD5 | DecimalByteParts is the decimal over an integer child: the MSP keeps the canonical width, capped at `i64`, a narrower MSP is an integer `Widen`, and decoding needs no scan (D16). CD7 is folded in, so one PR holds both. #10905 closed on 2026-11-05 | WD1, LT3, CD1, CD3, CD3b, #11284, the CD2 fixtures in 0.91.0 | Large |
8484
| CD6 | The `vortex.decimal` read plugin converts old nodes through `WidenArray::from_stored_decimal` | FX1, CD3, CD3b, CD4, the CD2 fixtures in 0.91.0 | Medium |
8585
| CD7 | One DecimalByteParts scheme on the canonical array, which stores a narrower MSP as an integer `Widen`, also next to lower parts (D17). Folded into CD5 on 2026-11-05. #10992 closed on 2026-11-05 | WD1, CD1, CD3b, CD5 | Medium |

‎research/decimal/agents/briefs/03-canonical-decimal.md‎

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -163,10 +163,9 @@ Tests: None. Benchmarks only.
163163
Benchmarks:
164164

165165
- In this PR, every case is new, so CodSpeed shows additions only.
166-
- LT3 reshapes these benches (D42). `sum` and `min_max` run on the MSP, `filter` stops at the DBP
167-
array, and `decode` and `canonicalize` become one `canonicalize`, which pays one widening copy
168-
for a narrow MSP. Compare cases stay the same, because they run on the MSP.
169-
`msp_i64_p18_canonical` stays the same. How the compressed tree decompress changes is open.
166+
- These benches keep their form through LT3 and CD5 (D42). After LT3, `sum` and `min_max` run on
167+
the MSP. After CD5, `decode` pays one widening copy for a narrow MSP and no scan. Compare cases
168+
stay the same, because they run on the MSP. `msp_i64_p18_canonical` stays the same.
170169
- After CD7, these in-memory inputs do not change. The golden snapshot and the optional compress
171170
case cover the scheme.
172171
- After CD4, `widen_*` in `cast_decimal.rs` stays similar or gets faster, because the execution of

‎research/decisions.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -66,9 +66,9 @@ decimal encoding.
6666
| D37 | The `vortex.decimal` read plugin converts old narrow nodes on the host, also on CUDA read paths. A GPU scan of such a node copies its buffer to the host | Old narrow nodes are rare, and CUDA has no `Widen` kernel |
6767
| D38 | The integer `Widen` PRs are written from scratch from their briefs. The local rewrite of #10223 and #10332 is a reference for tests and edge cases, not a base. FX3 ports the rewrite's dictionary fill fix as it is | The rewrite predates D25, D26, D28 and D36, so a cherry-pick would carry its old structure into the new PRs |
6868
| D39 | The encoding is renamed from `Narrow` to `Widen`, after the widening cast its decode applies ([#9156](https://github.com/vortex-data/vortex/issues/9156)). The compressor scheme stays `Narrow` and produces `Widen` arrays. The wire ID is `vortex.widen`. Plan IDs NR1 to NR6 become WD1 to WD6, and benchmark input `Wide` becomes `Canonical` | Encodings are named after the function their decode applies, like `FoR` is an addition |
69-
| D40 | `WidenArray::from_stored_decimal(parts, known_range: Precision<RangeInclusive<i256>>, ctx)` is the single owner of the choice between the canonical array and a `Widen` for stored decimal values. CD4 adds it, and CD6 uses it. Since 2026-11-05, DecimalByteParts decoding does not use it (D16). It first checks the buffer length, the alignment and the validity length, and returns an error on a mismatch. At the canonical width it returns the canonical array with no copy and no scan, and a device buffer stays on the device. Any other width copies the values to the host. `Precision::Absent` scans the valid values, and `Exact` is the smallest and the largest valid value. The child records either one as exact `Stat::Min` and `Stat::Max`. `Inexact` is only a bound, and the child records it as inexact. `WidenArray::encode_decimal` shares the core that picks the child. The function that picks the child dtype, `smallest_decimal_dtype`, is crate-private | One function owns the child label (D10), so two readers cannot choose different children for the same values. A false label gives wrong answers in the compare shortcut and in casts. A read never panics, so the checks come before any code that asserts on a length |
69+
| D40 | `WidenArray::from_stored_decimal(parts, known_range: Precision<RangeInclusive<i256>>, ctx)` is the single owner of the choice between the canonical array and a `Widen` for stored decimal values. CD4 adds it, and CD6 uses it. Since 2026-11-05, DecimalByteParts decoding does not use it (D16). It first checks the buffer length, the alignment and the validity length, and returns an error on a mismatch. At the canonical width it returns the canonical array with no copy and no scan, and a device buffer stays on the device. Any other width copies the values on the host. A device buffer is an error until #11284 reads legacy nodes on the GPU. `Precision::Absent` scans the valid values, and `Exact` is the smallest and the largest valid value. The child records either one as exact `Stat::Min` and `Stat::Max`. `Inexact` is only a bound, and the child records it as inexact. `WidenArray::encode_decimal` shares the core that picks the child. The function that picks the child dtype, `smallest_decimal_dtype`, is crate-private | One function owns the child label (D10), so two readers cannot choose different children for the same values. A false label gives wrong answers in the compare shortcut and in casts. A read never panics, so the checks come before any code that asserts on a length |
7070
| D41 | The PRs land in this order. CD3c, "Speed up the decimal widening copy", is a PR off `develop`. CD4 is off `develop` too and does not depend on CD3c. CD4b, "Export decimal Widen arrays to Arrow with one copy", is stacked on CD4. CD5 and CD6 are each stacked on CD4, and CD7 is stacked on CD5. CD3c, CD4 and CD4b have no release gate. CD3c merges first, so that the CodSpeed run of CD4 measures the fast copy. CD5 and CD6 wait for 0.91.0 (D31), and CD7 merges right after CD5. CD5, CD7 and CD6 aim for one release, with CD4b already merged. On 2026-11-05, #10905 (CD5) and #10992 (CD7) closed for the redesign of D16. LT3 lands on `develop` first. One PR for CD5, with CD7 folded in, stacks on LT3 and merges only after #11284. The new rule, the reader, and the writer must change together: a CD5 alone would need a temporary writer fix for the `narrowed_decimal` code that CD7 deletes, and a CD7 alone would put `Widen` MSPs in memory before the decoder and the GPU handle them. The PR can split later if review needs it. The scan design does not land in between, even if the replacements miss the release of CD6 | CD5 needs the decimal `Widen` and the helper of D40, so it moves onto CD4. Paths that need the canonical array, such as DuckDB, Arrow above precision 18 and the `canonicalize` bench, still pay one widening copy (D3), so CD3c makes that copy fast. CD4b removes one of the two copies of an Arrow export up to precision 18 |
71-
| D42 | The DecimalByteParts benchmarks in `encodings/decimal-byte-parts/benches/dbp_compute.rs` measure the operations on the DecimalByteParts array: `sum`, `min_max`, and both compare benches run on it, and `filter` stops at it. `decode` and `canonicalize` become one `canonicalize` bench with `execute::<DecimalArray>`, which tracks the widening copy. The canonical-width control stays. These changes land with LT3. How `decompress` in `vortex/benches/common_encoding_tree_throughput.rs` changes is open | DecimalByteParts never decodes to a `Widen` (D16), so a bench that stops at a `Widen` runs to the canonical array. The narrow form shows only in the operations that run on the parts. The 2026-10-29 version stopped `decode` and `filter` at a `Widen` with `execute_until::<Widen>` |
71+
| D42 | The DecimalByteParts benchmarks in `encodings/decimal-byte-parts/benches/dbp_compute.rs` keep their `develop` form: `decode` runs to the canonical array with `execute::<DecimalArray>`, which tracks the widening copy, `sum`, `min_max`, and both compare benches run on the DecimalByteParts array, and `filter` runs to the canonical array. The canonical-width control stays. LT3 changes no benchmark, and `decode` keeps its name, so CodSpeed keeps its history. CD5 only builds the narrow MSP inputs as a `Widen`. `decompress` in `vortex/benches/common_encoding_tree_throughput.rs` keeps its `develop` form too | DecimalByteParts never decodes to a `Widen` (D16), so a bench that stops at a `Widen` runs to the canonical array. The narrow form shows only in the operations that run on the parts. The 2026-10-29 version stopped `decode` and `filter` at a `Widen` with `execute_until::<Widen>` |
7272

7373
D1 to D6 date from 2026-10-19, D7 and D8 from 2026-10-22, and D12 to D38 from 2026-10-23. On
7474
2026-10-23 the maintainer also revised D9, D10 and D11. Their 2026-10-22 versions used `P(W)`, the

0 commit comments

Comments
 (0)