Tracking Issue: Fix Canonical Decimal

This is a tracking issue for fixing Vortex's canonical Decimal encoding.

Status

Decimal Widen and legacy vortex.decimal reads have merged, and DecimalByteParts is being redesigned so that a narrow MSP needs no scan.

Motivation

Our canonical Decimal encoding has been incorrect for too long.

The main (but not only) issue is that the canonicalization abstraction in Vortex has been broken by Decimal, where we can have multiple (up to 6) different possible canonical encodings for the same data. This breaks the abstraction because we want a SINGLE canonical (normal form) of logical data.

For example, right now if I have a Decimal(9, 2) (which means at most 9 integer digits, max integer 999_999_999, max decimal 9_999_999.99), but my decimal array looks like [1.00, 2.00, 1.27] where all of the integers fit in an i8 integer, then the canonical encoding allows us to implicitly narrow the buffer to 8 bits.

This is incredibly wrong because it means for the same array, we can have 6 different canonical representations (i8, i16, i32, i64, i128, i256).

The logical/physical overloading and mismatches have caused a lot of problems (see issue #5820). We seemingly have stamped out most of the bugs, but that doesn't mean the current implementation is not hacky.

There are 2 things that need to be done in order to fix this:

The canonical decimal encoding for a given decimal dtype must have exactly 1 canonical encoding / normal form.
Narrowing the data buffer must be a physical encoding concern, done via a new Widen encoding.

Design

For 1), we simply need to choose what physical encoding an array of Decimal(p, s) canonicalizes to. The simplest and most elegant solution is to pick the smallest power-of-2 bit-width buffer whose integer representation can fit every possible value in the domain of Decimal(p, s).

As an example, Decimal(9, 2) has a max integer value of 999,999,999, which means the smallest bit width it can fit into is 32 bits (i32::MAX == 2,147,483,647), so we can use 32-bit width buffers as the canonical encoding. On the other hand, while some values of Decimal(10, 2) can fit in 32 bits, the largest values must be stored in 64 bits, so we use 64-bit width buffers.

Of course, the issue is that this can potentially blow up the amount of memory and storage we use for representing potentially very small decimal values. But given that Vortex deals with compression quite well (via physical encodings), we can easily solve this by introducing a Widen array.

See Tracking Issue: Add Widen encoding for integers and decimals #10733

Backcompat conversion

The reader (#10995) converts a legacy vortex.decimal node (one whose stored width differs from its dtype's canonical width) through WidenArray::from_stored_decimal. The reader scans valid values for their minimum and maximum, then picks the smallest child precision that covers the values and the scale. The result is a Widen(Decimal) when the child has a narrower canonical width, or a canonical decimal array otherwise.

The stored metadata width identifies legacy nodes. Nodes already at the canonical width reuse their buffer without a scan or copy. The writer from #10946 also emits narrow vortex.decimal nodes for decimal Widen arrays, so this conversion covers ordinary IPC and writes without a compressor as well as older files.

The conversion can reuse a buffer only when the selected child precision has that canonical width. Decimal precision does not exactly match two's-complement widths, so some values fit the stored width but need a precision whose canonical width is larger.

Width	Largest precision for the width	Largest value of that precision	Largest value of the width
`i8`	2	99	127
`i16`	4	9,999	32,767
`i32`	9	999,999,999	2,147,483,647
`i64`	18	10^18 - 1	about 9.2 × 10^18
`i128`	38	10^38 - 1	about 1.7 × 10^38

For example, if an old array has type Decimal(9, s) but only contains integers up to 127, we currently store it as i8. The values 100 to 127 have 3 digits, so in the canonical decimal encoding world we ideally want to narrow it to Decimal(3, s). If we naively say "the largest decimal precision fitting i8 is 2, so we should use Decimal(2, s)", that would not cover the values 100-127 that we need. And since the canonical width of Decimal(3, s) is i16, the conversion copies the i8 values into a wider buffer.

To choose a valid child precision, the reader needs both extrema of the valid values, including negative values, and the scale. This does mean that there will be cases where we cannot narrow some old decimal arrays into a buffer that "fits" in this new world. When the needed precision has the same canonical width as the dtype, there is no Widen at all: an old Decimal(4, s) array with values up to 127 needs precision 3, and precisions 3 and 4 both use i16, so it becomes a plain Decimal(4, s) array at i16. See unresolved questions below.

Nodes stored wider than their dtype's canonical width, such as older Arrow Decimal128 imports at a low precision, use the same child selection and get a narrowing copy. Every narrowing copy writes zeros into null slots and fails on a valid value outside the target precision.

Serialization before `vortex.widen`

A non-executing array writer plugin replaces D26's serializer-only lowering. #11057 separates plugin read and write IDs (merged on 2026-11-02). #11002 closed without merging, and its writer work moved into #10946, which merged on 2026-11-03. That PR registers a writer-only WidenPlugin whose only write ID is vortex.decimal.

The writer adapts a concrete decimal leaf to a narrow vortex.decimal node under the parent logical dtype, without executing values or validity. The node stores the child width, preserving narrow storage without introducing a new wire ID. Integer Widen, non-decimal leaves, and nonserializable validity require explicit caller preparation.

#10995 (merged on 2026-11-05) reads legacy decimal nodes through the shared conversion, in a DecimalPlugin that owns all vortex.decimal serde. Because the read plugin receives stored statistics only after it returns, each legacy read needs a range scan. This is an ordinary path for new IPC and writes without a compressor as well as older files. The scan runs on the host, so a legacy node in device memory returns an error, and CUDA reads of legacy nodes fail until the CUDA executor reads them itself. The future vortex.widen format and its edition remain separate work in #10733, and vortex.widen would also stop WidenPlugin from writing legacy nodes.

Steps

The work lands as an in-memory change and then a wire change, so that the edition change stays small. A step lands as its own PR only if it needs no temporary code. Strict width decoding comes first, because the detection of old arrays reads the width field, and an unknown width currently decodes as I8 without an error.

PR plan (agents keep this list up to date)

Each line is one PR of the implementation plan. The briefs describe each PR. When a PR opens, add its link. When it merges, tick its box. When its scope changes, add a short note here. Record a change of plan in the research branch only when it changes a brief or a decision.

Follow-up issues found while planning:

Decimal scalars panic on a stored value outside the precision #10749 Decimal scalars panic on a stored value outside the precision. Return an error for decimal scalars outside the precision #10767 fixes scalar_at and the min_max kernel, merged on 2026-11-04. Decimal stats read from a file are not checked against the precision #10807 separately tracks cached decimal statistics that bypass that kernel. Other Scalar::decimal callers stay for later PRs
compat-gen skips old fixture versions in CI and stops at the first failure #10754 compat-gen skips old fixture versions in CI and stops at the first failure
Corrupt ALP-RD, RLE and RunEnd metadata can panic a file read #10756 Corrupt ALP-RD, RLE and RunEnd metadata can panic a file read. fix(alp): bounds-check ALP-RD stored metadata and left-parts codes #10760 fixes ALP-RD, merged on 2026-10-29. Return Serde errors for corrupt RLE and RunEnd metadata #10852 fixes RLE and RunEnd, open
Seven cast rules push nullability casts without trivially_cast_nullability #10750 Push-down cast rules without the nullability guard. Closed as not planned, because Document the nullability contract of cast rules #10798 documents them as valid

Related PRs outside this plan:

Bound decimal storage by precision #10305 and Add bounded decimal dispatch macros and migrate kernels #10314 bound decimal storage by precision and add bounded dispatch macros. Open. The plan does not build on them for now
perf(decimal-byte-parts): compare wide decimals through their parts #10146 compares wide decimals through their DecimalByteParts parts. Merged on 2026-11-03
Skip codegen of unreachable int to decimal carrier arms #10735 removes unreachable decimal carrier arms from the integer to decimal cast. Merged on 2026-10-26
Establish widening cast benchmark baselines #11052 (widening cast benchmark baselines) and [EXPERIMENT] Keep ARM integer widening loops compact #11050 (ARM loop experiment) closed without merging. Neither is a dependency

Unresolved questions

Is it okay that we are not able to narrow every decimal array exactly how it was before? ~~Note that this only affects canonical decimal arrays, not encodings like DecimalByteParts.~~ Yes, see explanations above.
Should DecimalByteParts decoding find the max of a narrow most significant part and return a Widen over its buffer instead of widening it? ~~This only matters for files written before the compressor narrows the decimal type first, so the decision waits for benchmarks that show the cost of widening.~~ Yes. Widening every narrow MSP made decoding up to 11x slower in CodSpeed (Decode DecimalByteParts into the canonical array or a decimal Widen #10905), and every default-written decimal column whose values are small for its precision has a narrow MSP. CD5 takes the range from min_max on the MSP and returns a Widen that reuses the MSP buffer (LT2, folded into CD5). No. The MSP keeps the canonical width as its logical dtype, and a narrower MSP is an integer Widen. The canonical width of a precision is the smallest width that holds 10^p - 1, so every value of a narrower integer is a valid Decimal(p) value, and the MSP needs no scan and no precision label. A decimal Widen child needs a precision label, which only a scan gives, and that scan made decoding 4.7 to 8.4 times as slow in CodSpeed (Decode DecimalByteParts into the canonical array or a decimal Widen #10905). DecimalByteParts is then the narrow form itself, and it decodes to the canonical array only when a consumer needs it.