Tracking Issue: Add Widen encoding for integers and decimals #10733

This is a tracking issue for adding a Widen encoding for integers and decimals, which stores logical values that can fit in a smaller bit width or precision in a smaller logical dtype.

• Adapted from Epic: Implement Narrow integer encoding #10020

Status

Integer Widen, its compute benchmarks, aggregates and arithmetic/cast kernels have merged (WD1 to WD4). Structural lifting (WD5) has no PR yet, and the internal-child migration experiment (#10943) closed without merging and is parked for later. The decimal writer plugin and decimal Widen merged on 2026-11-03 and 2026-11-04, and the rest of the decimal stack is open in #10701. No edition permits vortex.widen yet.

Motivation

We have the concept of "narrowing" data in a few places.

• In the compressor, if we have a Primitive integer array that does not contain very large values, we will call a narrow() method on the primitive array to shrink the logical dtype to fit the values. For example, if my values are all in [-128, 127], but the dtype is I64, then I can narrow this to I8. We use this with list offsets, for example. Note that this changes the dtype of the array.
• Narrowing can also happen with explicit casts (which are fallible), but this also changes the dtype.
• In the canonical decimal encoding, there is currently implicit narrowing of the buffer when possible. This is arguably very wrong, see Tracking Issue: Fix Canonical Decimal #10701 for more details.

Ideally, we want a way to keep the logical type of these values while having a better physical encoding that narrows data internally. This is why we want a Widen encoding.

Design

The encoding is named after what decoding does: like FoR is an addition, Widen is a widening cast of its child (see #9156). The compressor scheme that builds it narrows the data, so it keeps the name Narrow.

The Widen encoding will have a single array child that has a narrower logical dtype. For integers, the child has the same signedness and a strictly smaller power-of-two bit width. For decimals, the child has a lower precision whose canonical width is strictly smaller, with the same scale. A positive scale sets a lower bound on that precision. The child always has the same nullability as the parent, and supplies its validity.

For example, a Widen array of I64 can hold an I8 child (if it fits). Or, a Widen array of Decimal(18, 2) (with a 64-bit buffer) can hold a child of Decimal(9, 2) (with a 32-bit buffer).

We can constrain the child of the Widen array such that it must actually shrink the amount of data we need to store. This is only relevant for Decimal: having a Widen(Decimal(17, 2)) with dtype Decimal(18, 2) is completely pointless because it doesn't actually save any space. This is something we might be able to relax in the future, but it seems like a reasonable constraint for now.

Note that the compressor narrowing an internal child does not need to change. This is because list offsets, dictionary codes, run-end ends, patch indices, and the parts of DecimalByteParts have dtypes that their parent encoding chooses, so a narrow integer dtype there is "correct" by default, and these children technically do not need a Widen.

Compute

A consequence of making narrowing a physical concern rather than a logical one is that we will need to forward lots of compute into the Widen array's child.

Kernels run on the child at the narrow width and widen only when the result needs it. Any operation without a kernel falls back to canonicalization, which widens the child to the logical dtype.

Serialization

Widen is currently an in-memory encoding. The decimal writer in #10946 (merged on 2026-11-03) uses the split plugin read and write IDs from #11057 (merged on 2026-11-02). It includes the work folded from #11002, which closed without merging, and registers a writer-only WidenPlugin whose only write ID is vortex.decimal.

This replaces D26's serializer-only lowering. The writer adapts a concrete decimal leaf to the existing vortex.decimal wire form under the parent logical dtype, retaining the child width without executing values or validity. Integer Widen, non-decimal leaves, and nonserializable validity require explicit caller preparation. #10995, which is still open, reads non-canonical decimal nodes into the canonical array or a decimal Widen. Such reads need a range scan because stored statistics reach the array only after the plugin returns, including for ordinary new IPC and writes without a compressor.

The later wire change gives integers and decimals one ID, vortex.widen, behind an edition. Its proposed prost metadata has a typed oneof: the child PType for an integer, or the child precision for a decimal. It stores neither scale nor nullability, since both come from the parent. WR1 must adapt the writer plugin and add a working reader.

Relation to DecimalByteParts v2

Ideally, vortex.widen enters a preview edition together with DecimalByteParts v2, and later core together with it.

For decimal columns, we want the final compressor output to be a Widen over DecimalByteParts. Right now, decimals whose values need more than 64 bits get no compression beyond implicit narrowing, because DBPv1 can only store values up to 64 bits. For example, a Decimal(50, 2) column whose values fit in 128 bits is stored as a plain i128 buffer instead of an i256 one. If we remove the implicit narrowing (again, see #10701), those values are stored at the full canonical width.

A Widen over an i128 child brings back today's size, but only DBPv2 actually compresses these values, since it splits them into a 64-bit most significant part and u64 lower parts. So once we add Widen to replace Decimal's implicit narrowing, we will also want DecimalByteParts v2.

Steps

• Integer Widen in memory, with compute forwarding (rewrite of feat(array): add Narrow integer encoding #10223 and feat(array): push arithmetic and selection through Narrow #10332)
• Replace PrimitiveArrayExt::narrow with WidenArray::encode_values for internal children such as list offsets and dictionary codes (first part of feat(compress): retain Narrow integer children through compression #10333)
• Benchmarks for compute over integer Widen (Add compute benchmarks for Widen #10888)
• Benchmarks for compute over decimal Widen (Support decimal values in Widen #10946)
• Decimal Widen in memory (needed by Tracking Issue: Fix Canonical Decimal #10701)
• vortex.widen wire format for integers and decimals in a preview edition together with DecimalByteParts v2, with the compressor wrapping top-level integer and decimal columns in Widen (second part of feat(compress): retain Narrow integer children through compression #10333)
• vortex.widen and DecimalByteParts v2 in a core edition

Unresolved questions

• Should we store the entire DecimalDType in the Widen metadata, which includes scale? It is kind of pointless because the scale is always the same, but it also doesn't really hurt.
  • No, the metadata stores only the child precision. The scale and nullability of the child always equal those of the parent, so storing them only lets a file hold an invalid child. The format is in no edition yet, so this can still change.
• Should List and ListView offsets always have an I64 dtype, with Widen providing the narrow physical width? (From the discussion on Epic: Implement Narrow integer encoding #10020.)

Implementation history

• Research: [DO NOT MERGE] Research notes on extension types and casts #10633 and [DO NOT MERGE] Research notes on Decimal, Widen, and the type roadmap #11237
• feat(array): add Narrow integer encoding #10223
• feat(array): push arithmetic and selection through Narrow #10332
• feat(compress): retain Narrow integer children through compression #10333
• Avoid truncating dictionary fill codes #10745: dictionary fill_null no longer truncates the fill code (FX3)
• Widen dictionary codes in Dict::fill_null #10806: dictionary fill_null widens the codes instead of declining (follow-up of FX3)
• Renamed from Narrow to Widen on 2026-10-28.
• Add an in-memory Widen encoding for integers #10864: in-memory integer Widen (WD1)
• Add compute benchmarks for Widen #10888: compute benchmarks for Widen (WD2)
• Compute Widen aggregates on the values #10890: Widen aggregates run on the child (WD3)
• Run Widen arithmetic, cast, fill null, zip and between on the values #10891: Widen arithmetic, cast, fill null, zip and between run on the child (WD4)
• Separate the read and write IDs of array plugins #11057 (2026-11-02): separate plugin read and write IDs, so a Widen writer can emit existing wire forms without claiming their readers
• Support decimal values in Widen #10946 (2026-11-03): decimal values in Widen, with a writer-only plugin for decimal leaves (CD4 in Tracking Issue: Fix Canonical Decimal #10701)
• Dispatch zip kernels on either value branch #11138 (2026-11-03): zip kernels dispatch on either value branch, which fixes ZipExecuteAdaptor dispatches only on the if_true branch #10908
• Push decimal comparisons, between, and fills through Widen #11003, Export decimal Widen arrays to Arrow with one copy #10994, and Push decimal aggregates through Widen #11004 (2026-11-03): decimal compare, between, fills, Arrow export, and aggregates on Widen (CD4 and CD4b in Tracking Issue: Fix Canonical Decimal #10701)
• Add shared conversion to canonical decimal arrays or Widen #11005 (2026-11-04): WidenArray::from_stored_decimal converts stored decimal values into the canonical array or a decimal Widen (CD4 in Tracking Issue: Fix Canonical Decimal #10701)
• Keep same-scale decimal widening casts lazy #11006 (2026-11-04): same-scale decimal widening casts return a Widen (CD4 in Tracking Issue: Fix Canonical Decimal #10701)