Skip to content

Latest commit

 

History

History
180 lines (163 loc) · 36.3 KB

decisions.md

File metadata and controls

180 lines (163 loc) · 36.3 KB

Decisions

This file records the decisions that the maintainer made while reviewing these notes on 2026-10-19, 2026-10-22, 2026-10-23, 2026-10-26, 2026-10-28, 2026-10-29, 2026-10-30, 2026-11-03, and 2026-11-05. It also lists the questions that stay open. The other documents follow this file. If a document disagrees with it, this file wins.

"Settled" marks a choice of the maintainer. "Recommendation" marks a proposal of these notes that the maintainer has not confirmed yet.

The X, XQ, C, and CQ decisions live in research/decisions.md on ct/ext-types-research.

Decimal and new physical types

Issue #10701 tracks the fix of the canonical decimal encoding.

Settled

ID Decision Reason
D1 Decimal stays a built-in type. It behaves like a hardcoded extension type: opaque by default, with derives for one dtype and rules that the type owns. If the two become identical, a later change can remove the built-in type A move to an extension type adds no capability, and the built-in type can follow the same rules
D2 A canonical decimal array is a leaf with one buffer and validity. It has no data child A leaf is simpler than a wrapper over an integer child, and it is the same as the vortex.decimal wire node
D3 Each decimal dtype has exactly one canonical encoding. The precision fixes the width of the buffer Today one dtype has six physical forms, which causes the crashes in issue #5820
D4 A Widen encoding stores a decimal column as a child with a lower precision and the same scale Without it, D3 forces the full width on every column, which costs performance
D5 Vortex adds I128, I256, U128 and U256 as primitive integer types Wide decimals need an integer view, and wide integers are useful as columns
D6 FixedSizeBinary becomes a core type for opaque bytes of a fixed length, such as UUIDs and hashes Its defining operations (equality, hashing, byte order, cheap scalars) are core operations. Arrow and Parquet have it natively
D7 vortex.decimal keeps its wire ID. A reader finds an old node when its stored width differs from the canonical width of its dtype D3 removes forms and adds none. The versioning spec asks for a new ID only when readers must understand something new
D8 One Widen encoding, with one vortex.widen wire ID, covers integers and decimals. Integers narrow by width (issue #10020), and decimals narrow by precision. The metadata stores only the child PType of an integer child, or only the child precision of a decimal child, in a prost message with a typed oneof (D32). It stores no scale and no nullability. This settles DQ2 and DQ3 A release that permits an ID must read every valid instance of it. So the decimal reader ships in the same release as the first edition that permits vortex.widen. The child must have the scale and nullability of the parent, so metadata that stored them would only let a file hold an invalid child. The format is in no edition yet, so it can still change
D9 A reader converts an old vortex.decimal node, whose stored width differs from the canonical width of its dtype. One rule covers every such width, through WidenArray::from_stored_decimal (D40). The reader scans the valid values for their min and max, and it chooses the smallest precision that the values and the scale allow. The result is a Widen over a child at the canonical width of that precision, or the canonical array when that width is the canonical width of the column. The child reuses the buffer when its width is the stored width, and copies it otherwise. So a node stored wider than its canonical width can also become a Widen. A narrowing copy follows D21 A width does not give a precision. The read plugin cannot use stored statistics, because the deserializer attaches them after the plugin returns. Old nodes are rare, so the speed of this path matters little. One rule for every width keeps one owner of the child label (D40)
D10 A decimal Widen child has the same scale and nullability as its parent, and a canonical width strictly narrower than the canonical width of the parent. The producer chooses the child precision: the best precision that the values and the scale allow, or a larger one A Widen that does not reduce the width has no use. The integer Widen has the same rule. The rule can be relaxed later
D11 The canonical-width work lands as an in-memory change, and then as a wire change (D18). Each step of the in-memory change is its own PR, because none of the steps needs temporary code. The wire change lands in lockstep, if possible in the same release as the in-memory change. PRs #10305 and #10314 stay open, and the work ignores them for now Small PRs are easier to review, but not when a split needs temporary code or changes file contents twice. In one release, writers that select the preview edition never pay the i256 cost of D17
D12 Each precision has the smallest signed width that holds its values: i8 for precision 1 to 2, i16 for 3 to 4, i32 for 5 to 9, i64 for 10 to 18, i128 for 19 to 38 and i256 for 39 to 76. DecimalDType::canonical_values_type() owns the mapping. Its name differs from the stored-width accessor DecimalArray::values_type(), so code cannot mix the two up. This settles DQ1 It is the existing smallest_decimal_value_type. From precision 5, it is the width of the smallest Arrow decimal type that holds the precision
D13 Imports copy into the canonical width. They write zeros in null slots, and they fail on a value outside the precision. The Arrow import logic can be configured to handle such values in another way. For an export at a wider dtype, Vortex executes a Widen over the array arrow-rs does not check decimal values when it builds an array. An error is better than a silent change of the data
D14 DecimalArray::try_new checks the width and every valid value, with an ExecutionCtx (D30). new_unchecked skips the value check, for kernels that keep values legal. File reads trust the precision and never panic This is the rule of X7 for decimals, and the recommendation XQ3 for file reads
D15 Widen kernels pass work to the child, with canonicalization as the fallback. A Widen over a Widen becomes one Widen. A cast of a Widen passes to the child with the same target dtype, except that a wider integer target returns a Widen over the child. Another decimal cast to a precision with the same scale returns a Widen when the canonical width grows, changes only the dtype when the width stays, and checks every value when the precision drops A Widen holds exactly the values of its child, so work on the child gives the same result on less data
D16 DecimalByteParts is the decimal over an integer child. The MSP has the logical dtype of the signed integer at the canonical width of the precision (D12), capped at i64, with the nullability of the decimal. Every lower part has the dtype u64, and D29 bounds their count. Lower parts exist only above precision 18, so an array with lower parts always has an i64 MSP. An array above precision 18 with no lower parts also has an i64 MSP, which assembly sign-extends, because v1 files have that shape. An MSP stored at a narrower width is an integer Widen over the narrower child, in memory only. The plugin writes the child of that Widen as the MSP, with the narrow msp_ptype, so the wire form does not change, and the reader wraps a narrower MSP in a Widen without a scan. Older writers could store an MSP wider than its logical dtype. The reader finds such an MSP from the dtypes and narrows it with the copy of D21, which scans the values, so only these legacy nodes pay a scan. v2 is only in the unstable edition, so it takes the new rule with no legacy reads. Operations run on the parts wherever they can. With no lower parts, an aggregate runs the integer aggregate on the MSP and relabels the result as a decimal (LT3). Sum converts the integer sum to the decimal result dtype, which is null outside the result precision, and it uses the decimal sum on the decoded array when the integer sum overflows. A possible later refinement predicts the overflow from the dtypes and the batch length, with the bound of is_exact_sum in vortex-array/src/arrays/widen/compute/aggregate.rs, instead of the fallback at run time. Decoding returns the canonical array of the DecimalByteParts dtype, with a widening copy only for a narrower MSP. It never returns a decimal Widen, and it does not use WidenArray::from_stored_decimal (D40). #11284 adds a device Widen kernel that sign-extends on the device, for integer and decimal Widens, and the new decoding depends on it, because every narrow DecimalByteParts read on the GPU then has a Widen MSP. The GPU still rejects a legacy MSP that is wider than its logical dtype. This settles DQ4 The canonical width of a precision is the smallest width that holds 10^p - 1 (D12). So the largest value of any narrower signed width is below 10^p - 1, and every value of that width is a valid Decimal(p) value. A narrower integer MSP therefore needs no scan and no precision label. A decimal Widen child needs a precision label, and a narrow buffer gives a precision only after a scan. The 2026-10-29 version took the range of a narrow MSP from min_max and returned a decimal Widen over its buffer, which made decode 4.7 to 8.4 times as slow as on develop in CodSpeed (#10905). An integer MSP can also narrow next to lower parts, which a decimal Widen cannot: Decimal(38) values between 2^64 and 2^65 need an i128 decimal child, but their MSP holds only 1. The stricter constructor breaks no file, because the reader wraps or narrows every v1 MSP, and no released edition permits v2
D17 The compressor has one DecimalByteParts scheme. The scheme takes the canonical array, uses the value range to choose the number of lower parts, and stores a narrower MSP as an integer Widen (D16), also next to lower parts. The compressor executes every child to its canonical form before it compresses it, so the scheme narrows the MSP, compresses the narrow child, and then wraps the result in the Widen. This replaces narrowed_decimal and keeps v1 file contents, except that large values in null slots can give a narrower MSP. The scheme stays after the wire change, for writers whose edition does not permit vortex.widen. It is the only scheme that narrows decimals, also when an edition permits vortex.widen (D33). In an edition without the lower-parts format, precision 39 to 76 with values that need more than 64 bits is written as a plain i256 array. The core family holds the formats of the default writer, so default writers pay this cost until both formats enter core The canonical fix makes a narrow decimal array illegal in memory, so the scheme cannot call narrowed_decimal. File contents do not change before the wire change. The i256 cost affects only very wide decimals with values above 64 bits. The only way around it writes the old narrow form into new files, and it needs a special serializer
D18 vortex.widen stays in memory only until the maintainer decides to write it to files. Every new wire format enters a preview edition before core. vortex.widen and DecimalByteParts with lower parts enter one preview edition together, and later core together. Other IDs can join that edition, but this plan adds only these two. This settles DQ9 In-memory changes land first, so the edition change stays small
D19 The bar of no compute regression applies to the final state, not to the steps in between. Missing benchmarks land before the changes that they cover. A gap column holds values that fit a width but need the precision of the next width. In the final state, such a column uses the wider width in memory, and that cost is accepted. Since 2026-11-05, pushdown of sum, min, and max into the DecimalByteParts MSP (LT3) is required, and it lands before the new decoding of D16. DecimalByteParts has no gap columns, because its narrow MSP is an integer with no precision label A true child label needs the next width for such values. Disk size and exports do not change
D20 The canonical decimal work does not include the integer view, wide integers, FixedSizeBinary, decimal arithmetic across precisions and scales, casts that lose information, or the lower bound of a negative scale. Their recommendations are unreviewed future work. Epic #10730 tracks wide integers and FixedSizeBinary. Issue #9902 is its sub-issue for FixedSizeBinary, and wide integers get a tracking issue when that work is scheduled. The recommendation to always store min and max on the MSP (DQ8) is dropped The canonical fix needs none of them. Stored min and max exist only on a BitPacked MSP, and a slice drops them
D21 Every narrowing copy writes zeros in null slots, and it fails on a valid value outside the precision of the target dtype. The reads make such a copy when the stored width is wider than the width of the result: a canonical array or a Widen child. This happens for old vortex.decimal nodes, whose copies go through WidenArray::from_stored_decimal (D40), and for a legacy DecimalByteParts MSP that is wider than its logical dtype (D16). A read without a copy trusts the precision and does not scan The copy reads every value anyway, so the precision check costs nothing extra, and it matches the import rule of D13. A check without a copy needs a scan that the read does not otherwise make
D22 An integer Widen child has the same signedness and nullability as its parent, and a strictly smaller width. So an I64 column with values from 0 to 200 gets an I16 child, not a U8 child Mixed signedness needs more code in casts, in the compare shortcut and in the arithmetic bounds, and it saves at most one width. Part 1 of PR #10333 also keeps the signedness of internal buffers
D23 vortex.widen is only for columns with a fixed logical dtype: decimal columns and top-level integer columns. Which arrays the compressor wraps in a Widen is deferred (D27). Internal integer children keep changing their own dtype, through the deprecated encode_values (D28). These include list offsets, dictionary codes, run ends, patch and sparse indices, DateTimeParts, and VarBin and FSST offsets. The DecimalByteParts MSP has a fixed logical dtype instead, and it narrows with an in-memory integer Widen (D16). Its plugin writes the narrow child, so no vortex.widen reaches the wire. Whether List and ListView offsets get a fixed I64 dtype with Widen is an open question on issue #10733, and the decimal work does not depend on it The parent chooses the dtype of an internal child, so the child can narrow its dtype without a Widen. A column has a dtype that the user chose
D24 The compat fixtures are published from a release before the PRs that change how files are read (D31), because only the old writer can make narrow leaves. They cover only reads that succeed: narrow, wide and Arrow-style widths, gap values, high scales at narrow widths, nonzero values in null slots, and DecimalByteParts with a narrow MSP. Error cases, such as a value outside the precision in a narrowing copy, are unit tests The compat harness (vortex-test/compat-gen) expects the current reader to read every published fixture
D25 Generic structural encodings always sit above Widen. Reduce rules move Dict, RunEnd, Sparse, Constant, Masked, Chunked and Interleave above a Widen. The chunked and interleave merge rules of the local rewrite, which put Widen on top, are dropped. A follow-up issue covers the same rule for every encoding The Arrow and DuckDB exporters and the dict layout look for these encodings at the root. One direction for every rule cannot loop
D26 CD4 lowers a Widen to its canonical array in ArraySerializationTree::try_new (vortex-array/src/serde.rs). It executes the Widen to RecursiveCanonical, and the result inherits the statistics of the Widen. The lowering is crate-private, adds no public API, and applies to Widen only. It runs before the plugin lookup, and the recursion over the children lowers a nested Widen. A TODO(connor)[Widen] marks it. When vortex.widen gets a wire form (WR1), the lowering stays only for contexts that do not permit that ID. LT1 decides its end. ArraySession does not register Widen until WR1 gives it a wire form, because a registered VTable also gets a reader entry under vortex.widen that can only fail, which hides the allow_unknown foreign fallback and lists the ID in footers written without editions. Until WR1, Widen::serialize panics with a TODO(connor)[Widen], and the lowering keeps the serializer from calling it After CD4, a widening decimal cast returns a Widen, so IPC and writers without a compressor meet one in ordinary use. After CD5 and CD6, reads of narrow DecimalByteParts columns and of old nodes return one too. IPC, flat writes, the CUDA layout, Python serialize and pickling all pass through try_new. The lowered nodes intern only vortex.decimal, vortex.primitive and vortex.bool, so a context that does not permit vortex.widen accepts them, and footers never list it. The default file writer needed no lowering, because the compressor canonicalizes its input. Python __vortex_array_metadata__ calls array_serialize directly, so the lowering does not cover it
D27 Which arrays the compressor narrows is deferred to the design pass of WR3 The candidates include struct fields, list elements, extension storage, and arrays that layouts create, such as dict-layout codes and zone maps
D28 Widen has encode, which builds a Widen, and encode_values, which returns only the narrowed child with the same signedness. encode_values is #[deprecated]. It replaces PrimitiveArrayExt::narrow at every call site, and each call carries #[expect(deprecated)] with a TODO(connor) reason. The PR that moves the call sites checks that the latest release reads the files that it writes Every narrowing that changes a dtype is then marked for a later decision. Readers on develop accept signed internal children
D29 The DecimalByteParts constructor rejects a lower-part count whose assembled width is wider than the canonical width. Precision up to 18 allows no lower parts, 19 to 38 at most one, and 39 to 76 any count Decoding then never needs a narrowing copy after assembly. No file breaks, because no edition permits lower parts
D30 DecimalArray::try_new takes an ExecutionCtx, because the value check reads the validity, which can be lazy. Imports check their own values against the Arrow null buffer and then use the unchecked constructors, so a zero-copy import scans once This follows VarBinViewArray::try_new, and it keeps the context out of the import API
D31 Library changes land first. Only the PRs that change how files are read, CD5 (DecimalByteParts decoding) and CD6 (the read plugin), wait for the compat fixtures. Release 0.91.0 publishes them. CD7 merges right after CD5. This changes D24 The fixtures must be in the store when CI first runs the changed reader. Nothing else needs them. CD7 follows CD5 at once, so that the old scheme does not split decimals by their stored width for long (#10905)
D32 The vortex.widen metadata is a prost message with a typed oneof: the child PType of an integer parent, or the child precision of a decimal parent. A missing variant is an error, and the PType decodes strictly The typed oneof keeps the two kinds apart, and a missing variant cannot decode silently as U8
D33 The compressor uses a real Narrow scheme at first. If the compressor turns out to always want it, it can become a hard-coded step. The design pass of WR3 settles the cascade level that it uses. Since 2026-11-05, the scheme narrows only integers, and DecimalByteParts is the only scheme that narrows decimals A scheme fits the existing framework. The AlwaysUse contract and the cascade budget are the two costs to settle. A DecimalByteParts MSP that is an integer Widen reads with no scan and can narrow next to lower parts, while a narrow decimal node reads through the scan of D40, so a decimal Narrow scheme would only add a worse path. Floats work the same way: ALP turns them into integers, and the integer schemes do the rest
D34 Strict decoding covers the decimal width, the protobuf PType and the eleven PType fields of array and layout metadata, in one PR with three commits It is the same bug with the same fix
D35 The integer Widen lands as five stacked PRs: the encoding, benchmarks, aggregates, arithmetic and casts, and the structural rules. The PR that moves the call sites to encode_values (WD6) needs only WD1. Since 2026-10-28, it no longer waits for WD5. The dictionary fill-code fix lands before the fifth PR The benchmarks exist before the kernels change, and the fill-code fix guards the dictionary-codes rule
D36 An operation on a Widen fails only where the same operation on the parent dtype fails, and it gives the same result. An aggregate on a Widen runs in the accumulator of the parent dtype, and every partial that reaches it equals the partial that the parent builds from the decoded values. A kernel can run the same aggregate on the child and widen the child partial to the partial dtype of the parent, but only for an aggregate where decoding gives the same result. For integers these are MinMax, Min, Max, IsConstant, IsSorted, and Sum when the batch cannot overflow and the accumulator sums each batch from zero. For decimals these are MinMax, Min, Max, IsConstant, IsSorted, and Sum with skip_nans. CD4 adds the decimal Sum kernel. It runs Sum with the dtypes of the parent on the child, so the sum has the sum dtype of the parent. SumV2 and sums without skip_nans decline. MinMax on a Widen reads the exact cached extrema of the child through min_max. Every other aggregate keeps the default path Widen is a physical encoding, so it must not change behavior. Integer decoding is strictly increasing, one-to-one and keeps validity, so these partials widen exactly, and the child runs the same accumulator code, so edge cases match by construction. A decimal sum checks overflow at the precision of its input, so a child sum can overflow before the parent sum, and decimals need their own argument. Sum with skip_nans has the legacy Stat::Sum bridge, which sums each batch from zero. So the decimal kernel makes the same checked adds as the canonical batch, and its result is exact. Without the bridge, the canonical batch adds to the running sum, and the precision check runs only at the end. A kernel that sums the batch alone can then fail where the canonical batch does not. A running sum near the bound of Decimal(13, 0), the sum dtype of Decimal(3, 0), is a counterexample. A child from from_stored_decimal records its extrema (D40), so MinMax needs no scan
D37 The vortex.decimal read plugin converts old narrow nodes on the host, also on CUDA read paths. A GPU scan of such a node copies its buffer to the host Old narrow nodes are rare, and CUDA has no Widen kernel
D38 The integer Widen PRs are written from scratch from their briefs. The local rewrite of #10223 and #10332 is a reference for tests and edge cases, not a base. FX3 ports the rewrite's dictionary fill fix as it is The rewrite predates D25, D26, D28 and D36, so a cherry-pick would carry its old structure into the new PRs
D39 The encoding is renamed from Narrow to Widen, after the widening cast its decode applies (#9156). The compressor scheme stays Narrow and produces Widen arrays. The wire ID is vortex.widen. Plan IDs NR1 to NR6 become WD1 to WD6, and benchmark input Wide becomes Canonical Encodings are named after the function their decode applies, like FoR is an addition
D40 WidenArray::from_stored_decimal(parts, known_range: Precision<RangeInclusive<i256>>, ctx) is the single owner of the choice between the canonical array and a Widen for stored decimal values. CD4 adds it, and CD6 uses it. Since 2026-11-05, DecimalByteParts decoding does not use it (D16). It first checks the buffer length, the alignment and the validity length, and returns an error on a mismatch. At the canonical width it returns the canonical array with no copy and no scan, and a device buffer stays on the device. Any other width copies the values on the host. A device buffer is an error until #11284 reads legacy nodes on the GPU. Precision::Absent scans the valid values, and Exact is the smallest and the largest valid value. The child records either one as exact Stat::Min and Stat::Max. Inexact is only a bound, and the child records it as inexact. WidenArray::encode_decimal shares the core that picks the child. The function that picks the child dtype, smallest_decimal_dtype, is crate-private One function owns the child label (D10), so two readers cannot choose different children for the same values. A false label gives wrong answers in the compare shortcut and in casts. A read never panics, so the checks come before any code that asserts on a length
D41 The PRs land in this order. CD3c, "Speed up the decimal widening copy", is a PR off develop. CD4 is off develop too and does not depend on CD3c. CD4b, "Export decimal Widen arrays to Arrow with one copy", is stacked on CD4. CD5 and CD6 are each stacked on CD4, and CD7 is stacked on CD5. CD3c, CD4 and CD4b have no release gate. CD3c merges first, so that the CodSpeed run of CD4 measures the fast copy. CD5 and CD6 wait for 0.91.0 (D31), and CD7 merges right after CD5. CD5, CD7 and CD6 aim for one release, with CD4b already merged. On 2026-11-05, #10905 (CD5) and #10992 (CD7) closed for the redesign of D16. LT3 lands on develop first. One PR for CD5, with CD7 folded in, stacks on LT3 and merges only after #11284. The new rule, the reader, and the writer must change together: a CD5 alone would need a temporary writer fix for the narrowed_decimal code that CD7 deletes, and a CD7 alone would put Widen MSPs in memory before the decoder and the GPU handle them. The PR can split later if review needs it. The scan design does not land in between, even if the replacements miss the release of CD6 CD5 needs the decimal Widen and the helper of D40, so it moves onto CD4. Paths that need the canonical array, such as DuckDB, Arrow above precision 18 and the canonicalize bench, still pay one widening copy (D3), so CD3c makes that copy fast. CD4b removes one of the two copies of an Arrow export up to precision 18
D42 The DecimalByteParts benchmarks in encodings/decimal-byte-parts/benches/dbp_compute.rs keep their develop form: decode runs to the canonical array with execute::<DecimalArray>, which tracks the widening copy, sum, min_max, and both compare benches run on the DecimalByteParts array, and filter runs to the canonical array. The canonical-width control stays. LT3 changes no benchmark, and decode keeps its name, so CodSpeed keeps its history. CD5 only builds the narrow MSP inputs as a Widen. decompress in vortex/benches/common_encoding_tree_throughput.rs keeps its develop form too DecimalByteParts never decodes to a Widen (D16), so a bench that stops at a Widen runs to the canonical array. The narrow form shows only in the operations that run on the parts. The 2026-10-29 version stopped decode and filter at a Widen with execute_until::<Widen>

D1 to D6 date from 2026-10-19, D7 and D8 from 2026-10-22, and D12 to D38 from 2026-10-23. On 2026-10-23 the maintainer also revised D9, D10 and D11. Their 2026-10-22 versions used P(W), the largest precision whose canonical width is W, as the child precision of old data, and they planned the work as one complete change. A second review on 2026-10-23 added D21 to D24, and it changed D8, D9, D11, D16, D17 and D20. A planning review on the same day added D25 to D38, and it changed D8, D12, D14, D15, D16, D17, D18, D23 and D24. On 2026-10-26 the maintainer changed D26 and D36. D26 now says where the workaround lives. The 2026-10-23 version of D36 fed the child to the accumulator of the parent dtype, which rejects a batch of another dtype. On 2026-10-27, in the review of #10890, the maintainer changed D36 again: a kernel can run the aggregate on the child and widen its partial, for the aggregates where decoding gives the same result. The 2026-10-26 version allowed no partial cast, so its kernels rebuilt each partial from free functions whose edge cases differ from the accumulator. On 2026-10-28 the maintainer renamed the encoding to Widen (D39). The tracking issues and #10946 record two more changes from that day: CD4 adds no serialization lowering, so D26 moves to CD6, and WD6 no longer waits for WD5 (D35). On 2026-10-29 the maintainer approved a plan to land CD4 to CD7 without large regressions. The first version of #10905 widened every narrow MSP with a copy, and CodSpeed showed regressions of up to 11x. So DecimalByteParts decoding now returns a decimal Widen, LT2 becomes part of CD5, and CD5 depends on CD4 (D16). That plan changed D9 (one rule for every stored width), D16, D21 (the narrowing copies go through the helper), D26, D31 (the release is 0.91.0, and CD7 merges right after CD5) and D36 (the decimal Sum and MinMax kernels). It added D40 (the helper), D41 (the order and the stacks) and D42 (the decode benchmarks). D26 is settled: CD4 adds the lowering. This replaces the note from earlier on 2026-10-29 that moved D26 to CD6. Part 2 of the Decimal notes explains the design decisions from D7 on, and part 4 splits the work into PRs.

On 2026-11-05 the maintainer redesigned DecimalByteParts as the decimal over an integer child (D16). The MSP keeps the logical dtype of the canonical width, capped at i64, and a narrower MSP is an integer Widen, so decoding needs no scan. This closed #10905 (CD5) and #10992 (CD7). Aggregates with no lower parts run on the MSP, and #11284 adds a device Widen kernel that the new decoding needs. It changed D16, D17 (the MSP becomes a Widen), D19 (pushdown into the MSP is required), D21 (the DecimalByteParts copy is only for legacy MSPs), D23 (the MSP has a fixed logical dtype), D40 (DecimalByteParts no longer uses the helper), D41 (one PR for CD5 with CD7 folded in, on LT3), and D42 (the benchmarks run on the DecimalByteParts array). LT2 is dropped, and LT3 lands first. It also changed D33: the Narrow scheme of WR3 narrows only integers, and DecimalByteParts is the only scheme that narrows decimals.

Open

These questions belong to future work (D20). Their recommendations are not reviewed.

ID Question Recommendation
DQ5 Are wide integers user-visible columns, and which kernels do they get first? User-visible behind an edition. A small kernel set first: compare, arithmetic, cast, sum, min, max, hash and structural operations
DQ6 Does decimal arithmetic accept operands with different precision and scale? Yes, with the Arrow rules for the result type. Comparisons keep the rule of equal dtypes
DQ7 How does FixedSizeBinary appear on the wire? As a new core dtype tag behind an edition

The Widen pull requests

This review read the draft stack on 2026-10-19 and again on 2026-10-22. The state below is from 2026-10-30. The PR lists in #10701 and #10733 track the state of each new PR.

  • On 2026-10-30, these PRs of the plan have merged: FX1 (#10755), FX3 (#10745) and its follow-up #10806, CD1 (#10752), CD2 (#10753), CD3 (#10897) with #10903, CD3b (#10939), the decimal Interleave kernel (#10940), the benchmark of CD3c (#10990), and WD1 to WD4 (#10864, #10888, #10890, #10891). WD5 has no PR yet. The 0.91.0 release, which publishes the CD2 fixtures, is still a draft, so CD5 and CD6 wait for it.

  • CD4 now lands as a stack of six PRs. #11002 writes a Widen as its canonical array (the lowering of D26) and is off develop. On top of it, #10946 adds decimal values to Widen, #11003 runs compare, between and fills on the child, #11004 runs the aggregates on the child, #11005 adds WidenArray::from_stored_decimal (D40), and #11006 keeps same-scale widening casts lazy. CD4b (#10994) is stacked on #10946. CD5 (#10905) and CD6 (#10995) are stacked on #11005, and CD7 (#10992) is stacked on CD5. CD3c (#10989) is open off develop, and WD6 (#10943) is an open draft.

  • The draft PRs #10223, #10332 and #10333 date from 2026-10-06. The maintainer closed all three on 2026-10-23, and new PRs replace them. Issue #10733 tracks the new PRs.

  • A rewrite of #10223 and #10332 exists in five local commits, but it is not pushed. It makes Widen an in-memory encoding with no serialized form. The new PRs are written from scratch, and the rewrite is only a reference for tests and edge cases (D38). It predates #10681, which removed array_session(). The local branch has the name ct/narrow-pushdown, which is the head branch of the closed #10332, so the new PRs use new branch names.

  • The rewrite also fixes a fill code that the dictionary fill_null kernel truncates. That bug exists on develop without Widen, so the fix becomes its own PR.

  • The only integer-only check of #10223 and #10332 is the private validate_dtypes, so decimal support stays additive. When that check opens for decimals, the compare kernel and the sum pushdown need decimal code in the same change.

  • The CodSpeed reports on #10223 and #10332 compare against 3c87da0, because cc018ee had no run. Their gains, such as those of the FSST benchmarks, do not come from Widen. The new PRs need new reports.

  • The work of PR #10333 comes back in new PRs:

    1. WD1 adds WidenArray::encode_values, and WD6 replaces PrimitiveArrayExt::narrow with it at every call site (D28). Internal buffers then keep their signedness in every preset. Golden snapshots and file contents change, but no wire format changes.
    2. The serialization of D8, the edition entry and the Narrow scheme in the compressor are part of the wire change, WR1 to WR3 (D11, D18).
  • Part 4 of the Decimal notes lists every new PR, and the briefs in decimal/agents/briefs/ describe each one.

  • The CodSpeed report on #10333 shows take_map regressions of 22% and 11% in Simulation mode. The take_map benchmark builds u64 patch indices directly and does not call the narrowing code. The cause is not known. The new PRs get fresh CodSpeed runs, and the investigation happens there.

  • PR #10230 gave decimals an integer child, which D2 replaces. It closed on 2026-10-19. Part 3 of the Decimal notes replaces its steps. Its guard on the DecimalByteParts cast rule became FX2 (#10746), which closed without a merge on 2026-10-26. The guard was needed only while a canonical decimal held a lazy child. The rule pushes the cast into the MSP, which holds the validity, and the MSP cast fails on a null row, so no result was wrong. #10798 documents the two valid forms of a cast rule on CastReduce instead: a rule that builds a new Validity keeps the trivially_cast_nullability guard and a CastKernel, and a rule that pushes the cast into the children that hold its validity needs neither.

  • PR #10305 limits the decimal storage width to the canonical width, and PR #10314 adds bounded dispatch macros on top of it. Another contributor owns both, and both are open. The work ignores them for now (D11).