Decimal is a built-in dtype with the shape of an extension type: an integer storage value plus a
scale. On develop, the decimal array does not keep to that shape. One decimal dtype can have six
physical forms, and Decimal duplicates much of Primitive. These notes describe the target
design that the maintainer settled on 2026-10-19, 2026-10-22 and 2026-10-23, and they put the type
work that it needs in order. decisions.md records the decisions and the open
questions. Issue #10701 tracks the fix of the
canonical decimal encoding.
| Part | Content |
|---|---|
| 1. Current state | How Decimal works on develop, and what is wrong |
| 2. Design | The leaf, imports and Arrow, Widen, DecimalByteParts, the file format, and the future work |
| 3. Roadmap | The dependency graph of the type work, the independent changes, and the order of work |
| 4. Implementation plan | The PRs of the Widen and canonical decimal work, their dependencies and waves, with one brief for each PR |
- The width of the decimal buffer does not depend on the precision. So one dtype has up to six physical forms, every kernel must handle mixed widths, and some paths panic.
- Decided: a decimal array stays a leaf with one buffer and validity. Each decimal dtype has exactly one canonical encoding, because the precision fixes the width of the buffer. Each precision gets the smallest signed width that holds it.
- Decided:
DecimalArray::try_newchecks the width and every valid value. Imports copy into the canonical width, and they fail on a value outside the precision. File reads trust the precision. - Decided: a
Widenencoding stores a column as a child with a lower precision and the same scale. The child must have a narrower canonical width. OneWidenencoding serves integers and decimals, and its kernels pass work to the child. An integer child keeps the signedness of its parent. The wire form is only for columns with a fixed logical dtype, and internal integer children keep changing their own dtype. - Precision limits are powers of ten, and widths are powers of two. So a width does not give a
precision: an
i32holds 2,000,000,000, which needs precision 10. Each producer finds the child precision from the values. - Decided: a reader finds an old narrow
vortex.decimalnode by its stored width. It scans the values for their max, and it returns aWidenor a copy. Every narrowing copy writes zeros in null slots and fails on a value outside the precision. - Decided: DecimalByteParts stays the compressed form, and it is the decimal over an integer child.
Its MSP keeps the canonical width, capped at
i64, and a narrower MSP is an integerWiden, so decoding needs no scan. Operations run on its parts where they can, and it decodes to the canonical array of its own dtype. One compressor scheme builds it from the canonical array, so v1 file contents stay the same. - Decided:
vortex.widenstays in memory until the maintainer decides to write it. Then it enters one preview edition together with DecimalByteParts with lower parts. DecimalByteParts is the only scheme that narrows decimals (D33). Default writers usecore, so until both formats entercore, they write wide decimals with values above 64 bits as plaini256arrays. - Decided: the bar of no compute regression applies to the final state. A DecimalByteParts array keeps its narrow MSP in memory, and sum, min, and max run on the MSP.
- Decided: the work lands as 22 PRs in four tracks. Each PR has a brief that a coding agent can
implement alone. The integer
WidenPRs are written from scratch, with a local rewrite as a reference. Generic structural encodings always sit aboveWiden. A writer that cannot writevortex.widenwrites the canonical array, as a documented workaround. Which arrays the compressor narrows is deferred. - Arrow has the same decimal model and the same gap at the top of each width. Its width is part of the type, and from precision 5 the Vortex widths match the smallest Arrow types.
- Decided:
vortex.decimalkeeps its wire ID, and the decimal file format does not change. - Decided:
Decimalstays a built-in type that behaves like a hardcoded extension type. A move to an extension type is an optional later step. - Decided: Vortex adds
I128,I256,U128andU256, and FixedSizeBinary becomes a core type. Their designs are future work, outside the canonical decimal fix. So are the integer view, decimal arithmetic across precisions, and casts that lose information. The recommendations for this future work are not reviewed.
agents/briefs/: one brief for each PR of part 4, in four files.agents/report.md: the full report. It has file and line references, API sketches, the wire plan, the complete dependency graph with a reason for each edge, and the evidence log.agents/decimal-call-sites.md: every decimal-specific file with line counts, every use of the width function, and every canonical dispatch site.agents/ptype-match-inventory.md: everyPTypedispatch macro use, every exhaustivePTypematch, and every copy ofPTypein other languages and formats.
The notes come from reading develop at 5fc5cb8, RFC drafts, issues and pull requests. On
2026-10-19 the review also read the draft Widen PRs and PR #10230. On 2026-10-22 it read the
versioning spec, the array plugin and serialization code, and the DecimalByteParts code. It also
read the history of the decimal compression scheme, and PRs #10305 and #10314. On 2026-10-23 it
traced which statistics the compressors store on the DecimalByteParts parts, read the decimal types
of arrow-rs 59.3, and read the cast change of PR #10230. A second review on the same day read when
the reader attaches stored statistics, the DecimalByteParts constructor and compressor scheme, the
compat harness, and the edition families. A planning review on the same day had five agents draft
the PR briefs from the code at b70206818f, and the review checked their main claims in the code.
On 2026-10-26 it read the state of the first PRs, and PRs #10798 (the cast-rule contract), #10757
(the FoR scheme) and #10080 (error types without categories). One throwaway Rust test confirmed the
f32 cast disagreement and the ext_storage pushdown rules (probes R1 and R6 in the extension
notes). One short Python script checked byte order. Section 10 of the report lists the claims that
are not verified, which include every performance claim.