[DO NOT MERGE] Research notes on Decimal, Widen, and the type roadmap - #11237

Aligns the short-term nullability rule across the design documents, fixes
the count of typed wrappers in vortex-spatial, and uses a checked multiply
in the unit cast example.

Signed-off-by: Claude <[email protected]>
Claude-Session: https://claude.ai/code/session_01BKSPCwTaE2gzKwpxokMGXb

Research notes on making Decimal an opaque wrapper over an integer
child, the choice between an integer child and a FixedSizeBinary
child, a wide-integer tier, FixedSizeBinary as its own type, and a
dependency graph for the type work. The agents directory holds the
full report and the call-site and PType inventories.

The extension type notes now link to the Decimal notes, record the
position of the wrapper relative to generic encodings as an open
question, and state which parts of the Decimal work wait for the
extension phases.

Signed-off-by: Claude <[email protected]>
Claude-Session: https://claude.ai/code/session_01BKSPCwTaE2gzKwpxokMGXb

Derives are now a property of one extension dtype. A derive applies
only when every operand in a derived position has that dtype.
Operations across dtypes or metadata need explicit compatibility,
coercion or rewrite rules, and the notes do not assume them.

The design notes now define these parts:

- A Rust contract for Scalar equality and order that keeps PartialEq
  and PartialOrd consistent.
- Exact rules for the removal of ext_storage and ext_wrap pairs, for
  checked and unchecked wraps.
- A validation model with validated and unvalidated arrays, and a
  default validate_array that checks each value.
- Function declarations that authorize type-aware behavior, and a
  precise storage-only invariant.
- The position of the wrapper relative to generic encodings, as an
  open decision with a labeled recommendation.

A follow-up probe ran in vortex-btrblocks. It confirms the
DateTimeParts compare bug (EXT-Q3), the Decimal to f32 cast
disagreement, and a new finding about scalar storage checks
(EXT-V7). The notes also correct the U5 and U6 node mapping, a claim
about float equality, and a claim about the decimal wire form.

Signed-off-by: Connor Tsui <[email protected]>

U1 now also removes the storage target case in ExtScalar::cast.
Without that change, constants and the row path still cast an
extension value to its storage dtype after the array path stops.

The Decimal rewrite of a cast to an integer keeps the current rule: a
value outside the target range fails before truncation. Only min and
max wrap their result again, because is_sorted and is_constant return
Bool.

Signed-off-by: Connor Tsui <[email protected]>

The new research/decisions.md records the decisions from the review
on 2026-10-19 and the questions that stay open. The other notes
follow it.

Extension types now implement their behavior as rewrite rules on
arrays. The rules are registered with the type, and there is no
ext_wrap expression. A lift rule pulls generic structural arrays,
such as Constant and Dict, out from under the extension wrapper.
ExtensionArray::try_new checks values, and a new new_unchecked
serves kernels.

Decimal stays a leaf with one buffer and no data child. The
precision fixes the buffer width, so each decimal dtype has one
canonical encoding. A Narrow encoding stores a column at a lower
precision with the same scale. A reader wraps the narrow most
significant part of an old DecimalByteParts array in an integer
Narrow, without a copy.

The notes also list two costs of the one-width rule: an old I128
leaf at precision 39 to 76 needs a widening copy, and an Arrow
import at another width copies its buffer.

Signed-off-by: Connor Tsui <[email protected]>

Add decisions D21 to D24 and update D8, D9, D11, D16, D17 and D20
after a confirmation review of the decimal and Narrow design.

Signed-off-by: Connor Tsui <[email protected]>

#10223, #10332 and #10333 closed on 2026-10-23, and new PRs replace
them. Note that the local rewrite needs a rebase over #10681 and new
branch names.

Signed-off-by: Connor Tsui <[email protected]>

Split the work into 22 PRs with one brief for each, and record decisions
D25 to D35 from the planning review.

Signed-off-by: Connor Tsui <[email protected]>

Fix the dependency data, mark the PRs that share files, and explain how
agents reach the briefs and rebase stacks. Record D36 and D37, rename the
canonical width function, and let agents run any command the work needs.

Signed-off-by: Connor Tsui <[email protected]>

Treat the local rewrite as a reference for tests and edge cases, not as
a base (D38). Build every Narrow aggregate accumulator from the parent
dtype (D36), and explain D36 and D37 in the design notes.

Signed-off-by: Connor Tsui <[email protected]>

Lists the draft PRs and follow-up issues opened on 2026-10-23, and notes in the CD6 brief that it must update the round-trip check that CD2 adds.

Signed-off-by: Connor Tsui <[email protected]>

Records why FX2 was dropped (#10798 documents the cast-rule contract), the merged CD2 fixtures, the error types without categories (#10080), and the FoR scheme change (#10757). The tracking issues #10701 and #10733 now hold the state of each PR, so part 4 points to them instead of a status table.

Signed-off-by: Connor Tsui <[email protected]>

A preflight review checked the NR1 to NR4 briefs against develop. This commit
applies its corrections and the decisions made with it:

- D26: the serialization workaround is crate-private, in
  ArraySerializationTree::try_new. No public trait method is added.
- D36: an aggregate on a Narrow runs in the accumulator of the parent dtype.
  Kernels build the exact parent partial and never cast a partial state. The
  sum kernel serves only Sum with skip_nans.
- NR1 adds a take kernel instead of a TakeReduce rule (D25), names the
  helpers that NR4 reuses, and adds conformance tests.
- NR2 fixes its session, its input recipe and its workloads.
- NR4 declines operands that are neither a Narrow nor a constant.
- The stack lives in vortex2 and uses gh stack v0.0.8.

Signed-off-by: Connor Tsui <[email protected]>

A registered VTable also gets a reader entry under its wire ID. For
Narrow, which has no wire form yet, that entry can only fail, so it
hides the allow_unknown foreign fallback and lists vortex.narrow in
footers written without editions. NR1 (#10864) therefore leaves Narrow
out of ArraySession, lowers every Narrow before the serializer lookup,
and makes Narrow::serialize panic with a TODO. WR1 adds the
registration together with the working deserializer.

D26, the NR1 brief and the wire brief now say so. The NR1 brief also
gets the final deprecation note of encode_values.

Signed-off-by: Connor Tsui <[email protected]>

NR1 (#10864) no longer lowers a Narrow in ArraySerializationTree. In
NR1 nothing outside the tests creates a Narrow, so the branch protected
no real path, and a branch specific to Narrow does not belong in the
generic serializer while nothing needs it. Serializing a Narrow now
fails like it does for any other unregistered in-memory encoding.

D26 is now deferred to CD4, the first PR where a Narrow reaches writers
in ordinary use. CD4 chooses between a lowering specific to Narrow, a
generic lowering for arrays that cannot be written, and no lowering.
The NR1 brief, the CD4 and WR1 briefs, the design and the plan now say
so.

Signed-off-by: Connor Tsui <[email protected]>

NR1 (#10864) gave vortex-array/src/arrays/narrow/ the layout of the
other encodings: one file per operation in compute/, and the vtable in
vtable/. The integer Narrow brief now describes that layout, and the
NR3, NR4 and NR5 file tables put their code there. The CD4 and wire
briefs point to the same section.

Signed-off-by: Connor Tsui <[email protected]>

Signed-off-by: Connor Tsui <[email protected]>

Signed-off-by: Connor Tsui <[email protected]>

A check of the briefs against NR1 (#10864) found nine mismatches. The
NR3 kernels need a pub(crate) compute module, and the decode function
moves to array.rs, because the vtable modules are private. The
fastlanes test directory does not exist yet. There is no
ZipFalseReduce trait, so NR4 writes that rule by hand, and it reuses
integer_value instead of copying it. Narrow is not registered in
ArraySession. The other fixes update moved line numbers and the NR2
session recipe.

Signed-off-by: Connor Tsui <[email protected]>

A Narrow aggregate kernel can now run the aggregate on the child and widen
its partial, for the integer aggregates where decoding gives the same
result. The NR3 brief notes the design that landed.

Signed-off-by: Connor Tsui <[email protected]>

Signed-off-by: Connor Tsui <[email protected]>

X10 records that ExtensionStoragePushdown (#10566, #10616) is a
temporary bridge that phase 2 replaces, and part 3 explains how it
differs from the design. X11 records that derives are a goal: authors
opt in to common behavior with little code, and the Derives sketch is
not a decision. XQ7 asks which mechanism gives that opt-in.

The extension plan now adds ExtensionArray::new_unchecked in phase 1,
because the rules of phases 2 and 3 build their results with it. Phase
4 splits into the lift rule (4a), which waits for nothing and is the
extension case of #10748, and statistics with DateTimeParts (4b).
Phase 2 starts with a design pass for XQ2, XQ5 and XQ7, and its typing
covers lambda bodies.

The type graph gets the missing N2 to N4 edge, and the reason for N2
to N3 now follows the design notes. The design notes rename their
derive rules to DR1 to DR4 and their nullability steps to NU1 and NU2,
so that they no longer collide with D1 to D4 and N1 and N2.

Signed-off-by: Connor Tsui <[email protected]>

No commit since 5fc5cb8 fixed a finding, so U1 to U8 are all still
needed. Part 4 now gives line numbers at 1d09649. U5 also covers
Time, whose Span setters panic like those of Timestamp. U2 fixes the
DataFusion literal conversion of EXT-B1 in the same change, and checks
the dynamic filter path. U6 makes the claim in the unpack_native docs
true.

EXT-A1 records #10566 and #10616, which add composite aggregates on
storage for the temporal types and Uuid, while plain is_sorted and
min_max keep their old answers. #10246 closed without a merge. The
Duration and Interval PRs touch the code of U1, U2 and phase 3.

Signed-off-by: Connor Tsui <[email protected]>

WD1 to WD4, CD3, CD3b and the decimal Interleave kernel merged, and
CD4, CD5 and WD6 are open drafts. The plan now lists CD3b (#10939) and
the Interleave kernel (CD4p, #10940) as dependencies of CD4, and CD3b
as a dependency of CD6 and CD7. CD7 merges right after CD5 (#10905).

D26 moves to CD6, because CD4 (#10946) adds no serialization lowering:
the default writer executes a Widen, and reads of old files are the
first ordinary source of one. The CD6 and WR1 briefs now carry that
choice. D35 no longer makes WD6 wait for WD5. #10908 joins the
follow-up issues, and the plan suggests settling the mechanism of

Signed-off-by: Connor Tsui <[email protected]>
#10748 before WD5.

ExtensionStoragePushdown stays as the way to run an aggregate on the
storage of an extension type (X10). A declaration on the type registers
it, in place of the register_storage_ordered calls that crates write by
hand. Its results that are values of the input get the extension dtype
back inside the adapter (X12). Finding EXT-A3 and the urgent change U9
record the current behavior and the fix.

The lift rule of #10748 is not needed for correctness, so phase 4a has
low priority, and WD5 no longer waits for its mechanism.

Signed-off-by: Connor Tsui <[email protected]>

The first version of #10905 widened every narrow DecimalByteParts MSP
with a copy, and CodSpeed showed regressions of up to 11x. The
maintainer approved a plan on 2026-10-29: decoding returns the
canonical array or a decimal Widen, LT2 becomes part of CD5, and CD5
depends on CD4.

D9 now has one rule for every stored width, so a wider old node can
also become a Widen. D16, D21 and the new D40 make
WidenArray::from_stored_decimal the single owner of the choice between
the canonical array and a Widen. D26 is settled: CD4 lowers a Widen in
the serializer. D31 names 0.91.0, and D36 records the decimal Sum and
MinMax kernels. D41 records the order and the stacks, with two new
PRs, and D42 the decode benchmarks that stop at a Widen.

The implementation plan follows: the PR table, the graph, the waves,
the later work, the rules for every PR and the risks.

Signed-off-by: Connor Tsui <[email protected]>

The shared helpers now list WidenArray::from_stored_decimal and a
crate-private smallest_decimal_dtype. The CD4 brief lists the commits
that the plan added. The CD5 brief replaces the widening-copy design
with the canonical array or a decimal Widen. The CD6 brief reads every
old node through the helper, and D26 is no longer its decision. The
CD7 brief is stacked on CD5 and records the snapshots that change.

The wire brief points LT2 at CD5 and the D26 lowering at CD4.

Signed-off-by: Connor Tsui <[email protected]>

The design notes still said that the D26 lowering was deferred to CD6,
and the roadmap still said that DecimalByteParts decoding widens a
narrow MSP. CD4 adds the lowering, and decoding now returns a Widen
over a narrow MSP.

Signed-off-by: Connor Tsui <[email protected]>

Claims replace the derives sketch: a closed `Claim { Eq, Ord }` enum,
`ExtVTable::claims`, `DType::holds`, and a check when an extension dtype
is built (X13 to X16). Each function has one per-call `extension_use`
answer, and `Storage` carries the result mapping, so the compiler forces
it (X17 to X19). Rules live on `ExtVTable` under the orphan rule (X20).
The storage pushdown becomes the default path of aggregate binding
(X21). The notes also settle statistics, file reads, null slots and
boundaries (X22 to X24) and exact extension casts (C2), and they record
the lossy primitive casts as a known issue.

Part 5 gets a new phase plan, and the decimal report and roadmap follow
the new phases and settled questions.

Signed-off-by: Connor Tsui <[email protected]>

CD4 now lands as a stack of six PRs, and the CD3c benchmark merged. The
notes also name the open PRs for the urgent extension changes and the
fixes for #10756.

Signed-off-by: Connor Tsui <[email protected]>

Adds X25 to X34 and C3 to the decisions, and changes X15 and X19. An
extension type cannot choose its top-level nullability, owns its nested
nullability, and declares its storage from its metadata. Reads are
strict, the only generic extension cast changes the top-level
nullability, and `Timestamp` accepts every `i64`.

Updates the design, the plan (the declaration starts phase 1, a sixth
order constraint, the findings table), the urgent changes (U4 rework,
U5 merged, PR state) and the design notes (section 9 rewritten, new
section 6.5 on rewrites through generic encodings). Adds the findings
EXT-V9, EXT-N2 and EXT-S3.

Signed-off-by: Connor Tsui <[email protected]>

… research README

Signed-off-by: Connor Tsui <[email protected]>

Signed-off-by: Connor Tsui <[email protected]>

The phase graph of the extension plan now shows each PR of phase 0, its
state and the order constraints. Phase 1 depends on U4 and U5, and its
strict reads wait for the spatial fix. Phase 0 holds #10953, the spatial
fix and the importer change, and the structural part of X29 moves out of
phase 6. The PR state lists #11013 as merged and adds #11067.

In the type roadmap, N6 becomes the storage declaration (X25 to X30),
with hard edges from N1 and into N3, and the report matches.

Signed-off-by: Connor Tsui <[email protected]>

…on plan

Signed-off-by: Connor Tsui <[email protected]>

U2, U3, U4, U6, U7 and U8 merged on 2026-11-02 in #11012, #11014 and
#11016, and the DataFusion literal fix #11067 merged right after U2. So
order constraint 1 holds, and phase 1 starts from develop. The graph
shows these nodes as merged, and parts 2 and 4 name the PR that fixed
each finding.

#10953 now keeps i32 days and widens only the days part to i64 when a
value does not fit. It has no review, and it conflicts with the
DateTimeParts compare kernel that U2 changed. #11056 still waits on its
request for changes. #10661 replaces the closed #10482 for file sort
statistics.

In the type roadmap, only the N1 sentence changes.

Signed-off-by: Connor Tsui <[email protected]>

A maintainer commit merged develop into #10953 again on 2026-11-02, so the PR no longer conflicts and waits for a review. Part 4 records one local run of the split_temporal benchmark, because CodSpeed cannot compare the changed cases with develop.

Signed-off-by: Connor Tsui <[email protected]>

…11114

Each node of the graph is now one PR, with the ID that the PR plan in #11114 uses. The claims node points to #11115, the old phase 1 node splits into NU and CL, and the shared pushdown and the lift rule become separate low-priority nodes. The graph also links #10661, #10686 and #10793 as open PRs to coordinate with.

Signed-off-by: Connor Tsui <[email protected]>

Opacity in #11114 now lands with hardcoded opt-ins. `ExtensionUse` has
only `None`, `Structural` and `Specific`, and the built-in functions
answer `Specific` for fixed lists of types that match the claims of X16
(new decision X35). The claims, `ExtensionUse::Storage`,
`ResultMapping`, the generic storage rewrite and the storage paths move
to #11115 as CL1 to CL3 and SP1 to SP7. So #11114 no longer depends on
#11115.

The declarations of phase 2 now land before the typing check of phase
3. Before, the check could have rejected timestamp compares until TY1
landed. U9 wraps the results of the storage pushdown with a fixed
mapping, so it no longer waits for `ResultMapping`.

Signed-off-by: Connor Tsui <[email protected]>

Adds EXT-V10: the `Display` of `Date` panics on valid storage values
that jiff cannot hold. X31 now says so, and the plan gives the finding
a follow-up PR on `develop`. Parts 4 and 5 record the review of #10953
on 2026-11-03, the local review commits, the removal of the
cached-bounds fast path, and the benchmark against `develop`.

Signed-off-by: Connor Tsui <[email protected]>

SR merges into SD, because a storage declaration without the exact read
check reads an old form silently wrong. SD now needs SN1, which only
tightens the spatial validators. SN2 takes every Arrow importer of a
known type and builds on DP, which peels dictionary fields before the
importers run. V10 and DS become their own PRs.

Also record that #10953 is pushed at 7943b4e and waits for an
approval, that the 12.9% millisecond slowdown was measured at
d6e73bd, and that #11056 merged.

Signed-off-by: Connor Tsui <[email protected]>

TS (#10953) merged on 2026-11-03, and SN1 (#11183) and V10 (#11187)
merged on 2026-11-04. The dependency graph marks the three merged and
moves its state date to 2026-11-04. It gives SD, DP, FD and U9 their
open PRs (#11184, #11185, #11186 and #11188), and it drops #10408,
which closed without a merge. The text of parts 4 and 5 and the Fix
lines of EXT-V9, EXT-V10, EXT-N2 and EXT-S3 now record the same state.

Also record that AX, DS and SN2 are implemented on branches without a
PR, and add the approved one-pass design for the masked nulls of SN2.

Part 2 and decision X31 also record the fixes: #10953 made every i64
legal, #11183 made the spatial validators reject nullable nested
levels, #11187 gave the Date display its fallback, and #10408 closed.

Signed-off-by: Connor Tsui <[email protected]>

DP (#11185) merged on 2026-11-04. SN2, AX and DS are open drafts as

Signed-off-by: Connor Tsui <[email protected]>
#11223, #11224 and #11225, so the graph and parts 4 and 5 link them.

SN3 drops masked nulls below every nested level after SN2. TY5 declares the
type-specific functions, which the typing check needs. DS2 rejects DataFusion
columns that mix an extension type and plain storage. #11114 now lists all
three.

Signed-off-by: Connor Tsui <[email protected]>

Signed-off-by: Connor Tsui <[email protected]>

Signed-off-by: Connor Tsui <[email protected]>

Signed-off-by: Connor Tsui <[email protected]>

…#11284

Signed-off-by: Connor Tsui <[email protected]>