[DO NOT MERGE] Research notes on extension types and casts - #10633

Aligns the short-term nullability rule across the design documents, fixes
the count of typed wrappers in vortex-spatial, and uses a checked multiply
in the unit cast example.

Signed-off-by: Claude <[email protected]>
Claude-Session: https://claude.ai/code/session_01BKSPCwTaE2gzKwpxokMGXb

Research notes on making Decimal an opaque wrapper over an integer
child, the choice between an integer child and a FixedSizeBinary
child, a wide-integer tier, FixedSizeBinary as its own type, and a
dependency graph for the type work. The agents directory holds the
full report and the call-site and PType inventories.

The extension type notes now link to the Decimal notes, record the
position of the wrapper relative to generic encodings as an open
question, and state which parts of the Decimal work wait for the
extension phases.

Signed-off-by: Claude <[email protected]>
Claude-Session: https://claude.ai/code/session_01BKSPCwTaE2gzKwpxokMGXb

Derives are now a property of one extension dtype. A derive applies
only when every operand in a derived position has that dtype.
Operations across dtypes or metadata need explicit compatibility,
coercion or rewrite rules, and the notes do not assume them.

The design notes now define these parts:

- A Rust contract for Scalar equality and order that keeps PartialEq
  and PartialOrd consistent.
- Exact rules for the removal of ext_storage and ext_wrap pairs, for
  checked and unchecked wraps.
- A validation model with validated and unvalidated arrays, and a
  default validate_array that checks each value.
- Function declarations that authorize type-aware behavior, and a
  precise storage-only invariant.
- The position of the wrapper relative to generic encodings, as an
  open decision with a labeled recommendation.

A follow-up probe ran in vortex-btrblocks. It confirms the
DateTimeParts compare bug (EXT-Q3), the Decimal to f32 cast
disagreement, and a new finding about scalar storage checks
(EXT-V7). The notes also correct the U5 and U6 node mapping, a claim
about float equality, and a claim about the decimal wire form.

Signed-off-by: Connor Tsui <[email protected]>

U1 now also removes the storage target case in ExtScalar::cast.
Without that change, constants and the row path still cast an
extension value to its storage dtype after the array path stops.

The Decimal rewrite of a cast to an integer keeps the current rule: a
value outside the target range fails before truncation. Only min and
max wrap their result again, because is_sorted and is_constant return
Bool.

Signed-off-by: Connor Tsui <[email protected]>

The new research/decisions.md records the decisions from the review
on 2026-10-19 and the questions that stay open. The other notes
follow it.

Extension types now implement their behavior as rewrite rules on
arrays. The rules are registered with the type, and there is no
ext_wrap expression. A lift rule pulls generic structural arrays,
such as Constant and Dict, out from under the extension wrapper.
ExtensionArray::try_new checks values, and a new new_unchecked
serves kernels.

Decimal stays a leaf with one buffer and no data child. The
precision fixes the buffer width, so each decimal dtype has one
canonical encoding. A Narrow encoding stores a column at a lower
precision with the same scale. A reader wraps the narrow most
significant part of an old DecimalByteParts array in an integer
Narrow, without a copy.

The notes also list two costs of the one-width rule: an old I128
leaf at precision 39 to 76 needs a widening copy, and an Arrow
import at another width copies its buffer.

Signed-off-by: Connor Tsui <[email protected]>

Add decisions D21 to D24 and update D8, D9, D11, D16, D17 and D20
after a confirmation review of the decimal and Narrow design.

Signed-off-by: Connor Tsui <[email protected]>

#10223, #10332 and #10333 closed on 2026-10-23, and new PRs replace
them. Note that the local rewrite needs a rebase over #10681 and new
branch names.

Signed-off-by: Connor Tsui <[email protected]>

Split the work into 22 PRs with one brief for each, and record decisions
D25 to D35 from the planning review.

Signed-off-by: Connor Tsui <[email protected]>

Fix the dependency data, mark the PRs that share files, and explain how
agents reach the briefs and rebase stacks. Record D36 and D37, rename the
canonical width function, and let agents run any command the work needs.

Signed-off-by: Connor Tsui <[email protected]>

Treat the local rewrite as a reference for tests and edge cases, not as
a base (D38). Build every Narrow aggregate accumulator from the parent
dtype (D36), and explain D36 and D37 in the design notes.

Signed-off-by: Connor Tsui <[email protected]>

Lists the draft PRs and follow-up issues opened on 2026-10-23, and notes in the CD6 brief that it must update the round-trip check that CD2 adds.

Signed-off-by: Connor Tsui <[email protected]>

Records why FX2 was dropped (#10798 documents the cast-rule contract), the merged CD2 fixtures, the error types without categories (#10080), and the FoR scheme change (#10757). The tracking issues #10701 and #10733 now hold the state of each PR, so part 4 points to them instead of a status table.

Signed-off-by: Connor Tsui <[email protected]>

A registered VTable also gets a reader entry under its wire ID. For
Narrow, which has no wire form yet, that entry can only fail, so it
hides the allow_unknown foreign fallback and lists vortex.narrow in
footers written without editions. NR1 (#10864) therefore leaves Narrow
out of ArraySession, lowers every Narrow before the serializer lookup,
and makes Narrow::serialize panic with a TODO. WR1 adds the
registration together with the working deserializer.

D26, the NR1 brief and the wire brief now say so. The NR1 brief also
gets the final deprecation note of encode_values.

Signed-off-by: Connor Tsui <[email protected]>

NR1 (#10864) no longer lowers a Narrow in ArraySerializationTree. In
NR1 nothing outside the tests creates a Narrow, so the branch protected
no real path, and a branch specific to Narrow does not belong in the
generic serializer while nothing needs it. Serializing a Narrow now
fails like it does for any other unregistered in-memory encoding.

D26 is now deferred to CD4, the first PR where a Narrow reaches writers
in ordinary use. CD4 chooses between a lowering specific to Narrow, a
generic lowering for arrays that cannot be written, and no lowering.
The NR1 brief, the CD4 and WR1 briefs, the design and the plan now say
so.

Signed-off-by: Connor Tsui <[email protected]>

NR1 (#10864) gave vortex-array/src/arrays/narrow/ the layout of the
other encodings: one file per operation in compute/, and the vtable in
vtable/. The integer Narrow brief now describes that layout, and the
NR3, NR4 and NR5 file tables put their code there. The CD4 and wire
briefs point to the same section.

Signed-off-by: Connor Tsui <[email protected]>

Signed-off-by: Connor Tsui <[email protected]>

The first version of #10905 widened every narrow DecimalByteParts MSP
with a copy, and CodSpeed showed regressions of up to 11x. The
maintainer approved a plan on 2026-10-29: decoding returns the
canonical array or a decimal Widen, LT2 becomes part of CD5, and CD5
depends on CD4.

D9 now has one rule for every stored width, so a wider old node can
also become a Widen. D16, D21 and the new D40 make
WidenArray::from_stored_decimal the single owner of the choice between
the canonical array and a Widen. D26 is settled: CD4 lowers a Widen in
the serializer. D31 names 0.91.0, and D36 records the decimal Sum and
MinMax kernels. D41 records the order and the stacks, with two new
PRs, and D42 the decode benchmarks that stop at a Widen.

The implementation plan follows: the PR table, the graph, the waves,
the later work, the rules for every PR and the risks.

Signed-off-by: Connor Tsui <[email protected]>

The shared helpers now list WidenArray::from_stored_decimal and a
crate-private smallest_decimal_dtype. The CD4 brief lists the commits
that the plan added. The CD5 brief replaces the widening-copy design
with the canonical array or a decimal Widen. The CD6 brief reads every
old node through the helper, and D26 is no longer its decision. The
CD7 brief is stacked on CD5 and records the snapshots that change.

The wire brief points LT2 at CD5 and the D26 lowering at CD4.

Signed-off-by: Connor Tsui <[email protected]>

The design notes still said that the D26 lowering was deferred to CD6,
and the roadmap still said that DecimalByteParts decoding widens a
narrow MSP. CD4 adds the lowering, and decoding now returns a Widen
over a narrow MSP.

Signed-off-by: Connor Tsui <[email protected]>

Claims replace the derives sketch: a closed `Claim { Eq, Ord }` enum,
`ExtVTable::claims`, `DType::holds`, and a check when an extension dtype
is built (X13 to X16). Each function has one per-call `extension_use`
answer, and `Storage` carries the result mapping, so the compiler forces
it (X17 to X19). Rules live on `ExtVTable` under the orphan rule (X20).
The storage pushdown becomes the default path of aggregate binding
(X21). The notes also settle statistics, file reads, null slots and
boundaries (X22 to X24) and exact extension casts (C2), and they record
the lossy primitive casts as a known issue.

Part 5 gets a new phase plan, and the decimal report and roadmap follow
the new phases and settled questions.

Signed-off-by: Connor Tsui <[email protected]>

The plan now lists the five places where a change that lands alone
breaks working behavior, and maps every finding to the change or phase
that fixes it.

Signed-off-by: Connor Tsui <[email protected]>

CD4 now lands as a stack of six PRs, and the CD3c benchmark merged. The
notes also name the open PRs for the urgent extension changes and the
fixes for #10756.

Signed-off-by: Connor Tsui <[email protected]>

Adds X25 to X34 and C3 to the decisions, and changes X15 and X19. An
extension type cannot choose its top-level nullability, owns its nested
nullability, and declares its storage from its metadata. Reads are
strict, the only generic extension cast changes the top-level
nullability, and `Timestamp` accepts every `i64`.

Updates the design, the plan (the declaration starts phase 1, a sixth
order constraint, the findings table), the urgent changes (U4 rework,
U5 merged, PR state) and the design notes (section 9 rewritten, new
section 6.5 on rewrites through generic encodings). Adds the findings
EXT-V9, EXT-N2 and EXT-S3.

Signed-off-by: Connor Tsui <[email protected]>

… research README

Signed-off-by: Connor Tsui <[email protected]>

Signed-off-by: Connor Tsui <[email protected]>

The phase graph of the extension plan now shows each PR of phase 0, its
state and the order constraints. Phase 1 depends on U4 and U5, and its
strict reads wait for the spatial fix. Phase 0 holds #10953, the spatial
fix and the importer change, and the structural part of X29 moves out of
phase 6. The PR state lists #11013 as merged and adds #11067.

In the type roadmap, N6 becomes the storage declaration (X25 to X30),
with hard edges from N1 and into N3, and the report matches.

Signed-off-by: Connor Tsui <[email protected]>

…on plan

Signed-off-by: Connor Tsui <[email protected]>

A maintainer commit merged develop into #10953 again on 2026-11-02, so the PR no longer conflicts and waits for a review. Part 4 records one local run of the split_temporal benchmark, because CodSpeed cannot compare the changed cases with develop.

Signed-off-by: Connor Tsui <[email protected]>

Opacity in #11114 now lands with hardcoded opt-ins. `ExtensionUse` has
only `None`, `Structural` and `Specific`, and the built-in functions
answer `Specific` for fixed lists of types that match the claims of X16
(new decision X35). The claims, `ExtensionUse::Storage`,
`ResultMapping`, the generic storage rewrite and the storage paths move
to #11115 as CL1 to CL3 and SP1 to SP7. So #11114 no longer depends on
#11115.

The declarations of phase 2 now land before the typing check of phase
3. Before, the check could have rejected timestamp compares until TY1
landed. U9 wraps the results of the storage pushdown with a fixed
mapping, so it no longer waits for `ResultMapping`.

Signed-off-by: Connor Tsui <[email protected]>

Adds EXT-V10: the `Display` of `Date` panics on valid storage values
that jiff cannot hold. X31 now says so, and the plan gives the finding
a follow-up PR on `develop`. Parts 4 and 5 record the review of #10953
on 2026-11-03, the local review commits, the removal of the
cached-bounds fast path, and the benchmark against `develop`.

Signed-off-by: Connor Tsui <[email protected]>

SR merges into SD, because a storage declaration without the exact read
check reads an old form silently wrong. SD now needs SN1, which only
tightens the spatial validators. SN2 takes every Arrow importer of a
known type and builds on DP, which peels dictionary fields before the
importers run. V10 and DS become their own PRs.

Also record that #10953 is pushed at 7943b4e and waits for an
approval, that the 12.9% millisecond slowdown was measured at
d6e73bd, and that #11056 merged.

Signed-off-by: Connor Tsui <[email protected]>

TS (#10953) merged on 2026-11-03, and SN1 (#11183) and V10 (#11187)
merged on 2026-11-04. The dependency graph marks the three merged and
moves its state date to 2026-11-04. It gives SD, DP, FD and U9 their
open PRs (#11184, #11185, #11186 and #11188), and it drops #10408,
which closed without a merge. The text of parts 4 and 5 and the Fix
lines of EXT-V9, EXT-V10, EXT-N2 and EXT-S3 now record the same state.

Also record that AX, DS and SN2 are implemented on branches without a
PR, and add the approved one-pass design for the masked nulls of SN2.

Part 2 and decision X31 also record the fixes: #10953 made every i64
legal, #11183 made the spatial validators reject nullable nested
levels, #11187 gave the Date display its fallback, and #10408 closed.

Signed-off-by: Connor Tsui <[email protected]>

DP (#11185) merged on 2026-11-04. SN2, AX and DS are open drafts as

Signed-off-by: Connor Tsui <[email protected]>
#11223, #11224 and #11225, so the graph and parts 4 and 5 link them.

SN3 drops masked nulls below every nested level after SN2. TY5 declares the
type-specific functions, which the typing check needs. DS2 rejects DataFusion
columns that mix an extension type and plain storage. #11114 now lists all
three.

Signed-off-by: Connor Tsui <[email protected]>

Record the merged AX and DS, the new RT, SF and AF nodes, and the related PRs that closed or merged. Mark the nodes that wait for a new design of the function declaration, since FD (#11186) closed after review.

Signed-off-by: Connor Tsui <[email protected]>

Signed-off-by: Connor Tsui <[email protected]>