Repository navigation
Declare extension storage dtypes instead of validating them - #11184
Conversation
TS (#10953) merged on 2026-11-03, and SN1 (#11183) and V10 (#11187) merged on 2026-11-04. The dependency graph marks the three merged and moves its state date to 2026-11-04. It gives SD, DP, FD and U9 their open PRs (#11184, #11185, #11186 and #11188), and it drops #10408, which closed without a merge. The text of parts 4 and 5 and the Fix lines of EXT-V9, EXT-V10, EXT-N2 and EXT-S3 now record the same state. Also record that AX, DS and SN2 are implemented on branches without a PR, and add the approved one-pass design for the masked nulls of SN2. Part 2 and decision X31 also record the fixes: #10953 made every i64 legal, #11183 made the spatial validators reject nullable nested levels, #11187 gave the Date display its fallback, and #10408 closed. Signed-off-by: Connor Tsui <[email protected]>
3ff373a to
e6b34f0
Compare
f86ed0d to
1039771
Compare
1039771 to
b8cde46
Compare
Merging this PR will improve performance by 10.62%2 warnings
Performance Changes
Tip Curious why performance improved? Comment Comparing Footnotes
|
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.008x ➖, 0↑ 0↓)
|
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.015x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.004x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.013x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.009x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.990x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.003x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.000x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.988x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.998x ➖, 0↑ 0↓)
|
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.000x ➖, 0↑ 1↓)
|
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.996x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.994x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.006x ➖, 0↑ 1↓)
duckdb / parquet / ns (1.000x ➖, 1↑ 1↓)
|
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.015x ➖, 0↑ 3↓)
datafusion / parquet / ns (1.042x ➖, 0↑ 2↓)
duckdb / vortex-file-compressed / ns (0.985x ➖, 1↑ 1↓)
duckdb / parquet / ns (0.949x ➖, 1↑ 1↓)
|
Benchmarks: TPC-DS SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.997x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.996x ➖, 1↑ 0↓)
|
Each extension type validated the storage dtype that its caller passed, so the form of the storage was only as strict as the validator, and facts like the dimension of a Vector lived only in the storage. A nullability change also had to trust that the validator ignored the top-level nullability. A type now declares its storage dtype from its metadata with storage_dtype, and ExtDType::try_new applies the top-level nullability itself. validate_metadata checks the metadata, and deserialize_metadata receives the stored storage dtype, so the metadata can hold facts that its serialized bytes leave out. A read requires that the declared storage dtype equals the stored one exactly, including nested nullability, because arrays decode their storage child with the declared dtype. ForeignExtDType declares the storage that it was read with. The extension types and their callers move over in the following commits, which compile only together with this one. Signed-off-by: Connor Tsui <[email protected]>
Date and Time derive their storage from the time unit, and Timestamp always stores i64, so their declarations follow from the metadata that they already have. validate_metadata takes over the unit and time zone checks. The Date and Time unit errors now state the requirement, so the JNI tests expect the new text. ExtensionArray::try_new_from_vtable and Scalar::extension pass the nullability of their storage. When the storage dtype differs from the declaration, ExtensionArray::try_new and Scalar::try_extension_ref now name the type and both storage dtypes, as a read does. The test types declare the storage that their tests use. Apples carries its storage dtype as metadata, because its tests wrap several, and keeps it in the nullable form that a read rebuilds. The storage rejection tests become read tests. Signed-off-by: Connor Tsui <[email protected]>
Both storage dtypes follow from the type alone. The Uuid rejection tests become read tests, and one of them shows that a stored storage with nullable bytes fails to read with an error that names both dtypes. Signed-off-by: Connor Tsui <[email protected]>
The element type and dimension of a Vector, and the element type of a FixedShapeTensor, lived only in the storage dtype. The in-memory metadata now holds them, and deserialize_metadata reads them from the stored storage dtype, so the serialized bytes do not change. VectorMetadata holds the element type and dimension. It prints nothing, like the empty metadata before it, so printed dtypes stay the same. FixedShapeTensor rejects a shape whose dimensions or product do not fit the u32 list size, because storage_dtype cannot fail. Signed-off-by: Connor Tsui <[email protected]>
The coordinate dimension of the seven native geometry types lived only in the storage dtype. SpatialMetadata now holds it in memory, and validate_metadata requires it for a native type and rejects it for WKB. The serialized form keeps only the CRS, so the bytes and the printed dtypes do not change. Dimension becomes public, so code outside the crate can still build a native geometry dtype. Signed-off-by: Connor Tsui <[email protected]>
Reads now compare the complete stored storage dtype with the declaration, so spatial helpers only need to recover the dimension from the structure and field names. Keeping field type and nested nullability checks in the helpers would reject malformed storage before the read can report both complete dtypes. Regression cases cover nullable and wrongly typed coordinate fields for all seven native geometry types, including their enclosing lists. Signed-off-by: Connor Tsui <[email protected]>
The DuckDB, DataFusion, Python, and JNI conversions, the Arrow and compressor tests, and the benchmarks pass a nullability instead of a storage dtype. The UUID storage code in DuckDB, Python, and JNI goes away, because the Uuid declaration builds that storage. The Arrow adapter restores metadata against the native importer's inferred Vortex storage. The restored metadata must declare that storage exactly, so a side payload cannot change the representation promised by the field. Signed-off-by: Connor Tsui <[email protected]>
The guide described obsolete extension APIs and migration plans. Explain the implemented typed and erased dispatch path, metadata reconstruction, nullability, and strict storage checks using the current APIs. Signed-off-by: Connor Tsui <[email protected]>
b8cde46 to
3c109bf
Compare
Summary
Right now, an extension type validates whatever storage dtype the caller hands it, so one type can have several storage forms, and its metadata doesn't say which one a dtype uses. Each
ExtVTablenow declares its storage from its metadata, so the metadata deterministically fixes / decides the whole storage dtype except the top-level nullability.Reads then require the stored storage to equal that declaration exactly, including nested nullability. This has to be strict, since arrays decode their storage child with the declared dtype, so a stored form that differs would be misread instead of rejected. Every built-in type accepts the same stored forms as before, so existing files still read.
NB on nested nullability: this means that if we potentially need nulls in the storage, an extension type MUST require its storage type to be nullable, even if in some cases it might not be. I think this is a worthwhile tradeoff for clarity of implementation.
Changes
Some facts lived only in the storage dtype, like the element type of
VectorandFixedShapeTensorand the dimension of the native spatial types, so they move into the in-memoryMetadata.deserialize_metadatadecodesStoredMetadata(which is the oldMetadata, andmetadata_from_storagecombines it with storage to rebuild complete metadata. The serialized bytes don't change.Some other things:
ForeignExtDTypecan't derive its storage, since it doesn't know its type, so it keeps the stored storage in its metadata and declares that.Follow-up: a read error in a struct field still panics on first access (SF in #11114).
API Changes
ExtVTable::validate_dtypebecomesstorage_dtype(plus an optionalvalidate_metadata),deserialize_metadatareturnsStoredMetadata, andmetadata_from_storagereconstructs complete metadata from stored metadata and storage.ExtDType::try_newandtry_with_vtabletake a nullability in place of the storage dtype.FixedShapeTensorMetadata::newtakes the element type,Vector's metadata isVectorMetadata, metadata-first construction of native spatial dtypes needsSpatialMetadata::with_dimension, andSpatialMetadatano longer implementsprost::Message.ExtDTypeandExtensionArraygaintry_from_storageconstructors, andScalargainstry_extension_from_storage, with explicit-vtable variants. Existing metadata-first constructors remain available.FixedShapeTensorStoredMetadatasupplies shape and layout, andSpatialStoredMetadatasupplies CRS when constructing from storage.