Skip to content

Commit 3d37216

Browse files
connortsui20claude
andauthored
Add a Normalized encoding for tensor columns (#9138)
Adds `Normalized`, a norm-split array encoding for tensor columns with `normalized` and `norms` children. Tracking issue: #9129 `Normalized` owns the physical decomposition and validates the invariant between its children. Scalar functions remain operations over any well-typed input, while the encoding can guarantee that its direction child is unit-normalized and that its stored norm is valid. ## Changes - Renames `L2Denorm` and `encodings::l2_denorm` to `Normalized` and `encodings::normalized`. - `Normalized::try_new` validates exact normalized rows. `Normalized::new_unchecked` admits structurally valid lossy children whose stored norms remain authoritative. - Slice and filter push down into both children, preserving the encoding. - `NormalizedScheme` cascades compression into both children. - Cosine similarity, inner product, and L2 norm keep their read-through paths for `Normalized` arrays. ## API changes | Before | After | | --- | --- | | `scalar_fns::l2_denorm::*` | `encodings::normalized::*` | | `L2Denorm::try_new_array(n, s, ctx)` | `Normalized::try_new(n, s, ctx)` | | `unsafe L2Denorm::new_array_unchecked(n, s)` | `unsafe Normalized::new_unchecked(n, s)` | | `L2Denorm::new()` | removed | | `session.scalar_fns().register(L2Denorm)` | `session.arrays().register(Normalized)`, unconditional | | `ScalarFnArrayPlugin::new(L2Denorm)` | removed | The array ID is now `vortex.tensor.normalized`; the serialized child metadata layout is unchanged. `NormalizedScheme` remains out of `ALL_SCHEMES`. --------- Signed-off-by: Connor Tsui <[email protected]> Co-authored-by: Claude <[email protected]>
1 parent 3884b98 commit 3d37216

21 files changed

Lines changed: 1949 additions & 1366 deletions

File tree

‎Cargo.lock‎

Lines changed: 2 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎vortex-compressor/src/compressor/cascade.rs‎

Lines changed: 1 addition & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,6 @@ use vortex_array::arrays::fixed_size_list::FixedSizeListArraySlotsExt;
2323
use vortex_array::arrays::listview::ListViewArraySlotsExt;
2424
use vortex_array::arrays::listview::list_from_list_view;
2525
use vortex_array::arrays::masked::MaskedArraySlotsExt;
26-
use vortex_array::arrays::scalar_fn::AnyScalarFn;
2726
use vortex_array::arrays::struct_::StructArrayExt;
2827
use vortex_array::arrays::union::UnionArrayExt;
2928
use vortex_array::arrays::union::UnionArraySlotsExt;
@@ -177,10 +176,6 @@ impl CascadingCompressor {
177176
compress_ctx,
178177
exec_ctx,
179178
)?;
180-
// TODO(connor): HACK TO SUPPORT L2 DENORMALIZATION!!!
181-
if scheme_compressed.is::<AnyScalarFn>() {
182-
return Ok(scheme_compressed);
183-
}
184179

185180
// A constant extension array (that might be masked) is already in its terminal
186181
// representation, and compressing the storage separately cannot do better.
@@ -323,8 +318,7 @@ impl CascadingCompressor {
323318
let after_nbytes = compressed.nbytes();
324319
let actual_ratio = (after_nbytes != 0).then(|| before_nbytes as f64 / after_nbytes as f64);
325320

326-
// TODO(connor): HACK TO SUPPORT L2 DENORMALIZATION!!!
327-
let accepted = after_nbytes < before_nbytes || compressed.is::<AnyScalarFn>();
321+
let accepted = after_nbytes < before_nbytes;
328322

329323
trace::record_winner_compress_result(
330324
after_nbytes,

‎vortex-tensor/Cargo.toml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,3 +34,5 @@ prost = { workspace = true }
3434
[dev-dependencies]
3535
rstest = { workspace = true }
3636
vortex-array = { workspace = true, features = ["_test-harness"] }
37+
vortex-btrblocks = { workspace = true }
38+
vortex-mask = { workspace = true }

‎vortex-tensor/src/encodings/l2_denorm.rs‎

Lines changed: 0 additions & 61 deletions
This file was deleted.

‎vortex-tensor/src/encodings/mod.rs‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,9 @@
11
// SPDX-License-Identifier: Apache-2.0
22
// SPDX-FileCopyrightText: Copyright the Vortex contributors
33

4-
//! Encodings for the different tensor types.
4+
//! Array encodings for the different tensor types.
55
66
// TODO(connor):
77
// pub mod spherical; // Spherical transform on unit-normalized vectors.
88

9-
pub mod l2_denorm;
9+
pub mod normalized;

0 commit comments

Comments
 (0)