Tracking Issue: Add an Array variant to ScalarValue #7699

This is a tracking issue for adding a ScalarValue::Array(ArrayRef) variant to vortex-array/src/scalar, replacing the per-row Vec<Option<ScalarValue>> representation for List and FixedSizeList scalars.

This supersedes #6771 and revives the abandoned work in #6717.

Motivation

In the past, scalars used to live in a vortex-scalar crate, and vortex-array depended on vortex-scalar. This prevented us from adding an ArrayRef variant to ScalarValue since that would have caused a dependency cycle.

In #6523, we moved the entire vortex-scalar crate into a submodule of vortex-array. So we now have the ability to do this.

There are several benefits to having an Array variant in our scalars:

• The main benefit is that working with List, ListView, and FixedSizeList scalars becomes much more performant as we do not need to unwrap each row value into an Option<ScalarValue>
  • Given that there are important extension types that deal with these list types (Uuid, Vector, tensors in general)
• IN list predicates are very slow in Vortex
• There are many potential Constant optimizations for constant List arrays, and working with a slice of an ArrayRef in the ScalarValue is much more performant
• TODO are there other things? Likely related to ConstantArray and SparseArray?

Design

vortex-array/src/scalar/scalar_value.rs

pub enum ScalarValue {
    // ...
    Tuple(Vec<Option<ScalarValue>>),  // Struct scalars
    Array(ArrayRef),                  // NEW! List + FixedSizeList scalars
    Variant(Box<Scalar>),
}

vortex-proto/proto/scalar.proto

message ScalarValue {
  oneof kind {
    // ... (existing tags 1..11)
    StructValue struct_value = 9;     // renamed from ListValue
    ArrayValue  array_value  = 12;    // NEW!
  }
}

// TODO is this enough?
message ArrayValue {
  uint64           row_count    = 1;
  repeated string  encoding_ids = 2;  // self-describing; no shared ArrayContext
  bytes            data         = 3;  // ArrayRef::serialize output
}

IMPORTANT TODO!!!

We might need to enforce that the inner ArrayRef is recursively canonicalized at construction so that Hash/PartialEq/Display/unpack_native stay session-free. Constructors might need to also add &VortexSession. For backcompat, a StructValue payload paired with a List DType materializes as ScalarValue::Array.

I think this is the main blocker, but it is unclear if this is the only one...

Steps

TBD, probably want to break this down into more steps?

• Add ScalarValue::Array, rename List -> Struct, route list/FSL scalars through Array.
• Proto changes (StructValue rename, new ArrayValue tag 12, backcompat read path).
• Thread &VortexSession through scalar constructors and to_proto so we can call ArrayRef::serialize(ctx, session, options) and recursively canonicalize.
• Restore Uuid::unpack_native and the vortex-tensor constant test helpers disabled in ScalarValue::Array #6717.
• Update external call sites (vortex-datafusion, vortex-duckdb, vortex-python, vortex-ipc, parquet-variant, sequence::list_contains).

Unresolved questions

• Do we need to canonicalize the inner ArrayRef recursively at construction, or keep it lazy?
• Does ExtVTable::unpack_native need to add a &VortexSession / &mut ExecutionCtx parameter?
• Do we really need to be able to Hash all of our scalar and scalar views?
• Does the ScalarFnVTable::Options really need the Eq and Hash bounds? Things like ScalarFnRef and Expression rely on that for optimization and rule matching.

Implementation history

• Semi-related: Fix Scalar de/serialization with proto and outline nulls from ScalarValue #6309
• ScalarValue::Array #6717 (closed and stale)
• Rename ScalarValue::List to ScalarValue::Tuple #7672
• TODO