Skip to content

Commit 85ed8e4

Browse files
Add UnionValue and UnionScalar (#8838)
_Revived from #8791 since too many things have changed._ ## Summary Tracking issue: #8769 Adds scalar implementation for `Union` type in Vortex. ## Changes Adds `UnionValue` which exists inside of `ScalarValue` and the typed view `UnionScalar<'a>`. ```rust pub struct UnionValue { /// The type ID selecting a variant in the enclosing [`DType::Union`]. type_id: u8, /// The selected variant scalar. /// /// This is boxed to break the recursive layout between [`ScalarValue`] /// and [`UnionValue`]. value: `Option<Box<ScalarValue>>`, } ``` ### Inner Nulls with Type IDs The type above implies that type IDs MATTER for equality of nulls. 2 values with the same union type might both be null, **but if the type IDs are different, we consider them NOT equal.** Please respond below if you think this is the incorrect behavior. I think that this might be different from arrow's semantics, but because we are already distinguishing from them with the outer nullability, I think this is fine. ### Value type ~~I was debating if we should instead store a `Option<Box<ScalarValue>>` instead of just `Box<Scalar>` (which carries an extra dtype and an extra allocation even if the value is null). I think it is nice to have the dtype in there for implementation ease. Not sure if it's worth it.~~ We use `Option<Box<ScalarValue>>` instead of `Box<Scalar>` so that we do not store `DType` in multiple places. If we did so, we run into synchronization issues with casting. ### Casting For casting, this only allows identity and null casting (like everything else), otherwise it just panics with unimplemented. We probably want to figure this out later as it is quite valuable to cast union types. ## Default and Zero value semantics Finally, the "default value" and "zero value" semantics here are as such: - `default_value(nullable Union)` -> returns an outer null. - `default_value(non-nullable Union)` -> **panics** because there is no designated default variant. - `zero_value(Union)` -> **panics** because a Union has no canonical zero variant. - `is_zero()` on a null Union -> returns `None`. - `is_zero()` on a valid Union -> returns `Some(false)`, even when its child is numerically zero. Note that none of these behaviors depend on variant ordering. And also I do think that it is possible that we just choose the first variant of a union to be the default, I think this is kind of opinionated, and it also doesn't solve the issue where there are 0 variants (empty union / void type). So I would prefer if we do this in a separate change. --------- Signed-off-by: Connor Tsui <[email protected]> Co-authored-by: Joe Isaacs <[email protected]>
1 parent 994591b commit 85ed8e4

18 files changed

Lines changed: 1386 additions & 57 deletions

File tree

‎vortex-array/src/arrays/constant/vtable/operations.rs‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,3 +18,47 @@ impl OperationsVTable<Constant> for Constant {
1818
Ok(array.scalar.clone())
1919
}
2020
}
21+
22+
#[cfg(test)]
23+
mod tests {
24+
use vortex_error::VortexResult;
25+
26+
use crate::IntoArray;
27+
use crate::VortexSessionExecute;
28+
use crate::arrays::ConstantArray;
29+
use crate::dtype::DType;
30+
use crate::dtype::Nullability;
31+
use crate::dtype::PType;
32+
use crate::dtype::UnionVariants;
33+
use crate::scalar::Scalar;
34+
35+
#[test]
36+
fn scalar_at_preserves_union_scalar() -> VortexResult<()> {
37+
let variants = UnionVariants::try_new(
38+
["int", "string"].into(),
39+
vec![
40+
DType::Primitive(PType::I32, Nullability::Nullable),
41+
DType::Utf8(Nullability::NonNullable),
42+
],
43+
vec![5, 9],
44+
)?;
45+
46+
let scalar = Scalar::union(
47+
variants.clone(),
48+
5,
49+
Scalar::primitive(42_i32, Nullability::Nullable),
50+
Nullability::Nullable,
51+
)?;
52+
let array = ConstantArray::new(scalar.clone(), 3).into_array();
53+
let mut ctx = crate::array_session().create_execution_ctx();
54+
55+
assert_eq!(array.execute_scalar(1, &mut ctx)?, scalar);
56+
57+
let null = Scalar::null(DType::Union(variants, Nullability::Nullable));
58+
let array = ConstantArray::new(null.clone(), 3).into_array();
59+
60+
assert_eq!(array.execute_scalar(1, &mut ctx)?, null);
61+
62+
Ok(())
63+
}
64+
}

‎vortex-array/src/scalar/arbitrary.rs‎

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -95,7 +95,22 @@ pub fn random_scalar(u: &mut Unstructured, dtype: &DType) -> Result<Scalar> {
9595
)),
9696
)
9797
.vortex_expect("unable to construct random `Scalar`_"),
98-
DType::Union(..) => todo!("TODO(connor)[Union]: unimplemented"),
98+
DType::Union(variants, nullability) => {
99+
let child_index = u.choose_index(variants.len())?;
100+
101+
let child_dtype = variants
102+
.variant_by_index(child_index)
103+
.vortex_expect("chosen union child index must be valid");
104+
let child = random_scalar(u, &child_dtype)?;
105+
106+
Scalar::union(
107+
variants.clone(),
108+
variants.child_index_to_tag(child_index),
109+
child,
110+
*nullability,
111+
)
112+
.vortex_expect("generated union scalar must be valid")
113+
}
99114
DType::Variant(_) => todo!(),
100115
DType::Extension(..) => {
101116
unreachable!("Can't yet generate arbitrary scalars for ext dtype")

‎vortex-array/src/scalar/cast.rs‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -58,7 +58,10 @@ impl Scalar {
5858
DType::Binary(_) => self.as_binary().cast(target_dtype),
5959
DType::List(..) | DType::FixedSizeList(..) => self.as_list().cast(target_dtype),
6060
DType::Struct(..) => self.as_struct().cast(target_dtype),
61-
DType::Union(..) => todo!("TODO(connor)[Union]: unimplemented"),
61+
DType::Union(..) => vortex_bail!(
62+
"union scalar cast from {} to {target_dtype} is not supported (yet)",
63+
self.dtype()
64+
),
6265
DType::Variant(_) => vortex_bail!("Variant scalars can't be cast to {target_dtype}"),
6366
DType::Extension(..) => self.as_extension().cast(target_dtype),
6467
}

‎vortex-array/src/scalar/constructor.rs‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,20 +8,25 @@ use std::sync::Arc;
88
use vortex_buffer::BufferString;
99
use vortex_buffer::ByteBuffer;
1010
use vortex_error::VortexExpect;
11+
use vortex_error::VortexResult;
12+
use vortex_error::vortex_ensure_eq;
13+
use vortex_error::vortex_err;
1114
use vortex_error::vortex_panic;
1215

1316
use crate::dtype::DType;
1417
use crate::dtype::DecimalDType;
1518
use crate::dtype::NativePType;
1619
use crate::dtype::Nullability;
1720
use crate::dtype::PType;
21+
use crate::dtype::UnionVariants;
1822
use crate::dtype::extension::ExtDType;
1923
use crate::dtype::extension::ExtDTypeRef;
2024
use crate::dtype::extension::ExtVTable;
2125
use crate::scalar::DecimalValue;
2226
use crate::scalar::PValue;
2327
use crate::scalar::Scalar;
2428
use crate::scalar::ScalarValue;
29+
use crate::scalar::UnionValue;
2530

2631
// TODO(connor): Really, we want `try_` constructors that return errors instead of just panic.
2732
impl Scalar {
@@ -190,6 +195,48 @@ impl Scalar {
190195
.vortex_expect("unable to construct an extension `Scalar`")
191196
}
192197

198+
/// Creates a union scalar from a type ID and child scalar.
199+
///
200+
/// The selected child's dtype is verified and then discarded. Its raw value is stored so an
201+
/// inner null child remains distinct from a null at the outer union level.
202+
///
203+
/// # Errors
204+
///
205+
/// Returns an error if the type ID is not present in `variants` or if the child scalar's dtype
206+
/// does not exactly match the selected variant dtype.
207+
pub fn union(
208+
variants: UnionVariants,
209+
type_id: u8,
210+
child: Scalar,
211+
nullability: Nullability,
212+
) -> VortexResult<Self> {
213+
let child_index = variants.tag_to_child_index(type_id).ok_or_else(|| {
214+
vortex_err!(
215+
"union type ID {type_id} is not present in {:?}",
216+
variants.type_ids()
217+
)
218+
})?;
219+
220+
let expected_dtype = variants
221+
.variant_by_index(child_index)
222+
.vortex_expect("type ID resolved to a valid child index");
223+
224+
vortex_ensure_eq!(
225+
child.dtype(),
226+
&expected_dtype,
227+
"union type ID {type_id} selects child dtype {expected_dtype}, got {}",
228+
child.dtype()
229+
);
230+
231+
Self::try_new(
232+
DType::Union(variants, nullability),
233+
Some(ScalarValue::Union(UnionValue::new(
234+
type_id,
235+
child.into_value(),
236+
))),
237+
)
238+
}
239+
193240
/// Creates a new variant scalar from a row-specific nested scalar.
194241
///
195242
/// Use [`Scalar::null(DType::Variant(Nullability::Nullable))`][Scalar::null] for a top-level

‎vortex-array/src/scalar/display.rs‎

Lines changed: 48 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ impl Display for Scalar {
2020
DType::Binary(_) => write!(f, "{}", self.as_binary()),
2121
DType::List(..) | DType::FixedSizeList(..) => write!(f, "{}", self.as_list()),
2222
DType::Struct(..) => write!(f, "{}", self.as_struct()),
23-
DType::Union(..) => todo!("TODO(connor)[Union]: unimplemented"),
23+
DType::Union(..) => write!(f, "{}", self.as_union()),
2424
DType::Variant(_) => write!(f, "{}", self.as_variant()),
2525
DType::Extension(_) => write!(f, "{}", self.as_extension()),
2626
}
@@ -30,13 +30,15 @@ impl Display for Scalar {
3030
#[cfg(test)]
3131
mod tests {
3232
use vortex_buffer::ByteBuffer;
33+
use vortex_error::VortexResult;
3334

3435
use crate::dtype::DType;
3536
use crate::dtype::FieldName;
3637
use crate::dtype::Nullability::NonNullable;
3738
use crate::dtype::Nullability::Nullable;
3839
use crate::dtype::PType;
3940
use crate::dtype::StructFields;
41+
use crate::dtype::UnionVariants;
4042
use crate::extension::datetime::Date;
4143
use crate::extension::datetime::Time;
4244
use crate::extension::datetime::TimeUnit;
@@ -79,6 +81,51 @@ mod tests {
7981
);
8082
}
8183

84+
#[test]
85+
fn display_union() -> VortexResult<()> {
86+
let variants = UnionVariants::new(
87+
["int", "string"].into(),
88+
vec![
89+
DType::Primitive(PType::I32, Nullable),
90+
DType::Utf8(NonNullable),
91+
],
92+
)?;
93+
94+
let scalar = Scalar::union(
95+
variants.clone(),
96+
0,
97+
Scalar::primitive(42_i32, Nullable),
98+
Nullable,
99+
)?;
100+
101+
assert_eq!(format!("{scalar}"), "int(42i32)");
102+
let inner_null = Scalar::union(
103+
variants.clone(),
104+
0,
105+
Scalar::null(DType::Primitive(PType::I32, Nullable)),
106+
Nullable,
107+
)?;
108+
assert_eq!(format!("{inner_null}"), "int(null)");
109+
assert_eq!(
110+
format!("{}", Scalar::null(DType::Union(variants, Nullable))),
111+
"null"
112+
);
113+
114+
let struct_dtype = DType::struct_(
115+
[("field", DType::Primitive(PType::I32, NonNullable))],
116+
NonNullable,
117+
);
118+
let struct_scalar = Scalar::struct_(
119+
struct_dtype.clone(),
120+
[Scalar::primitive(42_i32, NonNullable)],
121+
);
122+
let struct_variants = UnionVariants::new(["record"].into(), vec![struct_dtype])?;
123+
let struct_union = Scalar::union(struct_variants, 0, struct_scalar, NonNullable)?;
124+
assert_eq!(format!("{struct_union}"), "record({field: 42i32})");
125+
126+
Ok(())
127+
}
128+
82129
#[test]
83130
fn display_utf8() {
84131
assert_eq!(

‎vortex-array/src/scalar/downcast.rs‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,8 @@ use crate::scalar::PrimitiveScalar;
1919
use crate::scalar::Scalar;
2020
use crate::scalar::ScalarValue;
2121
use crate::scalar::StructScalar;
22+
use crate::scalar::UnionScalar;
23+
use crate::scalar::UnionValue;
2224
use crate::scalar::Utf8Scalar;
2325
use crate::scalar::VariantScalar;
2426

@@ -156,6 +158,26 @@ impl Scalar {
156158
Some(ExtScalar::new_unchecked(self.dtype(), self.value()))
157159
}
158160

161+
/// Returns a view of the scalar as a union scalar.
162+
///
163+
/// # Panics
164+
///
165+
/// Panics if the scalar does not have a [`Union`](crate::dtype::DType::Union) type.
166+
pub fn as_union(&self) -> UnionScalar<'_> {
167+
self.as_union_opt()
168+
.vortex_expect("Failed to convert scalar to union")
169+
}
170+
171+
/// Returns a view of the scalar as a union scalar if it has a union type.
172+
pub fn as_union_opt(&self) -> Option<UnionScalar<'_>> {
173+
if !self.dtype().is_union() {
174+
return None;
175+
}
176+
177+
// Scalar construction has already validated the value against this union dtype.
178+
Some(UnionScalar::new_unchecked(self.dtype(), self.value()))
179+
}
180+
159181
/// Returns a view of the scalar as a variant scalar.
160182
///
161183
/// # Panics
@@ -273,6 +295,22 @@ impl ScalarValue {
273295
}
274296
}
275297

298+
/// Returns the union value, panicking if the value is not a [`Union`](ScalarValue::Union).
299+
pub fn as_union(&self) -> &UnionValue {
300+
match self {
301+
ScalarValue::Union(value) => value,
302+
_ => vortex_panic!("ScalarValue is not a Union"),
303+
}
304+
}
305+
306+
/// Returns the union value, panicking if the value is not a [`Union`](ScalarValue::Union).
307+
pub fn into_union(self) -> UnionValue {
308+
match self {
309+
ScalarValue::Union(value) => value,
310+
_ => vortex_panic!("ScalarValue is not a Union"),
311+
}
312+
}
313+
276314
/// Returns the row-specific scalar wrapped by a variant, panicking if the value is not a
277315
/// variant.
278316
pub fn as_variant(&self) -> &Scalar {

‎vortex-array/src/scalar/mod.rs‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,8 +4,8 @@
44
//! Scalar values and types for the Vortex system.
55
//!
66
//! This crate provides scalar types and values that can be used to represent individual data
7-
//! elements in the Vortex array system. [`Scalar`]s are composed of a logical data type ([`DType`])
8-
//! and an optional (encoding nullability) value ([`ScalarValue`]).
7+
//! elements in the Vortex array system. A [`Scalar`] pairs a logical data type ([`DType`]) with an
8+
//! optional non-null value ([`ScalarValue`]); [`None`] represents a null scalar.
99
//!
1010
//! Note that the implementations of `Scalar` are split into several different modules.
1111
//!

0 commit comments

Comments
 (0)