Repository navigation
Commit 85ed8e4
_Revived from #8791 since too
many things have changed._
## Summary
Tracking issue: #8769
Adds scalar implementation for `Union` type in Vortex.
## Changes
Adds `UnionValue` which exists inside of `ScalarValue` and the typed
view `UnionScalar<'a>`.
```rust
pub struct UnionValue {
/// The type ID selecting a variant in the enclosing [`DType::Union`].
type_id: u8,
/// The selected variant scalar.
///
/// This is boxed to break the recursive layout between [`ScalarValue`]
/// and [`UnionValue`].
value: `Option<Box<ScalarValue>>`,
}
```
### Inner Nulls with Type IDs
The type above implies that type IDs MATTER for equality of nulls. 2
values with the same union type might both be null, **but if the type
IDs are different, we consider them NOT equal.** Please respond below if
you think this is the incorrect behavior. I think that this might be
different from arrow's semantics, but because we are already
distinguishing from them with the outer nullability, I think this is
fine.
### Value type
~~I was debating if we should instead store a `Option<Box<ScalarValue>>`
instead of just `Box<Scalar>` (which carries an extra dtype and an extra
allocation even if the value is null). I think it is nice to have the
dtype in there for implementation ease. Not sure if it's worth it.~~
We use `Option<Box<ScalarValue>>` instead of `Box<Scalar>` so that we do
not store `DType` in multiple places. If we did so, we run into
synchronization issues with casting.
### Casting
For casting, this only allows identity and null casting (like everything
else), otherwise it just panics with unimplemented. We probably want to
figure this out later as it is quite valuable to cast union types.
## Default and Zero value semantics
Finally, the "default value" and "zero value" semantics here are as
such:
- `default_value(nullable Union)` -> returns an outer null.
- `default_value(non-nullable Union)` -> **panics** because there is no
designated default variant.
- `zero_value(Union)` -> **panics** because a Union has no canonical
zero variant.
- `is_zero()` on a null Union -> returns `None`.
- `is_zero()` on a valid Union -> returns `Some(false)`, even when its
child is numerically zero.
Note that none of these behaviors depend on variant ordering. And also I
do think that it is possible that we just choose the first variant of a
union to be the default, I think this is kind of opinionated, and it
also doesn't solve the issue where there are 0 variants (empty union /
void type). So I would prefer if we do this in a separate change.
---------
Signed-off-by: Connor Tsui <[email protected]>
Co-authored-by: Joe Isaacs <[email protected]>
1 parent 994591b commit 85ed8e4
18 files changed
Lines changed: 1386 additions & 57 deletions
File tree
- vortex-array/src
- arrays/constant/vtable
- scalar
- tests
- typed_view
- vortex-jni/src
- vortex-proto
- proto
- src/generated
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
95 | 95 | | |
96 | 96 | | |
97 | 97 | | |
98 | | - | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
99 | 114 | | |
100 | 115 | | |
101 | 116 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
58 | 58 | | |
59 | 59 | | |
60 | 60 | | |
61 | | - | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
62 | 65 | | |
63 | 66 | | |
64 | 67 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
11 | 14 | | |
12 | 15 | | |
13 | 16 | | |
14 | 17 | | |
15 | 18 | | |
16 | 19 | | |
17 | 20 | | |
| 21 | + | |
18 | 22 | | |
19 | 23 | | |
20 | 24 | | |
21 | 25 | | |
22 | 26 | | |
23 | 27 | | |
24 | 28 | | |
| 29 | + | |
25 | 30 | | |
26 | 31 | | |
27 | 32 | | |
| |||
190 | 195 | | |
191 | 196 | | |
192 | 197 | | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
193 | 240 | | |
194 | 241 | | |
195 | 242 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
23 | | - | |
| 23 | + | |
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
| |||
30 | 30 | | |
31 | 31 | | |
32 | 32 | | |
| 33 | + | |
33 | 34 | | |
34 | 35 | | |
35 | 36 | | |
36 | 37 | | |
37 | 38 | | |
38 | 39 | | |
39 | 40 | | |
| 41 | + | |
40 | 42 | | |
41 | 43 | | |
42 | 44 | | |
| |||
79 | 81 | | |
80 | 82 | | |
81 | 83 | | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
82 | 129 | | |
83 | 130 | | |
84 | 131 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
| 22 | + | |
| 23 | + | |
22 | 24 | | |
23 | 25 | | |
24 | 26 | | |
| |||
156 | 158 | | |
157 | 159 | | |
158 | 160 | | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
159 | 181 | | |
160 | 182 | | |
161 | 183 | | |
| |||
273 | 295 | | |
274 | 296 | | |
275 | 297 | | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
276 | 314 | | |
277 | 315 | | |
278 | 316 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
7 | | - | |
8 | | - | |
| 7 | + | |
| 8 | + | |
9 | 9 | | |
10 | 10 | | |
11 | 11 | | |
| |||
0 commit comments