Repository navigation
Commit bc9aa10
authored
Add Map logical type and scalar support (#9107)
## Rationale for this change
Vortex currently has no native logical map type, so Arrow maps must be
represented as lists of structs and lose map-specific semantics such as
non-null keys and the `keys_sorted` assertion. This provides the
type-level foundation for native Map support; #9108 builds on it with
canonical arrays and builders. Related to #428.
## What changes are included in this PR?
- Adds `DType::Map` and `MapDType`, including key/value dtypes, outer
nullability, sortedness, validation, coercion, display, equality, and
hashing.
- Adds map scalar construction and typed access through `Scalar::map`,
`Scalar::try_map`, and `MapScalar`.
- Adds map dtype and scalar serialization through Serde, Protobuf, and
FlatBuffers.
- Adds Arrow dtype/schema round-tripping, Vortex-to-Arrow map scalar
conversion, and DataFusion physical-schema reconciliation.
- Tests cover dtype and scalar validation, null and empty maps,
coercion, serialization round-trips, Arrow schema conversion, and nested
extension metadata.
## What APIs are changed? Are there any user-facing changes?
This adds the public `DType::Map` and `ScalarValue::Map` variants,
`MapDType`, `MapScalar`, and their constructors and accessors. Consumers
that exhaustively match `DType` or `ScalarValue` must handle the new
variants. The Protobuf and FlatBuffers dtype schemas also gain Map
variants.
This PR supports Map dtypes, schemas, and scalars, but not full Arrow
`MapArray` transport.
Signed-off-by: Adam Gutglick <[email protected]>1 parent fd43ebd commit bc9aa10
59 files changed
Lines changed: 1813 additions & 25 deletions
File tree
- encodings/sparse/src
- fuzz/src/array
- vortex-array/src
- aggregate_fn/fns
- is_sorted
- uncompressed_size_in_bytes
- arrays
- constant/vtable
- builders
- compute/conformance
- dtype
- arbitrary
- serde
- scalar_fn/fns/binary/compare
- scalar
- convert
- typed_view
- vortex-arrow/src
- executor
- vortex-datafusion/src/convert
- vortex-duckdb/src/convert
- vortex-ffi/src
- vortex-flatbuffers
- flatbuffers/vortex-dtype
- src/generated
- vortex-proto
- proto
- src/generated
- vortex-python/src
- dtype
- scalar
- vortex-row/src
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
153 | 153 | | |
154 | 154 | | |
155 | 155 | | |
| 156 | + | |
156 | 157 | | |
157 | 158 | | |
158 | 159 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
186 | 186 | | |
187 | 187 | | |
188 | 188 | | |
189 | | - | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
190 | 194 | | |
191 | 195 | | |
192 | 196 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
121 | 121 | | |
122 | 122 | | |
123 | 123 | | |
124 | | - | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
125 | 129 | | |
126 | 130 | | |
127 | 131 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
518 | 518 | | |
519 | 519 | | |
520 | 520 | | |
| 521 | + | |
521 | 522 | | |
522 | 523 | | |
523 | 524 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
149 | 149 | | |
150 | 150 | | |
151 | 151 | | |
152 | | - | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
153 | 157 | | |
154 | 158 | | |
155 | 159 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
125 | 125 | | |
126 | 126 | | |
127 | 127 | | |
128 | | - | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
129 | 133 | | |
130 | 134 | | |
131 | 135 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
102 | 102 | | |
103 | 103 | | |
104 | 104 | | |
105 | | - | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
106 | 110 | | |
107 | 111 | | |
108 | 112 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
148 | 148 | | |
149 | 149 | | |
150 | 150 | | |
151 | | - | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
152 | 156 | | |
153 | 157 | | |
154 | 158 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
246 | 246 | | |
247 | 247 | | |
248 | 248 | | |
| 249 | + | |
249 | 250 | | |
250 | 251 | | |
251 | 252 | | |
| |||
263 | 264 | | |
264 | 265 | | |
265 | 266 | | |
| 267 | + | |
266 | 268 | | |
267 | 269 | | |
268 | 270 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
236 | 236 | | |
237 | 237 | | |
238 | 238 | | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
239 | 242 | | |
240 | 243 | | |
241 | 244 | | |
| |||
294 | 297 | | |
295 | 298 | | |
296 | 299 | | |
| 300 | + | |
297 | 301 | | |
298 | 302 | | |
299 | 303 | | |
| |||
0 commit comments