Repository navigation
Commit 4f887ac
committed
[fix](lance) Source fixed-list contract facts from the reconstructed Arrow view
Verified against the pinned lance-core 9.1.0-beta.3 JNI bindings:
- Dataset.getLanceSchema() returns LanceFields whose children are always
empty for every fixed-size list (float32/float16, element nullable or
not): the element type is collapsed into the manifest logical-type
string (for example fixed_size_list:float:4). The previous contract
path required exactly one Lance child, so it rejected every real
vector column before the element facts were ever read.
- Element nullability has no slot in the manifest. It is dropped on
write and every read-side view synthesizes nullable=true (Java-side
LanceField.childrenForFixedSizeList() hardcodes true). Existing
datasets are equally affected; an SDK upgrade cannot backfill them.
- The element type is recoverable: LanceField.asArrowField()
synthesizes the element as the first child of the reconstructed Arrow
view.
The contract builder now takes vectorElementType and
vectorElementNullable from that synthesized child, copying the
synthesized nullability flag instead of pinning true so the contract
improves automatically if a future SDK preserves element nullability.
A missing Arrow view, an empty child list or a null child fails closed
with the bounded ERR_LANCE_INDEX_INVALID error, matching the other
malformed-provider-fact rejections.
Tests: the mock fixtures stub asArrowField() with a synthetic
always-nullable child exactly as the SDK produces it, and a new real
JNI fixture creates a schema-only dataset via Dataset.create (f32[4]
not-null top level, f16[3] nullable top level, int not null), reopens
it, and pins the reconstructed-schema contract: fixed_size_list with
dimensions 4/3, float32/float16 elements, vectorElementNullable=true,
and preserved top-level nullability.1 parent a53722f commit 4f887ac
5 files changed
Lines changed: 251 additions & 52 deletions
File tree
- fe/fe-core/src
- main/java/org/apache/doris/datasource/lance
- test/java/org/apache/doris
- datasource/lance
- nereids/trees/plans/commands
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
26 | 26 | | |
27 | 27 | | |
28 | 28 | | |
| 29 | + | |
29 | 30 | | |
30 | 31 | | |
31 | 32 | | |
| |||
47 | 48 | | |
48 | 49 | | |
49 | 50 | | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
50 | 63 | | |
51 | 64 | | |
52 | 65 | | |
| |||
104 | 117 | | |
105 | 118 | | |
106 | 119 | | |
107 | | - | |
108 | | - | |
109 | | - | |
110 | | - | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
111 | 131 | | |
112 | | - | |
| 132 | + | |
113 | 133 | | |
114 | 134 | | |
115 | 135 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
39 | 39 | | |
40 | 40 | | |
41 | 41 | | |
| 42 | + | |
| 43 | + | |
42 | 44 | | |
43 | 45 | | |
44 | 46 | | |
| |||
48 | 50 | | |
49 | 51 | | |
50 | 52 | | |
51 | | - | |
52 | 53 | | |
53 | 54 | | |
54 | 55 | | |
| |||
114 | 115 | | |
115 | 116 | | |
116 | 117 | | |
117 | | - | |
| 118 | + | |
118 | 119 | | |
119 | 120 | | |
120 | 121 | | |
| |||
178 | 179 | | |
179 | 180 | | |
180 | 181 | | |
181 | | - | |
182 | | - | |
183 | | - | |
184 | | - | |
185 | | - | |
186 | | - | |
187 | | - | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
188 | 189 | | |
189 | 190 | | |
190 | 191 | | |
191 | 192 | | |
192 | 193 | | |
193 | | - | |
| 194 | + | |
| 195 | + | |
194 | 196 | | |
195 | 197 | | |
196 | 198 | | |
| |||
342 | 344 | | |
343 | 345 | | |
344 | 346 | | |
345 | | - | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
346 | 350 | | |
347 | 351 | | |
348 | 352 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
0 commit comments