Repository navigation
Commit c91c8eb
fix(layout): check list offsets before the list reader uses them (#68)
## Summary
Closes #56. Part of #55.
The list reader trusted the offsets it decoded. When offsets decrease,
go negative or point past the elements, most reads return with no error,
and a sparse read of decreasing offsets panics in `vortex-mask`. Readers
now check offsets as they read them, so a malformed file fails the read
with an error.
## Changes
- `ListReader::fetch_raw_offsets` checks the offsets it returns. They
must be the expected count, castable to `u64` (negative values fail),
non-decreasing, and the last one can't be past the elements child's row
count. Every read path gets its offsets from here, so the range
arithmetic, `kept_elements` and `ListArray::new_unchecked` after it can
rely on that. Only the offsets a read uses get checked, so pruning
doesn't change and nothing scans the whole column on open. The
`row_range.end + 1` there is now checked too.
- `ListLayout::new` becomes `ListLayout::try_new`. It checks the
children the way deserialization does: elements match the list's element
dtype, offsets are non-nullable integers with at least one row, and
validity is non-nullable bool with one row per list. Before, it used
saturating subtraction and `vortex_expect`.
- The writer adds each chunk's element count with `checked_add`.
Not in this PR: which integer types the frozen format accepts for
offsets. That's a #54 decision. Readers still accept any integer type
that casts to `u64`.
## Testing
- `malformed_offsets_fail_the_read` covers decreasing, past-the-end and
negative offsets on full, bounded, sparse and length-only reads. Without
the check, 8 of the 12 cases return with no error, 2 fail later with
unrelated errors (an out-of-bounds slice and a filter length mismatch),
and 1 panics. Only one, a sparse read of negative offsets, already
failed cleanly.
- `try_new_rejects_mismatched_children` covers the wrong elements dtype,
float offsets, nullable offsets, no offsets and a validity row count
that doesn't match.
- `cargo test -p vortex-layout -p vortex-file` and `cargo clippy -p
vortex-layout -p vortex-file --all-targets`.
- Read a real list-layout file of nested event data through the data
source with the check in place. The bytes read and rows came out the
same as before.
---------
Signed-off-by: Rasheed Khoshnaw <[email protected]>
Signed-off-by: Christopher Wirt <[email protected]>
Co-authored-by: Christopher Wirt <[email protected]>1 parent 00130bd commit c91c8eb
4 files changed
Lines changed: 281 additions & 25 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
172 | 172 | | |
173 | 173 | | |
174 | 174 | | |
175 | | - | |
176 | | - | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
177 | 180 | | |
178 | 181 | | |
179 | 182 | | |
180 | 183 | | |
181 | | - | |
182 | | - | |
183 | | - | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
184 | 213 | | |
185 | 214 | | |
186 | | - | |
187 | | - | |
| 215 | + | |
| 216 | + | |
188 | 217 | | |
189 | 218 | | |
190 | 219 | | |
191 | 220 | | |
192 | 221 | | |
193 | 222 | | |
194 | 223 | | |
195 | | - | |
| 224 | + | |
196 | 225 | | |
197 | 226 | | |
198 | 227 | | |
| |||
272 | 301 | | |
273 | 302 | | |
274 | 303 | | |
| 304 | + | |
275 | 305 | | |
276 | 306 | | |
277 | 307 | | |
| |||
345 | 375 | | |
346 | 376 | | |
347 | 377 | | |
| 378 | + | |
| 379 | + | |
| 380 | + | |
| 381 | + | |
| 382 | + | |
| 383 | + | |
| 384 | + | |
| 385 | + | |
| 386 | + | |
| 387 | + | |
| 388 | + | |
| 389 | + | |
| 390 | + | |
| 391 | + | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
| 395 | + | |
| 396 | + | |
| 397 | + | |
| 398 | + | |
| 399 | + | |
| 400 | + | |
| 401 | + | |
| 402 | + | |
| 403 | + | |
| 404 | + | |
| 405 | + | |
| 406 | + | |
| 407 | + | |
| 408 | + | |
| 409 | + | |
| 410 | + | |
| 411 | + | |
| 412 | + | |
| 413 | + | |
| 414 | + | |
| 415 | + | |
| 416 | + | |
| 417 | + | |
| 418 | + | |
| 419 | + | |
| 420 | + | |
| 421 | + | |
| 422 | + | |
| 423 | + | |
| 424 | + | |
| 425 | + | |
| 426 | + | |
| 427 | + | |
| 428 | + | |
| 429 | + | |
| 430 | + | |
| 431 | + | |
| 432 | + | |
| 433 | + | |
| 434 | + | |
| 435 | + | |
| 436 | + | |
| 437 | + | |
| 438 | + | |
| 439 | + | |
| 440 | + | |
| 441 | + | |
| 442 | + | |
| 443 | + | |
348 | 444 | | |
349 | 445 | | |
350 | 446 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
26 | 26 | | |
27 | 27 | | |
28 | 28 | | |
| 29 | + | |
29 | 30 | | |
30 | 31 | | |
31 | 32 | | |
32 | 33 | | |
33 | 34 | | |
34 | 35 | | |
35 | 36 | | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
36 | 41 | | |
37 | 42 | | |
38 | 43 | | |
| |||
353 | 358 | | |
354 | 359 | | |
355 | 360 | | |
356 | | - | |
| 361 | + | |
| 362 | + | |
| 363 | + | |
357 | 364 | | |
358 | | - | |
| 365 | + | |
| 366 | + | |
| 367 | + | |
| 368 | + | |
359 | 369 | | |
360 | 370 | | |
361 | | - | |
| 371 | + | |
362 | 372 | | |
363 | 373 | | |
364 | 374 | | |
365 | | - | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
| 380 | + | |
| 381 | + | |
| 382 | + | |
366 | 383 | | |
367 | 384 | | |
368 | 385 | | |
| |||
496 | 513 | | |
497 | 514 | | |
498 | 515 | | |
499 | | - | |
500 | | - | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
501 | 519 | | |
502 | 520 | | |
503 | 521 | | |
| |||
742 | 760 | | |
743 | 761 | | |
744 | 762 | | |
| 763 | + | |
| 764 | + | |
| 765 | + | |
| 766 | + | |
| 767 | + | |
| 768 | + | |
| 769 | + | |
| 770 | + | |
| 771 | + | |
| 772 | + | |
| 773 | + | |
| 774 | + | |
| 775 | + | |
| 776 | + | |
| 777 | + | |
| 778 | + | |
| 779 | + | |
| 780 | + | |
| 781 | + | |
| 782 | + | |
| 783 | + | |
| 784 | + | |
| 785 | + | |
| 786 | + | |
| 787 | + | |
| 788 | + | |
| 789 | + | |
| 790 | + | |
| 791 | + | |
| 792 | + | |
| 793 | + | |
| 794 | + | |
| 795 | + | |
| 796 | + | |
| 797 | + | |
| 798 | + | |
| 799 | + | |
| 800 | + | |
| 801 | + | |
| 802 | + | |
| 803 | + | |
| 804 | + | |
| 805 | + | |
| 806 | + | |
| 807 | + | |
| 808 | + | |
| 809 | + | |
745 | 810 | | |
746 | 811 | | |
747 | 812 | | |
| |||
835 | 900 | | |
836 | 901 | | |
837 | 902 | | |
| 903 | + | |
| 904 | + | |
838 | 905 | | |
| 906 | + | |
839 | 907 | | |
840 | 908 | | |
841 | 909 | | |
| |||
1128 | 1196 | | |
1129 | 1197 | | |
1130 | 1198 | | |
| 1199 | + | |
| 1200 | + | |
| 1201 | + | |
| 1202 | + | |
| 1203 | + | |
| 1204 | + | |
| 1205 | + | |
| 1206 | + | |
| 1207 | + | |
| 1208 | + | |
| 1209 | + | |
| 1210 | + | |
| 1211 | + | |
| 1212 | + | |
| 1213 | + | |
| 1214 | + | |
| 1215 | + | |
| 1216 | + | |
| 1217 | + | |
| 1218 | + | |
| 1219 | + | |
| 1220 | + | |
| 1221 | + | |
| 1222 | + | |
| 1223 | + | |
| 1224 | + | |
| 1225 | + | |
| 1226 | + | |
| 1227 | + | |
| 1228 | + | |
| 1229 | + | |
| 1230 | + | |
| 1231 | + | |
| 1232 | + | |
| 1233 | + | |
| 1234 | + | |
| 1235 | + | |
| 1236 | + | |
| 1237 | + | |
| 1238 | + | |
| 1239 | + | |
| 1240 | + | |
| 1241 | + | |
| 1242 | + | |
| 1243 | + | |
| 1244 | + | |
| 1245 | + | |
| 1246 | + | |
| 1247 | + | |
| 1248 | + | |
| 1249 | + | |
| 1250 | + | |
| 1251 | + | |
| 1252 | + | |
| 1253 | + | |
| 1254 | + | |
| 1255 | + | |
| 1256 | + | |
| 1257 | + | |
| 1258 | + | |
| 1259 | + | |
| 1260 | + | |
| 1261 | + | |
| 1262 | + | |
| 1263 | + | |
| 1264 | + | |
| 1265 | + | |
| 1266 | + | |
| 1267 | + | |
| 1268 | + | |
| 1269 | + | |
| 1270 | + | |
| 1271 | + | |
| 1272 | + | |
| 1273 | + | |
| 1274 | + | |
| 1275 | + | |
| 1276 | + | |
| 1277 | + | |
| 1278 | + | |
| 1279 | + | |
| 1280 | + | |
| 1281 | + | |
| 1282 | + | |
1131 | 1283 | | |
1132 | 1284 | | |
1133 | 1285 | | |
| |||
2059 | 2211 | | |
2060 | 2212 | | |
2061 | 2213 | | |
2062 | | - | |
| 2214 | + | |
| 2215 | + | |
| 2216 | + | |
2063 | 2217 | | |
2064 | 2218 | | |
2065 | 2219 | | |
| |||
0 commit comments