Repository navigation
Commit 05437fd
committed
[fix](be) Narrow Lance Decimal256 columns that fit Decimal128 before decoding
### What problem does this PR solve?
Issue Number: None
Related PR: #69581
Problem Summary:
Reproduction: write a Lance dataset with a `pa.decimal256(18, 2)` column and
scan it with the Lance reader added in #69581. Lance keeps the 256-bit width
on disk, so the scanner returns an Arrow `Decimal256Array`, and the BE crashes
on the first non-empty batch. The same happens for any `decimal256(p, s)` with
p <= 38, also as a list, large_list, fixed_size_list, struct or map leaf. The
existing fixture only has decimal256 columns with precision 39 and 76, which
is why the tests did not catch it.
Root cause: `arrow_field_to_doris_type()` and the FE pick the Doris type from
the precision alone (DECIMAL32 up to 9, DECIMAL64 up to 18, DECIMAL128I up to
38), while `LanceArrowArrayNormalizer` passes the `Decimal256Array` through
unchanged. `DataTypeDecimalSerDe<TYPE_DECIMAL32/64/128I>::read_column_from_arrow`
casts the array with `dynamic_cast<const arrow::DecimalArray*>`, gets null for
a `Decimal256Array`, and dereferences it in `Value()`.
Fix: at schema binding, `LanceArrowArrayNormalizer::create()` records
`decimal128(p, s)` for a Decimal256 field with p <= 38 and marks it as needing
special handling, which parents inherit through the existing child loop.
`normalize_for_doris()` then narrows the array with
`arrow::compute::Cast(..., CastOptions::Safe())` on the query's Arrow memory
pool, the cast master's ADBC normalizer already uses
(`format/arrow/arrow_array_normalizer.cpp`). Nested parents rebuild their type
around the narrowed child through `set_lance_nested_type()`, as for BFloat16
leaves, before compaction. A value outside the declared precision, which is
invalid Arrow data (for example unscaled 10^18 in decimal(18, 2)), fails with
`narrow Lance Decimal256 field '<name>' to decimal128(18, 2) failed: Decimal
value does not fit in precision 18` instead of being truncated. Precision
39..76 keeps Decimal256 and DECIMAL256 as before.
Before: the BE dereferences a null pointer and crashes. After: the values are
read unchanged, and an out-of-precision value fails with an error that names
the field.
branch-4.1 and branch-4.2 have the same normalizer, so the same change applies
there.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- New: `LanceTableReaderTypeTest.NarrowsDecimal256WithDecimal128PrecisionDuringNormalization`,
`LanceTableReaderTypeTest.KeepsWideDecimal256UnchangedDuringNormalization`,
`LanceTableReaderTypeTest.ConvertsDecimal256WithDecimal128PrecisionToDorisDecimals`,
`LanceTableReaderTypeTest.NarrowsDecimal256LeavesInSlicedNestedArrays`,
`LanceTableReaderTypeTest.RejectsDecimal256ValuesOutsideDeclaredPrecision`,
`LanceTableReaderTypeTest.ReadsDecimal256WithDecimal128PrecisionFromLanceDataset`
(writes a decimal256(18, 2) dataset, checks that Lance stores Decimal256 and
that the reader returns DECIMAL64 values).
- All pass on macOS arm64 (ASAN) with the Lance suites. With the narrowing
disabled, the normalization cases fail and the sliced nested case then
crashes in `DataTypeDecimalSerDe::read_column_from_arrow`, as reported.
- Behavior changed: No (fixes a crash in the reader this PR adds)
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>1 parent 6714c7f commit 05437fd
3 files changed
Lines changed: 355 additions & 2 deletions
File tree
- be
- src/format_v2/lance
- test/format_v2/table
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
| 23 | + | |
23 | 24 | | |
24 | 25 | | |
25 | 26 | | |
| |||
318 | 319 | | |
319 | 320 | | |
320 | 321 | | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
321 | 346 | | |
322 | 347 | | |
323 | 348 | | |
| |||
488 | 513 | | |
489 | 514 | | |
490 | 515 | | |
491 | | - | |
492 | | - | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
| 524 | + | |
| 525 | + | |
| 526 | + | |
| 527 | + | |
| 528 | + | |
| 529 | + | |
493 | 530 | | |
494 | 531 | | |
495 | 532 | | |
| |||
532 | 569 | | |
533 | 570 | | |
534 | 571 | | |
| 572 | + | |
| 573 | + | |
| 574 | + | |
| 575 | + | |
| 576 | + | |
| 577 | + | |
| 578 | + | |
535 | 579 | | |
536 | 580 | | |
537 | 581 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
87 | 87 | | |
88 | 88 | | |
89 | 89 | | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
90 | 93 | | |
91 | 94 | | |
92 | 95 | | |
| |||
0 commit comments