Skip to content

Commit 05437fd

Browse files
committed
[fix](be) Narrow Lance Decimal256 columns that fit Decimal128 before decoding
### What problem does this PR solve? Issue Number: None Related PR: #69581 Problem Summary: Reproduction: write a Lance dataset with a `pa.decimal256(18, 2)` column and scan it with the Lance reader added in #69581. Lance keeps the 256-bit width on disk, so the scanner returns an Arrow `Decimal256Array`, and the BE crashes on the first non-empty batch. The same happens for any `decimal256(p, s)` with p <= 38, also as a list, large_list, fixed_size_list, struct or map leaf. The existing fixture only has decimal256 columns with precision 39 and 76, which is why the tests did not catch it. Root cause: `arrow_field_to_doris_type()` and the FE pick the Doris type from the precision alone (DECIMAL32 up to 9, DECIMAL64 up to 18, DECIMAL128I up to 38), while `LanceArrowArrayNormalizer` passes the `Decimal256Array` through unchanged. `DataTypeDecimalSerDe<TYPE_DECIMAL32/64/128I>::read_column_from_arrow` casts the array with `dynamic_cast<const arrow::DecimalArray*>`, gets null for a `Decimal256Array`, and dereferences it in `Value()`. Fix: at schema binding, `LanceArrowArrayNormalizer::create()` records `decimal128(p, s)` for a Decimal256 field with p <= 38 and marks it as needing special handling, which parents inherit through the existing child loop. `normalize_for_doris()` then narrows the array with `arrow::compute::Cast(..., CastOptions::Safe())` on the query's Arrow memory pool, the cast master's ADBC normalizer already uses (`format/arrow/arrow_array_normalizer.cpp`). Nested parents rebuild their type around the narrowed child through `set_lance_nested_type()`, as for BFloat16 leaves, before compaction. A value outside the declared precision, which is invalid Arrow data (for example unscaled 10^18 in decimal(18, 2)), fails with `narrow Lance Decimal256 field '<name>' to decimal128(18, 2) failed: Decimal value does not fit in precision 18` instead of being truncated. Precision 39..76 keeps Decimal256 and DECIMAL256 as before. Before: the BE dereferences a null pointer and crashes. After: the values are read unchanged, and an out-of-precision value fails with an error that names the field. branch-4.1 and branch-4.2 have the same normalizer, so the same change applies there. ### Release note None ### Check List (For Author) - Test: Unit Test - New: `LanceTableReaderTypeTest.NarrowsDecimal256WithDecimal128PrecisionDuringNormalization`, `LanceTableReaderTypeTest.KeepsWideDecimal256UnchangedDuringNormalization`, `LanceTableReaderTypeTest.ConvertsDecimal256WithDecimal128PrecisionToDorisDecimals`, `LanceTableReaderTypeTest.NarrowsDecimal256LeavesInSlicedNestedArrays`, `LanceTableReaderTypeTest.RejectsDecimal256ValuesOutsideDeclaredPrecision`, `LanceTableReaderTypeTest.ReadsDecimal256WithDecimal128PrecisionFromLanceDataset` (writes a decimal256(18, 2) dataset, checks that Lance stores Decimal256 and that the reader returns DECIMAL64 values). - All pass on macOS arm64 (ASAN) with the Lance suites. With the narrowing disabled, the normalization cases fail and the sliced nested case then crashes in `DataTypeDecimalSerDe::read_column_from_arrow`, as reported. - Behavior changed: No (fixes a crash in the reader this PR adds) - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent 6714c7f commit 05437fd

3 files changed

Lines changed: 355 additions & 2 deletions

File tree

‎be/src/format_v2/lance/lance_reader_helper.cpp‎

Lines changed: 46 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@
2020
#include <arrow/array.h>
2121
#include <arrow/builder.h>
2222
#include <arrow/c/bridge.h>
23+
#include <arrow/compute/cast.h>
2324
#include <arrow/extension_type.h>
2425
#include <arrow/type.h>
2526
#include <arrow/util/key_value_metadata.h>
@@ -318,6 +319,30 @@ Status convert_bfloat16_array(const std::shared_ptr<arrow::Array>& array,
318319
return Status::OK();
319320
}
320321

322+
// Narrow a Decimal256 array whose declared precision fits Decimal128. The safe cast keeps every
323+
// in-precision value and its scale, and rejects a value that exceeds the declared precision
324+
// instead of truncating it: unscaled 10^18 stored in a decimal256(18, 2) field fails with
325+
// "Decimal value does not fit in precision 18".
326+
Status narrow_decimal256_array(std::string_view field_name,
327+
const std::shared_ptr<arrow::Array>& array,
328+
const std::shared_ptr<arrow::DataType>& decimal128_type,
329+
arrow::MemoryPool* memory_pool,
330+
std::shared_ptr<arrow::Array>* normalized) {
331+
DORIS_CHECK(array != nullptr);
332+
DORIS_CHECK(memory_pool != nullptr);
333+
DORIS_CHECK(normalized != nullptr);
334+
arrow::compute::ExecContext exec_context(memory_pool);
335+
auto narrowed = arrow::compute::Cast(*array, decimal128_type,
336+
arrow::compute::CastOptions::Safe(), &exec_context);
337+
if (!narrowed.ok()) {
338+
return Status::InvalidArgument("narrow Lance Decimal256 field '{}' to {} failed: {}",
339+
field_name, decimal128_type->ToString(),
340+
narrowed.status().message());
341+
}
342+
*normalized = std::move(narrowed).ValueUnsafe();
343+
return Status::OK();
344+
}
345+
321346
// Rebuild a nested Arrow type after one or more child arrays changed physical type.
322347
Status set_lance_nested_type(std::string_view field_name,
323348
const std::shared_ptr<arrow::DataType>& source_type,
@@ -488,8 +513,20 @@ Status LanceArrowArrayNormalizer::create(const std::shared_ptr<arrow::Field>& fi
488513
result._storage_type = std::move(storage_type);
489514
result._unwrap_registered_extension = field->type()->id() == arrow::Type::EXTENSION;
490515
result._convert_bfloat16 = extension_kind == LanceExtensionKind::BFLOAT16;
491-
result._requires_special_handling =
492-
result._unwrap_registered_extension || result._convert_bfloat16;
516+
// Lance keeps the 256-bit carrier of pa.decimal256(18, 2), while the FE and
517+
// arrow_field_to_doris_type() type the column DECIMAL64 by precision. The DECIMAL32/64/128I
518+
// SerDes read only Decimal128Array, so a Decimal256Array reaching them made their dynamic_cast
519+
// return null and the BE crash on the first value, also for nested leaves. Precision 39..76
520+
// stays Decimal256 and maps to DECIMAL256.
521+
if (result._storage_type->id() == arrow::Type::DECIMAL256) {
522+
const auto& decimal = static_cast<const arrow::Decimal256Type&>(*result._storage_type);
523+
if (decimal.precision() <= arrow::Decimal128Type::kMaxPrecision) {
524+
result._decimal128_type = arrow::decimal128(decimal.precision(), decimal.scale());
525+
}
526+
}
527+
result._requires_special_handling = result._unwrap_registered_extension ||
528+
result._convert_bfloat16 ||
529+
result._decimal128_type != nullptr;
493530
result._child_normalizers.reserve(result._storage_type->num_fields());
494531
for (const auto& child_field : result._storage_type->fields()) {
495532
LanceArrowArrayNormalizer child_normalizer;
@@ -532,6 +569,13 @@ Status LanceArrowArrayNormalizer::normalize_for_doris(
532569
if (_convert_bfloat16) {
533570
return convert_bfloat16_array(storage_array, memory_pool, normalized);
534571
}
572+
if (_decimal128_type != nullptr) {
573+
// The cast reads only the visible slice and returns an offset-zero array, so no
574+
// compaction follows. For a nested leaf, the parent's call swaps in this array in the
575+
// child loop below and rebuilds its own type through set_lance_nested_type().
576+
return narrow_decimal256_array(_field_name, storage_array, _decimal128_type, memory_pool,
577+
normalized);
578+
}
535579

536580
// Normalize extension children before compacting the parent. Arrow cannot create a builder
537581
// for list/struct types whose child is still an ExtensionType (Arrow 24 returns

‎be/src/format_v2/lance/lance_reader_helper.h‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -87,6 +87,9 @@ class LanceArrowArrayNormalizer {
8787
private:
8888
std::string _field_name;
8989
std::shared_ptr<arrow::DataType> _storage_type;
90+
// decimal128(p, s) for a Decimal256 field with p <= 38, which Doris types DECIMAL32/64/128I
91+
// by precision; null for every other field.
92+
std::shared_ptr<arrow::DataType> _decimal128_type;
9093
std::vector<LanceArrowArrayNormalizer> _child_normalizers;
9194
bool _unwrap_registered_extension = false;
9295
bool _convert_bfloat16 = false;

0 commit comments

Comments
 (0)