Skip to content

Commit 1bb3537

Browse files
[fix](be) Prepare Parquet null maps before logical conversion (apache#69448)
### What problem does this PR solve? Issue Number: N/A Related PR: apache#63001 Problem Summary: A Parquet schema conversion can write past an empty destination null map and crash the BE. For example, a TVF infers `BIGINT` from one file and then reads `STRING` value `"xx"` from another file. `CastStringConverter` marks the failed cast as NULL, but `PhysicalToLogicalConverter` currently copies the source null map only after that conversion returns. Prepare the appended destination null-map slice before logical conversion. This preserves source NULLs and the existing destination prefix, lets converters mark additional failed casts, and retains COW isolation. The same boundary also covers other logical conversions that produce NULL, such as numeric overflow. Non-nullable conversion errors remain unchanged. Coverage added: - Converter unit tests for BYTE_ARRAY and FIXED_LEN_BYTE_ARRAY, nullable and non-nullable sources, invalid and valid strings, source NULLs, empty destinations, successive append batches, COW aliases, non-nullable errors, and numeric overflow. - Extend the existing `test_s3_tvf` suite with a mixed-schema Parquet regression: generate a BIGINT schema seed plus 4,096 STRING rows containing invalid values, numeric strings and source NULLs, then materialize all rows with both File Scanner V1 and V2 using `batch_size=1024` and non-strict casts. Compare their complete ordered results, including NULL positions. This differential check reuses the same S3 fixtures and needs no new golden output; fixed expected conversion values and null maps remain covered by the converter unit tests. Remove the separate crash-only Groovy suite. ### Release note Fix a BE crash when Parquet schema conversion produces NULL for an invalid value. ### Check List (For Author) - Test: - [x] Unit tests added; not executed locally. - [x] V1/V2 end-to-end differential regression added to the existing S3 TVF suite; not executed locally. - Passed: build hygiene, clang-format 16 formatting/check, `git diff --check`, Groovy syntax compilation. - `bash run-be-ut.sh --run --filter='ParquetColumnConvertTest.*' -j 4` was stopped while downloading the apache-orc submodule. The local third-party installation also lacks `libhs.a`; no test executable was produced. - `./run-regression-test.sh --run -d external_table_p0/tvf -s test_s3_tvf` failed before executing any suite: Maven 3.10 rejects existing duplicate Gson declarations in the regression framework POM. - `run-clang-tidy.sh` could not run because no compilation database was generated. Runtime and sanitizer validation remain for CI. - Behavior changed: - [x] Yes. Invalid nullable conversions produce NULL without accessing an unallocated null map. - Does this need documentation? - [x] No. <details> <summary>Self-review checkpoints</summary> - Goal: prepares the null map before any logical-converter failure write; the new unit cases reproduce that boundary. Runtime validation is pending. - Scope: one conversion boundary changed, with focused unit and crash regression tests. - Concurrency and locks: no new shared state or locking; columns belong to the reader, and destination aliases are detached using existing COW helpers. - Lifecycle and static initialization: no new lifetime mechanism or static state; source null maps remain independently owned. - Configuration: no new configuration; the regression restores all three changed session settings (`enable_file_scanner_v2`, `batch_size`, `enable_strict_cast`). - Compatibility: no symbol, protocol or storage-format changes. - Parallel paths: ORC already aligns destination nulls before logical conversion; File Scanner V2 uses a separate CAST implementation with allocated null maps and is now covered by the same end-to-end differential test. Consistent Parquet conversion paths are unchanged. - Conditions: no new production conditional branch; existing nullable handling is reused. - Coverage: source and conversion NULLs, valid values, two physical representations, successive batches, COW and non-nullable errors are covered; runtime execution is explicitly unverified. - Expected results: checked directly in converter unit tests; the end-to-end test checks successful materialization and compares complete V1/V2 results, including values and NULL positions, over multiple batches. - Observability: existing non-nullable conversion errors remain sufficient; no additional per-row logging is needed. - Transactions/persistence: not involved in the production change. - Data writes/atomicity: no production write-path changes; the regression writes only its unique temporary S3 prefix. - FE/BE variables: none added. - Performance: the existing null-map copy is moved before conversion, retaining linear cost and avoiding a second merge pass. - Other issues: no additional source-level issue found in the modified path; CI compilation, UT/sanitizer and external regression execution remain required. </details> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
1 parent 2524893 commit 1bb3537

3 files changed

Lines changed: 162 additions & 4 deletions

File tree

‎be/src/format/parquet/parquet_column_convert.h‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -385,14 +385,16 @@ class PhysicalToLogicalConverter {
385385
get_appended_null_map_start(src_physical_col, src_new_rows));
386386

387387
dst_logical_col = IColumn::mutate(std::move(dst_logical_col));
388-
const size_t dst_old_rows = get_mutable_inner_column_size(dst_logical_col);
389388
const size_t dst_old_null_map_size =
390389
get_null_map_size_or_inner_column_size(dst_logical_col);
390+
const size_t rows_to_convert = get_mutable_inner_column_size(src_logical_column);
391+
// Logical converters append all source rows and may mark failed casts as NULL (e.g.
392+
// "xx" -> BIGINT). Copy source nulls first so those writes have allocated entries and
393+
// are not overwritten after conversion. Keep any existing destination prefix intact.
394+
align_null_map(src_logical_column, dst_logical_col, dst_old_null_map_size, rows_to_convert,
395+
get_appended_null_map_start(src_logical_column, rows_to_convert));
391396
auto converted_column = dst_logical_col->assert_mutable();
392397
RETURN_IF_ERROR(_logical_converter->convert(src_logical_column, converted_column));
393-
const size_t dst_new_rows = get_mutable_inner_column_size(dst_logical_col) - dst_old_rows;
394-
align_null_map(src_logical_column, dst_logical_col, dst_old_null_map_size, dst_new_rows,
395-
get_appended_null_map_start(src_logical_column, dst_new_rows));
396398
return Status::OK();
397399
}
398400

‎be/test/format/parquet/parquet_column_convert_test.cpp‎

Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -424,6 +424,120 @@ TEST(ParquetColumnConvertTest,
424424
EXPECT_EQ(original_dst->get_data_at(0).to_string(), "zz");
425425
}
426426

427+
// Exercise both a direct BYTE_ARRAY source and a physical-to-logical temporary for
428+
// FIXED_LEN_BYTE_ARRAY. Both must prepare destination nulls before string casts.
429+
TEST(ParquetColumnConvertTest, StringCastFailuresPreserveNullsAcrossBatches) {
430+
for (const auto physical_type :
431+
{tparquet::Type::BYTE_ARRAY, tparquet::Type::FIXED_LEN_BYTE_ARRAY}) {
432+
for (const bool nullable_source : {false, true}) {
433+
SCOPED_TRACE(testing::Message() << "physical_type=" << physical_type
434+
<< ", nullable_source=" << nullable_source);
435+
FieldSchema field;
436+
field.parquet_schema.__set_type(physical_type);
437+
field.parquet_schema.__set_type_length(2);
438+
field.data_type =
439+
DataTypeFactory::instance().create_data_type(TYPE_STRING, nullable_source);
440+
const auto dst_type = DataTypeFactory::instance().create_data_type(TYPE_BIGINT, true);
441+
auto converter = PhysicalToLogicalConverter::get_converter(&field, field.data_type,
442+
dst_type, nullptr);
443+
ASSERT_TRUE(converter->support()) << converter->get_error_msg();
444+
ColumnPtr dst = dst_type->create_column();
445+
const std::vector<std::string> values {"xx", "05", "07", "09"};
446+
const std::vector<uint8_t> expected_nulls {1, 0, 0, 0};
447+
for (size_t batch = 0; batch < 2; ++batch) {
448+
// Retain an alias to ensure preparing the null map respects COW and does not
449+
// modify an existing batch through another owner.
450+
ColumnPtr dst_alias = dst;
451+
MutableColumnPtr nested;
452+
if (physical_type == tparquet::Type::BYTE_ARRAY) {
453+
nested = ColumnString::create();
454+
} else {
455+
nested = ColumnFixedLengthObject::create(2);
456+
}
457+
auto nulls = ColumnUInt8::create();
458+
for (size_t i = 0; i < values.size(); ++i) {
459+
nested->insert_data(values[i].data(), values[i].size());
460+
nulls->insert_value(i == 2);
461+
}
462+
ColumnPtr src;
463+
if (nullable_source) {
464+
src = ColumnNullable::create(std::move(nested), std::move(nulls));
465+
} else {
466+
src = std::move(nested);
467+
}
468+
const ColumnPtr src_alias = src;
469+
ASSERT_TRUE(converter->convert(src, field.data_type, dst_type, dst, false).ok());
470+
const auto& nullable = assert_cast<const ColumnNullable&>(*dst);
471+
const auto& numbers = assert_cast<const ColumnInt64&>(nullable.get_nested_column());
472+
ASSERT_EQ(nullable.get_null_map_data().size(), (batch + 1) * values.size());
473+
ASSERT_EQ(numbers.size(), nullable.get_null_map_data().size());
474+
for (size_t i = 0; i < numbers.size(); ++i) {
475+
const size_t value_index = i % values.size();
476+
EXPECT_EQ(nullable.get_null_map_data()[i],
477+
expected_nulls[value_index] || (nullable_source && value_index == 2));
478+
if (!nullable.get_null_map_data()[i]) {
479+
EXPECT_EQ(numbers.get_data()[i], std::stoi(values[value_index]));
480+
}
481+
}
482+
EXPECT_EQ(dst_alias->size(), batch * values.size());
483+
const auto& original = assert_cast<const ColumnNullable&>(*dst_alias);
484+
EXPECT_EQ(original.get_nested_column().size(), batch * values.size());
485+
if (nullable_source) {
486+
const auto& source = assert_cast<const ColumnNullable&>(*src_alias);
487+
EXPECT_EQ(source.get_null_map_data()[0], 0);
488+
EXPECT_EQ(source.get_null_map_data()[2], 1);
489+
}
490+
}
491+
}
492+
}
493+
}
494+
495+
TEST(ParquetColumnConvertTest, InvalidStringToNonNullableBigIntReturnsError) {
496+
FieldSchema field;
497+
field.parquet_schema.__set_type(tparquet::Type::BYTE_ARRAY);
498+
field.data_type = DataTypeFactory::instance().create_data_type(TYPE_STRING, false);
499+
const auto dst_type = DataTypeFactory::instance().create_data_type(TYPE_BIGINT, false);
500+
auto converter =
501+
PhysicalToLogicalConverter::get_converter(&field, field.data_type, dst_type, nullptr);
502+
ASSERT_TRUE(converter->support()) << converter->get_error_msg();
503+
auto strings = ColumnString::create();
504+
strings->insert_data("xx", 2);
505+
ColumnPtr src = std::move(strings);
506+
ColumnPtr dst = dst_type->create_column();
507+
auto status = converter->convert(src, field.data_type, dst_type, dst, false);
508+
EXPECT_FALSE(status.ok());
509+
EXPECT_NE(status.to_string().find("Failed to cast"), std::string::npos);
510+
}
511+
512+
TEST(ParquetColumnConvertTest, NumericOverflowMarksNullableDestination) {
513+
FieldSchema field;
514+
field.parquet_schema.__set_type(tparquet::Type::INT64);
515+
field.data_type = DataTypeFactory::instance().create_data_type(TYPE_BIGINT, true);
516+
const auto dst_type = DataTypeFactory::instance().create_data_type(TYPE_FLOAT, true);
517+
auto converter =
518+
PhysicalToLogicalConverter::get_converter(&field, field.data_type, dst_type, nullptr);
519+
ASSERT_TRUE(converter->support()) << converter->get_error_msg();
520+
auto values = ColumnInt64::create();
521+
values->insert_value(1LL << 40);
522+
values->insert_value(7);
523+
values->insert_value(9);
524+
auto nulls = ColumnUInt8::create();
525+
nulls->insert_value(0);
526+
nulls->insert_value(0);
527+
nulls->insert_value(1);
528+
ColumnPtr src = ColumnNullable::create(std::move(values), std::move(nulls));
529+
ColumnPtr dst = dst_type->create_column();
530+
ASSERT_TRUE(converter->convert(src, field.data_type, dst_type, dst, false).ok());
531+
const auto& nullable = assert_cast<const ColumnNullable&>(*dst);
532+
ASSERT_EQ(nullable.get_null_map_data().size(), 3);
533+
ASSERT_EQ(nullable.get_nested_column().size(), 3);
534+
EXPECT_EQ(nullable.get_null_map_data()[0], 1);
535+
EXPECT_EQ(nullable.get_null_map_data()[1], 0);
536+
EXPECT_EQ(nullable.get_null_map_data()[2], 1);
537+
EXPECT_FLOAT_EQ(assert_cast<const ColumnFloat32&>(nullable.get_nested_column()).get_data()[1],
538+
7.0F);
539+
}
540+
427541
TEST(ParquetColumnConvertTest, RejectInvalidUuidPhysicalWidth) {
428542
FieldSchema field;
429543
field.parquet_schema.__set_type(tparquet::Type::FIXED_LEN_BYTE_ARRAY);

‎regression-test/suites/external_table_p0/tvf/test_s3_tvf.groovy‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -169,4 +169,46 @@ suite("test_s3_tvf", "p0,external") {
169169
} finally {
170170
}
171171

172+
// Exercise schema conversion with both readers, including failed casts, source NULLs and
173+
// multiple batches. The BIGINT seed must be listed before the STRING file for TVF inference.
174+
String castPrefix = "s3://${getS3BucketName()}/regression/parquet_invalid_string_cast/${UUID.randomUUID()}"
175+
String castStorageProperties = """
176+
"s3.endpoint" = "${s3_endpoint}",
177+
"s3.region" = "${region}",
178+
"s3.access_key" = "${ak}",
179+
"s3.secret_key" = "${sk}"
180+
"""
181+
sql """SELECT CAST(1 AS BIGINT) AS id, CAST('seed' AS STRING) AS name
182+
INTO OUTFILE "${castPrefix}/00_seed/part_" FORMAT AS parquet
183+
PROPERTIES (${castStorageProperties})"""
184+
sql """SELECT CAST(CASE number % 4
185+
WHEN 0 THEN 'xx'
186+
WHEN 1 THEN '5'
187+
WHEN 2 THEN NULL
188+
ELSE '9' END AS STRING) AS id,
189+
CAST(number AS STRING) AS name
190+
FROM numbers("number" = "4096")
191+
INTO OUTFILE "${castPrefix}/01_strings/part_" FORMAT AS parquet
192+
PROPERTIES (${castStorageProperties})"""
193+
194+
def previousCastSettings = sql "SELECT @@enable_file_scanner_v2, @@batch_size, @@enable_strict_cast"
195+
try {
196+
sql "SET batch_size=1024"
197+
sql "SET enable_strict_cast=false"
198+
// Materialize every value; COUNT(*) could use footer statistics and skip conversion.
199+
// Compare the actual reader outputs rather than hardcoding expected rows in Groovy.
200+
def castResults = [false, true].collect { scannerV2 ->
201+
sql "SET enable_file_scanner_v2=${scannerV2}"
202+
sql """SELECT id, name FROM S3(
203+
"uri" = "${castPrefix}/{00_seed,01_strings}/*.parquet",
204+
"format" = "parquet",
205+
${castStorageProperties}) ORDER BY name"""
206+
}
207+
assertEquals(castResults[0], castResults[1],
208+
"V1 and V2 must preserve the same values and NULLs across mixed-schema batches")
209+
} finally {
210+
sql "SET enable_file_scanner_v2=${previousCastSettings[0][0]}"
211+
sql "SET batch_size=${previousCastSettings[0][1]}"
212+
sql "SET enable_strict_cast=${previousCastSettings[0][2]}"
213+
}
172214
}

0 commit comments

Comments
 (0)