Repository navigation
Commit 1bb3537
authored
[fix](be) Prepare Parquet null maps before logical conversion (apache#69448)
### What problem does this PR solve?
Issue Number: N/A
Related PR: apache#63001
Problem Summary:
A Parquet schema conversion can write past an empty destination null map
and crash the BE. For example, a TVF infers `BIGINT` from one file and
then reads `STRING` value `"xx"` from another file.
`CastStringConverter` marks the failed cast as NULL, but
`PhysicalToLogicalConverter` currently copies the source null map only
after that conversion returns.
Prepare the appended destination null-map slice before logical
conversion. This preserves source NULLs and the existing destination
prefix, lets converters mark additional failed casts, and retains COW
isolation. The same boundary also covers other logical conversions that
produce NULL, such as numeric overflow. Non-nullable conversion errors
remain unchanged.
Coverage added:
- Converter unit tests for BYTE_ARRAY and FIXED_LEN_BYTE_ARRAY, nullable
and non-nullable sources, invalid and valid strings, source NULLs, empty
destinations, successive append batches, COW aliases, non-nullable
errors, and numeric overflow.
- Extend the existing `test_s3_tvf` suite with a mixed-schema Parquet
regression: generate a BIGINT schema seed plus 4,096 STRING rows
containing invalid values, numeric strings and source NULLs, then
materialize all rows with both File Scanner V1 and V2 using
`batch_size=1024` and non-strict casts. Compare their complete ordered
results, including NULL positions. This differential check reuses the
same S3 fixtures and needs no new golden output; fixed expected
conversion values and null maps remain covered by the converter unit
tests. Remove the separate crash-only Groovy suite.
### Release note
Fix a BE crash when Parquet schema conversion produces NULL for an
invalid value.
### Check List (For Author)
- Test:
- [x] Unit tests added; not executed locally.
- [x] V1/V2 end-to-end differential regression added to the existing S3
TVF suite; not executed locally.
- Passed: build hygiene, clang-format 16 formatting/check, `git diff
--check`, Groovy syntax compilation.
- `bash run-be-ut.sh --run --filter='ParquetColumnConvertTest.*' -j 4`
was stopped while downloading the apache-orc submodule. The local
third-party installation also lacks `libhs.a`; no test executable was
produced.
- `./run-regression-test.sh --run -d external_table_p0/tvf -s
test_s3_tvf` failed before executing any suite: Maven 3.10 rejects
existing duplicate Gson declarations in the regression framework POM.
- `run-clang-tidy.sh` could not run because no compilation database was
generated. Runtime and sanitizer validation remain for CI.
- Behavior changed:
- [x] Yes. Invalid nullable conversions produce NULL without accessing
an unallocated null map.
- Does this need documentation?
- [x] No.
<details>
<summary>Self-review checkpoints</summary>
- Goal: prepares the null map before any logical-converter failure
write; the new unit cases reproduce that boundary. Runtime validation is
pending.
- Scope: one conversion boundary changed, with focused unit and crash
regression tests.
- Concurrency and locks: no new shared state or locking; columns belong
to the reader, and destination aliases are detached using existing COW
helpers.
- Lifecycle and static initialization: no new lifetime mechanism or
static state; source null maps remain independently owned.
- Configuration: no new configuration; the regression restores all three
changed session settings (`enable_file_scanner_v2`, `batch_size`,
`enable_strict_cast`).
- Compatibility: no symbol, protocol or storage-format changes.
- Parallel paths: ORC already aligns destination nulls before logical
conversion; File Scanner V2 uses a separate CAST implementation with
allocated null maps and is now covered by the same end-to-end
differential test. Consistent Parquet conversion paths are unchanged.
- Conditions: no new production conditional branch; existing nullable
handling is reused.
- Coverage: source and conversion NULLs, valid values, two physical
representations, successive batches, COW and non-nullable errors are
covered; runtime execution is explicitly unverified.
- Expected results: checked directly in converter unit tests; the
end-to-end test checks successful materialization and compares complete
V1/V2 results, including values and NULL positions, over multiple
batches.
- Observability: existing non-nullable conversion errors remain
sufficient; no additional per-row logging is needed.
- Transactions/persistence: not involved in the production change.
- Data writes/atomicity: no production write-path changes; the
regression writes only its unique temporary S3 prefix.
- FE/BE variables: none added.
- Performance: the existing null-map copy is moved before conversion,
retaining linear cost and avoiding a second merge pass.
- Other issues: no additional source-level issue found in the modified
path; CI compilation, UT/sanitizer and external regression execution
remain required.
</details>
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label1 parent 2524893 commit 1bb3537
3 files changed
Lines changed: 162 additions & 4 deletions
File tree
- be
- src/format/parquet
- test/format/parquet
- regression-test/suites/external_table_p0/tvf
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
385 | 385 | | |
386 | 386 | | |
387 | 387 | | |
388 | | - | |
389 | 388 | | |
390 | 389 | | |
| 390 | + | |
| 391 | + | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
| 395 | + | |
391 | 396 | | |
392 | 397 | | |
393 | | - | |
394 | | - | |
395 | | - | |
396 | 398 | | |
397 | 399 | | |
398 | 400 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
424 | 424 | | |
425 | 425 | | |
426 | 426 | | |
| 427 | + | |
| 428 | + | |
| 429 | + | |
| 430 | + | |
| 431 | + | |
| 432 | + | |
| 433 | + | |
| 434 | + | |
| 435 | + | |
| 436 | + | |
| 437 | + | |
| 438 | + | |
| 439 | + | |
| 440 | + | |
| 441 | + | |
| 442 | + | |
| 443 | + | |
| 444 | + | |
| 445 | + | |
| 446 | + | |
| 447 | + | |
| 448 | + | |
| 449 | + | |
| 450 | + | |
| 451 | + | |
| 452 | + | |
| 453 | + | |
| 454 | + | |
| 455 | + | |
| 456 | + | |
| 457 | + | |
| 458 | + | |
| 459 | + | |
| 460 | + | |
| 461 | + | |
| 462 | + | |
| 463 | + | |
| 464 | + | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
| 473 | + | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
| 477 | + | |
| 478 | + | |
| 479 | + | |
| 480 | + | |
| 481 | + | |
| 482 | + | |
| 483 | + | |
| 484 | + | |
| 485 | + | |
| 486 | + | |
| 487 | + | |
| 488 | + | |
| 489 | + | |
| 490 | + | |
| 491 | + | |
| 492 | + | |
| 493 | + | |
| 494 | + | |
| 495 | + | |
| 496 | + | |
| 497 | + | |
| 498 | + | |
| 499 | + | |
| 500 | + | |
| 501 | + | |
| 502 | + | |
| 503 | + | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
| 507 | + | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
| 524 | + | |
| 525 | + | |
| 526 | + | |
| 527 | + | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
| 531 | + | |
| 532 | + | |
| 533 | + | |
| 534 | + | |
| 535 | + | |
| 536 | + | |
| 537 | + | |
| 538 | + | |
| 539 | + | |
| 540 | + | |
427 | 541 | | |
428 | 542 | | |
429 | 543 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
169 | 169 | | |
170 | 170 | | |
171 | 171 | | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
172 | 214 | | |
0 commit comments