Repository navigation
Commit 9a982d3
[improvement](be) Import unshredded Parquet Variant values without rebuilding them
### What problem does this PR solve?
Issue Number: None
Related PR: #66714, #66910, #67016
Problem Summary: Port of branch-4.1 #66714 onto master's #67016 code.
When the native Parquet reader materializes an unshredded Variant column
(only metadata and value, no typed_value), master rebuilds every row
through VariantBatchBuilder (encode_variant_range), decoding and
re-encoding the whole value tree although the bytes already are Variant
Encoding V1. The direct path seek of #67016 does the same for each found
path through append_variant_rows, which re-encodes the rows into a
builder with a single metadata dictionary and splits a batch in halves
whenever its rows use different dictionaries.
Re-encoding also changes what is validated. The builder copies only the
metadata keys a value uses, so a corrupt key that the value does not use,
for example invalid UTF-8, disappears instead of being reported:
extracting a path from such a row returns a value. branch-4.x validates
every metadata dictionary when it imports the row, so the read fails. Two
#66910 tests cover this; #67016 left them out because master had no
importer.
This adds ColumnVariantV2::EncodedRowsAppender, as in branch-4.x. It
validates each distinct metadata dictionary once and every value, rejects
input that borrows bytes from the destination column, and then copies the
bytes, interning metadata through a hash index that it keeps across
batches. insert_encoded_rows and insert_encoded_batch share the new
_append_prevalidated_encoded_data. The Parquet reader uses the appender
for complete unshredded roots during materialization and for the direct
path seek, and reports VariantUnshreddedDirectImportTime, Rows and Bytes.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- BE UT: ColumnVariantV2Test (appender atomicity on a typed column,
rejection of rows borrowed from the destination, metadata interning
across chunks, empty appends) and VariantColumnReaderTest (direct
import of encoded bytes, a 4097-row import across the batch
boundary, profile counters on immediate and late failures, and the
two corrupt-metadata cases of #66910) pass, as do the other
Variant, JNI and Parquet schema suites (755 tests). With the
production change reverted, exactly the six new reader cases fail;
the two corrupt-metadata cases fail because the read throws nothing.
ColumnVariantV2Test.TypedComparisonBoundariesMatchEncodedWithoutMaterializing
fails on a local macOS build with and without this change, and
passes on CI.
- Behavior changed: Yes. Reading an unshredded Parquet Variant row whose
metadata has a corrupt key now fails instead of ignoring the key.
- Does this need documentation: No
Co-authored-by: daidai <[email protected]>
Co-authored-by: lihangyu <[email protected]>
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>1 parent cad4abb commit 9a982d3
7 files changed
Lines changed: 927 additions & 64 deletions
File tree
- be
- src
- core/column/variant_v2
- format_v2/parquet
- reader
- test
- core/column
- format_v2/parquet
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
119 | 119 | | |
120 | 120 | | |
121 | 121 | | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
122 | 150 | | |
123 | 151 | | |
124 | 152 | | |
| |||
204 | 232 | | |
205 | 233 | | |
206 | 234 | | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
207 | 239 | | |
208 | 240 | | |
209 | 241 | | |
| |||
282 | 314 | | |
283 | 315 | | |
284 | 316 | | |
| 317 | + | |
| 318 | + | |
285 | 319 | | |
286 | 320 | | |
287 | 321 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
107 | 107 | | |
108 | 108 | | |
109 | 109 | | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
110 | 116 | | |
111 | 117 | | |
112 | 118 | | |
| |||
355 | 361 | | |
356 | 362 | | |
357 | 363 | | |
| 364 | + | |
| 365 | + | |
| 366 | + | |
358 | 367 | | |
359 | 368 | | |
360 | 369 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
42 | 42 | | |
43 | 43 | | |
44 | 44 | | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
45 | 50 | | |
46 | 51 | | |
47 | 52 | | |
| |||
184 | 189 | | |
185 | 190 | | |
186 | 191 | | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
187 | 196 | | |
188 | 197 | | |
189 | 198 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
479 | 479 | | |
480 | 480 | | |
481 | 481 | | |
482 | | - | |
483 | | - | |
484 | | - | |
| 482 | + | |
| 483 | + | |
| 484 | + | |
| 485 | + | |
| 486 | + | |
| 487 | + | |
| 488 | + | |
| 489 | + | |
| 490 | + | |
| 491 | + | |
| 492 | + | |
| 493 | + | |
| 494 | + | |
| 495 | + | |
| 496 | + | |
| 497 | + | |
| 498 | + | |
| 499 | + | |
| 500 | + | |
| 501 | + | |
| 502 | + | |
| 503 | + | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
| 507 | + | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
| 524 | + | |
| 525 | + | |
| 526 | + | |
| 527 | + | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
| 531 | + | |
| 532 | + | |
| 533 | + | |
| 534 | + | |
| 535 | + | |
485 | 536 | | |
486 | 537 | | |
487 | 538 | | |
| |||
496 | 547 | | |
497 | 548 | | |
498 | 549 | | |
499 | | - | |
500 | | - | |
501 | | - | |
502 | | - | |
503 | | - | |
| 550 | + | |
| 551 | + | |
| 552 | + | |
| 553 | + | |
| 554 | + | |
| 555 | + | |
| 556 | + | |
| 557 | + | |
| 558 | + | |
| 559 | + | |
| 560 | + | |
| 561 | + | |
| 562 | + | |
| 563 | + | |
| 564 | + | |
| 565 | + | |
| 566 | + | |
| 567 | + | |
| 568 | + | |
| 569 | + | |
| 570 | + | |
| 571 | + | |
| 572 | + | |
| 573 | + | |
| 574 | + | |
| 575 | + | |
| 576 | + | |
| 577 | + | |
| 578 | + | |
| 579 | + | |
| 580 | + | |
| 581 | + | |
| 582 | + | |
| 583 | + | |
| 584 | + | |
| 585 | + | |
| 586 | + | |
| 587 | + | |
| 588 | + | |
| 589 | + | |
| 590 | + | |
| 591 | + | |
| 592 | + | |
| 593 | + | |
504 | 594 | | |
505 | 595 | | |
506 | 596 | | |
| |||
1041 | 1131 | | |
1042 | 1132 | | |
1043 | 1133 | | |
1044 | | - | |
1045 | | - | |
1046 | | - | |
1047 | | - | |
1048 | | - | |
1049 | | - | |
1050 | | - | |
1051 | | - | |
1052 | | - | |
1053 | | - | |
1054 | | - | |
1055 | | - | |
1056 | | - | |
1057 | | - | |
1058 | | - | |
1059 | | - | |
1060 | | - | |
1061 | | - | |
1062 | | - | |
1063 | | - | |
1064 | | - | |
1065 | 1134 | | |
1066 | 1135 | | |
1067 | 1136 | | |
| |||
1074 | 1143 | | |
1075 | 1144 | | |
1076 | 1145 | | |
| 1146 | + | |
1077 | 1147 | | |
1078 | 1148 | | |
1079 | 1149 | | |
| |||
1126 | 1196 | | |
1127 | 1197 | | |
1128 | 1198 | | |
1129 | | - | |
| 1199 | + | |
1130 | 1200 | | |
1131 | 1201 | | |
1132 | 1202 | | |
| |||
1503 | 1573 | | |
1504 | 1574 | | |
1505 | 1575 | | |
1506 | | - | |
| 1576 | + | |
1507 | 1577 | | |
1508 | 1578 | | |
1509 | 1579 | | |
| |||
0 commit comments