Repository navigation
[opt](paimon) report error when mapping variant type disable variant_v2 - #66466
Conversation
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
/review |
|
Codex automated review failed and did not complete. Error: All Codex review accounts are usage-limited; earliest retry is 2026-08-08T03:32:00Z. Please trigger /review again after that time. |
|
run buildall |
FE UT Coverage ReportIncrement line coverage |
|
run buildall |
1 similar comment
|
run buildall |
1c487ed to
79973fa
Compare
|
run buildall |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
FE Regression Coverage ReportIncrement line coverage |
79973fa to
4c99dd9
Compare
|
run buildall |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
FE UT Coverage ReportIncrement line coverage |
4c99dd9 to
7fd0218
Compare
|
run buildall |
FE UT Coverage ReportIncrement line coverage |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
FE Regression Coverage ReportIncrement line coverage |
…n table (#69227) ### What problem does this PR solve? Issue Number: None Related PR: #66466, #68242 Problem Summary: **In short.** Ports the primary-key Variant coverage of branch-4.1 #66466 to master: a Paimon deduplicate table whose second snapshot updates two keys, and three blocks of `test_paimon_catalog_variant` that read it through JNI. The rest of #66466 and the WAL fix #68242 are not ported, because master handles those cases another way; see below. **Background.** #66466 did three things on branch-4.x: 1. In BE, a generic gate in `TableReader::validate_file_mapping` rejects a physical Variant read from any file format but Parquet. #68242 then exempted WAL, which stores Doris blocks and supports Variant, because the gate broke group commit WAL replay. 2. In FE, `PaimonScanNode` rejects a query that projects a Variant column while Variant V2 is disabled. 3. In the tests, `run13.sql` gets `variant_primary_key_smoke`, and `test_paimon_catalog_variant` reads it. **The problem, and what it cost.** Every Variant table master's `test_paimon_catalog_variant` reads is append-only, so the regression does not check that a Paimon merge read keeps only the latest row of each key while it materializes Variant values. **How this PR fixes it.** - `run13.sql` creates `variant_primary_key_smoke` as on branch-4.x: a deduplicate primary-key table in Parquet, written in two snapshots. The second updates ids 1 and 2 and inserts id 4; id 3 holds a root array. - The JNI section of `test_paimon_catalog_variant` gets the three branch-4.x blocks: the whole Variant value, Variant paths cast to scalars, and a predicate on a Variant path. All three expect the latest value of ids 1 and 2. The other parts of #66466 and #68242 do not apply to master: | Branch-4.x change | Master | |---|---| | FE rejects a Variant projection while Variant V2 is disabled | #66858 made Variant V2 the default and removed the switch, so there is nothing to check | | BE rejects a physical Variant read from any format but Parquet | Would reject the native Variant reads of the ADBC reader (#68883). Master keeps the Iceberg-specific gate in `IcebergTableReader`, admits Arrow Variant reads in `FileScannerV2` only for ADBC, and plans a Paimon split that projects a Variant natively only when every file is Parquet or an ORC file whose schema predates the Variant column. Paimon 1.4.2 cannot create an ORC table with a VARIANT column (`Unsupported type: VARIANT`) | | WAL is exempted from that gate | WAL is not gated on master | **Results.** On a local cluster against the Iceberg docker environment, Spark reads `variant_primary_key_smoke` as four rows, with the second-snapshot values for ids 1 and 2. Doris returns the same rows in the three new blocks, and every other result block of the suite is unchanged. Co-authored-by: zhangstar333 <[email protected]>
What problem does this PR solve?
Problem Summary:
variant type mapping in paimon, should set enable_file_scanner_v2 , experimental_enable_variant_v2 to true.
and format should be parquet files. others will report error.
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)