Skip to content

[opt](paimon) report error when mapping variant type disable variant_v2 - #66466

Merged
yiguolei merged 4 commits into
apache:branch-4.1from
zhangstar333:branch-4.1
Aug 7, 2026
Merged

yiguolei merged 4 commits into
apache:branch-4.1from
zhangstar333:branch-4.1

Conversation

@zhangstar333

@zhangstar333 zhangstar333 commented Aug 5, 2026 •

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Problem Summary:
variant type mapping in paimon, should set enable_file_scanner_v2 , experimental_enable_variant_v2 to true.
and format should be parquet files. others will report error.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@zhangstar333
zhangstar333 requested a review from yiguolei as a code owner August 5, 2026 04:58
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zhangstar333

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

github-actions Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: All Codex review accounts are usage-limited; earliest retry is 2026-08-08T03:32:00Z.
Workflow run: https://github.com/apache/doris/actions/runs/30976737902

Please trigger /review again after that time.

@zhangstar333

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 20.00% (3/15) 🎉
Increment coverage report
Complete coverage report

@zhangstar333

Copy link
Copy Markdown
Contributor Author

run buildall

1 similar comment
@zhangstar333

Copy link
Copy Markdown
Contributor Author

run buildall

@zhangstar333

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 60.78% (31/51) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 58.26% (24672/42345)
Line Coverage 42.51% (248492/584562)
Region Coverage 38.54% (196168/508950)
Branch Coverage 39.94% (89806/224828)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 90.20% (46/51) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 73.76% (30443/41273)
Line Coverage 57.69% (334693/580187)
Region Coverage 54.51% (278196/510316)
Branch Coverage 55.53% (124707/224586)

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 0.00% (0/32) 🎉
Increment coverage report
Complete coverage report

@zhangstar333

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 60.78% (31/51) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 58.30% (24717/42395)
Line Coverage 42.57% (249167/585333)
Region Coverage 38.62% (196873/509768)
Branch Coverage 40.02% (90151/225286)

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 20.00% (3/15) 🎉
Increment coverage report
Complete coverage report

@zhangstar333

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 20.00% (3/15) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 60.78% (31/51) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 58.32% (24736/42412)
Line Coverage 42.60% (249472/585561)
Region Coverage 38.65% (197143/510015)
Branch Coverage 40.05% (90271/225374)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 90.20% (46/51) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 73.74% (30484/41338)
Line Coverage 57.73% (335497/581180)
Region Coverage 54.59% (279153/511375)
Branch Coverage 55.53% (125021/225132)

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 93.33% (14/15) 🎉
Increment coverage report
Complete coverage report

@yiguolei
yiguolei merged commit c3cfb38 into apache:branch-4.1 Aug 7, 2026
29 of 32 checks passed
morningman added a commit that referenced this pull request Oct 19, 2026
…n table (#69227)

### What problem does this PR solve?

Issue Number: None

Related PR: #66466, #68242

Problem Summary:

**In short.** Ports the primary-key Variant coverage of branch-4.1
#66466 to master: a Paimon deduplicate table whose second snapshot
updates two keys, and three blocks of `test_paimon_catalog_variant` that
read it through JNI. The rest of #66466 and the WAL fix #68242 are not
ported, because master handles those cases another way; see below.

**Background.** #66466 did three things on branch-4.x:

1. In BE, a generic gate in `TableReader::validate_file_mapping` rejects
a physical Variant read from any file format but Parquet. #68242 then
exempted WAL, which stores Doris blocks and supports Variant, because
the gate broke group commit WAL replay.
2. In FE, `PaimonScanNode` rejects a query that projects a Variant
column while Variant V2 is disabled.
3. In the tests, `run13.sql` gets `variant_primary_key_smoke`, and
`test_paimon_catalog_variant` reads it.

**The problem, and what it cost.** Every Variant table master's
`test_paimon_catalog_variant` reads is append-only, so the regression
does not check that a Paimon merge read keeps only the latest row of
each key while it materializes Variant values.

**How this PR fixes it.**

- `run13.sql` creates `variant_primary_key_smoke` as on branch-4.x: a
deduplicate primary-key table in Parquet, written in two snapshots. The
second updates ids 1 and 2 and inserts id 4; id 3 holds a root array.
- The JNI section of `test_paimon_catalog_variant` gets the three
branch-4.x blocks: the whole Variant value, Variant paths cast to
scalars, and a predicate on a Variant path. All three expect the latest
value of ids 1 and 2.

The other parts of #66466 and #68242 do not apply to master:

| Branch-4.x change | Master |
|---|---|
| FE rejects a Variant projection while Variant V2 is disabled | #66858
made Variant V2 the default and removed the switch, so there is nothing
to check |
| BE rejects a physical Variant read from any format but Parquet | Would
reject the native Variant reads of the ADBC reader (#68883). Master
keeps the Iceberg-specific gate in `IcebergTableReader`, admits Arrow
Variant reads in `FileScannerV2` only for ADBC, and plans a Paimon split
that projects a Variant natively only when every file is Parquet or an
ORC file whose schema predates the Variant column. Paimon 1.4.2 cannot
create an ORC table with a VARIANT column (`Unsupported type: VARIANT`)
|
| WAL is exempted from that gate | WAL is not gated on master |

**Results.** On a local cluster against the Iceberg docker environment,
Spark reads `variant_primary_key_smoke` as four rows, with the
second-snapshot values for ids 1 and 2. Doris returns the same rows in
the three new blocks, and every other result block of the suite is
unchanged.

Co-authored-by: zhangstar333 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants