Repository navigation
Commit 949a806
authored
[improvement](fe) Upgrade Paimon to 2.1.0 (apache#69343)
### What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
Problem Summary:
Upgrade the Paimon Java SDK from 1.4.2 to 2.1.0, the latest stable
release published on October 12, 2026 ([upstream
release](https://github.com/apache/paimon/releases/tag/release-2.1.0)).
Pin the FE connector and BE JNI scanner to the same shared
`paimon.version`, since tables, splits, and commit messages cross the
FE/BE boundary using Paimon's serialization.
Adapt the connector to the removal of Paimon's file-based `Privilege*`
API in 2.1.0, removing its obsolete catalog/table wrappers and
reflection-based delegate reconstruction. Preserve catalog query
authorization, fallback/chain branch semantics, statement-bound schemas
and snapshots, and Doris metadata-cache governance. Update local test
fixtures to `FileSystemSchemaManager` and the extended `Snapshot`
constructor. Give scanner tests the `java.nio` access already enabled in
the BE JVM for Arrow Variant buffer views. Update the oversized-write
negative test to the SDK's revised diagnostic while retaining its
snapshot/row atomicity checks. Retain the branch/scan API adaptations
and grouped format-file split fixture required since 2.0. Verify both
the default wrapping and explicitly enabled checked integer-overflow
modes in 2.1, as well as the corrected pre-epoch timestamp-to-date cast.
Keep the existing Rust reader eligibility rules and common/S3
compatibility patches. Fix changelog MERGE target-presence detection for
nullable primary keys: matched-only plans use their INNER JOIN
semantics, while plans with INSERT branches use a pre-join marker. Keep
that private marker outside user expression resolution. Preserve catalog
query authorization for the new `$file_indexes` system table and pin
both serialized table payloads to the catalog-published snapshot before
stripping their catalog loaders. Adapt timestamp precision repair to the
SDK's authorization projection validation and four-argument policy
callback; use real JSON policy definitions in serialized scanner
fixtures. Reject `scan.mode=latest-delta` in Doris `@options`, because
the existing full-snapshot pinning would silently widen a delta scan;
explicit `@incr` ranges remain available.
The FE and BE Java extensions must be deployed with the same SDK
version. This change does not establish compatibility between mixed
1.4.2/2.1.0 SDKs during a rolling upgrade.
### Paimon 2.1 feature impact on Doris
| Upstream change | Doris impact and scope of this PR |
| --- | --- |
| [Nullable primary keys](apache/paimon#9094),
opt-in with `primary-key.nullable=true` | Existing write-column mapping
respects the SDK field nullability. Added a real-table test of nullable
key metadata, FE/BE table serialization, and NULL-key deduplication
across commits through the Java reader. The cluster regression
additionally checks NULL-key deduplication across commits through JNI,
and checks a single-file table through both JNI and the actual Rust
Reader (verified from the completed query profile). Multi-file
primary-key merges retain the existing JNI fallback. Add Doris SQL
coverage for NULL-safe MERGE UPDATE/DELETE on nullable single/composite
keys, ordinary equality, INSERT branches, conditional priority,
duplicate-match atomicity and private-marker name collisions. |
| [`$file_indexes`](apache/paimon#9138) |
Automatically discovered through the SDK registry. Preserve deferred
catalog query checks and pin both backend payloads. FE scan-properties
and actual JNI reconstruction tests cover denied authorization and
snapshot 2 on disk while the catalog publishes only snapshot 1. |
| [`latest-delta` batch startup
mode](apache/paimon#9142) | Not exposed by this
upgrade: reject it explicitly in `@options` instead of silently
converting it to a full-snapshot read. A real three-snapshot test proves
the difference (one delta row versus three full-snapshot rows). |
| [Vector indexes](apache/paimon#8930) and
[full-text indexes](apache/paimon#9031) on DV
tables; [Multivalue array
indexes](apache/paimon#9282); BTree TopN | New
query acceleration needs Doris expression/search/TopN integration and
relevant runtime dependencies and validation. Bumping the SDK does not
enable these SQL capabilities. Existing reader selection remains
conservative. |
| [Arrow bundle writes](apache/paimon#9100) | An
optional batching API, not a new `file.format=arrow` storage format.
Doris keeps its current Arrow-to-Paimon-row writer; using the new bundle
API is a separate performance change requiring memory/lifecycle
validation and benchmarks. |
| [SchemaManager interface](apache/paimon#9174)
and [REST schema history
APIs](apache/paimon#9673) | Updated fixtures to
`FileSystemSchemaManager`. The REST additions inspect schema history;
they do not replace the SDK table's filesystem schema manager. Doris
does not call the new display APIs in this PR. |
| Manifest/index metadata and Variant handling | Both Java endpoints use
SDK 2.1.0; package and serialization tests passed. The scanner tests now
have the same `java.nio` access as the BE JVM. The targeted Java, native
and cluster checks below cover existing supported paths, but do not
establish native/Rust compatibility for every newly introduced table
option or file layout. |
| BLOB partial updates and first-row/data-evolution improvements |
Available upstream; the upgrade preserves Doris' existing write and
reader eligibility constraints. New combinations still need
external-table regression coverage before being advertised. |
| [Removal of file-based
privileges](apache/paimon#9186), arithmetic/cast
fixes, raw physical-metadata restrictions | Actual behavior changes,
described in the release note below; legacy privilege users must migrate
before upgrading. |
### Release note
Upgrade the Paimon Java SDK from 1.4.2 to 2.1.0. Fix NULL-safe MERGE
matched UPDATE/DELETE silently omitting rows with NULL primary keys.
Paimon 2.1 keeps wrapping integer `sum`/`product` overflow by default
and supports per-field `fields.<field>.sum.fail-on-overflow` /
`fields.<field>.product.fail-on-overflow` options for checked
arithmetic. Timestamp-to-date casts before the Unix epoch now use the
preceding calendar date. The SDK rejects raw `$files`,
`$file_key_ranges`, `$binlog`, and `$statistics` tables on query-auth
tables because column masks cannot be applied to their physical
metadata. Doris now preserves query authorization and pins
`$file_indexes` to the catalog-visible snapshot and explicitly rejects
the unimplemented `latest-delta` OPTIONS mode instead of returning a
full snapshot. Paimon's legacy file-based privilege system is no longer
supported by the upstream SDK; deployments using it must migrate
authorization before upgrading.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [x] Manual test
- **Build:** FE/Java extensions and ASAN BE passed after rebasing onto
master `5e3ed84abb0`; both Java endpoints package SDK 2.1.0. All 9 JNI
plugins passed layout checks.
- **Unit tests:** FE connector 844 passed, 1 live-connectivity test
skipped; Java scanner 175 passed; native Paimon/JNI/Rust 144 passed; S3
shade 1 passed. The MERGE follow-up passed 38 related FE tests.
- **Cluster regression:** 51 Paimon suites passed after rebase; the
MERGE follow-up passed 9 targeted suites, including the new nullable-key
regression. The opt-in KEY_DYNAMIC resource stress suite was skipped.
System-table expected output was regenerated with the runner and
replayed successfully.
- **Backward reads:** Paimon 1.4.2 append/PK data matched its Spark
oracle again; completed profiles confirmed Rust append reads and JNI
fallback for multi-file PK merges.
Scope: S3/filesystem Paimon catalog. File cache was disabled for a local
Btrfs inode-stat issue, and the host swap startup check was skipped.
Paimon HMS/REST catalogs and mixed-SDK rolling upgrades were not
validated.
- Behavior changed:
- [ ] No.
- [x] Yes. Uses Paimon 2.1.0 SDK behavior, including configurable
integer overflow checks, corrected pre-epoch timestamp-to-date casts,
removal of upstream file-based privileges, snapshot-safe
`$file_indexes`, and explicit rejection of unsupported `latest-delta`
OPTIONS reads.
- Does this need documentation?
- [x] No. Dependency upgrade and upstream behavior changes are recorded
in the release note.
- [ ] Yes.
### Self-review
- Validation follow-up scope: fix nullable-key MERGE target-presence
detection, then preserve deferred `$file_indexes` authorization and its
JNI source snapshot pin, adapt timestamp-policy validation and callbacks
to SDK 2.1, and refresh affected system-table expected output through
the regression runner.
- Goal and scope: shared SDK upgrade plus directly affected API/behavior
adaptations. New search and batching features remain separate work; no
new connector SPI or native ABI is introduced.
- Correctness and parallel paths: FE binding, serialized BE system-table
reconstruction, JNI SDK behavior, and reader selection were inspected.
`$file_indexes` preserves deferred catalog authorization and carries the
snapshot pin through both FE/BE payloads; timestamp policies validate
and execute against repaired precision; MERGE uses INNER JOIN semantics
or a null-extended target marker rather than testing a nullable key, and
user expressions retain their original binding scope; both OPTIONS
resolution and statement-fence pinning reject `latest-delta`.
- Concurrency and lifecycle: production adaptations add no threads,
locks, shared mutable state, native allocations, or resource owners.
Java fixtures close their catalog/writer/commit/reader resources; the
new cluster suite restores modified session variables in `finally` and
preserves tables for debugging. The existing writer-lifecycle regression
passed 120 successive inserts and verified compaction writer threads
return to zero after each phase.
- Performance: reuse the existing row repair and SDK policy evaluation;
bind source snapshots during FE planning. MERGE adds one constant target
projection only when INSERT branches need unmatched rows. No performance
improvement is claimed.
- Compatibility and configuration: upstream file privileges are removed,
and mixed SDK rolling upgrades remain unverified. No configuration keys,
Thrift/Protobuf definitions, new FE/BE variables, or persistence formats
are added by Doris. Storage access for `$file_indexes` uses the existing
configured catalog endpoint and base-table SELECT privilege; there is no
new artifact-download property.
- Tests and results: FE scan-properties tests cover denied
`$file_indexes` authorization and catalog-published snapshot bounds; an
actual serialized JNI reconstruction test covers the second payload.
Serialized timestamp-policy tests retain row filters, masking, fallback
and nested/hidden-column coverage. MERGE tests cover nullable
single/composite keys, ordinary and NULL-safe equality, branch priority,
duplicate-match rejection without row changes, and same-named user
columns; the new golden is generated and replayed by the regression
runner. The system-table golden now includes SDK 2.1's `operation` and
`writer_version` fields. Builds, unit tests, cluster replay and
legacy-data reads passed after rebase; limits are listed above.
- Other CI failures: the P0 view case passed in isolation; it shares and
drops `v_distinct_agg_rewrite` with `test_mtmv` in the same database.
The Iceberg failure was reproduced as `DESC digest` expecting nullable
`Yes` but returning `No`: [master's source-nullability
change](apache@c7ffbda3877) predates the
[test's old
expectation](apache@e83b06cb7b7). Those
Iceberg and P0 files are unchanged from the base; no unrelated golden
was modified.
- Independent optimizer limitation: a one-row constant source with
multiple NULL-safe join keys can become an empty relation even when an
internal OLAP table has a matching NULL row. This also reproduces in an
ordinary SELECT on an internal table; optimizer files are unchanged by
this PR. The composite-key DELETE regression uses a multi-row source
with an unmatched control. That standalone optimizer defect remains
outside this SDK adaptation.
- Transactions, writes and observability: no transaction/recovery
implementation is changed. The transaction regression confirms an
oversized failed insert publishes neither rows nor a snapshot. Completed
query profiles prove Rust selection and JNI fallback, rather than
relying only on enable flags. Unsupported delta mode fails with an
actionable diagnostic; existing system-table planning diagnostics apply
to snapshot pinning.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label1 parent f2117db commit 949a806
43 files changed
Lines changed: 860 additions & 591 deletions
File tree
- fe
- be-java-extensions/paimon-scanner
- src
- main/java/org/apache/doris/paimon
- test/java/org/apache/doris/paimon
- fe-connector/fe-connector-paimon/src
- main/java/org/apache/doris/connector/paimon
- test/java/org/apache/doris/connector/paimon
- fe-core/src
- main/java/org/apache/doris/nereids/rules/analysis
- test/java/org/apache/doris/nereids/rules/analysis
- regression-test
- data/external_table_p0/paimon
- suites/external_table_p0/paimon
- write
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
471 | 471 | | |
472 | 472 | | |
473 | 473 | | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
474 | 477 | | |
475 | 478 | | |
476 | 479 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
968 | 968 | | |
969 | 969 | | |
970 | 970 | | |
971 | | - | |
972 | | - | |
| 971 | + | |
| 972 | + | |
973 | 973 | | |
974 | 974 | | |
975 | 975 | | |
| |||
1039 | 1039 | | |
1040 | 1040 | | |
1041 | 1041 | | |
1042 | | - | |
| 1042 | + | |
1043 | 1043 | | |
1044 | 1044 | | |
1045 | 1045 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
146 | 146 | | |
147 | 147 | | |
148 | 148 | | |
149 | | - | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
150 | 166 | | |
151 | 167 | | |
152 | 168 | | |
153 | | - | |
| 169 | + | |
154 | 170 | | |
155 | | - | |
| 171 | + | |
| 172 | + | |
156 | 173 | | |
157 | 174 | | |
158 | 175 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
41 | 41 | | |
42 | 42 | | |
43 | 43 | | |
44 | | - | |
| 44 | + | |
45 | 45 | | |
46 | 46 | | |
47 | 47 | | |
| |||
263 | 263 | | |
264 | 264 | | |
265 | 265 | | |
266 | | - | |
| 266 | + | |
267 | 267 | | |
268 | 268 | | |
269 | 269 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
29 | 29 | | |
30 | 30 | | |
31 | 31 | | |
32 | | - | |
33 | | - | |
34 | 32 | | |
35 | 33 | | |
36 | 34 | | |
37 | | - | |
38 | 35 | | |
39 | 36 | | |
40 | 37 | | |
| |||
503 | 500 | | |
504 | 501 | | |
505 | 502 | | |
| 503 | + | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
| 507 | + | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
| 524 | + | |
| 525 | + | |
| 526 | + | |
| 527 | + | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
| 531 | + | |
| 532 | + | |
| 533 | + | |
| 534 | + | |
| 535 | + | |
| 536 | + | |
| 537 | + | |
| 538 | + | |
| 539 | + | |
| 540 | + | |
| 541 | + | |
| 542 | + | |
| 543 | + | |
| 544 | + | |
| 545 | + | |
| 546 | + | |
| 547 | + | |
| 548 | + | |
| 549 | + | |
| 550 | + | |
| 551 | + | |
| 552 | + | |
| 553 | + | |
| 554 | + | |
| 555 | + | |
| 556 | + | |
| 557 | + | |
| 558 | + | |
| 559 | + | |
506 | 560 | | |
507 | 561 | | |
508 | 562 | | |
| |||
636 | 690 | | |
637 | 691 | | |
638 | 692 | | |
639 | | - | |
640 | | - | |
641 | | - | |
642 | | - | |
643 | | - | |
644 | | - | |
645 | | - | |
646 | | - | |
647 | | - | |
648 | | - | |
649 | | - | |
650 | | - | |
651 | | - | |
652 | | - | |
653 | | - | |
654 | | - | |
655 | | - | |
656 | | - | |
657 | | - | |
658 | | - | |
659 | | - | |
660 | | - | |
661 | | - | |
662 | | - | |
663 | | - | |
664 | | - | |
665 | | - | |
666 | | - | |
667 | 693 | | |
668 | 694 | | |
669 | 695 | | |
| |||
708 | 734 | | |
709 | 735 | | |
710 | 736 | | |
711 | | - | |
712 | | - | |
713 | | - | |
714 | | - | |
715 | | - | |
716 | | - | |
717 | | - | |
718 | | - | |
719 | | - | |
720 | | - | |
721 | | - | |
722 | | - | |
723 | | - | |
724 | | - | |
725 | | - | |
726 | | - | |
727 | | - | |
728 | | - | |
729 | | - | |
730 | | - | |
731 | | - | |
732 | | - | |
733 | | - | |
734 | 737 | | |
735 | 738 | | |
736 | 739 | | |
| |||
1282 | 1285 | | |
1283 | 1286 | | |
1284 | 1287 | | |
| 1288 | + | |
| 1289 | + | |
| 1290 | + | |
1285 | 1291 | | |
1286 | 1292 | | |
1287 | 1293 | | |
| |||
0 commit comments