[improvement](fe) Upgrade Paimon to 2.1.0 by suxiaogang223 · Pull Request #69343 · apache/doris

suxiaogang223 · 2026-10-20T10:39:28Z

What problem does this PR solve?

Issue Number: N/A

Related PR: N/A

Problem Summary:

Upgrade the Paimon Java SDK from 1.4.2 to 2.1.0, the latest stable release published on October 12, 2026 (upstream release). Pin the FE connector and BE JNI scanner to the same shared paimon.version, since tables, splits, and commit messages cross the FE/BE boundary using Paimon's serialization.

Adapt the connector to the removal of Paimon's file-based Privilege* API in 2.1.0, removing its obsolete catalog/table wrappers and reflection-based delegate reconstruction. Preserve catalog query authorization, fallback/chain branch semantics, statement-bound schemas and snapshots, and Doris metadata-cache governance. Update local test fixtures to FileSystemSchemaManager and the extended Snapshot constructor. Give scanner tests the java.nio access already enabled in the BE JVM for Arrow Variant buffer views. Update the oversized-write negative test to the SDK's revised diagnostic while retaining its snapshot/row atomicity checks. Retain the branch/scan API adaptations and grouped format-file split fixture required since 2.0. Verify both the default wrapping and explicitly enabled checked integer-overflow modes in 2.1, as well as the corrected pre-epoch timestamp-to-date cast. Keep the existing Rust reader eligibility rules and common/S3 compatibility patches. Fix changelog MERGE target-presence detection for nullable primary keys: matched-only plans use their INNER JOIN semantics, while plans with INSERT branches use a pre-join marker. Keep that private marker outside user expression resolution. Preserve catalog query authorization for the new $file_indexes system table and pin both serialized table payloads to the catalog-published snapshot before stripping their catalog loaders. Adapt timestamp precision repair to the SDK's authorization projection validation and four-argument policy callback; use real JSON policy definitions in serialized scanner fixtures. Reject scan.mode=latest-delta in Doris @options, because the existing full-snapshot pinning would silently widen a delta scan; explicit @incr ranges remain available.

The FE and BE Java extensions must be deployed with the same SDK version. This change does not establish compatibility between mixed 1.4.2/2.1.0 SDKs during a rolling upgrade.

Paimon 2.1 feature impact on Doris

Upstream change	Doris impact and scope of this PR
Nullable primary keys, opt-in with `primary-key.nullable=true`	Existing write-column mapping respects the SDK field nullability. Added a real-table test of nullable key metadata, FE/BE table serialization, and NULL-key deduplication across commits through the Java reader. The cluster regression additionally checks NULL-key deduplication across commits through JNI, and checks a single-file table through both JNI and the actual Rust Reader (verified from the completed query profile). Multi-file primary-key merges retain the existing JNI fallback. Add Doris SQL coverage for NULL-safe MERGE UPDATE/DELETE on nullable single/composite keys, ordinary equality, INSERT branches, conditional priority, duplicate-match atomicity and private-marker name collisions.
`$file_indexes`	Automatically discovered through the SDK registry. Preserve deferred catalog query checks and pin both backend payloads. FE scan-properties and actual JNI reconstruction tests cover denied authorization and snapshot 2 on disk while the catalog publishes only snapshot 1.
`latest-delta` batch startup mode	Not exposed by this upgrade: reject it explicitly in `@options` instead of silently converting it to a full-snapshot read. A real three-snapshot test proves the difference (one delta row versus three full-snapshot rows).
Vector indexes and full-text indexes on DV tables; Multivalue array indexes; BTree TopN	New query acceleration needs Doris expression/search/TopN integration and relevant runtime dependencies and validation. Bumping the SDK does not enable these SQL capabilities. Existing reader selection remains conservative.
Arrow bundle writes	An optional batching API, not a new `file.format=arrow` storage format. Doris keeps its current Arrow-to-Paimon-row writer; using the new bundle API is a separate performance change requiring memory/lifecycle validation and benchmarks.
SchemaManager interface and REST schema history APIs	Updated fixtures to `FileSystemSchemaManager`. The REST additions inspect schema history; they do not replace the SDK table's filesystem schema manager. Doris does not call the new display APIs in this PR.
Manifest/index metadata and Variant handling	Both Java endpoints use SDK 2.1.0; package and serialization tests passed. The scanner tests now have the same `java.nio` access as the BE JVM. The targeted Java, native and cluster checks below cover existing supported paths, but do not establish native/Rust compatibility for every newly introduced table option or file layout.
BLOB partial updates and first-row/data-evolution improvements	Available upstream; the upgrade preserves Doris' existing write and reader eligibility constraints. New combinations still need external-table regression coverage before being advertised.
Removal of file-based privileges, arithmetic/cast fixes, raw physical-metadata restrictions	Actual behavior changes, described in the release note below; legacy privilege users must migrate before upgrading.

Release note

Upgrade the Paimon Java SDK from 1.4.2 to 2.1.0. Fix NULL-safe MERGE matched UPDATE/DELETE silently omitting rows with NULL primary keys. Paimon 2.1 keeps wrapping integer sum/product overflow by default and supports per-field fields.<field>.sum.fail-on-overflow / fields.<field>.product.fail-on-overflow options for checked arithmetic. Timestamp-to-date casts before the Unix epoch now use the preceding calendar date. The SDK rejects raw $files, $file_key_ranges, $binlog, and $statistics tables on query-auth tables because column masks cannot be applied to their physical metadata. Doris now preserves query authorization and pins $file_indexes to the catalog-visible snapshot and explicitly rejects the unimplemented latest-delta OPTIONS mode instead of returning a full snapshot. Paimon's legacy file-based privilege system is no longer supported by the upstream SDK; deployments using it must migrate authorization before upgrading.

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test
- Build: FE/Java extensions and ASAN BE passed after rebasing onto master 5e3ed84abb0; both Java endpoints package SDK 2.1.0. All 9 JNI plugins passed layout checks.
- Unit tests: FE connector 844 passed, 1 live-connectivity test skipped; Java scanner 175 passed; native Paimon/JNI/Rust 144 passed; S3 shade 1 passed. The MERGE follow-up passed 38 related FE tests.
- Cluster regression: 51 Paimon suites passed after rebase; the MERGE follow-up passed 9 targeted suites, including the new nullable-key regression. The opt-in KEY_DYNAMIC resource stress suite was skipped. System-table expected output was regenerated with the runner and replayed successfully.
- Backward reads: Paimon 1.4.2 append/PK data matched its Spark oracle again; completed profiles confirmed Rust append reads and JNI fallback for multi-file PK merges.
Scope: S3/filesystem Paimon catalog. File cache was disabled for a local Btrfs inode-stat issue, and the host swap startup check was skipped. Paimon HMS/REST catalogs and mixed-SDK rolling upgrades were not validated.
Behavior changed:
- No.
- Yes. Uses Paimon 2.1.0 SDK behavior, including configurable integer overflow checks, corrected pre-epoch timestamp-to-date casts, removal of upstream file-based privileges, snapshot-safe $file_indexes, and explicit rejection of unsupported latest-delta OPTIONS reads.
Does this need documentation?
- No. Dependency upgrade and upstream behavior changes are recorded in the release note.
- Yes.

Self-review

Validation follow-up scope: fix nullable-key MERGE target-presence detection, then preserve deferred $file_indexes authorization and its JNI source snapshot pin, adapt timestamp-policy validation and callbacks to SDK 2.1, and refresh affected system-table expected output through the regression runner.
Goal and scope: shared SDK upgrade plus directly affected API/behavior adaptations. New search and batching features remain separate work; no new connector SPI or native ABI is introduced.
Correctness and parallel paths: FE binding, serialized BE system-table reconstruction, JNI SDK behavior, and reader selection were inspected. $file_indexes preserves deferred catalog authorization and carries the snapshot pin through both FE/BE payloads; timestamp policies validate and execute against repaired precision; MERGE uses INNER JOIN semantics or a null-extended target marker rather than testing a nullable key, and user expressions retain their original binding scope; both OPTIONS resolution and statement-fence pinning reject latest-delta.
Concurrency and lifecycle: production adaptations add no threads, locks, shared mutable state, native allocations, or resource owners. Java fixtures close their catalog/writer/commit/reader resources; the new cluster suite restores modified session variables in finally and preserves tables for debugging. The existing writer-lifecycle regression passed 120 successive inserts and verified compaction writer threads return to zero after each phase.
Performance: reuse the existing row repair and SDK policy evaluation; bind source snapshots during FE planning. MERGE adds one constant target projection only when INSERT branches need unmatched rows. No performance improvement is claimed.
Compatibility and configuration: upstream file privileges are removed, and mixed SDK rolling upgrades remain unverified. No configuration keys, Thrift/Protobuf definitions, new FE/BE variables, or persistence formats are added by Doris. Storage access for $file_indexes uses the existing configured catalog endpoint and base-table SELECT privilege; there is no new artifact-download property.
Tests and results: FE scan-properties tests cover denied $file_indexes authorization and catalog-published snapshot bounds; an actual serialized JNI reconstruction test covers the second payload. Serialized timestamp-policy tests retain row filters, masking, fallback and nested/hidden-column coverage. MERGE tests cover nullable single/composite keys, ordinary and NULL-safe equality, branch priority, duplicate-match rejection without row changes, and same-named user columns; the new golden is generated and replayed by the regression runner. The system-table golden now includes SDK 2.1's operation and writer_version fields. Builds, unit tests, cluster replay and legacy-data reads passed after rebase; limits are listed above.
Other CI failures: the P0 view case passed in isolation; it shares and drops v_distinct_agg_rewrite with test_mtmv in the same database. The Iceberg failure was reproduced as DESC digest expecting nullable Yes but returning No: master's source-nullability change predates the test's old expectation. Those Iceberg and P0 files are unchanged from the base; no unrelated golden was modified.
Independent optimizer limitation: a one-row constant source with multiple NULL-safe join keys can become an empty relation even when an internal OLAP table has a matching NULL row. This also reproduces in an ordinary SELECT on an internal table; optimizer files are unchanged by this PR. The composite-key DELETE regression uses a multi-row source with an unmatched control. That standalone optimizer defect remains outside this SDK adaptation.
Transactions, writes and observability: no transaction/recovery implementation is changed. The transaction regression confirms an oversized failed insert publishes neither rows nor a snapshot. Completed query profiles prove Rust selection and JNI fallback, rather than relying only on enable flags. Unsupported delta mode fails with an actionable diagnostic; existing system-table planning diagnostics apply to snapshot pinning.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2026-10-20T10:39:33Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

suxiaogang223 · 2026-10-20T10:46:14Z

run buildall

hello-stephen · 2026-10-20T16:48:53Z

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

### What problem does this PR solve? Issue Number: N/A Related PR: apache#69343 Problem Summary: Upgrade the shared FE connector and BE scanner SDK pin from 2.0.0 to Paimon 2.1.0. Remove the upstream-deleted file-based privilege wrappers and their reflection-based reconstruction. Update filesystem schema and snapshot fixtures, verify default and checked integer overflow modes, and cover corrected pre-epoch date casts and query-auth file-statistics restrictions. Match scanner test JVM access to the existing BE java.nio opening required by Arrow Variant buffer views. ### Release note Upgrade the Paimon Java SDK from 1.4.2 to 2.1.0. The SDK removes legacy file-based privileges, rejects raw file statistics on query-auth tables, corrects pre-epoch timestamp-to-date casts, and supports per-field checked integer aggregation while preserving wrapping overflow by default. ### Check List (For Author) - Test: Unit Test / Manual test - FE connector: 831 passed, 1 live-connectivity test skipped. - BE scanner: 142 passed. S3 shade: 1 passed. - Maven validate/Checkstyle and plugin packaging passed. - Checked scanner duplicate classes, SPI isolation, dependency closure, and API stamps: 162 JARs, no failures. - Verified matching 2.1.0 SDKs and byte-for-byte S3 embedded runtime. - Full build blocked by missing local Paimon Rust third-party artifacts; native BE and cluster regression tests were not run. - Behavior changed: Yes, adopts the Paimon 2.1.0 behavior described above. - Does this need documentation: No, upstream changes are in the release note.

@incr

### What problem does this PR solve? Issue Number: N/A Related PR: apache#69343 Problem Summary: Paimon 2.1 adds file_indexes, whose backend reader re-plans files after Doris strips its catalog loader, and latest-delta, which Doris would incorrectly resolve to a full snapshot. Pin file_indexes to the catalog-published snapshot and reject the unimplemented delta OPTIONS mode with guidance to use explicit @incr ranges. Add persisted tests for both cases and for nullable primary-key metadata, serialization, and NULL-key deduplication across commits. ### Release note Fix catalog snapshot visibility for Paimon file_indexes and explicitly reject unsupported latest-delta OPTIONS reads instead of reading a full snapshot. ### Check List (For Author) - Test: Unit Test / Manual test: run-fe-ut.sh with temporary Paimon module scope, Checkstyle enabled, and install packaging; 834 passed and 1 live-connectivity test skipped. FE plugin ZIP produced. The runner was restored. Native BE/Rust and cluster regressions were not run because the local third-party Paimon native library/header are missing. - Behavior changed: Yes, pin new system-table reads and reject an unsupported startup mode rather than returning incorrect rows. - Does this need documentation: No, compatibility impact is recorded in the PR description and release note.

### What problem does this PR solve? Issue Number: N/A Related PR: apache#69343 Problem Summary: Exercise the upgraded SDK in a real Doris cluster. Verify nullable primary-key deduplication through JNI and a profile-confirmed Rust reader, bitmap file-index metadata, rejection of unsupported latest-delta OPTIONS, and wrapping versus checked sum overflow. Generate the expected output with the regression runner and verify it in a normal run. Update the oversized-write error assertion to Paimon 2.1's diagnostic while retaining the existing row and snapshot atomicity checks. ### Release note None ### Check List (For Author) - Test: Regression test / Unit Test / Manual test - New compatibility suite: generated output and normal replay passed. - Read compatibility matrix: 13 suites passed. - Metadata and transaction matrix: 6 suites passed. - Paimon ASAN native UT: 108 tests in 14 suites passed. - Java scanner: 142 tests passed; S3 shade: 1 test passed. - Native BE ASAN build and deployed plugin checks passed. - Manual backward read of 1.4.2 tables through JNI and Rust matched the Spark oracle. - Behavior changed: No (test coverage and updated SDK diagnostic assertion only) - Does this need documentation: No

suxiaogang223 · 2026-10-21T09:03:12Z

run buildall

suxiaogang223 · 2026-10-21T09:11:24Z

/review

github-actions

Request changes: two verified issues in the newly exposed $file_indexes path: [P1] loss of the deferred catalog query check and [P2] loss of the catalog snapshot pin during JNI reconstruction.

Static review covered all 35 changed paths at 967cf6af6255e3fb011e99460ef74b706e736322, using the supplied aggregate diff. No existing inline threads or additional user focus were supplied.

Critical checkpoint conclusions:

Goal and proof: the shared SDK upgrade and removed-API adaptations are consistent with Paimon 2.1, but the new system-table integration has the two inline defects. The direct backend-table test omits the real JNI source replacement; no denied-query test covers $file_indexes.
Scope and reuse: the production changes are focused on the upgrade and reuse the existing branch, cache, and scan helpers.
Concurrency and locks: no new Doris threads, locks, or shared mutable state. External catalog publication across FE/BE stages is relevant to the snapshot finding.
Lifecycle and initialization: no new resource owner or static initialization dependency. Catalog-loader removal is the lifecycle boundary responsible for the lost query check; existing cache/catalog cleanup remains intact.
Configuration: no new FE/BE deployment knob or dynamic configuration propagation. The new SDK startup literal is explicitly rejected for OPTIONS; existing manifest caps remain branch-specific.
Compatibility: both Java endpoints share SDK 2.1.0, and patch/shade linkage was inspected. Mixed 1.4.2/2.1.0 JNI splits and commit serialization are incompatible; coordinated same-SDK deployment is an explicit PR restriction, not an additional inline finding. Legacy file privileges are intentionally removed upstream and documented in the PR.
Thrift/Protobuf: no definitions, field IDs, or enum values change.
Parallel paths: normal, fallback/chain, REST schema binding, metadata-cache/statistics, JNI, native/Rust, and deferred system-source paths were traced. The two findings require separate snapshot and query-check fixes.
Conditions and errors: the latest-delta guard explains the full-snapshot conflict and directs users to explicit @incr; both OPTIONS resolution and statement fencing reach it. Incremental resets preserve explicit ranges. No further new selector defect was substantiated.
Test coverage: inspected the new nullable-key, persisted-index, overflow, timestamp, serde and regression cases. The transport and denied-query gaps above remain. Builds and tests were prohibited and were not run.
Expected results: ordered NULL-key merges, bitmap record counts, wrapping/checked arithmetic, pre-epoch date and revised diagnostic match the SDK source. Actual generation provenance and runtime output were not independently verified.
Observability: existing error context and metrics are sufficient for these API changes; the regression checks a completed Rust-reader profile. No separate new observability defect found.
Transactions and persistence: no Doris EditLog/recovery implementation changes. SDK commit messages retain explicit version framing; mixed SDK operation remains outside the declared deployment scope.
Writes and atomicity: the SDK write path and nullable-key mapping were inspected; the failed-write regression retains its row/snapshot atomicity assertions. Runtime crash/commit behavior was not validated by this static review.
FE/BE variables: no new protocol variable is added. Existing serialized-table and serialized-system-source payloads were traced end to end; their mismatch causes the snapshot finding.
Performance and memory: no new scan loop, large allocation, lock contention, or complexity regression was substantiated. Existing caps and conservative JNI fallback remain in effect.
Further review and user focus: all changed files and initial risks received dispositions, including a missed-issue recheck by every reviewer. There was no additional user focus.

Limits: source review only; author-reported builds, packages, native tests and cluster results were not rerun. Live HMS/REST services and mixed-SDK deployment were not exercised. The SDK system views retain their existing two-phase partition-marker and fallback-only inventory limitations and do not establish full statement MVCC. No candidate remains unverified; the two accepted defects remain to be fixed.

Review converged after 3 round(s).

hello-stephen · 2026-10-21T13:55:38Z

TPC-H: Total hot run time: 28054 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 967cf6af6255e3fb011e99460ef74b706e736322, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17591	5126	5010	5010
q2	2122	314	256	256
q3	10235	1352	797	797
q4	4676	454	339	339
q5	7413	781	525	525
q6	178	186	153	153
q7	738	787	605	605
q8	9251	1282	1013	1013
q9	5313	4161	4162	4161
q10	6832	1272	1027	1027
q11	444	246	229	229
q12	636	391	279	279
q13	18099	3477	2966	2966
q14	267	271	247	247
q15	q16	722	702	657	657
q17	1198	730	569	569
q18	6968	6167	6067	6067
q19	1092	956	581	581
q20	387	336	231	231
q21	4890	2091	2275	2091
q22	340	295	251	251
Total cold run time: 99392 ms
Total hot run time: 28054 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5546	5555	5551	5551
q2	652	552	543	543
q3	5198	5397	4880	4880
q4	1999	2129	1564	1564
q5	5205	5142	5216	5142
q6	331	215	158	158
q7	2313	2042	1800	1800
q8	3029	2665	2607	2607
q9	7979	7918	7866	7866
q10	3723	3645	3253	3253
q11	600	427	596	427
q12	691	705	511	511
q13	3416	3921	3211	3211
q14	310	313	298	298
q15	q16	730	778	682	682
q17	8205	7529	6951	6951
q18	12991	12188	12779	12188
q19	877	765	765	765
q20	2125	2138	1891	1891
q21	5517	4523	4673	4523
q22	517	446	412	412
Total cold run time: 71954 ms
Total hot run time: 65223 ms

hello-stephen · 2026-10-21T14:05:25Z

TPC-DS: Total hot run time: 150200 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 967cf6af6255e3fb011e99460ef74b706e736322, data reload: false

query5	4310	586	440	440
query6	430	203	190	190
query7	4802	438	235	235
query8	321	161	155	155
query9	8803	3073	3072	3072
query10	481	312	276	276
query11	5972	3848	3574	3574
query12	154	93	88	88
query13	1259	453	326	326
query14	6128	3929	3614	3614
query14_1	3363	3431	3414	3414
query15	208	199	178	178
query16	939	446	410	410
query17	857	663	521	521
query18	2425	435	305	305
query19	192	161	135	135
query20	86	81	84	81
query21	205	129	105	105
query22	12898	13164	12832	12832
query23	13435	12807	12305	12305
query23_1	12506	12494	12596	12494
query24	7405	819	459	459
query24_1	473	478	468	468
query25	534	419	358	358
query26	1272	258	138	138
query27	2760	444	270	270
query28	4619	1912	1941	1912
query29	1587	604	456	456
query30	304	222	188	188
query31	1008	941	862	862
query32	150	109	92	92
query33	613	302	245	245
query34	997	845	489	489
query35	744	765	639	639
query36	832	827	760	760
query37	127	99	87	87
query38	1802	1761	1687	1687
query39	797	796	742	742
query39_1	755	746	760	746
query40	217	119	97	97
query41	71	68	66	66
query42	87	87	84	84
query43	342	373	315	315
query44	1369	701	712	701
query45	189	181	169	169
query46	849	980	575	575
query47	2993	3030	2790	2790
query48	305	306	256	256
query49	575	395	289	289
query50	679	274	204	204
query51	10584	10763	10204	10204
query52	83	80	70	70
query53	180	199	150	150
query54	247	191	193	191
query55	70	66	61	61
query56	235	218	199	199
query57	1662	1602	1574	1574
query58	279	255	252	252
query59	2169	2093	2008	2008
query60	275	246	219	219
query61	147	142	140	140
query62	376	321	282	282
query63	185	155	152	152
query64	2737	969	771	771
query65	3310	3256	3238	3238
query66	1804	409	312	312
query67	20113	20041	19931	19931
query68	3016	972	599	599
query69	370	282	236	236
query70	801	815	762	762
query71	284	232	221	221
query72	2690	2500	2174	2174
query73	520	518	287	287
query74	4606	4497	4365	4365
query75	2332	2291	1885	1885
query76	2323	1002	594	594
query77	343	362	268	268
query78	9732	9666	9070	9070
query79	978	848	502	502
query80	1169	408	336	336
query81	585	314	276	276
query82	574	130	105	105
query83	303	190	173	173
query84	311	135	99	99
query85	860	432	371	371
query86	389	242	227	227
query87	2030	1941	1804	1804
query88	3605	2668	2625	2625
query89	334	278	238	238
query90	1905	167	176	167
query91	157	147	125	125
query92	94	88	90	88
query93	977	981	571	571
query94	617	301	273	273
query95	574	338	360	338
query96	665	467	232	232
query97	2436	2409	2335	2335
query98	159	152	149	149
query99	773	775	661	661
Total cold run time: 234910 ms
Total hot run time: 150200 ms

hello-stephen · 2026-10-21T14:10:17Z

ClickBench: Total hot run time: 24.19 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 967cf6af6255e3fb011e99460ef74b706e736322, data reload: false

query1	0.00	0.00	0.00
query2	0.09	0.04	0.05
query3	0.27	0.07	0.09
query4	1.62	0.09	0.09
query5	0.23	0.25	0.26
query6	1.17	0.69	0.68
query7	0.04	0.01	0.01
query8	0.05	0.04	0.04
query9	0.40	0.33	0.35
query10	0.58	0.59	0.61
query11	0.20	0.15	0.14
query12	0.18	0.15	0.14
query13	0.48	0.49	0.49
query14	1.02	1.01	1.01
query15	0.64	0.61	0.60
query16	0.35	0.34	0.36
query17	1.12	1.14	1.15
query18	0.22	0.19	0.20
query19	2.15	1.97	2.00
query20	0.02	0.01	0.02
query21	15.47	0.18	0.13
query22	4.92	0.05	0.06
query23	16.22	0.32	0.12
query24	3.19	0.62	0.41
query25	0.11	0.06	0.05
query26	1.17	0.20	0.15
query27	0.05	0.04	0.04
query28	3.75	0.75	0.37
query29	12.51	4.05	3.21
query30	0.28	0.12	0.10
query31	2.77	0.55	0.32
query32	3.22	0.60	0.49
query33	3.24	3.14	3.22
query34	15.36	3.98	3.40
query35	3.35	3.37	3.33
query36	0.54	0.43	0.41
query37	0.09	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.03	0.04
query40	0.18	0.14	0.15
query41	0.09	0.03	0.03
query42	0.04	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 97.49 s
Total hot run time: 24.19 s

hello-stephen · 2026-10-21T15:02:57Z

FE Regression Coverage Report

Increment line coverage 100% (0/0) 🎉
Increment coverage report
Complete coverage report

### What problem does this PR solve? Issue Number: N/A Related PR: N/A Problem Summary: Doris currently uses Paimon 1.4.2. Upgrade the shared SDK version to 2.0.0 so the FE connector and BE JNI scanner continue to serialize and deserialize with the same SDK. Adapt test fixtures to the new Table branch API, Snapshot operation parameter, and grouped format-file splits. Update integer aggregation tests to expect the checked overflow behavior introduced by Paimon 2.0.0, while preserving the existing Rust reader eligibility rules and the common/S3 compatibility patches. ### Release note Upgrade the Paimon Java SDK from 1.4.2 to 2.0.0. Integer sum and product aggregation now report overflow instead of returning wrapped values, following Paimon 2.0.0 behavior. ### Check List (For Author) - Test: Unit tests through run-fe-ut.sh with a temporary, restored Paimon-only module scope: FE connector 836 passed and 1 live-connectivity test skipped; BE scanner 144 passed. Maven validate/Checkstyle passed for the Paimon modules and their dependencies. Verified that FE/BE test SDK artifacts are version 2.0.0 and that S3 shade removes outer multi-release classes while retaining 1,227 embedded classes. Full build.sh build was blocked by missing LLVM 20 in the local third-party environment; external-cluster regression tests were not run. - Behavior changed: Yes, use Paimon 2.0.0 SDK behavior, including checked integer aggregation overflow. - Does this need documentation: No

### What problem does this PR solve? Issue Number: N/A Related PR: apache#69343 Problem Summary: Upgrade the shared FE connector and BE scanner SDK pin from 2.0.0 to Paimon 2.1.0. Remove the upstream-deleted file-based privilege wrappers and their reflection-based reconstruction. Update filesystem schema and snapshot fixtures, verify default and checked integer overflow modes, and cover corrected pre-epoch date casts and query-auth file-statistics restrictions. Match scanner test JVM access to the existing BE java.nio opening required by Arrow Variant buffer views. ### Release note Upgrade the Paimon Java SDK from 1.4.2 to 2.1.0. The SDK removes legacy file-based privileges, rejects raw file statistics on query-auth tables, corrects pre-epoch timestamp-to-date casts, and supports per-field checked integer aggregation while preserving wrapping overflow by default. ### Check List (For Author) - Test: Unit Test / Manual test - FE connector: 831 passed, 1 live-connectivity test skipped. - BE scanner: 142 passed. S3 shade: 1 passed. - Maven validate/Checkstyle and plugin packaging passed. - Checked scanner duplicate classes, SPI isolation, dependency closure, and API stamps: 162 JARs, no failures. - Verified matching 2.1.0 SDKs and byte-for-byte S3 embedded runtime. - Full build blocked by missing local Paimon Rust third-party artifacts; native BE and cluster regression tests were not run. - Behavior changed: Yes, adopts the Paimon 2.1.0 behavior described above. - Does this need documentation: No, upstream changes are in the release note.

@incr

### What problem does this PR solve? Issue Number: N/A Related PR: apache#69343 Problem Summary: Paimon 2.1 adds file_indexes, whose backend reader re-plans files after Doris strips its catalog loader, and latest-delta, which Doris would incorrectly resolve to a full snapshot. Pin file_indexes to the catalog-published snapshot and reject the unimplemented delta OPTIONS mode with guidance to use explicit @incr ranges. Add persisted tests for both cases and for nullable primary-key metadata, serialization, and NULL-key deduplication across commits. ### Release note Fix catalog snapshot visibility for Paimon file_indexes and explicitly reject unsupported latest-delta OPTIONS reads instead of reading a full snapshot. ### Check List (For Author) - Test: Unit Test / Manual test: run-fe-ut.sh with temporary Paimon module scope, Checkstyle enabled, and install packaging; 834 passed and 1 live-connectivity test skipped. FE plugin ZIP produced. The runner was restored. Native BE/Rust and cluster regressions were not run because the local third-party Paimon native library/header are missing. - Behavior changed: Yes, pin new system-table reads and reject an unsupported startup mode rather than returning incorrect rows. - Does this need documentation: No, compatibility impact is recorded in the PR description and release note.

### What problem does this PR solve? Issue Number: N/A Related PR: apache#69343 Problem Summary: Exercise the upgraded SDK in a real Doris cluster. Verify nullable primary-key deduplication through JNI and a profile-confirmed Rust reader, bitmap file-index metadata, rejection of unsupported latest-delta OPTIONS, and wrapping versus checked sum overflow. Generate the expected output with the regression runner and verify it in a normal run. Update the oversized-write error assertion to Paimon 2.1's diagnostic while retaining the existing row and snapshot atomicity checks. ### Release note None ### Check List (For Author) - Test: Regression test / Unit Test / Manual test - New compatibility suite: generated output and normal replay passed. - Read compatibility matrix: 13 suites passed. - Metadata and transaction matrix: 6 suites passed. - Paimon ASAN native UT: 108 tests in 14 suites passed. - Java scanner: 142 tests passed; S3 shade: 1 test passed. - Native BE ASAN build and deployed plugin checks passed. - Manual backward read of 1.4.2 tables through JNI and Rust matched the Spark oracle. - Behavior changed: No (test coverage and updated SDK diagnostic assertion only) - Does this need documentation: No

### What problem does this PR solve? Issue Number: N/A Related PR: apache#69343 Problem Summary: Paimon 2.1 file_indexes defers catalog query checks until the BE read, where its catalog loader has already been removed. The second serialized system-source payload also lost the catalog snapshot bound when JNI rebuilt the wrapper. Transfer the deferred query check to the FE and pin the transported source. Adapt timestamp-policy projection validation and the four-argument authorization callback to the 2.1 reader API, and serialize real JSON policy definitions in scanner tests. Regenerate the affected snapshots system-table output for the added SDK columns. ### Release note Keep catalog query authorization and catalog-published snapshots when reading Paimon file_indexes, and preserve timestamp-policy filtering and masking with SDK 2.1. ### Check List (For Author) - Test: Unit Test: FE connector 844 passed/1 skipped, Java scanner 175, native Paimon/JNI/Rust 144, S3 shade 1. Regression test: 51 Paimon suites passed; opt-in KEY_DYNAMIC stress skipped. Manual test: ASAN BE and FE/Java builds, plugin layout checks, legacy 1.4.2 data reads, CI failure triage. - Behavior changed: Yes, preserve query authorization, snapshot bounds and timestamp policy enforcement with Paimon 2.1 - Does this need documentation: No

suxiaogang223 · 2026-10-22T08:16:16Z

run buildall

suxiaogang223 · 2026-10-22T08:16:42Z

/review

github-actions

Reviewed all 39 changed paths at 9337b7c using static analysis. Request changes for one new P1: nullable primary keys newly supported by SDK 2.1 invalidate Doris MERGE's target-presence test, silently dropping matched UPDATE/DELETE operations on NULL keys.

The two prior file_indexes reports are addressed at this head: deferred catalog query checks run on the FE before loader removal (4335228948), and the transported system-source payload receives the snapshot pin before JNI rebuild (4335228960). They are not reposted, and no existing P0/P1 finding remains applicable. No additional user focus was supplied.

Critical checkpoint conclusions:

Goal and proof: the shared SDK upgrade and API adaptations are covered by focused tests, but the nullable-key tests exercise insert/read/deduplication rather than Doris MERGE; the inline finding identifies the missing SQL semantic case.
Scope and reuse: changes stay within the Paimon SDK migration, directly affected wrappers/callbacks, and compatibility tests; existing branch and runtime-option helpers are reused.
Concurrency and locking: changed production paths add no threads, locks, or new shared mutable state. No introduced lock-order or heavy-under-lock problem was found.
Lifecycle and initialization: source preparation retains the authentication/classloader scope; reader/test resources close on the examined exit paths. No new resource owner or static initialization dependency was introduced.
Configuration: no Doris configuration key was added. The common OPTIONS validation rejects latest-delta with an actionable error before snapshot pinning can widen that request.
Compatibility: both Java endpoints inherit the shared SDK version. Legacy file-based privileges are removed upstream and require migration. Mixed 1.4.2/2.1 Java table/commit serialization is not generally compatible; coordinated FE/BE SDK deployment remains the PR's documented requirement.
Shared definitions: no Thrift/Protobuf or connector SPI surface changed; release-branch field-ID and SPI-baseline checks are not applicable.
Parallel paths: traced FE metadata/schema binding, both serialized system-table payloads, JNI reconstruction, fallback/chain branches, Rust selection, and nullable fixed-bucket routing. No additional introduced issue was substantiated in these paths.
Conditional behavior: file_indexes joins the existing deferred-check/snapshot paths; selected masks and logical timestamp precision are preserved through the new SDK callback. Removed privilege-specific branches correspond to removed SDK classes.
Test coverage: inspected new serialized-source/JNI controls, allow/deny cases, timestamp/nested/hidden-field policy fixtures, nullable-key roundtrips and reader profiles, overflow modes, delta rejection, and failed-write atomicity checks. NULL-key MERGE UPDATE/DELETE needs coverage with the fix.
Changed results: appended nullable snapshot fields and DESC rows agree with the SDK schema; obsolete golden blocks have no remaining qt callers; new ordered result values and the revised write-error substring agree with the inspected mechanisms. Golden generation provenance was not independently verified.
Observability: changed failure paths retain exceptions and existing scanner diagnostics. No newly introduced critical stage requiring additional telemetry was identified.
Transactions and persistence: no Doris EditLog, replay, transaction ownership, or failover state machine was changed. Commit framing remains aligned for matching SDK deployments; mixed-SDK limits are stated above.
Data writes and atomicity: the existing failed-insert regression retains row/snapshot checks. The new P1 occurs before writing and can silently omit an intended row change; it requires correction or explicit rejection.
FE/BE transport: no Doris protocol field was added. The source payload now preserves the catalog pin; both endpoints' Java object serialization depends on matching SDK jars.
Performance and memory: no new native allocation, cache owner, executor, or unbounded hot-loop work was introduced in the changed code; no additional substantiated performance defect was found.
Other concerns and user focus: no additional introduced finding beyond the inline P1. There were no extra user focus points.

No builds, tests, or source edits were performed under this review contract. Runtime packaging, HMS/REST integration, native/Rust execution, and mixed-version rollout were not independently exercised. Convergence describes the completed static review process, not runtime certification.

Review converged after 2 round(s).

### What problem does this PR solve? Issue Number: N/A Related PR: apache#69343 Problem Summary: With Paimon 2.1 nullable primary keys, a NULL-safe MERGE can join a target row whose first primary key is NULL. Changelog branch selection then treated that row as absent and silently dropped matched UPDATE and DELETE changes. Treat every surviving INNER JOIN row as matched for MATCHED-only MERGE. For MERGE with INSERT branches, project a non-null presence marker on the target before the outer join and inspect its null-extended output by ExprId. Keep private marker slots outside the user expression-binding scope so same-named source columns retain their values. Add nullable single/composite-key SQL coverage, branch and cardinality checks, marker-name collision coverage, and focused planner unit tests. ### Release note Fix matched MERGE UPDATE and DELETE silently omitting Paimon rows with NULL primary keys under NULL-safe equality. ### Check List (For Author) - Test: Unit Test: 38 related FE tests passed. Regression test: 9 Paimon suites passed, including nullable-key MERGE with generated/replayed expected output and all six existing write suites containing MERGE. Manual test: FE build and before/after NULL-key UPDATE/DELETE reproduction. - Behavior changed: Yes, matched NULL-key rows are updated/deleted and target presence is independent of key values - Does this need documentation: No

suxiaogang223 · 2026-10-22T10:52:02Z

/review

github-actions

Reviewed all 43 changed paths at b6ad0f2. No new actionable findings. Review converged in round 1: both complete reviewers and the separate risk reviewer returned NO_NEW_VALUABLE_FINDINGS on the unchanged empty inline set; all candidates and risk focuses have evidence-based dispositions. No additional user review focus was supplied.

The earlier nullable-key MERGE P1, file_indexes deferred-check P1, and JNI source snapshot P2 are resolved on this head. MATCHED-only joins use their INNER JOIN semantics; INSERT-capable plans use the null-extended target marker by ExprId without exposing it to user expression binding. file_indexes performs the deferred catalog check before loader removal, and its separately transported source carries the catalog snapshot pin through actual JNI reconstruction. No existing P0/P1 remains to carry as a blocking comment ID.

Inspected flows: catalog/cache -> bound schema and scan selectors -> FE split planning plus serialized table/source -> JNI reconstruction and policy callbacks or conservative Rust selection; MERGE join -> branch selection -> row-change projection -> cardinality assertions -> existing commit/rollback boundary.

Critical checkpoint conclusions:

Checkpoint	Conclusion
Goal and evidence	Shared SDK 2.1 upgrade and affected adapters are consistent on static inspection; targeted unit and SQL tests cover the changed behavior.
Scope and reuse	Changes stay focused on SDK adaptations, file_indexes transport, unsupported delta startup, and nullable target presence; existing branch/cardinality and connector helpers are reused.
Concurrency and locks	New state is query/read local; no new worker, shared mutable state, lock acquisition order, or metadata-lock IO region was found.
Lifecycle and initialization	Reader wrappers retain normal close propagation; cache construction cleanup, catalog-loader removal, fallback ownership and static initialization remain consistent.
Configuration	No new global configuration; existing option precedence and branch-local caps remain. latest-delta fails explicitly before full-snapshot conversion.
Compatibility	FE/scanner share paimon.version. SDK schema/Snapshot fixtures, serialized rule definitions, and callback signatures are adapted. FE and BE Java extensions must deploy the same SDK.
Thrift/protobuf	No changed definitions or field IDs; release-branch ID comparison is inapplicable.
Parallel paths	Checked data/system scans, source-payload reconstruction, REST/filesystem schema provenance, explicit selectors, fallback/chain boundaries, and native/JNI/Rust selection.
Conditional checks	New checks have concrete triggers: unsupported delta semantics, join shape and lost marker. Nullable target detection follows actual join output nullability.
Test coverage	Includes NULL single/composite keys, ordinary/NULL-safe matches, branch priority, insert-only paths, marker collisions, duplicate failures, actual source pinning, denied deferred checks, and overflow modes.
Changed results	New golden labels match ordered SQL calls; nullable rows, overflow values and added snapshot fields are consistent. Removed stale labels have no corresponding suite calls.
Observability	Unsupported modes and invariant failures remain explicit; existing scan profiles and transaction errors cover the changed paths.
Persistence and failover	No Doris EditLog/replay or persisted transaction format changes; existing schema/snapshot coordinates and rollback paths remain.
Writes and atomicity	Private marker stays outside final sink output. Cardinality checks group full key tuples, including NULLs; failure still prevents commit and follows rollback.
FE/BE variables	Existing transport keys are reused; both system-table payload paths and the SDK policy callback boundary were checked.
Performance and memory	One small constant marker projection is added only for INSERT-capable MERGE. No new unbounded allocation, executor growth or asymptotic regression was established.
Other concerns	No unresolved PR-introduced candidate remains after the missed-issue rechecks and final sweep.

Validation limits: this invocation was static only; no builds or tests were run, and golden generation was not independently executed. Mixed 1.4.2/2.1.0 rolling deployments and all newly introduced upstream layouts are not validated. Removal of upstream file-based privileges requires the migration already noted in the PR. Existing two-phase system-table metadata timing and the independent constant-source composite NULL-safe optimizer limitation remain understood limitations, not new findings from this change.

Review converged after 1 round(s).

suxiaogang223 · 2026-10-22T12:18:03Z

run buildall

hello-stephen · 2026-10-22T14:34:18Z

TPC-H: Total hot run time: 28278 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b6ad0f2571d7da3bfa9c53a3c24484271566de45, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17590	5096	5105	5096
q2	2139	320	251	251
q3	10245	1364	809	809
q4	4681	458	331	331
q5	7426	802	529	529
q6	178	181	151	151
q7	745	792	593	593
q8	9263	1336	1087	1087
q9	5583	4154	4189	4154
q10	6841	1268	1020	1020
q11	429	242	221	221
q12	632	389	276	276
q13	18084	3490	3024	3024
q14	270	268	253	253
q15	q16	730	706	656	656
q17	1169	745	562	562
q18	6990	6124	6111	6111
q19	1093	955	572	572
q20	372	334	229	229
q21	4748	2102	2276	2102
q22	338	291	251	251
Total cold run time: 99546 ms
Total hot run time: 28278 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5572	5594	5568	5568
q2	680	583	521	521
q3	5205	5395	4938	4938
q4	1986	2132	1542	1542
q5	5275	5150	5430	5150
q6	255	198	150	150
q7	2342	2060	1859	1859
q8	3043	2689	2590	2590
q9	8117	7934	7974	7934
q10	3714	3642	3213	3213
q11	764	471	410	410
q12	713	706	537	537
q13	3554	4007	3228	3228
q14	324	334	307	307
q15	q16	742	712	680	680
q17	8392	7448	6935	6935
q18	13027	12157	12821	12157
q19	904	782	765	765
q20	2125	2138	1907	1907
q21	5584	4553	4666	4553
q22	498	435	414	414
Total cold run time: 72816 ms
Total hot run time: 65358 ms

hello-stephen · 2026-10-22T14:44:03Z

TPC-DS: Total hot run time: 150264 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b6ad0f2571d7da3bfa9c53a3c24484271566de45, data reload: false

query5	4313	594	442	442
query6	416	223	192	192
query7	4813	428	224	224
query8	318	168	147	147
query9	8789	3095	3108	3095
query10	491	302	255	255
query11	6063	3854	3611	3611
query12	141	91	84	84
query13	1249	437	322	322
query14	6112	3898	3612	3612
query14_1	3412	3396	3401	3396
query15	206	205	184	184
query16	929	432	418	418
query17	854	649	539	539
query18	2421	418	304	304
query19	210	171	138	138
query20	84	84	84	84
query21	206	128	105	105
query22	12969	12993	12861	12861
query23	13441	12785	12329	12329
query23_1	12442	12465	12491	12465
query24	7374	814	462	462
query24_1	462	475	476	475
query25	535	419	363	363
query26	1263	262	138	138
query27	2764	436	287	287
query28	4603	1965	1956	1956
query29	1599	609	445	445
query30	314	221	183	183
query31	1025	941	808	808
query32	148	97	90	90
query33	528	308	247	247
query34	897	849	499	499
query35	740	778	649	649
query36	803	818	744	744
query37	117	97	88	88
query38	1779	1711	1663	1663
query39	807	787	743	743
query39_1	727	734	739	734
query40	221	128	97	97
query41	71	69	69	69
query42	89	83	84	83
query43	357	365	310	310
query44	1293	686	704	686
query45	181	177	166	166
query46	814	962	565	565
query47	3048	2844	2939	2844
query48	287	305	213	213
query49	574	406	286	286
query50	659	265	201	201
query51	10249	10572	10306	10306
query52	92	89	69	69
query53	185	201	150	150
query54	229	201	173	173
query55	68	65	61	61
query56	228	224	197	197
query57	1656	1612	1534	1534
query58	283	258	257	257
query59	2160	2157	1978	1978
query60	278	239	220	220
query61	147	142	143	142
query62	386	325	286	286
query63	183	153	160	153
query64	2748	943	756	756
query65	3294	3245	3260	3245
query66	1788	408	317	317
query67	20171	19995	19822	19822
query68	2919	1009	597	597
query69	405	286	233	233
query70	862	793	718	718
query71	293	233	217	217
query72	2703	2602	2295	2295
query73	510	513	288	288
query74	4599	4488	4319	4319
query75	2321	2273	1904	1904
query76	2317	1004	576	576
query77	336	373	268	268
query78	9785	9736	9165	9165
query79	955	888	511	511
query80	730	422	326	326
query81	536	323	279	279
query82	564	128	104	104
query83	259	215	190	190
query84	326	126	93	93
query85	904	409	370	370
query86	353	233	239	233
query87	1995	1995	1836	1836
query88	3605	2650	2631	2631
query89	326	291	241	241
query90	1753	163	162	162
query91	171	152	129	129
query92	94	77	86	77
query93	990	995	568	568
query94	496	303	270	270
query95	566	322	377	322
query96	669	477	232	232
query97	2478	2418	2316	2316
query98	156	154	146	146
query99	763	768	667	667
Total cold run time: 233606 ms
Total hot run time: 150264 ms

hello-stephen · 2026-10-22T14:48:57Z

ClickBench: Total hot run time: 24.59 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b6ad0f2571d7da3bfa9c53a3c24484271566de45, data reload: false

query1	0.01	0.01	0.01
query2	0.10	0.05	0.04
query3	0.26	0.09	0.09
query4	1.60	0.10	0.09
query5	0.24	0.24	0.24
query6	1.17	0.69	0.68
query7	0.04	0.01	0.00
query8	0.05	0.04	0.04
query9	0.41	0.33	0.33
query10	0.58	0.58	0.62
query11	0.20	0.14	0.14
query12	0.18	0.16	0.15
query13	0.50	0.49	0.49
query14	1.04	1.01	1.01
query15	0.63	0.61	0.62
query16	0.34	0.35	0.35
query17	1.15	1.19	1.17
query18	0.20	0.19	0.19
query19	2.15	2.01	2.03
query20	0.02	0.01	0.02
query21	15.50	0.18	0.13
query22	4.95	0.06	0.05
query23	16.27	0.32	0.12
query24	2.99	0.62	0.68
query25	0.13	0.05	0.04
query26	1.58	0.21	0.16
query27	0.05	0.05	0.03
query28	3.85	0.72	0.36
query29	12.48	4.22	3.25
query30	0.28	0.13	0.11
query31	2.78	0.56	0.31
query32	3.21	0.60	0.52
query33	3.25	3.16	3.15
query34	15.32	3.99	3.38
query35	3.37	3.33	3.38
query36	0.56	0.44	0.44
query37	0.08	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.04	0.03
query40	0.18	0.14	0.14
query41	0.09	0.04	0.03
query42	0.03	0.03	0.03
query43	0.05	0.03	0.04
Total cold run time: 97.96 s
Total hot run time: 24.59 s

hello-stephen · 2026-10-22T15:36:15Z

FE UT Coverage Report

Increment line coverage 80.00% (12/15) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-10-22T17:13:20Z

FE Regression Coverage Report

Increment line coverage 60.00% (12/20) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-10-22T19:14:09Z

TPC-H: Total hot run time: 28678 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b6ad0f2571d7da3bfa9c53a3c24484271566de45, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17600	5191	5089	5089
q2	2094	326	277	277
q3	10205	1387	857	857
q4	4676	467	333	333
q5	7449	834	543	543
q6	189	190	161	161
q7	756	803	597	597
q8	9288	1366	1065	1065
q9	5457	4226	4213	4213
q10	6838	1290	1024	1024
q11	444	249	230	230
q12	648	408	286	286
q13	18081	3534	3032	3032
q14	275	280	245	245
q15	q16	726	726	683	683
q17	1196	759	581	581
q18	7065	6170	6122	6122
q19	1107	955	618	618
q20	389	340	230	230
q21	5096	2231	2358	2231
q22	347	314	261	261
Total cold run time: 99926 ms
Total hot run time: 28678 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5654	5599	5625	5599
q2	665	573	526	526
q3	5227	5403	4905	4905
q4	2027	2165	1618	1618
q5	5227	5142	5670	5142
q6	274	199	157	157
q7	2370	2113	1915	1915
q8	3103	2917	2750	2750
q9	8116	8123	7904	7904
q10	3753	3666	3288	3288
q11	615	434	403	403
q12	735	696	541	541
q13	3538	3950	3236	3236
q14	312	310	282	282
q15	q16	699	711	630	630
q17	8255	7521	7273	7273
q18	13092	12274	12867	12274
q19	951	840	846	840
q20	2158	2192	1927	1927
q21	5752	4791	4889	4791
q22	497	460	420	420
Total cold run time: 73020 ms
Total hot run time: 66421 ms

hello-stephen · 2026-10-22T19:23:59Z

TPC-DS: Total hot run time: 150628 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b6ad0f2571d7da3bfa9c53a3c24484271566de45, data reload: false

query5	4316	588	446	446
query6	431	207	186	186
query7	4816	450	226	226
query8	329	156	156	156
query9	8769	3152	3129	3129
query10	469	309	261	261
query11	5999	3850	3578	3578
query12	147	99	85	85
query13	1259	445	321	321
query14	6144	3965	3668	3668
query14_1	3457	3694	3519	3519
query15	207	202	180	180
query16	948	451	435	435
query17	834	650	528	528
query18	2411	410	300	300
query19	185	168	126	126
query20	87	79	81	79
query21	208	123	103	103
query22	13165	12916	12808	12808
query23	13516	12848	12469	12469
query23_1	12526	12474	12606	12474
query24	7355	814	455	455
query24_1	469	471	470	470
query25	516	399	342	342
query26	1294	258	136	136
query27	2773	446	276	276
query28	4608	1950	1932	1932
query29	1573	582	431	431
query30	305	227	183	183
query31	1017	927	833	833
query32	144	93	97	93
query33	537	303	240	240
query34	898	855	505	505
query35	748	773	647	647
query36	816	817	738	738
query37	128	96	85	85
query38	1800	1731	1671	1671
query39	801	831	746	746
query39_1	736	730	748	730
query40	222	124	99	99
query41	70	68	68	68
query42	86	85	89	85
query43	356	364	316	316
query44	1393	693	684	684
query45	186	181	162	162
query46	827	966	574	574
query47	2941	2945	2452	2452
query48	307	315	218	218
query49	587	416	305	305
query50	674	277	213	213
query51	10541	10702	10250	10250
query52	79	81	73	73
query53	190	214	159	159
query54	241	213	197	197
query55	78	69	64	64
query56	238	241	223	223
query57	1649	1621	1597	1597
query58	294	271	274	271
query59	2134	2112	1958	1958
query60	303	273	245	245
query61	173	166	169	166
query62	390	335	285	285
query63	193	173	174	173
query64	2881	1032	747	747
query65	3340	3277	3257	3257
query66	1794	426	303	303
query67	20506	20150	20013	20013
query68	2919	1029	607	607
query69	381	280	251	251
query70	838	781	779	779
query71	289	236	209	209
query72	2732	2576	2289	2289
query73	509	536	295	295
query74	4660	4522	4357	4357
query75	2359	2281	1900	1900
query76	2311	1018	608	608
query77	346	392	270	270
query78	9982	9741	9161	9161
query79	1744	840	532	532
query80	1227	427	351	351
query81	590	327	274	274
query82	576	136	109	109
query83	304	194	181	181
query84	307	122	97	97
query85	860	428	379	379
query86	395	246	237	237
query87	2019	1937	1818	1818
query88	3745	2697	2677	2677
query89	334	278	248	248
query90	1859	167	169	167
query91	158	153	131	131
query92	98	88	92	88
query93	1125	1023	578	578
query94	630	294	268	268
query95	581	402	308	308
query96	683	490	231	231
query97	2451	2425	2304	2304
query98	174	159	162	159
query99	773	759	650	650
Total cold run time: 237116 ms
Total hot run time: 150628 ms

hello-stephen · 2026-10-22T19:28:52Z

ClickBench: Total hot run time: 24.43 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b6ad0f2571d7da3bfa9c53a3c24484271566de45, data reload: false

query1	0.01	0.01	0.00
query2	0.09	0.05	0.05
query3	0.26	0.08	0.09
query4	1.61	0.08	0.11
query5	0.23	0.25	0.26
query6	1.18	0.68	0.67
query7	0.04	0.01	0.01
query8	0.05	0.03	0.03
query9	0.40	0.36	0.36
query10	0.59	0.61	0.60
query11	0.20	0.14	0.14
query12	0.18	0.15	0.15
query13	0.48	0.51	0.50
query14	1.04	1.02	1.03
query15	0.64	0.61	0.61
query16	0.35	0.36	0.35
query17	1.14	1.14	1.16
query18	0.22	0.19	0.20
query19	2.06	1.95	1.92
query20	0.02	0.02	0.01
query21	15.50	0.21	0.13
query22	4.78	0.05	0.05
query23	16.27	0.31	0.12
query24	2.85	0.65	0.43
query25	0.13	0.05	0.06
query26	1.55	0.22	0.16
query27	0.05	0.04	0.05
query28	3.38	0.72	0.36
query29	12.49	4.11	3.32
query30	0.28	0.13	0.10
query31	2.78	0.56	0.32
query32	3.21	0.60	0.51
query33	3.26	3.34	3.18
query34	15.37	3.95	3.40
query35	3.35	3.34	3.32
query36	0.56	0.44	0.44
query37	0.09	0.06	0.07
query38	0.05	0.04	0.03
query39	0.04	0.03	0.03
query40	0.17	0.14	0.15
query41	0.08	0.03	0.02
query42	0.04	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 97.11 s
Total hot run time: 24.43 s

hello-stephen · 2026-10-22T21:03:36Z

FE Regression Coverage Report

Increment line coverage 80.00% (12/15) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-10-22T21:40:30Z

FE Regression Coverage Report

Increment line coverage 80.00% (12/15) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-10-22T22:16:52Z

FE Regression Coverage Report

Increment line coverage 80.00% (12/15) 🎉
Increment coverage report
Complete coverage report

dismiss

github-actions · 2026-10-23T05:12:07Z

PR approved by at least one committer and no changes requested.

…ble source (#69569) ### What problem does this PR solve? Issue Number: None Related PR: #68897, #69343 Problem Summary: **In short.** This PR only adds tests. Some Paimon system tables are planned on the BE. For them, the JNI scanner rebuilds the system table from the source table that the FE sends in `doris.serialized-system-source`. In a version-managed catalog, that payload must be pinned to the snapshot the catalog publishes. #69343 added this pin. This PR adds the two cases its tests do not cover: a `scan.fallback-branch` pair, and an explicit relation snapshot that must still take precedence over the pin. **Background.** Six Paimon system tables are planned on the BE: `$files`, `$file_indexes`, `$partitions`, `$manifests`, `$statistics` and `$table_indexes` (`PaimonScanParams#resolvesSnapshotOnBackend`). For these tables the FE sends two payloads: - `serialized_table` is built by `tableForBackend`. - `doris.serialized-system-source` was added in #66247 so that a smaller BE can cap the manifest parallelism. The FE always sends it when the system table has a source. When it is present, `PaimonJniScanner#applyBackendManifestParallelism` rebuilds the system table from this payload instead of from `serialized_table`. A version-managed catalog (REST) owns the committed snapshot pointer. Once the catalog loader is removed, Paimon falls back to listing the snapshot directory, and that directory can already contain a snapshot the catalog has not published. #65867 therefore pins `scan.snapshot-id` to the catalog's pointer (`PaimonScanPlanProvider#pinCatalogSnapshot`). **The problem, and what it cost.** Before #69343, only `serialized_table` carried that pin. The side channel shipped the source unpinned, and the BE rebuilt from the side channel. So in a REST catalog these six system tables were planned on the newest snapshot in storage rather than the published one, on every branch. Inside the publication window, `$partitions` could list partitions of a snapshot the catalog had not published yet. The review of #68897 pointed this out for chain tables. #69343 (the Paimon 2.1.0 upgrade) now prepares the side-channel source in the same order as `tableForBackend` (`prepareSystemSource`): 1. Pin the catalog snapshot, branch by branch. 2. Apply the relation's `@options` over the pin. 3. Drop the catalog loader. Its tests cover `$file_indexes` on a single-branch table, on the FE and through the JNI rebuild. **How this PR fixes it.** Two FE unit tests in `PaimonBackendBoundTableTest` read the side channel out of `paimon.options_json`: - `systemSourcePinsEachFallbackBranchToItsOwnCatalogSnapshot` builds `$partitions` over a fallback pair whose two catalogs point at snapshots 2 and 1. Each branch of the transported source carries its own pointer and no catalog loader. A chain table nests the same pairs. - `relationSnapshotOutranksTheCatalogPinOnTheSystemSource` uses a catalog that points at snapshot 2 and `@options('scan.snapshot-id'='1')`. The transported source says 1. Paimon 2.1.0 still has the two limitations documented on `pinCatalogSnapshot`: - `$files` emits its marker splits from `SnapshotReader#partitions`. - `$snapshots` and `$buckets` read without `scan.snapshot-id`. So that javadoc stays as written. **Results.** | `prepareSystemSource` | `PaimonBackendBoundTableTest` | |---|---| | as on master (pin, then `@options`) | 38 / 38 pass | | without the pin | the new fallback case and the existing `$file_indexes` case fail | | pin applied after the relation options | the new relation-snapshot case fails (`expected: <1> but was: <2>`) | All 847 tests of fe-connector-paimon pass, with 1 skipped as before, and checkstyle passes. No regression suite is added: it would need a version-managed (REST) Paimon catalog whose pointer lags the snapshot directory, and the docker fixtures do not provide one.

@incr

### What problem does this PR solve? Related PR: apache#69343, apache#69493 Problem Summary: Bring the Paimon 2.1 upgrade on branch-4.2 in line with master's compatibility handling. A nullable primary key cannot identify whether a MERGE target row is present: MATCHED-only NULL-safe joins lost valid updates/deletes, while outer joins need a separate presence marker. Use the existing join shape and bind user expressions before exposing the private marker. Reject latest-delta OPTIONS before snapshot pinning can silently turn a delta read into a full read. Match master after removal of the legacy privilege delegates and carry over SDK compatibility tests, file-index snapshot coverage and generated regression results. ### Release note Fix Paimon MERGE matched UPDATE/DELETE for nullable primary keys. Reject scan.mode=latest-delta in OPTIONS with guidance to use explicit @incr ranges. ### Check List (For Author) - Test: Added/adapted FE and JNI unit tests and master's external regression suites. Static self-review and whitespace checks completed (preserving generated result-file separators). No local build, Checkstyle, unit tests or regression execution, as requested. Result files are copied from master apache#69343 rather than generated locally. - Behavior changed: Yes; preserve NULL-key MERGE matches and reject an unsupported delta mode instead of reading a full snapshot. - Does this need documentation: No; the rejection provides the supported alternative.

### What problem does this PR solve? Related PR: apache#69886, apache#70075, apache#69343 Problem Summary: Master now provides paimon-rust v0.5.0-rc1 and Java Paimon 2.1.0. Rebase the writer on master instead of carrying separate dependency upgrades. Reuse the upstream Java v14 serializer with bounded accumulated output, remove the v11 and legacy DataFileMeta encoding patch, and regenerate the real two-writer interop fixture. Reject old protocol requests before writer admission. Preserve master's vector-search field 42 and use optional field 43 for the shared descriptor; retain master's SPI additions and bump the write diagnostics surface to API 30. The previous BE UT crash was caused by a validation test bypassing init() and reading an uninitialized scan-params pointer. Explicitly initialize its missing-descriptor input. ### Release note Rust Paimon writes use the Java v14 commit protocol matching master's Paimon 2.1.0 SDK and inherit master's paimon-rust dependency version. ### Check List (For Author) - Test: Third-party build with Rust 1.94.0; real C FFI append, codec bounds, version rejection, message ranges and resource release; FE/Java build with Checkstyle; Paimon connector 904 tests (one skip); connector SPI 152 tests; six Paimon JNI static layout checks; Thrift generation/field audit; clang-format 16, header hygiene, shell syntax and whitespace checks. JNI Paimon 186 and FE integration 36 tests passed. BE compilation, native tests, clang-tidy against the new build and SQL regression are deferred at the user's request to leave resources for another PR. - Behavior changed: Yes; Rust commit messages now use v14 instead of v11. - Does this need documentation: Yes; existing feature documentation remains pending with the draft PR.

### What problem does this PR solve? Issue Number: N/A Related PR: apache#70103, apache#69343 Problem Summary: The backported Paimon merge planner adds its target-presence marker to the left join input, but PaimonMergeCommand still places the source on the left. Unmatched source rows therefore select MATCHED and write NULL target keys instead of inserting. Reuse MergeUtils to keep the target on the left and preserve the source with RIGHT OUTER JOIN when INSERT is present. Add command-level tests for both join layouts. Keep logical splits when the compatibility case asserts Rust execution and use the 4.2 MERGE diagnostics in nullable-key error expectations. ### Release note Fix Paimon MERGE INTO incorrectly classifying unmatched source rows as matched on branch-4.2, including NULL key writes to non-nullable primary keys. ### Check List (For Author) - Test: FE/JNI build and Checkstyle; 490 FE unit tests passed, 2 environment-gated tests skipped; 47 regression framework unit tests passed; all 6 affected Paimon regression suites passed against existing golden outputs using the PR's successful CI ASAN BE and the repository's Spark/Paimon fixtures. - Behavior changed: Yes, unmatched Paimon MERGE source rows select INSERT. - Does this need documentation: No, restores the intended MERGE semantics.

suxiaogang223 requested review from 924060929, CalvinKirs and morningman as code owners October 20, 2026 10:39

suxiaogang223 changed the title ~~[improvement](fe) Upgrade Paimon to 2.0.0~~ [improvement](fe) Upgrade Paimon to 2.1.0 Oct 21, 2026

github-actions Bot requested changes Oct 21, 2026

View reviewed changes

Comment thread fe/pom.xml Resolved

Comment thread ...or/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanParams.java Resolved

suxiaogang223 and others added 5 commits October 22, 2026 15:22

suxiaogang223 force-pushed the codex/upgrade-paimon-2.0 branch from 967cf6a to 9337b7c Compare October 22, 2026 08:14

github-actions Bot previously requested changes Oct 22, 2026

View reviewed changes

Comment thread fe/pom.xml Resolved

suxiaogang223 requested review from englefly, morrySnow and starocean999 as code owners October 22, 2026 10:49

github-actions Bot reviewed Oct 22, 2026

View reviewed changes

924060929 approved these changes Oct 23, 2026

View reviewed changes

CalvinKirs approved these changes Oct 23, 2026

View reviewed changes

suxiaogang223 merged commit 949a806 into apache:master Oct 23, 2026
46 of 47 checks passed

suxiaogang223 deleted the codex/upgrade-paimon-2.0 branch October 23, 2026 03:51

github-actions Bot added the approved Indicates a PR has been approved by one committer. label Oct 23, 2026

morningman mentioned this pull request Oct 24, 2026

[test](paimon) Cover the catalog snapshot pin on the Paimon system-table source #69569

Merged

3 of 16 tasks

suxiaogang223 mentioned this pull request Nov 3, 2026

branch-4.2: [improvement](fe) Upgrade Paimon to 2.1.0 #70103

Open

4 tasks

suxiaogang223 mentioned this pull request Nov 4, 2026

[feature](paimon) Add SQL and TVF ANN search with Paimon 2.1 #69493

Open

3 tasks done

suxiaogang223 mentioned this pull request Nov 4, 2026

[feature](paimon) Support writes using paimon-rust #69886

Open

5 of 12 tasks

Conversation

suxiaogang223 commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Paimon 2.1 feature impact on Doris

Release note

Check List (For Author)

Self-review

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Oct 20, 2026

Uh oh!

suxiaogang223 commented Oct 20, 2026

Uh oh!

hello-stephen commented Oct 20, 2026

FE UT Coverage Report

Uh oh!

suxiaogang223 commented Oct 21, 2026

Uh oh!

suxiaogang223 commented Oct 21, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

hello-stephen commented Oct 21, 2026

Uh oh!

hello-stephen commented Oct 21, 2026

Uh oh!

hello-stephen commented Oct 21, 2026

Uh oh!

hello-stephen commented Oct 21, 2026

FE Regression Coverage Report

Uh oh!

suxiaogang223 commented Oct 22, 2026

Uh oh!

suxiaogang223 commented Oct 22, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

suxiaogang223 commented Oct 22, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

suxiaogang223 commented Oct 22, 2026

Uh oh!

hello-stephen commented Oct 22, 2026

Uh oh!

hello-stephen commented Oct 22, 2026

Uh oh!

hello-stephen commented Oct 22, 2026

Uh oh!

hello-stephen commented Oct 22, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Oct 22, 2026

FE Regression Coverage Report

Uh oh!

hello-stephen commented Oct 22, 2026

Uh oh!

hello-stephen commented Oct 22, 2026

Uh oh!

hello-stephen commented Oct 22, 2026

Uh oh!

hello-stephen commented Oct 22, 2026

FE Regression Coverage Report

Uh oh!

hello-stephen commented Oct 22, 2026

FE Regression Coverage Report

Uh oh!

hello-stephen commented Oct 22, 2026

FE Regression Coverage Report

Uh oh!

Uh oh!

github-actions Bot commented Oct 23, 2026

Uh oh!

suxiaogang223 commented Oct 20, 2026 •

edited

Loading