[core][flink][python] Support nullable primary keys - #9094

### What problem does this PR solve?

Issue Number: N/A

Related PR: N/A

Problem Summary:

Upgrade the Paimon Java SDK from 1.4.2 to 2.1.0, the latest stable
release published on October 12, 2026 ([upstream
release](https://github.com/apache/paimon/releases/tag/release-2.1.0)).
Pin the FE connector and BE JNI scanner to the same shared
`paimon.version`, since tables, splits, and commit messages cross the
FE/BE boundary using Paimon's serialization.

Adapt the connector to the removal of Paimon's file-based `Privilege*`
API in 2.1.0, removing its obsolete catalog/table wrappers and
reflection-based delegate reconstruction. Preserve catalog query
authorization, fallback/chain branch semantics, statement-bound schemas
and snapshots, and Doris metadata-cache governance. Update local test
fixtures to `FileSystemSchemaManager` and the extended `Snapshot`
constructor. Give scanner tests the `java.nio` access already enabled in
the BE JVM for Arrow Variant buffer views. Update the oversized-write
negative test to the SDK's revised diagnostic while retaining its
snapshot/row atomicity checks. Retain the branch/scan API adaptations
and grouped format-file split fixture required since 2.0. Verify both
the default wrapping and explicitly enabled checked integer-overflow
modes in 2.1, as well as the corrected pre-epoch timestamp-to-date cast.
Keep the existing Rust reader eligibility rules and common/S3
compatibility patches. Fix changelog MERGE target-presence detection for
nullable primary keys: matched-only plans use their INNER JOIN
semantics, while plans with INSERT branches use a pre-join marker. Keep
that private marker outside user expression resolution. Preserve catalog
query authorization for the new `$file_indexes` system table and pin
both serialized table payloads to the catalog-published snapshot before
stripping their catalog loaders. Adapt timestamp precision repair to the
SDK's authorization projection validation and four-argument policy
callback; use real JSON policy definitions in serialized scanner
fixtures. Reject `scan.mode=latest-delta` in Doris `@options`, because
the existing full-snapshot pinning would silently widen a delta scan;
explicit `@incr` ranges remain available.

The FE and BE Java extensions must be deployed with the same SDK
version. This change does not establish compatibility between mixed
1.4.2/2.1.0 SDKs during a rolling upgrade.

### Paimon 2.1 feature impact on Doris

| Upstream change | Doris impact and scope of this PR |
| --- | --- |
| [Nullable primary keys](apache/paimon#9094),
opt-in with `primary-key.nullable=true` | Existing write-column mapping
respects the SDK field nullability. Added a real-table test of nullable
key metadata, FE/BE table serialization, and NULL-key deduplication
across commits through the Java reader. The cluster regression
additionally checks NULL-key deduplication across commits through JNI,
and checks a single-file table through both JNI and the actual Rust
Reader (verified from the completed query profile). Multi-file
primary-key merges retain the existing JNI fallback. Add Doris SQL
coverage for NULL-safe MERGE UPDATE/DELETE on nullable single/composite
keys, ordinary equality, INSERT branches, conditional priority,
duplicate-match atomicity and private-marker name collisions. |
| [`$file_indexes`](apache/paimon#9138) |
Automatically discovered through the SDK registry. Preserve deferred
catalog query checks and pin both backend payloads. FE scan-properties
and actual JNI reconstruction tests cover denied authorization and
snapshot 2 on disk while the catalog publishes only snapshot 1. |
| [`latest-delta` batch startup
mode](apache/paimon#9142) | Not exposed by this
upgrade: reject it explicitly in `@options` instead of silently
converting it to a full-snapshot read. A real three-snapshot test proves
the difference (one delta row versus three full-snapshot rows). |
| [Vector indexes](apache/paimon#8930) and
[full-text indexes](apache/paimon#9031) on DV
tables; [Multivalue array
indexes](apache/paimon#9282); BTree TopN | New
query acceleration needs Doris expression/search/TopN integration and
relevant runtime dependencies and validation. Bumping the SDK does not
enable these SQL capabilities. Existing reader selection remains
conservative. |
| [Arrow bundle writes](apache/paimon#9100) | An
optional batching API, not a new `file.format=arrow` storage format.
Doris keeps its current Arrow-to-Paimon-row writer; using the new bundle
API is a separate performance change requiring memory/lifecycle
validation and benchmarks. |
| [SchemaManager interface](apache/paimon#9174)
and [REST schema history
APIs](apache/paimon#9673) | Updated fixtures to
`FileSystemSchemaManager`. The REST additions inspect schema history;
they do not replace the SDK table's filesystem schema manager. Doris
does not call the new display APIs in this PR. |
| Manifest/index metadata and Variant handling | Both Java endpoints use
SDK 2.1.0; package and serialization tests passed. The scanner tests now
have the same `java.nio` access as the BE JVM. The targeted Java, native
and cluster checks below cover existing supported paths, but do not
establish native/Rust compatibility for every newly introduced table
option or file layout. |
| BLOB partial updates and first-row/data-evolution improvements |
Available upstream; the upgrade preserves Doris' existing write and
reader eligibility constraints. New combinations still need
external-table regression coverage before being advertised. |
| [Removal of file-based
privileges](apache/paimon#9186), arithmetic/cast
fixes, raw physical-metadata restrictions | Actual behavior changes,
described in the release note below; legacy privilege users must migrate
before upgrading. |

### Release note

Upgrade the Paimon Java SDK from 1.4.2 to 2.1.0. Fix NULL-safe MERGE
matched UPDATE/DELETE silently omitting rows with NULL primary keys.
Paimon 2.1 keeps wrapping integer `sum`/`product` overflow by default
and supports per-field `fields.<field>.sum.fail-on-overflow` /
`fields.<field>.product.fail-on-overflow` options for checked
arithmetic. Timestamp-to-date casts before the Unix epoch now use the
preceding calendar date. The SDK rejects raw `$files`,
`$file_key_ranges`, `$binlog`, and `$statistics` tables on query-auth
tables because column masks cannot be applied to their physical
metadata. Doris now preserves query authorization and pins
`$file_indexes` to the catalog-visible snapshot and explicitly rejects
the unimplemented `latest-delta` OPTIONS mode instead of returning a
full snapshot. Paimon's legacy file-based privilege system is no longer
supported by the upstream SDK; deployments using it must migrate
authorization before upgrading.

### Check List (For Author)

- Test
    - [x] Regression test
    - [x] Unit Test
    - [x] Manual test

- **Build:** FE/Java extensions and ASAN BE passed after rebasing onto
master `5e3ed84abb0`; both Java endpoints package SDK 2.1.0. All 9 JNI
plugins passed layout checks.
- **Unit tests:** FE connector 844 passed, 1 live-connectivity test
skipped; Java scanner 175 passed; native Paimon/JNI/Rust 144 passed; S3
shade 1 passed. The MERGE follow-up passed 38 related FE tests.
- **Cluster regression:** 51 Paimon suites passed after rebase; the
MERGE follow-up passed 9 targeted suites, including the new nullable-key
regression. The opt-in KEY_DYNAMIC resource stress suite was skipped.
System-table expected output was regenerated with the runner and
replayed successfully.
- **Backward reads:** Paimon 1.4.2 append/PK data matched its Spark
oracle again; completed profiles confirmed Rust append reads and JNI
fallback for multi-file PK merges.

Scope: S3/filesystem Paimon catalog. File cache was disabled for a local
Btrfs inode-stat issue, and the host swap startup check was skipped.
Paimon HMS/REST catalogs and mixed-SDK rolling upgrades were not
validated.

- Behavior changed:
    - [ ] No.
- [x] Yes. Uses Paimon 2.1.0 SDK behavior, including configurable
integer overflow checks, corrected pre-epoch timestamp-to-date casts,
removal of upstream file-based privileges, snapshot-safe
`$file_indexes`, and explicit rejection of unsupported `latest-delta`
OPTIONS reads.

- Does this need documentation?
- [x] No. Dependency upgrade and upstream behavior changes are recorded
in the release note.
    - [ ] Yes.

### Self-review

- Validation follow-up scope: fix nullable-key MERGE target-presence
detection, then preserve deferred `$file_indexes` authorization and its
JNI source snapshot pin, adapt timestamp-policy validation and callbacks
to SDK 2.1, and refresh affected system-table expected output through
the regression runner.

- Goal and scope: shared SDK upgrade plus directly affected API/behavior
adaptations. New search and batching features remain separate work; no
new connector SPI or native ABI is introduced.
- Correctness and parallel paths: FE binding, serialized BE system-table
reconstruction, JNI SDK behavior, and reader selection were inspected.
`$file_indexes` preserves deferred catalog authorization and carries the
snapshot pin through both FE/BE payloads; timestamp policies validate
and execute against repaired precision; MERGE uses INNER JOIN semantics
or a null-extended target marker rather than testing a nullable key, and
user expressions retain their original binding scope; both OPTIONS
resolution and statement-fence pinning reject `latest-delta`.
- Concurrency and lifecycle: production adaptations add no threads,
locks, shared mutable state, native allocations, or resource owners.
Java fixtures close their catalog/writer/commit/reader resources; the
new cluster suite restores modified session variables in `finally` and
preserves tables for debugging. The existing writer-lifecycle regression
passed 120 successive inserts and verified compaction writer threads
return to zero after each phase.
- Performance: reuse the existing row repair and SDK policy evaluation;
bind source snapshots during FE planning. MERGE adds one constant target
projection only when INSERT branches need unmatched rows. No performance
improvement is claimed.
- Compatibility and configuration: upstream file privileges are removed,
and mixed SDK rolling upgrades remain unverified. No configuration keys,
Thrift/Protobuf definitions, new FE/BE variables, or persistence formats
are added by Doris. Storage access for `$file_indexes` uses the existing
configured catalog endpoint and base-table SELECT privilege; there is no
new artifact-download property.
- Tests and results: FE scan-properties tests cover denied
`$file_indexes` authorization and catalog-published snapshot bounds; an
actual serialized JNI reconstruction test covers the second payload.
Serialized timestamp-policy tests retain row filters, masking, fallback
and nested/hidden-column coverage. MERGE tests cover nullable
single/composite keys, ordinary and NULL-safe equality, branch priority,
duplicate-match rejection without row changes, and same-named user
columns; the new golden is generated and replayed by the regression
runner. The system-table golden now includes SDK 2.1's `operation` and
`writer_version` fields. Builds, unit tests, cluster replay and
legacy-data reads passed after rebase; limits are listed above.
- Other CI failures: the P0 view case passed in isolation; it shares and
drops `v_distinct_agg_rewrite` with `test_mtmv` in the same database.
The Iceberg failure was reproduced as `DESC digest` expecting nullable
`Yes` but returning `No`: [master's source-nullability
change](c7ffbda3877) predates the
[test's old
expectation](e83b06cb7b7). Those
Iceberg and P0 files are unchanged from the base; no unrelated golden
was modified.
- Independent optimizer limitation: a one-row constant source with
multiple NULL-safe join keys can become an empty relation even when an
internal OLAP table has a matching NULL row. This also reproduces in an
ordinary SELECT on an internal table; optimizer files are unchanged by
this PR. The composite-key DELETE regression uses a multi-row source
with an unmatched control. That standalone optimizer defect remains
outside this SDK adaptation.
- Transactions, writes and observability: no transaction/recovery
implementation is changed. The transaction regression confirms an
oversized failed insert publishes neither rows nor a snapshot. Completed
query profiles prove Rust selection and JNI fallback, rather than
relying only on enable flags. Unsupported delta mode fails with an
actionable diagnostic; existing system-table planning diagnostics apply
to snapshot pinning.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label