Repository navigation
[fix](paimon) Read Paimon tables on BE without reconstructing the catalog metastore - #65867
Merged
Merged
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
/review |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Automated review completed. I found no substantiated blocking issue in the current one-file
change.
Part 1.3 checkpoint conclusions:
- Goal and proof: this restores the unrelocated
HiveConfand shims classes needed when FE
serializes an HMS-backed Paimon table and the BE Paimon JNI scanner deserializes it. Both
legacy and Format V2 affected reads converge on that boundary. The PR reports rebuilt-artifact
smoke coverage fornew HiveConf(), theSerializableHiveConfround trip, andgetUser.
Existing HMS regression coverage exercises forced JNI, incremental, time-travel, and
system-table modes; the named branch product test was not run here. - Scope and clarity: this is a focused one-file dependency repair. Wildcard transitive pruning
keeps the addition to the two small Hive artifacts rather than restoring the former catalog
shade. - Concurrency and lifecycle: no threading, locking, mutable-state ownership, or resource
lifecycle changes. The existing process-lifetime scanner classloader behavior is unchanged. - Configuration and compatibility: no configuration, protocol, storage-format, persistence, or
serialized-wrapper contract changes. The Hive versions remain within the existing 3.1 family. - Parallel paths: legacy and Format V2 JNI modes use the same Paimon scanner artifact; native
and C++ reader paths do not cross this Java deserialization boundary, and other scanners are
isolated. - Tests and results: no result files changed. Visible CI style, license, and dependency checks
pass, but they did not execute the deployed Paimon fat jar. The review instructions prohibited
local builds/tests, so validation here is static plus the PR's reported manual artifact tests. - Observability, transactions, writes, FE-BE variables, and performance: no logging/metric,
transaction, data-write, or FE-BE variable behavior changes. The only identified cost is the
modest jar/startup footprint; no query hot path changes. - Risk closure: the wildcard-pruned external dependency exercised by
HiveConfis already
supplied by the supported BE parent classpath, and the direct Hive jars follow the existing
assembly, deployment, and parent-first loading path. I found no substantiated closure,
collision, or version issue.
User focus: no additional focus was provided.
Contributor
Author
|
run buildall |
Contributor
FE UT Coverage ReportIncrement line coverage `` 🎉 |
Contributor
TPC-H: Total hot run time: 29445 ms |
Contributor
TPC-DS: Total hot run time: 176734 ms |
Contributor
ClickBench: Total hot run time: 25.04 s |
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29714 ms |
Contributor
TPC-DS: Total hot run time: 176867 ms |
Contributor
ClickBench: Total hot run time: 24.88 s |
morningman
changed the title
Contributor
Author
|
run buildall |
Contributor
Author
|
/review |
Contributor
TPC-H: Total hot run time: 23371 ms |
Contributor
|
Codex automated review failed and did not complete. Error: You've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Jul 30th, 2026 7:58 AM. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
Contributor
TPC-DS: Total hot run time: 118007 ms |
Contributor
ClickBench: Total hot run time: 18.55 s |
Contributor
Author
|
/review |
yiguolei
previously approved these changes
Jul 24, 2026
github-actions
Bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
Contributor
|
PR approved by anyone and no changes requested. |
Contributor
|
Codex automated review failed and did not complete. Error: You've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Jul 30th, 2026 7:58 AM. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
Contributor
Author
|
/review |
Contributor
|
Codex automated review failed and did not complete. Error: You've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Jul 30th, 2026 7:58 AM. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
Contributor
Author
|
/review |
Contributor
There was a problem hiding this comment.
Request changes: the catalog-less BE table reconstruction still loses branch-local catalog state in two fallback paths.
Blocking findings
- The incremental
$partitions@incrtimestamp range is decided and resolved through the main branch only. The fallback branch can either read beyond its catalog-visible pointer or interpret the main branch's explicit snapshot IDs as its own history. - Reapplying resolved relation
OPTIONSafter both loaders are removed discards the fallback selector already resolved on FE and can make FE and BE use different fallback snapshots.
Both are P1 correctness issues, with branch-skew regression-test requirements in the inline comments.
Checkpoint conclusions
- Goal and scope: Moving the Paimon FileIO providers into the scanner artifact and removing the BE catalog loader directly address the reported classloading and metastore-reconstruction failures. All eight changed files are relevant and the implementation is localized, but fallback behavior prevents the end-to-end objective from being correct in every advertised branch, incremental, and OPTIONS mode.
- Correctness and lifecycle: The captured system/base table publication ordering, decorator peeling, five deferred-reader classifications, and branch-by-branch deferred authorization transfer are coherent. The two inline fallback defects remain blocking. Existing live threads on catalog-visible-empty /
$snapshots/$buckets, implicit fallback pin translation,$filesmarker skew, and the missing artifact-level classloader regression are intentionally not duplicated and are not cleared by this review. - Configuration, compatibility, and special paths: Ordinary single-branch, schema-evolution, authorization, rollback/publication, time-travel, fallback, incremental, and FE-planned versus BE-replanned paths were traced. No journal, persistence, transaction, wire-format, or user-configuration compatibility change was introduced. The relevant RBAC boundary is in-model; the deferred authorization transfer itself did not yield another new finding.
- Concurrency and performance: Volatile publication of the captured table generation is ordered correctly. The added work is planning-time metadata and authorization lookup, with no new hot-loop or execution-path performance blocker found.
- Tests and evidence: The added unit tests cover several generation, authorization, and snapshot cases, but not the two branch-skew cases above, and the existing packaging-level classloader-test concern remains live. This runner is review-only, so no build or tests were run; conclusions are based on static tracing and current GitHub CI state.
- Observability and writes: No new persistence/write protocol or logging/metrics contract applies.
- User focus: No additional review focus was provided; the complete PR was reviewed.
Three bounded review rounds converged on head e692e978351eedf61ad382ff38d37f9a06948029: all reviewers returned NO_NEW_VALUABLE_FINDINGS against this exact two-comment set. This review requests changes on the two new, distinct findings while retaining the existing unresolved threads.
| if (range == null || !dataTable.catalogEnvironment().supportsVersionManagement()) { | ||
| return incrementalParams; | ||
| } | ||
| SnapshotManager snapshotManager = dataTable.snapshotManager(); |
Contributor
There was a problem hiding this comment.
[P1] Bind this incremental range against each fallback branch
This helper makes both the rewrite decision and endpoint resolution through dataTable.snapshotManager(), which a FallbackReadFileStoreTable delegates to its main branch. If the requested end is older than the main latest, line 352 leaves the timestamp range unchanged even when it extends past the fallback catalog pointer; after loader removal, that branch can include an unpublished or rollback-retained snapshot. If this method does rewrite, Paimon's fallback option rewrite copies the resulting main incremental-between IDs unchanged because it translates only scan.snapshot-id, so disjoint branch histories fail out of range and overlapping IDs select the wrong commits. Please resolve/cap the range separately with both captured branch managers (or reject this composition), and test different branch IDs and catalog times.
| return rebuiltSysTable.copy(incrementalParams); | ||
| } | ||
| if (theScanParams.isOptions()) { | ||
| return PaimonScanParams.applyOptions(rebuiltSysTable, |
Contributor
There was a problem hiding this comment.
[P1] Preserve the fallback selector already resolved on FE
The processed FE wrapper has already applied this OPTIONS selector to the catalog-backed fallback pair, where Paimon maps the main snapshot to the fallback branch through that branch's catalog-visible SnapshotManager. Here the code rebuilds an unoptioned pair with CatalogEnvironment.empty() and applies only the main selector again, so FallbackReadFileStoreTable.copy repeats the translation against filesystem state. After fallback rollback/publication skew, FE can plan F7 while the serialized BE table selects retained F8; this also reaches OPTIONS-capable FE-planned system tables such as $ro. Please preserve each selected branch before removing its loader, and add an encode/decode fallback-skew OPTIONS test.
Contributor
Author
|
/review |
Contributor
There was a problem hiding this comment.
Request changes remains warranted on reviewed head 0d83512fab416c1b4b53d7441ea7d876b140e521.
I found no new non-duplicate inline issue, so this review intentionally adds zero inline comments. The current head still does not preserve branch-local catalog state across the catalog-less FE-to-BE handoff:
- The fallback selector already resolved on FE is translated again after both branch loaders are removed.
- Fallback incremental ranges cannot be bounded by a main-only ID range or left for loaderless branch managers to resolve.
- The same catalog-bound limitation remains for the fallback snapshot pin and for metadata readers that ignore
scan.snapshot-id, including catalog-visible-empty state. - The classloader move is statically coherent, but the production parent/child artifact boundary still lacks the already-requested assembled-jar
FileIO.getregression.
Critical checkpoint conclusions:
- Task goal and proof: Removing the metastore loader and co-locating the Paimon FileIO SPI address the reported BE classloading failure on ordinary paths. The goal is not fully met for the advertised fallback/OPTIONS/incremental and catalog-bound metadata paths above. The added FE tests exercise generation capture, reconstruction, authorization, and selected system tables, but not those branch-skew cases or the assembled classloader boundary.
- Scope and minimality: The production changes are focused on packaging and table reconstruction. The helper boundaries are understandable, but correctness depends on a broad Paimon system-table/fallback lifecycle rather than only local serialization.
- Concurrency:
sysBaseTableis written under the same monitor before volatile publication ofpaimonSysTable, so readers acquire a matching wrapper/base generation. Scan-node table state is query-local. No new lock ordering or shared-collection race was found. - Lifecycle: Capturing the system wrapper and base together fixes refresh-generation skew, and rebuilding fallback branches independently preserves captured schemas. The unresolved lifecycle break is intentional loader removal before every branch-local selector/range has been frozen.
- Configuration: No Doris configuration item is added or changed; dynamic Paimon options remain relation-local copies.
- Compatibility and parallel paths: No thrift, persisted metadata, storage format, function symbol, or transaction protocol changes are introduced. I traced data tables, fixed-row and schema-derived system tables, native/JNI consumers, version-managed and plain catalogs, decorated/fallback tables, publication/rollback/empty states, and OPTIONS/incremental modes. The live threads above cover the remaining wrong parallel paths.
- Conditional checks: The five pin-capable BE-resolved system tables are classified correctly, and
$filesauthorization is transferred branch by branch. The fallback early return in incremental binding and the post-loader OPTIONS reapplication have concrete failure paths already documented in the live threads. - Tests and results: This was a review-only environment, so no build or test suite was run. Static shell parsing passed. Current CI shows CheckStyle and repository checks green, but the extension workflow did not exercise the assembled scanner/preload classloaders; that does not prove the runtime packaging contract.
- Authorization and error handling: Under the repository threat model, limited-user RBAC is in scope. The earlier fallback authorization loss is fixed: main and fallback
$filesbranches are authorized before their loaders are removed. No new silent error or distinct RBAC bypass was found. - Observability: The new build guard has specific error messages and the FE retains existing planning timing/error reporting. No additional hot-path metric or log is necessary.
- Transactions, persistence, writes, and FE-BE variables: Not applicable beyond the existing serialized-table handoff. There is no Doris write, EditLog, failover, or new FE-BE field.
- Performance and other invariants: Reconstruction, authorization, and snapshot binding occur once per scan initialization rather than per row or split. No distinct CPU, memory, transaction-visibility, or MoW invariant issue was found.
No additional user review focus was provided; the complete PR was reviewed.
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29374 ms |
Contributor
TPC-DS: Total hot run time: 177151 ms |
Contributor
ClickBench: Total hot run time: 24.96 s |
Contributor
FE Regression Coverage ReportIncrement line coverage |
4 of 8 tasks
… BE reads Reading an object-store-backed Paimon table over a REST/DLF catalog failed on BE when resolving the latest snapshot from the filesystem: NoClassDefFoundError: org/apache/paimon/fs/FileIOLoader The paimon OSS/S3 FileIO plugins (paimon-s3 -> S3Loader, paimon-jindo -> JindoLoader) were bundled in preload-extensions, which sits on BE's JVM system (app) classpath, but the org.apache.paimon.fs.FileIOLoader interface they implement lives in paimon-common, bundled only in paimon-scanner's own JniScannerClassLoader. When a BE read resolves the latest snapshot from the filesystem, RESTTokenFileIO.fileIO() calls FileIO.get() -> ServiceLoader.load(FileIOLoader.class, ...), which eagerly instantiates every registered provider. JniScannerClassLoader is a plain parent-first URLClassLoader, so the app classloader defines S3Loader/JindoLoader and cannot resolve the child-only FileIOLoader interface -> the whole discovery aborts. Co-locate the plugins with the FileIOLoader interface: move paimon-s3 / paimon-jindo from preload-extensions to paimon-scanner. paimon-scanner already carries a complete paimon runtime, and the assembly's metaInf-services handler merges the plugins' service files with paimon-common's, so all five FileIOLoader providers (local/hadoop/viewfs/s3/jindo) and the interface end up in one classloader. The Jindo SDK stays on the app classpath (start_be.sh adds jindofs to DORIS_CLASSPATH), so paimon-scanner still reaches it via parent delegation. Both jars must be rebuilt and redeployed: paimon-scanner (now carries the plugins) and preload-extensions (must stop advertising the un-linkable providers on the shared classpath). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01DHwqsiGWNosZCeS4ecK97e
…ld time The FileIOLoader interface (paimon-common) and every provider implementing it must sit in the same jar: JniScannerClassLoader delegates parent-first, so a provider left on the shared preload-extensions (JVM app) classpath cannot resolve the child-only interface and ServiceLoader discovery aborts at runtime with "NoClassDefFoundError: FileIOLoader". Unit tests all run in one classloader and cannot reproduce that, so the invariant is asserted on the built artifacts instead: paimon-scanner's merged META-INF/services/org.apache.paimon.fs.FileIOLoader must list S3Loader and JindoLoader, and preload-extensions must bundle no org/apache/paimon/(s3|jindo) class at all. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
…isting build.sh runs with `set -eo pipefail`. `unzip -l <jar> | grep -q` closes the pipe on the first match, so unzip dies of SIGPIPE and the pipeline reports 141 even though grep matched: the preload-extensions guard then took its false branch and let the forbidden provider through. The listing is 120k entries (11 MB) against a 64 KB pipe buffer, so any match outside the last few lines hits this - the guard failed exactly when a provider had leaked in, and stayed quiet on a clean jar, which is why the build never showed it. Let grep consume the whole listing instead. The paimon-scanner guard had a second, unrelated hole: `unzip -p` exits 11 when the services entry is missing altogether, which under `set -e` killed the build with "caution: filename not matched" before the guard could explain itself - its primary failure mode. Tolerate that exit and let the message print. Also record the fallback-branch limit of the BE-side snapshot pin: for a table with `scan.fallback-branch`, `FallbackReadFileStoreTable#copyWithoutTimeTravel` derives the fallback branch's own bound through `SnapshotManager#earlierOrEqualTimeMills`, and that branch has no catalog loader left either, so it resolves against the snapshot directory like the other metadata-only gaps already noted there. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
…alog metastore Every table loaded from a metastore-backed Paimon catalog (HMS / DLF) carries a Paimon HiveCatalogLoader in its CatalogEnvironment. The BE only reads - via FE-resolved splits and the object store - and never needs the catalog, yet deserializing that loader forces the whole Hive metastore stack onto the BE classpath: HiveConf, the metastore API, and, when a system table resolves its latest snapshot, even the metastore client (DLF's ProxyMetaStoreClient and its REST stack). PaimonScanPlanProvider now serializes a catalog-less table to the BE. A data table is rebuilt from fileIO / location / schema; a system table is rebuilt over a catalog-less base so SnapshotManager lists the snapshot directory instead of calling the metastore. Everything the loader used to do on the BE is done on the FE first, so dropping it changes no rows: - authorization: $files re-plans on the BE through DataTableScan#plan(), whose Catalog#authTableQuery becomes a permanent allow without the loader, so it is run here - per branch on a scan.fallback-branch pair, whose fallback branch carries its own CatalogEnvironment. - snapshot: for the five system tables that pick their snapshot inside the BE reader, the catalog-visible snapshot is pinned via copyWithoutTimeTravel, so a version-managed (REST / DLF) catalog cannot expose an unpublished or rollback-orphaned snapshot. Row-type-following tables ($audit_log, $ro, ...) are deliberately left unpinned: pinning would rewind the BE schema. - incremental range: @incr on $partitions resolves its timestamp endpoints against the catalog and hands the BE the explicit id range paimon would have computed, except on a fallback pair, where an id range cannot be expressed. - schema generation: getSysTableHandle builds the wrapper over the base Table its handle already carries and keeps that base on the sys handle, so the FE and the BE never plan on two different generations. The decorators paimon stacks on a loaded table are peeled down to the fallback pair, which must stay on top or $ro silently drops every fallback-only partition. - relation scan params: the @options / @incr copy the FE applied to the original wrapper is re-applied to the rebuilt one. With the FE no longer sending a catalog loader, paimon-scanner does not need the Hive artifacts on the BE side either; they are removed in the same change. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29367 ms |
Contributor
TPC-DS: Total hot run time: 177636 ms |
Contributor
ClickBench: Total hot run time: 24.79 s |
Contributor
Author
|
/review |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
There was a problem hiding this comment.
Automated review result: request changes.
Review completion: COMPLETE. Three bounded review rounds converged on head 8db5036d2e2db28b10a0d519a2cfb3e1d82ff418 and this exact one-comment payload; all three Round 3 reviewers returned NO_NEW_VALUABLE_FINDINGS. No additional user-provided review focus was supplied, so all ten authoritative changed files were reviewed.
Blocking finding
- The eager fallback-branch authorization can reject a
$filesquery that main completely satisfies, even though Paimon would not invoke fallback's authorizedplan()or return fallback files. The inline P1 is distinct from the earlier missing-authorization thread, which applies when fallback does contribute.
Existing threads already cover the fallback OPTIONS/incremental branch-state defects, catalog-visible snapshot/marker limitations, and the missing assembled parent/child classloader regression; those issues were verified at this head and intentionally not duplicated.
Critical checkpoint conclusions
- Goal, scope, and proof: Removing catalog loaders from the serialized BE table and moving Paimon FileIO providers into the scanner artifact are focused changes that address the reported classloading/metastore reconstruction failure. The accepted P1 shows that loader-free authorization transfer is not yet behaviorally equivalent for every advertised fallback
$filespath. - Correctness, lifecycle, parallel paths, and conditions: Captured system wrapper/base generations, handle copies, decorator peeling, all 17 Paimon system readers, normal/OPTIONS/INCR paths, version-managed and plain catalogs, direct and privilege-decorated fallback tables, and native/JNI consumers were traced. The exact wrapper/base generation survives the production property path; the new unconditional fallback check is the only non-duplicate issue. All other substantiated snapshot/selector cases map to named live threads.
- Concurrency, configuration, and compatibility: No new shared-state race, lock-order issue, Doris setting, persisted format, storage format, public symbol, Thrift field, or rolling-upgrade contract was introduced. The compatibility impact is the user-visible false denial at the existing serialized-table boundary.
- Tests and results: The changed FE tests were reviewed statically and cover reconstruction, generation capture, pinning, and eager branch authorization, but not a main-complete/main-allowed/fallback-denied plan. No build or test suite was run because the authoritative review contract prohibits it and this worktree is not initialized; only
bash -n build.shwas run as a syntax check. The already-requested assembled-jar classloader test remains a separate live thread. - Errors and observability: Existing exceptions and build diagnostics are adequate for explicit failures, but this denial needs a behavioral test rather than a new metric or log. No additional observability defect was found.
- Transactions, persistence, writes, and variable propagation: Not applicable beyond the existing FE-to-BE serialized Java
Tablepayload; this patch adds no Doris transaction, EditLog/failover, data-write, or new variable-propagation path. - Performance and other invariants: The added reconstruction and authorization work is per scan initialization, not per row. No independent material CPU, memory, I/O, or hot-path regression was substantiated.
The final live-state sweep confirmed the PR head/base, normalized diff content, diff position 172, current review threads, and a fully classified ledger with no unresolved candidate.
| if (undecorated instanceof FallbackReadFileStoreTable) { | ||
| FallbackReadFileStoreTable fallbackReadTable = (FallbackReadFileStoreTable) undecorated; | ||
| authorizeBranch(fallbackReadTable.wrapped()); | ||
| authorizeBranch(fallbackReadTable.fallback()); |
Contributor
There was a problem hiding this comment.
[P1] Preserve conditional fallback authorization
FallbackReadScan.plan() always plans the main scan, but invokes fallbackScan.plan() only when listPartitions() finds partitions not completed by main; that listing itself does not authorize. This unconditional call therefore rejects a $files query for a user denied on the fallback branch even when every selected partition is satisfied by main and Paimon would never invoke fallback's authorized plan() or return fallback files. This is the inverse of the earlier missing-auth case: please transfer the fallback authorization with the same branch-use condition (while retaining the check when fallback contributes), and cover a main-complete/main-allowed/fallback-denied case.
CalvinKirs
approved these changes
Jul 31, 2026
wyxxxcat
pushed a commit
to wyxxxcat/doris
that referenced
this pull request
Aug 17, 2026
…alog metastore (apache#65867) ### What problem does this PR solve? Reading a Paimon table over a metastore-backed or REST catalog (HMS / DLF) fails on BE whenever the JNI reader is used — most visibly system tables (e.g. `$snapshots`, `$files`), which always go through JNI, and also branch / time-travel / incremental reads. Two failures were observed, both caused by the BE rebuilding a catalog it does not need: ``` # HMS catalog [JNI_ERROR] NoClassDefFoundError: org/apache/hadoop/hive/conf/HiveConf # DLF (REST) catalog [JNI_ERROR] ClassNotFoundException: com.aliyun.datalake.metastore.hive2.ProxyMetaStoreClient ``` **Root cause.** A table loaded from a metastore-backed Paimon catalog carries a Paimon `CatalogLoader` (e.g. `HiveCatalogLoader`) in its `CatalogEnvironment`. When FE serializes that table to BE, `SnapshotManager#latestSnapshotId` resolves the latest snapshot through the catalog's `SnapshotLoader`, which on BE reconstructs the catalog's metastore client and its whole Hive / DLF-REST stack — even though the BE only reads (via FE-resolved splits and the object store) and the snapshots already live there. This was previously masked by `java-udf`'s ~122MB `hive-catalog-shade` on BE's shared classpath; apache#65733 replaced it with the slim `hive-udf-shade`, exposing the dependency. **Fix 1 — serialize a catalog-less table to the BE (`PaimonScanNode`).** - data table: rebuild via `FileStoreTableFactory` with an empty `CatalogEnvironment`. A `FileStoreTable` is fully defined by fileIO / location / schema, and its dynamic options (time travel, incremental) are already merged into the schema by `copy(...)`, so nothing is lost except the catalog loader. - system table: rebuild it over such a catalog-less data table via `SystemTableLoader`. With no catalog loader, `SnapshotManager#latestSnapshotId` lists the snapshot directory on the filesystem instead of calling the metastore, so the BE never reconstructs the catalog and no longer needs any Hive / metastore classes. **Fix 2 — co-locate the Paimon FileIO plugins with `paimon-scanner` (packaging).** Reading the snapshot from the filesystem makes the BE materialize the object-store `FileIO` lazily via `FileIO.get()` → `ServiceLoader.load(FileIOLoader.class, ...)`, which eagerly instantiates every registered provider. The OSS/S3 plugins (`paimon-s3` → `S3Loader`, `paimon-jindo` → `JindoLoader`) were bundled in `preload-extensions`, on BE's JVM system (app) classpath, but the `org.apache.paimon.fs.FileIOLoader` interface they implement ships in `paimon-common`, bundled only in paimon-scanner's own `JniScannerClassLoader`. That loader is parent-first, so the app classloader defines `S3Loader` / `JindoLoader` and cannot resolve the child-only `FileIOLoader` → `NoClassDefFoundError: FileIOLoader`, which aborts discovery. Moving the two plugins into `paimon-scanner` puts the whole FileIO SPI (interface + all providers) in one classloader; the Jindo SDK stays on the app classpath (`start_be.sh` adds `jindofs` to `DORIS_CLASSPATH`) and is still reachable via parent delegation. Regression tests to re-run: `io.trino.tests.product.paimon.TestPaimonSparkCompatibility` (system-table reads) and `external_table_p2/paimon/test_paimon_dlf_rest_catalog`. ### Release note Fix Paimon reads (system tables, branch / time-travel / incremental) failing on BE over metastore-backed or REST catalogs (HMS / DLF) with `NoClassDefFoundError` (`HiveConf` / `FileIOLoader`) or `ClassNotFoundException: ProxyMetaStoreClient`. ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [x] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - Behavior changed: - [x] No. - [ ] Yes. - Does this need documentation? - [x] No. - [ ] Yes. --------- Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
3 of 16 tasks
morningman
added a commit
that referenced
this pull request
Oct 25, 2026
…ble source (#69569) ### What problem does this PR solve? Issue Number: None Related PR: #68897, #69343 Problem Summary: **In short.** This PR only adds tests. Some Paimon system tables are planned on the BE. For them, the JNI scanner rebuilds the system table from the source table that the FE sends in `doris.serialized-system-source`. In a version-managed catalog, that payload must be pinned to the snapshot the catalog publishes. #69343 added this pin. This PR adds the two cases its tests do not cover: a `scan.fallback-branch` pair, and an explicit relation snapshot that must still take precedence over the pin. **Background.** Six Paimon system tables are planned on the BE: `$files`, `$file_indexes`, `$partitions`, `$manifests`, `$statistics` and `$table_indexes` (`PaimonScanParams#resolvesSnapshotOnBackend`). For these tables the FE sends two payloads: - `serialized_table` is built by `tableForBackend`. - `doris.serialized-system-source` was added in #66247 so that a smaller BE can cap the manifest parallelism. The FE always sends it when the system table has a source. When it is present, `PaimonJniScanner#applyBackendManifestParallelism` rebuilds the system table from this payload instead of from `serialized_table`. A version-managed catalog (REST) owns the committed snapshot pointer. Once the catalog loader is removed, Paimon falls back to listing the snapshot directory, and that directory can already contain a snapshot the catalog has not published. #65867 therefore pins `scan.snapshot-id` to the catalog's pointer (`PaimonScanPlanProvider#pinCatalogSnapshot`). **The problem, and what it cost.** Before #69343, only `serialized_table` carried that pin. The side channel shipped the source unpinned, and the BE rebuilt from the side channel. So in a REST catalog these six system tables were planned on the newest snapshot in storage rather than the published one, on every branch. Inside the publication window, `$partitions` could list partitions of a snapshot the catalog had not published yet. The review of #68897 pointed this out for chain tables. #69343 (the Paimon 2.1.0 upgrade) now prepares the side-channel source in the same order as `tableForBackend` (`prepareSystemSource`): 1. Pin the catalog snapshot, branch by branch. 2. Apply the relation's `@options` over the pin. 3. Drop the catalog loader. Its tests cover `$file_indexes` on a single-branch table, on the FE and through the JNI rebuild. **How this PR fixes it.** Two FE unit tests in `PaimonBackendBoundTableTest` read the side channel out of `paimon.options_json`: - `systemSourcePinsEachFallbackBranchToItsOwnCatalogSnapshot` builds `$partitions` over a fallback pair whose two catalogs point at snapshots 2 and 1. Each branch of the transported source carries its own pointer and no catalog loader. A chain table nests the same pairs. - `relationSnapshotOutranksTheCatalogPinOnTheSystemSource` uses a catalog that points at snapshot 2 and `@options('scan.snapshot-id'='1')`. The transported source says 1. Paimon 2.1.0 still has the two limitations documented on `pinCatalogSnapshot`: - `$files` emits its marker splits from `SnapshotReader#partitions`. - `$snapshots` and `$buckets` read without `scan.snapshot-id`. So that javadoc stays as written. **Results.** | `prepareSystemSource` | `PaimonBackendBoundTableTest` | |---|---| | as on master (pin, then `@options`) | 38 / 38 pass | | without the pin | the new fallback case and the existing `$file_indexes` case fail | | pin applied after the relation options | the new relation-snapshot case fails (`expected: <1> but was: <2>`) | All 847 tests of fe-connector-paimon pass, with 1 skipped as before, and checkstyle passes. No regression suite is added: it would need a version-managed (REST) Paimon catalog whose pointer lags the snapshot directory, and the docker fixtures do not provide one.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Reading a Paimon table over a metastore-backed or REST catalog (HMS / DLF) fails on
BE whenever the JNI reader is used — most visibly system tables (e.g.
$snapshots,$files), which always go through JNI, and also branch / time-travel / incrementalreads. Two failures were observed, both caused by the BE rebuilding a catalog it does
not need:
Root cause. A table loaded from a metastore-backed Paimon catalog carries a Paimon
CatalogLoader(e.g.HiveCatalogLoader) in itsCatalogEnvironment. When FEserializes that table to BE,
SnapshotManager#latestSnapshotIdresolves the latestsnapshot through the catalog's
SnapshotLoader, which on BE reconstructs the catalog'smetastore client and its whole Hive / DLF-REST stack — even though the BE only reads
(via FE-resolved splits and the object store) and the snapshots already live there.
This was previously masked by
java-udf's ~122MBhive-catalog-shadeon BE's sharedclasspath; #65733 replaced it with the slim
hive-udf-shade, exposing the dependency.Fix 1 — serialize a catalog-less table to the BE (
PaimonScanNode).FileStoreTableFactorywith an emptyCatalogEnvironment.A
FileStoreTableis fully defined by fileIO / location / schema, and its dynamicoptions (time travel, incremental) are already merged into the schema by
copy(...),so nothing is lost except the catalog loader.
SystemTableLoader.With no catalog loader,
SnapshotManager#latestSnapshotIdlists the snapshot directoryon the filesystem instead of calling the metastore, so the BE never reconstructs the
catalog and no longer needs any Hive / metastore classes.
Fix 2 — co-locate the Paimon FileIO plugins with
paimon-scanner(packaging).Reading the snapshot from the filesystem makes the BE materialize the object-store
FileIOlazily viaFileIO.get()→ServiceLoader.load(FileIOLoader.class, ...),which eagerly instantiates every registered provider. The OSS/S3 plugins
(
paimon-s3→S3Loader,paimon-jindo→JindoLoader) were bundled inpreload-extensions, on BE's JVM system (app) classpath, but theorg.apache.paimon.fs.FileIOLoaderinterface they implement ships inpaimon-common,bundled only in paimon-scanner's own
JniScannerClassLoader. That loader isparent-first, so the app classloader defines
S3Loader/JindoLoaderand cannotresolve the child-only
FileIOLoader→NoClassDefFoundError: FileIOLoader, whichaborts discovery. Moving the two plugins into
paimon-scannerputs the whole FileIOSPI (interface + all providers) in one classloader; the Jindo SDK stays on the app
classpath (
start_be.shaddsjindofstoDORIS_CLASSPATH) and is still reachablevia parent delegation.
Regression tests to re-run:
io.trino.tests.product.paimon.TestPaimonSparkCompatibility(system-table reads) andexternal_table_p2/paimon/test_paimon_dlf_rest_catalog.Release note
Fix Paimon reads (system tables, branch / time-travel / incremental) failing on BE over
metastore-backed or REST catalogs (HMS / DLF) with
NoClassDefFoundError(
HiveConf/FileIOLoader) orClassNotFoundException: ProxyMetaStoreClient.Check List (For Author)
Test
Behavior changed:
Does this need documentation?