Repository navigation
[fix](paimon) Keep chain groups in Paimon fallback rebuilds and reject chain table scans clearly - #68897
Conversation
…t chain table scans clearly ### What problem does this PR solve? Issue Number: None Related PR: apache#68885 Problem Summary: Paimon 1.4 builds a chain table as a fallback pair whose other branch is a ChainGroupReadTable over the snapshot and the delta branch, and the outer pair accepts no other nested fallback. Seven places in the Paimon connector and the BE JNI scanner rebuild fallback pairs branch by branch, and they rebuilt the chain group as a plain pair or as a plain table: the BE manifest-parallelism bound (taken when the FE cap is below the BE's core count) and its FE twin, the catalog-loader strip and the catalog snapshot pin of the table sent to the BE, and the restore of a statement pin (pinned fallback snapshots and bound schemas). Every data scan of a chain table failed in planning with Paimon's "This is a bug, perhaps there is a recursive call.", and a $files scan could fail the same way on the BE. The rebuilds now keep the chain group. With the group kept, a scan still cannot read a chain table: Doris plans neither the partitions a chain table reads from its snapshot and delta branches nor the chain splits. A table whose main branch is empty read as empty, and one with main data failed on the partition values of the chain splits. A data scan of a chain table now fails at planning with a clear message. Its system tables, and a branch read on its own (@Branch), are plain tables and still read. ### Release note Reading a Paimon chain table (chain-table.enabled) fails with "Reading Paimon chain table ... is not supported yet" instead of an internal Paimon error. ### Check List (For Author) - Test: Unit Test, Manual test - PaimonJniScannerTest, PaimonReaderOptionsTest: the manifest bound keeps the chain group. PaimonBackendBoundTableTest: the catalog-loader strip keeps both chain branches; a data scan of a chain table fails and a branch read on its own resolves. PaimonStatementSchemaTest: restoring a pinned chain table keeps the group and reads the rows Paimon reads at the same snapshot. - A local cluster with a file-system chain table built through the Paimon API: before, every SELECT failed with the recursive-call error; after, it fails with the new message, while $files and @Branch(snapshot) / @Branch(delta) read the branch data. - Behavior changed: Yes. A data scan of a Paimon chain table fails with a clear message instead of a Paimon internal error. - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
|
/review |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Static review of PR #68897 at f64111fb2aa6691ca47ad41e65100112560bfa9f (base dd456ab7a894f54c6553415223c6d5c9c7d4e035). Request changes: two distinct inline findings remain after two complete review rounds and a final sweep of all nine changed files. There were no existing blocking inline threads or user focus points. No builds or tests were run under the review contract.
Critical checkpoints
| Checkpoint | Conclusion |
|---|---|
| Goal and proof | Preserving Paimon's nested ChainGroupReadTable through FE and BE table reconstruction, while refusing unsupported ordinary chain scans, is mostly implemented. New unit tests check subtype retention, pinned SDK reads, and ordinary-handle refusal. The system-handle escape and mixed-version path lack coverage and produce the two findings below. |
| Scope and clarity | The shared pair-rebuilder keeps the change focused. Both findings are at changed control-flow and serialization boundaries. |
| Concurrency and locks | Rebuilds are per-scan table copies. The change adds no shared mutable state, threads, locks, or lock-order dependency. |
| Lifecycle and ownership | The FE removes the catalog loader before serialization; the BE reconstructs a table per scanner initialization. No new static initialization, ownership cycle, or release path was found. |
| Configuration | No new setting is introduced. The existing chain option and manifest-parallelism cap drive the affected paths; dynamic configuration behavior is unchanged. |
| Compatibility | The serialized system-source object graph changes. A previous BE that lowers a child manifest cap cannot rebuild the new nested group, so a rolling upgrade can fail (P2). No thrift/protobuf definition changes. |
| Parallel paths | FE runtime-safe normalization, snapshot/schema pinning, catalog snapshot pinning, loader removal, and BE cap normalization were traced. Direct snapshot/delta branch reads remain plain tables. The $ro system wrapper is still a data-read path through the new exception (P1). |
| Conditions and errors | The ordinary-table guard has a clear rationale and includes the table name in its error. Its blanket isSystemTable() exemption admits $ro despite the same unsupported chain split/partition semantics. |
| Tests and results | The five added test methods assert the constructed table shapes and one SDK read result. They do not exercise a Doris $ro scan with a delta-only partition or a new-FE/old-BE serialized source. Assertions were reviewed statically; test results were not observed. |
| Observability | Existing connector exceptions identify ordinary rejected scans; no new metrics or log-heavy path is needed for the local reconstruction. The two uncovered paths need correctness fixes rather than additional logging. |
| Transactions, persistence, and writes | Production changes are read-side only; Doris EditLog, transaction state, and data writes are untouched. Snapshot/schema pins were traced through the nested branches and showed no separate inconsistency. |
| FE–BE variables | No new thrift field is added, but existing DORIS_SERIALIZED_SYSTEM_SOURCE and manifest-cap options now carry a different table graph; this is the P2 compatibility boundary. |
| Performance and memory | Reconstruction visits the small fixed branch tree once per scan; no new hot-loop allocation, retained graph, or material performance regression was found. |
| Other issues | Security-sensitive artifact or HTTP behavior, BE nullable columns, and C++ memory safety are not changed. The existing main-only $files inventory behavior was checked and not posted as a separate finding. |
The review is complete as a static review. The accepted inline findings are the only substantiated issues remaining.
| */ | ||
| Table resolveScanTable(PaimonTableHandle paimonHandle) { | ||
| Table table = resolveTable(paimonHandle); | ||
| if (!paimonHandle.isSystemTable() && PaimonTableDecorators.isChainTable(table)) { |
There was a problem hiding this comment.
[P1] Reject chain-backed $ro reads at this boundary. The new isSystemTable() exception leaves t$ro open even though it is a data scan over the same outer chain pair. In Paimon 1.4.2, ReadOptimizedTable.newScan() passes a callback through the outer fallback; for the nested ChainGroupReadTable, that callback reads its delegated snapshot branch rather than invoking the chain planner. A partition present only in delta, even with files eligible for the read-optimized scan, can therefore disappear from SELECT ... FROM t$ro; an emitted fallback DataSplit also does not meet the group reader's ChainSplit requirement. The SDK behavior predates this guard, but the new selective refusal leaves this unsupported read exposed. Reject chain-backed $ro at handle construction or scanning until it has a chain-aware path, and cover a delta-only partition. See ReadOptimizedTable and ChainGroupReadTable.
There was a problem hiding this comment.
Not changing this: t$ro on a chain table returns what Paimon itself returns, on master and with this PR.
$ro is Paimon's view of the latest full compaction. ReadOptimizedTable.newScan() hands its callback to the outer fallback pair, and for the nested group that callback plans with the snapshot branch's reader, so a partition held only by the delta branch is not in Paimon's own t$ro either; the delta branch's compacted files are increments, not that partition. The check at this line could not catch $ro in any case: the handle's table is a ReadOptimizedTable, not a FileStoreTable, so isChainTable is false for it.
Checked with Paimon 1.4.2 in jshell on a chain table with main 20231231, snapshot 20240101 and a delta-only 20240102, every branch fully compacted. Paimon's own t$ro returns [20231231 9 z, 20240101 1 a, 20240101 2 b], and reading the same splits through master's backend payload and through this PR's returns the same three rows. With this PR Paimon logs the same "Reading from supplemental branch has problems" error per fallback split that its own table logs.
Refusing $ro would fail a read that works on master. A chain-aware $ro, if Paimon defines one, belongs with the chain read support this PR leaves out.
| if (table instanceof FallbackReadFileStoreTable) { | ||
| // A chain table nests its snapshot and delta branches in a pair of their own; rebuild those too. | ||
| FallbackReadFileStoreTable pair = (FallbackReadFileStoreTable) table; | ||
| return PaimonTableDecorators.rebuildFallbackPair(pair, |
There was a problem hiding this comment.
[P2] Keep this system-source payload consumable by the previous BE. It now serializes a preserved ChainGroupReadTable for chain-backed system views. During a rolling upgrade, a prior-head BE with fewer CPUs can lower a child scan.manifest.parallelism value, rebuild that group as a plain FallbackReadFileStoreTable (its pre-change scanner behavior), and then fail when constructing the outer pair because Paimon requires a nested ChainGroupReadTable. For example, an FE cap of 16, BE cap of 8, and snapshot-branch value of 16 trigger this on a supported $files scan with a main-branch marker. Gate affected system scans or send a payload the previous BE can consume. See FallbackReadFileStoreTable.
There was a problem hiding this comment.
Confirmed, and not changed, for three reasons:
- It only happens when the FE is upgraded before the BE. Doris upgrades BEs first, and this PR's BE reads both the new payload and the previous FE's, in which the group has been replaced by a plain table at the snapshot branch (
testBackendCapStillReadsTheChainTablePayloadOfAnOlderFrontend, 26eebd6). - It fails the scan at scanner initialization rather than returning rows, and stops once that BE is upgraded.
- The payload the previous BE can consume is that flattened one, and it loses the delta branch. Paimon resolves
$partitionson the backend: for a chain table whose delta branch alone holds 20240102, the flattened payload lists[20231231, 20240101], while the payload with the group lists Paimon's own[20231231, 20240101, 20240102](Paimon 1.4.2 in jshell). That would trade a temporary error for a silently wrong result.
The trigger is wider than the example: calling the pre-PR scanner's cap path with an FE cap of 16 fails whenever the BE has more cores than the cap, since the absent option is then filled in and the nested pair rebuilt, not only when a branch sets a larger value. 26eebd6 also covers the system-source path (testBackendCapKeepsTheChainGroupOfAChainTableSystemSource), and the PR description now states the BE-first upgrade order and that $partitions lists the delta-only partitions.
TPC-H: Total hot run time: 29264 ms |
TPC-DS: Total hot run time: 151268 ms |
ClickBench: Total hot run time: 24.27 s |
FE Regression Coverage ReportIncrement line coverage |
FE UT Coverage ReportIncrement line coverage `` 🎉 |
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: The backend scanner bounds Paimon's manifest
parallelism by rebuilding the transported table, and this PR made that
rebuild keep a chain table's ChainGroupReadTable. Only the plain-table
overload was covered. A system table over a chain table ($partitions,
$files, ...) is rebuilt from the transported source instead, and the
backend resolves $partitions itself, so losing the group there drops the
partitions only the delta branch holds. A backend upgraded before the FE
also still receives the payload of the previous FE, in which the chain
group has been replaced by a plain table at the snapshot branch, and
must keep reading it as before.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- PaimonJniScannerTest
.testBackendCapKeepsTheChainGroupOfAChainTableSystemSource:
a $partitions wrapper rebuilt from a chain source keeps the group.
- PaimonJniScannerTest
.testBackendCapStillReadsTheChainTablePayloadOfAnOlderFrontend:
the previous FE's pair stays a plain fallback pair.
- Behavior changed: No
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
|
run buildall |
|
/review |
There was a problem hiding this comment.
Static review complete for apache/doris PR #68897 at head 26eebd6752bad84f1804b1de130494899699c9cc. I found two new P2 issues, both attached inline: the chain system-source payload can expose an unpublished delta partition in $partitions, and a $files marker/read race can reach a split that Paimon's reader cannot cast. Review convergence completed in three rounds; every final-round reviewer returned NO_NEW_VALUABLE_FINDINGS. No additional user focus was supplied.
Critical checkpoints:
- Goal and proof: The change keeps
ChainGroupReadTablethrough FE/BE fallback rebuilds and gives ordinary chain data scans a clear unsupported error. The new unit tests check group shape, cap handling, and direct restored chain rows. They do not prove the actual BE system-source selection or the two metadata read cases reported inline. - Scope and conditions: The edits are focused on the chain guard and six fallback rebuild sites. I traced the
isChainTablecondition through normal, branch, and system handles, plus nested schema and snapshot options; no other concrete guard defect was found. - Concurrency: External Paimon commits can occur between FE split generation and BE reader creation. The
$filestwo-phase marker/replan path is the M2 failure. This patch adds no shared mutable state, locks, or lock ordering. - Lifecycle: FE copies and serializes both a main table payload and a separate system-source payload. BE cap rebuilding chooses the latter and caches that reconstructed wrapper. The source misses the catalog visibility pin on the former payload, causing M1. No new global initialization, reference cycle, or release issue was found.
- Configuration: No new configuration key or dynamic configuration behavior is added. Existing chain and manifest options were traced through all three leaves; no separate cap error was substantiated.
- Compatibility: No Thrift/protobuf or storage format definition changes. New BE accepts the prior FE's flattened payload. The old-BE/new-FE construction failure is already covered by inline thread 4238874498 and its BE-first rollout discussion; I did not repost it.
- Parallel paths: Ordinary chain data reads are rejected; branch reads resolve a plain leaf; system views rebuild from a separate source. The existing
$roP1 thread 4238874496 was checked against the author's Paimon SDK parity evidence and was not independently confirmed as a new regression at this head, so it is not carried as an existing blocker. - Tests and results: I inspected all changed Java tests and their assertions. No regression output was changed. The review contract prohibited builds and tests, so this is static evidence only; the author's reported test runs were not independently validated. End-to-end cases for version-managed
$partitionspublication and$filesmarker/read timing are missing. - Observability and errors: The new ordinary-data rejection includes the table name. I found no independent logging/metrics gap. M2 surfaces as a reader exception rather than a valid result.
- Transactions, persistence, and writes: No Doris EditLog, transaction, or persistent format path changes. I checked the write-binding consumer of loader removal and found no separate write regression. M1 and M2 concern the visibility of external Paimon commits during reads.
- FE/BE variables: The existing serialized system-source transport is the decisive parallel payload; no new wire key was added. The BE's unconditional rebuild from it loses the main payload's pin.
- Performance and remaining issues: Recursive work is bounded by the three chain leaves, and no separate CPU, allocation, or cache issue was substantiated. Final sweep of all nine changed files and prior candidates left no unresolved point.
| FallbackReadFileStoreTable pair = (FallbackReadFileStoreTable) table; | ||
| return PaimonTableDecorators.rebuildFallbackPair(pair, | ||
| dropCatalogLoaderOfBranches(pair.wrapped()), dropCatalogLoaderOfBranches(pair.other())); | ||
| } |
There was a problem hiding this comment.
[P2] Pin the chain system source before stripping its catalog loaders. For a plain chain-backed $partitions read in a version-managed catalog, this recursion retains the delta leaf in doris.serialized-system-source, but that source never receives the pinCatalogSnapshot applied to serializedTable. The BE always rebuilds from the side-channel source, so if delta snapshot 2 exists on storage while its catalog pointer remains at 1, Paimon's partition listing can return the unpublished delta-only partition. Carry catalog-visible coordinates for each branch into the source the BE actually uses, then test a delta-only partition in that publication window. See Paimon PartitionsRead and ChainGroupReadTable.
There was a problem hiding this comment.
Confirmed, but it is not specific to this PR or to chain tables, so not changed here.
The side-channel source has never carried the catalog pin, for any branch. The FE sends doris.serialized-system-source for every system table that has a source (backendManifestParallelismCap always returns a value), and the BE then always rebuilds the system table from that source (PaimonJniScanner.applyBackendManifestParallelism), so the pinCatalogSnapshot applied to serializedTable never reaches the BE's plan. On master that already means a $partitions or $files read in a version-managed catalog plans every branch, main and snapshot alike, on the newest snapshot in storage rather than on the catalog pointer; this has been so since #66247 added the side channel. This PR only adds the delta leaf to that source. Before it, the flattened payload had no delta leaf at all, so $partitions never listed a delta-only partition, published or not, which is the larger error.
The fix for all of these is to pin the side-channel source the way buildBackendTable pins serializedTable: before the relation's @options are applied, so that a user's time travel still wins. That changes every system-table read in a version-managed catalog, so it should be its own PR with its own tests rather than part of this chain-table one.
| // Each branch owns an independent planner setting; a smaller sibling is not an | ||
| // execution ceiling and must never throttle the other branch. | ||
| if (pair instanceof ChainGroupReadTable) { | ||
| // A chain table nests this group over its snapshot and delta branches. The group plans |
There was a problem hiding this comment.
[P2] Keep chain fallback splits out of $files's DataSplit reader. FE can emit a $files marker for main partition P, then main can drop P before BE reads that marker. With this preserved group in the system-source payload, BE replanning falls back to FallbackSplitImpl(ChainSplit) for snapshot-branch P; Paimon FilesRead casts every replanned split to DataSplit, so the query fails with ClassCastException. The previous flattened snapshot fallback returned FallbackDataSplit, which passed that cast. Align marker and reader snapshots or prevent this wrapper from reaching $files until its reader can handle the chain split, and test the drop-after-marker case.
There was a problem hiding this comment.
Confirmed, and not changed: in that window the BE now fails the way Paimon itself fails on the same table.
With this PR the BE reads $files through the table Paimon builds for a chain table. FilesTable.FilesRead.createReader replans each marker on that table and casts every split it gets back to DataSplit, and the fallback pair returns the chain split for a partition its main branch no longer has, so Spark or Flink reading t$files in the same window would raise the same ClassCastException.
The window is the two-phase $files gap the pinCatalogSnapshot javadoc already records: the FE takes its markers from the latest snapshot, and the BE replans later. With master's flattened payload the same race did not fail, but it returned the snapshot branch's files under a partition the FE had just attributed to main, that is, rows of another branch. An error is the better outcome there, and the race needs main to drop a partition between planning and the read.
TPC-H: Total hot run time: 29069 ms |
TPC-DS: Total hot run time: 151999 ms |
ClickBench: Total hot run time: 24.66 s |
FE UT Coverage ReportIncrement line coverage `` 🎉 |
FE Regression Coverage ReportIncrement line coverage |
|
PR approved by anyone and no changes requested. |
1 similar comment
|
PR approved by anyone and no changes requested. |
github-actions
Bot
added
approved
|
PR approved by at least one committer and no changes requested. |
1 similar comment
|
PR approved by at least one committer and no changes requested. |
…ble source (#69569) ### What problem does this PR solve? Issue Number: None Related PR: #68897, #69343 Problem Summary: **In short.** This PR only adds tests. Some Paimon system tables are planned on the BE. For them, the JNI scanner rebuilds the system table from the source table that the FE sends in `doris.serialized-system-source`. In a version-managed catalog, that payload must be pinned to the snapshot the catalog publishes. #69343 added this pin. This PR adds the two cases its tests do not cover: a `scan.fallback-branch` pair, and an explicit relation snapshot that must still take precedence over the pin. **Background.** Six Paimon system tables are planned on the BE: `$files`, `$file_indexes`, `$partitions`, `$manifests`, `$statistics` and `$table_indexes` (`PaimonScanParams#resolvesSnapshotOnBackend`). For these tables the FE sends two payloads: - `serialized_table` is built by `tableForBackend`. - `doris.serialized-system-source` was added in #66247 so that a smaller BE can cap the manifest parallelism. The FE always sends it when the system table has a source. When it is present, `PaimonJniScanner#applyBackendManifestParallelism` rebuilds the system table from this payload instead of from `serialized_table`. A version-managed catalog (REST) owns the committed snapshot pointer. Once the catalog loader is removed, Paimon falls back to listing the snapshot directory, and that directory can already contain a snapshot the catalog has not published. #65867 therefore pins `scan.snapshot-id` to the catalog's pointer (`PaimonScanPlanProvider#pinCatalogSnapshot`). **The problem, and what it cost.** Before #69343, only `serialized_table` carried that pin. The side channel shipped the source unpinned, and the BE rebuilt from the side channel. So in a REST catalog these six system tables were planned on the newest snapshot in storage rather than the published one, on every branch. Inside the publication window, `$partitions` could list partitions of a snapshot the catalog had not published yet. The review of #68897 pointed this out for chain tables. #69343 (the Paimon 2.1.0 upgrade) now prepares the side-channel source in the same order as `tableForBackend` (`prepareSystemSource`): 1. Pin the catalog snapshot, branch by branch. 2. Apply the relation's `@options` over the pin. 3. Drop the catalog loader. Its tests cover `$file_indexes` on a single-branch table, on the FE and through the JNI rebuild. **How this PR fixes it.** Two FE unit tests in `PaimonBackendBoundTableTest` read the side channel out of `paimon.options_json`: - `systemSourcePinsEachFallbackBranchToItsOwnCatalogSnapshot` builds `$partitions` over a fallback pair whose two catalogs point at snapshots 2 and 1. Each branch of the transported source carries its own pointer and no catalog loader. A chain table nests the same pairs. - `relationSnapshotOutranksTheCatalogPinOnTheSystemSource` uses a catalog that points at snapshot 2 and `@options('scan.snapshot-id'='1')`. The transported source says 1. Paimon 2.1.0 still has the two limitations documented on `pinCatalogSnapshot`: - `$files` emits its marker splits from `SnapshotReader#partitions`. - `$snapshots` and `$buckets` read without `scan.snapshot-id`. So that javadoc stays as written. **Results.** | `prepareSystemSource` | `PaimonBackendBoundTableTest` | |---|---| | as on master (pin, then `@options`) | 38 / 38 pass | | without the pin | the new fallback case and the existing `$file_indexes` case fail | | pin applied after the relation options | the new relation-snapshot case fails (`expected: <1> but was: <2>`) | All 847 tests of fe-connector-paimon pass, with 1 skipped as before, and checkstyle passes. No regression suite is added: it would need a version-managed (REST) Paimon catalog whose pointer lags the snapshot directory, and the docker fixtures do not provide one.
What problem does this PR solve?
Issue Number: None
Related PR: #68885
Problem Summary:
In short. Master cannot read a Paimon 1.4 chain table: every data scan fails in planning with Paimon's internal
This is a bug, perhaps there is a recursive call.This PR keeps the chain group intact wherever the Paimon connector and the BE JNI scanner rebuild a fallback pair, and makes a data scan of a chain table fail with a clear "not supported yet" message, because the rest of the scan path does not plan chain partitions and chain splits yet. The review of #68885 (closed since) found the BE half of the problem.Background. Paimon 1.4 builds a chain table (
chain-table.enabledwithscan.fallback-snapshot-branchandscan.fallback-delta-branch) asFallbackReadFileStoreTable(main, ChainGroupReadTable(snapshot, delta), true).ChainGroupReadTableis itself a fallback pair. It plans and reads the chain: a partition missing from the snapshot branch is read as the latest earlier snapshot partition merged with the delta branch. Paimon'sFallbackReadFileStoreTableconstructor rejects any other pair nested inside a pair. The Paimon connector and the BE JNI scanner rebuild fallback pairs branch by branch in six places:PaimonJniScanner, taken when the FE cap is below the BE's core count) and on the FE (PaimonReaderOptions);dropCatalogLoader) and the catalog snapshot pin (pinCatalogSnapshot) of the table sent to the BE;copyWithPinnedFallback) and bound schemas (restoreBoundSchema).The problem, and what it cost. Each of these recursed into the chain group and rebuilt it either as a plain pair, which the outer constructor rejects, or as a plain table of the snapshot branch, which silently drops the delta branch. On a local FE + BE with a file-system Paimon catalog and a chain table created and written through the Paimon API:
SELECT * FROM chain_tThis is a bug, perhaps there is a recursive call.(first fromPaimonScanParams.restoreBoundSchema)$filesscan when the FE cap is below the BE's core countKeeping the chain group is not enough to read a chain table. Doris plans neither the partitions a chain table reads from its branches nor the chain splits. With only the rebuilds fixed, a chain table whose main branch is empty reads as empty (0 partitions, 0 splits), and one with main data fails with
Fail to parse columnsFromPath, expected: [dt], filePath: connector://virtual. branch-4.x has the same manifest-bound rebuild and no chain support either.How this PR fixes it.
PaimonTableDecorators.rebuildFallbackPairrebuilds aChainGroupReadTableas a group and any other pair as before. The FE manifest bound,copyWithPinnedFallbackandrestoreBoundSchemause it, anddropCatalogLoaderandpinCatalogSnapshotnow recurse into the nested pair. The BE manifest bound keeps the group the same way.PaimonScanPlanProvider.resolveScanTablerejects a data scan of a chain table (a fallback pair over aChainGroupReadTable) withReading Paimon chain table db.tbl (chain-table.enabled) is not supported yet. System tables and a branch read on its own (@branch(...)) are plain tables and are not affected.Results. Same cluster and table:
SELECT ... FROM chain_tThis is a bug, perhaps there is a recursive call.Reading Paimon chain table d.chain_t (chain-table.enabled) is not supported yetchain_t$fileschain_t@branch(snapshot),chain_t@branch(delta)System tables over a chain table now read through the chain group too. Paimon resolves
$partitionson the backend, and for a chain table whose delta branch alone holds partition 20240102 it returns[20231231, 20240101]from master's backend payload and[20231231, 20240101, 20240102], Paimon's own answer, from this PR's (Paimon 1.4.2 in jshell on the same payloads, not on the cluster).chain_t$rois unchanged: it returns Paimon's own read-optimized rows, which leave out a partition only the delta branch holds.Release note
Reading a Paimon chain table (
chain-table.enabled) fails withReading Paimon chain table ... is not supported yetinstead of an internal Paimon error.Check List (For Author)
Test
PaimonJniScannerTest.testBackendManifestCapKeepsTheChainGroupOfAChainTable,PaimonReaderOptionsTest.testRuntimeSafeTableKeepsTheChainGroupOfAChainTable: the BE and FE manifest bounds keep the chain group. The BE case fails on master with the recursive-call error.PaimonBackendBoundTableTest.chainTableKeepsItsChainGroupWithoutTheCatalogLoader: on master the chain group comes back as a plainPrimaryKeyFileStoreTable.PaimonBackendBoundTableTest.scanOfAChainTableFailsClearly: a data scan of a chain table fails, and a branch read on its own resolves.PaimonStatementSchemaTest.statementPinKeepsTheChainOfAChainTable: a real chain table, pinned and restored, keeps the group and reads the same rows as Paimon's own read at that snapshot.PaimonJniScannerTest.testBackendCapKeepsTheChainGroupOfAChainTableSystemSource(a$partitionswrapper rebuilt from a chain source keeps the group) andtestBackendCapStillReadsTheChainTablePayloadOfAnOlderFrontend(the previous FE's payload, with the group replaced by a plain table at the snapshot branch, stays a plain pair). Rebuilding the group as a plain pair, or every pair as a group, makes them fail; the paimon-scanner module tests (116) pass.038e2274ef6; the connector code it touches is the same there.Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)