Repository navigation
Commit 76fb880
authored
[test](paimon) Cover the catalog snapshot pin on the Paimon system-table source (apache#69569)
### What problem does this PR solve?
Issue Number: None
Related PR: apache#68897, apache#69343
Problem Summary:
**In short.** This PR only adds tests. Some Paimon system tables are
planned on the BE. For them, the JNI scanner rebuilds the system table
from the source table that the FE sends in
`doris.serialized-system-source`. In a version-managed catalog, that
payload must be pinned to the snapshot the catalog publishes. apache#69343
added this pin. This PR adds the two cases its tests do not cover: a
`scan.fallback-branch` pair, and an explicit relation snapshot that must
still take precedence over the pin.
**Background.** Six Paimon system tables are planned on the BE:
`$files`, `$file_indexes`, `$partitions`, `$manifests`, `$statistics`
and `$table_indexes` (`PaimonScanParams#resolvesSnapshotOnBackend`). For
these tables the FE sends two payloads:
- `serialized_table` is built by `tableForBackend`.
- `doris.serialized-system-source` was added in apache#66247 so that a smaller
BE can cap the manifest parallelism. The FE always sends it when the
system table has a source. When it is present,
`PaimonJniScanner#applyBackendManifestParallelism` rebuilds the system
table from this payload instead of from `serialized_table`.
A version-managed catalog (REST) owns the committed snapshot pointer.
Once the catalog loader is removed, Paimon falls back to listing the
snapshot directory, and that directory can already contain a snapshot
the catalog has not published. apache#65867 therefore pins `scan.snapshot-id`
to the catalog's pointer (`PaimonScanPlanProvider#pinCatalogSnapshot`).
**The problem, and what it cost.** Before apache#69343, only
`serialized_table` carried that pin. The side channel shipped the source
unpinned, and the BE rebuilt from the side channel. So in a REST catalog
these six system tables were planned on the newest snapshot in storage
rather than the published one, on every branch. Inside the publication
window, `$partitions` could list partitions of a snapshot the catalog
had not published yet. The review of apache#68897 pointed this out for chain
tables.
apache#69343 (the Paimon 2.1.0 upgrade) now prepares the side-channel source
in the same order as `tableForBackend` (`prepareSystemSource`):
1. Pin the catalog snapshot, branch by branch.
2. Apply the relation's `@options` over the pin.
3. Drop the catalog loader.
Its tests cover `$file_indexes` on a single-branch table, on the FE and
through the JNI rebuild.
**How this PR fixes it.** Two FE unit tests in
`PaimonBackendBoundTableTest` read the side channel out of
`paimon.options_json`:
- `systemSourcePinsEachFallbackBranchToItsOwnCatalogSnapshot` builds
`$partitions` over a fallback pair whose two catalogs point at snapshots
2 and 1. Each branch of the transported source carries its own pointer
and no catalog loader. A chain table nests the same pairs.
- `relationSnapshotOutranksTheCatalogPinOnTheSystemSource` uses a
catalog that points at snapshot 2 and
`@options('scan.snapshot-id'='1')`. The transported source says 1.
Paimon 2.1.0 still has the two limitations documented on
`pinCatalogSnapshot`:
- `$files` emits its marker splits from `SnapshotReader#partitions`.
- `$snapshots` and `$buckets` read without `scan.snapshot-id`.
So that javadoc stays as written.
**Results.**
| `prepareSystemSource` | `PaimonBackendBoundTableTest` |
|---|---|
| as on master (pin, then `@options`) | 38 / 38 pass |
| without the pin | the new fallback case and the existing
`$file_indexes` case fail |
| pin applied after the relation options | the new relation-snapshot
case fails (`expected: <1> but was: <2>`) |
All 847 tests of fe-connector-paimon pass, with 1 skipped as before, and
checkstyle passes. No regression suite is added: it would need a
version-managed (REST) Paimon catalog whose pointer lags the snapshot
directory, and the docker fixtures do not provide one.1 parent 20ee7a3 commit 76fb880
1 file changed
Lines changed: 65 additions & 0 deletions
File tree
- fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
562 | 562 | | |
563 | 563 | | |
564 | 564 | | |
| 565 | + | |
| 566 | + | |
| 567 | + | |
| 568 | + | |
| 569 | + | |
| 570 | + | |
| 571 | + | |
| 572 | + | |
| 573 | + | |
| 574 | + | |
| 575 | + | |
| 576 | + | |
| 577 | + | |
| 578 | + | |
| 579 | + | |
| 580 | + | |
| 581 | + | |
| 582 | + | |
| 583 | + | |
| 584 | + | |
| 585 | + | |
| 586 | + | |
| 587 | + | |
| 588 | + | |
| 589 | + | |
| 590 | + | |
| 591 | + | |
| 592 | + | |
| 593 | + | |
| 594 | + | |
| 595 | + | |
| 596 | + | |
| 597 | + | |
| 598 | + | |
| 599 | + | |
| 600 | + | |
| 601 | + | |
| 602 | + | |
| 603 | + | |
| 604 | + | |
| 605 | + | |
| 606 | + | |
| 607 | + | |
| 608 | + | |
| 609 | + | |
| 610 | + | |
| 611 | + | |
| 612 | + | |
| 613 | + | |
| 614 | + | |
| 615 | + | |
| 616 | + | |
| 617 | + | |
| 618 | + | |
| 619 | + | |
| 620 | + | |
| 621 | + | |
565 | 622 | | |
566 | 623 | | |
567 | 624 | | |
| |||
1023 | 1080 | | |
1024 | 1081 | | |
1025 | 1082 | | |
| 1083 | + | |
| 1084 | + | |
| 1085 | + | |
| 1086 | + | |
| 1087 | + | |
| 1088 | + | |
| 1089 | + | |
| 1090 | + | |
1026 | 1091 | | |
1027 | 1092 | | |
1028 | 1093 | | |
| |||
0 commit comments