Skip to content

Commit 76fb880

Browse files
authored
[test](paimon) Cover the catalog snapshot pin on the Paimon system-table source (apache#69569)
### What problem does this PR solve? Issue Number: None Related PR: apache#68897, apache#69343 Problem Summary: **In short.** This PR only adds tests. Some Paimon system tables are planned on the BE. For them, the JNI scanner rebuilds the system table from the source table that the FE sends in `doris.serialized-system-source`. In a version-managed catalog, that payload must be pinned to the snapshot the catalog publishes. apache#69343 added this pin. This PR adds the two cases its tests do not cover: a `scan.fallback-branch` pair, and an explicit relation snapshot that must still take precedence over the pin. **Background.** Six Paimon system tables are planned on the BE: `$files`, `$file_indexes`, `$partitions`, `$manifests`, `$statistics` and `$table_indexes` (`PaimonScanParams#resolvesSnapshotOnBackend`). For these tables the FE sends two payloads: - `serialized_table` is built by `tableForBackend`. - `doris.serialized-system-source` was added in apache#66247 so that a smaller BE can cap the manifest parallelism. The FE always sends it when the system table has a source. When it is present, `PaimonJniScanner#applyBackendManifestParallelism` rebuilds the system table from this payload instead of from `serialized_table`. A version-managed catalog (REST) owns the committed snapshot pointer. Once the catalog loader is removed, Paimon falls back to listing the snapshot directory, and that directory can already contain a snapshot the catalog has not published. apache#65867 therefore pins `scan.snapshot-id` to the catalog's pointer (`PaimonScanPlanProvider#pinCatalogSnapshot`). **The problem, and what it cost.** Before apache#69343, only `serialized_table` carried that pin. The side channel shipped the source unpinned, and the BE rebuilt from the side channel. So in a REST catalog these six system tables were planned on the newest snapshot in storage rather than the published one, on every branch. Inside the publication window, `$partitions` could list partitions of a snapshot the catalog had not published yet. The review of apache#68897 pointed this out for chain tables. apache#69343 (the Paimon 2.1.0 upgrade) now prepares the side-channel source in the same order as `tableForBackend` (`prepareSystemSource`): 1. Pin the catalog snapshot, branch by branch. 2. Apply the relation's `@options` over the pin. 3. Drop the catalog loader. Its tests cover `$file_indexes` on a single-branch table, on the FE and through the JNI rebuild. **How this PR fixes it.** Two FE unit tests in `PaimonBackendBoundTableTest` read the side channel out of `paimon.options_json`: - `systemSourcePinsEachFallbackBranchToItsOwnCatalogSnapshot` builds `$partitions` over a fallback pair whose two catalogs point at snapshots 2 and 1. Each branch of the transported source carries its own pointer and no catalog loader. A chain table nests the same pairs. - `relationSnapshotOutranksTheCatalogPinOnTheSystemSource` uses a catalog that points at snapshot 2 and `@options('scan.snapshot-id'='1')`. The transported source says 1. Paimon 2.1.0 still has the two limitations documented on `pinCatalogSnapshot`: - `$files` emits its marker splits from `SnapshotReader#partitions`. - `$snapshots` and `$buckets` read without `scan.snapshot-id`. So that javadoc stays as written. **Results.** | `prepareSystemSource` | `PaimonBackendBoundTableTest` | |---|---| | as on master (pin, then `@options`) | 38 / 38 pass | | without the pin | the new fallback case and the existing `$file_indexes` case fail | | pin applied after the relation options | the new relation-snapshot case fails (`expected: <1> but was: <2>`) | All 847 tests of fe-connector-paimon pass, with 1 skipped as before, and checkstyle passes. No regression suite is added: it would need a version-managed (REST) Paimon catalog whose pointer lags the snapshot directory, and the docker fixtures do not provide one.
1 parent 20ee7a3 commit 76fb880

1 file changed

Lines changed: 65 additions & 0 deletions

File tree

‎fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonBackendBoundTableTest.java‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -562,6 +562,63 @@ public void fileIndexesReadOnlyTheCatalogPublishedSnapshot(@TempDir Path warehou
562562
Assertions.assertEquals(2L, onDisk.snapshotManager().latestSnapshotIdFromFileSystem());
563563
}
564564

565+
@Test
566+
public void systemSourcePinsEachFallbackBranchToItsOwnCatalogSnapshot(@TempDir Path warehouse)
567+
throws Exception {
568+
// JNI rebuilds a system table that plans on the BE from doris.serialized-system-source rather
569+
// than from serialized_table, so that payload must carry each branch's own catalog pointer too.
570+
FileStoreTable onDisk = commit(newRealTable(warehouse, "fallback_system_source"), 2);
571+
VersionManagedCatalog mainCatalog = new VersionManagedCatalog(warehouse, true);
572+
VersionManagedCatalog fallbackCatalog = new VersionManagedCatalog(warehouse, true);
573+
mainCatalog.pointAt(onDisk, 2L);
574+
fallbackCatalog.pointAt(onDisk, 1L);
575+
Map<String, String> mainOptions = new HashMap<>();
576+
mainOptions.put("scan.fallback-branch", "fb");
577+
Map<String, String> fallbackOptions = new HashMap<>();
578+
fallbackOptions.put("branch", "fb");
579+
FileStoreTable pair = new FallbackReadFileStoreTable(
580+
withCatalogEnvironment(onDisk.copyWithoutTimeTravel(mainOptions),
581+
catalogEnvironment(mainCatalog)),
582+
withCatalogEnvironment(onDisk.copyWithoutTimeTravel(fallbackOptions),
583+
catalogEnvironment(fallbackCatalog)),
584+
true);
585+
PaimonTableHandle handle = sysHandle("partitions", pair);
586+
handle.setPaimonTable(SystemTableLoader.load("partitions", pair));
587+
handle.setSystemTableSource(pair);
588+
589+
FileStoreTable transported = transportedSystemSource(handle);
590+
591+
Assertions.assertTrue(transported instanceof FallbackReadFileStoreTable);
592+
FallbackReadFileStoreTable transportedPair = (FallbackReadFileStoreTable) transported;
593+
Assertions.assertEquals("2", transportedPair.wrapped().options().get("scan.snapshot-id"));
594+
Assertions.assertEquals("1", transportedPair.other().options().get("scan.snapshot-id"),
595+
"the fallback branch must use its own catalog pointer, not its newest snapshot file");
596+
Assertions.assertNull(transportedPair.wrapped().catalogEnvironment().catalogLoader());
597+
Assertions.assertNull(transportedPair.other().catalogEnvironment().catalogLoader());
598+
}
599+
600+
@Test
601+
public void relationSnapshotOutranksTheCatalogPinOnTheSystemSource(@TempDir Path warehouse)
602+
throws Exception {
603+
// The catalog pin is taken first and the relation's @options are applied over it, so an explicit
604+
// snapshot still selects what the BE rebuilds the system table on.
605+
VersionManagedCatalog catalog = new VersionManagedCatalog(warehouse, true);
606+
FileStoreTable onDisk = commit(newRealTable(warehouse, "options_system_source"), 2);
607+
catalog.pointAt(onDisk, 2L);
608+
FileStoreTable versionManaged = withCatalogEnvironment(onDisk, catalogEnvironment(catalog));
609+
Map<String, String> relationOptions = PaimonScanParams.markAsOptions(
610+
Collections.singletonMap("scan.snapshot-id", "1"));
611+
PaimonTableHandle handle = sysHandle("partitions", versionManaged).withScanOptions(relationOptions);
612+
handle.setPaimonTable(PaimonScanParams.applyOptions(
613+
SystemTableLoader.load("partitions", versionManaged), relationOptions));
614+
handle.setSystemTableSource(versionManaged);
615+
616+
FileStoreTable transported = transportedSystemSource(handle);
617+
618+
Assertions.assertEquals("1", transported.options().get("scan.snapshot-id"));
619+
Assertions.assertTrue(catalog.loadSnapshotCalls > 0, "the catalog pin is still resolved on the FE");
620+
}
621+
565622
@Test
566623
public void keepsThePinOffSystemTablesWhoseRowTypeFollowsTheBaseTable(@TempDir Path warehouse)
567624
throws Exception {
@@ -1023,6 +1080,14 @@ private static void assertPairMatchesTheFeGeneration(FileStoreTable forBackend,
10231080
}
10241081

10251082
/** Decodes {@code paimon.serialized_table} the way {@code PaimonJniScanner#initTable} does. */
1083+
/** The catalog-less source that JNI rebuilds a backend-planned system table from. */
1084+
private static FileStoreTable transportedSystemSource(PaimonTableHandle handle) throws Exception {
1085+
Map<String, String> props = provider().getScanNodeProperties(null, handle,
1086+
Collections.emptyList(), Optional.empty());
1087+
Map<?, ?> options = JsonSerdeUtil.fromJson(props.get("paimon.options_json"), Map.class);
1088+
return deserializeTable((String) options.get("doris.serialized-system-source"));
1089+
}
1090+
10261091
private static FileStoreTable deserializeTable(String encoded) throws Exception {
10271092
byte[] raw = encoded.getBytes(StandardCharsets.UTF_8);
10281093
byte[] bytes;

0 commit comments

Comments
 (0)