Skip to content

Commit cf79a1a

Browse files
morningmanclaude
andcommitted
[fix](paimon) Check binary partitions of fallback splits under the options of their branch
### What problem does this PR solve? Issue Number: None Related PR: #69571 Problem Summary: Follow-up to the scan-planning check of binary partitions in this PR, for two cases found in review. The check takes the naming option and the file schemas from the branch that produced each split, but only when the table has a schema dictionary. For $audit_log and $row_tracking, which are read through JNI, there is none, so the check used the wrapper's own options, and an AuditLogTable reports the options of the fallback pair, which are the main branch's. With scan.fallback-branch set, a partition missing on the main branch is still planned and read under the fallback branch's options. A table whose main branch keeps the default 'partition.legacy-name' = 'true' and holds only a NULL partition, while its fallback branch uses 'false' and holds a non-NULL binary partition, was therefore rejected for $audit_log although Paimon reads the fallback rows; in the reverse layout the scan planned and Paimon's reader failed with FileNotFoundException at execution. For the wrappers read through JNI the check now resolves the table that produced the splits from the wrapper's pinned source, as the timestamp-history checks already do, and picks each split's branch from it as it does for data tables. $binlog exposes its partition columns as arrays and is not checked. A scan.file-creation-time-millis read plans a fallback table through the pair's SnapshotReader, which reads the wrapped branch and returns plain DataSplits. Picking the branch cast every split to FallbackSplit, so such a scan of a table with a non-NULL binary partition failed with a ClassCastException (the same cast on the native timestamp path predates this PR). A split that is not a FallbackSplit now maps to the wrapped branch. ### Release note None (follow-up inside the PR) ### Check List (For Author) - Test: Unit Test - PaimonScanPlanProviderTest: a fallback pair whose main branch holds only a NULL partition and whose fallback branch holds a non-NULL binary partition plans the data table and $audit_log, and the $audit_log splits read back both rows through Paimon, when the fallback branch names partitions by bytes; both fail at planning with the legacy-name error when only the main branch does; a scan.file-creation-time-millis read of a fallback pair with a non-NULL binary partition plans the main branch's file - fe-connector-paimon module tests: 853 run, 0 failures, 0 errors, 1 skipped (pre-existing) - Behavior changed: Yes. $audit_log and $row_tracking of a fallback table check binary partitions under the options of the branch that produced each split, and a file-creation-time read of a fallback table with a binary partition no longer fails with a ClassCastException. - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent 65a1682 commit cf79a1a

2 files changed

Lines changed: 117 additions & 4 deletions

File tree

‎fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java‎

Lines changed: 16 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -992,6 +992,12 @@ && hasTrustworthyLimitAccounting(table)
992992
// $ro wraps the pinned file-store table; resolve its schema dictionary once, only if native is considered.
993993
java.util.function.Supplier<Table> legacyOrcSchemaTable = com.google.common.base.Suppliers.memoize(
994994
() -> resolveSchemaDictTable(table, paimonHandle));
995+
// The wrappers read through JNI report the options of the main branch, but their splits come from a branch
996+
// of the pinned source, which the timestamp-history checks resolve the same way.
997+
java.util.function.Supplier<Table> binaryPartitionSourceTable = paimonHandle.isForceJni()
998+
? com.google.common.base.Suppliers.memoize(() -> withSchemaHistoryAuthentication(
999+
() -> resolveTimestampHistoryTable(table, paimonHandle)))
1000+
: legacyOrcSchemaTable;
9951001
int[] binaryPartitionFields = binaryPartitionFields(table);
9961002
Map<FileStoreTable, Map<Long, Boolean>> legacyPartitionNameSchemas = new java.util.IdentityHashMap<>();
9971003
Map<String, Boolean> fileExistence = new HashMap<>();
@@ -1006,7 +1012,7 @@ && hasTrustworthyLimitAccounting(table)
10061012
}
10071013
if (binaryPartitionFields.length > 0) {
10081014
checkBinaryPartitionFiles(paimonHandle, table, dataSplit, binaryPartitionFields,
1009-
legacyOrcSchemaTable, legacyPartitionNameSchemas, fileExistence);
1015+
binaryPartitionSourceTable, legacyPartitionNameSchemas, fileExistence);
10101016
}
10111017

10121018
Map<String, String> partitionValues = getPartitionInfoMap(
@@ -2294,6 +2300,11 @@ static FileStoreTable legacyOrcSplitTable(FileStoreTable table, DataSplit split)
22942300
FileStoreTable base = PaimonTableDecorators.unwrapToFallbackOrBase(table);
22952301
if (base instanceof FallbackReadFileStoreTable) {
22962302
FallbackReadFileStoreTable pair = (FallbackReadFileStoreTable) base;
2303+
if (!(split instanceof FallbackReadFileStoreTable.FallbackSplit)) {
2304+
// A split planned through the pair's SnapshotReader (a file-creation-time read) comes from its
2305+
// wrapped branch.
2306+
return pair.wrapped();
2307+
}
22972308
boolean fallback = ((FallbackReadFileStoreTable.FallbackSplit) split).isFallback();
22982309
// Match the SDK reader's branch selection before reading either options or historical schemas.
22992310
return PaimonReaderOptions.isWrappedFirst(pair) != fallback ? pair.wrapped() : pair.other();
@@ -2316,9 +2327,10 @@ private static int[] binaryPartitionFields(Table table) {
23162327
* is the Java array object ({@code [B@1b6d3586}) rather than its bytes, and every resolution of the name
23172328
* differs. A file written under such a name, or looked up under one, is never found: Paimon's own read fails,
23182329
* while the BE skips a missing file of an external table and would silently drop the partition's rows. A file
2319-
* with an external path is found by the path its metadata records. Without a schema dictionary (system tables
2320-
* other than {@code $ro}, which the JNI reader reads with Paimon's own reader) the schemas of the files are not
2321-
* consulted.
2330+
* with an external path is found by the path its metadata records. A system wrapper reports the options of the
2331+
* main branch while a fallback split is planned and read under its own branch, so the options and schemas come
2332+
* from the branch that produced the split; for the wrappers read through JNI that branch is resolved from the
2333+
* pinned source of the wrapper.
23222334
*
23232335
* <p>The schema a file refers to does not prove how the file was named, because dynamic options
23242336
* ({@code spark.paimon.*}, Flink hints, {@code Table.copy}) keep the schema ID: a file whose schema uses legacy

‎fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonScanPlanProviderTest.java‎

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -80,6 +80,7 @@
8080
import org.apache.paimon.table.source.RawFile;
8181
import org.apache.paimon.table.source.Split;
8282
import org.apache.paimon.table.system.ReadOptimizedTable;
83+
import org.apache.paimon.table.system.SystemTableLoader;
8384
import org.apache.paimon.types.DataField;
8485
import org.apache.paimon.types.DataType;
8586
import org.apache.paimon.types.DataTypes;
@@ -1085,6 +1086,91 @@ public void externalBinaryPartitionDeletionVectorAtExternalPathStaysReadable(@Te
10851086
}
10861087
}
10871088

1089+
@Test
1090+
public void fallbackBinaryPartitionsUseTheNamingOfTheirBranch(@TempDir Path warehouse) throws Exception {
1091+
// A partition missing on the main branch is planned and read under the options of the fallback branch,
1092+
// while the table and its $audit_log wrapper report the options of the main branch.
1093+
for (boolean fallbackUsesBytes : new boolean[] {true, false}) {
1094+
try (Catalog catalog = new FileSystemCatalog(LocalFileIO.create(),
1095+
new org.apache.paimon.fs.Path(warehouse.resolve("fallback_bytes_" + fallbackUsesBytes).toUri()))) {
1096+
catalog.createDatabase("db", false);
1097+
Map<String, String> byBytes = Collections.singletonMap("partition.legacy-name", "false");
1098+
// The main branch only holds the NULL partition, which every naming finds.
1099+
FileStoreTable main = (FileStoreTable) binaryPartitionTable(catalog, "main",
1100+
fallbackUsesBytes ? Collections.emptyMap() : byBytes, (byte[]) null);
1101+
FileStoreTable fallback = (FileStoreTable) binaryPartitionTable(catalog, "fallback",
1102+
fallbackUsesBytes ? byBytes : Collections.emptyMap(), new byte[] {0x41, 0x42});
1103+
FallbackReadFileStoreTable pair = new FallbackReadFileStoreTable(main, fallback, true);
1104+
if (!fallbackUsesBytes) {
1105+
Map<String, Supplier<List<ConnectorScanRange>>> plans = new LinkedHashMap<>();
1106+
plans.put("table", () -> planBinaryPartitionScan(pair, false));
1107+
plans.put("$audit_log", () -> planSystemTableScan("audit_log", pair));
1108+
plans.forEach((name, plan) -> {
1109+
DorisConnectorException error = Assertions.assertThrows(DorisConnectorException.class,
1110+
plan::get, name);
1111+
Assertions.assertTrue(error.getMessage().startsWith("Cannot read Paimon table db.main: the"
1112+
+ " table uses legacy partition names ('partition.legacy-name' = 'true'), under which"
1113+
+ " Paimon names partition key = 0x4142"), error.getMessage());
1114+
});
1115+
continue;
1116+
}
1117+
List<ConnectorScanRange> ranges = planBinaryPartitionScan(pair, false);
1118+
Assertions.assertEquals(2, ranges.size());
1119+
for (ConnectorScanRange range : ranges) {
1120+
String path = range.getPath().orElseThrow(AssertionError::new);
1121+
Assertions.assertTrue(LocalFileIO.create().exists(new org.apache.paimon.fs.Path(path)), path);
1122+
}
1123+
Assertions.assertTrue(ranges.stream().anyMatch(range -> range.getPath().get().contains("/key=AB/")));
1124+
1125+
List<ConnectorScanRange> auditLog = planSystemTableScan("audit_log", pair);
1126+
Assertions.assertEquals(2, auditLog.size());
1127+
List<String> rows = new ArrayList<>();
1128+
SystemTableLoader.load("audit_log", pair).newReadBuilder().newRead()
1129+
.createReader(deserializeJniSplits(auditLog))
1130+
.forEachRemaining(row -> rows.add(row.getString(0) + " " + (row.isNullAt(2) ? "NULL"
1131+
: java.util.HexFormat.of().withUpperCase().formatHex(row.getBinary(2)))));
1132+
rows.sort(Comparator.naturalOrder());
1133+
Assertions.assertEquals(Arrays.asList("+I 4142", "+I NULL"), rows);
1134+
}
1135+
}
1136+
}
1137+
1138+
@Test
1139+
public void fileCreationTimeScanOfFallbackTableChecksBinaryPartitionsOfTheMainBranch(@TempDir Path warehouse)
1140+
throws Exception {
1141+
// A scan.file-creation-time-millis read plans a fallback table through its SnapshotReader, which reads the
1142+
// main branch and returns plain DataSplits rather than the FallbackSplits of the fallback scan.
1143+
try (Catalog catalog = new FileSystemCatalog(LocalFileIO.create(),
1144+
new org.apache.paimon.fs.Path(warehouse.toUri()))) {
1145+
catalog.createDatabase("db", false);
1146+
Map<String, String> byBytes = Collections.singletonMap("partition.legacy-name", "false");
1147+
FileStoreTable main = (FileStoreTable) binaryPartitionTable(catalog, "main", byBytes,
1148+
new byte[] {0x41, 0x42});
1149+
FileStoreTable fallback = (FileStoreTable) binaryPartitionTable(catalog, "fallback", byBytes,
1150+
new byte[] {0x43, 0x44});
1151+
FallbackReadFileStoreTable pair = new FallbackReadFileStoreTable(main, fallback, true);
1152+
RecordingPaimonCatalogOps ops = new RecordingPaimonCatalogOps();
1153+
ops.table = pair;
1154+
PaimonScanPlanProvider provider = new PaimonScanPlanProvider(
1155+
PaimonCatalogProperties.of(Collections.emptyMap()), ops);
1156+
PaimonTableHandle handle = new PaimonTableHandle("db", "main", Collections.emptyList(),
1157+
Collections.emptyList()).withScanOptions(PaimonScanParams.markAsOptions(
1158+
PaimonScanParams.resolveOptions(pair, Collections.singletonMap(
1159+
CoreOptions.SCAN_FILE_CREATION_TIME_MILLIS.key(), "0"))));
1160+
List<ConnectorScanRange> ranges = provider.planScan(
1161+
sessionWithProps(Collections.emptyMap(), new TestStatementScope()),
1162+
ConnectorScanRequest.builder(handle, Arrays.asList(new PaimonColumnHandle("id", 0),
1163+
new PaimonColumnHandle("key", 1))).build());
1164+
Assertions.assertEquals(1, ranges.size());
1165+
String path = ranges.get(0).getPath().orElseThrow(AssertionError::new);
1166+
Assertions.assertTrue(path.contains("/key=AB/"), path);
1167+
1168+
DataSplit split = (DataSplit) pair.newSnapshotReader().read().splits().get(0);
1169+
Assertions.assertFalse(split instanceof FallbackReadFileStoreTable.FallbackSplit);
1170+
Assertions.assertSame(main, PaimonScanPlanProvider.legacyOrcSplitTable(pair, split));
1171+
}
1172+
}
1173+
10881174
private static Table binaryPartitionTable(Catalog catalog, String name, Map<String, String> options,
10891175
byte[]... keys) throws Exception {
10901176
Identifier id = Identifier.create("db", name);
@@ -1120,6 +1206,21 @@ private static List<ConnectorScanRange> planBinaryPartitionScan(Table table, boo
11201206
new PaimonColumnHandle("key", 1))).countPushdown(countPushdown).build());
11211207
}
11221208

1209+
/** Plans a scan of {@code source$kind} the way the connector metadata builds its system table handle. */
1210+
private static List<ConnectorScanRange> planSystemTableScan(String kind, FileStoreTable source) {
1211+
RecordingPaimonCatalogOps ops = new RecordingPaimonCatalogOps();
1212+
ops.table = source;
1213+
PaimonScanPlanProvider provider = new PaimonScanPlanProvider(
1214+
PaimonCatalogProperties.of(Collections.emptyMap()), ops);
1215+
PaimonTableHandle handle = PaimonTableHandle.forSystemTable("db", source.name(), kind,
1216+
PaimonScanParams.requiresPaimonReader(kind));
1217+
handle.setPaimonTable(SystemTableLoader.load(kind, source));
1218+
handle.setSysBaseTable(source);
1219+
handle.setSystemTableSource(source);
1220+
return provider.planScan(sessionWithProps(Collections.emptyMap(), new TestStatementScope()),
1221+
ConnectorScanRequest.builder(handle, Collections.emptyList()).build());
1222+
}
1223+
11231224
private static Table deletionVectorBinaryPartitionTable(Catalog catalog, String name, Path warehouse)
11241225
throws Exception {
11251226
Map<String, String> options = new HashMap<>();

0 commit comments

Comments
 (0)