Skip to content

Commit 7d16729

Browse files
authored
[fix](paimon) Preserve schema options across renamed-field reads (apache#69545)
### What problem does this PR solve? After renaming a Paimon sequence or aggregation field, current and historical reads can fail with a missing-field error. Paimon's table copy selects the historical columns but overlays current table options, so `sequence.field` and `fields.<column>.aggregate-function` can reference names that do not exist in the selected schema. JNI transport also rebuilds tables without their catalog loader. The SDK factory can reapply a snapshot selector during that rebuild, reverting a schema-only rename before the bound schema is restored. Build with selectors temporarily removed and then restore the complete bound schema. Restore the selected schema and its physical options together before applying query options. Preserve the resolved table's read-state keys and branch policy, including absent keys, so historical options cannot resurrect a removed scan mode or change fallback priority during parallelism normalization. Preserve explicit catalog overrides and the SDK's fallback-branch snapshot translation. Apply the same logic to scan planning, metadata, statistics, and the source used to rebuild system-table wrappers, so binding and JNI serialization agree. Add real-table tests for snapshot, timestamp, tag, and OPTIONS reads after renames, current reads after schema-only renames, catalog-loader removal and serialization, catalog override precedence, fallback branches, empty-table timestamps, and read-optimized system tables. Add an end-to-end regression covering sequence semantics and repeated aggregation-field renames with `force_jni_scanner=false` and `true`. ### Release note Fix Paimon current and historical reads after renaming sequence or aggregation fields. ### Check List (For Author) - Test: - Baseline: 8 historical-read cases fail with missing-field errors. Two additional catalog-backed current-read cases fail during loader removal. Review regressions reproduce a removed startup mode being restored and different branch rows at capacities 64 and 128. - Full Paimon connector reactor install: 866 tests, 0 failures, 0 errors, 1 existing skipped; all 21 new tests pass. - FE Checkstyle: passed. FE/BE build and connector packaging: passed. - End-to-end regression: all 26 query checkpoints passed on a running FE/BE cluster; generated expected output was reviewed and a second comparison run passed (1 suite, 0 failures, 0 skips). - Behavior changed: Yes. Current reads retain their statement-bound schema during JNI transport, and historical reads use field-dependent options from the selected schema. - Does this need documentation: No.
1 parent 8ebb909 commit 7d16729

9 files changed

Lines changed: 915 additions & 35 deletions

File tree

‎fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonCatalogOps.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -312,7 +312,7 @@ public Table getTable(Identifier identifier) throws Catalog.TableNotExistExcepti
312312
// so this must not invalidate: REFRESH, Doris writes and DDL invalidate through CatalogMetaCache.
313313
Table table = catalog.getTable(identifier);
314314
Map<String, String> optionsForCopy = PaimonTableOptions.forCopy(tableOptions);
315-
return optionsForCopy.isEmpty() ? table : table.copy(optionsForCopy);
315+
return optionsForCopy.isEmpty() ? table : PaimonScanParams.copyWithHistoricalSchema(table, optionsForCopy);
316316
}
317317

318318
@Override

‎fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnectorMetadata.java‎

Lines changed: 25 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -420,15 +420,11 @@ private Table resolveSystemTableAt(ConnectorSession session,
420420
? paimonHandle
421421
: (PaimonTableHandle) applySnapshot(session, paimonHandle, snapshot);
422422
Map<String, String> scanOptions = pinned.getScanOptions();
423-
if (PaimonScanParams.preservesBoundSchema(scanOptions)) {
423+
if (PaimonScanParams.preservesBoundSchema(scanOptions) || PaimonScanParams.isOptionsPin(scanOptions)) {
424424
// Schema-derived wrappers must be built over the bound source before exposing their fields.
425425
return new PaimonScanPlanProvider(catalogProperties, catalogOps, context).resolveScanTable(pinned);
426426
}
427-
Table table = resolveTable(pinned);
428-
if (scanOptions != null && !scanOptions.isEmpty() && PaimonScanParams.isOptionsPin(scanOptions)) {
429-
return PaimonScanParams.applyOptions(table, scanOptions);
430-
}
431-
return table;
427+
return resolveTable(pinned);
432428
}
433429

434430
/**
@@ -870,7 +866,8 @@ private Optional<ConnectorMvccSnapshot> resolveTimeTravelAuthenticated(
870866
resolved = PaimonScanParams.pinOptionsToSnapshot(
871867
spec.getOptions(), spec.getLatestSnapshotFence().getAsLong());
872868
} else {
873-
resolved = PaimonScanParams.resolveOptions(table, spec.getOptions());
869+
resolved = PaimonScanParams.resolveOptions(table, spec.getOptions(),
870+
PaimonTableOptions.extractCompatible(catalogProperties.getRaw()));
874871
}
875872
String pinnedTag = resolved.get(CoreOptions.SCAN_TAG_NAME.key());
876873
if (pinnedTag != null) {
@@ -1755,6 +1752,13 @@ private ConnectorTableKey partitionViewCacheKey(PaimonTableHandle paimonHandle)
17551752
* the partition columns and escapes path-special characters in the name via the Paimon SDK.
17561753
*/
17571754
private List<ConnectorPartitionInfo> collectPartitions(PaimonTableHandle paimonHandle) {
1755+
if (paimonHandle.getScanOptions().isEmpty()) {
1756+
return collectPartitionsAuthenticated(paimonHandle);
1757+
}
1758+
return readMetadataAuthenticated(() -> collectPartitionsAuthenticated(paimonHandle));
1759+
}
1760+
1761+
private List<ConnectorPartitionInfo> collectPartitionsAuthenticated(PaimonTableHandle paimonHandle) {
17581762
if (PaimonScanParams.isPinnedEmptyScan(paimonHandle.getScanOptions())) {
17591763
// Do not reopen latest metadata after the statement fenced an empty table.
17601764
return Collections.emptyList();
@@ -1767,7 +1771,8 @@ private List<ConnectorPartitionInfo> collectPartitions(PaimonTableHandle paimonH
17671771
}
17681772

17691773
Table resolvedTable = resolveTable(paimonHandle);
1770-
Map<String, String> scanOptions = paimonHandle.getScanOptions();
1774+
Map<String, String> scanOptions = PaimonScanParams.withCatalogOptions(
1775+
paimonHandle.getScanOptions(), PaimonTableOptions.extractCompatible(catalogProperties.getRaw()));
17711776
boolean optionsPin = PaimonScanParams.isOptionsPin(scanOptions);
17721777
Table table;
17731778
if (optionsPin) {
@@ -1778,8 +1783,9 @@ private List<ConnectorPartitionInfo> collectPartitions(PaimonTableHandle paimonH
17781783
// partitions so EXPLAIN and block-rule accounting describe the same version as the scan.
17791784
Table partitionTable = snapshotId == null
17801785
? resolvedTable
1781-
: resolvedTable.copy(Collections.singletonMap(
1782-
CoreOptions.SCAN_SNAPSHOT_ID.key(), snapshotId));
1786+
: PaimonScanParams.copyWithHistoricalSchema(resolvedTable, PaimonScanParams.withCatalogOptions(
1787+
Collections.singletonMap(CoreOptions.SCAN_SNAPSHOT_ID.key(), snapshotId),
1788+
PaimonTableOptions.extractCompatible(catalogProperties.getRaw())));
17831789
// Partition projection never opens a data reader, so reader-only settings must not
17841790
// invalidate metadata that a later relation-scoped override can make safe.
17851791
// Metadata planning can also touch Paimon's global manifest executor, so apply the
@@ -1949,6 +1955,11 @@ public Optional<ConnectorTableStatistics> getTableStatistics(
19491955
if (snapshot == null) {
19501956
return getTableStatistics(session, handle);
19511957
}
1958+
return readMetadataAuthenticated(() -> getTableStatisticsAtSnapshot(session, handle, snapshot));
1959+
}
1960+
1961+
private Optional<ConnectorTableStatistics> getTableStatisticsAtSnapshot(
1962+
ConnectorSession session, ConnectorTableHandle handle, ConnectorMvccSnapshot snapshot) {
19521963
long rowCount;
19531964
try {
19541965
PaimonTableHandle pinned = (PaimonTableHandle) applySnapshot(session, handle, snapshot);
@@ -1957,17 +1968,18 @@ public Optional<ConnectorTableStatistics> getTableStatistics(
19571968
// first commit even though execution is still required to scan zero rows.
19581969
return Optional.empty();
19591970
}
1960-
Map<String, String> scanOptions = pinned.getScanOptions();
1971+
Map<String, String> scanOptions = PaimonScanParams.withCatalogOptions(
1972+
pinned.getScanOptions(), PaimonTableOptions.extractCompatible(catalogProperties.getRaw()));
19611973
// applyOptions removes this Doris-only marker, but execution still filters files by it.
19621974
// This also covers scan.creation-time-millis when it resolves to a file-creation scan.
19631975
if (PaimonScanParams.getPinnedFileCreationTime(scanOptions).isPresent()) {
19641976
return Optional.empty();
19651977
}
19661978
Table table = resolveTable(pinned);
1967-
if (scanOptions != null && !scanOptions.isEmpty()) {
1979+
if (scanOptions != null && !scanOptions.isEmpty() && !pinned.isSystemTable()) {
19681980
table = PaimonScanParams.isOptionsPin(scanOptions)
19691981
? PaimonScanParams.applyOptions(table, scanOptions)
1970-
: table.copy(scanOptions);
1982+
: PaimonScanParams.copyWithHistoricalSchema(table, scanOptions);
19711983
}
19721984
table = PaimonReaderOptions.runtimeSafeTable(table);
19731985
table = runtimeSafeSystemTable(pinned, table, scanOptions);

‎fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonReaderOptions.java‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -189,6 +189,14 @@ public static Table runtimeSafeSystemTable(
189189
}
190190
return rebuilt;
191191
}
192+
// Non-file sources cannot be rebuilt through SystemTableLoader; retain their wrapper's
193+
// own copy semantics after independently validating the source's effective options.
194+
if (scanOptions != null && !scanOptions.isEmpty()) {
195+
systemTable = PaimonScanParams.isOptionsPin(scanOptions)
196+
? PaimonScanParams.applyOptions(systemTable, scanOptions)
197+
: PaimonScanParams.copyWithHistoricalSchema(systemTable,
198+
PaimonIncrementalScanParams.applyResetsIfIncremental(scanOptions));
199+
}
192200
return runtimeSafeTable(systemTable);
193201
}
194202

@@ -207,7 +215,7 @@ public static Table runtimeSafeSystemSource(Table sourceTable, Map<String, Strin
207215
if (scanOptions != null && !scanOptions.isEmpty()) {
208216
// Incremental ranges are relation state too; rebuilding a capped system wrapper from the
209217
// undecorated source must not silently turn @incr back into a latest scan.
210-
effectiveSource = sourceTable.copy(
218+
effectiveSource = PaimonScanParams.copyWithHistoricalSchema(sourceTable,
211219
PaimonIncrementalScanParams.applyResetsIfIncremental(scanOptions));
212220
}
213221
return runtimeSafeTable(effectiveSource);

‎fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanParams.java‎

Lines changed: 81 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,9 @@
2424
import org.apache.paimon.Snapshot;
2525
import org.apache.paimon.options.ConfigOption;
2626
import org.apache.paimon.options.FallbackKey;
27+
import org.apache.paimon.options.Options;
2728
import org.apache.paimon.schema.TableSchema;
29+
import org.apache.paimon.table.ChainGroupReadTable;
2830
import org.apache.paimon.table.FallbackReadFileStoreTable;
2931
import org.apache.paimon.table.FileStoreTable;
3032
import org.apache.paimon.table.Table;
@@ -103,6 +105,13 @@ public final class PaimonScanParams {
103105

104106
private static final Set<String> INHERITED_READ_STATE_KEYS = buildInheritedReadStateKeys();
105107

108+
private static final Set<String> SCHEMA_INDEPENDENT_READ_KEYS = ImmutableSet.<String>builder()
109+
.addAll(INHERITED_READ_STATE_KEYS)
110+
.add(CoreOptions.BRANCH.key())
111+
.add(CoreOptions.SCAN_PRIMARY_BRANCH.key())
112+
.add(CoreOptions.SCAN_FALLBACK_BRANCH.key())
113+
.build();
114+
106115
private static final Set<String> TIME_TRAVEL_SELECTOR_KEYS = ImmutableSet.of(
107116
CoreOptions.SCAN_TIMESTAMP_MILLIS.key(),
108117
CoreOptions.SCAN_WATERMARK.key(),
@@ -204,13 +213,60 @@ public static Table applyOptions(Table table, Map<String, String> options) {
204213
}
205214
// Apply the relation override first, then normalize each physical planning branch. A
206215
// single combined copy would flatten independent fallback values into one sibling's cap.
207-
Table effectiveTable = PaimonReaderOptions.runtimeSafeTable(table.copy(isolatedOptions));
216+
Table effectiveTable = PaimonReaderOptions.runtimeSafeTable(
217+
copyWithHistoricalSchema(table, isolatedOptions, options));
208218
// Validate after every copy so relation options participate in the documented
209219
// relation > catalog > physical precedence before the effective value is judged.
210220
PaimonReaderOptions.validateEffectiveTable(effectiveTable);
211221
return effectiveTable;
212222
}
213223

224+
static Table copyWithHistoricalSchema(Table table, Map<String, String> options) {
225+
return copyWithHistoricalSchema(table, userOptions(options), options);
226+
}
227+
228+
private static Table copyWithHistoricalSchema(
229+
Table table, Map<String, String> dynamicOptions, Map<String, String> provenance) {
230+
if (!(table instanceof FileStoreTable) || table instanceof ChainGroupReadTable
231+
|| PaimonTableDecorators.isChainTable(table)) {
232+
// Chain groups have their own option filtering and are not ordinary fallback readers.
233+
return table.copy(dynamicOptions);
234+
}
235+
if (table instanceof FallbackReadFileStoreTable) {
236+
FallbackReadFileStoreTable pair = (FallbackReadFileStoreTable) table;
237+
// Each branch has independent snapshot IDs. Keep the SDK's timestamp translation
238+
// before restoring either schema; copying one historical schema onto the pair is invalid.
239+
return PaimonTableDecorators.rebuildFallbackPair(pair,
240+
(FileStoreTable) copyWithHistoricalSchema(pair.wrapped(), dynamicOptions, provenance),
241+
(FileStoreTable) copyWithHistoricalSchema(pair.other(),
242+
PaimonTableDecorators.otherOptions(pair, dynamicOptions), provenance));
243+
}
244+
FileStoreTable dataTable = (FileStoreTable) table;
245+
Map<String, String> effectiveOptions = new HashMap<>(table.options());
246+
dynamicOptions.forEach((key, value) -> {
247+
if (value == null) {
248+
effectiveOptions.remove(key);
249+
} else {
250+
effectiveOptions.put(key, value);
251+
}
252+
});
253+
Optional<Snapshot> snapshot;
254+
try {
255+
snapshot = TimeTravelUtil.tryTravelToSnapshot(Options.fromMap(effectiveOptions),
256+
dataTable.snapshotManager(), dataTable.tagManager());
257+
} catch (Exception e) {
258+
// Match the SDK's schema-selection fallback (for example a timestamp on an empty table).
259+
// Its normal copy below still owns option validation and the eventual scan error.
260+
return table.copy(dynamicOptions);
261+
}
262+
if (snapshot.isPresent() && snapshot.get().schemaId() != dataTable.schema().id()) {
263+
// Paimon overlays current options onto historical columns. Restore the matching
264+
// sequence/aggregation options first, or renamed fields fail before scan planning.
265+
dataTable = restoreBoundSchema(dataTable, snapshot.get().schemaId(), provenance, "");
266+
}
267+
return dataTable.copy(dynamicOptions);
268+
}
269+
214270
public static FileStoreTable applyOptionsWithoutTimeTravel(
215271
FileStoreTable table, Map<String, String> options) {
216272
Map<String, String> tableOptions = userOptions(options);
@@ -282,6 +338,17 @@ private static FileStoreTable restoreBoundSchema(
282338
merged.put(key, value);
283339
}
284340
});
341+
// Read selectors and branch topology belong to the resolved table, not its historical schema.
342+
// Preserve absent keys too: reviving an old mode can invalidate a snapshot selector, while
343+
// reviving an old branch policy can change fallback priority when reader caps rebuild the pair.
344+
SCHEMA_INDEPENDENT_READ_KEYS.forEach(key -> {
345+
String value = table.options().get(key);
346+
if (value == null) {
347+
merged.remove(key);
348+
} else {
349+
merged.put(key, value);
350+
}
351+
});
285352
// Explicit catalog overrides can equal old physical values; equality is not provenance.
286353
options.forEach((key, value) -> {
287354
if (key.startsWith(INTERNAL_PREFIX + "catalog-option.")) {
@@ -353,6 +420,11 @@ public static boolean usesStatementSnapshot(Map<String, String> options) {
353420
}
354421

355422
public static Map<String, String> resolveOptions(Table table, Map<String, String> options) {
423+
return resolveOptions(table, options, java.util.Collections.emptyMap());
424+
}
425+
426+
static Map<String, String> resolveOptions(
427+
Table table, Map<String, String> options, Map<String, String> catalogOptions) {
356428
validateOptions(options);
357429
if (!hasStartupOptions(options)) {
358430
return options;
@@ -388,7 +460,7 @@ public static Map<String, String> resolveOptions(Table table, Map<String, String
388460
return resolveFileCreationTime(options, fileStoreTable, creationTime);
389461
}
390462

391-
Table selectedTable = applyOptions(table, options);
463+
Table selectedTable = applyOptions(table, withCatalogOptions(options, catalogOptions));
392464
if ("compacted-full".equalsIgnoreCase(options.get(CoreOptions.SCAN_MODE.key()))) {
393465
Long snapshotId = compactedFullSnapshotId((FileStoreTable) selectedTable);
394466
return snapshotId == null
@@ -508,9 +580,13 @@ private static Map<String, String> resolvedEmptyOptions(Map<String, String> opti
508580
}
509581

510582
private static Map<String, String> userOptions(Map<String, String> options) {
511-
return options.entrySet().stream()
512-
.filter(entry -> !entry.getKey().startsWith(INTERNAL_PREFIX))
513-
.collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));
583+
Map<String, String> result = new HashMap<>();
584+
options.forEach((key, value) -> {
585+
if (!key.startsWith(INTERNAL_PREFIX)) {
586+
result.put(key, value);
587+
}
588+
});
589+
return result;
514590
}
515591

516592
/**

‎fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java‎

Lines changed: 16 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -361,10 +361,11 @@ Table resolveScanTable(PaimonTableHandle paimonHandle) {
361361
}
362362

363363
private <T> T withBoundSchemaAuthentication(PaimonTableHandle handle, Supplier<T> action) {
364-
if (context == null || !PaimonScanParams.preservesBoundSchema(handle.getScanOptions())) {
364+
if (context == null || (handle.getScanOptions().isEmpty()
365+
&& PaimonTableOptions.extractCompatible(catalogProps.getRaw()).isEmpty())) {
365366
return action.get();
366367
}
367-
// Restoring a bound schema can read FileIO after table resolution has left the authenticated scope.
368+
// Historical and statement schema restoration read FileIO after table resolution leaves its scope.
368369
try {
369370
return context.executeAuthenticated(action::get);
370371
} catch (RuntimeException e) {
@@ -376,20 +377,17 @@ private <T> T withBoundSchemaAuthentication(PaimonTableHandle handle, Supplier<T
376377

377378
private Map<String, String> effectiveScanOptions(PaimonTableHandle handle) {
378379
Map<String, String> options = handle.getScanOptions();
379-
if (PaimonScanParams.preservesBoundSchema(options)) {
380-
// Replayed catalogs already ignored legacy options during binding; scans must use the same policy.
381-
return PaimonScanParams.withCatalogOptions(options,
382-
PaimonTableOptions.extractCompatible(catalogProps.getRaw()));
383-
}
384-
return options;
380+
// An explicit catalog value may equal latest's value but differ from the historical schema.
381+
return PaimonScanParams.withCatalogOptions(options,
382+
PaimonTableOptions.extractCompatible(catalogProps.getRaw()));
385383
}
386384

387385
private Table applyScanOptions(PaimonTableHandle paimonHandle, Table table) {
388386
Map<String, String> scanOptions = effectiveScanOptions(paimonHandle);
389387
Table finalTable = table;
390-
if (scanOptions != null && !scanOptions.isEmpty()
391-
&& !(paimonHandle.isSystemTable() && PaimonScanParams.preservesBoundSchema(scanOptions))) {
392-
// Statement-fenced system wrappers are rebuilt below from their schema-bound source.
388+
if (scanOptions != null && !scanOptions.isEmpty() && !paimonHandle.isSystemTable()) {
389+
// System wrappers are rebuilt below from their schema-bound source. Copying the wrapper
390+
// first would time-travel inside the SDK before historical field options can be restored.
393391
if (table instanceof FileStoreTable
394392
&& PaimonScanParams.preservesBoundSchema(scanOptions)) {
395393
// A statement fence owns data visibility, not schema time travel. Reusing Table.copy
@@ -406,7 +404,8 @@ private Table applyScanOptions(PaimonTableHandle paimonHandle, Table table) {
406404
// scan.snapshot-id/scan.mode here (the single Table.copy chokepoint shared by both the
407405
// native/JNI scan path and the JNI serialized-table path) so a stale persisted pin on the
408406
// base table cannot hijack incremental-between. Non-incremental pins pass through unchanged.
409-
finalTable = table.copy(PaimonIncrementalScanParams.applyResetsIfIncremental(scanOptions));
407+
finalTable = PaimonScanParams.copyWithHistoricalSchema(table,
408+
PaimonIncrementalScanParams.applyResetsIfIncremental(scanOptions));
410409
}
411410
}
412411
finalTable = PaimonReaderOptions.runtimeSafeTable(finalTable);
@@ -1799,10 +1798,12 @@ private static FileStoreTable dropCatalogLoaderOfBranches(FileStoreTable table)
17991798
}
18001799

18011800
private static FileStoreTable rebuildWithoutCatalogLoader(FileStoreTable branch) {
1802-
// The factory evaluates scan.snapshot-id and can rewind a schema-only ALTER. Restore
1803-
// the already-bound schema after removing the loader so JNI reads the planned field ids.
1801+
// The factory invokes copy(Map) before returning, so selectors can rewind a schema-only
1802+
// ALTER and fail field-option validation before the bound schema can be restored.
1803+
Map<String, String> factoryOptions = new HashMap<>(branch.schema().options());
1804+
PaimonScanParams.inheritedReadStateKeys().forEach(factoryOptions::remove);
18041805
return FileStoreTableFactory.createWithoutFallbackBranch(
1805-
branch.fileIO(), branch.location(), branch.schema(), new Options(),
1806+
branch.fileIO(), branch.location(), branch.schema().copy(factoryOptions), new Options(),
18061807
CatalogEnvironment.empty()).copy(branch.schema());
18071808
}
18081809

0 commit comments

Comments
 (0)