Skip to content

Commit 61a8b38

Browse files
morningmanclaude
andcommitted
[fix](catalog) restore jfs hive-write hadoop config + iceberg system-table scan projection
External Regression on this branch (build 1000131) surfaced two real regressions that pass on master but fail here; both are fixed with FE regression tests. 1. Hive INSERT into a JuiceFS (jfs://) backed table failed on the BE with "No FileSystem for scheme jfs". HiveWritePlanProvider.buildHadoopConfig() sourced the BE writer hadoop config only from the typed getStorageProperties() pipeline, which is EMPTY for schemes whose fe-filesystem plugin has no typed bind() (jfs, oss-hdfs), so fs.<scheme>.impl / juicefs.* never reached the BE. It now also merges getBackendStorageProperties() -- the same source the hive READ path (HiveScanPlanProvider) already uses, which carries the fs./juicefs.* passthrough. The typed overlay is applied last so object-store / HDFS writes stay byte-identical; the default getBackendStorageProperties() is emptyMap so nothing else changes. This also covers the oss-hdfs (JindoFS) sibling. 2. Selecting an iceberg $files / $data_files system table threw BufferUnderflowException. IcebergScanPlanProvider planned the metadata-table scan without a column projection, so the SDK eagerly materialised readable_metrics for every row (MetricsUtil.readableMetricsStruct -> Conversions.fromByteBuffer underflows on complex/boolean bound columns) even when the query selected only a scalar such as file_size_in_bytes. doPlanSystemTableScan now projects the scan to the requested columns (scan.select(...)), applied only in the system-table path; $position_deletes and the shared buildScan (data scans) are unaffected. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_014ZngdXGBAXNDRe7jA42mAS
1 parent 7b3821f commit 61a8b38

5 files changed

Lines changed: 82 additions & 2 deletions

File tree

‎fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveWritePlanProvider.java‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -312,11 +312,19 @@ private static String resolveTextCompressionDefault(ConnectorSession session) {
312312
return textCompression;
313313
}
314314

315-
// Mirror iceberg buildHadoopConfig: BE-canonical static catalog creds from the typed fe-filesystem
316-
// StorageProperties (AWS_* for object stores, dfs/hadoop for HDFS). Hive has no vended overlay.
315+
// Mirror iceberg buildHadoopConfig: BE-canonical static catalog creds (AWS_* for object stores, dfs/hadoop
316+
// for HDFS). Hive has no vended overlay.
317317
private Map<String, String> buildHadoopConfig() {
318318
Map<String, String> merged = new HashMap<>();
319319
if (context != null) {
320+
// Backend properties from the catalog's parsed storage map, the SAME source the hive READ path uses
321+
// (HiveScanPlanProvider.getBackendStorageProperties) and legacy HiveTableSink emitted. It carries the
322+
// resolved hadoop./dfs./fs./juicefs.* passthrough, so it is the sole source of fs.<scheme>.impl for
323+
// schemes whose fe-filesystem plugin has no typed bind() (jfs, oss-hdfs) — without it a jfs write ships
324+
// the BE writer a config with no fs.jfs.impl and libhdfs fails "No FileSystem for scheme jfs". Emitted
325+
// first so the typed getStorageProperties() overlay still wins wherever it produces a value (object
326+
// stores / HDFS), keeping their behavior byte-identical.
327+
merged.putAll(context.getBackendStorageProperties());
320328
for (StorageProperties sp : context.getStorageProperties()) {
321329
sp.toBackendProperties().ifPresent(b -> merged.putAll(b.toMap()));
322330
}

‎fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveWritePlanProviderTest.java‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -488,6 +488,27 @@ public void planWriteHadoopConfigFromStorageProperties() {
488488
planSink(client, ctx, handle()).getHadoopConfig().get("AWS_ACCESS_KEY"));
489489
}
490490

491+
@Test
492+
public void planWriteHadoopConfigIncludesBackendStoragePropertiesForUntypedFsSchemes() {
493+
// C1 regression (External Regression build 1000131, test_jfs_hms_catalog_read): the jfs/oss-hdfs
494+
// fe-filesystem plugins have no typed bind(), so getStorageProperties() is EMPTY for a jfs catalog and
495+
// fs.jfs.impl (+ juicefs.*) would be dropped from the BE writer hadoopConfig -> libhdfs fails
496+
// "No FileSystem for scheme jfs" on INSERT. buildHadoopConfig must ALSO merge
497+
// getBackendStorageProperties() — the SAME source the hive READ path (HiveScanPlanProvider) uses, which
498+
// carries the fs./juicefs.* passthrough. MUTATION: dropping the getBackendStorageProperties() merge from
499+
// buildHadoopConfig -> fs.jfs.impl absent -> red.
500+
RecordingHmsClient client = new RecordingHmsClient();
501+
client.table = tableBuilder().build();
502+
RecordingConnectorContext ctx = new RecordingConnectorContext();
503+
// Typed storageProperties intentionally EMPTY (models a jfs catalog whose plugin has no typed binding);
504+
// the fs.<scheme>.impl passthrough is available only via the backend-storage source.
505+
ctx.backendStorageProperties = Collections.singletonMap("fs.jfs.impl", "io.juicefs.JuiceFileSystem");
506+
507+
Assertions.assertEquals("io.juicefs.JuiceFileSystem",
508+
planSink(client, ctx, handle()).getHadoopConfig().get("fs.jfs.impl"),
509+
"the backend-storage passthrough (fs.<scheme>.impl) must reach the BE writer hadoopConfig");
510+
}
511+
491512
// ───────────────────────────── no transaction ─────────────────────────────
492513

493514
@Test

‎fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/RecordingConnectorContext.java‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,11 @@ final class RecordingConnectorContext implements ConnectorContext {
4848
* {@code sp.toBackendProperties().toMap()}); default none. */
4949
List<StorageProperties> storageProperties = Collections.emptyList();
5050

51+
/** BE-canonical backend props the fake returns from {@link #getBackendStorageProperties()} — the read-path
52+
* source that carries the {@code fs./juicefs.*} passthrough for untyped fe-filesystem schemes (jfs, oss-hdfs)
53+
* whose typed {@link #getStorageProperties()} binding is empty; default none. */
54+
Map<String, String> backendStorageProperties = Collections.emptyMap();
55+
5156
/** Broker addresses the fake returns from {@link #getBrokerAddresses()}; default none, so a FILE_BROKER
5257
* write fails loud ("No alive broker.") unless a test populates it. */
5358
List<ConnectorBrokerAddress> brokerAddresses = Collections.emptyList();
@@ -85,6 +90,11 @@ public List<StorageProperties> getStorageProperties() {
8590
return storageProperties;
8691
}
8792

93+
@Override
94+
public Map<String, String> getBackendStorageProperties() {
95+
return backendStorageProperties;
96+
}
97+
8898
@Override
8999
public List<ConnectorBrokerAddress> getBrokerAddresses() {
90100
return brokerAddresses;

‎fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -747,6 +747,16 @@ private List<ConnectorScanRange> doPlanSystemTableScan(IcebergTableHandle handle
747747
return doPlanPositionDeletesSystemTableScan(handle, metadataTable, columns, filter, session);
748748
}
749749
TableScan scan = buildScan(metadataTable, handle, filter, session);
750+
// Project the metadata-table scan to ONLY the requested columns (legacy IcebergScanNode parity: FE keeps
751+
// the fields requested by BE in the projection). Without it the iceberg SDK materialises EVERY metadata
752+
// column per row for the $files/$data_files family — including readable_metrics, whose bound conversion
753+
// (MetricsUtil.readableMetricsStruct -> Conversions.fromByteBuffer) throws BufferUnderflowException on
754+
// boolean/complex bound columns — even when the query selects only a scalar such as file_size_in_bytes.
755+
// BE's IcebergSysTableJniScanner resolves required_fields by name against this projected task schema.
756+
List<String> projectedColumns = requestedLowerNames(columns);
757+
if (!projectedColumns.isEmpty()) {
758+
scan = scan.select(projectedColumns);
759+
}
750760
List<ConnectorScanRange> ranges = new ArrayList<>();
751761
try (CloseableIterable<FileScanTask> tasks = scan.planFiles()) {
752762
for (FileScanTask task : tasks) {

‎fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -746,6 +746,37 @@ public void getScanNodePropertiesForSysHandleStillEmitsLocationCreds() {
746746
Assertions.assertEquals("jni", props.get("file_format_type"));
747747
}
748748

749+
@Test
750+
public void planSystemTableScanProjectsRequestedColumnsExcludingReadableMetrics() {
751+
// WHY (regression — External Regression build 1000131, test_iceberg_system_table_projection): a
752+
// $data_files/$files sys scan MUST be projected to only the requested columns. Without the
753+
// scan.select(...) projection in doPlanSystemTableScan the iceberg SDK materialises EVERY metadata
754+
// column per row — including readable_metrics, whose bound conversion
755+
// (MetricsUtil.readableMetricsStruct -> Conversions.fromByteBuffer) throws BufferUnderflowException on
756+
// complex/boolean bound columns — even when the query selects only a scalar such as file_size_in_bytes.
757+
// The projected schema travels INSIDE the serialized FileScanTask that BE's IcebergSysTableJniScanner
758+
// deserializes and iterates (asDataTask().rows()). MUTATION: dropping scan.select(projectedColumns) in
759+
// doPlanSystemTableScan -> readable_metrics stays in the task schema -> red.
760+
Table table = createTable("t1", SCHEMA, PartitionSpec.unpartitioned());
761+
table.newAppend()
762+
.appendFile(dataFile(table.spec(), "s3://b/db/t1/f1.parquet", 1024, null, null))
763+
.commit();
764+
IcebergScanPlanProvider provider = new IcebergScanPlanProvider(Collections.emptyMap(), opsReturning(table));
765+
766+
List<ConnectorScanRange> ranges = provider.planScan(
767+
null, IcebergTableHandle.forSystemTable("db1", "t1", "data_files", -1L, null, -1L),
768+
Collections.singletonList(new IcebergColumnHandle("file_size_in_bytes", 1)), Optional.empty());
769+
770+
Assertions.assertEquals(1, ranges.size(), "one data file -> one metadata split");
771+
FileScanTask task = SerializationUtil.deserializeFromBase64(
772+
((IcebergScanRange) ranges.get(0)).getSerializedSplit());
773+
Assertions.assertNotNull(task.schema().findField("file_size_in_bytes"),
774+
"the requested column must survive in the projected task schema");
775+
Assertions.assertNull(task.schema().findField("readable_metrics"),
776+
"readable_metrics must NOT be in the projected task schema: its bound conversion "
777+
+ "BufferUnderflows on complex/boolean bound columns when materialised unrequested");
778+
}
779+
749780
// ---------------------------------------------------------------------
750781
// $position_deletes (upstream #65135 port): the ONE sys table BE reads with a native reader
751782
// ---------------------------------------------------------------------

0 commit comments

Comments
 (0)