Skip to content

Commit 03f4c12

Browse files
morningmanclaude
andcommitted
[fix](catalog) hudi H4: lowercase JNI reader column names
Batch-0 (apache#65185 reverify) H4. After the HMS SPI cutover, a hudi-on-HMS table with mixed-case Avro columns (e.g. Id/Name/Addr) crashed every MOR/JNI split. Root cause: HudiScanPlanProvider.planScan built the JNI reader column list with `.map(Schema.Field::name)` (original case). BE HadoopHudiJniScanner.initRequiredColumnsAndTypes keys hudiColNameToType by these names, then resolves each requiredField with an EXACT lower-case containsKey; mixed-case "Id" misses lower-case "id" -> throw. The two sibling name paths were already lower-cased (avroSchemaToColumns:905, HudiSchemaUtils.buildField:137); legacy HudiScanNode:223 also emits lower-case names. This one path regressed. Fix: extract the inline transform into a package-private static `jniColumnNames(Schema)` that lower-cases each field name (toLowerCase(Locale.ROOT)); column ORDER preserved so the parallel columnTypes list stays positionally aligned. Extraction makes the transform offline-unit-testable (planScan needs a live metaClient). Test: HudiSchemaParityTest.jniColumnNamesAreLowerCased asserts the mixed-case Id/Name/Addr fixture lower-cases. MOR/JNI end-to-end read with mixed-case columns is live-gated (real hudi cluster). Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
1 parent 17a9aaf commit 03f4c12

2 files changed

Lines changed: 35 additions & 2 deletions

File tree

‎fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java‎

Lines changed: 23 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,7 @@
5757
import java.util.Collections;
5858
import java.util.LinkedHashMap;
5959
import java.util.List;
60+
import java.util.Locale;
6061
import java.util.Map;
6162
import java.util.Optional;
6263
import java.util.function.Function;
@@ -177,8 +178,12 @@ public List<ConnectorScanRange> planScan(
177178
// InternalSchema at the instant, D3) keeps the latest schema — byte-identical.
178179
Schema jniSchema = HudiSchemaUtils.resolveJniColumnSchema(
179180
schemaResolver, metaClient, avroSchema, hudiHandle.getQueryInstant());
180-
columnNames = jniSchema.getFields().stream()
181-
.map(Schema.Field::name).collect(Collectors.toList());
181+
// JNI reader column names MUST be lower-cased: BE HadoopHudiJniScanner.initRequiredColumnsAndTypes
182+
// keys hudiColNameToType by these names, then looks each requiredField up as an EXACT lower-case key
183+
// (mixed-case "Id" would miss lower-case "id" -> throw, crashing every MOR/JNI split). Matches the
184+
// Doris-column path (avroSchemaToColumns:905) and the native history_schema_info dict
185+
// (HudiSchemaUtils.buildField:137); legacy HudiScanNode:223 also emits lower-case names.
186+
columnNames = jniColumnNames(jniSchema);
182187
columnTypes = jniSchema.getFields().stream()
183188
.map(f -> HudiTypeMapping.toHiveTypeString(f.schema()))
184189
.collect(Collectors.toList());
@@ -827,6 +832,22 @@ private static String unescapePathName(String path) {
827832
return sb.toString();
828833
}
829834

835+
/**
836+
* The JNI (MOR-realtime) reader's required-column names, LOWER-CASED. BE
837+
* {@code HadoopHudiJniScanner.initRequiredColumnsAndTypes} builds {@code hudiColNameToType} keyed by these
838+
* names and resolves each requiredField with an EXACT lower-case {@code containsKey}, so a mixed-case Avro
839+
* name ({@code "Id"}) would miss the lower-case slot ({@code "id"}) and throw, crashing every MOR/JNI split.
840+
* Byte-consistent with the Doris-column casing ({@code HudiConnectorMetadata.avroSchemaToColumns} /
841+
* {@code HudiSchemaUtils.buildField}) and legacy {@code HudiScanNode} (which emits lower-case Column names).
842+
* Column ORDER is preserved so the parallel {@code columnTypes} list stays positionally aligned. Package-
843+
* private static for offline unit testing (planScan itself needs a live metaClient).
844+
*/
845+
static List<String> jniColumnNames(Schema jniSchema) {
846+
return jniSchema.getFields().stream()
847+
.map(f -> f.name().toLowerCase(Locale.ROOT))
848+
.collect(Collectors.toList());
849+
}
850+
830851
/**
831852
* Detect file format from file path suffix. Package-private static so the ported incremental relations
832853
* ({@code COWIncrementalRelation}) stamp the required explicit {@code fileFormat} on their native

‎fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiSchemaParityTest.java‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -148,6 +148,18 @@ public void avroSchemaToColumnsPreservesMetaCommitTimeAsBindableStringColumn() {
148148
"_hoodie_commit_time must be STRING so the window compare is lexicographic over Hudi instants");
149149
}
150150

151+
@Test
152+
public void jniColumnNamesAreLowerCased() {
153+
// H4: HudiScanPlanProvider.planScan feeds these names into THudiFileDesc.column_names, which BE
154+
// (HadoopHudiJniScanner.initRequiredColumnsAndTypes) keys a map by and then resolves each requiredField
155+
// as an EXACT lower-case containsKey. A mixed-case Avro name ("Id") missing the lower-case slot ("id")
156+
// throws and crashes every MOR/JNI split, so the JNI column-name list MUST be lower-cased — consistent
157+
// with avroSchemaToColumns and legacy HudiScanNode. RED before the fix: raw-case "Id"/"Name"/"Addr".
158+
Assertions.assertEquals(
159+
Arrays.asList("id", "name", "price", "event_date", "created_at", "tags", "props", "addr"),
160+
HudiScanPlanProvider.jniColumnNames(schema()));
161+
}
162+
151163
@Test
152164
public void testTopLevelNameLoweredButNestedStructNamePreserved() {
153165
List<ConnectorColumn> columns = HudiConnectorMetadata.avroSchemaToColumns(schema());

0 commit comments

Comments
 (0)