Skip to content

Commit 0ff75bb

Browse files
morningmanclaude
andcommitted
[fix](catalog) fe-connector-trino: carry STRUCT field names, and enforce the complex-type shape
WHY IT WAS WRONG: TrinoTypeMapping's ROW branch collected each field's TYPE and never read field.getName(). fe-core's converter does not reject a nameless STRUCT -- it fills in "col" + index -- so the user saw struct<col0:int,col1:text> in DESCRIBE and every `SELECT s.a` written against the actual source schema was rejected at analysis with "the specified field name a was not found". There is no way to reach the field by its real name at all; only positional access still works, which is exactly why a smoke test would not catch it. Field names are part of this type's equals()/hashCode(), so a nameless STRUCT is not "slightly incomplete" -- it has the wrong identity. This is a regression this migration introduced. The pre-migration fe-core mapping checked field.getName().isPresent() and used the real name when there was one. FIX (connector): collect the names and build through ConnectorType.structOf. Anonymous ROW fields are named by POSITION rather than reusing the legacy mapping's single "col" for all of them -- duplicate names are just as unreachable, and "col" + index matches what fe-core and four other connectors already fall back to. ARRAY and MAP move to the factories too, so "complex types go through a factory" is visible at a glance. FIX (public type): the canonical ConnectorType constructor now checks the shape, which covers every convenience constructor and factory since they all funnel through it: ARRAY has one child, MAP two, STRUCT at least one child with a same-length, null-free, parallel field-name list. This is enforced eagerly because a violation is undetectable downstream -- fe-core also turns a childless ARRAY/MAP into ARRAY<NULL>/MAP<NULL,NULL> in silence -- so the mistake surfaces far from the mapping that made it. One rule is deliberately looser than it might look: the four optional per-child lists (nullability, comment, field id, comment-specified) may not be LONGER than the children, but a SHORTER one stays legal. All four accessors read out of range as "not carried for that index" and fall back to a default; that is documented behavior the legacy factories rely on, so requiring exact length would hard-fail a supported state. Every production construction site was checked: only TrinoTypeMapping (fixed here) and EsTypeMapping (a single-child ARRAY, valid) use a bare constructor; the rest use factories that fill names and types in one loop. The existing complex-type tests across fe-core, hms, hudi, iceberg, maxcompute, paimon and hive act as the regression net and all still pass. USER-VISIBLE: STRUCT columns in a trino catalog now show their real field names instead of col0/col1. External table schemas are not persisted, so a reload is enough. A query that hard-coded col0 to work around this stops working -- it was relying on the defect, and a field genuinely named col0 is unaffected. No test depended on the old names. Mutation-verified: dropping the names again, and separately removing the shape check, each turns the corresponding tests red. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: catalog-spi-review-20
1 parent 27bf2be commit 0ff75bb

4 files changed

Lines changed: 307 additions & 17 deletions

File tree

‎fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorType.java‎

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@
2020
import java.util.Arrays;
2121
import java.util.Collections;
2222
import java.util.List;
23+
import java.util.Locale;
2324
import java.util.Objects;
2425

2526
/**
@@ -50,6 +51,26 @@
5051
* from {@link #equals(Object)}/{@link #hashCode()}. A connector that tracks a stable per-field id (iceberg
5152
* field-ids) carries them here so fe-core can stamp the Doris child column tree's {@code uniqueId}, which the
5253
* engine's nested access-path rewrite and the BE field-id scan path match nested leaves by.</p>
54+
*
55+
* <p><b>Shape contract for complex types</b> (enforced in the canonical constructor, so every factory
56+
* and convenience constructor is covered):</p>
57+
* <ul>
58+
* <li>{@code ARRAY} carries exactly one child; {@code MAP} exactly two (key, value).</li>
59+
* <li>{@code STRUCT} carries at least one child, and {@link #getFieldNames()} is a parallel list —
60+
* same length, same order, no null entries. A STRUCT may not omit or partially supply its
61+
* field names.</li>
62+
* <li>No optional per-child list (nullability, comment, field id, comment-specified) may be
63+
* <em>longer</em> than {@link #getChildren()}. Shorter stays legal — those four are read
64+
* index-tolerantly, so a missing entry means "not carried" and falls back to its default.</li>
65+
* <li>The type name is matched case-insensitively, so {@code "Struct"} cannot dodge the check. Any
66+
* other type name is left alone: {@code typeName} is a free-form string with no vocabulary, so
67+
* "this is not a complex type" cannot be inferred from it.</li>
68+
* </ul>
69+
*
70+
* <p>This is checked eagerly because a violation is invisible downstream: fe-core's converter fills a
71+
* missing STRUCT field name with {@code "col" + index} and turns a childless ARRAY/MAP into
72+
* {@code ARRAY<NULL>}/{@code MAP<NULL,NULL>} without any error, so the mistake surfaces much later as
73+
* "field name not found" at query analysis, far from the connector that made it.</p>
5374
*/
5475
public final class ConnectorType {
5576

@@ -145,6 +166,60 @@ public ConnectorType(String typeName, int precision, int scale,
145166
this.childrenCommentSpecified = childrenCommentSpecified == null
146167
? Collections.emptyList()
147168
: Collections.unmodifiableList(childrenCommentSpecified);
169+
validateShape();
170+
}
171+
172+
/**
173+
* Fails loud on a malformed complex type: see the shape contract in the class javadoc. Runs on the
174+
* already-normalized fields, so every constructor and factory funnels through it.
175+
*/
176+
private void validateShape() {
177+
switch (typeName.toUpperCase(Locale.ROOT)) {
178+
case "ARRAY":
179+
requireChildCount("ARRAY", 1);
180+
break;
181+
case "MAP":
182+
requireChildCount("MAP", 2);
183+
break;
184+
case "STRUCT":
185+
if (children.isEmpty()) {
186+
throw new IllegalArgumentException("STRUCT requires at least one child type");
187+
}
188+
if (fieldNames.size() != children.size()) {
189+
throw new IllegalArgumentException("STRUCT field name count (" + fieldNames.size()
190+
+ ") must match child type count (" + children.size() + ")");
191+
}
192+
if (fieldNames.contains(null)) {
193+
throw new IllegalArgumentException("STRUCT field names must not contain null: " + fieldNames);
194+
}
195+
break;
196+
default:
197+
// Unknown type name: nothing to assert. Not a complex-type tag as far as we can tell, and
198+
// typeName is free-form, so we must not conclude "then it has no children".
199+
return;
200+
}
201+
requireParallel("children nullability", childrenNullable.size());
202+
requireParallel("children comments", childrenComments.size());
203+
requireParallel("children field ids", childrenFieldIds.size());
204+
requireParallel("children comment-specified flags", childrenCommentSpecified.size());
205+
}
206+
207+
private void requireChildCount(String tag, int expected) {
208+
if (children.size() != expected) {
209+
throw new IllegalArgumentException(
210+
tag + " requires exactly " + expected + " child type(s), got " + children.size());
211+
}
212+
}
213+
214+
private void requireParallel(String what, int size) {
215+
// Deliberately only rejects "longer than children". A SHORTER list is a documented, supported
216+
// state for these four: every accessor (isChildNullable / getChildComment / getChildFieldId /
217+
// isChildCommentSpecified) reads out of range as "not carried for that index" and falls back to
218+
// its default. Only an entry with no child to belong to is unambiguously a caller bug.
219+
if (size > children.size()) {
220+
throw new IllegalArgumentException(typeName.toUpperCase(Locale.ROOT) + " " + what + " count ("
221+
+ size + ") must not exceed child type count (" + children.size() + ")");
222+
}
148223
}
149224

150225
/** Factory: simple type with no parameters. */
@@ -242,6 +317,10 @@ public List<ConnectorType> getChildren() {
242317
return children;
243318
}
244319

320+
/**
321+
* The STRUCT field names, parallel to {@link #getChildren()} — same length, same order, no nulls
322+
* (see the shape contract in the class javadoc). Empty for non-STRUCT types.
323+
*/
245324
public List<String> getFieldNames() {
246325
return fieldNames;
247326
}
Lines changed: 178 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,178 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
package org.apache.doris.connector.api;
19+
20+
import org.junit.jupiter.api.Assertions;
21+
import org.junit.jupiter.api.Test;
22+
23+
import java.util.Arrays;
24+
import java.util.Collections;
25+
import java.util.List;
26+
27+
/**
28+
* Pins the complex-type shape contract enforced by the {@link ConnectorType} constructor.
29+
*
30+
* <p><b>WHY this matters:</b> a complex type is described by PARALLEL lists — one of child types, one
31+
* of STRUCT field names, plus optional per-child metadata. Nothing downstream can detect that they
32+
* disagree: fe-core's converter fills a missing STRUCT field name with {@code "col" + index} and turns
33+
* a childless ARRAY/MAP into {@code ARRAY<NULL>}/{@code MAP<NULL,NULL>}, all silently. The connector
34+
* that got it wrong compiles, the table loads, DESCRIBE prints something — and the user only finds out
35+
* when a query that names a real sub-field is rejected as "field name not found", by which point the
36+
* error site is nowhere near the type mapping that caused it. So the invariant is enforced where the
37+
* mistake is made: at construction.
38+
*/
39+
public class ConnectorTypeTest {
40+
41+
private static final ConnectorType INT = ConnectorType.of("INT");
42+
private static final ConnectorType STR = ConnectorType.of("STRING");
43+
44+
private static ConnectorType struct(List<ConnectorType> children, List<String> names) {
45+
return new ConnectorType("STRUCT", -1, -1, children, names);
46+
}
47+
48+
// ---------- STRUCT: field names are mandatory and parallel ----------
49+
50+
@Test
51+
public void testStructWithoutFieldNamesRejected() {
52+
// Exactly what the trino ROW mapping used to build.
53+
IllegalArgumentException e = Assertions.assertThrows(IllegalArgumentException.class,
54+
() -> new ConnectorType("STRUCT", -1, -1, Arrays.asList(INT, STR)));
55+
// Both counts belong in the message: "they disagree" without the numbers does not locate the bug.
56+
Assertions.assertTrue(e.getMessage().contains("(0)") && e.getMessage().contains("(2)"), e.getMessage());
57+
}
58+
59+
@Test
60+
public void testStructWithTooFewFieldNamesRejected() {
61+
IllegalArgumentException e = Assertions.assertThrows(IllegalArgumentException.class,
62+
() -> struct(Arrays.asList(INT, STR), Collections.singletonList("a")));
63+
Assertions.assertTrue(e.getMessage().contains("(1)") && e.getMessage().contains("(2)"), e.getMessage());
64+
}
65+
66+
@Test
67+
public void testStructWithTooManyFieldNamesRejected() {
68+
Assertions.assertThrows(IllegalArgumentException.class,
69+
() -> struct(Collections.singletonList(INT), Arrays.asList("a", "b")));
70+
}
71+
72+
@Test
73+
public void testStructWithNoChildrenRejected() {
74+
Assertions.assertThrows(IllegalArgumentException.class,
75+
() -> struct(Collections.emptyList(), Collections.emptyList()));
76+
}
77+
78+
@Test
79+
public void testStructWithNullFieldNameRejected() {
80+
// A null name is unresolvable in exactly the same way a missing one is.
81+
Assertions.assertThrows(IllegalArgumentException.class,
82+
() -> struct(Arrays.asList(INT, STR), Arrays.asList("a", null)));
83+
}
84+
85+
@Test
86+
public void testLowercaseStructTagStillValidated() {
87+
// The tag is compared case-insensitively so a spelling variant cannot dodge the check.
88+
Assertions.assertThrows(IllegalArgumentException.class,
89+
() -> new ConnectorType("struct", -1, -1, Arrays.asList(INT, STR)));
90+
}
91+
92+
// ---------- ARRAY / MAP: fixed arity ----------
93+
94+
@Test
95+
public void testArrayArityEnforced() {
96+
Assertions.assertThrows(IllegalArgumentException.class,
97+
() -> new ConnectorType("ARRAY", -1, -1, Collections.emptyList()));
98+
Assertions.assertThrows(IllegalArgumentException.class,
99+
() -> new ConnectorType("ARRAY", -1, -1, Arrays.asList(INT, STR)));
100+
}
101+
102+
@Test
103+
public void testMapArityEnforced() {
104+
Assertions.assertThrows(IllegalArgumentException.class,
105+
() -> new ConnectorType("MAP", -1, -1, Collections.singletonList(INT)));
106+
Assertions.assertThrows(IllegalArgumentException.class,
107+
() -> new ConnectorType("MAP", -1, -1, Arrays.asList(INT, STR, INT)));
108+
}
109+
110+
// ---------- optional per-child lists ----------
111+
112+
@Test
113+
public void testOptionalListLongerThanChildrenRejected() {
114+
// An entry with no child to belong to cannot be interpreted at all.
115+
Assertions.assertThrows(IllegalArgumentException.class,
116+
() -> new ConnectorType("STRUCT", -1, -1, Collections.singletonList(INT),
117+
Collections.singletonList("a"), Arrays.asList(true, false),
118+
Collections.emptyList(), Collections.emptyList()));
119+
Assertions.assertThrows(IllegalArgumentException.class,
120+
() -> new ConnectorType("ARRAY", -1, -1, Collections.singletonList(INT),
121+
Collections.emptyList(), Collections.emptyList(), Collections.emptyList(),
122+
Arrays.asList(1, 2)));
123+
}
124+
125+
@Test
126+
public void testOptionalListShorterThanChildrenStillLegal() {
127+
// Deliberately NOT rejected: these four are read index-tolerantly ("not carried for that index"
128+
// -> default), which is the documented behavior every legacy factory relies on. Tightening this
129+
// to exact-length would hard-fail a supported state.
130+
ConnectorType ct = new ConnectorType("STRUCT", -1, -1, Arrays.asList(INT, STR),
131+
Arrays.asList("a", "b"), Collections.singletonList(false),
132+
Collections.emptyList(), Collections.emptyList());
133+
Assertions.assertFalse(ct.isChildNullable(0));
134+
Assertions.assertTrue(ct.isChildNullable(1));
135+
Assertions.assertNull(ct.getChildComment(0));
136+
Assertions.assertEquals(-1, ct.getChildFieldId(0));
137+
}
138+
139+
// ---------- every legal construction path must stay legal ----------
140+
141+
@Test
142+
public void testFactoriesRemainValid() {
143+
Assertions.assertEquals(1, ConnectorType.arrayOf(INT).getChildren().size());
144+
Assertions.assertEquals(1, ConnectorType.arrayOf(INT, false).getChildren().size());
145+
Assertions.assertEquals(2, ConnectorType.mapOf(STR, INT).getChildren().size());
146+
Assertions.assertEquals(2, ConnectorType.mapOf(STR, INT, false).getChildren().size());
147+
148+
List<String> names = Arrays.asList("a", "b");
149+
List<ConnectorType> types = Arrays.asList(INT, STR);
150+
Assertions.assertEquals(names, ConnectorType.structOf(names, types).getFieldNames());
151+
Assertions.assertEquals(names, ConnectorType.structOf(names, types,
152+
Arrays.asList(true, false), Arrays.asList("c1", "c2")).getFieldNames());
153+
Assertions.assertEquals(names, ConnectorType.structOf(names, types,
154+
Arrays.asList(true, false), Arrays.asList("c1", "c2"),
155+
Arrays.asList(true, true)).getFieldNames());
156+
}
157+
158+
@Test
159+
public void testWithChildrenFieldIdsRemainsValid() {
160+
// The iceberg usage: one id per child on ARRAY (1), MAP (2) and STRUCT (N).
161+
Assertions.assertEquals(7, ConnectorType.arrayOf(INT)
162+
.withChildrenFieldIds(Collections.singletonList(7)).getChildFieldId(0));
163+
Assertions.assertEquals(9, ConnectorType.mapOf(STR, INT)
164+
.withChildrenFieldIds(Arrays.asList(8, 9)).getChildFieldId(1));
165+
Assertions.assertEquals(3, ConnectorType.structOf(Arrays.asList("a", "b"), Arrays.asList(INT, STR))
166+
.withChildrenFieldIds(Arrays.asList(2, 3)).getChildFieldId(1));
167+
}
168+
169+
@Test
170+
public void testNonComplexTypesUnaffected() {
171+
// typeName is a free-form string with no vocabulary, so an unrecognized tag must not be
172+
// second-guessed - we cannot conclude "then it has no children".
173+
Assertions.assertEquals("JSONB", ConnectorType.of("JSONB").getTypeName());
174+
Assertions.assertEquals(10, ConnectorType.of("DECIMALV3", 10, 2).getPrecision());
175+
Assertions.assertEquals("SOMETHING",
176+
new ConnectorType("SOMETHING", -1, -1, Arrays.asList(INT, STR)).getTypeName());
177+
}
178+
}

‎fe/fe-connector/fe-connector-trino/src/main/java/org/apache/doris/connector/trino/TrinoTypeMapping.java‎

Lines changed: 19 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -90,26 +90,28 @@ public static ConnectorType toConnectorType(Type type) {
9090
int precision = Math.min(((TimestampWithTimeZoneType) type).getPrecision(), 6);
9191
return new ConnectorType("DATETIMEV2", precision, -1);
9292
} else if (type instanceof io.trino.spi.type.ArrayType) {
93-
ConnectorType elementType = toConnectorType(
94-
((io.trino.spi.type.ArrayType) type).getElementType());
95-
List<ConnectorType> children = new ArrayList<>();
96-
children.add(elementType);
97-
return new ConnectorType("ARRAY", -1, -1, children);
93+
return ConnectorType.arrayOf(toConnectorType(
94+
((io.trino.spi.type.ArrayType) type).getElementType()));
9895
} else if (type instanceof io.trino.spi.type.MapType) {
99-
ConnectorType keyType = toConnectorType(
100-
((io.trino.spi.type.MapType) type).getKeyType());
101-
ConnectorType valueType = toConnectorType(
102-
((io.trino.spi.type.MapType) type).getValueType());
103-
List<ConnectorType> children = new ArrayList<>();
104-
children.add(keyType);
105-
children.add(valueType);
106-
return new ConnectorType("MAP", -1, -1, children);
96+
return ConnectorType.mapOf(
97+
toConnectorType(((io.trino.spi.type.MapType) type).getKeyType()),
98+
toConnectorType(((io.trino.spi.type.MapType) type).getValueType()));
10799
} else if (type instanceof RowType) {
108-
List<ConnectorType> children = new ArrayList<>();
109-
for (RowType.Field field : ((RowType) type).getFields()) {
110-
children.add(toConnectorType(field.getType()));
100+
// Carry the field NAMES, not just the field types: Doris resolves STRUCT sub-field access by
101+
// name, so a nameless STRUCT makes fe-core invent "col0"/"col1" and every `s.a` written against
102+
// the real schema fails analysis with "field name a was not found".
103+
List<RowType.Field> rowFields = ((RowType) type).getFields();
104+
List<String> names = new ArrayList<>(rowFields.size());
105+
List<ConnectorType> fieldTypes = new ArrayList<>(rowFields.size());
106+
for (int i = 0; i < rowFields.size(); i++) {
107+
RowType.Field field = rowFields.get(i);
108+
// Trino ROW fields may be anonymous (RowType.anonymousRow). Name them BY POSITION so each
109+
// one still gets a distinct, resolvable name; the legacy fe-core mapping named every
110+
// anonymous field "col", which collides and is unaddressable once there is more than one.
111+
names.add(field.getName().orElse("col" + i));
112+
fieldTypes.add(toConnectorType(field.getType()));
111113
}
112-
return new ConnectorType("STRUCT", -1, -1, children);
114+
return ConnectorType.structOf(names, fieldTypes);
113115
} else {
114116
throw new IllegalArgumentException("Cannot transform unknown Trino type: " + type);
115117
}

‎fe/fe-connector/fe-connector-trino/src/test/java/org/apache/doris/connector/trino/TrinoTypeMappingTest.java‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,9 @@
4040
import org.junit.jupiter.api.Assertions;
4141
import org.junit.jupiter.api.Test;
4242

43+
import java.util.Arrays;
44+
import java.util.Collections;
45+
4346
/**
4447
* Unit tests for {@link TrinoTypeMapping}: every supported Trino SPI type must map to
4548
* the Doris {@link ConnectorType} name (and precision/scale/children) that the rest of
@@ -130,6 +133,34 @@ public void testStructCarriesFieldTypes() {
130133
Assertions.assertEquals(2, ct.getChildren().size());
131134
Assertions.assertEquals("INT", ct.getChildren().get(0).getTypeName());
132135
Assertions.assertEquals("STRING", ct.getChildren().get(1).getTypeName());
136+
// WHY the names matter: Doris resolves STRUCT sub-field access BY NAME. Drop them here and
137+
// fe-core invents "col0"/"col1", so DESCRIBE shows fabricated names and every `SELECT s.a`
138+
// written against the source schema is rejected at analysis time as "field a was not found".
139+
// The user then has no way at all to reach the field by its real name.
140+
Assertions.assertEquals(Arrays.asList("a", "b"), ct.getFieldNames());
141+
Assertions.assertEquals(ct.getChildren().size(), ct.getFieldNames().size());
142+
}
143+
144+
@Test
145+
public void testAnonymousStructFieldsNamedByPosition() {
146+
// Trino ROWs can be anonymous. Each field still needs a DISTINCT resolvable name; the legacy
147+
// fe-core mapping named them all "col", which collides as soon as there is more than one.
148+
ConnectorType ct = TrinoTypeMapping.toConnectorType(
149+
RowType.anonymousRow(IntegerType.INTEGER, VarcharType.VARCHAR));
150+
Assertions.assertEquals(Arrays.asList("col0", "col1"), ct.getFieldNames());
151+
}
152+
153+
@Test
154+
public void testNestedStructCarriesInnerFieldNames() {
155+
// row(a int, b row(c int)) - the recursive path has to carry names too, not just the top level.
156+
RowType inner = RowType.rowType(RowType.field("c", IntegerType.INTEGER));
157+
ConnectorType ct = TrinoTypeMapping.toConnectorType(RowType.rowType(
158+
RowType.field("a", IntegerType.INTEGER),
159+
RowType.field("b", inner)));
160+
Assertions.assertEquals(Arrays.asList("a", "b"), ct.getFieldNames());
161+
ConnectorType nested = ct.getChildren().get(1);
162+
Assertions.assertEquals("STRUCT", nested.getTypeName());
163+
Assertions.assertEquals(Collections.singletonList("c"), nested.getFieldNames());
133164
}
134165

135166
@Test

0 commit comments

Comments
 (0)