Skip to content

Commit b1b936b

Browse files
morningmanGabriel39claude
committed
[test](fe) Cover Iceberg Variant write gating and effective-type scan gating
### What problem does this PR solve? Issue Number: None Related PR: #66302, #66413 Problem Summary: branch-4.1 #66302 (reading Iceberg Variant from Parquet) added FE unit tests for three guards: Iceberg VARIANT columns are read-only for data-file writes, a delete-only MERGE ships `writes_data_files=false` to BE, and the old-backend Variant fence follows the slot's effective (pruned) type. master ported the feature into the connector framework in #66413 (`IcebergWritePlanProvider`, `PluginDrivenScanNode`, `PluginDrivenTableSink`), and the production behavior is equivalent, but three of those assertions have no master counterpart: - No test reads `TIcebergMergeSink.writes_data_files`; only the engine-to-handle half of the delete-only MERGE flag is covered. - `rejectsVariantDataWritesButAllowsDeleteOnlyMerge` only checks a STRUCT nested plain `VARIANT`. The engine actually hands Iceberg VARIANT to the write path as the `VARIANT_COMPUTE_V2` carrier, and neither a root carrier column nor the read-only message is checked, so dropping the carrier arm of `containsVariant` passes every FE unit test. - `projectsComputeVariant` is only tested with a slot that projects a Variant; nothing checks that a slot whose pruned type no longer contains the Variant is not fenced off old backends. This adds those assertions in master's terms: - `IcebergWritePlanProviderTest#planWriteMergeSinkShipsWhetherTheWriteProducesDataFiles`: a delete-only MERGE ships `writes_data_files=false`, an UPDATE ships true; the field is always set. - `IcebergWritePlanProviderTest#rejectsRootAndNestedComputeVariantCarrier`: a root and a STRUCT-nested `VARIANT_COMPUTE_V2` column are rejected with the read-only message for data writes and accepted for delete-only writes. - `PluginDrivenScanNodeCompatibilityTest#projectsComputeVariantFollowsTheEffectiveSlotType`: the gate answers false for the pruned type and true for the full type of the same slot. ### Release note None ### Check List (For Author) - Test: Unit Test - `run-fe-ut.sh --run` on `IcebergWritePlanProviderTest`, `PluginDrivenScanNodeCompatibilityTest`, `IcebergTypeMappingReadTest`, `ConnectorColumnConverterTest`, `PruneNestedColumnTest`, `VariantPruningLogicTest`, `PluginDrivenTableSinkTest`: 232 tests, 0 failures - `mvn checkstyle:check` on fe-core and fe-connector-iceberg: 0 violations - Behavior changed: No - Does this need documentation: No Co-authored-by: Gabriel <[email protected]> Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent 038e227 commit b1b936b

2 files changed

Lines changed: 71 additions & 0 deletions

File tree

‎fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergWritePlanProviderTest.java‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -130,6 +130,29 @@ public void rejectsVariantDataWritesButAllowsDeleteOnlyMerge() {
130130
() -> IcebergWritePlanProvider.validateWriteSchema(partialInsert));
131131
}
132132

133+
@Test
134+
public void rejectsRootAndNestedComputeVariantCarrier() {
135+
// The engine hands an Iceberg VARIANT column to the write path as the VARIANT_COMPUTE_V2 carrier
136+
// (ConnectorColumnConverter), not as plain VARIANT, so the read-only gate must recognize the carrier
137+
// both as a root column and nested in a complex type. MUTATION: dropping the VARIANT_COMPUTE_V2 arm
138+
// of containsVariant -> red.
139+
ConnectorType carrier = ConnectorType.of("VARIANT_COMPUTE_V2");
140+
ConnectorColumn rootVariant = new ConnectorColumn("v", carrier, null, true, null);
141+
ConnectorColumn nestedVariant = new ConnectorColumn("payload",
142+
ConnectorType.structOf(Collections.singletonList("nested"), Collections.singletonList(carrier)),
143+
null, true, null);
144+
for (ConnectorColumn column : Arrays.asList(rootVariant, nestedVariant)) {
145+
DorisConnectorException exception = Assertions.assertThrows(DorisConnectorException.class,
146+
() -> IcebergWritePlanProvider.validateWriteSchema(Collections.singletonList(column), true),
147+
column.getName());
148+
Assertions.assertTrue(exception.getMessage().contains("VARIANT")
149+
&& exception.getMessage().contains("read-only"), exception.getMessage());
150+
// A delete-only MERGE writes no data file, so the same schema stays usable for it.
151+
Assertions.assertDoesNotThrow(() -> IcebergWritePlanProvider.validateWriteSchema(
152+
Collections.singletonList(column), false), column.getName());
153+
}
154+
}
155+
133156
private static InMemoryCatalog freshCatalog() {
134157
InMemoryCatalog catalog = new InMemoryCatalog();
135158
catalog.initialize("test", Collections.emptyMap());
@@ -1828,6 +1851,29 @@ public void planWriteMergeSinkDoesNotRequireCardinalityCheckForUpdate() {
18281851
+ " legal UPDATEs whose predicate matches a row through several source rows");
18291852
}
18301853

1854+
@Test
1855+
public void planWriteMergeSinkShipsWhetherTheWriteProducesDataFiles() {
1856+
// BE opens the data-file writer only when writes_data_files is true. A delete-only SQL MERGE must
1857+
// ship false so it neither builds a writer it never uses nor trips the writer-side schema checks
1858+
// (an Iceberg VARIANT target is deletable but not writable). MUTATION: dropping setWritesDataFiles,
1859+
// or shipping a constant -> one of the two plans below carries the wrong value -> red.
1860+
Table table = partitionedSortedTable(freshCatalog());
1861+
TIcebergMergeSink deleteOnlyMerge = planMergeSink(table, contextWithStorage(),
1862+
new WriteHandle(new IcebergTableHandle("db1", "t1"))
1863+
.writeOperation(WriteOperation.MERGE)
1864+
.writesDataFiles(false)
1865+
.requireMergeCardinalityCheck(true));
1866+
Assertions.assertTrue(deleteOnlyMerge.isSetWritesDataFiles(),
1867+
"the field must always be set so BE never has to guess from an unset field");
1868+
Assertions.assertFalse(deleteOnlyMerge.isWritesDataFiles());
1869+
1870+
TIcebergMergeSink update = planMergeSink(table, contextWithStorage(),
1871+
new WriteHandle(new IcebergTableHandle("db1", "t1"))
1872+
.writeOperation(WriteOperation.UPDATE));
1873+
Assertions.assertTrue(update.isSetWritesDataFiles());
1874+
Assertions.assertTrue(update.isWritesDataFiles());
1875+
}
1876+
18311877
@Test
18321878
public void planWriteBuildsMergeSinkWithTableDerivedFields() {
18331879
Table table = partitionedSortedTable(freshCatalog());

‎fe/fe-core/src/test/java/org/apache/doris/datasource/scan/PluginDrivenScanNodeCompatibilityTest.java‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,12 +18,17 @@
1818
package org.apache.doris.datasource.scan;
1919

2020
import org.apache.doris.analysis.SlotDescriptor;
21+
import org.apache.doris.analysis.SlotId;
2122
import org.apache.doris.analysis.TableSample;
2223
import org.apache.doris.analysis.TupleDescriptor;
24+
import org.apache.doris.analysis.TupleId;
2325
import org.apache.doris.catalog.ArrayType;
2426
import org.apache.doris.catalog.Column;
2527
import org.apache.doris.catalog.PartitionItem;
28+
import org.apache.doris.catalog.StructField;
29+
import org.apache.doris.catalog.StructType;
2630
import org.apache.doris.catalog.TableIf;
31+
import org.apache.doris.catalog.Type;
2732
import org.apache.doris.common.Config;
2833
import org.apache.doris.common.UserException;
2934
import org.apache.doris.common.jmockit.Deencapsulation;
@@ -157,6 +162,26 @@ public void translatedScanTuplePreservesNestedComputeVariantCarrier() {
157162
.types.get(1).scalar_type.variant_is_v2);
158163
}
159164

165+
@Test
166+
public void projectsComputeVariantFollowsTheEffectiveSlotType() {
167+
// Nested-column pruning narrows the slot type but keeps the original Column, so a scan that pruned
168+
// the Variant child away decodes no Variant and must not be fenced off old backends.
169+
// MUTATION: deciding from the slot's Column instead of its effective type -> red.
170+
StructType full = new StructType(
171+
new StructField("label", Type.STRING),
172+
new StructField("payload", new ConnectorComputeVariantType()));
173+
StructType pruned = new StructType(new StructField("label", Type.STRING));
174+
TupleDescriptor tuple = new TupleDescriptor(new TupleId(0));
175+
SlotDescriptor slot = new SlotDescriptor(new SlotId(1), tuple.getId());
176+
slot.setColumn(new Column("info", full));
177+
tuple.addSlot(slot);
178+
179+
slot.setType(pruned);
180+
Assertions.assertFalse(PluginDrivenScanNode.projectsComputeVariant(tuple));
181+
slot.setType(full);
182+
Assertions.assertTrue(PluginDrivenScanNode.projectsComputeVariant(tuple));
183+
}
184+
160185
@Test
161186
public void oldBackendAllowsOnlyFullyPrecomputedVariantCountPlans() throws UserException {
162187
ConnectorScanRange countRange = new ConnectorScanRange() {

0 commit comments

Comments
 (0)