Repository navigation
Commit b1b936b
[test](fe) Cover Iceberg Variant write gating and effective-type scan gating
### What problem does this PR solve?
Issue Number: None
Related PR: #66302, #66413
Problem Summary: branch-4.1 #66302 (reading Iceberg Variant from Parquet) added FE unit tests
for three guards: Iceberg VARIANT columns are read-only for data-file writes, a delete-only
MERGE ships `writes_data_files=false` to BE, and the old-backend Variant fence follows the
slot's effective (pruned) type. master ported the feature into the connector framework in
#66413 (`IcebergWritePlanProvider`, `PluginDrivenScanNode`, `PluginDrivenTableSink`), and the
production behavior is equivalent, but three of those assertions have no master counterpart:
- No test reads `TIcebergMergeSink.writes_data_files`; only the engine-to-handle half of the
delete-only MERGE flag is covered.
- `rejectsVariantDataWritesButAllowsDeleteOnlyMerge` only checks a STRUCT nested plain
`VARIANT`. The engine actually hands Iceberg VARIANT to the write path as the
`VARIANT_COMPUTE_V2` carrier, and neither a root carrier column nor the read-only message is
checked, so dropping the carrier arm of `containsVariant` passes every FE unit test.
- `projectsComputeVariant` is only tested with a slot that projects a Variant; nothing checks
that a slot whose pruned type no longer contains the Variant is not fenced off old backends.
This adds those assertions in master's terms:
- `IcebergWritePlanProviderTest#planWriteMergeSinkShipsWhetherTheWriteProducesDataFiles`:
a delete-only MERGE ships `writes_data_files=false`, an UPDATE ships true; the field is
always set.
- `IcebergWritePlanProviderTest#rejectsRootAndNestedComputeVariantCarrier`: a root and a
STRUCT-nested `VARIANT_COMPUTE_V2` column are rejected with the read-only message for data
writes and accepted for delete-only writes.
- `PluginDrivenScanNodeCompatibilityTest#projectsComputeVariantFollowsTheEffectiveSlotType`:
the gate answers false for the pruned type and true for the full type of the same slot.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- `run-fe-ut.sh --run` on `IcebergWritePlanProviderTest`,
`PluginDrivenScanNodeCompatibilityTest`, `IcebergTypeMappingReadTest`,
`ConnectorColumnConverterTest`, `PruneNestedColumnTest`, `VariantPruningLogicTest`,
`PluginDrivenTableSinkTest`: 232 tests, 0 failures
- `mvn checkstyle:check` on fe-core and fe-connector-iceberg: 0 violations
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Gabriel <[email protected]>
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>1 parent 038e227 commit b1b936b
2 files changed
Lines changed: 71 additions & 0 deletions
File tree
- fe
- fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg
- fe-core/src/test/java/org/apache/doris/datasource/scan
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
130 | 130 | | |
131 | 131 | | |
132 | 132 | | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
133 | 156 | | |
134 | 157 | | |
135 | 158 | | |
| |||
1828 | 1851 | | |
1829 | 1852 | | |
1830 | 1853 | | |
| 1854 | + | |
| 1855 | + | |
| 1856 | + | |
| 1857 | + | |
| 1858 | + | |
| 1859 | + | |
| 1860 | + | |
| 1861 | + | |
| 1862 | + | |
| 1863 | + | |
| 1864 | + | |
| 1865 | + | |
| 1866 | + | |
| 1867 | + | |
| 1868 | + | |
| 1869 | + | |
| 1870 | + | |
| 1871 | + | |
| 1872 | + | |
| 1873 | + | |
| 1874 | + | |
| 1875 | + | |
| 1876 | + | |
1831 | 1877 | | |
1832 | 1878 | | |
1833 | 1879 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
| 21 | + | |
21 | 22 | | |
22 | 23 | | |
| 24 | + | |
23 | 25 | | |
24 | 26 | | |
25 | 27 | | |
| 28 | + | |
| 29 | + | |
26 | 30 | | |
| 31 | + | |
27 | 32 | | |
28 | 33 | | |
29 | 34 | | |
| |||
157 | 162 | | |
158 | 163 | | |
159 | 164 | | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
160 | 185 | | |
161 | 186 | | |
162 | 187 | | |
| |||
0 commit comments