Skip to content

Commit f0b2128

Browse files
morningmanclaude
andcommitted
[improvement](fe) Reject only lossy casts that reach a Variant leaf of the write target
### What problem does this PR solve? Issue Number: None Related PR: #69194 Problem Summary: VariantWritePlanValidator (#69194) follows the lineage of every target column that contains a Variant and rejects an implicit Variant-to-non-Variant cast on the way, because common-type analysis can turn an object or array Variant into SQL NULL before the sink encodes it back. It traced the whole target column and every argument of every expression, so it also rejected writes that lose nothing: - a scalar field of a composite target computed from a Variant, such as named_struct('matched', COALESCE(v['n'], 0) > 1, 'payload', v) into STRUCT<matched BOOLEAN, payload VARIANT>; - a Variant that is only compared or only picks an element: v['n'] > 0, NULLIF(1, IF(TRUE, v, 1)), the index or key of element_at. The validator now traces a demand per ExprId: which parts of a value (none, all, or per struct field / array element / map key and value) reach a Variant leaf. The target type gives the demand, aligned position by position to the source type, and all of a value of another shape is traced. A closed set of structural expressions passes the demand on (named_struct, struct, array, map, element_at, casts between values of the same shape, the results of IF / CASE / NULLIF / NVL, aliases, window functions, set operations, CTEs and scalar subqueries); any other expression traces all of its arguments as before. The second argument of NULLIF, the index or key of element_at, and the implicit cast a comparison, IN, LIKE / REGEXP, AND / OR or NOT applies directly to an operand are not traced; the operand under that cast still is, so IF(TRUE, v, 1) > 0 stays rejected, as do ELT, SUBSTRING and MAX_BY arguments. A CTE output is traced again only for a part not traced yet. The error now suggests an explicit CAST. ### Release note Writes into an external table column that contains a VARIANT no longer fail when a Variant is implicitly cast only to compute a non-Variant field of the target (for example the BOOLEAN field of a STRUCT), only to be compared (comparison, IN, LIKE, REGEXP, AND, OR, NOT), or only to choose a value (the second argument of NULLIF, the index or key of element_at). The error for a lossy cast now suggests an explicit CAST. ### Check List (For Author) - Test: Regression test / Unit Test - Unit Test: VariantWritePlanValidatorTest, 6 new tests; 19 mutations of the new rules each fail at least one test - Regression test: test_paimon_write_variant_errors (new struct and comparison cases), run locally against the iceberg docker environment - Behavior changed: Yes (the writes in the release note are accepted; the error message gains a hint) - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent cefe023 commit f0b2128

4 files changed

Lines changed: 695 additions & 160 deletions

File tree

0 commit comments

Comments
 (0)