Repository navigation
[Feature](function) Support function DEFAULT - #57531
Conversation
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
0b10e76 to
5ca6a29
Compare
|
run buildall |
FE UT Coverage ReportIncrement line coverage |
ClickBench: Total hot run time: 29.57 s |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
FE Regression Coverage ReportIncrement line coverage |
|
run buildall |
ClickBench: Total hot run time: 28.79 s |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
|
run buildall |
TPC-DS: Total hot run time: 189481 ms |
ClickBench: Total hot run time: 27.5 s |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
1 similar comment
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
|
run buildall |
FE UT Coverage ReportIncrement line coverage |
TPC-DS: Total hot run time: 186913 ms |
ClickBench: Total hot run time: 27.66 s |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
FE Regression Coverage ReportIncrement line coverage |
FE UT Coverage ReportIncrement line coverage |
|
run p0 |
FE Regression Coverage ReportIncrement line coverage |
| | CASE value=expression whenClause+ (ELSE elseExpression=expression)? END #simpleCase | ||
| | name=CAST LEFT_PAREN expression AS castDataType RIGHT_PAREN #cast | ||
| | name=TRY_CAST LEFT_PAREN expression AS castDataType RIGHT_PAREN #tryCast | ||
| | DEFAULT LEFT_PAREN expression RIGHT_PAREN #defaultValue |
There was a problem hiding this comment.
should be DEFAULT LEFT_PAREN qualifiedName RIGHT_PAREN because you only support get default value from a column
|
run buildall |
|
PR approved by at least one committer and no changes requested. |
TPC-H: Total hot run time: 30813 ms |
TPC-DS: Total hot run time: 172507 ms |
ClickBench: Total hot run time: 26.69 s |
FE UT Coverage ReportIncrement line coverage |
FE Regression Coverage ReportIncrement line coverage |
Return the default value of column
```sql
CREATE TABLE test_default_scalar (
c_id INT,
c_bool BOOLEAN NULL DEFAULT 1,
c_tinyint TINYINT NULL DEFAULT 7,
c_smallint SMALLINT NULL DEFAULT 32000,
c_int INT NULL DEFAULT 2147483647,
c_bigint BIGINT NULL DEFAULT 9223372036854775807,
c_largeint LARGEINT NULL DEFAULT '170141183460469231731687303715884105727',
c_float FLOAT NULL DEFAULT 3.125,
c_double DOUBLE NULL DEFAULT 2.718281828,
c_decimal DECIMAL(27, 9) NULL DEFAULT '123456789.123456789',
c_decimal_compact DECIMAL(18, 4) NULL DEFAULT '99999.1234',
c_pi DOUBLE NULL DEFAULT PI,
c_e DOUBLE NULL DEFAULT E,
c_char CHAR(8) NULL DEFAULT 'charDemo',
c_varchar VARCHAR(32) NULL DEFAULT '',
c_string STRING NULL DEFAULT 'plain string',
c_datetime DATETIME NULL DEFAULT '2025-10-25 11:22:33',
c_date DATE NULL DEFAULT '2025-10-31',
c_json JSON NULL,
c_ipv4 IPV4 NULL DEFAULT '192.168.1.1',
c_ipv6 IPV6 NULL DEFAULT '2001:db8::1',
c_array_int ARRAY<INT> NULL DEFAULT '[]',
c_array_string ARRAY<STRING> NULL,
c_map_str_int MAP<STRING, INT> NULL,
c_struct STRUCT<f1:INT,f2:STRING,f3:BOOLEAN> NULL,
c_variant VARIANT NULL
) PROPERTIES ( 'replication_num' = '1');
SELECT
default(c_id) AS default_id,
default(c_bool) AS default_c_bool,
default(c_tinyint) AS default_c_tinyint,
default(c_smallint) AS default_c_smallint,
default(c_int) AS default_c_int,
default(c_bigint) AS default_c_bigint,
default(c_largeint) AS default_c_largeint,
default(c_float) AS default_c_float,
default(c_double) AS default_c_double,
default(c_decimal) AS default_c_decimal,
default(c_decimal_compact) AS default_c_decimal_compact,
default(c_char) AS default_c_char,
default(c_varchar) AS default_c_varchar,
default(c_string) AS default_c_string,
default(c_datetime) AS default_datetime,
default(c_date) AS defaut_time,
default(c_json) AS default_c_json,
default(c_ipv4) AS default_c_ipv4,
default(c_ipv6) AS default_c_ipv6,
default(c_array_int) AS default_c_array_int,
default(c_array_string) AS default_c_array_string,
default(c_map_str_int) AS default_c_map_str_int,
default(c_struct) AS default_c_struct,
default(c_variant) AS default_c_variant
FROM test_default_scalar
LIMIT 1;
```
```text
+------------+----------------+-------------------+--------------------+---------------+---------------------+-----------------------------------------+-----------------+------------------+---------------------+---------------------------+----------------+-------------------+------------------+---------------------+-------------+----------------+----------------+----------------+---------------------+------------------------+-----------------------+------------------+-------------------+
| default_id | default_c_bool | default_c_tinyint | default_c_smallint | default_c_int | default_c_bigint | default_c_largeint | default_c_float | default_c_double | default_c_decimal | default_c_decimal_compact | default_c_char | default_c_varchar | default_c_string | default_datetime | defaut_time | default_c_json | default_c_ipv4 | default_c_ipv6 | default_c_array_int | default_c_array_string | default_c_map_str_int | default_c_struct | default_c_variant |
+------------+----------------+-------------------+--------------------+---------------+---------------------+-----------------------------------------+-----------------+------------------+---------------------+---------------------------+----------------+-------------------+------------------+---------------------+-------------+----------------+----------------+----------------+---------------------+------------------------+-----------------------+------------------+-------------------+
| NULL | 1 | 7 | 32000 | 2147483647 | 9223372036854775807 | 170141183460469231731687303715884105727 | 3.125 | 2.718281828 | 123456789.123456789 | 99999.1234 | charDemo | | plain string | 2025-10-25 11:22:33 | 2025-10-31 | NULL | 192.168.1.1 | 2001:db8::1 | [] | NULL | NULL | NULL | NULL |
+------------+----------------+-------------------+--------------------+---------------+---------------------+-----------------------------------------+-----------------+------------------+---------------------+---------------------------+----------------+-------------------+------------------+---------------------+-------------+----------------+----------------+----------------+---------------------+------------------------+-----------------------+------------------+-------------------+
```
…hema ### What problem does this PR solve? Issue Number: None Related PR: apache#66538, apache#57531, apache#65851 Problem Summary: An Iceberg write-default exists only in the writer schema that a write statement pins (IcebergWriteSchemaContext); the columns of the read schema carry no default. Master resolves DEFAULT(column) against that writer schema only in UPDATE / MERGE SET and in MERGE INSERT VALUES, and everywhere else falls back to the read column. On a v3 table whose column d has write-default 42, INSERT ... VALUES (DEFAULT(d)) and INSERT ... SELECT DEFAULT(d) without FROM fail with "Unknown column", INSERT ... SELECT DEFAULT(d) FROM t writes NULL, UPDATE ... WHERE d = DEFAULT(d) and MERGE ... ON ... d = DEFAULT(d) compare d with NULL and silently match no row, and SELECT DEFAULT(d) FROM t returns NULL. branch-4.1 and branch-4.2 (apache#66538) resolve all of these against the pinned write target, by name against the target in INSERT ... SELECT, and reject DEFAULT(column) on an Iceberg column in a statement that does not write the table. This change adds the connector capability SUPPORTS_WRITE_ONLY_DEFAULTS, declared by the Iceberg connector and inherited by iceberg-on-HMS tables. For a table that declares it, InsertUtils resolves DEFAULT(column) in INSERT VALUES and in the select lists of INSERT ... SELECT by name against the pinned writer schema, and RewriteDefaultExpression takes the default of a DEFAULT(column) bound to the table from the pinned writer column with the same field id, which covers the UPDATE and MERGE conditions, and rejects it when the statement pinned no writer schema for the table. Hive, Paimon and OLAP tables keep the current behavior. The new enum constant changes the connector plugin API surface, so the plugin API version goes from 20.0 to 21.0. ### Release note Fix: On Iceberg tables, DEFAULT(column) in INSERT VALUES, INSERT ... SELECT and the WHERE / ON conditions of UPDATE and MERGE now resolves to the column's write-default. Before, it failed, wrote NULL, or matched no row. SELECT DEFAULT(column) on an Iceberg table now fails instead of returning NULL. ### Check List (For Author) - Test: Regression test / Unit Test - test_iceberg_initial_defaults: writes rows 20-22 with DEFAULT(default_int) in VALUES and in INSERT ... SELECT with and without FROM, then updates them with UPDATE ... WHERE default_int = DEFAULT(default_int) and MERGE ... ON ... = DEFAULT(...), for Parquet and ORC; SELECT DEFAULT(default_int) must fail. Passes locally; with master's FE it fails at the first INSERT with "Unknown column 'default_int'" - InsertUtilsTest: VALUES and INSERT ... SELECT (from a staging relation and without FROM) resolve against the pinned writer schema; unknown column fails; a table without the capability is unchanged - RewriteDefaultExpressionTest: pinned writer column wins, no pinned schema fails, a table without the capability keeps the read default - IcebergConnectorTest, HiveConnectorMetadataSiblingDelegationTest, ConnectorPluginSurfaceTest - Behavior changed: Yes. See the release note. - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
…hema ### What problem does this PR solve? Issue Number: None Related PR: apache#66538, apache#57531, apache#65851 Problem Summary: An Iceberg write-default exists only in the writer schema that a write statement pins (IcebergWriteSchemaContext); the columns of the read schema carry no default. Master resolves DEFAULT(column) against that writer schema only in UPDATE / MERGE SET and in MERGE INSERT VALUES, and everywhere else falls back to the read column. On a v3 table whose column d has write-default 42, INSERT ... VALUES (DEFAULT(d)) and INSERT ... SELECT DEFAULT(d) without FROM fail with "Unknown column", INSERT ... SELECT DEFAULT(d) FROM t writes NULL, UPDATE ... WHERE d = DEFAULT(d) and MERGE ... ON ... d = DEFAULT(d) compare d with NULL and silently match no row, and SELECT DEFAULT(d) FROM t returns NULL. branch-4.1 and branch-4.2 (apache#66538) resolve all of these against the pinned write target, by name against the target in INSERT ... SELECT, and reject DEFAULT(column) on an Iceberg column in a statement that does not write the table. This change adds the connector capability SUPPORTS_WRITE_ONLY_DEFAULTS, declared by the Iceberg connector and inherited by iceberg-on-HMS tables. For a table that declares it, InsertUtils resolves DEFAULT(column) in INSERT VALUES and in the select lists of INSERT ... SELECT by name against the pinned writer schema, and RewriteDefaultExpression takes the default of a DEFAULT(column) bound to the table from the pinned writer column with the same field id, which covers the UPDATE and MERGE conditions, and rejects it when the statement pinned no writer schema for the table. Hive, Paimon and OLAP tables keep the current behavior. The new enum constant changes the connector plugin API surface, so the plugin API version goes from 20.0 to 21.0. ### Release note Fix: On Iceberg tables, DEFAULT(column) in INSERT VALUES, INSERT ... SELECT and the WHERE / ON conditions of UPDATE and MERGE now resolves to the column's write-default. Before, it failed, wrote NULL, or matched no row. SELECT DEFAULT(column) on an Iceberg table now fails instead of returning NULL. ### Check List (For Author) - Test: Regression test / Unit Test - test_iceberg_initial_defaults: writes rows 20-22 with DEFAULT(default_int) in VALUES and in INSERT ... SELECT with and without FROM, then updates them with UPDATE ... WHERE default_int = DEFAULT(default_int) and MERGE ... ON ... = DEFAULT(...), for Parquet and ORC; SELECT DEFAULT(default_int) must fail. Passes locally; with master's FE it fails at the first INSERT with "Unknown column 'default_int'" - InsertUtilsTest: VALUES and INSERT ... SELECT (from a staging relation and without FROM) resolve against the pinned writer schema; unknown column fails; a table without the capability is unchanged - RewriteDefaultExpressionTest: pinned writer column wins, no pinned schema fails, a table without the capability keeps the read default - IcebergConnectorTest, HiveConnectorMetadataSiblingDelegationTest, ConnectorPluginSurfaceTest - Behavior changed: Yes. See the release note. - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
…hema ### What problem does this PR solve? Issue Number: None Related PR: apache#66538, apache#57531, apache#65851 Problem Summary: An Iceberg write-default exists only in the writer schema that a write statement pins (IcebergWriteSchemaContext); the columns of the read schema carry no default. Master resolves DEFAULT(column) against that writer schema only in UPDATE / MERGE SET and in MERGE INSERT VALUES, and everywhere else falls back to the read column. On a v3 table whose column d has write-default 42, INSERT ... VALUES (DEFAULT(d)) and INSERT ... SELECT DEFAULT(d) without FROM fail with "Unknown column", INSERT ... SELECT DEFAULT(d) FROM t writes NULL, UPDATE ... WHERE d = DEFAULT(d) and MERGE ... ON ... d = DEFAULT(d) compare d with NULL and silently match no row, and SELECT DEFAULT(d) FROM t returns NULL. branch-4.1 and branch-4.2 (apache#66538) resolve all of these against the pinned write target, by name against the target in INSERT ... SELECT, and reject DEFAULT(column) on an Iceberg column in a statement that does not write the table. This change adds the connector capability SUPPORTS_WRITE_ONLY_DEFAULTS, declared by the Iceberg connector and inherited by iceberg-on-HMS tables. For a table that declares it, InsertUtils resolves DEFAULT(column) in INSERT VALUES and in the select lists of INSERT ... SELECT by name against the pinned writer schema, and RewriteDefaultExpression takes the default of a DEFAULT(column) bound to the table from the pinned writer column with the same field id, which covers the UPDATE and MERGE conditions, and rejects it when the statement pinned no writer schema for the table. Hive, Paimon and OLAP tables keep the current behavior. The new enum constant changes the connector plugin API surface, so the plugin API version goes from 20.0 to 21.0. ### Release note Fix: On Iceberg tables, DEFAULT(column) in INSERT VALUES, INSERT ... SELECT and the WHERE / ON conditions of UPDATE and MERGE now resolves to the column's write-default. Before, it failed, wrote NULL, or matched no row. SELECT DEFAULT(column) on an Iceberg table now fails instead of returning NULL. ### Check List (For Author) - Test: Regression test / Unit Test - test_iceberg_initial_defaults: writes rows 20-22 with DEFAULT(default_int) in VALUES and in INSERT ... SELECT with and without FROM, then updates them with UPDATE ... WHERE default_int = DEFAULT(default_int) and MERGE ... ON ... = DEFAULT(...), for Parquet and ORC; SELECT DEFAULT(default_int) must fail. Passes locally; with master's FE it fails at the first INSERT with "Unknown column 'default_int'" - InsertUtilsTest: VALUES and INSERT ... SELECT (from a staging relation and without FROM) resolve against the pinned writer schema; unknown column fails; a table without the capability is unchanged - RewriteDefaultExpressionTest: pinned writer column wins, no pinned schema fails, a table without the capability keeps the read default - IcebergConnectorTest, HiveConnectorMetadataSiblingDelegationTest, ConnectorPluginSurfaceTest - Behavior changed: Yes. See the release note. - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
…hema ### What problem does this PR solve? Issue Number: None Related PR: apache#66538, apache#57531, apache#65851 Problem Summary: An Iceberg write-default exists only in the writer schema that a write statement pins (IcebergWriteSchemaContext); the columns of the read schema carry no default. Master resolves DEFAULT(column) against that writer schema only in UPDATE / MERGE SET and in MERGE INSERT VALUES, and everywhere else falls back to the read column. On a v3 table whose column d has write-default 42, INSERT ... VALUES (DEFAULT(d)) and INSERT ... SELECT DEFAULT(d) without FROM fail with "Unknown column", INSERT ... SELECT DEFAULT(d) FROM t writes NULL, UPDATE ... WHERE d = DEFAULT(d) and MERGE ... ON ... d = DEFAULT(d) compare d with NULL and silently match no row, and SELECT DEFAULT(d) FROM t returns NULL. branch-4.1 and branch-4.2 (apache#66538) resolve all of these against the pinned write target, by name against the target in INSERT ... SELECT, and reject DEFAULT(column) on an Iceberg column in a statement that does not write the table. This change adds the connector capability SUPPORTS_WRITE_ONLY_DEFAULTS, declared by the Iceberg connector and inherited by iceberg-on-HMS tables. For a table that declares it, InsertUtils resolves DEFAULT(column) in INSERT VALUES and in the select lists of INSERT ... SELECT by name against the pinned writer schema, and RewriteDefaultExpression takes the default of a DEFAULT(column) bound to the table from the pinned writer column with the same field id, which covers the UPDATE and MERGE conditions, and rejects it when the statement pinned no writer schema for the table. Hive, Paimon and OLAP tables keep the current behavior. The new enum constant changes the connector plugin API surface, so the plugin API version goes from 21.0 to 22.0. ### Release note Fix: On Iceberg tables, DEFAULT(column) in INSERT VALUES, INSERT ... SELECT and the WHERE / ON conditions of UPDATE and MERGE now resolves to the column's write-default. Before, it failed, wrote NULL, or matched no row. SELECT DEFAULT(column) on an Iceberg table now fails instead of returning NULL. ### Check List (For Author) - Test: Regression test / Unit Test - test_iceberg_initial_defaults: writes rows 20-22 with DEFAULT(default_int) in VALUES and in INSERT ... SELECT with and without FROM, then updates them with UPDATE ... WHERE default_int = DEFAULT(default_int) and MERGE ... ON ... = DEFAULT(...), for Parquet and ORC; SELECT DEFAULT(default_int) must fail. Passes locally; with master's FE it fails at the first INSERT with "Unknown column 'default_int'" - InsertUtilsTest: VALUES and INSERT ... SELECT (from a staging relation and without FROM) resolve against the pinned writer schema; unknown column fails; a table without the capability is unchanged - RewriteDefaultExpressionTest: pinned writer column wins, no pinned schema fails, a table without the capability keeps the read default - IcebergConnectorTest, HiveConnectorMetadataSiblingDelegationTest, ConnectorPluginSurfaceTest - Behavior changed: Yes. See the release note. - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
…hema ### What problem does this PR solve? Issue Number: None Related PR: apache#66538, apache#57531, apache#65851 Problem Summary: An Iceberg write-default exists only in the writer schema that a write statement pins (IcebergWriteSchemaContext); the columns of the read schema carry no default. Master resolves DEFAULT(column) against that writer schema only in UPDATE / MERGE SET and in MERGE INSERT VALUES, and everywhere else falls back to the read column. On a v3 table whose column d has write-default 42, INSERT ... VALUES (DEFAULT(d)) and INSERT ... SELECT DEFAULT(d) without FROM fail with "Unknown column", INSERT ... SELECT DEFAULT(d) FROM t writes NULL, UPDATE ... WHERE d = DEFAULT(d) and MERGE ... ON ... d = DEFAULT(d) compare d with NULL and silently match no row, and SELECT DEFAULT(d) FROM t returns NULL. branch-4.1 and branch-4.2 (apache#66538) resolve all of these against the pinned write target, by name against the target in INSERT ... SELECT, and reject DEFAULT(column) on an Iceberg column in a statement that does not write the table. This change adds the connector capability SUPPORTS_WRITE_ONLY_DEFAULTS, declared by the Iceberg connector and inherited by iceberg-on-HMS tables. For a table that declares it, InsertUtils resolves DEFAULT(column) in INSERT VALUES and in the select lists of INSERT ... SELECT by name against the pinned writer schema, and RewriteDefaultExpression takes the default of a DEFAULT(column) bound to the table from the pinned writer column with the same field id, which covers the UPDATE and MERGE conditions, and rejects it when the statement pinned no writer schema for the table. Hive, Paimon and OLAP tables keep the current behavior. The new enum constant changes the connector plugin API surface, so the plugin API version goes from 24.0 to 25.0. ### Release note Fix: On Iceberg tables, DEFAULT(column) in INSERT VALUES, INSERT ... SELECT and the WHERE / ON conditions of UPDATE and MERGE now resolves to the column's write-default. Before, it failed, wrote NULL, or matched no row. SELECT DEFAULT(column) on an Iceberg table now fails instead of returning NULL. ### Check List (For Author) - Test: Regression test / Unit Test - test_iceberg_initial_defaults: writes rows 20-22 with DEFAULT(default_int) in VALUES and in INSERT ... SELECT with and without FROM, then updates them with UPDATE ... WHERE default_int = DEFAULT(default_int) and MERGE ... ON ... = DEFAULT(...), for Parquet and ORC; SELECT DEFAULT(default_int) must fail. Passes locally; with master's FE it fails at the first INSERT with "Unknown column 'default_int'" - InsertUtilsTest: VALUES and INSERT ... SELECT (from a staging relation and without FROM) resolve against the pinned writer schema; unknown column fails; a table without the capability is unchanged - RewriteDefaultExpressionTest: pinned writer column wins, no pinned schema fails, a table without the capability keeps the read default - IcebergConnectorTest, HiveConnectorMetadataSiblingDelegationTest, ConnectorPluginSurfaceTest - Behavior changed: Yes. See the release note. - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
…hema ### What problem does this PR solve? Issue Number: None Related PR: apache#66538, apache#57531, apache#65851 Problem Summary: An Iceberg write-default exists only in the writer schema that a write statement pins (IcebergWriteSchemaContext); the columns of the read schema carry no default. Master resolves DEFAULT(column) against that writer schema only in UPDATE / MERGE SET and in MERGE INSERT VALUES, and everywhere else falls back to the read column. On a v3 table whose column d has write-default 42, INSERT ... VALUES (DEFAULT(d)) and INSERT ... SELECT DEFAULT(d) without FROM fail with "Unknown column", INSERT ... SELECT DEFAULT(d) FROM t writes NULL, UPDATE ... WHERE d = DEFAULT(d) and MERGE ... ON ... d = DEFAULT(d) compare d with NULL and silently match no row, and SELECT DEFAULT(d) FROM t returns NULL. branch-4.1 and branch-4.2 (apache#66538) resolve all of these against the pinned write target, by name against the target in INSERT ... SELECT, and reject DEFAULT(column) on an Iceberg column in a statement that does not write the table. This change adds the connector capability SUPPORTS_WRITE_ONLY_DEFAULTS, declared by the Iceberg connector and inherited by iceberg-on-HMS tables. For a table that declares it, InsertUtils resolves DEFAULT(column) in INSERT VALUES and in the select lists of INSERT ... SELECT by name against the pinned writer schema, and RewriteDefaultExpression takes the default of a DEFAULT(column) bound to the table from the pinned writer column with the same field id, which covers the UPDATE and MERGE conditions, and rejects it when the statement pinned no writer schema for the table. Hive, Paimon and OLAP tables keep the current behavior. The new enum constant changes the connector plugin API surface, so the plugin API version goes from 25.0 to 26.0. ### Release note Fix: On Iceberg tables, DEFAULT(column) in INSERT VALUES, INSERT ... SELECT and the WHERE / ON conditions of UPDATE and MERGE now resolves to the column's write-default. Before, it failed, wrote NULL, or matched no row. SELECT DEFAULT(column) on an Iceberg table now fails instead of returning NULL. ### Check List (For Author) - Test: Regression test / Unit Test - test_iceberg_initial_defaults: writes rows 20-22 with DEFAULT(default_int) in VALUES and in INSERT ... SELECT with and without FROM, then updates them with UPDATE ... WHERE default_int = DEFAULT(default_int) and MERGE ... ON ... = DEFAULT(...), for Parquet and ORC; SELECT DEFAULT(default_int) must fail. Passes locally; with master's FE it fails at the first INSERT with "Unknown column 'default_int'" - InsertUtilsTest: VALUES and INSERT ... SELECT (from a staging relation and without FROM) resolve against the pinned writer schema; unknown column fails; a table without the capability is unchanged - RewriteDefaultExpressionTest: pinned writer column wins, no pinned schema fails, a table without the capability keeps the read default - IcebergConnectorTest, HiveConnectorMetadataSiblingDelegationTest, ConnectorPluginSurfaceTest - Behavior changed: Yes. See the release note. - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
…hema ### What problem does this PR solve? Issue Number: None Related PR: apache#66538, apache#57531, apache#65851 Problem Summary: An Iceberg write-default exists only in the writer schema that a write statement pins (IcebergWriteSchemaContext); the columns of the read schema carry no default. Master resolves DEFAULT(column) against that writer schema only in UPDATE / MERGE SET and in MERGE INSERT VALUES, and everywhere else falls back to the read column. On a v3 table whose column d has write-default 42, INSERT ... VALUES (DEFAULT(d)) and INSERT ... SELECT DEFAULT(d) without FROM fail with "Unknown column", INSERT ... SELECT DEFAULT(d) FROM t writes NULL, UPDATE ... WHERE d = DEFAULT(d) and MERGE ... ON ... d = DEFAULT(d) compare d with NULL and silently match no row, and SELECT DEFAULT(d) FROM t returns NULL. branch-4.1 and branch-4.2 (apache#66538) resolve all of these against the pinned write target, by name against the target in INSERT ... SELECT, and reject DEFAULT(column) on an Iceberg column in a statement that does not write the table. This change adds the connector capability SUPPORTS_WRITE_ONLY_DEFAULTS, declared by the Iceberg connector and inherited by iceberg-on-HMS tables. For a table that declares it, InsertUtils resolves DEFAULT(column) in INSERT VALUES and in the select lists of INSERT ... SELECT by name against the pinned writer schema, and RewriteDefaultExpression takes the default of a DEFAULT(column) bound to the table from the pinned writer column with the same field id, which covers the UPDATE and MERGE conditions, and rejects it when the statement pinned no writer schema for the table. Hive, Paimon and OLAP tables keep the current behavior. The new enum constant changes the connector plugin API surface, so the plugin API version goes from 25.0 to 26.0. ### Release note Fix: On Iceberg tables, DEFAULT(column) in INSERT VALUES, INSERT ... SELECT and the WHERE / ON conditions of UPDATE and MERGE now resolves to the column's write-default. Before, it failed, wrote NULL, or matched no row. SELECT DEFAULT(column) on an Iceberg table now fails instead of returning NULL. ### Check List (For Author) - Test: Regression test / Unit Test - test_iceberg_initial_defaults: writes rows 20-22 with DEFAULT(default_int) in VALUES and in INSERT ... SELECT with and without FROM, then updates them with UPDATE ... WHERE default_int = DEFAULT(default_int) and MERGE ... ON ... = DEFAULT(...), for Parquet and ORC; SELECT DEFAULT(default_int) must fail. Passes locally; with master's FE it fails at the first INSERT with "Unknown column 'default_int'" - InsertUtilsTest: VALUES and INSERT ... SELECT (from a staging relation and without FROM) resolve against the pinned writer schema; unknown column fails; a table without the capability is unchanged - RewriteDefaultExpressionTest: pinned writer column wins, no pinned schema fails, a table without the capability keeps the read default - IcebergConnectorTest, HiveConnectorMetadataSiblingDelegationTest, ConnectorPluginSurfaceTest - Behavior changed: Yes. See the release note. - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
…hema ### What problem does this PR solve? Issue Number: None Related PR: apache#66538, apache#57531, apache#65851 Problem Summary: An Iceberg write-default exists only in the writer schema that a write statement pins (IcebergWriteSchemaContext); the columns of the read schema carry no default. Master resolves DEFAULT(column) against that writer schema only in UPDATE / MERGE SET and in MERGE INSERT VALUES, and everywhere else falls back to the read column. On a v3 table whose column d has write-default 42, INSERT ... VALUES (DEFAULT(d)) and INSERT ... SELECT DEFAULT(d) without FROM fail with "Unknown column", INSERT ... SELECT DEFAULT(d) FROM t writes NULL, UPDATE ... WHERE d = DEFAULT(d) and MERGE ... ON ... d = DEFAULT(d) compare d with NULL and silently match no row, and SELECT DEFAULT(d) FROM t returns NULL. branch-4.1 and branch-4.2 (apache#66538) resolve all of these against the pinned write target, by name against the target in INSERT ... SELECT, and reject DEFAULT(column) on an Iceberg column in a statement that does not write the table. This change adds the connector capability SUPPORTS_WRITE_ONLY_DEFAULTS, declared by the Iceberg connector and inherited by iceberg-on-HMS tables. For a table that declares it, InsertUtils resolves DEFAULT(column) in INSERT VALUES and in the select lists of INSERT ... SELECT by name against the pinned writer schema, and RewriteDefaultExpression takes the default of a DEFAULT(column) bound to the table from the pinned writer column with the same field id, which covers the UPDATE and MERGE conditions, and rejects it when the statement pinned no writer schema for the table. Hive, Paimon and OLAP tables keep the current behavior. The new enum constant changes the connector plugin API surface, so the plugin API version goes from 26.0 to 27.0. ### Release note Fix: On Iceberg tables, DEFAULT(column) in INSERT VALUES, INSERT ... SELECT and the WHERE / ON conditions of UPDATE and MERGE now resolves to the column's write-default. Before, it failed, wrote NULL, or matched no row. SELECT DEFAULT(column) on an Iceberg table now fails instead of returning NULL. ### Check List (For Author) - Test: Regression test / Unit Test - test_iceberg_initial_defaults: writes rows 20-22 with DEFAULT(default_int) in VALUES and in INSERT ... SELECT with and without FROM, then updates them with UPDATE ... WHERE default_int = DEFAULT(default_int) and MERGE ... ON ... = DEFAULT(...), for Parquet and ORC; SELECT DEFAULT(default_int) must fail. Passes locally; with master's FE it fails at the first INSERT with "Unknown column 'default_int'" - InsertUtilsTest: VALUES and INSERT ... SELECT (from a staging relation and without FROM) resolve against the pinned writer schema; unknown column fails; a table without the capability is unchanged - RewriteDefaultExpressionTest: pinned writer column wins, no pinned schema fails, a table without the capability keeps the read default - IcebergConnectorTest, HiveConnectorMetadataSiblingDelegationTest, ConnectorPluginSurfaceTest - Behavior changed: Yes. See the release note. - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
…hema (#69488) ### What problem does this PR solve? Issue Number: None Related PR: #66538, #57531, #65851 Problem Summary: **In short.** On an Iceberg table, `DEFAULT(column)` anywhere except `UPDATE ... SET` and `MERGE ... VALUES` reads the default from the read schema, which carries no Iceberg write-default. This PR resolves it against the write schema that the statement pinned for its target, as branch-4.1 and branch-4.2 do. **Background.** An Iceberg `write-default` applies to rows written after it is set. So it lives in the writer schema that a write statement pins (`IcebergWriteSchemaContext`), not in the columns of the read schema. #65851 resolves `DEFAULT(column)` against that pinned schema in `UPDATE ... SET`, `MERGE ... UPDATE SET` and `MERGE ... INSERT VALUES` (`ConnectorWriteSchemaUtils`). Every other `DEFAULT(column)` either fails at binding or reaches `RewriteDefaultExpression` (#57531), which reads the default from the bound column. branch-4.1 #66538, also in branch-4.2, resolves these against the pinned write target too. It rejects `DEFAULT(column)` on an Iceberg column in a statement that does not write the table. **The problem, and what it cost.** Reproduced on a local cluster with a `file://` format-version 3 table: `CREATE TABLE t (id INT, c INT)`, `ALTER TABLE t ADD COLUMN d INT DEFAULT 42`, one row `(0, 0)`, so `d = 42`. | Statement | Master today | After this PR (same as branch-4.2) | |---|---|---| | `INSERT INTO t (id, c, d) VALUES (1, 1, DEFAULT(d))` | `Unknown column 'd' in 'table list'` | writes 42 | | `INSERT INTO t (id, c, d) SELECT 2, 2, DEFAULT(d) FROM t WHERE id = 0` | writes NULL | writes 42 | | `INSERT INTO t (id, c, d) SELECT 3, 3, DEFAULT(d)` | `Unknown column 'd'` | writes 42 | | `UPDATE t SET c = 100 WHERE d = DEFAULT(d)` | updates nothing: `d = NULL` makes the plan an empty set | updates every row, `d = 42` is pushed down to the scan | | `MERGE INTO t USING (SELECT 0 AS id) s ON t.id = s.id AND t.d = DEFAULT(t.d) WHEN MATCHED THEN UPDATE SET c = 300` | matches nothing | updates row 0 | | `SELECT DEFAULT(d) FROM t` | NULL | `DEFAULT(d) must reference a column of the table that this statement writes` | The `UPDATE` and `MERGE` cases fail silently: the statement succeeds and changes nothing. **How this PR fixes it.** - fe-core cannot tell from a column whether its connector keeps defaults in the read schema. Hive does: an HMS default shows up in both schemas. Iceberg does not. The new connector capability `SUPPORTS_WRITE_ONLY_DEFAULTS` says which kind a table is. The Iceberg connector declares it. The Hive connector passes it on to iceberg-on-HMS tables, like the other per-table capabilities, but only through the two-argument `getTableSchema`. Since #69299, a statement that pins the table at its latest snapshot reads the schema from the snapshot overload, which reflects none of them, so on an iceberg-on-HMS table such a statement keeps master's results (for example, `UPDATE t SET c = 100 WHERE d = DEFAULT(d)` still changes no row). Reflecting the sibling's capabilities there changes planning for all of them and is left to a follow-up PR. - For a target that declares it, `InsertUtils.normalizePlan` resolves `DEFAULT(column)` in `INSERT ... VALUES` and in the select lists of `INSERT ... SELECT`, including the replacements of `SELECT * REPLACE(...)`, against the pinned write schema. It does this by name, before the source relations are bound, so the name always refers to a target column, as in branch-4.2. In MySQL, `DEFAULT(column)` in a `SELECT` refers to a source column instead. A derived table in `FROM` is a query of its own and is left alone: `DEFAULT(column)` in it still names a column of its own `FROM` clause. - `RewriteDefaultExpression` takes the default for a `DEFAULT(column)` bound to such a table from the pinned write column with the same field id. That covers the `WHERE` of `UPDATE` and the `ON` of `MERGE`. A MERGE resolves it whatever its WHEN branches, also when they all delete, as branch-4.2 does. If the statement pinned no write schema for that table, it fails. A `DELETE` pins the writer schema too, for its delete sink, but it writes no row values, so `RowLevelDmlCommand` marks its target and `DEFAULT(column)` on it fails with `DEFAULT(d) cannot be used in a DELETE: the defaults of this table apply only to the rows that a statement writes`, as in branch-4.2, whose DELETE installs no write schema context. Master reads NULL there, so `DELETE FROM t WHERE d = DEFAULT(d)` deletes nothing; resolved to the write-default it would delete every row with `d = 42`. - A resolved default is cast to the type of its writer column before the expression around it is bound, as `RewriteDefaultExpression` already does for a bound `DEFAULT(column)`. The parsed literal `-1` is a TINYINT, so without the cast `bit_count(DEFAULT(c))` on an INT column would count 8 bits instead of 32. The external `UPDATE`, `MERGE` and changelog paths use the same helper. `RewriteDefaultExpression` rejects a bound `DEFAULT(column)` whose writer column has another type than the read column the statement was bound with, which happens only when the table changed outside Doris after Doris cached it, and asks to refresh the table: the expressions around it were bound against the read type. - Hive, Paimon and internal tables do not declare the capability, so they behave as before. - The new enum constant changes the connector plugin API surface, so `connector.plugin.api.version` goes from 26.0 to 27.0, as for the other recent changes to that surface (#69299, #69588, #69298, #69683 and #69835 took 22.0 to 26.0). **Where `DEFAULT(column)` resolves on an Iceberg target (direct Iceberg catalogs).** | Statement | Position of `DEFAULT(column)` | Result | branch-4.2 | |---|---|---|---| | `INSERT` / `INSERT OVERWRITE` | `VALUES` rows | the target column's write-default, typed as the column | same | | | the INSERT's own select lists (with or without `FROM`, set operation branches, `SELECT * REPLACE`) | the target column's write-default, by name, before binding | same, except `* REPLACE`, which branch-4.2 binds to the source | | | `GROUP BY` / `ROLLUP` outputs, a derived table in `FROM`, `WITH` definitions, subqueries, `WHERE` / `ON` | bound like any `DEFAULT(column)` (last row) | same, except derived tables, which branch-4.2 resolves against the target | | | a `VALUES` branch of a set operation | fails with an unknown column, as on master | same | | | a qualifier other than the target (`DEFAULT(src.c)`, `DEFAULT(ctl.db.t.c)`) in the positions above | fails: `column in assignment list is invalid` | same | | `UPDATE` | `SET` / `WHERE` | write-default | same | | `MERGE` | `ON` and `WHEN ... AND` conditions, whatever the WHEN branches, also when they all delete | write-default | same | | | `UPDATE SET` / `INSERT VALUES` | write-default | same | | `DELETE` (also `DELETE ... USING`) | anywhere | fails: `cannot be used in a DELETE` | fails: its DELETE has no write schema context | | any | a bound `DEFAULT(x.c)` | `x` is the written table: its write-default by field id; an OLAP or Hive table: that column's default; another Iceberg table: fails with `must reference a column of the table that this statement writes` | same | A bound `DEFAULT(column)` whose writer column has another type than the read column fails and asks to refresh the table. Where `DEFAULT(column)` is bound rather than resolved by name (the third row), it keeps its meaning from before this PR and names a column of its own `FROM` clause, as on master and branch-4.2: for example in `INSERT INTO t SELECT id, DEFAULT(d) FROM src GROUP BY id, d` with an OLAP or Hive `src`, it takes the default of `src.d`. **Results.** - Local cluster, with the table above: the table shows what each statement did before and after. `EXPLAIN UPDATE t SET c = 5 WHERE d = DEFAULT(d)` goes from an empty set to a scan with `d = 42` pushed down to Iceberg. - Unit tests: 6 new tests in `InsertUtilsTest`, 6 in the new `RewriteDefaultExpressionTest` and 1 in `RowLevelDmlCommandTest`. They cover VALUES, and INSERT ... SELECT both from a staging relation whose own column is also `name` and without FROM: each resolves to the pinned default, and an unknown column fails. `DEFAULT(column)` inside a derived table, qualified or not and also under a UNION branch, is left to binding. A `SELECT * REPLACE(DEFAULT(name) AS name)` replacement resolves like the rest of the select list and keeps the star's qualifier and `EXCEPT` list. The pinned write column wins over the read column. A statement that writes nothing fails. A table without the capability is unchanged. With master's `InsertUtils` and `RewriteDefaultExpression`, the 5 tests of the fix fail and the 2 tests of the unchanged path pass. The `SELECT * REPLACE` test fails with this PR's previous `InsertUtils`, where the replacement stays `DEFAULT(name)`. `IcebergConnectorTest`, `HiveConnectorMetadataSiblingDelegationTest` and `ConnectorPluginSurfaceTest` pass. So do the fe-core tests around the write schema: `ConnectorWriteSchemaUtilsTest`, `ExternalRowLevelUpdatePlanBuilderTest`, `ExternalRowLevelMergePlanBuilderTest`, `ChangelogRowLevelDmlTransformTest` and `PluginDrivenExternalTableTest`. Two of the tests pin the typing: `bit_count(DEFAULT(bits))` gets the default as an INT in VALUES and in SELECT, and fails without the cast; a BIGINT writer column over an INT read column is rejected, directly and under an IF bound to INT. `DEFAULT(column)` on the target of a DELETE fails even with a pinned write-default, and only a DELETE marks its target (an UPDATE does not); skipping the check, or marking UPDATE instead of DELETE, makes exactly these two tests fail. `PositionDeleteRowLevelDmlTransformTest`, `PhysicalPlanTranslatorIcebergRowLevelDmlTest`, `DeleteFromCommandTest` and `UpdateCommandTest` pass. - Regression: `test_iceberg_initial_defaults` now writes rows 20 to 22 with `DEFAULT(default_int)` in VALUES and in INSERT ... SELECT with and without FROM. It then updates them with `WHERE default_int = DEFAULT(default_int)` and `MERGE ... ON ... = DEFAULT(...)`, for Parquet and ORC, and checks the rows in the `.out`. A plain `SELECT DEFAULT(default_int)` must fail, and so must a `DELETE ... WHERE default_int = DEFAULT(default_int)` on rows 20 to 22, which hold the write-default. The suite passed locally before the DELETE block was added, and the whole suite, DELETE block included, has since passed in two External regression runs on CI. With master's FE it fails at the first of these statements: `Unknown column 'default_int' in 'table list'`.
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Release note
Return the default value of column
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)