You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit fbe6a9d
Browse filesBrowse the repository at this point in the historyBrowse files
committed
[fix](paimon) Cast static overwrite partition values to their column types
### What problem does this PR solve?
Issue Number: None
Related PR: #65868
Problem Summary: An INSERT OVERWRITE with a static PARTITION clause named the
Paimon partition by the literal as written: PluginDrivenInsertCommandContext
keeps Literal.getStringValue() of the user's literal, and PaimonWriteBinding
handed it to Paimon's withOverwrite unchanged, while the written rows carry the
literal cast to the column type. On a local filesystem catalog,
PARTITION (p = true) on an INT column failed with NumberFormatException,
PARTITION (dt = 20240101) on a DATE column failed with "the changes in
dt=19723/ does not belong to this partition", and an overwrite of a TIMESTAMP
WITH LOCAL TIME ZONE partition failed the same way, because Paimon parses that
value as local time in the FE JVM zone while the rows were written in the
session zone. An overwrite without rows passes Paimon's check and replaced the
wrong partition silently: an empty LTZ overwrite deleted the partition that
lies the session's offset from the JVM zone away and kept the named one, and
an empty overwrite of a value equal to partition.default-name deleted the NULL
partition, which is what Paimon reads that string as.
branch-4.1 #65868 casts each static partition literal to its column type,
moves an LTZ value from the session zone to the FE JVM zone in Paimon's
timestamp format, and rejects a value equal to partition.default-name. Master
builds the static partition spec in fe-core and hands it to connectors through
ConnectorWriteHandle, so the cast values are offered by a new SPI getter,
getCastStaticPartitionSpec, beside the spec as written. fe-core casts each
literal to the type of its column in the bound target schema, the cast that
constant folding applies to the value materialized into the rows, and only
when a connector asks: Iceberg, MaxCompute and Hive keep reading the spec as
written and see no change. Paimon's write binding resolves its static
partition from the cast values; an LTZ column mapped to TIMESTAMPTZ yields a
UTC instant with its offset, which moves to the JVM zone as well. A literal
that does not fit its column type now fails the statement, as on branch-4.x,
where master wrote NULL into the partition column.
### Release note
Fix INSERT OVERWRITE ... PARTITION on Paimon tables failing, or replacing the
wrong partition, when a static partition value is not spelled in its column
type (p = true on an INT column, dt = 20240101 on a DATE column) or the column
is TIMESTAMP WITH LOCAL TIME ZONE. A static partition value equal to the
table's partition.default-name, or one that is not a valid value of its column
type, is now rejected.
### Check List (For Author)
- Test: Unit Test, Manual test; regression test added but not run locally
- PaimonWriteBindingTest ports the branch-4.x cases for the cast value, the
partition.default-name rejection and the LTZ zone move, and adds the
TIMESTAMPTZ mapping and SQL NULL. PluginDrivenTableSinkBindingTest checks
the cast values on the write handle, the session-zone forms of DATETIMEV2
and TIMESTAMPTZ values, and that an invalid value fails only when cast.
ConnectorWriteHandleTest pins the SPI default. Reading the spec as
written, dropping the default-name check, skipping either zone move, or
skipping the cast in fe-core makes them fail.
- test_paimon_write_transaction gains the INT, DATE, LTZ and default-name
cases. It needs the Paimon docker environment and was not run locally;
the same statements on a local filesystem Paimon catalog give the
asserted rows with this change and fail or delete the wrong partition
on master.
- Behavior changed: Yes (see Release note)
- Does this need documentation: No
Co-authored-by: Socrates <[email protected]>
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnectorMetadata.java
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonWriteBinding.java
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonWritePlanProvider.java
+1-2Lines changed: 1 addition & 2 deletions
Original file line number
Diff line number
Diff line change
@@ -152,8 +152,7 @@ public ConnectorSinkPlan planWrite(ConnectorSession session, ConnectorWriteHandl
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonBackendBoundTableTest.java
0 commit comments