You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit f7332ed
Browse filesBrowse the repository at this point in the historyBrowse files
[fix](external) Preserve external binary and timestamp mappings (apache#68532)
### What problem does this PR solve?
Related PRs: apache#67784, apache#68441
This combines splits 4 and 5 of apache#67784 on branch-4.1. External binary
columns retain their bytes as VARBINARY, and external instant timestamps
retain their time semantics as TIMESTAMPTZ instead of being exposed as
unzoned DATETIME. Iceberg and Paimon write bindings and partition
metadata preserve microseconds, negative epochs, and both instants in a
DST overlap.
- Unify Hive, Iceberg, Paimon and JDBC binary mappings, including
fixed-width and UUID values. Preserve the existing text contract for
Hive binary partition columns.
- Map instant types consistently across Hive, Iceberg, Paimon, JDBC,
Hudi, Trino connector and MaxCompute; local timestamp types remain
DATETIMEV2. Align Java/JNI carriers, nested values, JDBC projections and
predicates, and MaxCompute microsecond writes with these declared types.
Bind JDBC instant writes with dialect-aware zoned parameters, including
a supported NULL bind for Trino/Presto. Project Trino and Presto scalar
and nested-array instant timestamps to UTC before JDBC decoding.
PostgreSQL instants outside the supported range map to NULL, including
source NOT NULL columns; predicates involving these timestamps remain
local so filtering observes the decoded values.
- Normalize the legacy catalog mapping properties on CREATE/ALTER. After
master metadata replay, journal ordinary ALTER records for old
missing/false markers before query readiness; replay and migration are
idempotent. This changes persisted catalog metadata: rolling back FE
binaries alone does not restore the old properties. External files are
not rewritten. During a mixed-version FE upgrade, old and new FEs can
expose different external schemas before a new master journals this
migration; compatibility within that rolling-upgrade window is outside
this change. Old-FE/new-BE compatibility for Iceberg FIXED writes,
Hudi/MySQL timestamp reads, and MaxCompute timestamp writes is also
intentionally outside this PR; those review findings are acknowledged
and not fixed.
- Retain Iceberg/Paimon write validation, binary-safe partition
transport, and dynamic/static/hybrid overwrite behavior. Support Iceberg
binary bucket hashing and byte-counted truncate partitions, preserving
NULLs and raw binary prefixes. Route legacy Paimon ORC LTZ files through
the SDK when native decoding cannot preserve the logical instant. Reject
static Paimon LTZ overwrite keys that are ambiguous in the FE JVM time
zone.
- Permit VARBINARY in ordinary views while retaining native OLAP, CTAS
and MTMV restrictions and binary computation rejections. File TVF binary
mapping remains an explicit opt-in with default false; file timestamp
mapping follows logical annotations.
- Update affected regression schemas/results and use explicit supported
text representations where fixtures require binary computation. Preserve
CDC's JSON carriers: binary values use Base64 text and timestamps retain
the existing unzoned representation, including array elements.
- Reject VARBINARY arguments to IN/NOT IN and least/greatest/nullif
during analysis while preserving CASE/IF/COALESCE value selection.
Existing binary aggregate support and legality gaps remain outside this
change.
Validation:
- Binary partition review follow-up: reproduced both
unsupported-transform failures, then passed 34 related ASAN BE tests and
clang-format 16. Added Parquet/ORC append/overwrite regressions with
Spark byte and partition-pruning checks; Groovy parsing passed, external
execution pending CI. Bucket expectations were checked against Iceberg
Java transforms.
- JDBC write review follow-up: 25 local unit tests and FE/JDBC
Checkstyle passed after six expected failures were reproduced. Added
per-dialect JNI binding checks and five opt-in server write round trips;
77 real-driver encoding checks passed. Thirteen server integration tests
were skipped without configured connections. Corrected a PostgreSQL
regression COUNT expectation to JDBC Long; the numeric result was
already correct.
- JDBC review fixes: reproduced four failing FE test methods, then
passed 48 FE tests, 17 JDBC unit tests, and FE/JDBC Checkstyle. Added
PostgreSQL SQL regressions for decoded NULL filtering, LIMIT, null-safe
equality, NOT NULL metadata and COUNT; added Presto scalar/nested/TVF
projection tests and optional driver integration coverage. A real
PrestoSQL JDBC 350 parser check passed 18 cases across three JVM zones.
Seven server integration tests were skipped because connections were not
configured; external SQL execution remains pending CI.
- Earlier follow-up: corrected three Iceberg FE UT expectations for
mandatory logical mappings while retaining generation/authenticator
checks; 152 related FE tests, 17 JDBC unit tests and FE/JDBC Checkstyle
passed.
- A real Trino JDBC 435 parser check passed 18 cases across three JVM
zones; live-server Trino integration coverage was added but not executed
locally.
- Review follow-up: 59 targeted FE tests and FE Checkstyle passed after
reproducing four failing test methods before the fix. Coverage includes
CDC binary schemas and binary comparison rejection under both coercion
modes.
- 507 related FE unit tests passed across the main run and targeted
rerun after correcting the view assertion; FE and connector Checkstyle
passed.
- 34 Java connector unit tests passed, covering timezone mismatches,
nested timestamps, nulls, negative epochs, DST overlaps, and microsecond
writes.
- ASAN BE build and 265 related BE tests passed, including UUID
partition-byte transport.
- clang-format 16 passed for all 15 affected C/C++ files; AST parsing
passed for all 43 newly affected Groovy suites. Existing split-4 SQL
tests retain Spark epoch/byte comparisons.
- External SQL regression and real-server JDBC integration execution are
pending CI. Golden results were reviewed, not regenerated against
external services locally.
### Release note
Preserve binary and instant timestamp types across external catalogs and
connectors, migrate legacy catalog mapping properties, and align
Iceberg/Paimon writes and partition metadata. Existing external result
schemas can change from STRING/DATETIME to VARBINARY/TIMESTAMPTZ.
### Check List (For Author)
- [x] Regression tests added and affected expectations updated (external
integration execution pending CI).
- [x] Unit tests.
- [x] Behavior changed: external type mapping and persisted catalog
properties, connector timestamp handling, and external read/write
bindings.
- [x] Compatibility and rollback impact described above.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note.
- [ ] Confirm test cases.
- [ ] Confirm document.
- [ ] Add branch pick label if needed.
0 commit comments