You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 575c5ac
Browse filesBrowse the repository at this point in the historyBrowse files
[feat](datalake) Support nanosecond timestamps in Iceberg and Paimon (apache#68934)
### What problem does this PR solve?
Iceberg `timestamp_ns` was unsupported, and Paimon `timestamp(9)` was
exposed as `DATETIMEV2(6)`, losing the final three fractional digits.
Map these unzoned types bidirectionally to Doris `TIMESTAMP_NS` and
preserve all nine digits through reads, inserts and CTAS, including
arrays, maps and structs.
- Represent Iceberg `timestamp`, `timestamptz`, `timestamp_ns`, and
`timestamptz_ns` with four distinct BE classes and type IDs. Arrow
unit/timezone conversion and nested Parquet nanosecond detection
dispatch directly on these IDs.
- Add nanosecond materialization to format V2 Parquet/ORC readers and
Iceberg writers. Format V1 readers remain unchanged. Nanosecond Iceberg
schemas use Parquet 2.6; ORC uses a UTC wall-clock carrier to avoid
historical timezone-offset differences at the signed nanosecond
boundaries, preserving negative-fraction correction across external
stripe writer timezones.
- Support Iceberg identity, calendar and bucket partitions, including
date-only static partitions and exact nanosecond overwrite/conflict
filters. Support nanosecond predicate pushdown and filtered rewrites,
ADD COLUMN defaults, and statement-scoped write defaults. Omit unsafe
ORC timestamp bounds and derived MTMV RANGE boundaries.
- Select Paimon's JNI reader for projected or filter-only high-precision
fields, including nested fields, and use nanosecond Arrow transport for
writes. Preserve physical metadata columns when the SDK can expose real
file/row positions. Paimon precisions 7 and 8 also map to
`TIMESTAMP_NS`.
Iceberg nanosecond schemas require format version 3 and the format V2
reader (`enable_file_scanner_v2=true`). High-precision Paimon reads use
JNI, which may cost more than native scans. The existing ORC-645
rejection of ambiguous pre-epoch fractions is retained; Parquet supports
those values.
### Release note
Support reading and writing Iceberg `timestamp_ns` and Paimon
`timestamp(9)` as Doris `TIMESTAMP_NS`, preserving nanosecond precision.
### Validation
- External regression follow-up: `paimon_timestamp_types`,
`test_paimon_minio`, and `test_paimon_table_properties` all passed
locally after regenerating their goldens from actual execution. Coverage
retains scanner-V2-disabled JNI reads, both S3/MinIO property aliases,
and SHOW CREATE TABLE for precision-7/8/9 columns.
Current review fixes:
- 270 focused FE tests passed, including real Iceberg schema/transaction
commits, rewrite partition pruning, and Paimon ORC/Parquet scan
planning. Connector packages and FE Checkstyle passed.
- 263 BE tests passed under ASAN. External ORC writer fixtures cover
GMT, Asia/Shanghai and America/New_York, epoch crossings, fractional
precision, rounding and sparse reads.
- `test_iceberg_timestamp_ns` and `test_paimon_timestamp_ns` passed
locally with rebuilt FE connector plugins. New assertions cover filtered
rewrites, explicit/omitted-column inserts with defaults, date-only
static overwrite, and nanosecond columns with physical metadata. The
Iceberg suite uses the existing newer REST fixture because the baseline
1.10 server incorrectly rescales nanosecond schema defaults. ORC
decoding changes were validated separately by the ASAN tests above.
- All nine changed C++ files passed clang-format 16, clang-tidy and
build hygiene checks.
Earlier feature validation:
- Full FE and ASAN BE builds; FE Checkstyle, clang-format 16 and
clang-tidy over all 25 changed C++ files.
- 2,112 relevant Java tests: zero failures, five existing skips.
- 117 relevant BE tests under ASAN, including independent Arrow/ORC file
read-back, signed bounds, dictionary decoding and overflow handling.
- Three end-to-end regression suites passed against the generated
goldens: `test_iceberg_timestamp_ns`, `test_paimon_timestamp_ns`, and
`paimon_timestamp_types`. Coverage includes Iceberg format V2 reads,
Paimon JNI reads with both scanner settings, session time zones, nested
values, filters, signed bounds, CTAS, Iceberg partition
transforms/overwrite, and Paimon primary-key upsert.
- Independent Flink read-back of four Doris-written Paimon tables: 28
rows across Parquet/ORC and append/primary-key tables, with exact
nanosecond bounds and upserts.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [x] Manual test (independent Flink read-back described above)
- Behavior changed:
- [x] Yes. External unzoned nanosecond timestamps retain nine fractional
digits.
- Does this need documentation?
- [x] Yes. The external type-mapping tables should describe
`TIMESTAMP_NS`.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
0 commit comments