Repository navigation
Commit 388e935
authored
[fix](file-format) Align Parquet and ORC timestamp semantics (#68441)
### What problem does this PR solve?
Related PR: #67784. This is the third split, following #68297 and
#68381, targeting `branch-4.1`.
Parquet reads and writes can conflate wall-clock timestamps with UTC
instants. ORC decoding truncates nanoseconds while pruning and writer
statistics use different precision, which can shift values or discard
matching rows.
- Keep unannotated INT96 as DATETIMEV2 and carry an explicit
`hive.parquet.time-zone` contract through schema inference, scanning,
nested/filter-only projections and Hive writes. An unset or empty
property selects wall-clock INT96; a named zone decodes values
normalized in that zone. Scanner selection and condition-cache keys
preserve this interpretation.
- Default Parquet OUTFILE/EXPORT to INT64 logical timestamps: DATETIMEV2
has `isAdjustedToUTC=false`, while TIMESTAMPTZ retains instant
semantics. Keep explicit `enable_int96_timestamps=true`, validate its
boolean value, and pass it through EXPORT. Hive writes retain INT96.
- Round ORC nanoseconds to microseconds consistently, distinguish
civil-time carry from instant carry at DST transitions, and reject
invalid nanoseconds or years before narrowing into packed storage. Align
aggregate statistics, conservative SARG bounds and writer bounds; reject
the pre-epoch fractional interval that ORC cannot write losslessly.
- Preserve Paimon timestamp semantics recursively on the FE wire and
apply the effective file types before TableReader builds mappings and
blocks. Keep nested overrides for full and predicate-only projections,
and isolate Paimon from Hive INT96 timezone properties. Advertise native
Parquet in hybrid Paimon/Hudi scans, including deferred Hudi splits, so
scanner selection honors the contract.
This split preserves existing external type-mapping defaults. It
includes the Paimon read-side prerequisites for the INT96 change;
remaining Iceberg/Paimon bindings stay in the next split. It adds no
binary computation support.
Unannotated raw INT96 intentionally remains DATETIMEV2. This PR does not
restore the old mapping override or add a mixed-version writer
compatibility gate.
### Release note
Parquet OUTFILE/EXPORT now defaults to INT64 logical timestamps; set
`enable_int96_timestamps=true` when INT96 output is required.
`hive.parquet.time-zone` explicitly controls INT96 normalization. ORC
timestamp rounding and pruning now agree at microsecond and DST
boundaries; lossy pre-epoch writes fail explicitly.
### Validation
- Before the review fixes: built the official ASAN BE unit-test target
and passed 622 selected tests; passed 43 selected FE tests and parsed 21
changed regression Groovy scripts.
- Review-fix validation: all 446 selected BE tests in 18 suites passed
under ASAN, including real INT96 decoding through
PaimonReader/TableReader, hidden nested filters, NULLs and both
timestamp-mapping settings. All 104 FE tests in five classes passed,
including recursive Thrift round trips and eager/deferred hybrid scan
planning.
- FE Checkstyle and clang-format 16 checks passed for all affected
files.
- Regression evidence: five new FE tests and four BE tests failed before
their fixes. The full native Paimon read test additionally reproduced a
nested predicate-only column type mismatch before fixing semantic
propagation from the annotated file schema.
- External SQL/Hive/HDFS regressions were not executed locally; full
regression validation is pending CI.
### External regression follow-up
Updated timestamp goldens and scanner assertions for native ORC
rounding, raw INT96 reads, and required V2 dispatch. The INT64 export
cases now read their own output files. The malformed timestamp case
explicitly selects strict conversion and restores the previous setting.
Existing exact pruning counters and ordered result checks remain
covered.
Validation: 360 related ASAN BE tests passed, all eight changed Groovy
scripts parsed successfully, and updated CI result rows and golden
structure were checked. The Parquet profile regression now checks the
same exact V2 pruning counters across all four legacy min/max and
lazy-read switch combinations: versioned Parquet requires V2, which
always applies safe metadata pruning and lazy materialization. Follow-up
validation passed all 93 selected Parquet reader/statistics ASAN tests,
Groovy compilation, and 28 profile checks against recorded CI metrics
(not a full SQL regression run). Full external regression reruns are
pending CI. The remaining Lance ranking, object-storage timeout, OBS
dependency and large-string memory failures also occur on independent PR
builds and are outside this change.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- Behavior changed:
- [x] Yes. Timestamp representation, interpretation and precision
changes are described above.
- Does this need documentation?
- [x] Yes. The INT64 default and `hive.parquet.time-zone` should be
reflected in the file-format documentation.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label1 parent e0b5cf2 commit 388e935
123 files changed
Lines changed: 3839 additions & 770 deletions
File tree
- be
- src
- core/data_type_serde
- exec
- operator
- sink/writer
- format_v2
- orc
- parquet
- reader
- native
- table
- format
- parquet
- table/hive
- transformer
- test
- exec
- scan
- format_v2
- orc
- parquet
- table
- format
- arrow
- transformer
- fe/fe-core/src
- main/java/org/apache/doris
- common/util
- datasource
- hive
- hudi/source
- paimon
- source
- property/fileformat
- load
- nereids/trees/plans/commands
- planner
- tablefunction
- test/java/org/apache/doris
- datasource
- hive
- hudi/source
- paimon
- source
- property/fileformat
- load
- nereids/trees/plans/commands
- planner
- tablefunction
- gensrc/thrift
- regression-test
- data
- export_p0/outfile/parquet
- external_table_p0
- export/hive_read/parquet
- iceberg
- paimon
- tvf
- orc_tvf
- suites
- datatype_p0/nested_types/query
- external_table_p0
- export/hive_read/parquet
- hive
- ddl
- write
- iceberg
- write
- paimon
- tvf
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
73 | 73 | | |
74 | 74 | | |
75 | 75 | | |
76 | | - | |
77 | | - | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
78 | 94 | | |
79 | 95 | | |
80 | 96 | | |
| |||
838 | 854 | | |
839 | 855 | | |
840 | 856 | | |
| 857 | + | |
| 858 | + | |
| 859 | + | |
| 860 | + | |
| 861 | + | |
| 862 | + | |
| 863 | + | |
841 | 864 | | |
842 | 865 | | |
843 | 866 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
56 | 56 | | |
57 | 57 | | |
58 | 58 | | |
59 | | - | |
60 | | - | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
61 | 74 | | |
62 | 75 | | |
63 | 76 | | |
| |||
492 | 505 | | |
493 | 506 | | |
494 | 507 | | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
495 | 515 | | |
496 | 516 | | |
497 | 517 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
41 | 41 | | |
42 | 42 | | |
43 | 43 | | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
44 | 91 | | |
45 | 92 | | |
46 | 93 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
| 27 | + | |
27 | 28 | | |
28 | 29 | | |
29 | 30 | | |
| |||
32 | 33 | | |
33 | 34 | | |
34 | 35 | | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
35 | 49 | | |
36 | 50 | | |
37 | 51 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
33 | 33 | | |
34 | 34 | | |
35 | 35 | | |
| 36 | + | |
36 | 37 | | |
37 | 38 | | |
38 | 39 | | |
| |||
147 | 148 | | |
148 | 149 | | |
149 | 150 | | |
150 | | - | |
151 | | - | |
152 | | - | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
153 | 171 | | |
154 | 172 | | |
155 | 173 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
50 | 50 | | |
51 | 51 | | |
52 | 52 | | |
53 | | - | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
54 | 58 | | |
55 | 59 | | |
56 | 60 | | |
| |||
97 | 101 | | |
98 | 102 | | |
99 | 103 | | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
100 | 110 | | |
101 | 111 | | |
102 | 112 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
| 22 | + | |
| 23 | + | |
22 | 24 | | |
23 | 25 | | |
24 | 26 | | |
| |||
101 | 103 | | |
102 | 104 | | |
103 | 105 | | |
| 106 | + | |
104 | 107 | | |
105 | 108 | | |
106 | 109 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
31 | 31 | | |
32 | 32 | | |
33 | 33 | | |
34 | | - | |
| 34 | + | |
35 | 35 | | |
36 | 36 | | |
37 | | - | |
| 37 | + | |
| 38 | + | |
38 | 39 | | |
39 | 40 | | |
40 | 41 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
27 | | - | |
| 27 | + | |
| 28 | + | |
28 | 29 | | |
29 | 30 | | |
30 | 31 | | |
31 | | - | |
| 32 | + | |
| 33 | + | |
32 | 34 | | |
33 | 35 | | |
34 | 36 | | |
| |||
41 | 43 | | |
42 | 44 | | |
43 | 45 | | |
| 46 | + | |
44 | 47 | | |
45 | 48 | | |
46 | 49 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
232 | 232 | | |
233 | 233 | | |
234 | 234 | | |
235 | | - | |
236 | | - | |
237 | | - | |
238 | | - | |
239 | | - | |
240 | | - | |
241 | | - | |
242 | | - | |
243 | | - | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
244 | 239 | | |
245 | 240 | | |
246 | 241 | | |
| |||
0 commit comments