Skip to content

Commit d3772cb

Browse files
authored
[fix](iceberg) Preserve timestamp precision and semantics in metadata scans (apache#68985)
### What problem does this PR solve? Issue Number: N/A Related PR: apache#68276 (supersedes its metadata-scanner fix; this PR targets master) Problem Summary: Iceberg metadata table timestamps are encoded as epoch microseconds. The JNI scanner converts them through milliseconds, losing fractional precision. It also applies the session time zone to timestamp-without-zone partition values, changing their wall-clock value. For example, a `$files.partition` timestamp of `1234567` microseconds is truncated to `1970-01-01 00:00:01.234000` rather than `1970-01-01 00:00:01.234567`. Switching from UTC to Asia/Shanghai incorrectly shifts timestamp-without-zone values by eight hours. This PR: - Decodes signed epoch microseconds with floor division/modulus, preserving pre-epoch fractions. - Propagates Iceberg types through struct/list/map values to distinguish wall-clock timestamps from instants. - Resolves requested field types by name while continuing to read projected rows by position. - Adds unit tests and a `$files.partition` regression covering `-1000001`, `-1`, `1`, `999`, `1000`, `1234567`, and NULL, UTC/Asia/Shanghai, both timestamp mapping settings, and a projected partition child. Scope is the Java Iceberg metadata scanner. This PR does not modify regular data-table timestamp reads or the FE/native `$position_deletes` serialization path. There are no storage-format or configuration changes. ### Release note Fix microsecond precision loss and incorrect session-time-zone shifts for timestamp-without-zone values in Iceberg metadata tables. ### Check List (For Author) - Test - [x] Regression test - `test_iceberg_files_timestamp_partition`: generated the golden output, checked its timestamp values, and reran normal comparison successfully (1 suite, 0 failures/skips). - `test_iceberg_system_table_projection`: passed. - `test_iceberg_position_deletes_sys_table`: passed. - [x] Unit Test - `IcebergSysTableColumnValueTest`: 2 tests passed; the FE/Java-extension reactor completed successfully. - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason Local External tests used an isolated Iceberg fixture stack and the newly packaged metadata-scanner plugin deployed to the personal Doris BE. The full FE/BE deployment build was interrupted during C++ submodule downloads; FE/BE executables were not rebuilt for these runs. These are targeted local results, not full community External/P0/coverage results. - Behavior changed: - [ ] No. - [x] Yes. Metadata timestamps retain microseconds; timestamp-without-zone values no longer shift with the session time zone. Timestamp-with-zone values preserve their instant semantics. - Does this need documentation? - [x] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
1 parent 846f932 commit d3772cb

6 files changed

Lines changed: 265 additions & 11 deletions

File tree

‎fe/be-java-extensions/iceberg-metadata-scanner/src/main/java/org/apache/doris/iceberg/IcebergSysTableColumnValue.java‎

Lines changed: 28 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,8 @@
2020
import org.apache.doris.jni.spi.vec.ColumnValue;
2121

2222
import org.apache.iceberg.StructLike;
23+
import org.apache.iceberg.types.Type;
24+
import org.apache.iceberg.types.Types;
2325

2426
import java.math.BigDecimal;
2527
import java.math.BigInteger;
@@ -38,14 +40,20 @@ public class IcebergSysTableColumnValue implements ColumnValue {
3840

3941
private final Object fieldData;
4042
private final String timezone;
43+
private final Type icebergType;
4144

4245
public IcebergSysTableColumnValue(Object fieldData) {
4346
this(fieldData, DEFAULT_TIME_ZONE);
4447
}
4548

4649
public IcebergSysTableColumnValue(Object fieldData, String timezone) {
50+
this(fieldData, timezone, null);
51+
}
52+
53+
public IcebergSysTableColumnValue(Object fieldData, String timezone, Type icebergType) {
4754
this.fieldData = fieldData;
4855
this.timezone = timezone;
56+
this.icebergType = icebergType;
4957
}
5058

5159
@Override
@@ -136,14 +144,22 @@ public LocalDate getDate() {
136144

137145
@Override
138146
public LocalDateTime getDateTime() {
139-
Instant instant = Instant.ofEpochMilli((((long) fieldData) / 1000));
140-
return LocalDateTime.ofInstant(instant, ZoneId.of(timezone));
147+
// NTZ is a wall-clock value; only timestamptz mapped to DATETIME uses the session zone.
148+
ZoneId zone = icebergType instanceof Types.TimestampType
149+
&& !((Types.TimestampType) icebergType).shouldAdjustToUTC()
150+
? ZoneOffset.UTC : ZoneId.of(timezone);
151+
return LocalDateTime.ofInstant(timestampInstant(), zone);
141152
}
142153

143154
@Override
144155
public LocalDateTime getTimeStampTz() {
145-
Instant instant = Instant.ofEpochMilli((((long) fieldData) / 1000));
146-
return LocalDateTime.ofInstant(instant, ZoneId.of("UTC"));
156+
return LocalDateTime.ofInstant(timestampInstant(), ZoneOffset.UTC);
157+
}
158+
159+
private Instant timestampInstant() {
160+
long micros = (long) fieldData;
161+
return Instant.ofEpochSecond(Math.floorDiv(micros, 1_000_000L),
162+
Math.floorMod(micros, 1_000_000L) * 1_000L);
147163
}
148164

149165
@Override
@@ -162,16 +178,19 @@ public byte[] getBytes() {
162178
public void unpackArray(List<ColumnValue> values) {
163179
List<?> items = (List<?>) fieldData;
164180
for (Object item : items) {
165-
values.add(new IcebergSysTableColumnValue(item, timezone));
181+
values.add(new IcebergSysTableColumnValue(item, timezone,
182+
icebergType == null ? null : icebergType.asListType().elementType()));
166183
}
167184
}
168185

169186
@Override
170187
public void unpackMap(List<ColumnValue> keys, List<ColumnValue> values) {
171188
Map<?, ?> data = (Map<?, ?>) fieldData;
172189
data.forEach((key, value) -> {
173-
keys.add(new IcebergSysTableColumnValue(key, timezone));
174-
values.add(new IcebergSysTableColumnValue(value, timezone));
190+
keys.add(new IcebergSysTableColumnValue(key, timezone,
191+
icebergType == null ? null : icebergType.asMapType().keyType()));
192+
values.add(new IcebergSysTableColumnValue(value, timezone,
193+
icebergType == null ? null : icebergType.asMapType().valueType()));
175194
});
176195
}
177196

@@ -180,7 +199,8 @@ public void unpackStruct(List<Integer> structFieldIndex, List<ColumnValue> value
180199
StructLike record = (StructLike) fieldData;
181200
for (Integer fieldIndex : structFieldIndex) {
182201
Object rawValue = record.get(fieldIndex, Object.class);
183-
values.add(new IcebergSysTableColumnValue(rawValue, timezone));
202+
values.add(new IcebergSysTableColumnValue(rawValue, timezone,
203+
icebergType == null ? null : icebergType.asStructType().fields().get(fieldIndex).type()));
184204
}
185205
}
186206
}

‎fe/be-java-extensions/iceberg-metadata-scanner/src/main/java/org/apache/doris/iceberg/IcebergSysTableJniScanner.java‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@
2727
import org.apache.iceberg.FileScanTask;
2828
import org.apache.iceberg.StructLike;
2929
import org.apache.iceberg.io.CloseableIterator;
30+
import org.apache.iceberg.types.Type;
3031
import org.slf4j.Logger;
3132
import org.slf4j.LoggerFactory;
3233

@@ -44,6 +45,7 @@ public class IcebergSysTableJniScanner extends JniScanner {
4445
private final PreExecutionAuthenticator preExecutionAuthenticator;
4546
private final FileScanTask scanTask;
4647
private final int requiredFieldCount;
48+
private final Type[] icebergTypes;
4749
private final String timezone;
4850
private CloseableIterator<StructLike> reader;
4951

@@ -57,6 +59,13 @@ public IcebergSysTableJniScanner(int batchSize, Map<String, String> params) {
5759
"required_fields should not be empty");
5860
String[] requiredFields = requiredFieldsParam.split(",");
5961
this.requiredFieldCount = requiredFields.length;
62+
this.icebergTypes = new Type[requiredFieldCount];
63+
for (int i = 0; i < requiredFieldCount; i++) {
64+
// Resolve types by name, but read projected ROWS by position below. StaticDataTask.schema()
65+
// can be the full schema even when its rows contain only the requested fields.
66+
this.icebergTypes[i] = scanTask.schema().findType(requiredFields[i]);
67+
Preconditions.checkNotNull(icebergTypes[i], "Missing Iceberg metadata field %s", requiredFields[i]);
68+
}
6069
this.timezone = params.getOrDefault("time_zone", TimeZone.getDefault().getID());
6170
Map<String, String> hadoopOptionParams = params.entrySet().stream()
6271
.filter(kv -> kv.getKey().startsWith(HADOOP_OPTION_PREFIX))
@@ -104,7 +113,7 @@ protected int getNext() throws IOException {
104113
// narrowed StructProjection, so a full-schema ordinal overruns the projected row (upstream
105114
// #65262 -- reverting this to a by-name/schema() lookup reintroduces ArrayIndexOutOfBounds).
106115
Object value = row.get(i, Object.class);
107-
ColumnValue columnValue = new IcebergSysTableColumnValue(value, timezone);
116+
ColumnValue columnValue = new IcebergSysTableColumnValue(value, timezone, icebergTypes[i]);
108117
appendData(i, columnValue);
109118
}
110119
rows++;
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
package org.apache.doris.iceberg;
19+
20+
import org.apache.doris.jni.spi.vec.ColumnValue;
21+
22+
import org.apache.iceberg.PartitionData;
23+
import org.apache.iceberg.types.Types;
24+
import org.junit.jupiter.api.Assertions;
25+
import org.junit.jupiter.api.Test;
26+
27+
import java.time.LocalDateTime;
28+
import java.util.ArrayList;
29+
import java.util.Arrays;
30+
import java.util.Collections;
31+
import java.util.List;
32+
33+
public class IcebergSysTableColumnValueTest {
34+
@Test
35+
public void preservesMicrosAndTimestampSemantics() {
36+
long[] micros = {-1000001L, -1L, 1L, 999L, 1000L, 1234567L};
37+
String[] expected = {"1969-12-31T23:59:58.999999", "1969-12-31T23:59:59.999999",
38+
"1970-01-01T00:00:00.000001", "1970-01-01T00:00:00.000999",
39+
"1970-01-01T00:00:00.001", "1970-01-01T00:00:01.234567"};
40+
for (int i = 0; i < micros.length; i++) {
41+
LocalDateTime utc = LocalDateTime.parse(expected[i]);
42+
for (String zone : Arrays.asList("UTC", "Asia/Shanghai")) {
43+
ColumnValue ntz = new IcebergSysTableColumnValue(micros[i], zone, Types.TimestampType.withoutZone());
44+
ColumnValue tz = new IcebergSysTableColumnValue(micros[i], zone, Types.TimestampType.withZone());
45+
Assertions.assertEquals(utc, ntz.getDateTime());
46+
Assertions.assertEquals(zone.equals("UTC") ? utc : utc.plusHours(8), tz.getDateTime());
47+
Assertions.assertEquals(utc, tz.getTimeStampTz());
48+
}
49+
}
50+
}
51+
52+
@Test
53+
public void propagatesTypesThroughNestedContainers() {
54+
Types.StructType struct = Types.StructType.of(
55+
Types.NestedField.optional(1, "ts", Types.TimestampType.withoutZone()),
56+
Types.NestedField.optional(2, "tz", Types.TimestampType.withZone()));
57+
PartitionData record = new PartitionData(struct);
58+
record.set(0, -1L);
59+
record.set(1, -1L);
60+
List<ColumnValue> fields = new ArrayList<>();
61+
new IcebergSysTableColumnValue(record, "Asia/Shanghai", struct)
62+
.unpackStruct(Arrays.asList(1, 0), fields);
63+
LocalDateTime utc = LocalDateTime.parse("1969-12-31T23:59:59.999999");
64+
Assertions.assertEquals(utc.plusHours(8), fields.get(0).getDateTime());
65+
Assertions.assertEquals(utc, fields.get(1).getDateTime());
66+
67+
List<ColumnValue> elements = new ArrayList<>();
68+
new IcebergSysTableColumnValue(Collections.singletonList(-1L), "Asia/Shanghai",
69+
Types.ListType.ofOptional(3, Types.TimestampType.withoutZone())).unpackArray(elements);
70+
Assertions.assertEquals(utc, elements.get(0).getDateTime());
71+
72+
List<ColumnValue> keys = new ArrayList<>();
73+
List<ColumnValue> values = new ArrayList<>();
74+
new IcebergSysTableColumnValue(Collections.singletonMap(-1L, -1L), "Asia/Shanghai",
75+
Types.MapType.ofOptional(4, 5, Types.TimestampType.withoutZone(), Types.TimestampType.withZone()))
76+
.unpackMap(keys, values);
77+
Assertions.assertEquals(utc, keys.get(0).getDateTime());
78+
Assertions.assertEquals(utc.plusHours(8), values.get(0).getDateTime());
79+
}
80+
}
Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
-- This file is automatically generated. You should know what you did if you want to edit this
2+
-- !false_UTC --
3+
{"ts":"1969-12-31 23:59:58.999999", "tz":"1969-12-31 23:59:58.999999+00:00"}
4+
{"ts":"1969-12-31 23:59:59.999999", "tz":"1969-12-31 23:59:59.999999+00:00"}
5+
{"ts":"1970-01-01 00:00:00.000001", "tz":"1970-01-01 00:00:00.000001+00:00"}
6+
{"ts":"1970-01-01 00:00:00.000999", "tz":"1970-01-01 00:00:00.000999+00:00"}
7+
{"ts":"1970-01-01 00:00:00.001000", "tz":"1970-01-01 00:00:00.001000+00:00"}
8+
{"ts":"1970-01-01 00:00:01.234567", "tz":"1970-01-01 00:00:01.234567+00:00"}
9+
{"ts":null, "tz":null}
10+
11+
-- !false_UTC_ntz --
12+
\N
13+
1969-12-31T23:59:58.999999
14+
1969-12-31T23:59:59.999999
15+
1970-01-01T00:00:00.000001
16+
1970-01-01T00:00:00.000999
17+
1970-01-01T00:00:00.001
18+
1970-01-01T00:00:01.234567
19+
20+
-- !false_Asia_Shanghai --
21+
{"ts":"1969-12-31 23:59:58.999999", "tz":"1970-01-01 07:59:58.999999+08:00"}
22+
{"ts":"1969-12-31 23:59:59.999999", "tz":"1970-01-01 07:59:59.999999+08:00"}
23+
{"ts":"1970-01-01 00:00:00.000001", "tz":"1970-01-01 08:00:00.000001+08:00"}
24+
{"ts":"1970-01-01 00:00:00.000999", "tz":"1970-01-01 08:00:00.000999+08:00"}
25+
{"ts":"1970-01-01 00:00:00.001000", "tz":"1970-01-01 08:00:00.001000+08:00"}
26+
{"ts":"1970-01-01 00:00:01.234567", "tz":"1970-01-01 08:00:01.234567+08:00"}
27+
{"ts":null, "tz":null}
28+
29+
-- !false_Asia_Shanghai_ntz --
30+
\N
31+
1969-12-31T23:59:58.999999
32+
1969-12-31T23:59:59.999999
33+
1970-01-01T00:00:00.000001
34+
1970-01-01T00:00:00.000999
35+
1970-01-01T00:00:00.001
36+
1970-01-01T00:00:01.234567
37+
38+
-- !true_UTC --
39+
{"ts":"1969-12-31 23:59:58.999999", "tz":"1969-12-31 23:59:58.999999+00:00"}
40+
{"ts":"1969-12-31 23:59:59.999999", "tz":"1969-12-31 23:59:59.999999+00:00"}
41+
{"ts":"1970-01-01 00:00:00.000001", "tz":"1970-01-01 00:00:00.000001+00:00"}
42+
{"ts":"1970-01-01 00:00:00.000999", "tz":"1970-01-01 00:00:00.000999+00:00"}
43+
{"ts":"1970-01-01 00:00:00.001000", "tz":"1970-01-01 00:00:00.001000+00:00"}
44+
{"ts":"1970-01-01 00:00:01.234567", "tz":"1970-01-01 00:00:01.234567+00:00"}
45+
{"ts":null, "tz":null}
46+
47+
-- !true_UTC_ntz --
48+
\N
49+
1969-12-31T23:59:58.999999
50+
1969-12-31T23:59:59.999999
51+
1970-01-01T00:00:00.000001
52+
1970-01-01T00:00:00.000999
53+
1970-01-01T00:00:00.001
54+
1970-01-01T00:00:01.234567
55+
56+
-- !true_Asia_Shanghai --
57+
{"ts":"1969-12-31 23:59:58.999999", "tz":"1970-01-01 07:59:58.999999+08:00"}
58+
{"ts":"1969-12-31 23:59:59.999999", "tz":"1970-01-01 07:59:59.999999+08:00"}
59+
{"ts":"1970-01-01 00:00:00.000001", "tz":"1970-01-01 08:00:00.000001+08:00"}
60+
{"ts":"1970-01-01 00:00:00.000999", "tz":"1970-01-01 08:00:00.000999+08:00"}
61+
{"ts":"1970-01-01 00:00:00.001000", "tz":"1970-01-01 08:00:00.001000+08:00"}
62+
{"ts":"1970-01-01 00:00:01.234567", "tz":"1970-01-01 08:00:01.234567+08:00"}
63+
{"ts":null, "tz":null}
64+
65+
-- !true_Asia_Shanghai_ntz --
66+
\N
67+
1969-12-31T23:59:58.999999
68+
1969-12-31T23:59:59.999999
69+
1970-01-01T00:00:00.000001
70+
1970-01-01T00:00:00.000999
71+
1970-01-01T00:00:00.001
72+
1970-01-01T00:00:01.234567
73+

0 commit comments

Comments
 (0)