Skip to content

Commit 76d3b7e

Browse files
authored
[fix](test) Fix external_table_p0 DATETIME baselines and Iceberg object counting (apache#66970)
### What problem does this PR solve? Related PR: apache#66873 Problem Summary: Two unrelated `external_table_p0` failures, both in test code only. No BE/FE code is touched. #### 1. `remote_doris` DATETIME baselines are stale after apache#66873 apache#66873 changed the Arrow Flight result schema so `DATETIMEV2` is exposed as a **timezone-naive** Arrow timestamp. Both sides of the conversion agree — `write_column_to_arrow` and `read_column_from_arrow` both decode an empty Arrow timezone as UTC — so a cross-cluster `type=doris` read now round-trips the whole DATETIME domain. Under the previous timezone-aware encoding, `0000-01-01 00:00:00` was written as an epoch below the minimum representable value once the session timezone was applied, `from_unixtime` rejected it, and the value came back as an invalid datetime (rendered empty). `test_remote_doris_all_types_select` and `test_remote_doris_statistics` are the only suites that store that boundary value in a `DATETIME` column read over an arrow-flight Doris catalog, so they are the only ones whose baselines still record the old broken output: ``` test_remote_doris_all_types_select.out scalar c_datetime : → 0000-01-01T00:00 array a_datetime : [""] → ["0000-01-01 00:00:00"] test_remote_doris_statistics.out c_datetime min/max : 2023-10-01 12:34:56 / (empty) → 0000-01-01 00:00:00 / 9999-12-31 23:59:59 ``` The new values are exactly the ones the test inserts, and the refreshed baseline is now **identical cell for cell** to `test_query_remote_doris_as_olap_table_select`, which reads the same table definition and the same rows with `use_arrow_flight = false`. Making the two transports agree is what apache#66873 set out to do, so that equality is the check that these values are right. This is a master-wide failure, not specific to any PR: across ~60 recent `Doris_External_Regression` builds, every build whose base contains 910b741 fails these two suites, and every build with an older base passes them. Both are currently muted in TeamCity, so the mutes should be removed once this lands. #### 2. `test_iceberg_write_merge_duplicate_source_negative` is flaky (13 of 50 recent runs) ``` mc: <ERROR> Unable to initialize new alias from the provided credentials. Get "http://172.17.0.6:19001/probe-.../?location=": dial tcp 172.17.0.6:19001: i/o timeout ``` The suite counted data objects by starting a throwaway `minio/mc` container per call. MinIO runs on the isolated `doris--iceberg` compose network (`168.38.0.0/24`) and only publishes its API on the host, while the ad-hoc container joins the default bridge — so every call had to hairpin from that bridge back to the published port. That path times out while the agent starts and stops other containers, and with four calls per run the suite failed about a quarter of the time. MinIO itself was healthy in every failing run, and this was the only suite in the repo using that pattern. Count the objects with the S3 SDK in the suite's own JVM instead, against the same endpoint the catalog is already configured with — the same approach `test_iceberg_variant_read.groovy` already uses. Two incidental bugs go away with it: - the listing now paginates over continuation tokens, so a table with more than 1000 objects can no longer be undercounted; - the prefix is anchored at `.../data/` rather than `.../data`, so a sibling directory sharing the prefix is not swept in.
1 parent be88cea commit 76d3b7e

3 files changed

Lines changed: 54 additions & 25 deletions

File tree

‎regression-test/data/external_table_p0/remote_doris/test_remote_doris_all_types_select.out‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,12 @@
11
-- This file is automatically generated. You should know what you did if you want to edit this
22
-- !sql --
3-
2025-05-18T01:00 true -128 -32768 -2147483648 -9223372036854775808 -1234567890123456790 -123.456 -123456.789 -123457 -123456789012346 -1234567890123456789012345678 1970-01-01 A Hello Hello, Doris! ["apple", "banana", "orange"] {"Emily":101, "age":25} {"f1":11, "f2":3.14, "f3":"Emily"} {"k1":"v31","k2":300,"k3":[123,456],"k4":[],"k5":{"i1":"iv1"}}
3+
2025-05-18T01:00 true -128 -32768 -2147483648 -9223372036854775808 -1234567890123456790 -123.456 -123456.789 -123457 -123456789012346 -1234567890123456789012345678 1970-01-01 0000-01-01T00:00 A Hello Hello, Doris! ["apple", "banana", "orange"] {"Emily":101, "age":25} {"f1":11, "f2":3.14, "f3":"Emily"} {"k1":"v31","k2":300,"k3":[123,456],"k4":[],"k5":{"i1":"iv1"}}
44
2025-05-18T02:00 \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N
55
2025-05-18T03:00 false 127 32767 2147483647 9223372036854775807 1234567890123456789 123.456 123456.789 123457 123456789012346 1234567890123456789012345678 9999-12-31 9999-12-31T23:59:59 [] {} {"f1":11, "f2":3.14, "f3":"Emily"} {}
66
2025-05-18T04:00 true 0 0 0 0 0 0.0 0 0 0 0 2023-10-01 2023-10-01T12:34:56 A Hello Hello, Doris! ["apple", "banana", "orange"] {"Emily":101, "age":25} {"f1":11, "f2":3.14, "f3":"Emily"} []
77

88
-- !sql --
9-
2025-05-18T01:00 [1] [-128] [-32768] [-2147483648] [-9223372036854775808] [-1234567890123456790] [-123.456] [-123456.789] [-123457] [-123456789012346] [-1234567890123456789012345678] ["0000-01-01"] [""] ["A"] ["Hello"] ["Hello, Doris!"]
9+
2025-05-18T01:00 [1] [-128] [-32768] [-2147483648] [-9223372036854775808] [-1234567890123456790] [-123.456] [-123456.789] [-123457] [-123456789012346] [-1234567890123456789012345678] ["0000-01-01"] ["0000-01-01 00:00:00"] ["A"] ["Hello"] ["Hello, Doris!"]
1010
2025-05-18T02:00 [null] [null] [null] [null] [null] [null] [null] [null] [null] [null] [null] [null] [null] [null] [null] [null]
1111
2025-05-18T03:00 [0] [127] [32767] [2147483647] [9223372036854775807] [1234567890123456789] [123.456] [123456.789] [123457] [123456789012346] [1234567890123456789012345678] ["9999-12-31"] ["9999-12-31 23:59:59"] [""] [""] [""]
1212
2025-05-18T04:00 [1] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] ["2023-10-01"] ["2023-10-01 12:34:56"] ["A"] ["Hello"] ["Hello, Doris!"]

‎regression-test/data/external_table_p0/remote_doris/test_remote_doris_statistics.out‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ c_bigint 4 3 1 -9223372036854775808 9223372036854775807 32
44
c_boolean 4 2 1 0 1 4
55
c_char 4 2 1 A 2
66
c_date 4 3 1 1970-01-01 9999-12-31 16
7-
c_datetime 4 3 1 2023-10-01 12:34:56 32
7+
c_datetime 4 3 1 0000-01-01 00:00:00 9999-12-31 23:59:59 32
88
c_decimal18 4 3 1 -123456789012346 123456789012346 32
99
c_decimal32 4 3 1 -1234567890123456789012345678 1234567890123456789012345678 64
1010
c_decimal9 4 3 1 -123457 123457 16

‎regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_merge_duplicate_source_negative.groovy‎

Lines changed: 51 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,12 @@
1515
// specific language governing permissions and limitations
1616
// under the License.
1717

18+
import com.amazonaws.auth.AWSStaticCredentialsProvider
19+
import com.amazonaws.auth.BasicAWSCredentials
20+
import com.amazonaws.client.builder.AwsClientBuilder.EndpointConfiguration
21+
import com.amazonaws.services.s3.AmazonS3ClientBuilder
22+
import com.amazonaws.services.s3.model.ListObjectsV2Request
23+
1824
suite("test_iceberg_write_merge_duplicate_source_negative",
1925
"p0,external,iceberg,external_docker,external_docker_iceberg,nonConcurrent") {
2026
String enabled = context.config.otherConfigs.get("enableIcebergTest")
@@ -27,21 +33,36 @@ suite("test_iceberg_write_merge_duplicate_source_negative",
2733
String externalEnvIp = context.config.otherConfigs.get("externalEnvIp")
2834
String catalogName = "test_iceberg_write_merge_duplicate_source_negative"
2935
String dbName = "iceberg_write_merge_duplicate_source_negative_db"
30-
String dockerCommand = context.config.otherConfigs.get("externalDockerCommand") ?: "docker"
31-
String mcImage = "minio/mc:RELEASE.2025-01-17T23-25-50Z"
3236

33-
def countDataObjects = { String objectPath ->
34-
def stdout = new StringBuilder()
35-
def stderr = new StringBuilder()
36-
def process = new ProcessBuilder("/bin/bash", "-c",
37-
"${dockerCommand} run --rm --entrypoint /bin/sh ${mcImage} -c "
38-
+ "'/usr/bin/mc alias set minio http://${externalEnvIp}:${minioPort} "
39-
+ "admin password >/dev/null "
40-
+ "&& /usr/bin/mc find ${objectPath} --type f | wc -l'").start()
41-
process.consumeProcessOutput(stdout, stderr)
42-
process.waitForOrKill(30000)
43-
assertEquals(0, process.exitValue(), "Failed to list Iceberg data objects: ${stderr}")
44-
return stdout.toString().trim() as long
37+
// MinIO only publishes its API on the host, while its container sits on the isolated
38+
// `doris--iceberg` compose network. Listing objects from this JVM reuses the very endpoint the
39+
// catalog is configured with. Shelling out to a throwaway `minio/mc` container instead puts the
40+
// client on the default bridge, so every call has to hairpin back to the published port, which
41+
// intermittently times out while the agent starts and stops other containers.
42+
def minioClient = AmazonS3ClientBuilder.standard()
43+
.withEndpointConfiguration(
44+
new EndpointConfiguration("http://${externalEnvIp}:${minioPort}", "us-east-1"))
45+
.withPathStyleAccessEnabled(true)
46+
.withCredentials(new AWSStaticCredentialsProvider(
47+
new BasicAWSCredentials("admin", "password")))
48+
.build()
49+
50+
def countDataObjects = { String bucket, String prefix ->
51+
long objectCount = 0
52+
String continuationToken = null
53+
while (true) {
54+
def listing = minioClient.listObjectsV2(new ListObjectsV2Request()
55+
.withBucketName(bucket)
56+
.withPrefix(prefix)
57+
.withContinuationToken(continuationToken))
58+
// Directory placeholder keys are not data objects, mirroring `mc find --type f`.
59+
objectCount += listing.getObjectSummaries().count { !it.getKey().endsWith("/") }
60+
if (!listing.isTruncated()) {
61+
break
62+
}
63+
continuationToken = listing.getNextContinuationToken()
64+
}
65+
return objectCount
4566
}
4667

4768
sql """drop catalog if exists ${catalogName}"""
@@ -81,11 +102,18 @@ suite("test_iceberg_write_merge_duplicate_source_negative",
81102
String committedFile = (sql """
82103
select file_path from duplicate_source_target\$files order by file_path limit 1
83104
""")[0][0].toString()
84-
int dataDirectoryEnd = committedFile.indexOf('/data/') + '/data'.length()
85-
assertTrue(dataDirectoryEnd >= '/data'.length(), "Unexpected Iceberg data path: ${committedFile}")
86-
String dataObjectPath = committedFile.substring(0, dataDirectoryEnd)
87-
.replaceFirst('^s3a?://warehouse', 'minio/warehouse')
88-
long objectsBefore = countDataObjects(dataObjectPath)
105+
// file_path is `s3://<bucket>/<key>`; split it into the bucket and the table's data prefix.
106+
int schemeEnd = committedFile.indexOf("://")
107+
assertTrue(schemeEnd > 0, "Unexpected Iceberg data path: ${committedFile}")
108+
String bucketAndKey = committedFile.substring(schemeEnd + "://".length())
109+
int bucketEnd = bucketAndKey.indexOf('/')
110+
assertTrue(bucketEnd > 0, "Unexpected Iceberg data path: ${committedFile}")
111+
String dataBucket = bucketAndKey.substring(0, bucketEnd)
112+
String objectKey = bucketAndKey.substring(bucketEnd + 1)
113+
int dataDirectoryEnd = objectKey.indexOf('/data/')
114+
assertTrue(dataDirectoryEnd > 0, "Unexpected Iceberg data path: ${committedFile}")
115+
String dataObjectPrefix = objectKey.substring(0, dataDirectoryEnd + '/data/'.length())
116+
long objectsBefore = countDataObjects(dataBucket, dataObjectPrefix)
89117

90118
long snapshotsBefore =
91119
(sql """select count(*) from duplicate_source_target\$snapshots""")[0][0] as long
@@ -126,7 +154,7 @@ suite("test_iceberg_write_merge_duplicate_source_negative",
126154
(sql """select count(*) from duplicate_source_target\$snapshots""")[0][0] as long)
127155
assertEquals(filesBefore,
128156
(sql """select count(*) from duplicate_source_target\$files""")[0][0] as long)
129-
assertEquals(objectsBefore, countDataObjects(dataObjectPath))
157+
assertEquals(objectsBefore, countDataObjects(dataBucket, dataObjectPrefix))
130158
order_qt_duplicate_source_atomic_state """
131159
select id, region, payload
132160
from duplicate_source_target
@@ -135,7 +163,7 @@ suite("test_iceberg_write_merge_duplicate_source_negative",
135163

136164
// A sibling delete close can fail only after the data side has closed successfully. The outer
137165
// MERGE still owns and must remove those unpublished data objects.
138-
long siblingFailureObjectsBefore = countDataObjects(dataObjectPath)
166+
long siblingFailureObjectsBefore = countDataObjects(dataBucket, dataObjectPrefix)
139167
try {
140168
GetDebugPoint().enableDebugPointForAllBEs("VIcebergDeleteSink.close.inject_failure")
141169
test {
@@ -162,10 +190,11 @@ suite("test_iceberg_write_merge_duplicate_source_negative",
162190
(sql """select count(*) from duplicate_source_target\$snapshots""")[0][0] as long)
163191
assertEquals(filesBefore,
164192
(sql """select count(*) from duplicate_source_target\$files""")[0][0] as long)
165-
assertEquals(siblingFailureObjectsBefore, countDataObjects(dataObjectPath))
193+
assertEquals(siblingFailureObjectsBefore, countDataObjects(dataBucket, dataObjectPrefix))
166194
order_qt_sibling_close_atomic_state """
167195
select id, region, payload
168196
from duplicate_source_target
169197
order by id
170198
"""
199+
minioClient.shutdown()
171200
}

0 commit comments

Comments
 (0)