Repository navigation
Commit 76d3b7e
authored
[fix](test) Fix external_table_p0 DATETIME baselines and Iceberg object counting (apache#66970)
### What problem does this PR solve?
Related PR: apache#66873
Problem Summary:
Two unrelated `external_table_p0` failures, both in test code only. No
BE/FE code is touched.
#### 1. `remote_doris` DATETIME baselines are stale after apache#66873
apache#66873 changed the Arrow Flight result schema so `DATETIMEV2` is exposed
as a **timezone-naive** Arrow timestamp. Both sides of the conversion
agree — `write_column_to_arrow` and `read_column_from_arrow` both decode
an empty Arrow timezone as UTC — so a cross-cluster `type=doris` read
now round-trips the whole DATETIME domain.
Under the previous timezone-aware encoding, `0000-01-01 00:00:00` was
written as an epoch below the minimum representable value once the
session timezone was applied, `from_unixtime` rejected it, and the value
came back as an invalid datetime (rendered empty).
`test_remote_doris_all_types_select` and `test_remote_doris_statistics`
are the only suites that store that boundary value in a `DATETIME`
column read over an arrow-flight Doris catalog, so they are the only
ones whose baselines still record the old broken output:
```
test_remote_doris_all_types_select.out
scalar c_datetime : → 0000-01-01T00:00
array a_datetime : [""] → ["0000-01-01 00:00:00"]
test_remote_doris_statistics.out
c_datetime min/max : 2023-10-01 12:34:56 / (empty)
→ 0000-01-01 00:00:00 / 9999-12-31 23:59:59
```
The new values are exactly the ones the test inserts, and the refreshed
baseline is now **identical cell for cell** to
`test_query_remote_doris_as_olap_table_select`, which reads the same
table definition and the same rows with `use_arrow_flight = false`.
Making the two transports agree is what apache#66873 set out to do, so that
equality is the check that these values are right.
This is a master-wide failure, not specific to any PR: across ~60 recent
`Doris_External_Regression` builds, every build whose base contains
910b741 fails these two suites, and every build with an older base
passes them. Both are currently muted in TeamCity, so the mutes should
be removed once this lands.
#### 2. `test_iceberg_write_merge_duplicate_source_negative` is flaky
(13 of 50 recent runs)
```
mc: <ERROR> Unable to initialize new alias from the provided credentials.
Get "http://172.17.0.6:19001/probe-.../?location=": dial tcp 172.17.0.6:19001: i/o timeout
```
The suite counted data objects by starting a throwaway `minio/mc`
container per call. MinIO runs on the isolated `doris--iceberg` compose
network (`168.38.0.0/24`) and only publishes its API on the host, while
the ad-hoc container joins the default bridge — so every call had to
hairpin from that bridge back to the published port. That path times out
while the agent starts and stops other containers, and with four calls
per run the suite failed about a quarter of the time. MinIO itself was
healthy in every failing run, and this was the only suite in the repo
using that pattern.
Count the objects with the S3 SDK in the suite's own JVM instead,
against the same endpoint the catalog is already configured with — the
same approach `test_iceberg_variant_read.groovy` already uses. Two
incidental bugs go away with it:
- the listing now paginates over continuation tokens, so a table with
more than 1000 objects can no longer be undercounted;
- the prefix is anchored at `.../data/` rather than `.../data`, so a
sibling directory sharing the prefix is not swept in.1 parent be88cea commit 76d3b7e
3 files changed
Lines changed: 54 additions & 25 deletions
File tree
- regression-test
- data/external_table_p0/remote_doris
- suites/external_table_p0/iceberg/write
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | | - | |
| 3 | + | |
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
9 | | - | |
| 9 | + | |
10 | 10 | | |
11 | 11 | | |
12 | 12 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
7 | | - | |
| 7 | + | |
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
15 | 15 | | |
16 | 16 | | |
17 | 17 | | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
18 | 24 | | |
19 | 25 | | |
20 | 26 | | |
| |||
27 | 33 | | |
28 | 34 | | |
29 | 35 | | |
30 | | - | |
31 | | - | |
32 | 36 | | |
33 | | - | |
34 | | - | |
35 | | - | |
36 | | - | |
37 | | - | |
38 | | - | |
39 | | - | |
40 | | - | |
41 | | - | |
42 | | - | |
43 | | - | |
44 | | - | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
45 | 66 | | |
46 | 67 | | |
47 | 68 | | |
| |||
81 | 102 | | |
82 | 103 | | |
83 | 104 | | |
84 | | - | |
85 | | - | |
86 | | - | |
87 | | - | |
88 | | - | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
89 | 117 | | |
90 | 118 | | |
91 | 119 | | |
| |||
126 | 154 | | |
127 | 155 | | |
128 | 156 | | |
129 | | - | |
| 157 | + | |
130 | 158 | | |
131 | 159 | | |
132 | 160 | | |
| |||
135 | 163 | | |
136 | 164 | | |
137 | 165 | | |
138 | | - | |
| 166 | + | |
139 | 167 | | |
140 | 168 | | |
141 | 169 | | |
| |||
162 | 190 | | |
163 | 191 | | |
164 | 192 | | |
165 | | - | |
| 193 | + | |
166 | 194 | | |
167 | 195 | | |
168 | 196 | | |
169 | 197 | | |
170 | 198 | | |
| 199 | + | |
171 | 200 | | |
0 commit comments