Skip to content

Commit 954fde3

Browse files
morningmanclaude
andcommitted
branch-4.0: [fix](docker)(case) Restore Kerberos/Paimon external env and fix flaky file cache cases
The daily external regression pipeline for branch-4.0 fails 10 cases. Eight of them turned red at the same build and stayed red; the other two are long-standing. 1. Kerberos: the lightweight Kerberos environment creates every principal with `addprinc -randkey`, so `/keytabs/*.keytab` holds a different key on every container start. Deployments that run Doris on hosts other than the Docker host provision `/keytabs` out of band and cannot follow that rotation, so every Kerberos catalog fails with `GeneralSecurityException: Checksum failed`. The old image-based environment shipped fixed keys, which is why this only appeared after the switch. Derive the keys from a fixed password and export them with `ktadd -norandkey` instead: kdc.conf already pins a single enctype and the salt is fixed, so the keytab is now reproducible across rebuilds. 2. Paimon HMS: `create_paimon_tables.hql` only runs from `init-hive-baseline.sh`, which is skipped whenever the Hive baseline tarball is restored (the default `refresh` mode). The Paimon HMS databases are therefore missing on every baseline-restore run and `test_paimon_hms_catalog` fails with `Unknown database 'hdfs_db'`. Move the bootstrap into a `paimon_hms` module so it is reached through `refresh-hive-modules.sh` as well; `init-hive-baseline.sh` now goes through the same module so the HQL is not executed twice on a full init. 3. `test_es_query` moved from `external_table_p0` to `external_table_p2` but still hardcodes `regression_test_external_table_p0_es`, while the framework derives the database from the suite directory. Use `context.dbName`. 4. `test_file_cache_query_limit` curls `externalEnvIp` for the backend brpc and HTTP endpoints. That is the third-party Docker host, which runs no backend at all in a multi-host deployment, so `file_cache_capacity` is never found. Resolve the backend from `show backends`, and skip when more than one backend is alive, which the case already documents as its precondition. 5. `test_file_cache_statistics` asserts `normal_queue_curr_size < normal_queue_max_size`. A queue's `max_size` is a soft limit: `try_reserve_from_other_queue` lets a queue grow past its own share while the whole cache still fits, evicting from the under-used queues instead. Whether the assertion holds depends on how full the other queues happen to be, which is why it is flaky. Assert the bound the backend actually enforces, the total cache capacity. Items 1 and 2 apply to master and branch-4.1 as well and should be forward ported. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
1 parent 7275af3 commit 954fde3

8 files changed

Lines changed: 92 additions & 20 deletions

File tree

‎docker/thirdparties/docker-compose/hive/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,7 @@ Modules are refreshed incrementally: only modules whose content SHA changed are
6060
| `test` | `scripts/data/test/` | Lightweight smoke-test datasets |
6161
| `preinstalled_hql` | `scripts/create_preinstalled_scripts/*.hql` | ~77 HQL files, executed in parallel via `xargs -P` |
6262
| `view` | `scripts/create_view_scripts/create_view.hql` | View definitions |
63+
| `paimon_hms` | `scripts/create_external_paimon_scripts/create_paimon_tables.hql` | Paimon tables registered in HMS (`hdfs_db`, `ali_db`, …); only selected when `enablePaimonHms=true` |
6364

6465
### Layer 3 — Version-Specific File Selection
6566

‎docker/thirdparties/docker-compose/hive/README_ZH.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,7 @@ Hive 启动被拆分为三层互相独立的抽象:
6060
| `test` | `scripts/data/test/` | 轻量级冒烟测试数据 |
6161
| `preinstalled_hql` | `scripts/create_preinstalled_scripts/*.hql` | 约 77 个 HQL 文件,通过 `xargs -P` 并行执行 |
6262
| `view` | `scripts/create_view_scripts/create_view.hql` | View 定义 |
63+
| `paimon_hms` | `scripts/create_external_paimon_scripts/create_paimon_tables.hql` | 注册到 HMS 的 Paimon 表(`hdfs_db`、`ali_db` 等),仅在 `enablePaimonHms=true` 时纳入 |
6364

6465
### Layer 3 — 按版本自动选文件
6566

‎docker/thirdparties/docker-compose/hive/scripts/hive-module-lib.sh‎

Lines changed: 23 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -22,27 +22,39 @@ set -eo pipefail
2222
. /mnt/scripts/hive-common-lib.sh
2323

2424
BOOTSTRAP_GROUPS="$(bootstrap_normalize_groups "${HIVE_BOOTSTRAP_GROUPS:-}")"
25-
DEFAULT_MODULES=(default multi_catalog partition_type statistics tvf regression test preinstalled_hql view)
25+
DEFAULT_MODULES=(default multi_catalog partition_type statistics tvf regression test preinstalled_hql view paimon_hms)
2626
LAST_REFRESH_DETAIL=""
2727
HIVE_HQL_PARALLEL="${HIVE_HQL_PARALLEL:-${LOAD_PARALLEL}}"
2828

2929
ensure_hive_state_layout
3030

31+
# The Paimon HMS tables need the Paimon storage handler and the object storage
32+
# credentials that only the Hive3 stack is configured with, so the module opts
33+
# in via the same flag the settings env files carry.
34+
paimon_hms_enabled() {
35+
[[ "${enablePaimonHms:-false}" == "true" ]]
36+
}
37+
3138
normalize_hive_modules() {
3239
local raw_modules="${1:-}"
3340
local cleaned_modules="${raw_modules// /}"
3441
local module=""
3542
local normalized=()
3643

3744
if [[ -z "${cleaned_modules}" || "${cleaned_modules}" == "all" ]]; then
38-
printf '%s\n' "${DEFAULT_MODULES[@]}"
45+
for module in "${DEFAULT_MODULES[@]}"; do
46+
if [[ "${module}" == "paimon_hms" ]] && ! paimon_hms_enabled; then
47+
continue
48+
fi
49+
echo "${module}"
50+
done
3951
return 0
4052
fi
4153

4254
IFS=',' read -r -a normalized <<<"${cleaned_modules}"
4355
for module in "${normalized[@]}"; do
4456
case "${module}" in
45-
default|multi_catalog|partition_type|statistics|tvf|regression|test|preinstalled_hql|view)
57+
default|multi_catalog|partition_type|statistics|tvf|regression|test|preinstalled_hql|view|paimon_hms)
4658
echo "${module}"
4759
;;
4860
*)
@@ -126,6 +138,9 @@ calc_module_sha() {
126138
view)
127139
files+=("/mnt/scripts/create_view_scripts/create_view.hql")
128140
;;
141+
paimon_hms)
142+
files+=("/mnt/scripts/create_external_paimon_scripts/create_paimon_tables.hql")
143+
;;
129144
*)
130145
echo "Unknown module for sha: ${module}" >&2
131146
return 1
@@ -314,6 +329,11 @@ refresh_module() {
314329
LAST_REFRESH_DETAIL="create_view.hql"
315330
run_hive_hql /mnt/scripts/create_view_scripts/create_view.hql "create_view.hql"
316331
;;
332+
paimon_hms)
333+
LAST_REFRESH_DETAIL="create_paimon_tables.hql"
334+
run_hive_hql /mnt/scripts/create_external_paimon_scripts/create_paimon_tables.hql \
335+
"create_paimon_tables.hql"
336+
;;
317337
*)
318338
echo "Unknown module for refresh: ${module}" >&2
319339
return 1

‎docker/thirdparties/docker-compose/hive/scripts/init-hive-baseline.sh‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,10 @@ copy_to_hdfs_if_selected "paimon1"
3131
copy_to_hdfs_if_selected "tvf_data"
3232
copy_to_hdfs_if_selected "preinstalled_data"
3333

34-
if [[ ${enablePaimonHms:-false} == "true" ]]; then
35-
run_hive_hql /mnt/scripts/create_external_paimon_scripts/create_paimon_tables.hql "create_paimon_table.hql"
34+
# Go through the module framework rather than calling the HQL directly: this
35+
# script only runs on the full-init path, while a baseline restore reaches the
36+
# Paimon tables through refresh-hive-modules.sh. Recording the module state
37+
# here also keeps the follow-up refresh pass from running the same HQL twice.
38+
if paimon_hms_enabled; then
39+
refresh_module paimon_hms
3640
fi

‎docker/thirdparties/docker-compose/kerberos/entrypoint-hive-master.sh‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@ readonly HTTP_PRINCIPAL="HTTP/${HOST}@${REALM}"
3737
readonly HIVE_PRINCIPAL="hive/${HOST}@${REALM}"
3838
readonly HIVE_CLIENT_PRINCIPAL="hive/presto-master.docker.cluster@${REALM}"
3939
readonly PRESTO_CLIENT_PRINCIPAL="presto-server/presto-master.docker.cluster@${REALM}"
40+
readonly PRINCIPAL_PASSWORD="doris-kerberos-test"
4041

4142
declare -a SERVICE_PIDS=()
4243

@@ -70,12 +71,18 @@ wait_for_port() {
7071
return 1
7172
}
7273

74+
# Keys must stay identical across container rebuilds. Deployments that run
75+
# Doris on separate hosts from this container provision /keytabs out of band,
76+
# so a key that is re-randomized on every start makes every such client fail
77+
# the AS-REP decryption with "GeneralSecurityException: Checksum failed".
78+
# A fixed password plus the single fixed enctype in kdc.conf yields a stable
79+
# key, and -norandkey exports that key instead of rolling a new one.
7380
create_keytab() {
7481
local principal=$1
7582
local keytab=$2
7683

77-
kadmin.local -r "${REALM}" -q "addprinc -randkey ${principal}"
78-
kadmin.local -r "${REALM}" -q "ktadd -k ${keytab} ${principal}"
84+
kadmin.local -r "${REALM}" -q "addprinc -pw ${PRINCIPAL_PASSWORD} ${principal}"
85+
kadmin.local -r "${REALM}" -q "ktadd -k ${keytab} -norandkey ${principal}"
7986
}
8087

8188
report_stage() {

‎regression-test/suites/external_table_p0/cache/test_file_cache_query_limit.groovy‎

Lines changed: 18 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -52,9 +52,24 @@ suite("test_file_cache_query_limit", "external_docker,hive,external_docker_hive,
5252

5353
sql """set enable_file_cache=true"""
5454

55-
// Check backend configuration prerequisites
5655
// Note: This test case assumes a single backend scenario. Testing with single backend is logically equivalent
5756
// to testing with multiple backends having identical configurations, but simpler in logic.
57+
// The assumption is load-bearing rather than cosmetic: the HTTP calls below clear and inspect ONE backend's
58+
// file cache while the queries are served by the whole cluster, so with several backends the inspected cache
59+
// never reflects what the query actually cached. Skip instead of reporting a false failure.
60+
def aliveBackends = sql_return_maparray("show backends").findAll {
61+
it.Alive.toString().equalsIgnoreCase("true")
62+
}
63+
if (aliveBackends.size() != 1) {
64+
logger.info("skip test_file_cache_query_limit: it assumes a single backend, found ${aliveBackends.size()}")
65+
return
66+
}
67+
// The backend HTTP/brpc endpoints must be addressed by the backend's own host. externalEnvIp is the
68+
// third-party docker host (hive/es/...), which in a multi-host deployment runs no backend at all, so
69+
// curling it silently yields no file cache metrics.
70+
String beHost = aliveBackends[0].Host
71+
72+
// Check backend configuration prerequisites
5873
def enableFileCacheResult = sql """show backend config like 'enable_file_cache';"""
5974
logger.info("enable_file_cache configuration: " + enableFileCacheResult)
6075
assertFalse(enableFileCacheResult.size() == 0 || !enableFileCacheResult[0][3].equalsIgnoreCase("true"),
@@ -139,7 +154,7 @@ suite("test_file_cache_query_limit", "external_docker,hive,external_docker_hive,
139154
String brpc_port = brpcPortResult[0][3]
140155

141156
// Search file cache capacity
142-
def command = ["curl", "-X", "POST", "${externalEnvIp}:${brpc_port}/vars"]
157+
def command = ["curl", "-X", "POST", "${beHost}:${brpc_port}/vars"]
143158
def stringCommand = command.collect{it.toString()}
144159
def process = new ProcessBuilder(stringCommand as String[]).redirectErrorStream(true).start()
145160

@@ -160,7 +175,7 @@ suite("test_file_cache_query_limit", "external_docker,hive,external_docker_hive,
160175
logger.info("========================= Start running file cache base test ========================")
161176

162177
// Clear file cache
163-
command = ["curl", "-X", "POST", "${externalEnvIp}:${webserver_port}/api/file_cache?op=clear&sync=true"]
178+
command = ["curl", "-X", "POST", "${beHost}:${webserver_port}/api/file_cache?op=clear&sync=true"]
164179
stringCommand = command.collect{it.toString()}
165180
process = new ProcessBuilder(stringCommand as String[]).redirectErrorStream(true).start()
166181

‎regression-test/suites/external_table_p0/cache/test_file_cache_statistics.groovy‎

Lines changed: 29 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -31,8 +31,8 @@ final String HIT_RATIO_5M_METRIC_FALSE_MSG = HIT_RATIO_CHECK_FAILED_PREFIX + "hi
3131

3232
// Constants for normal queue check
3333
final String NORMAL_QUEUE_CHECK_FAILED_PREFIX = "Normal queue check failed: "
34-
final String NORMAL_QUEUE_SIZE_VALIDATION_FAILED_MSG = NORMAL_QUEUE_CHECK_FAILED_PREFIX + "size validation failed (curr_size should be > 0 and < max_size)"
35-
final String NORMAL_QUEUE_ELEMENTS_VALIDATION_FAILED_MSG = NORMAL_QUEUE_CHECK_FAILED_PREFIX + "elements validation failed (curr_elements should be > 0 and < max_elements)"
34+
final String NORMAL_QUEUE_SIZE_VALIDATION_FAILED_MSG = NORMAL_QUEUE_CHECK_FAILED_PREFIX + "size validation failed (curr_size should be > 0 and <= total cache capacity)"
35+
final String NORMAL_QUEUE_ELEMENTS_VALIDATION_FAILED_MSG = NORMAL_QUEUE_CHECK_FAILED_PREFIX + "elements validation failed (curr_elements should be > 0)"
3636

3737
// Constants for hit and read counts check
3838
final String HIT_AND_READ_COUNTS_CHECK_FAILED_PREFIX = "Hit and read counts check failed: "
@@ -174,8 +174,20 @@ suite("test_file_cache_statistics", "external_docker,hive,external_docker_hive,p
174174
// ===== Normal Queue Metrics Check =====
175175
// curr_size / curr_elements are monitor-published; poll until populated (> 0) across paths.
176176
// max_size / max_elements come from the queue's static capacity (not monitor-published), so
177-
// they are read once without polling. SUM across paths preserves the curr < max inequality
178-
// (sum of per-path curr < sum of per-path max, since each curr < max).
177+
// they are read once without polling.
178+
//
179+
// A queue's own max_size is a SOFT limit, not a bound to assert against: when a queue is over
180+
// its share, BlockFileCache::try_reserve_from_other_queue lets it keep growing as long as the
181+
// WHOLE cache still fits (`_cur_cache_size + size > _capacity && cur_queue_size + size >
182+
// cur_queue_max_size` is the only rejection), evicting from the under-used queues instead --
183+
// see the "Hit the soft limit by self" branch and is_overflow(), which compares against
184+
// _capacity alone. So normal_queue_curr_size legitimately exceeds normal_queue_max_size
185+
// whenever the ttl/index/disposable queues are not full, which depends on whichever cases ran
186+
// before on this shared cache. Asserting curr < max encoded that non-invariant and was flaky.
187+
//
188+
// The hard bound the BE actually enforces is the per-cache _capacity, and by construction in
189+
// get_file_cache_settings() capacity == normal + index + ttl + disposable max sizes (the
190+
// normal/query queue is defined as the remainder). Sum across paths and assert against that.
179191
pollMetric('normal_queue_curr_size', { it > 0 }, metricPollTimeoutSeconds)
180192
pollMetric('normal_queue_curr_elements', { it > 0 }, metricPollTimeoutSeconds)
181193

@@ -188,16 +200,25 @@ suite("test_file_cache_statistics", "external_docker,hive,external_docker_hive,p
188200
def normalQueueMaxElementsSum = cacheMetricSum('normal_queue_max_elements')
189201
logger.info("normal_queue_max_elements sum: " + normalQueueMaxElementsSum)
190202

203+
def indexQueueMaxSizeSum = cacheMetricSum('index_queue_max_size')
204+
def ttlQueueMaxSizeSum = cacheMetricSum('ttl_queue_max_size')
205+
def disposableQueueMaxSizeSum = cacheMetricSum('disposable_queue_max_size')
206+
Double cacheCapacitySum = (normalQueueMaxSizeSum == null || indexQueueMaxSizeSum == null
207+
|| ttlQueueMaxSizeSum == null || disposableQueueMaxSizeSum == null) ? null
208+
: normalQueueMaxSizeSum + indexQueueMaxSizeSum + ttlQueueMaxSizeSum + disposableQueueMaxSizeSum
209+
logger.info("total file cache capacity sum (normal+index+ttl+disposable max_size): " + cacheCapacitySum)
210+
191211
boolean hasNormalQueueCurrSize = normalQueueCurrSizeSum != null && normalQueueCurrSizeSum > 0
192212
boolean hasNormalQueueMaxSize = normalQueueMaxSizeSum != null && normalQueueMaxSizeSum > 0
193213
boolean hasNormalQueueCurrElements = normalQueueCurrElementsSum != null && normalQueueCurrElementsSum > 0
194214
boolean hasNormalQueueMaxElements = normalQueueMaxElementsSum != null && normalQueueMaxElementsSum > 0
195215

196-
// Check if current size is less than max size and current elements is less than max elements
216+
// The queue must be in use and must stay within the cache's hard capacity. max_elements is only
217+
// logged: element counts are not bounded by the sum of the per-queue element caps either, since
218+
// a block may be smaller than max_file_block_size.
197219
boolean normalQueueSizeValid = hasNormalQueueCurrSize && hasNormalQueueMaxSize &&
198-
normalQueueCurrSizeSum < normalQueueMaxSizeSum
199-
boolean normalQueueElementsValid = hasNormalQueueCurrElements && hasNormalQueueMaxElements &&
200-
normalQueueCurrElementsSum < normalQueueMaxElementsSum
220+
cacheCapacitySum != null && normalQueueCurrSizeSum <= cacheCapacitySum
221+
boolean normalQueueElementsValid = hasNormalQueueCurrElements && hasNormalQueueMaxElements
201222

202223
logger.info("Normal queue metrics check result - size valid: ${normalQueueSizeValid}, " +
203224
"elements valid: ${normalQueueElementsValid}")

‎regression-test/suites/external_table_p2/es/test_es_query.groovy‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -193,9 +193,12 @@ suite("test_es_query", "p2,external") {
193193
}
194194
}
195195

196-
def query_catalogs = { ->
196+
def query_catalogs = { ->
197197
sql """switch internal"""
198-
sql """use regression_test_external_table_p0_es"""
198+
// test_v1/test_v2 live in this suite's own database, which the
199+
// framework derives from the suite directory. Hardcoding the name
200+
// breaks whenever the suite moves between p0 and p2.
201+
sql """use ${context.dbName}"""
199202
executeWithRetry("""select * from test_v1 where test2='text#1'""", "sql01", 30)
200203
order_qt_sql01 """select * from test_v1 where test2='text#1'"""
201204
order_qt_sql02 """select * from test_v1 where esquery(test2, '{"match":{"test2":"text#1"}}')"""

0 commit comments

Comments
 (0)