Skip to content

Commit 82ad886

Browse files
morningmanclaude
andcommitted
[fix](test) drive data to converge adaptive timeout in test_routine_load_adaptive_param restore phase
The restore-phase checkTaskTimeout("100") is flaky and timed out after 60s (build 970376). Root cause is observability, not a product bug: a routine load task's adaptive timeout is only (re)computed when the task is actually scheduled WITH data to consume (updateAdaptiveTimeout runs after the hasMoreDataToConsume gate in RoutineLoadTaskScheduler). Once a job drains its data it reaches EOF and the renewed task stays idle with txnId == -1, inheriting the timeout from the previous schedule round. So when the single restore batch is consumed atomically by one task that still carries the isEof=false inertia from the debug-on phase (timeout 3600), the EOF timeout 100 is never computed on any task, and checkTaskTimeout's "txnId != -1 && timeout == 100" condition can never be met. Add checkTaskTimeoutWithData, which feeds a small batch each poll round to force an isEof task to be scheduled and recompute the timeout, and reads the value without skipping txnId == -1 (after EOF the converged value settles on the idle task). Use it for the restore phase; leave checkTaskTimeout (increase phase, where the converged value lives on the running task) untouched. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
1 parent da471d4 commit 82ad886

2 files changed

Lines changed: 37 additions & 1 deletion

File tree

‎regression-test/framework/src/main/groovy/org/apache/doris/regression/util/RoutineLoadTestUtils.groovy‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -114,6 +114,40 @@ class RoutineLoadTestUtils {
114114
}
115115
}
116116

117+
// Verify that the adaptive task timeout converges to expectedTimeout when the job is caught up
118+
// (EOF). The adaptive timeout is only (re)computed when a task is actually scheduled WITH data
119+
// to consume; once a job drains its data the renewed task stays idle (txnId == -1) and keeps the
120+
// timeout from the previous schedule round, so the EOF timeout is never observed on its own.
121+
// Drive a fresh small batch each round to force an isEof task to be scheduled and recompute the
122+
// timeout, then read whatever task is visible (the running task, or the renewed idle one that
123+
// inherits the just-converged value). Unlike checkTaskTimeout we do NOT skip txnId == -1 here,
124+
// because after EOF the converged value naturally settles on an idle task.
125+
static void checkTaskTimeoutWithData(Closure sqlRunner, KafkaProducer producer, List<String> topics,
126+
String jobName, String expectedTimeout, int maxAttempts = 60) {
127+
def count = 0
128+
while (true) {
129+
sendTestDataToKafka(producer, topics)
130+
def res = sqlRunner.call("SHOW ROUTINE LOAD TASK WHERE JobName = '${jobName}'")
131+
if (res.size() > 0) {
132+
def txnId = res[0][1].toString()
133+
def timeout = res[0][6].toString()
134+
logger.info("res: ${res[0].toString()}")
135+
logger.info("txnId: ${txnId}, timeout: ${timeout}, expected: ${expectedTimeout}")
136+
if (timeout == expectedTimeout) {
137+
Assert.assertEquals(expectedTimeout, timeout)
138+
break;
139+
}
140+
}
141+
if (count > maxAttempts) {
142+
Assert.fail("Timeout waiting for task timeout to converge to ${expectedTimeout} for job ${jobName}")
143+
break;
144+
} else {
145+
sleep(1000)
146+
count++
147+
}
148+
}
149+
}
150+
117151
static int waitForTaskFinish(Closure sqlRunner, String job, String tableName, int expectedMinRows = 0, int maxAttempts = 60) {
118152
return waitForTaskFinishInternal(sqlRunner, job, tableName, expectedMinRows, maxAttempts, false)
119153
}

‎regression-test/suites/load_p0/routine_load/test_routine_load_adaptive_param.groovy‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -84,7 +84,9 @@ suite("test_routine_load_adaptive_param","nonConcurrent") {
8484
logger.info("---test restore adaptively---")
8585
RoutineLoadTestUtils.sendTestDataToKafka(producer, kafkaCsvTpoics)
8686
RoutineLoadTestUtils.waitForTaskFinish(runSql, job, tableName, 4)
87-
RoutineLoadTestUtils.checkTaskTimeout(runSql, job, "100")
87+
// After EOF the adaptive timeout only converges when an isEof task is scheduled with
88+
// data, so keep feeding small batches until the task timeout restores to the job timeout.
89+
RoutineLoadTestUtils.checkTaskTimeoutWithData(runSql, producer, kafkaCsvTpoics, job, "100")
8890
} finally {
8991
sql "stop routine load for ${job}"
9092
}

0 commit comments

Comments
 (0)