[Fix-17436][Workflow]Task timeout kill throw exception #17437

njnu-seafish · 2025-08-21T02:29:14Z

Purpose of the pull request

Brief change log

add string blank check
remove redundant cancelApplication call

Verify this pull request

After fixing the bug, timeout kill logs:

2025-08-21 09:30:25.259 INFO [WorkerRpcServer-methodInvoker-155] - Publish TaskExecutorKillLifecycleEvent: {
"taskInstanceId" : 1341,
"eventCreateTime" : 1755739825259,
"type" : "KILL"
}
2025-08-21 09:30:25.302 INFO [PhysicalTaskExecutorEventBusCoordinator-eventbus-coordinator-worker-22] - Begin killing task instance, processId: 1115493
2025-08-21 09:30:25.826 INFO [PhysicalTaskExecutorEventBusCoordinator-eventbus-coordinator-worker-22] - prepare to parse pid, raw pid string: sudo(1115493)---1341.sh(1115497)---sleep(1115570)

2025-08-21 09:30:26.370 INFO [PhysicalTaskExecutorEventBusCoordinator-eventbus-coordinator-worker-22] - Sending SIGINT to process group: 1115493 1115497 1115570, command: sudo -u dolphinscheduler -i kill -s SIGINT 1115493 1115497 1115570
2025-08-21 09:30:37.408 INFO [PhysicalTaskExecutorEventBusCoordinator-eventbus-coordinator-worker-22] - Kill command: sudo -u dolphinscheduler -i kill -s SIGINT 1115493 1115497 1115570, timed out, still running PIDs: 1115493 1115497 1115570
2025-08-21 09:30:37.961 INFO [PhysicalTaskExecutorEventBusCoordinator-eventbus-coordinator-worker-22] - Sending SIGTERM to process group: 1115493 1115497 1115570, command: sudo -u dolphinscheduler -i kill -s SIGTERM 1115493 1115497 1115570
2025-08-21 09:30:38.156 ERROR [exclusive-task-executor-container-worker-1] - process has failure due to timeout kill, the timeout value is:60, the taskTimeoutStrategy is:FAILED
2025-08-21 09:30:38.157 INFO [exclusive-task-executor-container-worker-1] - process has killed. execute path:/data01/dolphinscheduler/exec/process/1341, processId:1115493 ,exitStatusCode:-1 ,processWaitForStatus:false ,processExitValue:143
2025-08-21 09:30:38.157 INFO [exclusive-task-executor-container-worker-1] - Publish TaskExecutorFailedLifecycleEvent: {
"taskInstanceId" : 1341,
"eventCreateTime" : 1755739838157,
"type" : "FAILED",
"workflowInstanceId" : 1273,
"workflowInstanceHost" : "192.168.30.11:5678",
"taskInstanceHost" : "192.168.30.121:1234",
"appIds" : null,
"endTime" : 1755739838157,
"latestReportTime" : null
}
2025-08-21 09:30:38.157 INFO [exclusive-task-executor-container-worker-1] - Publish TaskExecutorFinalizeLifecycleEvent: {
"taskInstanceId" : 1341,
"eventCreateTime" : 1755739838157,
"type" : "FINALIZE"
}
2025-08-21 09:30:38.676 INFO [PhysicalTaskExecutorEventBusCoordinator-eventbus-coordinator-worker-22] - Successfully killed process tree using SIGTERM, processId: 1115493
2025-08-21 09:30:38.676 INFO [PhysicalTaskExecutorEventBusCoordinator-eventbus-coordinator-worker-22] - Process tree for task: 1341 is killed or already finished, pid: 1115493
2025-08-21 09:30:38.677 INFO [PhysicalTaskExecutorEventBusCoordinator-eventbus-coordinator-worker-22] - Get appIds from worker 192.168.30.121:1234, taskLogPath: /data01/dolphinscheduler/20250821/149143631011392/7/1273/1341.log
2025-08-21 09:30:38.677 INFO [PhysicalTaskExecutorEventBusCoordinator-eventbus-coordinator-worker-22] - Start finding appId in /data01/dolphinscheduler/20250821/149143631011392/7/1273/1341.log, fetch way: log
2025-08-21 09:30:38.677 INFO [PhysicalTaskExecutorEventBusCoordinator-eventbus-coordinator-worker-22] - The appId is empty
2025-08-21 09:30:38.677 INFO [PhysicalTaskExecutorEventBusCoordinator-eventbus-coordinator-worker-22] - Success fire TaskExecutorKillLifecycleEvent: {
"taskInstanceId" : 1341,
"eventCreateTime" : 1755739825259,
"type" : "KILL"
}
2025-08-21 09:30:38.718 INFO [PhysicalTaskExecutorEventBusCoordinator-eventbus-coordinator-worker-20] - Success fire TaskExecutorFailedLifecycleEvent: {
"taskInstanceId" : 1341,
"eventCreateTime" : 1755739838157,
"type" : "FAILED",
"workflowInstanceId" : 1273,
"workflowInstanceHost" : "192.168.30.11:5678",
"taskInstanceHost" : "192.168.30.121:1234",
"appIds" : null,
"endTime" : 1755739838157,
"latestReportTime" : null
}
2025-08-21 09:30:38.768 INFO [PhysicalTaskExecutorEventBusCoordinator-eventbus-coordinator-worker-21] -

********************************* Finalize task instance ************************************

2025-08-21 09:30:38.768 INFO [PhysicalTaskExecutorEventBusCoordinator-eventbus-coordinator-worker-21] - FINALIZE_SESSION
2025-08-21 09:30:38.768 INFO [PhysicalTaskExecutorEventBusCoordinator-eventbus-coordinator-worker-21] - Success fire TaskExecutorFinalizeLifecycleEvent: {
"taskInstanceId" : 1341,
"eventCreateTime" : 1755739838157,
"type" : "FINALIZE"
}

Pull Request Notice

If your pull request contains incompatible change, you should also add it to docs/docs/en/guide/upgrade/incompatible.md

...r-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/utils/ProcessUtils.java

ruanwenjun · 2025-09-03T12:07:07Z

...k-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/AbstractCommandExecutor.java

+        String exitLogMessage = (EXIT_CODE_KILL == exitCode || EXIT_CODE_HARD_KILL == exitCode) ? "process has killed."
+                : "process has exited.";


It's better to check from status in taskExecutionContext rather then check from the exit code, we don't care about the exit, only kill from ds then we think it's kill.

It's better to check from status in taskExecutionContext rather then check from the exit code, we don't care about the exit, only kill from ds then we think it's kill.

Here it simply outputs the final execution status of the process, indicating whether it exited normally or was killed.

ruanwenjun · 2025-09-03T12:08:49Z

...r-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/utils/ProcessUtils.java

            String pids = getPidsStr(processId);
            String[] pidArray = PID_PATTERN.split(pids);
-            if (pidArray.length == 0) {
+            if (StringUtils.isBlank(pids) || pidArray.length == 0) {


ifStringUtils.isEmpty(pids) then we don't need to execute line 117.

ruanwenjun · 2025-09-03T12:09:25Z

...r-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/utils/ProcessUtils.java

+            List<Integer> pidList = Arrays.stream(pidArray).filter(StringUtils::isNotBlank)
+                    .map(s -> {
+                        try {
+                            return Integer.parseInt(s.trim());
+                        } catch (NumberFormatException e) {
+                            log.warn("Invalid PID string ignored: {}", s);
+                            return null;
+                        }
+                    })
+                    .filter(Objects::nonNull)
+                    .collect(Collectors.toList());


It's better to change getPidsStr return a validate pid list, rather than parse the pid in up layer.

ruanwenjun · 2025-09-03T12:10:09Z

...r-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/utils/ProcessUtils.java

+                            return Integer.parseInt(s.trim());
+                        } catch (NumberFormatException e) {
+                            log.warn("Invalid PID string ignored: {}", s);
+                            return null;


Throw exception here, catch the exception will hide the bug.

…into Fix-17436

ruanwenjun

Pretty good.

njnu-seafish · 2025-09-09T09:32:53Z

@SbloodyS Could u plz help review when available? Thanks.

SbloodyS

+1

sonarqubecloud · 2025-09-09T10:55:23Z

Quality Gate failed

Failed conditions
42.9% Coverage on New Code (required ≥ 60%)

See analysis details on SonarQube Cloud

…hod (apache#17005) [Fix-17316][Task-API] Add check process status after killing task (apache#17320) [Fix-17436][Workflow]Task timeout kill throw exception (apache#17437 改造支持seatunnel集群模式停止任务

苏义超 added 2 commits August 21, 2025 10:12

add string blank check

1975953

remove redundant cancelApplication

3b1eed1

njnu-seafish requested review from Gallardot, SbloodyS and caishunfeng as code owners August 21, 2025 02:29

github-actions bot added the backend label Aug 21, 2025

github-actions bot assigned njnu-seafish Aug 21, 2025

njnu-seafish changed the title ~~[Fix 17436][Workflow]Task timeout kill throw exception~~ [Fix-17436][Workflow]Task timeout kill throw exception Aug 21, 2025

SbloodyS added the bug Something isn't working label Aug 21, 2025

SbloodyS added this to the 3.3.1 milestone Aug 21, 2025

github-advanced-security bot found potential problems Aug 21, 2025

View reviewed changes

...r-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/utils/ProcessUtils.java Fixed Show fixed Hide fixed

Merge branch 'dev' into Fix-17436

3493cd8

SbloodyS modified the milestones: 3.3.1, 3.3.2 Aug 25, 2025

SbloodyS requested a review from ruanwenjun August 28, 2025 06:23

njnu-seafish and others added 5 commits September 1, 2025 11:23

Merge branch 'apache:dev' into Fix-17436

5fbab09

do with NumberFormatException

6ecde23

add string blank check

19850fa

remove redundant cancelApplication

6c8e8ab

do with NumberFormatException

48d28cc

ruanwenjun force-pushed the Fix-17436 branch from 6ecde23 to 48d28cc Compare September 3, 2025 12:02

ruanwenjun reviewed Sep 3, 2025

View reviewed changes

苏义超 added 2 commits September 4, 2025 09:13

Merge branch 'Fix-17436' of github.com:njnu-seafish/dolphinscheduler …

2f72344

…into Fix-17436

refactor getPidList function

eb542cf

github-actions bot added the test label Sep 4, 2025

njnu-seafish requested a review from ruanwenjun September 4, 2025 03:44

Merge branch 'dev' into Fix-17436

8daca27

ruanwenjun approved these changes Sep 5, 2025

View reviewed changes

Merge branch 'dev' into Fix-17436

aabdc61

SbloodyS approved these changes Sep 9, 2025

View reviewed changes

SbloodyS merged commit bfb9f17 into apache:dev Sep 9, 2025
70 of 71 checks passed

davidzollo pushed a commit to davidzollo/dolphinscheduler that referenced this pull request Oct 27, 2025

[Fix-17436][Workflow]Task timeout kill throw exception (apache#17437)

383ba93

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Fix-17436][Workflow]Task timeout kill throw exception #17437

[Fix-17436][Workflow]Task timeout kill throw exception #17437

Uh oh!

njnu-seafish commented Aug 21, 2025

Uh oh!

Uh oh!

ruanwenjun Sep 3, 2025

Uh oh!

njnu-seafish Sep 4, 2025

Uh oh!

ruanwenjun Sep 4, 2025

Uh oh!

ruanwenjun Sep 3, 2025

Uh oh!

ruanwenjun Sep 3, 2025

Uh oh!

ruanwenjun Sep 3, 2025

Uh oh!

ruanwenjun left a comment

Uh oh!

njnu-seafish commented Sep 9, 2025

Uh oh!

SbloodyS left a comment

Uh oh!

sonarqubecloud bot commented Sep 9, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		String exitLogMessage = (EXIT_CODE_KILL == exitCode \|\| EXIT_CODE_HARD_KILL == exitCode) ? "process has killed."
		: "process has exited.";

[Fix-17436][Workflow]Task timeout kill throw exception #17437

[Fix-17436][Workflow]Task timeout kill throw exception #17437

Uh oh!

Conversation

njnu-seafish commented Aug 21, 2025

Purpose of the pull request

Brief change log

Verify this pull request

Pull Request Notice

Uh oh!

Uh oh!

ruanwenjun Sep 3, 2025

Choose a reason for hiding this comment

Uh oh!

njnu-seafish Sep 4, 2025

Choose a reason for hiding this comment

Uh oh!

ruanwenjun Sep 4, 2025

Choose a reason for hiding this comment

Uh oh!

ruanwenjun Sep 3, 2025

Choose a reason for hiding this comment

Uh oh!

ruanwenjun Sep 3, 2025

Choose a reason for hiding this comment

Uh oh!

ruanwenjun Sep 3, 2025

Choose a reason for hiding this comment

Uh oh!

ruanwenjun left a comment

Choose a reason for hiding this comment

Uh oh!

njnu-seafish commented Sep 9, 2025

Uh oh!

SbloodyS left a comment

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud bot commented Sep 9, 2025

Quality Gate failed

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants