[DSIP-61][Master] Refactor thread pool and state event orchestration in master #16327

ruanwenjun · 2024-07-16T11:33:41Z

Purpose of the pull request

close #16423

Brief change log

Verify this pull request

This pull request is code cleanup without any test coverage.

(or)

This pull request is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(or)

Pull Request Notice

If your pull request contain incompatible change, you should also add it to docs/docs/en/guide/upgrede/incompatible.md

...heduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/ExecutorController.java

+        String[] processInstanceIdArray = processInstanceIds.split(Constants.COMMA);
+        List<String> errorMessage = new ArrayList<>();
+        for (String strProcessInstanceId : processInstanceIdArray) {
+            int processInstanceId = Integer.parseInt(strProcessInstanceId);


...heduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/ExecutorController.java

+            int processInstanceId = Integer.parseInt(strProcessInstanceId);
+            try {
+                execService.controlWorkflowInstance(loginUser, processInstanceId, executeType);
+                log.info("Success do action {} on workflowInstance: {}", executeType, processInstanceId);


...heduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/ExecutorController.java

+            } catch (Exception e) {
+                errorMessage.add("Failed do action " + executeType + " on workflowInstance: " + processInstanceId
+                        + "reason: " + e.getMessage());
+                log.error("Failed do action {} on workflowInstance: {}, error: {}", executeType, processInstanceId, e);


sonarqubecloud · 2024-07-16T12:33:24Z

Quality Gate failed

Failed conditions
39.7% Coverage on New Code (required ≥ 60%)

See analysis details on SonarCloud

github-advanced-security

CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

...hinscheduler/server/master/engine/task/lifecycle/handler/TaskStartLifecycleEventHandler.java

...er-master/src/main/java/org/apache/dolphinscheduler/server/master/engine/WorkflowEngine.java

...uler-master/src/main/java/org/apache/dolphinscheduler/server/master/config/MasterConfig.java

SbloodyS

LGTM. That's really a huge job. 👍🏻

caishunfeng

Incomplete review

...org/apache/dolphinscheduler/api/executor/workflow/PauseWorkflowInstanceExecutorDelegate.java

...dolphinscheduler/api/executor/workflow/RecoverSuspendedWorkflowInstanceExecutorDelegate.java

.../java/org/apache/dolphinscheduler/api/executor/workflow/TriggerWorkflowExecutorDelegate.java

...n/java/org/apache/dolphinscheduler/api/executor/workflow/WorkflowInstanceControlRequest.java

...rc/main/java/org/apache/dolphinscheduler/extract/master/transportor/ITaskExecutionEvent.java

caishunfeng · 2024-08-23T09:51:31Z

...rc/main/java/org/apache/dolphinscheduler/extract/master/transportor/ITaskExecutionEvent.java

 package org.apache.dolphinscheduler.extract.master.transportor;

-public interface ITaskInstanceExecutionEvent {
+public interface ITaskExecutionEvent {


Suggest to add event source, which can help troubleshoot.

Yes, it is needed, e.g. distinguish the event from the task executor or user operation. I add todo here.

...aster/src/main/java/org/apache/dolphinscheduler/server/master/cluster/MasterSlotManager.java

...ter/src/main/java/org/apache/dolphinscheduler/server/master/engine/TaskGroupCoordinator.java

...er-master/src/main/java/org/apache/dolphinscheduler/server/master/engine/WorkflowEngine.java

...c/main/java/org/apache/dolphinscheduler/server/master/engine/WorkflowEventBusFireWorker.java

...aster/src/main/java/org/apache/dolphinscheduler/server/master/engine/WorkflowRepository.java

...pache/dolphinscheduler/server/master/engine/command/handler/ReRunWorkflowCommandHandler.java

.../dolphinscheduler/server/master/engine/command/handler/RecoverFailureTaskCommandHandler.java

...ain/java/org/apache/dolphinscheduler/server/master/engine/graph/AbstractSuccessorParser.java

...c/main/java/org/apache/dolphinscheduler/server/master/engine/graph/WorkflowGraphFactory.java

caishunfeng · 2024-08-26T02:59:44Z

...main/java/org/apache/dolphinscheduler/server/master/engine/system/event/SystemEventType.java

+    GLOBAL_MASTER_FAILOVER,
+    MASTER_FAILOVER,


It's better to add some comments for the difference.

caishunfeng · 2024-08-26T03:04:36Z

...e/dolphinscheduler/server/master/engine/task/client/PhysicalTaskExecutorClientDelegator.java

+
+@Slf4j
+@Component
+public class PhysicalTaskExecutorClientDelegator implements ITaskExecutorClientDelegator {


Is the PhysicalTask means The tasks which execute on worker?

.../apache/dolphinscheduler/server/master/engine/task/statemachine/AbstractTaskStateAction.java

...n/java/org/apache/dolphinscheduler/server/master/runner/TaskExecutionRunnableRepository.java

...r/src/test/java/org/apache/dolphinscheduler/server/master/AbstractMasterIntegrationTest.java

sonarqubecloud · 2024-08-26T06:22:03Z

Quality Gate failed

Failed conditions
7.0% Coverage on New Code (required ≥ 60%)
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarCloud

Catch issues before they fail your Quality Gate with our IDE extension SonarLint

caishunfeng

LGTM overall.

SbloodyS

+1

ruanwenjun requested review from SbloodyS and caishunfeng as code owners July 16, 2024 11:33

github-actions bot added backend test labels Jul 16, 2024

github-actions bot assigned ruanwenjun Jul 16, 2024

ruanwenjun force-pushed the dev_wenjun_fixSerialWaitWorkflowCannotKill branch 3 times, most recently from d29aba5 to d6052b8 Compare July 16, 2024 11:56

github-advanced-security bot found potential problems Jul 16, 2024

View reviewed changes

ruanwenjun marked this pull request as draft July 16, 2024 13:32

ruanwenjun force-pushed the dev_wenjun_fixSerialWaitWorkflowCannotKill branch from d6052b8 to 465806d Compare July 17, 2024 01:12

github-actions bot added the document label Jul 17, 2024

ruanwenjun force-pushed the dev_wenjun_fixSerialWaitWorkflowCannotKill branch from 465806d to 4adedb5 Compare July 23, 2024 15:39

github-actions bot added the UI ui and front end related label Jul 23, 2024

github-advanced-security bot found potential problems Jul 23, 2024

View reviewed changes

ruanwenjun force-pushed the dev_wenjun_fixSerialWaitWorkflowCannotKill branch from 4adedb5 to 0b40177 Compare July 25, 2024 06:41

This was referenced Jul 26, 2024

[DSIP-57][Audit] Revert PR 14981 code #16370

Merged

[DSIP-57][Audit] Refactor PR 14981 code #16367

Closed

[Improvement][Test] Add api-test cases for workflow and task #16384

Closed

ruanwenjun force-pushed the dev_wenjun_fixSerialWaitWorkflowCannotKill branch 2 times, most recently from f5a524f to c816086 Compare August 14, 2024 14:32

ruanwenjun mentioned this pull request Aug 15, 2024

[Fix-16382] Fix the bug of async master task casthread pool invocations ramp-up #16461

Merged

ruanwenjun force-pushed the dev_wenjun_fixSerialWaitWorkflowCannotKill branch 2 times, most recently from 41a627e to a101f89 Compare August 15, 2024 12:01

ruanwenjun changed the title ~~Fix serial wait workflow instance cannot stop~~ [DSIP-61][Master] Refactor thread pool and state event orchestration in master Aug 15, 2024

ruanwenjun force-pushed the dev_wenjun_fixSerialWaitWorkflowCannotKill branch 3 times, most recently from 8999a0d to 2118c1f Compare August 15, 2024 12:54

SbloodyS mentioned this pull request Aug 16, 2024

[Bug] [Process Instance ] Version 3.2.2 Workflow scheduling uses a serial wait for execution strategy, and task blocking accumulation occurs after master overloads #16474

Closed

3 tasks

ruanwenjun force-pushed the dev_wenjun_fixSerialWaitWorkflowCannotKill branch 4 times, most recently from c143676 to 8f6286a Compare August 21, 2024 02:46

SbloodyS reviewed Aug 21, 2024

View reviewed changes

...hinscheduler/server/master/engine/task/lifecycle/handler/TaskStartLifecycleEventHandler.java Outdated Show resolved Hide resolved

SbloodyS reviewed Aug 21, 2024

View reviewed changes

...er-master/src/main/java/org/apache/dolphinscheduler/server/master/engine/WorkflowEngine.java Outdated Show resolved Hide resolved

...er-master/src/main/java/org/apache/dolphinscheduler/server/master/engine/WorkflowEngine.java Outdated Show resolved Hide resolved

ruanwenjun force-pushed the dev_wenjun_fixSerialWaitWorkflowCannotKill branch from 8f6286a to a3eed5b Compare August 21, 2024 06:23

SbloodyS reviewed Aug 21, 2024

View reviewed changes

...uler-master/src/main/java/org/apache/dolphinscheduler/server/master/config/MasterConfig.java Show resolved Hide resolved

ruanwenjun force-pushed the dev_wenjun_fixSerialWaitWorkflowCannotKill branch from a3eed5b to c320f08 Compare August 21, 2024 06:54

ruanwenjun requested a review from SbloodyS August 21, 2024 07:45

SbloodyS previously approved these changes Aug 21, 2024

View reviewed changes

caishunfeng reviewed Aug 23, 2024

View reviewed changes

ruanwenjun dismissed SbloodyS’s stale review via b5a7d6f August 23, 2024 14:35

ruanwenjun force-pushed the dev_wenjun_fixSerialWaitWorkflowCannotKill branch from b5a7d6f to 23872cf Compare August 23, 2024 14:48

ruanwenjun requested review from SbloodyS and caishunfeng August 23, 2024 15:41

caishunfeng reviewed Aug 26, 2024

View reviewed changes

ruanwenjun added 3 commits August 26, 2024 13:45

[Refactor] Refactor the master threadpool and event engine

5cb97e6

Resolve comments

4433752

Fix comments from shunfeng

109d07e

ruanwenjun force-pushed the dev_wenjun_fixSerialWaitWorkflowCannotKill branch from 5f5fe56 to 109d07e Compare August 26, 2024 05:45

ruanwenjun requested a review from caishunfeng August 26, 2024 06:37

caishunfeng approved these changes Aug 26, 2024

View reviewed changes

SbloodyS approved these changes Aug 26, 2024

View reviewed changes

ruanwenjun merged commit 9448806 into apache:dev Aug 26, 2024

ruanwenjun deleted the dev_wenjun_fixSerialWaitWorkflowCannotKill branch August 26, 2024 07:19

SbloodyS mentioned this pull request Nov 21, 2024

[Bug] [dolphinscheduler - master]switch task type Endless loop #16827

Closed

3 tasks

saowu mentioned this pull request Feb 7, 2025

[Bug] [Task] A single task instance runs or runs backwards only once #16997

Open

3 tasks

[DSIP-61][Master] Refactor thread pool and state event orchestration in master #16327

[DSIP-61][Master] Refactor thread pool and state event orchestration in master #16327

Uh oh!

Conversation

ruanwenjun commented Jul 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose of the pull request

Brief change log

Verify this pull request

Pull Request Notice

Uh oh!

Check notice

Check failure

Check failure

sonarqubecloud bot commented Jul 16, 2024

Quality Gate failed

Uh oh!

github-advanced-security bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SbloodyS left a comment

Choose a reason for hiding this comment

Uh oh!

caishunfeng left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

caishunfeng Aug 23, 2024

Choose a reason for hiding this comment

Uh oh!

ruanwenjun Aug 23, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

caishunfeng Aug 26, 2024

Choose a reason for hiding this comment

Uh oh!

ruanwenjun Aug 26, 2024

Choose a reason for hiding this comment

Uh oh!

caishunfeng Aug 26, 2024

Choose a reason for hiding this comment

Uh oh!

ruanwenjun Aug 26, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sonarqubecloud bot commented Aug 26, 2024

Quality Gate failed

Uh oh!

caishunfeng left a comment

Choose a reason for hiding this comment

Uh oh!

SbloodyS left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

ruanwenjun commented Jul 16, 2024 •

edited

Loading