Skip to content

[Feature-12456][task plugin] Add new task plugin for apache linkis #12693

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
zhongjiajie merged 10 commits into from
Nov 17, 2022
Merged

[Feature-12456][task plugin] Add new task plugin for apache linkis #12693

zhongjiajie merged 10 commits into from
Nov 17, 2022

Conversation

@shangeyao
Copy link
Contributor

@shangeyao shangeyao commented Nov 3, 2022

Purpose of the pull request

close #12456

Brief change log

Add new task plugin for apache linkis

@github-actions github-actions bot added backend UI ui and front end related labels Nov 3, 2022
@ruanwenjun ruanwenjun added this to the 3.2.0 milestone Nov 4, 2022
caishunfeng
caishunfeng requested changes Nov 4, 2022
View reviewed changes
...ler-task-linkis/src/main/java/org/apache/dolphinscheduler/plugin/task/linkis/LinkisTask.java Outdated

// todo split handle to submit and track
@Override
public void handle(TaskCallBack taskCallBack) throws TaskException {
Copy link
Contributor

@caishunfeng caishunfeng Nov 4, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please split into submit and track if this is a remote task, and then when task failover, it will not submit remote task again and just track status.

@github-actions github-actions bot removed the document label Nov 7, 2022
@codecov-commenter
Copy link

codecov-commenter commented Nov 7, 2022 •
edited
Loading

Codecov Report

Merging #12693 (9e1cf63) into dev (6d8befa) will decrease coverage by 0.21%.
The diff coverage is 0.60%.

❗ Current head 9e1cf63 differs from pull request most recent head 22163a8. Consider uploading reports for the commit 22163a8 to get more accurate results

@@             Coverage Diff              @@
##                dev   #12693      +/-   ##
============================================
- Coverage     39.46%   39.25%   -0.22%     
+ Complexity     4247     4241       -6     
============================================
  Files          1052     1058       +6     
  Lines         39860    39976     +116     
  Branches       4582     4585       +3     
============================================
- Hits          15731    15692      -39     
- Misses        22349    22503     +154     
- Partials       1780     1781       +1
Impacted Files Coverage Δ
...dolphinscheduler/plugin/task/linkis/Constants.java 0.00% <0.00%> (ø)
...nscheduler/plugin/task/linkis/LinkisJobStatus.java 0.00% <0.00%> (ø)
...olphinscheduler/plugin/task/linkis/LinkisTask.java 0.00% <0.00%> (ø)
...cheduler/plugin/task/linkis/LinkisTaskChannel.java 0.00% <0.00%> (ø)
...r/plugin/task/linkis/LinkisTaskChannelFactory.java 0.00% <0.00%> (ø)
...scheduler/plugin/task/linkis/LinkisParameters.java 33.33% <33.33%> (ø)
...r/plugin/registry/zookeeper/ZookeeperRegistry.java 43.54% <0.00%> (-6.46%) ⬇️
...er/master/dispatch/host/assign/RandomSelector.java 77.77% <0.00%> (-5.56%) ⬇️
...e/dolphinscheduler/remote/NettyRemotingClient.java 50.00% <0.00%> (-2.78%) ⬇️
...eduler/api/dto/schedule/ScheduleCreateRequest.java 83.33% <0.00%> (-2.39%) ⬇️
... and 32 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

github-advanced-security[bot]
github-advanced-security bot found potential problems Nov 7, 2022
View reviewed changes
...ler-task-linkis/src/main/java/org/apache/dolphinscheduler/plugin/task/linkis/LinkisTask.java
TaskResponse commandExecuteResult = shellCommandExecutor.run(command);
String status = findStatus(commandExecuteResult.getResultString());
LinkisJobStatus jobStatus = LinkisJobStatus.convertFromJobStatusString(status);
switch (jobStatus) {

Check warning

Code scanning / CodeQL

Missing enum case in switch

Switch statement does not have a case for [SHUTTINGDOWN](1). Switch statement does not have a case for [UNKNOWN](2). Switch statement does not have a case for [TIMEOUT](3). Switch statement does not have a case for [RUNNING](4). Switch statement does not have a case for [SCHEDULED](5). Switch statement does not have a case for [WAIT_FOR_RETRY](6). Switch statement does not have a case for [INITED](7). Switch statement does not have a case for [SUBMITTING](8). Switch statement does not have a case for [UNSUBMITTED](9).
@shangeyao
Copy link
Contributor Author

shangeyao commented Nov 8, 2022

please review again @caishunfeng

@shangeyao shangeyao requested review from caishunfeng and removed request for Amy0104 November 8, 2022 03:00
caishunfeng
caishunfeng previously approved these changes Nov 15, 2022
View reviewed changes
Copy link
Contributor

@caishunfeng caishunfeng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM of the backend part.

@caishunfeng
Copy link
Contributor

caishunfeng commented Nov 15, 2022

Please add the doc in this pr too. @shangeyao

@caishunfeng caishunfeng added the miss:docs missing documents in PR label Nov 15, 2022
zhongjiajie
zhongjiajie reviewed Nov 16, 2022
View reviewed changes
Copy link
Member

@zhongjiajie zhongjiajie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Others looking good

docs/docs/en/guide/task/linkis.md
@@ -0,0 +1,49 @@
# Apache Linkis
Copy link
Member

@zhongjiajie zhongjiajie Nov 16, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, due to we add new docs in this PR, we should also add it into https://github.com/apache/dolphinscheduler/blob/dev/docs/configs/docsdev.js

docs/docs/en/guide/task/linkis.md Outdated
### Attention

- No need to fill `sh ./bin/linkis-cli` in the configuration column, it has been configured in advance.
- The default configuration is asynchronous submission. You do not need to configure the `-- async` parameter.
Copy link
Member

@zhongjiajie zhongjiajie Nov 16, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- The default configuration is asynchronous submission. You do not need to configure the `-- async` parameter.
- The default configuration is asynchronous submission. You do not need to configure the `--async` parameter.

docs/docs/en/guide/task/linkis.md Outdated

### Config example

```Config
Copy link
Member

@zhongjiajie zhongjiajie Nov 16, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure about that, but should this be shell syntax instead of config?

Suggested change
```Config
```sh

@sonarqubecloud
Copy link

sonarqubecloud bot commented Nov 16, 2022

SonarCloud Quality Gate failed.    Quality Gate failed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 13 Code Smells

0.5% 0.5% Coverage
1.6% 1.6% Duplication

@zhongjiajie zhongjiajie added the 3.2.0 for 3.2.0 version label Nov 16, 2022
devosend
devosend approved these changes Nov 16, 2022
View reviewed changes
Copy link
Contributor

@devosend devosend left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zhongjiajie zhongjiajie merged commit 2dbc796 into apache:dev Nov 17, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zhongjiajie zhongjiajie zhongjiajie left review comments

@devosend devosend devosend approved these changes

@songjianet songjianet songjianet approved these changes

@Tianqi-Dotes Tianqi-Dotes Awaiting requested review from Tianqi-Dotes

@Amy0104 Amy0104 Awaiting requested review from Amy0104

@caishunfeng caishunfeng Awaiting requested review from caishunfeng caishunfeng is a code owner

Assignees

@shangeyao shangeyao

Labels

3.2.0 for 3.2.0 version backend document feature new feature miss:docs missing documents in PR UI ui and front end related

Projects

None yet

Milestone

3.2.0

Development

Successfully merging this pull request may close these issues.

[Feature][task plugin] Add new task plugin for apache linkis

7 participants

@shangeyao @codecov-commenter @caishunfeng @devosend @zhongjiajie @songjianet @ruanwenjun