[Feature][plugin] Add databend datasource in datasource plugin #13866

hantmac · 2023-04-04T06:30:26Z

Purpose of the pull request

Add databend datasource in datasource plugin and close #13857

EricGao888

Could u plz add related docs? Thx : )

EricGao888

Backend part LGTM, would u like to double check? @SbloodyS @caishunfeng

codecov-commenter · 2023-04-11T03:10:17Z

Codecov Report

Merging #13866 (ec09f1b) into dev (fba47b3) will increase coverage by 0.02%.
The diff coverage is 71.42%.

❗ Current head ec09f1b differs from pull request most recent head 8155a6a. Consider uploading reports for the commit 8155a6a to get more accurate results

@@             Coverage Diff              @@
##                dev   #13866      +/-   ##
============================================
+ Coverage     38.92%   38.94%   +0.02%     
- Complexity     4470     4488      +18     
============================================
  Files          1164     1170       +6     
  Lines         42445    42506      +61     
  Branches       4761     4762       +1     
============================================
+ Hits          16520    16556      +36     
- Misses        24105    24128      +23     
- Partials       1820     1822       +2

Impacted Files	Coverage Δ
...eduler/api/service/impl/DataSourceServiceImpl.java	`49.40% <ø> (ø)`
...cheduler/common/constants/DataSourceConstants.java	`0.00% <ø> (ø)`
.../datasource/databend/DatabendDataSourceClient.java	`0.00% <0.00%> (ø)`
.../org/apache/dolphinscheduler/spi/enums/DbType.java	`0.00% <0.00%> (ø)`
...olphinscheduler/plugin/task/api/TaskConstants.java	`60.00% <ø> (ø)`
...dolphinscheduler/plugin/task/datax/DataxUtils.java	`0.00% <0.00%> (ø)`
...inscheduler/server/worker/config/WorkerConfig.java	`3.84% <0.00%> (ø)`
...datasource/databend/DatabendDataSourceChannel.java	`50.00% <50.00%> (ø)`
...rce/databend/DatabendDataSourceChannelFactory.java	`66.66% <66.66%> (ø)`
...ce/databend/param/DatabendDataSourceProcessor.java	`80.43% <80.43%> (ø)`
... and 2 more

... and 3 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

...rg/apache/dolphinscheduler/plugin/datasource/databend/param/DatabendDataSourceProcessor.java

+
+        String[] hostSeperator = connectionParams.getAddress().split(Constants.DOUBLE_SLASH);
+        String[] hostPortArray = hostSeperator[hostSeperator.length - 1].split(Constants.COMMA);
+        databendDatasourceParamDTO.setPort(Integer.parseInt(hostPortArray[0].split(Constants.COLON)[1]));


davidzollo · 2023-04-14T06:27:21Z

Please add license, https://github.com/apache/dolphinscheduler/actions/runs/4663325682/jobs/8254607689?pr=13866
and you can refer the doc, https://github.com/apache/dolphinscheduler/blob/dev/docs/docs/en/contribute/join/DS-License.md

davidzollo

+1
LGTM

fuchanghai · 2023-04-19T09:56:24Z

Hi @hantmac Can you do some tests related to parsing parameters? as follows

hantmac · 2023-04-19T09:57:26Z

Hi @hantmac Can you do some tests related to parsing parameters? as follows

Of course.

fuchanghai

looking forward to perfecting simple parsing parameter tests

hantmac · 2023-04-20T13:27:22Z

I have run ./mvnw spotless:apply and ./mvnw spotless:check locally and successful, why the CI has this kind of error?

hantmac · 2023-04-22T01:57:07Z

looking forward to perfecting simple parsing parameter tests

Hi bro, Would u like to help me review the tests ❤?

fuchanghai · 2023-04-23T09:23:46Z

@SbloodyS it doesn't look like a problem with the code ,plz rerun CI

WUWEI111222333 · 2023-04-24T07:53:45Z

LGTM,I am waiting for this

caishunfeng

LGTM

zhongjiajie · 2023-04-24T09:32:38Z

docs/docs/en/guide/datasource/databend.md

+
+## Native Supported
+
+Yes, could use this datasource by default.


Chinese and English version is different?

should also add new docs in

dolphinscheduler/docs/configs/docsdev.js

Lines 1011 to 1014 in c0126b7

{

title: 'SSH',

link: '/zh-cn/docs/dev/user_doc/guide/datasource/ssh.html',

},

and

dolphinscheduler/docs/configs/docsdev.js

Lines 330 to 333 in c0126b7

{

title: 'SSH',

link: '/en-us/docs/dev/user_doc/guide/datasource/ssh.html',

},

should also add new docs in

dolphinscheduler/docs/configs/docsdev.js

Lines 1011 to 1014 in c0126b7

{

title: 'SSH',

link: '/zh-cn/docs/dev/user_doc/guide/datasource/ssh.html',

},

and

dolphinscheduler/docs/configs/docsdev.js

Lines 330 to 333 in c0126b7

{

title: 'SSH',

link: '/en-us/docs/dev/user_doc/guide/datasource/ssh.html',

},

we need to add some ci for it, or change our website template

Add this line for databend but where is the content of /en-us/docs/dev/user_doc/guide/datasource/ssh.html?

it will auto-generated during website build, from markdown to html

dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-databend/pom.xml

dolphinscheduler-ui/src/views/projects/task/components/node/fields/use-datax.ts

zhongjiajie · 2023-04-24T09:41:44Z

@devosend and @calvinjiang do you have time to take a look the frontend part?

zhongjiajie

LGTM, waiting for CI, BTW, databend is good, I had watch it for a while

zhongjiajie · 2023-04-24T12:48:51Z

spotless failed, please run mvn spotless:apply locally and commit again

zhongjiajie · 2023-04-24T12:55:28Z

I had update it by myself

sonarqubecloud · 2023-04-24T13:27:18Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
18 Code Smells

74.2% Coverage
0.0% Duplication

hantmac · 2023-04-24T14:14:50Z

Thanks all of you!

alextinng · 2023-04-25T09:00:46Z

does connectionParams contains plaintext password?

hantmac · 2023-04-26T02:13:16Z

does connectionParams contains plaintext password?

Yes, you can get the password using the code

        DatabendDataSourceParamDTO databendDataSourceParamDTO = new DatabendDataSourceParamDTO();
        databendDataSourceParamDTO.setHost("localhost");
        databendDataSourceParamDTO.setDatabase("default");
        databendDataSourceParamDTO.setUserName("root");
        databendDataSourceParamDTO.setPort(8000);
        databendDataSourceParamDTO.setPassword("123456");

feat: add databend data source in datasource plugin

c20ff6a

hantmac requested review from SbloodyS, caishunfeng, kezhenxu94 and zhuangchong as code owners April 4, 2023 06:30

github-actions bot added the backend label Apr 4, 2023

add UI for databend

66f2df3

hantmac requested a review from songjianet as a code owner April 6, 2023 06:31

github-actions bot added the UI ui and front end related label Apr 6, 2023

neat

0e23a7d

EricGao888 assigned hantmac Apr 6, 2023

EricGao888 added 3.2.0 for 3.2.0 version miss:docs missing documents in PR labels Apr 6, 2023

EricGao888 added this to the 3.2.0 milestone Apr 6, 2023

EricGao888 reviewed Apr 6, 2023

View reviewed changes

hantmac added 2 commits April 6, 2023 16:12

fix pom

fcab6b5

add docs and dtabend logo

cdf5fd4

hantmac requested review from Tianqi-Dotes and zhongjiajie as code owners April 10, 2023 03:39

github-actions bot added the document label Apr 10, 2023

hantmac requested a review from EricGao888 April 10, 2023 14:01

EricGao888 added the first time contributor First-time contributor label Apr 11, 2023

fix docs style

ff81d74

EricGao888 reviewed Apr 11, 2023

View reviewed changes

EricGao888 added the feature new feature label Apr 11, 2023

github-advanced-security bot found potential problems Apr 11, 2023

View reviewed changes

add License for databend

a9b0754

davidzollo previously approved these changes Apr 17, 2023

View reviewed changes

fuchanghai requested changes Apr 19, 2023

View reviewed changes

add more test cases

f3134fd

hantmac dismissed SbloodyS’s stale review via f3134fd April 19, 2023 13:05

add license header

4cbd8b2

Merge remote-tracking branch 'dol-up/dev' into feat/databend-source

62b11b0

caishunfeng previously approved these changes Apr 24, 2023

View reviewed changes

fuchanghai previously approved these changes Apr 24, 2023

View reviewed changes

zhongjiajie requested changes Apr 24, 2023

View reviewed changes

zhongjiajie mentioned this pull request Apr 24, 2023

[Doc][toc] new docs check or change website template #14004

Closed

3 tasks

fix

7724715

hantmac dismissed stale reviews from fuchanghai and caishunfeng via 7724715 April 24, 2023 12:43

zhongjiajie previously approved these changes Apr 24, 2023

View reviewed changes

style

8155a6a

zhongjiajie dismissed their stale review via 8155a6a April 24, 2023 12:54

zhongjiajie approved these changes Apr 24, 2023

View reviewed changes

zhongjiajie merged commit db6c061 into apache:dev Apr 24, 2023


		## Native Supported

		Yes, could use this datasource by default.

	{
	title: 'SSH',
	link: '/zh-cn/docs/dev/user_doc/guide/datasource/ssh.html',
	},

	{
	title: 'SSH',
	link: '/en-us/docs/dev/user_doc/guide/datasource/ssh.html',
	},

[Feature][plugin] Add databend datasource in datasource plugin #13866

[Feature][plugin] Add databend datasource in datasource plugin #13866

Uh oh!

Conversation

hantmac commented Apr 4, 2023

Purpose of the pull request

Uh oh!

EricGao888 left a comment

Choose a reason for hiding this comment

Uh oh!

EricGao888 left a comment

Choose a reason for hiding this comment

Uh oh!

codecov-commenter commented Apr 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Check notice

davidzollo commented Apr 14, 2023

Uh oh!

davidzollo left a comment

Choose a reason for hiding this comment

Uh oh!

fuchanghai commented Apr 19, 2023

Uh oh!

hantmac commented Apr 19, 2023

Uh oh!

fuchanghai left a comment

Choose a reason for hiding this comment

Uh oh!

hantmac commented Apr 20, 2023

Uh oh!

hantmac commented Apr 22, 2023

Uh oh!

fuchanghai commented Apr 23, 2023

Uh oh!

WUWEI111222333 commented Apr 24, 2023

Uh oh!

caishunfeng left a comment

Choose a reason for hiding this comment

Uh oh!

zhongjiajie Apr 24, 2023

Choose a reason for hiding this comment

Uh oh!

zhongjiajie Apr 24, 2023

Choose a reason for hiding this comment

Uh oh!

zhongjiajie Apr 24, 2023

Choose a reason for hiding this comment

Uh oh!

hantmac Apr 24, 2023

Choose a reason for hiding this comment

Uh oh!

zhongjiajie Apr 24, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

zhongjiajie commented Apr 24, 2023

Uh oh!

zhongjiajie left a comment

Choose a reason for hiding this comment

Uh oh!

zhongjiajie commented Apr 24, 2023

Uh oh!

zhongjiajie commented Apr 24, 2023

Uh oh!

sonarqubecloud bot commented Apr 24, 2023

Uh oh!

hantmac commented Apr 24, 2023

Uh oh!

alextinng commented Apr 25, 2023

Uh oh!

hantmac commented Apr 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

codecov-commenter commented Apr 11, 2023 •

edited

Loading

hantmac commented Apr 26, 2023 •

edited

Loading