[Improve][Connector-V2] Support hadoop ha and kerberos for paimon sink #6585

dailai · 2024-03-27T02:58:46Z

Purpose of this pull request

Support hadoop-ha and kerberos for paimon sink
Fix the bug of date type in paimon sink. We should convert date type to int when we sink paimon ranther then timestamp. Refer: org.apache.paimon.data.BinaryWriter#createValueSetter. Otherwise, when we use flink to query the date data in paimon, the date data is not correct

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update change log that in connector document. For more details you can refer to connector-v2
2. Update plugin-mapping.properties and add new connector information in it
3. Update the pom file of seatunnel-dist
Update the release-note.

hailin0 · 2024-03-27T10:38:58Z

docs/en/connector-v2/sink/Paimon.md

 | warehouse                   | String | Yes      | -                            | Paimon warehouse path                                                                                                                                            |
 | database                    | String | Yes      | -                            | The database you want to access                                                                                                                                  |
 | table                       | String | Yes      | -                            | The table you want to access                                                                                                                                     |
-| hdfs_site_path              | String | No       | -                            |                                                                                                                                                                  |


You should be compatible with older versions

You should be compatible with older versions

Yes, the code is compatible with older versions. I've updated the docs.

…affecting other hadoop filesystem jobs

...n/src/main/java/org/apache/seatunnel/connectors/seatunnel/paimon/sink/PaimonSinkFactory.java

…e factory for paimon sink

hailin0 · 2024-04-07T05:04:51Z

LGTM

cc @EricJoy2048

Carl-Zhou-CN · 2024-04-07T11:28:40Z

link #6076

Carl-Zhou-CN · 2024-04-07T11:40:08Z

@dailai hi,Since ci cannot test your test case, can you run the test case locally and provide screenshots

Carl-Zhou-CN · 2024-04-07T11:43:20Z

...imon/src/main/java/org/apache/seatunnel/connectors/seatunnel/paimon/config/PaimonConfig.java

+        return argument;
+    }
+
+    @VisibleForTesting


Can we remove this annotation? It is not intended for testing purposes.

@Carl-Zhou-CN
This annotation is meaningful to prevent method permissions from being changed.

ok, it's my oversight

dailai · 2024-04-08T00:01:12Z

@dailai hi,Since ci cannot test your test case, can you run the test case locally and provide screenshots

Carl-Zhou-CN

LGTM

Carl-Zhou-CN · 2024-04-08T01:39:02Z

cc @EricJoy2048

EricJoy2048 · 2024-04-08T06:20:37Z

LGTM, Thanks for your contribution.

apache#6585)

dailai added 2 commits March 27, 2024 10:46

[Fix][Paimon] Fix write problem of the datatype with date

cfbeace

[Feature][Paimon] Support hadoop-ha and kerberos

2f2385d

hailin0 added improve paimon labels Mar 27, 2024

hailin0 reviewed Mar 27, 2024

View reviewed changes

[Improve][Connector-V2] update the docs for paimon sink

7038a7d

dailai changed the title ~~[Feature][Connector-V2] Support hadoop ha and kerberos for paimon sink~~ [Improve][Connector-V2] Support hadoop ha and kerberos for paimon sink Mar 27, 2024

hailin0 added this to the 2.3.5 milestone Mar 29, 2024

dailai requested a review from hailin0 April 1, 2024 05:31

[Improve][Connector-V2] Enable the fs cache for paimon sink to avoid …

7025b05

…affecting other hadoop filesystem jobs

dailai force-pushed the support-hadoop-ha-and-kerberos branch from 8cc8559 to 7025b05 Compare April 3, 2024 01:04

[Improve][Connector-V2] Fix the method name in PaimonSecurityContext

36f1c42

hailin0 reviewed Apr 3, 2024

View reviewed changes

...n/src/main/java/org/apache/seatunnel/connectors/seatunnel/paimon/sink/PaimonSinkFactory.java Outdated Show resolved Hide resolved

[Improve][Connector-V2] Add hadoop_conf_path to the option rule of th…

71b849d

…e factory for paimon sink

dailai requested a review from hailin0 April 3, 2024 08:22

Carl-Zhou-CN reviewed Apr 7, 2024

View reviewed changes

dailai requested a review from Carl-Zhou-CN April 8, 2024 00:02

Carl-Zhou-CN approved these changes Apr 8, 2024

View reviewed changes

github-actions bot added approved reviewed labels Apr 8, 2024

EricJoy2048 approved these changes Apr 8, 2024

View reviewed changes

EricJoy2048 merged commit 20b62f3 into apache:dev Apr 8, 2024

dailai deleted the support-hadoop-ha-and-kerberos branch April 10, 2024 06:56

chaorongzhi pushed a commit to chaorongzhi/seatunnel that referenced this pull request Aug 21, 2024

[Improve][Connector-V2] Support hadoop ha and kerberos for paimon sink (

893c90b

apache#6585)

[Improve][Connector-V2] Support hadoop ha and kerberos for paimon sink #6585

[Improve][Connector-V2] Support hadoop ha and kerberos for paimon sink #6585

Uh oh!

Conversation

dailai commented Mar 27, 2024

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

Uh oh!

hailin0 Mar 27, 2024

Choose a reason for hiding this comment

Uh oh!

dailai Mar 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hailin0 commented Apr 7, 2024

Uh oh!

Carl-Zhou-CN commented Apr 7, 2024

Uh oh!

Carl-Zhou-CN commented Apr 7, 2024

Uh oh!

Carl-Zhou-CN Apr 7, 2024

Choose a reason for hiding this comment

Uh oh!

hailin0 Apr 7, 2024

Choose a reason for hiding this comment

Uh oh!

Carl-Zhou-CN Apr 7, 2024

Choose a reason for hiding this comment

Uh oh!

dailai commented Apr 8, 2024

Uh oh!

Carl-Zhou-CN left a comment

Choose a reason for hiding this comment

Uh oh!

Carl-Zhou-CN commented Apr 8, 2024

Uh oh!

EricJoy2048 commented Apr 8, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dailai Mar 27, 2024 •

edited

Loading