[Future][Connector-V2]Support the automatic creation of non-primary key table #9219

hawk9821 · 2025-04-23T08:23:16Z

Purpose of this pull request

Support the automatic creation of non-primary key table ( append-only table) when Source is the primary key table
Fix losing the primary key when loading the Schema in Paimon Source

Does this PR introduce any user-facing change?

How was this patch tested?

UT: PaimonCatalogPrimaryTest
E2E: PaimonSinkCDCIT#testChangelogLookup

Check list

If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update plugin-mapping.properties and add new connector information in it
2. Update the pom file of seatunnel-dist
3. Add ci label in label-scope-conf
4. Add e2e testcase in seatunnel-e2e
5. Update connector plugin_config
Update the release-note.

Copilot

Pull Request Overview

This pull request introduces support for the automatic creation of non-primary key (append-only) tables when the source is a primary key table and fixes the issue of losing primary key information when loading the schema in Paimon Source.

Adds a new unit test to verify primary key preservation.
Updates schema conversion and sink configuration to handle the non-primary key scenario.
Introduces a new error code and configuration option for non-primary key validation.

Reviewed Changes

Copilot reviewed 6 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
seatunnel-connectors-v2/connector-paimon/src/test/java/org/apache/seatunnel/connectors/seatunnel/paimon/catalog/PaimonCatalogPrimaryTest.java	Adds tests to verify primary key behavior.
seatunnel-connectors-v2/connector-paimon/src/main/java/org/apache/seatunnel/connectors/seatunnel/paimon/utils/SchemaUtil.java	Updates schema conversion to clear primary keys when non-primary key is enabled.
seatunnel-connectors-v2/connector-paimon/src/main/java/org/apache/seatunnel/connectors/seatunnel/paimon/exception/PaimonConnectorErrorCode.java	Introduces a new error code for non-primary key check errors.
seatunnel-connectors-v2/connector-paimon/src/main/java/org/apache/seatunnel/connectors/seatunnel/paimon/config/PaimonSinkOptions.java	Adds a new NON_PRIMARY_KEY option with a description that appears ambiguous.
seatunnel-connectors-v2/connector-paimon/src/main/java/org/apache/seatunnel/connectors/seatunnel/paimon/config/PaimonSinkConfig.java	Implements validation to throw an exception when non-primary key is true but primary keys are provided.
seatunnel-connectors-v2/connector-paimon/src/main/java/org/apache/seatunnel/connectors/seatunnel/paimon/catalog/PaimonCatalog.java	Adjusts catalog conversion logic and sets primary key and constraint key based on the schema.

Files not reviewed (1)

seatunnel-e2e/seatunnel-connector-v2-e2e/connector-paimon-e2e/src/test/resources/changelog_paimon_to_paimon.conf: Language not supported

Comments suppressed due to low confidence (2)

seatunnel-connectors-v2/connector-paimon/src/main/java/org/apache/seatunnel/connectors/seatunnel/paimon/config/PaimonSinkConfig.java:55

The exception for the non-primary key check is thrown with an empty error message. Provide a descriptive message to aid debugging.

throw new PaimonConnectorException(PaimonConnectorErrorCode.NON_PRIMARY_KEY_CHECK_ERROR, "");

seatunnel-connectors-v2/connector-paimon/src/main/java/org/apache/seatunnel/connectors/seatunnel/paimon/catalog/PaimonCatalog.java:276

Typo in variable name 'primaryKyes'. Consider renaming it to 'primaryKeys' for clarity.

List<String> primaryKyes = schema.primaryKeys();

Copilot · 2025-04-23T08:27:55Z

...src/main/java/org/apache/seatunnel/connectors/seatunnel/paimon/config/PaimonSinkOptions.java

                    .defaultValue(DataSaveMode.APPEND_DATA)
                    .withDescription("data_save_mode");
+
+    public static final Option<Boolean> NON_PRIMARY_KEY =


The description for the NON_PRIMARY_KEY option does not match its functionality (clearing primary keys). Consider updating the description to reflect that setting this option to true disables primary key handling.

Hisoka-X · 2025-04-24T11:14:06Z

...on/src/main/java/org/apache/seatunnel/connectors/seatunnel/paimon/catalog/PaimonCatalog.java

+            List<ConstraintKey.ConstraintKeyColumn> constraintKeyColumns =
+                    primaryKyes.stream()
+                            .map(
+                                    pk ->
+                                            new ConstraintKey.ConstraintKeyColumn(
+                                                    pk, ConstraintKey.ColumnSortType.ASC))
+                            .collect(Collectors.toList());
+            builder.constraintKey(
+                    ConstraintKey.of(
+                            ConstraintKey.ConstraintType.UNIQUE_KEY, "uk", constraintKeyColumns));


I think if a field is a primary key, it must be unique. So do not put it into constraintKey.

...src/main/java/org/apache/seatunnel/connectors/seatunnel/paimon/config/PaimonSinkOptions.java

Wrong click

...imon/src/main/java/org/apache/seatunnel/connectors/seatunnel/paimon/config/PaimonConfig.java

Hisoka-X · 2025-06-09T03:41:31Z

...nel-connector-v2-e2e/connector-paimon-e2e/src/test/resources/changelog_paimon_to_paimon.conf

-    paimon.table.write-props = {
-      write-only = true
-    }


why remove this?

this test case requires an apped only table (non-primary key table), and the write-only parameter controls Whether to perform compression when writing.

Hisoka-X

Please update the docs.

hawk9821 · 2025-06-20T08:52:10Z

Please update the docs.

updated

…ey tables

Hisoka-X · 2025-06-25T03:23:57Z

wating test case passes.

…ey table (apache#9219)

github-actions bot added connectors-v2 e2e paimon labels Apr 23, 2025

hailin0 requested a review from Copilot April 23, 2025 08:27

Copilot AI reviewed Apr 23, 2025

View reviewed changes

Hisoka-X previously approved these changes Apr 24, 2025

View reviewed changes

github-actions bot added approved reviewed labels Apr 24, 2025

github-actions bot removed approved reviewed labels Apr 24, 2025

hawk9821 force-pushed the paimon_source_pk branch 3 times, most recently from 3738339 to bd2c8b6 Compare April 27, 2025 07:06

hawk9821 force-pushed the paimon_source_pk branch 3 times, most recently from 7a22afd to 7a42d4a Compare June 8, 2025 15:08

Hisoka-X reviewed Jun 9, 2025

View reviewed changes

...imon/src/main/java/org/apache/seatunnel/connectors/seatunnel/paimon/config/PaimonConfig.java Outdated Show resolved Hide resolved

Hisoka-X reviewed Jun 9, 2025

View reviewed changes

hawk9821 force-pushed the paimon_source_pk branch from 32b92e8 to ded511a Compare June 16, 2025 00:41

Hisoka-X reviewed Jun 20, 2025

View reviewed changes

github-actions bot added the document label Jun 20, 2025

[Future][Connector-V2]Support the automatic creation of non-primary k…

fb6db69

…ey tables

hawk9821 force-pushed the paimon_source_pk branch from 450e580 to fb6db69 Compare June 24, 2025 11:18

Hisoka-X approved these changes Jun 25, 2025

View reviewed changes

github-actions bot added approved reviewed labels Jun 25, 2025

corgy-w approved these changes Jun 26, 2025

View reviewed changes

corgy-w merged commit 93e539c into apache:dev Jun 26, 2025
5 checks passed

hawk9821 deleted the paimon_source_pk branch June 27, 2025 01:51

dybyte pushed a commit to dybyte/seatunnel that referenced this pull request Jul 23, 2025

[Future][Connector-V2]Support the automatic creation of non-primary k…

622ebb7

…ey table (apache#9219)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Future][Connector-V2]Support the automatic creation of non-primary key table #9219

[Future][Connector-V2]Support the automatic creation of non-primary key table #9219

Uh oh!

hawk9821 commented Apr 23, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Apr 23, 2025

Uh oh!

Hisoka-X Apr 24, 2025

Uh oh!

Uh oh!

Uh oh!

Hisoka-X Jun 9, 2025

Uh oh!

hawk9821 Jun 20, 2025

Uh oh!

Hisoka-X left a comment

Uh oh!

hawk9821 commented Jun 20, 2025

Uh oh!

Hisoka-X commented Jun 25, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

[Future][Connector-V2]Support the automatic creation of non-primary key table #9219

[Future][Connector-V2]Support the automatic creation of non-primary key table #9219

Uh oh!

Conversation

hawk9821 commented Apr 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Apr 23, 2025

Choose a reason for hiding this comment

Uh oh!

Hisoka-X Apr 24, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Hisoka-X Jun 9, 2025

Choose a reason for hiding this comment

Uh oh!

hawk9821 Jun 20, 2025

Choose a reason for hiding this comment

Uh oh!

Hisoka-X left a comment

Choose a reason for hiding this comment

Uh oh!

hawk9821 commented Jun 20, 2025

Uh oh!

Hisoka-X commented Jun 25, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

hawk9821 commented Apr 23, 2025 •

edited

Loading