[Feature][Connector-V2] Support like predicate pushdown in paimon #9484

xiaochen-zhou · 2025-06-20T18:02:07Z

Purpose of this pull request

Support like predicate pushdown in paimon

Does this PR introduce any user-facing change?

No

How was this patch tested?

Add new tests

Check list

If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update plugin-mapping.properties and add new connector information in it
2. Update the pom file of seatunnel-dist
3. Add ci label in label-scope-conf
4. Add e2e testcase in seatunnel-e2e
5. Update connector plugin_config

Hisoka-X · 2025-06-24T03:50:46Z

cc @hawk9821 as well.

Hisoka-X

Please update the docs.

xiaochen-zhou · 2025-06-24T04:28:19Z

Please update the docs.

OK.

hawk9821 · 2025-06-24T11:38:06Z

...he/seatunnel/connectors/seatunnel/paimon/source/converter/SqlToPaimonPredicateConverter.java

+            Object rightVal =
+                    convertValueByPaimonDataType(rowType, column.getColumnName(), rightPredicate);
+
+            Pattern BEGIN_PATTERN = Pattern.compile("([^%]+)%");


%keyword% not supported . I think should be printe some logs or throw exception.

Hisoka-X · 2025-06-25T13:33:13Z

docs/en/connector-v2/source/Paimon.md


 The filter condition of the table read. For example: `select * from st_test where id > 100`. If not specified, all rows are read.
-Currently, where conditions only support <, <=, >, >=, =, !=, or, and,is null, is not null, between...and, in, not in, and others are not supported.
+Currently, where conditions only support <, <=, >, >=, =, !=, or, and,is null, is not null, between...and, in, not in, like(only support startWith) ,and others are not supported.


Suggested change

Currently, where conditions only support <, <=, >, >=, =, !=, or, and,is null, is not null, between...and, in, not in, like(only support startWith) ,and others are not supported.

Currently, where conditions only support <, <=, >, >=, =, !=, or, and,is null, is not null, between...and, in, not in, like(pattern matching with prefix only) ,and others are not supported.

Hisoka-X · 2025-06-25T13:33:27Z

docs/zh/connector-v2/source/Paimon.md

 读取表格的筛选条件，例如：`select * from st_test where id > 100`。如果未指定，则将读取所有记录。 

-目前，`where` 支持`<, <=, >, >=, =, !=, or, and,is null, is not null, between...and, in , not in`，其他暂不支持。 
+目前，`where` 支持`<, <=, >, >=, =, !=, or, and,is null, is not null, between...and, in , not in, like(only support startWith)`，其他暂不支持。 


Hisoka-X · 2025-06-25T13:33:32Z

...he/seatunnel/connectors/seatunnel/paimon/source/converter/SqlToPaimonPredicateConverter.java

+                return builder.startsWith(columnIndex, BinaryString.fromString(matcher.group(1)));
+            } else {
+                throw new IllegalArgumentException(
+                        "Unsupported expression type: "


Copilot

Pull Request Overview

This pull request adds support for like predicate pushdown in Paimon by enhancing the SQL predicate conversion logic, updating end-to-end tests, and revising documentation.

Enhanced SQL to Paimon predicate conversion to support prefix matching via the LIKE operator.
Introduced new end-to-end tests to validate the functionality.
Updated documentation (both English and Chinese) to reflect supported where clause conditions.

Reviewed Changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
seatunnel-e2e/seatunnel-connector-v2-e2e/connector-paimon-e2e/src/test/resources/paimon_to_assert_with_filter8.conf	New test configuration file for LIKE predicate tests
seatunnel-e2e/seatunnel-connector-v2-e2e/connector-paimon-e2e/src/test/java/org/apache/seatunnel/e2e/connector/paimon/PaimonSinkCDCIT.java	Added test execution for new filter config; increased wait durations
seatunnel-connectors-v2/connector-paimon/src/test/java/org/apache/seatunnel/connectors/seatunnel/paimon/source/converter/SqlToPaimonConverterTest.java	Added test validating LIKE predicate conversion for prefix matching
seatunnel-connectors-v2/connector-paimon/src/main/java/org/apache/seatunnel/connectors/seatunnel/paimon/source/converter/SqlToPaimonPredicateConverter.java	Updated SQL predicate converter to handle LIKE expressions with prefix matching
docs/zh/connector-v2/source/Paimon.md, docs/en/connector-v2/source/Paimon.md	Updated documentation to include support for like (prefix matching) in the where clause

Copilot · 2025-06-26T02:51:18Z

...ctor-paimon-e2e/src/test/java/org/apache/seatunnel/e2e/connector/paimon/PaimonSinkCDCIT.java

                container.executeJob("/changelog_fake_cdc_sink_paimon_case1_ddl.conf");
        Assertions.assertEquals(0, writeResult.getExitCode());
-        TimeUnit.SECONDS.sleep(20);
+        TimeUnit.SECONDS.sleep(120);


The increased sleep duration to 120 seconds may unnecessarily prolong test execution; consider using a condition-based waiting mechanism to detect readiness instead of a fixed sleep.

Suggested change

TimeUnit.SECONDS.sleep(120);

given().ignoreExceptions()

.await()

.atMost(120, TimeUnit.SECONDS)

.untilAsserted(() -> {

// Check readiness condition

Container.ExecResult readinessCheck =

container.executeJob("/changelog_fake_cdc_sink_paimon_case1_readiness_check.conf");

Assertions.assertEquals(0, readinessCheck.getExitCode());

});

Copilot · 2025-06-26T02:51:18Z

...he/seatunnel/connectors/seatunnel/paimon/source/converter/SqlToPaimonPredicateConverter.java

+            Object rightVal =
+                    convertValueByPaimonDataType(rowType, column.getColumnName(), rightPredicate);
+
+            Pattern BEGIN_PATTERN = Pattern.compile("([^%]+)%");


[nitpick] Consider defining the compiled pattern as a static final constant to avoid recompiling it on each method invocation, which can improve performance and readability.

Suggested change

Pattern BEGIN_PATTERN = Pattern.compile("([^%]+)%");

Hisoka-X · 2025-06-26T13:22:14Z

...he/seatunnel/connectors/seatunnel/paimon/source/converter/SqlToPaimonPredicateConverter.java

+                throw new IllegalArgumentException(
+                        "Unsupported expression type: "
+                                + expression.getClass().getSimpleName()
+                                + ", only support LikeExpression with 'startWith' pattern ");


Suggested change

+ ", only support LikeExpression with 'startWith' pattern ");

+ ", only support LikeExpression pattern matching with prefix");

xiaochen-zhou · 2025-06-28T01:45:50Z

@Hisoka-X @hawk9821 done.

Hisoka-X

LGTM. But we should waiting #9467 be merged first.

Hisoka-X · 2025-07-02T12:51:50Z

Hi @xiaochen-zhou . Please rebase on dev to retrigger ci. Thanks

…ache#9484)

[Feature][Connector-V2] Support like predicate pushdown in paimon

de2fba0

github-actions bot added connectors-v2 paimon labels Jun 20, 2025

davidzollo requested a review from Copilot June 21, 2025 12:32

This comment was marked as outdated.

Sign in to view

Hisoka-X reviewed Jun 24, 2025

View reviewed changes

hawk9821 reviewed Jun 24, 2025

View reviewed changes

[Feature][Connector-V2] Support like predicate pushdown in paimon

43d72fb

github-actions bot added the document label Jun 24, 2025

[Fix][Connector-V2] add e2e

3ac0204

github-actions bot added the e2e label Jun 25, 2025

Hisoka-X reviewed Jun 25, 2025

View reviewed changes

xiaochen-zhou added 2 commits June 26, 2025 07:33

[Fix][Connector-V2] update e2e

c334d96

[Fix][Connector-V2] update e2e

ef70f0c

nielifeng requested a review from Copilot June 26, 2025 02:50

Copilot AI reviewed Jun 26, 2025

View reviewed changes

Hisoka-X reviewed Jun 26, 2025

View reviewed changes

[Feature][Connector-V2] update e2e

0cc8fbe

Hisoka-X reviewed Jun 28, 2025

View reviewed changes

xiaochen-zhou requested review from Hisoka-X and hawk9821 July 1, 2025 04:12

Merge branch 'dev' into paimon-like-predicate

42ac276

Hisoka-X approved these changes Jul 3, 2025

View reviewed changes

github-actions bot added approved reviewed labels Jul 3, 2025

corgy-w approved these changes Jul 4, 2025

View reviewed changes

corgy-w merged commit a19720c into apache:dev Jul 4, 2025
5 checks passed

dybyte pushed a commit to dybyte/seatunnel that referenced this pull request Jul 23, 2025

[Feature][Connector-V2] Support like predicate pushdown in paimon (ap…

d8cb330

…ache#9484)

	Currently, where conditions only support <, <=, >, >=, =, !=, or, and,is null, is not null, between...and, in, not in, like(only support startWith) ,and others are not supported.
	Currently, where conditions only support <, <=, >, >=, =, !=, or, and,is null, is not null, between...and, in, not in, like(pattern matching with prefix only) ,and others are not supported.

-        TimeUnit.SECONDS.sleep(120);
+        given().ignoreExceptions()
+                .await()
+                .atMost(120, TimeUnit.SECONDS)
+                .untilAsserted(() -> {
+                    // Check readiness condition
+                    Container.ExecResult readinessCheck =
+                            container.executeJob("/changelog_fake_cdc_sink_paimon_case1_readiness_check.conf");
+                    Assertions.assertEquals(0, readinessCheck.getExitCode());
+                });

	+ ", only support LikeExpression with 'startWith' pattern ");
	+ ", only support LikeExpression pattern matching with prefix");

[Feature][Connector-V2] Support like predicate pushdown in paimon #9484

[Feature][Connector-V2] Support like predicate pushdown in paimon #9484

Uh oh!

Conversation

xiaochen-zhou commented Jun 20, 2025

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

Uh oh!

This comment was marked as outdated.

Uh oh!

Hisoka-X commented Jun 24, 2025

Uh oh!

Hisoka-X left a comment

Choose a reason for hiding this comment

Uh oh!

xiaochen-zhou commented Jun 24, 2025

Uh oh!

hawk9821 Jun 24, 2025

Choose a reason for hiding this comment

Uh oh!

Hisoka-X Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

Hisoka-X Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

Hisoka-X Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Jun 26, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jun 26, 2025

Choose a reason for hiding this comment

Uh oh!

Hisoka-X Jun 26, 2025

Choose a reason for hiding this comment

Uh oh!

xiaochen-zhou commented Jun 28, 2025

Uh oh!

Hisoka-X left a comment

Choose a reason for hiding this comment

Uh oh!

Hisoka-X commented Jul 2, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants