[Feature][Sink] File support new format: maxwell_json,canal_json,debezium_json (#9278) #9336

dyp12 · 2025-05-19T08:27:17Z

Purpose of this pull request

[Feature][Sink] File support new format: maxwell_json,canal_json,debezium_json (close #9278)

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update plugin-mapping.properties and add new connector information in it
2. Update the pom file of seatunnel-dist
3. Add ci label in label-scope-conf
4. Add e2e testcase in seatunnel-e2e
5. Update connector plugin_config

…bezium_json Signed-off-by: dyp12 <[email protected]>

Signed-off-by: dyp12 <[email protected]>

Copilot

Pull Request Overview

This pull request adds support for three new JSON file formats—maxwell_json, canal_json, and debezium_json—by introducing new serialization schemas and write strategies, and updating the file format configuration.

Added overloaded constructors with Charset support in the serialization schema classes
Created new write strategy classes for maxwell, canal, and debezium JSON formats
Updated the FileFormat enum to include new file format options

Reviewed Changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
seatunnel-formats/seatunnel-format-json/src/main/java/org/apache/seatunnel/format/json/maxwell/MaxWellJsonSerializationSchema.java	Added an overloaded constructor accepting Charset for maxwell JSON serialization
seatunnel-formats/seatunnel-format-json/src/main/java/org/apache/seatunnel/format/json/debezium/DebeziumJsonSerializationSchema.java	Added an overloaded constructor accepting Charset for debezium JSON serialization
seatunnel-formats/seatunnel-format-json/src/main/java/org/apache/seatunnel/format/json/canal/CanalJsonSerializationSchema.java	Added an overloaded constructor accepting Charset for canal JSON serialization
seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/writer/MaxWellJsonWriteStrategy.java	New write strategy implementation for maxwell JSON format
seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/writer/DebeziumJsonWriteStrategy.java	New write strategy implementation for debezium JSON format
seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/writer/CanalJsonWriteStrategy.java	New write strategy implementation for canal JSON format
seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/config/FileFormat.java	Updated the file format enum to include new JSON file formats

Comments suppressed due to low confidence (3)

seatunnel-formats/seatunnel-format-json/src/main/java/org/apache/seatunnel/format/json/maxwell/MaxWellJsonSerializationSchema.java:35

[nitpick] Consider renaming the class to 'MaxwellJsonSerializationSchema' to maintain consistent naming with the file format 'maxwell_json'.

public class MaxWellJsonSerializationSchema implements SerializationSchema {

seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/config/FileFormat.java:155

Consider updating the exception message to 'file format debezium json type does not support reading' for improved clarity.

throw new UnsupportedOperationException("file format debezium json type not support read");

seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/config/FileFormat.java:168

Consider updating this exception message to 'file format maxwell json type does not support reading' for consistency and clarity.

throw new UnsupportedOperationException("file format maxwell json type not support read");

Copilot · 2025-05-20T01:25:04Z

...ile-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/config/FileFormat.java

+
+        @Override
+        public ReadStrategy getReadStrategy() {
+            throw new UnsupportedOperationException("file format canal json type not support read");


Consider updating the exception message to improve clarity and grammar, e.g., 'file format canal json type does not support reading'.

Suggested change

throw new UnsupportedOperationException("file format canal json type not support read");

throw new UnsupportedOperationException("File format 'canal_json' does not support reading.");

hailin0 · 2025-05-20T01:50:12Z

waiting for ci passed

Signed-off-by: dyp12 <[email protected]>

Copilot

Pull Request Overview

This pull request adds support for new file formats (maxwell_json, canal_json, debezium_json) to the File sink.

Introduces new constructors accepting Charset in the JSON serialization schemas.
Implements new write strategies for maxwell_json, canal_json, and debezium_json in the file connector.
Extends the FileFormat enum to include entries for the new JSON formats.

Reviewed Changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
seatunnel-formats/seatunnel-format-json/src/main/java/org/apache/seatunnel/format/json/maxwell/MaxWellJsonSerializationSchema.java	Added a new constructor to support Charset for maxwell_json serialization.
seatunnel-formats/seatunnel-format-json/src/main/java/org/apache/seatunnel/format/json/debezium/DebeziumJsonSerializationSchema.java	Added a new constructor to support Charset for debezium_json serialization.
seatunnel-formats/seatunnel-format-json/src/main/java/org/apache/seatunnel/format/json/canal/CanalJsonSerializationSchema.java	Added a new constructor to support Charset for canal_json serialization.
seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/writer/*.java	Implements new write strategies for maxwell_json, debezium_json, and canal_json formats with a similar approach to existing JSON strategies.
seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/config/FileFormat.java	Extended the enum to handle the new JSON formats with appropriate write strategies and read strategy exceptions.

Comments suppressed due to low confidence (1)

seatunnel-formats/seatunnel-format-json/src/main/java/org/apache/seatunnel/format/json/maxwell/MaxWellJsonSerializationSchema.java:53

[nitpick] The class and file name 'MaxWellJsonSerializationSchema' use a mixed casing ('MaxWell') which might be inconsistent with the naming of the supported format ('maxwell_json') in the enum. Consider reviewing the naming for consistency.

public MaxWellJsonSerializationSchema(SeaTunnelRowType rowType, Charset charset) {

dyp12 · 2025-05-20T07:45:06Z

waiting for ci passed

@hailin0 SqlServerSchemaChangeIT>AbstractSchemaChangeBaseIT.testMysqlCdcWithSchemaEvolutionCaseExactlyOnce:291->AbstractSchemaChangeBaseIT.assertSchemaEvolution:296 » ConditionTimeout
i need to setup something ? always fail because of timeout

liunaijie · 2025-05-21T03:34:16Z

waiting for ci passed

@hailin0 SqlServerSchemaChangeIT>AbstractSchemaChangeBaseIT.testMysqlCdcWithSchemaEvolutionCaseExactlyOnce:291->AbstractSchemaChangeBaseIT.assertSchemaEvolution:296 » ConditionTimeout i need to setup something ? always fail because of timeout

Just retry it, CI is unstable

Signed-off-by: dyp12 <[email protected]>

Hisoka-X

Please update all file series docs. Thanks

dyp12 · 2025-05-22T06:55:50Z

Please update all file series docs. Thanks

sorry，i will update it

Signed-off-by: dyp12 <[email protected]>

Hisoka-X

Please add test case, you can refer

seatunnel/seatunnel-connectors-v2/connector-file/connector-file-local/src/test/java/org/apache/seatunnel/connectors/seatunnel/file/local/LocalFileTest.java

Line 52 in fbf7872

public class LocalFileTest {

Signed-off-by: dyp12 <[email protected]>

Hisoka-X

LGTM if ci passes. Thanks @dyp12 !

corgy-w · 2025-05-26T02:57:36Z

Try ci again

dyp12 · 2025-05-27T02:24:51Z

Try ci again

i rea

Try ci again

Re-run failed jobs so many times, jdbc-connectors-it-part-7 always fail

Hisoka-X · 2025-05-27T03:23:19Z

Please rebase on dev. Already fixed in #9360

dyp12 · 2025-05-27T09:12:47Z

Please rebase on dev. Already fixed in #9360

ci passes,Thank you for your help

…zium_json (apache#9278) (apache#9336)

wubx · 2025-08-07T02:43:51Z

Thank you very much for your contribution.
I have test this pr:

update t01 set c1='debezium_json' where id=7;

debezium format

{"before":{"id":7,"c1":"test"},"after":null,"op":"d"}
{"before":null,"after":{"id":7,"c1":"debezium_json"},"op":"c"}

maxwell format
update t01 set c1='debezium_json10' where id=7;

{"data":{"id":7,"c1":"debezium_json"},"type":"DELETE"}
{"data":{"id":7,"c1":"debezium_json10"},"type":"INSERT"}

From the usage, a time field(ts_ms) is required in this format to indicate the generation time of the record. This makes it convenient to obtain the status of the last record.

Full debezium json , source section is also very useful. If the file contains sources that are useful for batch Change Data Capture (CDC) for multiple tables.

{
    "before": {
        "id": 111,
        "name": "scooter",
        "description": "Big 2-wheel scooter ",
        "weight": 5.18
    },
    "after": {
        "id": 111,
        "name": "scooter",
        "description": "Big 2-wheel scooter ",
        "weight": 5.17
    },
    "source": {
        "version": "1.1.1.Final",
        "connector": "mysql",
        "name": "dbserver1",
        "ts_ms": 1589362330000,
        "snapshot": "false",
        "db": "inventory",
        "table": "products",
        "server_id": 223344,
        "gtid": null,
        "file": "mysql-bin.000003",
        "pos": 2090,
        "row": 0,
        "thread": 2,
        "query": null
    },
    "op": "u",
    "ts_ms": 1589362330904,
    "transaction": null
}

It is recommended to add the "ts" field at least.If you have obtained the other information, it is recommended to keep it.

wubx · 2025-08-07T02:55:06Z

Create new issue to improve it.
#9675

dyp12 added 2 commits May 19, 2025 16:18

[Feature][Sink] S3File support new format: maxwell_json,canal_json,de…

444c4b4

…bezium_json Signed-off-by: dyp12 <[email protected]>

[Feature][Sink] S3File support new format: maxwell_json,canal_json,de…

dbec1bb

…bezium_json Signed-off-by: dyp12 <[email protected]>

github-actions bot added connectors-v2 format file labels May 19, 2025

dyp12 added 6 commits May 19, 2025 16:35

[Feature][Sink] S3File support new format: maxwell_json,canal_json,de…

f6f5545

…bezium_json Signed-off-by: dyp12 <[email protected]>

[Feature][Sink] S3File support new format: maxwell_json,canal_json,de…

a1ce07e

…bezium_json Signed-off-by: dyp12 <[email protected]>

[Feature][Sink] S3File support new format: maxwell_json,canal_json,de…

eca3e35

…bezium_json Signed-off-by: dyp12 <[email protected]>

[Feature][Sink] S3File support new format: maxwell_json,canal_json,de…

4d5a84e

…bezium_json Signed-off-by: dyp12 <[email protected]>

code format

1094bde

Signed-off-by: dyp12 <[email protected]>

code format

cce11da

Signed-off-by: dyp12 <[email protected]>

nielifeng requested a review from Copilot May 20, 2025 01:24

Copilot AI reviewed May 20, 2025

View reviewed changes

code format

3cc6e7c

Signed-off-by: dyp12 <[email protected]>

nielifeng requested a review from Copilot May 20, 2025 07:35

Copilot AI reviewed May 20, 2025

View reviewed changes

dyp12 added 2 commits May 21, 2025 19:38

re ci

ca5ae19

Signed-off-by: dyp12 <[email protected]>

re ci

d1cb280

Signed-off-by: dyp12 <[email protected]>

Hisoka-X reviewed May 22, 2025

View reviewed changes

update doc

f3708fc

Signed-off-by: dyp12 <[email protected]>

github-actions bot added the document label May 23, 2025

Hisoka-X reviewed May 23, 2025

View reviewed changes

add test case

a5aaa24

Signed-off-by: dyp12 <[email protected]>

Hisoka-X approved these changes May 23, 2025

View reviewed changes

github-actions bot added approved reviewed labels May 23, 2025

Merge branch 'apache:dev' into dev

a70ed71

Merge branch 'apache:dev' into dev

080985b

hailin0 approved these changes May 27, 2025

View reviewed changes

hailin0 merged commit a1bfbb2 into apache:dev May 27, 2025
4 checks passed

dybyte pushed a commit to dybyte/seatunnel that referenced this pull request Jul 23, 2025

[Feature][Sink] File support new format: maxwell_json,canal_json,debe…

3b2d4dc

…zium_json (apache#9278) (apache#9336)

dyp12 mentioned this pull request Sep 1, 2025

[Feature][Format] Improve maxwell_json,canal_json,debezium_json format support merge update_before and update_after #9805

Merged

3 tasks

	throw new UnsupportedOperationException("file format canal json type not support read");
	throw new UnsupportedOperationException("File format 'canal_json' does not support reading.");

[Feature][Sink] File support new format: maxwell_json,canal_json,debezium_json (#9278) #9336

[Feature][Sink] File support new format: maxwell_json,canal_json,debezium_json (#9278) #9336

Uh oh!

Conversation

dyp12 commented May 19, 2025 • edited by Hisoka-X Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI May 20, 2025

Choose a reason for hiding this comment

Uh oh!

hailin0 commented May 20, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

dyp12 commented May 20, 2025

Uh oh!

liunaijie commented May 21, 2025

Uh oh!

Hisoka-X left a comment

Choose a reason for hiding this comment

Uh oh!

dyp12 commented May 22, 2025

Uh oh!

Hisoka-X left a comment

Choose a reason for hiding this comment

Uh oh!

Hisoka-X left a comment

Choose a reason for hiding this comment

Uh oh!

corgy-w commented May 26, 2025

Uh oh!

dyp12 commented May 27, 2025

Uh oh!

Hisoka-X commented May 27, 2025

Uh oh!

dyp12 commented May 27, 2025

Uh oh!

Uh oh!

wubx commented Aug 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wubx commented Aug 7, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

dyp12 commented May 19, 2025 •

edited by Hisoka-X

Loading

wubx commented Aug 7, 2025 •

edited

Loading