Skip to content

[FLINK-35243][cdc] Supports more schema change event types & backfill pre-schema payload for sink #3296

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
yuxiqian wants to merge 9 commits into from
Closed

[FLINK-35243][cdc] Supports more schema change event types & backfill pre-schema payload for sink #3296

yuxiqian wants to merge 9 commits into from

Conversation

@yuxiqian
Copy link
Member

@yuxiqian yuxiqian commented May 6, 2024 •
edited
Loading

This PR closes FLINK-35243:

  • Supports AlterColumnCommentEvent, AlterTableCommentEvent, DropTableEvent, RenameTableEvent, and TruncateTableEvent in pipeline framework
  • Adds SchemaChangeEventWithPreSchema interface, allowing SchemaRegistry backfilling schema before change payload
  • Fixed MySQL pipeline source doesn't capture ALTER TABLE ... MODIFY COLUMN DDL
  • Adds corresponding test cases

This requires #3339 being merged first.

@yuxiqian yuxiqian force-pushed the FLINK-35243 branch 3 times, most recently from 7ed8f0a to e8ce8ed Compare May 8, 2024 09:39
@yuxiqian
Copy link
Member Author

yuxiqian commented May 8, 2024

@PatrickRen @ruanhang1993 PTAL

@yuxiqian yuxiqian force-pushed the FLINK-35243 branch 2 times, most recently from f22c3a4 to 1e8ea73 Compare May 9, 2024 06:11
@melin
Copy link

melin commented May 10, 2024

DropTableEvent TruncateTableEvent can be closed. Sometimes, tables cannot be deleted synchronously.

@yuxiqian
Copy link
Member Author

yuxiqian commented May 10, 2024 •
edited
Loading

DropTableEvent TruncateTableEvent can be closed. Sometimes, tables cannot be deleted synchronously.

Hi @melin, thanks for your comments. It's a common use case to apply some "safe" schema change events (like AddColumnEvent) and ignore "dangerous" events (like Truncate / Drop tables). So we're designing a new feature allowing users to configure schema change behaviour for each type of schema change events in FLINK-35242. Also, we're trying to expose interfaces allowing Sink connectors to report their supported kind of schema change events.

This PR is still in early draft stage, so feel free to leave your comments if you have any other concerns.

@yuxiqian yuxiqian marked this pull request as draft May 31, 2024 02:34
@github-actions github-actions bot added the cli label May 31, 2024
@yuxiqian yuxiqian changed the title [FLINK-35243][cdc] Support table schema change events & PreSchema backfill [FLINK-35243][cdc] Supports more schema change event types & backfill pre-schema payload for sink May 31, 2024
@melin
Copy link

melin commented Aug 2, 2024

support add/drop primary key event

@yuxiqian
Copy link
Member Author

yuxiqian commented Aug 2, 2024

Thanks for @melin's suggestion, will be working on this after #3339 got merged.

@yuxiqian
Copy link
Member Author

yuxiqian commented Aug 8, 2024

Seems it's not meaningful to rebase, since there are too many conflicts. Will open another PR to address this.

@yuxiqian yuxiqian closed this Aug 8, 2024
@yuxiqian yuxiqian mentioned this pull request Aug 8, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

cli common composer doris-pipeline-connector e2e-tests migration-tests mysql-pipeline-connector runtime starrocks-pipeline-connector values-pipeline-connector

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@yuxiqian @melin