[cdc-base] Support Scan Newly Added Tables feature
#1838
Conversation
Jiabao-Sun
left a comment
Jiabao-Sun
left a comment
There was a problem hiding this comment.
Thanks @sap1ens for this contribution.
I left some comments.
...va/com/ververica/cdc/connectors/base/source/assigner/state/PendingSplitsStateSerializer.java
Show resolved
Hide resolved
...in/java/com/ververica/cdc/connectors/base/source/enumerator/IncrementalSourceEnumerator.java
Outdated
Show resolved
Hide resolved
...in/java/com/ververica/cdc/connectors/base/source/enumerator/IncrementalSourceEnumerator.java
Outdated
Show resolved
Hide resolved
...in/java/com/ververica/cdc/connectors/base/source/enumerator/IncrementalSourceEnumerator.java
Outdated
Show resolved
Hide resolved
...in/java/com/ververica/cdc/connectors/base/source/enumerator/IncrementalSourceEnumerator.java
Outdated
Show resolved
Hide resolved
...c-base/src/main/java/com/ververica/cdc/connectors/base/source/reader/external/FetchTask.java
Outdated
Show resolved
Hide resolved
| @Override | ||
| public boolean isScanNewlyAddedTableEnabled() { | ||
| return scanNewlyAddedTableEnabled; | ||
| } | ||
|
|
There was a problem hiding this comment.
Are we missing the scan.newly-added-table.enabled config option for table api?
There was a problem hiding this comment.
Hmm, I don't see the Table API implementation in the flink-cdc-base module... Am I missing something?
There was a problem hiding this comment.
Maybe we need to pass the scan.newly-added-table.enabled option via OracleTableSourceFactory and MongoDBTableSourceFactory.
There was a problem hiding this comment.
Well, I think actually enabling the option to be usable in connectors implementing the Incremental Snapshotting framework should be in a different pull request. Partially because of the scope of the change, but also because it might require connector-specific changes. And it definitely requires a lot of connector-specific testing.
Can you share your branch? |
sap1ens
force-pushed
the
cdc-base-scanNewlyAddedTable
branch
from
89a190a to
3f871ad
Compare
|
Closing this PR since this feature has been implemented in #3024. |
4 participants
Resolves #1152
Adding support for the
Scan Newly Added Tablesfeature by bringing changes from the existing MySQL connector.I'll have a follow-up PR to add support for Postgres once #1823 and this PR are merged. I've tested this extensively for the Postgres connector.
NOTE: this PR only adds common functionality. Adding support per connector will require separate PRs.