-
Notifications
You must be signed in to change notification settings - Fork 2.2k
[Feature][Connector-V2]Jdbc chunk split add snapshotSplitColumn config #7794 #7840
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
# Conflicts: # plugin-mapping.properties
Hisoka-X
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please add test case.
… update test case
Test case have been added, take a look. |
Hisoka-X
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
cc @hailin0
...se/src/main/java/org/apache/seatunnel/connectors/cdc/base/dialect/JdbcDataSourceDialect.java
Outdated
Show resolved
Hide resolved
...eatunnel/connectors/cdc/base/source/enumerator/splitter/AbstractJdbcSourceChunkSplitter.java
Outdated
Show resolved
Hide resolved
...se/src/main/java/org/apache/seatunnel/connectors/cdc/base/dialect/JdbcDataSourceDialect.java
Outdated
Show resolved
Hide resolved
...eatunnel/connectors/cdc/base/source/enumerator/splitter/AbstractJdbcSourceChunkSplitter.java
Outdated
Show resolved
Hide resolved
…ain/java/org/apache/seatunnel/connectors/cdc/base/source/enumerator/splitter/AbstractJdbcSourceChunkSplitter.java Co-authored-by: Jia Fan <[email protected]>
…ain/java/org/apache/seatunnel/connectors/cdc/base/dialect/JdbcDataSourceDialect.java Co-authored-by: Jia Fan <[email protected]>
…ain/java/org/apache/seatunnel/connectors/cdc/base/dialect/JdbcDataSourceDialect.java Co-authored-by: Jia Fan <[email protected]>
...eatunnel/connectors/cdc/base/source/enumerator/splitter/AbstractJdbcSourceChunkSplitter.java
Outdated
Show resolved
Hide resolved
Hisoka-X
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @XenosK !
|
@Hisoka-X 这里2.3.12也是依旧的逻辑, 可能要提醒一下用户, 这边代码 如果table-names-config配置后 , 这个指定列的字段, 必须要是unique里面的 ; 特别是对于mysql8.0的表结构如下的情况 对于这个视图的结构 在mysql , 本次pr的代码设置他的table-names-config为id并不会生效, 他会走目前代码的逻辑, 会从unique key里面取spilt key;此时并不会取到用户设置的id字段 这里最好调整一下mysqlcdc的文档2.3.12 , 如果需要的话我可以提供一些边界情况的测试结果集附录在文档上 可以看一下这个patch |
|
核心逻辑是: 先看用户配置的 snapshotSplitColumn 但它会先判断:配置列是否属于 dialect.getUniqueKeys() 返回的 unique keys 先遍历 PrimaryKey 列(id) 这就解释了整个现象链: 我的 snapshotSplitColumn=id 被判定 “not unique key” id 是 BIGINT 以为“配置强制生效”,但实际上被静默忽略,只能从日志里看出来;并且再加上官方的mysqlcdc文档, 都不知道如何下手去调整(因为已经按照文档去设置了指定的分片键了) |
Purpose of this pull request
fix: #7794
Does this PR introduce any user-facing change?
How was this patch tested?
Check list
New License Guide
release-note.