[pipeline-connector] Ability to provide a schema replacement #2908
Conversation
WholeWorld-Timothy
force-pushed
the
full-database-matching-rules
branch
from
7c9ff13 to
6e21b89
Compare
WholeWorld-Timothy
changed the title
WholeWorld-Timothy
force-pushed
the
full-database-matching-rules
branch
from
6e21b89 to
e1e246e
Compare
|
PTAL @banmoy |
|
Hi, @WholeWorld-Timothy this idea looks good to me. |
|
The current mode is a support for modification mode, and I think it is a good idea to add a table name prefix or a table name suffix based on this placeholder, so I can try it out. |
WholeWorld-Timothy
force-pushed
the
full-database-matching-rules
branch
from
e1e246e to
2ca1150
Compare
WholeWorld-Timothy
changed the title
|
PTAL @lvyanquan |
| TableId.parse( | ||
| replaceBy.getNamespace(), | ||
| replaceBy.getSchemaName(), | ||
| replaceBy.getTableName().replace("<>", tableId.getTableName())); |
There was a problem hiding this comment.
We need to extract this literal into a static constant, and determine a value that will not be accidentally used in the table name.
There was a problem hiding this comment.
I changed a version and put the definition of the <> symbol into the configuration file, just like this form:
route:
- source-table: mrtdb.[\S]*
sink-table: mrtdb_sd.sd_s<>
replace-symbol: s<>
In this way, the user can define the replacement symbols by themselves. Is it any better?
There was a problem hiding this comment.
+1. Please add some tests for these scenes.
There was a problem hiding this comment.
Two methods, RouteFunctionTest#testSchemaChangeRouting And RouteFunctionTest#testTableNameChangeRouting were modified to test this scenes.
There was a problem hiding this comment.
To expose this parameter to the users, please add description to cdc-pipeline.md.
And add this configuration to testParsingFullDefinition.
By the way, fix this typo, thanks😊.
WholeWorld-Timothy
force-pushed
the
full-database-matching-rules
branch
from
2ca1150 to
fb23579
Compare
WholeWorld-Timothy
force-pushed
the
full-database-matching-rules
branch
from
fb23579 to
87f73cf
Compare
This comment was marked as duplicate.
This comment was marked as duplicate.
Sorry, something went wrong.
|
My local compilation and testing are correct, so the error in check may not be caused by the code? How can rerun these check ? I click rerun in github desktop doesn t work |
|
Thanks for your contribution, please rebase the master branch and I'll take a look on the CI failure later. Besides, there are some doc modifications which seems to be unnecessary, could you please revert them back? |
|
Hi @WholeWorld-Timothy, could you please rebase this PR with latest cc @whhe |
|
Closed by #3428 |
6 participants
In our usage scenario, the source schema of mysql and the target schema of starrocks are often different, but the table names and column names are consistent, and this pr provides this capability.
Maintain a route configuration as follows:
Represents the table below the test will be transferred to the test02, where the symbol <> the representative table name is unchanged, just change the schema. For example, the test.table would be replaced with the test02.table.
And maintain a route configuration as follows:
Represents the table below the test will be transferred to the test02, Replace the original table name with the <> symbol. For example, the test.table would be replaced with the test02.s02_table.
And the user can define the replacement symbols by themselves as follows:
Represents the table below the test will be transferred to the test02, Replace the original table name with the s<> symbol.