-
Notifications
You must be signed in to change notification settings - Fork 2.2k
Open
Labels
Description
Code of Conduct
- I agree to follow this project's Code of Conduct
Search before asking
- I had searched in the issues and found no similar issues.
Describe the proposal
This issue has two proposal:
- move all parameter that user can adjust to a single class, connector use
XXXSourceConfig/ XXXSourceOptions.
UpdateEnvCommandOptionsto describe the parameters we can adjust in configenvpart.
UpdateServerConfigOptionsto describe the parameters we can adjust aboutZeta Engineconfig (config in SeaTunnel.yaml).
The benefits:
- We can easily know what parameter can be adjusted, that can help us to tune the performance.
- When version release, we can easily know which parameter has been changed, which can break the user version upgrade, which is added in new version, So we can record this into version upgrade document (although we don't have it yet).
- refactor connector, implement
TableSourceFactory,TableSinkFactoryinterface and implementcreateSource,createSink,optionRulemethod.
In optionRule method, describe which parameter is necessary, which is optional, which are the conflicts with other parameter. (need action1 done)
Move the connector creation logic from Connector to ConnectorFactory.
The benefits:
- Know the relationship between parameters. Use the
optionRulecheck the config before create connector instance. - This can be used in SeaTunnelWeb.
Welcome to do this with me
Task list
| type | assign to | status | |
|---|---|---|---|
| ZetaConfig | ServerConfigOptions | @liunaijie | Done |
| EnvConfig | EnvCommonOptions | @liunaijie | Done |
| Connector | connector-activemq | @liunaijie | Done |
| Connector | connector-amazondynamodb | @liunaijie | Done |
| Connector | connector-amazonsqs | @liunaijie | Done |
| Connector | connector-assert | @liunaijie | Done |
| Connector | connector-cassandra | @liunaijie | Done |
| Connector | connector-clickhouse | @lyne7-sc | Done |
| Connector | connector-console | @liunaijie | Done |
| Connector | connector-datahub | @liunaijie | Done |
| Connector | connector-dingtalk | @liunaijie | Done |
| Connector | connector-doris | @liunaijie | Done |
| Connector | connector-druid | @akulabs8 | Done |
| Connector | connector-easysearch | @liunaijie | Done |
| Connector | connector-elasticsearch | @fcb-xiaobo | Done |
| Connector | connector-email | @xdu-chenrj | Done |
| Connector | connector-fake | @liunaijie | Done |
| Connector | connector-file-cos | @misi1987107 | Done |
| Connector | connector-file-ftp | @misi1987107 | Done |
| Connector | connector-file-hadoop | @misi1987107 | Done |
| Connector | connector-file-jindo-oss | @misi1987107 | Done |
| Connector | connector-file-local | @misi1987107 | Done |
| Connector | connector-file-obs | @misi1987107 | Done |
| Connector | connector-file-oss | @misi1987107 | Done |
| Connector | connector-file-s3 | @litiliu | Done |
| Connector | connector-file-sftp | @misi1987107 | Done |
| Connector | connector-google-firestore | @liunaijie | Done |
| Connector | connector-google-sheets | @liunaijie | Done |
| Connector | connector-hbase | @liunaijie | Done |
| Connector | connector-http-feishu | @liunaijie | Done |
| Connector | connector-http-github | @liunaijie | Done |
| Connector | connector-http-gitlab | @liunaijie | Done |
| Connector | connector-http-jira | @liunaijie | Done |
| Connector | connector-http-klaviyo | @liunaijie | Done |
| Connector | connector-http-lemlist | @liunaijie | Done |
| Connector | connector-http-myhours | @liunaijie | Done |
| Connector | connector-http-notion | @liunaijie | Done |
| Connector | connector-http-onesignal | @liunaijie | Done |
| Connector | connector-http-persistiq | @liunaijie | Done |
| Connector | connector-http-wechat | @liunaijie | Done |
| Connector | connector-hudi | @liunaijie | Done |
| Connector | connector-iceberg | @liunaijie | Done |
| Connector | connector-influxdb | @liunaijie | Done |
| Connector | connector-iotdb | @liunaijie | Done |
| Connector | connector-jdbc | @liunaijie | Done |
| Connector | connector-kafka | @wattt3 | Done |
| Connector | connector-kudu | @liunaijie | Done |
| Connector | connector-maxcompute | @liunaijie | Done |
| Connector | connector-milvus | @liunaijie | Done |
| Connector | connector-mongodb | @liunaijie | Done |
| Connector | connector-neo4j | @liunaijie | Done |
| Connector | connector-openmldb | @liunaijie | Done |
| Connector | connector-paimon | @liunaijie | Done |
| Connector | connector-prometheus | @liunaijie | Done |
| Connector | connector-pulsar | @liunaijie | Done |
| Connector | connector-qdrant | @liunaijie | Done |
| Connector | connector-rabbitmq | @liunaijie | Done |
| Connector | connector-redis | @akulabs8 | Done |
| Connector | connector-rocketmq | @liunaijie | Done |
| Connector | connector-selectdb-cloud | @liunaijie | Done |
| Connector | connector-sentry | @liunaijie | Done |
| Connector | connector-slack | @akulabs8 | Done |
| Connector | connector-sls | @liunaijie | Done |
| Connector | connector-socket | @liunaijie | Done |
| Connector | connector-starrocks | @fcb-xiaobo | Done |
| Connector | connector-tablestore | @liunaijie | Done |
| Connector | connector-tdengine | @liunaijie | Done |
| Connector | connector-typesense | @liunaijie | Done |
| Connector | connector-web3j | @akulabs8 | Done |
| Connector | connector-cdc-mongodb | @liunaijie | WIP |
| Connector | connector-cdc-mysql | @liunaijie | WIP |
| Connector | connector-cdc-opengauss | @liunaijie | WIP |
| Connector | connector-cdc-oracle | @liunaijie | WIP |
| Connector | connector-cdc-postgres | @liunaijie | WIP |
| Connector | connector-cdc-sqlserver | @liunaijie | WIP |
| Connector | connector-cdc-tidb | @liunaijie | WIP |
| Connector | connector-s3-redshift | @liunaijie | WIP |
| Connector | connector-hive | @liunaijie | WIP |
What should a Pull Request accomplish?
-
List and describe the connector parameters in the
Factory.optionRule()method.
While doing so, we can move all parameters into the XXXXOptions class to unify the code style and centralize the parameters into a single file. -
Move the connector creation logic into
Factory.createSource/createSink.
Implement thecreateSource/createSinkmethods in Factory and instantiate the corresponding connector.
Are you willing to submit PR?
- Yes I am willing to submit a PR!
hailin0, liugddx and akulabs8