[INLONG-8948][Sort] Add Redis connector on flink 1.15 #9836
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
XiaoYou201
changed the title
Contributor
|
@XiaoYou201 please add more details about this PR. |
Member
|
Please fix your uts and build |
EMsnap
reviewed
Mar 18, 2024
...-connectors/redis/src/main/java/org/apache/inlong/sort/redis/sink/RedisDynamicTableSink.java
Outdated
Show resolved
Hide resolved
aloyszhang
reviewed
Mar 18, 2024
...connectors/redis/src/main/java/org/apache/inlong/sort/redis/common/config/RedisDataType.java
Outdated
Show resolved
Hide resolved
aloyszhang
reviewed
Mar 18, 2024
inlong-sort/sort-flink/sort-flink-v1.15/sort-connectors/pom.xml
Outdated
Show resolved
Hide resolved
aloyszhang
reviewed
Mar 18, 2024
inlong-sort/sort-flink/sort-flink-v1.15/sort-connectors/redis/pom.xml
Outdated
Show resolved
Hide resolved
aloyszhang
reviewed
Mar 18, 2024
...edis/src/main/java/org/apache/inlong/sort/redis/common/handler/InlongJedisConfigHandler.java
Outdated
Show resolved
Hide resolved
aloyszhang
previously approved these changes
Mar 19, 2024
EMsnap
reviewed
Mar 19, 2024
...k-v1.15/sort-connectors/redis/src/test/java/org/apache/inlong/sort/redis/RedisTableTest.java
Outdated
Show resolved
Hide resolved
EMsnap
reviewed
Mar 19, 2024
...k-v1.15/sort-connectors/redis/src/test/java/org/apache/inlong/sort/redis/RedisTableTest.java
Show resolved
Hide resolved
...k-v1.15/sort-connectors/redis/src/test/java/org/apache/inlong/sort/redis/RedisTableTest.java
Outdated
Show resolved
Hide resolved
EMsnap
reviewed
Mar 19, 2024
Member
EMsnap
left a comment
EMsnap
left a comment
There was a problem hiding this comment.
should put declaration for copied codes under licenses/inlong-sort-connectors/LICENSE , thanks
EMsnap
reviewed
Mar 19, 2024
Member
EMsnap
left a comment
EMsnap
left a comment
There was a problem hiding this comment.
add dependency in inlong-sort/sort-core/pom.xml under flink v1.15 , and also add in src/main/assemblies/sort-connectors-v1.15.xml
… add in sort-connectors-v1.15.xml.
EMsnap
approved these changes
Mar 20, 2024
aloyszhang
approved these changes
Mar 20, 2024
dockerzhang
changed the title
dockerzhang
approved these changes
Mar 20, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Prepare a Pull Request
[INLONG-9835] [Feature][Sort] Add Redis connector on flink 1.15
Fixes [Feature][Sort] Add Redis connector on flink 1.15 #8948
Motivation
Due to the version of flink update to 1.15.4, the redis connector should be updated and tested.
Modifications
According to the official website(https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sourcessinks/) introduce.The Metadata, Planning, and Runtime please refer to official website.
Accomplish customize source and sink about the redis-connector of flink 1.15.4. This work includes three main parts as follow.
1. RedisDynamicTableFactory
This part mainly override two methods.
This method firtst validate the support commands and then return a RedisDynamicTableSource object.
This method firtst validate the support commands and then return a RedisDynamicTableSink object.
2. RedisDynamicTableSource And RedisDynamicTableSink
This class validate the config properties and pass those properties and RedisRowDataLookupFunction object into TableFunctionProvider.of() method to generate a LookupRuntimeProvider object which get data from redis servers.
This class validate the config properties and pass those properties and RedisSinkFunction object into SinkFunctionProvider.of() method to generate SinkRuntimeProvider object which sink data to redis servers.
3. RedisLookupFunction And RedisSinkFunction
This class has a method called eval(), the eval() include the logical that get data from redis servers.Now, it has four request types to get different data (GET, HGET, ZREVRANK, ZSCORE).Due to flexible interface design, The request type can be quickly expand.
This class has a method called flushInternal(), include the logical that sink data to redis servers.Now, it has three request types to sink different data (PLAIN, HASH, BITMAP).
Documentation