Bounded column value partition function plus multple column partitioning #1
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds following features -
Added a new partition function
BoundedColumnValuePartitionFunctionto be able to partition segments on column values such that partitionId still remains integer. You can configure different column values on which you want to partition segments in a newfunctionConfigproperty.Example Config -
PartitionId is generated based on position in
columnValues. PartitionId would1for Maths,2for English and so on.PartitionId
0is reserved for any other subject which are not present in given partitionConfig.The new config
functionConfigis also saved with column metadata in metadata.properties and in zookeeper segment metadata.Broker can also prune segments based on this partition function