Skip to content

Conversation

@klsince
Copy link
Contributor

@klsince klsince commented Oct 14, 2022

This PR is part of the work to support multi-datadir for Pinot server as tracked by #8843.

Following up on #9306, this one adds the controller side periodic task SegmentTierAssigner to calculate the target tier for segment and set it in SegmentZKMetadata, as goal state to guide servers to put segments into the right datadir (i.e. tier).

The next PR will connect the two parts, i.e. reusing SegmentReloadMessage to notify servers to do the migration.

Release Note

New configs to turn on tier assigner:

  1. controller.segmentTierAssigner.frequencyPeriod: -1 by default, disabled
  2. controller.segmentTierAssigner.initialDelayInSeconds: 120s-300s w/ jitter by default

Extended API to query segment tiers

e.g. curl -X GET "http://localhost:9000/segments/airlineStats_OFFLINE/tiers?type=OFFLINE" -H "accept: application/json"
{
  "tableName": "airlineStats_OFFLINE",
  "segmentTiers": {
    "airlineStats_OFFLINE_16071_16071_0": {
      "targetTier": "coldTier",
      "Server_192.168.0.104_7000": "coldTier"
    },
    "airlineStats_OFFLINE_16072_16072_0": {
      "targetTier": "coldTier",
      "Server_192.168.0.104_7000": "coldTier"
    },
...

@codecov-commenter
Copy link

codecov-commenter commented Oct 14, 2022

Codecov Report

Merging #9598 (4824266) into master (d396b97) will increase coverage by 44.04%.
The diff coverage is 58.57%.

@@              Coverage Diff              @@
##             master    #9598       +/-   ##
=============================================
+ Coverage     25.84%   69.89%   +44.04%     
- Complexity       44     4977     +4933     
=============================================
  Files          1926     1939       +13     
  Lines        103305   103725      +420     
  Branches      15688    15739       +51     
=============================================
+ Hits          26702    72497    +45795     
+ Misses        73903    26119    -47784     
- Partials       2700     5109     +2409     
Flag Coverage Δ
integration1 25.81% <15.71%> (-0.04%) ⬇️
integration2 24.51% <15.71%> (?)
unittests1 67.30% <ø> (?)
unittests2 15.60% <58.57%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...ler/helix/core/relocation/SegmentTierAssigner.java 46.66% <46.66%> (ø)
.../apache/pinot/controller/util/TableTierReader.java 88.46% <70.58%> (+88.46%) ⬆️
...apache/pinot/controller/BaseControllerStarter.java 82.42% <100.00%> (+6.50%) ⬆️
...va/org/apache/pinot/controller/ControllerConf.java 57.95% <100.00%> (+8.91%) ⬆️
...e/pinot/controller/util/ServerTableTierReader.java 85.29% <100.00%> (+85.29%) ⬆️
...data/manager/realtime/DefaultSegmentCommitter.java 0.00% <0.00%> (-80.00%) ⬇️
...er/api/resources/LLCSegmentCompletionHandlers.java 43.56% <0.00%> (-18.82%) ⬇️
...altime/ServerSegmentCompletionProtocolHandler.java 51.88% <0.00%> (-6.61%) ⬇️
...core/startree/operator/StarTreeFilterOperator.java 86.61% <0.00%> (-1.41%) ⬇️
...pache/pinot/core/query/utils/idset/EmptyIdSet.java 25.00% <0.00%> (ø)
... and 1402 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@klsince klsince force-pushed the segment_tier_assigner branch from 1f252e0 to 57a1203 Compare October 14, 2022 17:44
@Jackie-Jiang Jackie-Jiang added feature release-notes Referenced by PRs that need attention when compiling the next release notes labels Oct 14, 2022
@klsince klsince force-pushed the segment_tier_assigner branch 3 times, most recently from 1a900e0 to 55a9458 Compare October 15, 2022 05:35
Copy link
Contributor

@npawar npawar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please also add an example to the PR description, regarding the changes expected to the the API response

@klsince klsince force-pushed the segment_tier_assigner branch from 0b36a80 to 4824266 Compare October 18, 2022 00:45
Copy link
Contributor

@npawar npawar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm!

@npawar npawar merged commit fa95569 into apache:master Oct 18, 2022
@klsince klsince deleted the segment_tier_assigner branch October 18, 2022 17:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature release-notes Referenced by PRs that need attention when compiling the next release notes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants