Skip to content

Allow to configure multi-datadirs as instance configs and a Quickstart example about them #9705

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
npawar merged 3 commits into from
Dec 7, 2022
Merged

Allow to configure multi-datadirs as instance configs and a Quickstart example about them #9705

npawar merged 3 commits into from
Dec 7, 2022

Conversation

@klsince
Copy link
Contributor

@klsince klsince commented Nov 1, 2022 •
edited
Loading

This PR is part of the work to support multi-datadir for Pinot server as tracked by #8843.

This one makes it simpler and more consistent to configure the feature. And added a QuickStart example about how this works with the typical configs.

Release Note

One can config tier and its datadir as instance configs to be consistent across tables, e.g.

"pinot.server.instance.tierConfigs.0.tierName": "hotTier"
"pinot.server.instance.tierConfigs.0.dataDir: "/tmp/multidir_test/hotTier"
"pinot.server.instance.tierConfigs.1.tierName": "coldTier"
"pinot.server.instance.tierConfigs.1.dataDir": "/tmp/multidir_test/coldTier"

The datadir from instance configs can be overwritten by tierConfigs in TableConfig, e.g.

  "tierConfigs": [
    {
      "name": "hotTier",
      "segmentSelectorType": "time",
      "segmentAge": "3130d",
      "storageType": "pinot_server",
      "serverTag": "DefaultTenant_OFFLINE",
      "tierBackendProperties": {
          "dataDir": "/foo/bar/baz", <--- overwrite the instance config
      }
    },
    {
      "name": "coldTier",
      "segmentSelectorType": "time",
      "segmentAge": "3140d",
      "storageType": "pinot_server",
      "serverTag": "DefaultTenant_OFFLINE"
    }
  ]

But the instance configs are more recommended to keep datadirs consistent across tables within a cluster.

@klsince klsince force-pushed the config_multi_datadirs_instance_config branch from 74f46ba to 120e673 Compare November 1, 2022 21:49
@klsince klsince changed the title Quickstart example for multi-datadirs support on server Allow to configure multi-datadirs as instance configs and a Quickstart example about them Nov 1, 2022
@codecov-commenter
Copy link

codecov-commenter commented Nov 1, 2022 •
edited
Loading

Codecov Report

Merging #9705 (2ec5616) into master (173916e) will decrease coverage by 1.43%.
The diff coverage is 78.26%.

@@             Coverage Diff              @@
##             master    #9705      +/-   ##
============================================
- Coverage     70.46%   69.02%   -1.44%     
+ Complexity     5535     5064     -471     
============================================
  Files          1982     1982              
  Lines        106449   106496      +47     
  Branches      16131    16140       +9     
============================================
- Hits          75006    73512    -1494     
- Misses        26213    27833    +1620     
+ Partials       5230     5151      -79
Flag Coverage Δ
integration1 ?
integration2 24.65% <5.79%> (+0.16%) ⬆️
unittests1 67.96% <83.33%> (+0.01%) ⬆️
unittests2 15.81% <5.79%> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...roller/helix/core/relocation/SegmentRelocator.java 47.14% <0.00%> (-16.43%) ⬇️
...ent/local/data/manager/TableDataManagerConfig.java 36.20% <55.00%> (+9.89%) ⬆️
.../apache/pinot/controller/util/TableTierReader.java 84.21% <57.14%> (-4.26%) ⬇️
...che/pinot/common/utils/config/TierConfigUtils.java 74.54% <95.83%> (+4.09%) ⬆️
.../pinot/core/data/manager/BaseTableDataManager.java 74.86% <100.00%> (+0.28%) ⬆️
...indexsegment/immutable/ImmutableSegmentLoader.java 85.04% <100.00%> (+0.14%) ⬆️
.../local/loader/TierBasedSegmentDirectoryLoader.java 90.81% <100.00%> (+1.92%) ⬆️
...local/segment/index/loader/IndexLoadingConfig.java 81.56% <100.00%> (+0.21%) ⬆️
...ment/spi/loader/SegmentDirectoryLoaderContext.java 92.50% <100.00%> (+0.83%) ⬆️
...g/apache/pinot/server/api/resources/ErrorInfo.java 0.00% <0.00%> (-100.00%) ⬇️
... and 148 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@klsince klsince force-pushed the config_multi_datadirs_instance_config branch from 120e673 to b3c3e4d Compare December 6, 2022 05:19
npawar
npawar reviewed Dec 6, 2022
View reviewed changes
@klsince klsince force-pushed the config_multi_datadirs_instance_config branch 2 times, most recently from be2896d to 3bd6b06 Compare December 7, 2022 00:24
npawar
npawar approved these changes Dec 7, 2022
View reviewed changes
Copy link
Contributor

@npawar npawar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for the config change. lgtm

@klsince klsince force-pushed the config_multi_datadirs_instance_config branch from 3bd6b06 to 2ec5616 Compare December 7, 2022 04:38
@npawar npawar merged commit 8cb4cef into apache:master Dec 7, 2022
@klsince klsince deleted the config_multi_datadirs_instance_config branch December 7, 2022 20:22
@KKcorps KKcorps added feature documentation release-notes Referenced by PRs that need attention when compiling the next release notes labels Dec 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@npawar npawar npawar approved these changes

Assignees

No one assigned

Labels

documentation feature release-notes Referenced by PRs that need attention when compiling the next release notes

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@klsince @codecov-commenter @npawar @KKcorps