Skip to content

Sharded RocksDB reliability improvements #9665

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
jzhou77 merged 1 commit into from
Mar 16, 2023
Merged

Sharded RocksDB reliability improvements #9665

jzhou77 merged 1 commit into from
Mar 16, 2023

Conversation

@xy-54321
Copy link
Contributor

@xy-54321 xy-54321 commented Mar 11, 2023

  • Add knob to inject delay to limit FetchKeys' write rate
  • Return server_overloaded rather than fail the storage server on timeouts
  • Add delay when opening multiple key value stores to avoid memory exhaustion
  • Add rocksdb rate limiter to control write rate
  • More metrics in storage server to track data move load

Code-Reviewer Section

The general pull request guidelines can be found here.

Please check each of the following things and check all boxes before accepting a PR.

  • The PR has a description, explaining both the problem and the solution.
  • The description mentions which forms of testing were done and the testing seems reasonable.
  • Every function/class/actor that was touched is reasonably well documented.

For Release-Branches

If this PR is made against a release-branch, please also check the following:

  • This change/bugfix is a cherry-pick from the next younger branch (younger release-branch or main if this is the youngest branch)
  • There is a good reason why this PR needs to go into a release branch and this reason is documented (either in the description above or in a linked GitHub issue)

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Mar 11, 2023

Result of foundationdb-pr-clang on Linux CentOS 7

  • Commit ID: 68133d5
  • Duration 0:21:58
  • Result: ❌ FAILED
  • Error: Error while executing command: ctest -j ${NPROC} --no-compress-output -T test --output-on-failure. Reason: exit status 8
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Mar 11, 2023

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

  • Commit ID: 68133d5
  • Duration 0:25:36
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Mar 11, 2023

Result of foundationdb-pr on Linux CentOS 7

  • Commit ID: 68133d5
  • Duration 0:29:11
  • Result: ❌ FAILED
  • Error: Error while executing command: ctest -j ${NPROC} --no-compress-output -T test --output-on-failure. Reason: exit status 8
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@fdb-windows-ci
Copy link
Collaborator

fdb-windows-ci commented Mar 11, 2023

Doxense CI Report for Windows 10

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Mar 11, 2023

Result of foundationdb-pr-macos on macOS Ventura 13.x

  • Commit ID: 68133d5
  • Duration 0:32:51
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Mar 11, 2023

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

  • Commit ID: 68133d5
  • Duration 1:21:31
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

jzhou77
jzhou77 requested changes Mar 13, 2023
View reviewed changes
fdbclient/ServerKnobs.cpp Outdated
init( STORAGE_DURABILITY_LAG_HARD_MAX, 2000e6 ); if( smallStorageTarget ) STORAGE_DURABILITY_LAG_HARD_MAX = 100e6;
init( STORAGE_DURABILITY_LAG_SOFT_MAX, 250e6 ); if( smallStorageTarget ) STORAGE_DURABILITY_LAG_SOFT_MAX = 10e6;
init( STORAGE_INCLUDE_FEED_STORAGE_QUEUE, true ); if ( randomize && BUGGIFY ) STORAGE_INCLUDE_FEED_STORAGE_QUEUE = false;
init (STORAGE_FETCH_KEYS_DELAY, 0.0); if ( randomize && BUGGIFY ) STORAGE_FETCH_KEYS_DELAY = deterministicRandom()->random01() * 100;
Copy link
Contributor

@jzhou77 jzhou77 Mar 13, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

change tab to space within this line.

@xy-54321 xy-54321 requested a review from jzhou77 March 14, 2023 18:53
@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Mar 14, 2023

Result of foundationdb-pr on Linux CentOS 7

  • Commit ID: 62cdfe8
  • Duration 0:30:06
  • Result: ❌ FAILED
  • Error: Error while executing command: ctest -j ${NPROC} --no-compress-output -T test --output-on-failure. Reason: exit status 8
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@fdb-windows-ci
Copy link
Collaborator

fdb-windows-ci commented Mar 14, 2023

Doxense CI Report for Windows 10

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Mar 14, 2023

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

  • Commit ID: 62cdfe8
  • Duration 1:29:26
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Mar 15, 2023

Result of foundationdb-pr-clang on Linux CentOS 7

  • Commit ID: 62cdfe8
  • Duration 3:00:32
  • Result: ❌ FAILED
  • Error: Error while executing command: ctest -j ${NPROC} --no-compress-output -T test --output-on-failure. Reason: exit status 8
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@jzhou77 jzhou77 closed this Mar 16, 2023
@jzhou77 jzhou77 reopened this Mar 16, 2023
@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Mar 16, 2023

Result of foundationdb-pr on Linux CentOS 7

  • Commit ID: 62cdfe8
  • Duration 0:30:06
  • Result: ❌ FAILED
  • Error: Error while executing command: ctest -j ${NPROC} --no-compress-output -T test --output-on-failure. Reason: exit status 8
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Mar 16, 2023

Result of foundationdb-pr-clang on Linux CentOS 7

  • Commit ID: 62cdfe8
  • Duration 1:00:57
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Mar 16, 2023

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

  • Commit ID: 62cdfe8
  • Duration 1:23:34
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

@jzhou77 jzhou77 merged commit 423197a into apple:release-7.2 Mar 16, 2023
xy-54321 pushed a commit to xy-54321/foundationdb that referenced this pull request May 4, 2023
Cherrypick apple#9665
72c383a
xy-54321 pushed a commit to xy-54321/foundationdb that referenced this pull request May 9, 2023
Cherrypick apple#9665
5fc7089
xy-54321 pushed a commit to xy-54321/foundationdb that referenced this pull request May 10, 2023
Cherrypick apple#9665
6c5b0ed
xy-54321 pushed a commit to xy-54321/foundationdb that referenced this pull request May 10, 2023
Cherrypick apple#9665
abd45c4
xy-54321 pushed a commit to xy-54321/foundationdb that referenced this pull request May 10, 2023
Cherrypick apple#9665
bc3dd06
xy-54321 pushed a commit to xy-54321/foundationdb that referenced this pull request May 10, 2023
Cherrypick apple#9665
8f0c593
xy-54321 pushed a commit to xy-54321/foundationdb that referenced this pull request May 11, 2023
Cherrypick apple#9665
4d3850f
xy-54321 added a commit that referenced this pull request May 11, 2023
@xy-54321
[release-7.3] Cherrypick Sharded RocksDB fixes. (#10203)
df370e3 
* Cherrypick #9665

* Cherrypick #9746

* Log background error and add knobs for memory tuning. (#9841)

* error logger

* recovery mode

* Range deletion memory usage improvements  (#10048)

* Fix build error.

* initialize counter

* Log physical shard size in KVS

* fix build error
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jzhou77 jzhou77 jzhou77 approved these changes

+1 more reviewer

@liquid-helium liquid-helium liquid-helium approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@xy-54321 @foundationdb-ci @fdb-windows-ci @jzhou77 @liquid-helium