Skip to content

Add compaction related knobs. #9746

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
jzhou77 merged 1 commit into from
Mar 22, 2023
Merged

Add compaction related knobs. #9746

jzhou77 merged 1 commit into from
Mar 22, 2023

Conversation

@xy-54321
Copy link
Contributor

@xy-54321 xy-54321 commented Mar 17, 2023

Code-Reviewer Section

The general pull request guidelines can be found here.

Please check each of the following things and check all boxes before accepting a PR.

  • The PR has a description, explaining both the problem and the solution.
  • The description mentions which forms of testing were done and the testing seems reasonable.
  • Every function/class/actor that was touched is reasonably well documented.

For Release-Branches

If this PR is made against a release-branch, please also check the following:

  • This change/bugfix is a cherry-pick from the next younger branch (younger release-branch or main if this is the youngest branch)
  • There is a good reason why this PR needs to go into a release branch and this reason is documented (either in the description above or in a linked GitHub issue)

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Mar 17, 2023

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

  • Commit ID: 3ad6646
  • Duration 0:26:00
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Mar 17, 2023

Result of foundationdb-pr-macos on macOS Ventura 13.x

  • Commit ID: 3ad6646
  • Duration 0:32:41
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@fdb-windows-ci
Copy link
Collaborator

fdb-windows-ci commented Mar 17, 2023

Doxense CI Report for Windows 10

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Mar 17, 2023

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

  • Commit ID: 3ad6646
  • Duration 1:24:49
  • Result: ❌ FAILED
  • Error: Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Mar 17, 2023

Result of foundationdb-pr on Linux CentOS 7

  • Commit ID: 3ad6646
  • Duration 2:08:44
  • Result: ❌ FAILED
  • Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Mar 17, 2023

Result of foundationdb-pr-clang on Linux CentOS 7

  • Commit ID: 3ad6646
  • Duration 3:00:30
  • Result: ❌ FAILED
  • Error: Error while executing command: ctest -j ${NPROC} --no-compress-output -T test --output-on-failure. Reason: exit status 8
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Mar 20, 2023

Result of foundationdb-pr-clang on Linux CentOS 7

  • Commit ID: 9d61270
  • Duration 0:23:06
  • Result: ❌ FAILED
  • Error: Error while executing command: ctest -j ${NPROC} --no-compress-output -T test --output-on-failure. Reason: exit status 8
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Mar 20, 2023

Result of foundationdb-pr on Linux CentOS 7

  • Commit ID: 9d61270
  • Duration 0:27:50
  • Result: ❌ FAILED
  • Error: Error while executing command: ctest -j ${NPROC} --no-compress-output -T test --output-on-failure. Reason: exit status 8
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@fdb-windows-ci
Copy link
Collaborator

fdb-windows-ci commented Mar 20, 2023

Doxense CI Report for Windows 10

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Mar 20, 2023

Result of foundationdb-pr-clang on Linux CentOS 7

  • Commit ID: dc47bdf
  • Duration 0:18:55
  • Result: ❌ FAILED
  • Error: Error while executing command: ctest -j ${NPROC} --no-compress-output -T test --output-on-failure. Reason: exit status 8
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Mar 20, 2023

Result of foundationdb-pr on Linux CentOS 7

  • Commit ID: dc47bdf
  • Duration 0:28:46
  • Result: ❌ FAILED
  • Error: Error while executing command: ctest -j ${NPROC} --no-compress-output -T test --output-on-failure. Reason: exit status 8
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Mar 20, 2023

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

  • Commit ID: 9d61270
  • Duration 1:24:50
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

@fdb-windows-ci
Copy link
Collaborator

fdb-windows-ci commented Mar 20, 2023

Doxense CI Report for Windows 10

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Mar 20, 2023

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

  • Commit ID: dc47bdf
  • Duration 1:29:09
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

@xy-54321 xy-54321 force-pushed the compaction-7.2 branch 3 times, most recently from efd6f69 to 08db65a Compare March 21, 2023 17:27
@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Mar 21, 2023

Result of foundationdb-pr on Linux CentOS 7

  • Commit ID: efd6f69
  • Duration 0:08:40
  • Result: ❌ FAILED
  • Error: reference not found for primary source and source version efd6f69bf5b1b4bc7dec2549c6cf28ae47d8638b
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Mar 21, 2023

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

  • Commit ID: efd6f69
  • Duration 0:08:45
  • Result: ❌ FAILED
  • Error: reference not found for primary source and source version efd6f69bf5b1b4bc7dec2549c6cf28ae47d8638b
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Mar 21, 2023

Result of foundationdb-pr-clang on Linux CentOS 7

  • Commit ID: efd6f69
  • Duration 0:08:44
  • Result: ❌ FAILED
  • Error: reference not found for primary source and source version efd6f69bf5b1b4bc7dec2549c6cf28ae47d8638b
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

neethuhaneesha
neethuhaneesha reviewed Mar 21, 2023
View reviewed changes
fdbserver/KeyValueStoreShardedRocksDB.actor.cpp Outdated
continue;
}

// Do not compact clear range.
Copy link
Contributor

@neethuhaneesha neethuhaneesha Mar 21, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does this comment mean?

Copy link
Contributor Author

@xy-54321 xy-54321 Mar 21, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a TODO for myself:)

@fdb-windows-ci
Copy link
Collaborator

fdb-windows-ci commented Mar 21, 2023

Doxense CI Report for Windows 10

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Mar 21, 2023

Result of foundationdb-pr-clang on Linux CentOS 7

  • Commit ID: 08db65a
  • Duration 0:20:07
  • Result: ❌ FAILED
  • Error: Error while executing command: ctest -j ${NPROC} --no-compress-output -T test --output-on-failure. Reason: exit status 8
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Mar 21, 2023

Result of foundationdb-pr-clang on Linux CentOS 7

  • Commit ID: cd8d226
  • Duration 0:22:20
  • Result: ❌ FAILED
  • Error: Error while executing command: ctest -j ${NPROC} --no-compress-output -T test --output-on-failure. Reason: exit status 8
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Mar 21, 2023

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

  • Commit ID: cd8d226
  • Duration 0:25:54
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Mar 21, 2023

Result of foundationdb-pr on Linux CentOS 7

  • Commit ID: cd8d226
  • Duration 0:30:48
  • Result: ❌ FAILED
  • Error: Error while executing command: ctest -j ${NPROC} --no-compress-output -T test --output-on-failure. Reason: exit status 8
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Mar 21, 2023

Result of foundationdb-pr on Linux CentOS 7

  • Commit ID: 08db65a
  • Duration 0:30:08
  • Result: ❌ FAILED
  • Error: Error while executing command: ctest -j ${NPROC} --no-compress-output -T test --output-on-failure. Reason: exit status 8
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

liquid-helium
liquid-helium reviewed Mar 21, 2023
View reviewed changes
fdbserver/KeyValueStoreShardedRocksDB.actor.cpp
Copy link
Contributor

@liquid-helium liquid-helium Mar 21, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have some TraceEvent to print all the initial RocksDB options?

Copy link
Contributor Author

@xy-54321 xy-54321 Mar 21, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's part of the rocksdb log.

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Mar 21, 2023

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

  • Commit ID: cd8d226
  • Duration 1:21:53
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Mar 21, 2023

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

  • Commit ID: 08db65a
  • Duration 1:23:30
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Mar 21, 2023

Result of foundationdb-pr-clang on Linux CentOS 7

  • Commit ID: f2ddbee
  • Duration 0:20:55
  • Result: ❌ FAILED
  • Error: Error while executing command: ctest -j ${NPROC} --no-compress-output -T test --output-on-failure. Reason: exit status 8
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Mar 21, 2023

Result of foundationdb-pr on Linux CentOS 7

  • Commit ID: f2ddbee
  • Duration 0:31:20
  • Result: ❌ FAILED
  • Error: Error while executing command: ctest -j ${NPROC} --no-compress-output -T test --output-on-failure. Reason: exit status 8
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Mar 21, 2023

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

  • Commit ID: f2ddbee
  • Duration 1:23:26
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

@fdb-windows-ci
Copy link
Collaborator

fdb-windows-ci commented Mar 21, 2023

Doxense CI Report for Windows 10

@jzhou77 jzhou77 merged commit 0dda2ab into apple:release-7.2 Mar 22, 2023
xy-54321 pushed a commit to xy-54321/foundationdb that referenced this pull request May 4, 2023
Cherrypick apple#9746
1742cfe
xy-54321 pushed a commit to xy-54321/foundationdb that referenced this pull request May 9, 2023
Cherrypick apple#9746
ff6b96c
xy-54321 pushed a commit to xy-54321/foundationdb that referenced this pull request May 10, 2023
Cherrypick apple#9746
2eace5c
xy-54321 pushed a commit to xy-54321/foundationdb that referenced this pull request May 10, 2023
Cherrypick apple#9746
fa821c0
xy-54321 pushed a commit to xy-54321/foundationdb that referenced this pull request May 10, 2023
Cherrypick apple#9746
eccab1c
xy-54321 pushed a commit to xy-54321/foundationdb that referenced this pull request May 10, 2023
Cherrypick apple#9746
eb6498b
xy-54321 pushed a commit to xy-54321/foundationdb that referenced this pull request May 11, 2023
Cherrypick apple#9746
f4884a7
xy-54321 added a commit that referenced this pull request May 11, 2023
@xy-54321
[release-7.3] Cherrypick Sharded RocksDB fixes. (#10203)
df370e3 
* Cherrypick #9665

* Cherrypick #9746

* Log background error and add knobs for memory tuning. (#9841)

* error logger

* recovery mode

* Range deletion memory usage improvements  (#10048)

* Fix build error.

* initialize counter

* Log physical shard size in KVS

* fix build error
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@neethuhaneesha neethuhaneesha neethuhaneesha approved these changes

@jzhou77 jzhou77 jzhou77 approved these changes

+1 more reviewer

@liquid-helium liquid-helium liquid-helium approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@xy-54321 @foundationdb-ci @fdb-windows-ci @neethuhaneesha @jzhou77 @liquid-helium