cherrypick multiple fixes [release-7.2] #9732

jzhou77 · 2023-03-16T23:26:11Z

cherrypick

20230317-225145-jzhou-91e21575ab43e825
20230317-012444-jzhou-73895584efd10958

Code-Reviewer Section

The general pull request guidelines can be found here.

Please check each of the following things and check all boxes before accepting a PR.

The PR has a description, explaining both the problem and the solution.
The description mentions which forms of testing were done and the testing seems reasonable.
Every function/class/actor that was touched is reasonably well documented.

For Release-Branches

If this PR is made against a release-branch, please also check the following:

This change/bugfix is a cherry-pick from the next younger branch (younger release-branch or main if this is the youngest branch)
There is a good reason why this PR needs to go into a release branch and this reason is documented (either in the description above or in a linked GitHub issue)

… we need to throw that error to terminate the SS. (apple#9712)

* fix KeyRangeRef::isCovered() * reproduce bug * more unit test added * fmt

… failed

In FastTriggeredWatchesWorkload, if the randomized new value for the given `setKey` happens to be the same as the current value, the following will hold - `first` is true, and - `getDuration` is 0, and - assertion `lastReadVersion - ver >= SERVER_KNOBS->MAX_VERSIONS_IN_FLIGHT || lastReadVersion - ver < SERVER_KNOBS->VERSIONS_PER_SECOND * (25 + getDuration)` will fail To fix this, change the assertion to ``` assert(first || lastReadVersion - ver >= SERVER_KNOBS->MAX_VERSIONS_IN_FLIGHT || lastReadVersion - ver < SERVER_KNOBS->VERSIONS_PER_SECOND * (25 + getDuration)); ``` Test plan: Apply to the fix on top of the commit reported by Joshua test. Rerun the command to make sure it passes.

This reverts commit c2939ba.

Change `StringRef()` to `""_sr`. Co-authored-by: A.J. Beamon <[email protected]>

The ruby bindings are not currently installable. We can reproduce this with a build from source: ``` foundationdb/tmp on  main [$?] via △ v3.25.2 zsh ❯ gem install ./bindings/ruby/fdb-7.3.0.gem ERROR: While executing gem ... (Gem::Package::PathError) installing into parent path /Users/andrew/projects/foundationdb/LICENSE of /Users/andrew/.rbenv/versions/3.2.1/lib/ruby/gems/3.2.0/gems/fdb-7.3.0 is not allowed ``` The problem is that the gemspec is interpolating the source directory into the `files` array - and while this allows the gem to build, it prevents it from actually being installed. To fix it, we borrow similar code from the python bindings to copy the necessary files and license into the build directory before building the gem. This allows the gem to be installed: ``` foundationdb/tmp on  main [!?] via △ v3.25.2 zsh ❯ gem install ./bindings/ruby/fdb-7.3.0.gem Successfully installed fdb-7.3.0 Parsing documentation for fdb-7.3.0 Installing ri documentation for fdb-7.3.0 Done installing documentation for fdb after 0 seconds 1 gem installed ```

While fixing the gem build, I noticed that the gem will not run on arm64 machines, like the new spiffy M1/M2 MacBooks. On a whim, I tried just removing that restriction... and it does seem to work: ``` foundationdb/bindings/ruby/lib on  ahayworth/fix-ruby-binding-build [?] via 💎 v3.2.1 took 2s zsh ❯ pry [1] pry(main)> require 'fdb' => true [2] pry(main)> FDB.api_version 720 LoadError: FoundationDB API only supported on x86_64 (not arm64) from /Users/andrew/.rbenv/versions/3.2.1/lib/ruby/gems/3.2.0/gems/fdb-7.3.0/lib/fdbimpl.rb:40:in `<module:FDBC>' [3] pry(main)> ```

foundationdb-ci · 2023-03-16T23:47:08Z

Result of foundationdb-pr-clang on Linux CentOS 7

Commit ID: 838e2d1
Duration 0:20:44
Result: ❌ FAILED
Error: Error while executing command: ctest -j ${NPROC} --no-compress-output -T test --output-on-failure. Reason: exit status 8
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2023-03-16T23:59:51Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: 838e2d1
Duration 0:33:27
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2023-03-17T00:07:08Z

Result of foundationdb-pr on Linux CentOS 7

Commit ID: 838e2d1
Duration 0:40:48
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

fdb-windows-ci · 2023-03-17T00:36:22Z

Doxense CI Report for Windows 10

Commit ID: 838e2d1
Result: ✔️ SUCCEEDED
Build Logs (available for 30 days)

foundationdb-ci · 2023-03-17T00:54:33Z

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

Commit ID: 838e2d1
Duration 1:28:09
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

fdb-windows-ci · 2023-03-17T01:42:45Z

Doxense CI Report for Windows 10

Commit ID: 79d0687
Result: ✔️ SUCCEEDED
Build Logs (available for 30 days)

foundationdb-ci · 2023-03-17T01:48:04Z

Result of foundationdb-pr-clang on Linux CentOS 7

Commit ID: 79d0687
Duration 0:21:40
Result: ❌ FAILED
Error: Error while executing command: ctest -j ${NPROC} --no-compress-output -T test --output-on-failure. Reason: exit status 8
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2023-03-17T01:52:29Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 79d0687
Duration 0:26:08
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2023-03-17T01:59:27Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: 79d0687
Duration 0:33:04
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2023-03-17T02:26:04Z

Result of foundationdb-pr on Linux CentOS 7

Commit ID: 79d0687
Duration 0:59:37
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2023-03-17T02:46:51Z

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

Commit ID: 79d0687
Duration 1:20:25
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

…ite, the simulator could kill too many machines

…elliteTLogReplicationFactor and satelliteTLogUsableDcs. Update trace event with more info about the updated policy.

foundationdb-ci · 2023-03-17T20:45:07Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 10962e1
Duration 0:25:28
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2023-03-17T20:48:52Z

Result of foundationdb-pr on Linux CentOS 7

Commit ID: 10962e1
Duration 0:29:10
Result: ❌ FAILED
Error: Error while executing command: ctest -j ${NPROC} --no-compress-output -T test --output-on-failure. Reason: exit status 8
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

flowguru

#9744 seems not merged yet, do we want to wait for it to finalize?

foundationdb-ci · 2023-03-17T20:53:03Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: 10962e1
Duration 0:33:23
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2023-03-17T20:56:11Z

Result of foundationdb-pr-clang on Linux CentOS 7

Commit ID: 10962e1
Duration 0:36:28
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

jzhou77 · 2023-03-17T21:02:56Z

#9744 seems not merged yet, do we want to wait for it to finalize?

Yes.

fdb-windows-ci · 2023-03-17T21:30:24Z

Doxense CI Report for Windows 10

Commit ID: 10962e1
Result: ✔️ SUCCEEDED
Build Logs (available for 30 days)

foundationdb-ci · 2023-03-17T21:40:00Z

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

Commit ID: 10962e1
Duration 1:20:16
Result: ❌ FAILED
Error: Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

… proxies when getting key server locations

foundationdb-ci · 2023-03-17T23:04:18Z

Result of foundationdb-pr-clang on Linux CentOS 7

Commit ID: 7427645
Duration 0:20:52
Result: ❌ FAILED
Error: Error while executing command: ctest -j ${NPROC} --no-compress-output -T test --output-on-failure. Reason: exit status 8
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

fdb-windows-ci · 2023-03-17T23:06:01Z

Doxense CI Report for Windows 10

Commit ID: 7427645
Result: ✔️ SUCCEEDED
Build Logs (available for 30 days)

foundationdb-ci · 2023-03-17T23:09:35Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 7427645
Duration 0:26:12
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2023-03-17T23:16:41Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: 7427645
Duration 0:33:17
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2023-03-18T00:06:33Z

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

Commit ID: 7427645
Duration 1:22:50
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2023-03-18T01:44:00Z

Result of foundationdb-pr on Linux CentOS 7

Commit ID: 7427645
Duration 3:00:36
Result: ❌ FAILED
Error: Error while executing command: ctest -j ${NPROC} --no-compress-output -T test --output-on-failure. Reason: exit status 8
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

jzhou77 · 2023-03-19T03:08:11Z

single_process_external_client_fdbcli_tests failure should be unrelated:

Traceback (most recent call last):
  File "/codebuild/output/src215849086/src/github.com/apple/foundationdb/tests/TestRunner/tmp_cluster.py", line 223, in <module>
    print(f.read())
  File "/codebuild/output/src215849086/src/github.com/apple/foundationdb/tests/TestRunner/tmp_cluster.py", line 59, in __exit__
    shutil.rmtree(self.tmp_dir)
  File "/opt/rh/rh-python38/root/usr/lib64/python3.8/shutil.py", line 718, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/opt/rh/rh-python38/root/usr/lib64/python3.8/shutil.py", line 655, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/opt/rh/rh-python38/root/usr/lib64/python3.8/shutil.py", line 675, in _rmtree_safe_fd
    onerror(os.unlink, fullname, sys.exc_info())
  File "/opt/rh/rh-python38/root/usr/lib64/python3.8/shutil.py", line 673, in _rmtree_safe_fd
    os.unlink(entry.name, dir_fd=topfd)
FileNotFoundError: [Errno 2] No such file or directory: 'fdbmonitor.lock'

sfc-gh-abeamon and others added 16 commits March 16, 2023 14:25

When a storage server fails to register (e.g. due to worker_removed),…

7ab24a7

… we need to throw that error to terminate the SS. (apple#9712)

fixing copy constructor error

def3a81

fix unit test failure because of implicit uint16_t conversion to int

39fe27e

Fix KeyRangeRef::isCovered() (apple#9675)

b781467

* fix KeyRangeRef::isCovered() * reproduce bug * more unit test added * fmt

The isHostname function could cause an unknown_error if regex parsing…

e8944a6

… failed

Fixing double-completing of destination servers in busy map (apple#9629)

743b04b

Fix: the consistency check did not properly report failed tests

152b3fb

Revert "Deflake test FastTriggeredWatches"

00c9e01

This reverts commit c2939ba.

address comments

8f87b0a

Update fdbserver/workloads/FastTriggeredWatches.actor.cpp

9a4b1a5

Change `StringRef()` to `""_sr`. Co-authored-by: A.J. Beamon <[email protected]>

Rename a variable

4606433

Bumps minimum cmake required version.

3948d88

Fix misplaced link in fdbcli documentation

3961bca

Fix unit test failure

79d0687

jzhou77 requested review from flowguru and halfprice March 17, 2023 17:29

jzhou77 requested a review from xis19 March 17, 2023 17:29

jzhou77 marked this pull request as ready for review March 17, 2023 17:33

sfc-gh-abeamon added 2 commits March 17, 2023 12:52

If one region is being shared between the remote primary and a satell…

909b06f

…ite, the simulator could kill too many machines

Fix check that excludes satellites from consideration to consider sat…

10962e1

…elliteTLogReplicationFactor and satelliteTLogUsableDcs. Update trace event with more info about the updated policy.

jzhou77 mentioned this pull request Mar 17, 2023

Injected disk error in AsyncFileChaos is not tracked and causing StorageServerDurabilityError #9747

Open

halfprice previously approved these changes Mar 17, 2023

View reviewed changes

flowguru previously approved these changes Mar 17, 2023

View reviewed changes

The consistency check should retry if it couldn't find all the commit…

7427645

… proxies when getting key server locations

jzhou77 dismissed stale reviews from flowguru and halfprice via 7427645 March 17, 2023 22:43

jzhou77 requested review from flowguru and halfprice March 18, 2023 02:41

halfprice approved these changes Mar 18, 2023

View reviewed changes

flowguru approved these changes Mar 19, 2023

View reviewed changes

jzhou77 merged commit 9d50781 into apple:release-7.2 Mar 19, 2023

cherrypick multiple fixes [release-7.2] #9732

cherrypick multiple fixes [release-7.2] #9732

Uh oh!

Conversation

jzhou77 commented Mar 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code-Reviewer Section

For Release-Branches

Uh oh!

foundationdb-ci commented Mar 16, 2023

Result of foundationdb-pr-clang on Linux CentOS 7

Uh oh!

foundationdb-ci commented Mar 16, 2023

Result of foundationdb-pr-macos on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Mar 17, 2023

Result of foundationdb-pr on Linux CentOS 7

Uh oh!

fdb-windows-ci commented Mar 17, 2023

Doxense CI Report for Windows 10

Uh oh!

foundationdb-ci commented Mar 17, 2023

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

Uh oh!

fdb-windows-ci commented Mar 17, 2023

Doxense CI Report for Windows 10

Uh oh!

foundationdb-ci commented Mar 17, 2023

Result of foundationdb-pr-clang on Linux CentOS 7

Uh oh!

foundationdb-ci commented Mar 17, 2023

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Mar 17, 2023

Result of foundationdb-pr-macos on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Mar 17, 2023

Result of foundationdb-pr on Linux CentOS 7

Uh oh!

foundationdb-ci commented Mar 17, 2023

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

Uh oh!

foundationdb-ci commented Mar 17, 2023

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Mar 17, 2023

Result of foundationdb-pr on Linux CentOS 7

Uh oh!

flowguru left a comment

Choose a reason for hiding this comment

Uh oh!

foundationdb-ci commented Mar 17, 2023

Result of foundationdb-pr-macos on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Mar 17, 2023

Result of foundationdb-pr-clang on Linux CentOS 7

Uh oh!

jzhou77 commented Mar 17, 2023

Uh oh!

fdb-windows-ci commented Mar 17, 2023

Doxense CI Report for Windows 10

Uh oh!

foundationdb-ci commented Mar 17, 2023

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

Uh oh!

foundationdb-ci commented Mar 17, 2023

Result of foundationdb-pr-clang on Linux CentOS 7

Uh oh!

fdb-windows-ci commented Mar 17, 2023

Doxense CI Report for Windows 10

Uh oh!

foundationdb-ci commented Mar 17, 2023

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Mar 17, 2023

Result of foundationdb-pr-macos on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Mar 18, 2023

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

Uh oh!

jzhou77 commented Mar 16, 2023 •

edited

Loading