Skip to content

Fixing double-completing of destination servers in busy map #9629

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
sfc-gh-jslocum merged 2 commits into from
Mar 9, 2023
Merged

Fixing double-completing of destination servers in busy map #9629

sfc-gh-jslocum merged 2 commits into from
Mar 9, 2023

Conversation

@sfc-gh-jslocum
Copy link
Collaborator

@sfc-gh-jslocum sfc-gh-jslocum commented Mar 8, 2023 •
edited
Loading

Found by 71.2.7 release correctness tests, and appears to be a rare latent bug

Correctness in progress

Code-Reviewer Section

The general pull request guidelines can be found here.

Please check each of the following things and check all boxes before accepting a PR.

  • The PR has a description, explaining both the problem and the solution.
  • The description mentions which forms of testing were done and the testing seems reasonable.
  • Every function/class/actor that was touched is reasonably well documented.

For Release-Branches

If this PR is made against a release-branch, please also check the following:

  • This change/bugfix is a cherry-pick from the next younger branch (younger release-branch or main if this is the youngest branch)
  • There is a good reason why this PR needs to go into a release branch and this reason is documented (either in the description above or in a linked GitHub issue)

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Mar 8, 2023

Result of foundationdb-pr-clang-ide on Linux CentOS 7

  • Commit ID: 7970ffd
  • Duration 0:14:41
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Mar 8, 2023

Result of foundationdb-pr-clang on Linux CentOS 7

  • Commit ID: 7970ffd
  • Duration 0:36:50
  • Result: ❌ FAILED
  • Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Mar 8, 2023

Result of foundationdb-pr on Linux CentOS 7

  • Commit ID: 7970ffd
  • Duration 0:53:52
  • Result: ❌ FAILED
  • Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Mar 9, 2023

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

  • Commit ID: 7970ffd
  • Duration 1:23:29
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

@fdb-windows-ci
Copy link
Collaborator

fdb-windows-ci commented Mar 9, 2023

Doxense CI Report for Windows 10

@sfc-gh-xwang
Copy link
Collaborator

sfc-gh-xwang commented Mar 9, 2023

It's interesting we have a chance

CODE_PROBE(true, "move to removed server", probe::decoration::rare);

is hit, it has been 0 hit for a long time. Maybe it's the reason this double remove never caught in the past

@sfc-gh-jslocum
Copy link
Collaborator Author

sfc-gh-jslocum commented Mar 9, 2023

It's interesting we have a chance

CODE_PROBE(true, "move to removed server", probe::decoration::rare);

is hit, it has been 0 hit for a long time. Maybe it's the reason this double remove never caught in the past

Yeah. Based on the behavior it also seems likely that hitting this code block doesn't always trigger this double counting. I'm assuming I had added this extra completeDest() because i found it leaked slots in testing when i originally added the feature.

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Mar 9, 2023

Result of foundationdb-pr-clang-ide on Linux CentOS 7

  • Commit ID: 545bdbe
  • Duration 0:20:29
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@fdb-windows-ci
Copy link
Collaborator

fdb-windows-ci commented Mar 9, 2023

Doxense CI Report for Windows 10

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Mar 9, 2023

Result of foundationdb-pr on Linux CentOS 7

  • Commit ID: 545bdbe
  • Duration 1:08:36
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

sfc-gh-jslocum added a commit to sfc-gh-jslocum/foundationdb that referenced this pull request Mar 9, 2023
@sfc-gh-jslocum
Fixing double-completing of destination servers in busy map (Cherry-P…
932b9cd 
…ick apple#9629 to snowflake/release-71.3)
@sfc-gh-jslocum sfc-gh-jslocum mentioned this pull request Mar 9, 2023
Merged
5 tasks
@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Mar 9, 2023

Result of foundationdb-pr-clang on Linux CentOS 7

  • Commit ID: 545bdbe
  • Duration 1:10:21
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@sfc-gh-jslocum sfc-gh-jslocum merged commit 0c718f7 into apple:main Mar 9, 2023
@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Mar 9, 2023

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

  • Commit ID: 545bdbe
  • Duration 1:22:42
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

sfc-gh-jslocum added a commit to sfc-gh-jslocum/foundationdb that referenced this pull request Mar 10, 2023
@sfc-gh-jslocum
Fixing double-completing of destination servers in busy map (Cherry-P…
32c5045 
…ick apple#9629 to snowflake/release-71.2)
@sfc-gh-jslocum sfc-gh-jslocum mentioned this pull request Mar 10, 2023
Merged
5 tasks
sfc-gh-akejriwal pushed a commit that referenced this pull request Mar 14, 2023
@sfc-gh-jslocum
Fixing double-completing of destination servers in busy map (Cherry-P…
9160a57 
…ick #9629 to snowflake/release-71.3) (#9641)
sfc-gh-jslocum added a commit that referenced this pull request Mar 16, 2023
@sfc-gh-jslocum
Fixing double-completing of destination servers in busy map (Cherry-P…
0493a0a 
…ick #9629 to snowflake/release-71.2) (#9652)
jzhou77 pushed a commit to jzhou77/foundationdb that referenced this pull request Mar 16, 2023
@sfc-gh-jslocum @jzhou77
Fixing double-completing of destination servers in busy map (apple#9629)
743b04b
@jzhou77 jzhou77 mentioned this pull request Mar 16, 2023
Merged
18 tasks
@jzhou77 jzhou77 mentioned this pull request Apr 5, 2023
Merged
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sfc-gh-xwang sfc-gh-xwang sfc-gh-xwang approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@sfc-gh-jslocum @foundationdb-ci @fdb-windows-ci @sfc-gh-xwang