Skip to content

Buggify GRV proxy to return errors and fix bugs found #7844

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
sfc-gh-mpilman merged 13 commits into from
Aug 15, 2022
Merged

Buggify GRV proxy to return errors and fix bugs found #7844

sfc-gh-mpilman merged 13 commits into from
Aug 15, 2022

Conversation

@jzhou77
Copy link
Contributor

@jzhou77 jzhou77 commented Aug 10, 2022 •
edited
Loading

GRV proxy may return proxy_memory_limit_exceeded and batch_transaction_throttled errors, which are not properly handled at many places.

100k 20220811-220634-jzhou-2e8c8744412b8112 passed.

Code-Reviewer Section

The general pull request guidelines can be found here.

Please check each of the following things and check all boxes before accepting a PR.

  • The PR has a description, explaining both the problem and the solution.
  • The description mentions which forms of testing were done and the testing seems reasonable.
  • Every function/class/actor that was touched is reasonably well documented.

For Release-Branches

If this PR is made against a release-branch, please also check the following:

  • This change/bugfix is a cherry-pick from the next younger branch (younger release-branch or main if this is the youngest branch)
  • There is a good reason why this PR needs to go into a release branch and this reason is documented (either in the description above or in a linked GitHub issue)

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Aug 10, 2022

Result of foundationdb-pr on Linux CentOS 7

  • Commit ID: c4c6aeb
  • Duration 0:47:52
  • Result: ❌ FAILED
  • Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
  • Build Logs (available for 30 days)

@fdb-windows-ci
Copy link
Collaborator

fdb-windows-ci commented Aug 10, 2022

Doxense CI Report for Windows 10

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Aug 10, 2022

Result of foundationdb-pr-macos on macOS BigSur 11.5.2

  • Commit ID: 737b964
  • Duration 0:43:31
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Logs (available for 30 days)

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Aug 10, 2022

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

  • Commit ID: c4c6aeb
  • Duration 1:54:50
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Logs (available for 30 days)

@fdb-windows-ci
Copy link
Collaborator

fdb-windows-ci commented Aug 10, 2022

Doxense CI Report for Windows 10

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Aug 10, 2022

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

  • Commit ID: 737b964
  • Duration 1:45:14
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Logs (available for 30 days)

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Aug 10, 2022

Result of foundationdb-pr on Linux CentOS 7

  • Commit ID: 737b964
  • Duration 2:15:24
  • Result: ❌ FAILED
  • Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
  • Build Logs (available for 30 days)

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Aug 10, 2022

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

  • Commit ID: c01c2f2
  • Duration 1:47:10
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Logs (available for 30 days)

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Aug 11, 2022

Result of foundationdb-pr on Linux CentOS 7

  • Commit ID: c01c2f2
  • Duration 2:16:37
  • Result: ❌ FAILED
  • Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
  • Build Logs (available for 30 days)

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Aug 11, 2022

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

  • Commit ID: 41e941c
  • Duration 2:20:01
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Logs (available for 30 days)

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Aug 11, 2022

Result of foundationdb-pr on Linux CentOS 7

  • Commit ID: 41e941c
  • Duration 2:20:41
  • Result: ❌ FAILED
  • Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
  • Build Logs (available for 30 days)

@fdb-windows-ci
Copy link
Collaborator

fdb-windows-ci commented Aug 11, 2022

Doxense CI Report for Windows 10

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Aug 11, 2022

Result of foundationdb-pr on Linux CentOS 7

  • Commit ID: 5a0a366
  • Duration 0:59:12
  • Result: ❌ FAILED
  • Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
  • Build Logs (available for 30 days)

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Aug 11, 2022

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

  • Commit ID: 5a0a366
  • Duration 1:40:35
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Logs (available for 30 days)

@fdb-windows-ci
Copy link
Collaborator

fdb-windows-ci commented Aug 11, 2022

Doxense CI Report for Windows 10

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Aug 11, 2022

Result of foundationdb-pr-macos on macOS BigSur 11.5.2

  • Commit ID: 4017a7a
  • Duration 0:45:12
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Logs (available for 30 days)

@jzhou77 jzhou77 marked this pull request as ready for review August 11, 2022 18:25
@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Aug 11, 2022

Result of foundationdb-pr on Linux CentOS 7

  • Commit ID: 4017a7a
  • Duration 0:52:34
  • Result: ❌ FAILED
  • Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
  • Build Logs (available for 30 days)

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Aug 11, 2022

Result of foundationdb-pr on Linux CentOS 7

  • Commit ID: f6f5f9e
  • Duration 0:10:42
  • Result: ❌ FAILED
  • Error: reference not found for primary source and source version f6f5f9eb2afb1072d0796294beea48d765999f7f
  • Build Logs (available for 30 days)

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Aug 11, 2022

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

  • Commit ID: f6f5f9e
  • Duration 0:11:00
  • Result: ❌ FAILED
  • Error: reference not found for primary source and source version f6f5f9eb2afb1072d0796294beea48d765999f7f
  • Build Logs (available for 30 days)

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Aug 11, 2022

Result of foundationdb-pr-macos on macOS BigSur 11.5.2

  • Commit ID: f6f5f9e
  • Duration 0:11:05
  • Result: ❌ FAILED
  • Error: reference not found for primary source and source version f6f5f9eb2afb1072d0796294beea48d765999f7f
  • Build Logs (available for 30 days)

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Aug 11, 2022

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

  • Commit ID: 4017a7a
  • Duration 1:42:09
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Logs (available for 30 days)

@fdb-windows-ci
Copy link
Collaborator

fdb-windows-ci commented Aug 11, 2022

Doxense CI Report for Windows 10

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Aug 11, 2022

Result of foundationdb-pr on Linux CentOS 7

  • Commit ID: ade54d4
  • Duration 1:18:02
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Logs (available for 30 days)

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Aug 11, 2022

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

  • Commit ID: ade54d4
  • Duration 1:39:31
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Logs (available for 30 days)

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Aug 12, 2022

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

  • Commit ID: 40afd38
  • Duration 1:45:17
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Logs (available for 30 days)

sfc-gh-mpilman
sfc-gh-mpilman previously approved these changes Aug 12, 2022
View reviewed changes
Copy link
Collaborator

@sfc-gh-mpilman sfc-gh-mpilman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing this

xis19
xis19 reviewed Aug 12, 2022
View reviewed changes
fdbserver/BackupWorker.actor.cpp
self->cx->ssVersionVectorCache.applyDelta(reply.ssVersionVectorDelta);
return reply.version;
try {
GetReadVersionRequest request(span.context,
Copy link
Collaborator

@xis19 xis19 Aug 12, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this request can be put outside the loop.

Copy link
Contributor Author

@jzhou77 jzhou77 Aug 12, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is intentional. Same request object in an actor can cause old response being replayed, i.e., not delivering new responses.

@fdb-windows-ci
Copy link
Collaborator

fdb-windows-ci commented Aug 12, 2022

Doxense CI Report for Windows 10

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Aug 12, 2022

Result of foundationdb-pr on Linux CentOS 7

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Aug 12, 2022

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

  • Commit ID: 8fb6d59
  • Duration 1:23:41
  • Result: ❌ FAILED
  • Error: Error in UPLOAD_ARTIFACTS phase: [fdb-kubernetes-tests: [report files not found in build]]
  • Build Logs (available for 30 days)
  • Build Artifact (available for 30 days)

@jzhou77
Increase waitForLowInFlight timeout
92cae85 
Seeing a CheckRelocation test failed with 300s timeout.
@fdb-windows-ci
Copy link
Collaborator

fdb-windows-ci commented Aug 12, 2022

Doxense CI Report for Windows 10

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Aug 12, 2022

Result of foundationdb-pr-macos on macOS BigSur 11.5.2

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Aug 12, 2022

Result of foundationdb-pr on Linux CentOS 7

@foundationdb-ci
Copy link
Contributor

foundationdb-ci commented Aug 12, 2022

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

  • Commit ID: 92cae85
  • Duration 1:28:22
  • Result: ❌ FAILED
  • Error: Error in UPLOAD_ARTIFACTS phase: [fdb-kubernetes-tests: [report files not found in build]]
  • Build Logs (available for 30 days)
  • Build Artifact (available for 30 days)

@sfc-gh-mpilman sfc-gh-mpilman merged commit eb16947 into apple:main Aug 15, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sfc-gh-mpilman sfc-gh-mpilman sfc-gh-mpilman approved these changes

@sfc-gh-abeamon sfc-gh-abeamon Awaiting requested review from sfc-gh-abeamon

@sfc-gh-clin sfc-gh-clin Awaiting requested review from sfc-gh-clin

@liquid-helium liquid-helium Awaiting requested review from liquid-helium

@xis19 xis19 Awaiting requested review from xis19

Assignees

@sfc-gh-mpilman sfc-gh-mpilman

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@jzhou77 @foundationdb-ci @fdb-windows-ci @xis19 @sfc-gh-mpilman