Skip to content

Support Testing S3BlobStore backup in simulation #4179

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
sfc-gh-satherton merged 26 commits into from
Jan 26, 2021
Merged

Support Testing S3BlobStore backup in simulation #4179

sfc-gh-satherton merged 26 commits into from
Jan 26, 2021

Conversation

@sfc-gh-tclinkenbeard
Copy link
Collaborator

@sfc-gh-tclinkenbeard sfc-gh-tclinkenbeard commented Dec 25, 2020 •
edited
Loading

This PR is resolves part of #4178.

Changes in this PR:

  • Add a new SimExternalConnection class
  • Add a connectExternal method to INetworkConnections
  • Use SimExternalConnection to make S3BlobStore deterministic when used in simulation
  • Added test file for S3 backups

Style

  • All variable and function names make sense.
  • The code is properly formatted (consider running git clang-format).

Performance

  • All CPU-hot paths are well optimized.
  • The proper containers are used (for example std::vector vs VectorRef).
  • There are no new known SlowTask traces.

Testing

  • The code was sufficiently tested in simulation.
  • If there are new parameters or knobs, different values are tested in simulation.
  • ASSERT, ASSERT_WE_THINK, and TEST macros are added in appropriate places.
  • Unit tests were added for new algorithms and data structure that make sense to unit-test
  • If this is a bugfix: there is a test that can easily reproduce the bug.

@sfc-gh-tclinkenbeard
Remove BlobStore.actor.cpp
c63024c 
This file was replaced by S3BlobStore.actor.cpp
@sfc-gh-tclinkenbeard
Added SimExternalConnection (most methods not implemented yet)
fed1a03
@sfc-gh-tclinkenbeard
Implement SimExternalConnection::connect
9f22ae7
@sfc-gh-tclinkenbeard
Implemented SimExternalConnection::getPeerAddress
87282ed
@sfc-gh-tclinkenbeard
Implemented SimExternalConnection::write
1e7ea8e
@sfc-gh-tclinkenbeard
Implemented SimExternalConnection::read
16476ec
@sfc-gh-tclinkenbeard
Made SimExternalConnection reading and writing deterministic
8cb029b
@sfc-gh-tclinkenbeard
Implemented SimExternalConnection::onReadable
94d7c17
@sfc-gh-tclinkenbeard
Remove unnecessary copies from INetworkConnections interface
2018575
@sfc-gh-tclinkenbeard
Added fdbrpc/SimExternalClient unit test
555c3d9
@sfc-gh-tclinkenbeard
Miscellaneous improvements to SimExternalConnection unit test
917da45
@sfc-gh-tclinkenbeard
Add random delays to SimExternalConnection
d9b5309
@sfc-gh-tclinkenbeard
Reduce sleep in SimExternalConnection::write
1dfa2ac
@sfc-gh-tclinkenbeard
Use connectExernal in INetworkConnections::connect
a7202a5
@sfc-gh-tclinkenbeard
Remove invalid assertion from SimExternalConnection::read
0b45c96
@sfc-gh-tclinkenbeard
Update blob backup test files
1dc0343
@sfc-gh-tclinkenbeard
Fix Azure blob storage backup test
2a8f971
@sfc-gh-tclinkenbeard sfc-gh-tclinkenbeard marked this pull request as ready for review December 25, 2020 03:23
@sfc-gh-tclinkenbeard
Move SendBufferIterator out of network.h
913401a 
This class is only needed for implementation in Net2.actor.cpp and
SimExternalConnection.actor.cpp, so this class should not be included
everywhere network.h is included
@sfc-gh-tclinkenbeard
Use platform::getEnvironmentVar instead of std::getenv
abb7cd4
@sfc-gh-tclinkenbeard
Merge remote-tracking branch 'origin/master' into run-minio-joshua
61a29ec
@sfc-gh-tclinkenbeard
Copy link
Collaborator Author

sfc-gh-tclinkenbeard commented Jan 14, 2021

The Windows CMake error is:

LINK : fatal error LNK1104: cannot open file 'libboost_date_time-vc141-mt-s-x64-1_72.lib' [C:\Users\fdb\jenkins\foundationdb-ci.foundationdb.org\workspace\jobs\prb-cmake-windows\work\fdbserver\fdbserver.vcxproj]
Build step 'Execute Windows batch command' marked build as failure

I'm not sure how to proceed with debugging this without access to the Jenkins hosts.

@sfc-gh-satherton sfc-gh-satherton merged commit d718e7b into apple:master Jan 26, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sfc-gh-satherton sfc-gh-satherton sfc-gh-satherton approved these changes

Assignees

@sfc-gh-satherton sfc-gh-satherton

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@sfc-gh-tclinkenbeard @sfc-gh-satherton