Add (un)lockRange APIs #2

jzhou77 · 2020-07-15T04:08:14Z

TODOs

Verify when database is locked, can't lock/unlock ranges.
Mixed lock/unlock requests
Multiple lock/unlock ops in one request
Limit on total locked ranges, or the size of cache.
Add test workload, simulating leader election via locking.
Add API with timeout for locking.

High Level Ideas

Proxies keep an in memory cache of locked ranges, which are persisted in \xff key space
The cache has a version, similar to metadata version, that is updated for each modification to the cache
Each client GRV reply piggybacks the lock version, so client can request the lock cache from proxies if its local one becomes stale
Each client operation is compared against its lock cache, so read/write to locked ranges raises an error
Client can request to lock a range via a transaction, which updates the lock version to the commit version. To prevent concurrent modification to the lock version, the lock version key is added to the read conflict ranges.

Guarantees

After a range is locked, further access to the range is denied. This is because locking transaction updates lock version. Thus any transaction after this one will see the new lock version, and requests the new lock cache.
After a range is unlocked, further access to the range is allowed. This follows the same reasoning as above.
Locks are modified via a series transactions, where each one sees the result of previous one. This is because concurrent modifications to the lock cache is not allowed, and later transaction must see the lock version of previous one.

Caveats

If the client caches read version, it may still access locked ranges. To solve this problem, we could add lock version key to each transaction's read conflict range.
Locks are modified by trusted client. If the client is malicious, it can arbitrarily lock ranges to disrupt normal operations. The client side library must be correct in adding or removing locks.

TODOs

Add client side cache
Add RPC to retrieve snapshots of locks from commit proxies
Persist locks to \xff key space and recover from it by proxies, i.e., reconstruct RangeLockCache.
~~CommitProxy should check LockRequest transaction and fail it for lock conflict -- hook into post resolution processing~~
~~Expire old entries in \xff key space~~
Check transaction's access with local range lock cache
Pass cycle and lock range workload
Efficient implementation of RangeLockCache using PTree (and KeyRangeMap?).
~~Refactor metadata version and range lock version into a vector for future extensions~~

Mechanism:

Modify existing LockDatabase so that:

a bool lock becomes a KeyRangeMap of locks
locks has a generation ID
GRV response has the generation ID that client can invalidate its cache
lock request has to be last in a batch? Otherwise, lock checking is needed at server side

Client sends lock request in a transaction:

	tr->atomicOp(rangeLockKey, lockRequest.toValue(), MutationRef::LockRange);
	tr->atomicOp(rangeLockVersionKey, rangeLockVersionRequiredValue, MutationRef::SetVersionstampedValue);

The lock request is sent to a special system key rangeLockKey. If transaction commits, the special system key rangeLockVersionKey is updated with the commit version, which can then be used by client to check if local cache is stale. rangeLockVersionRequiredValue is 14 (=8 + 2 + 4) bytes 0x00 in the format of (Version, sub, size).

Each client transaction calls getReadVersion, which will get back a rangeLockVersion. Then the transaction makes sure the corresponding range locks are present so that each get/set can be checked on locks. This lock check can fail the transaction early. If the client doesn't have range locks, the client will try to get it from one of the commit proxies, which can reply with the locks or an error range_locks_invalid_version().

On the server side, mutations to rangeLockKey is considered a metadata mutation, thus propagated to all proxies. Each proxy will apply the mutation in its local RangeLockCache, along with the updates to rangeLockVersionKey.

~~Add RangeLockCache to resolvers (and check for each transaction?)~~ This doesn't work, because resolver doesn't know if the transaction is finally committed or not, so its cache is inaccurate. Only proxy's cache is right.
- ~~Pass cycle and lock range workload~~
- ~~Add a knob and disable resolver check~~
implement union_like_traits (which you can find in ObjectSerializerTraits.h) for enum class LockMode. Additionally you also need to generate a file identifier (which you can do by defining FileIdentifierFor)
This doesn't work, still generate compiling error "flow/serialize.h:108:4: error: member reference base type 'LockMode' is not a structure or union"

template <>
struct union_like_traits<LockMode> : std::true_type {
	using Member = LockMode;
	using alternatives = pack<LockMode>;
	template <class Context>
	static uint8_t index(const Member&, Context&) {
		return 0;
	}
	template <class Context>
	static bool empty(const Member&, Context&) {
		return false;
	}

	template <int i, class Context>
	static const index_t<i, alternatives>& get(const Member& member, Context&) {
		static_assert(i == 0, "LockMode only has one member");
		return member;
	}

	template <size_t i, class Alternative, class Context>
	static void assign(Member& member, const Alternative& a, Context&) {
		static_assert(i == 0, "LockMode only has one member");
		member = a;
	}
};

template <>
struct FileIdentifierFor<LockMode> {
	constexpr static FileIdentifier value = 1587434;
};

Review comments and integrate extended getRestoreSet API into fast restore

This follows how metadataVersionKey is implemented in PR apple#1213. Specifically, the GetReadVersionReply will return the range lock version that the client can use to check if its local cache is valid. This change only wires server side components. The next step is to create a workload that will set the range lock version and check the version.

The lock/unlock range operation is stored on txn state store, thus can't be used with other ops (mutations) on normal key ranges.

The range lock is intended for clients. Using it on server side can block transactions: each transaction needs to check if lock cache is stale; if so, it tries to get the cache from commit proxies, which can cause timeout errors, especially during recovery time. For instance, DD can be blocked. Since fdbserver doesn't mutate data, it is safe to ignore range locks.

During recovery, the range lock version should be "true-up" to the version stored in the rangeLockVersionKey. Otherwise, the default version 0 is setting version back. This resolves the problem that the client get a cache version that is less than the requested version.

This range_locks_access_denied causes transaction retry, so there is no harm. In contrast, a timeout error would be raised and can cause client to fail.

We'll need a full reset to trigger the refresh of rangeLock caches. When getting unknown_result and retrying, short-circuit unlock request as NOOP.

This lock owner UID is stored as part of the value for keys in "lockedKeyRanges" key space so that it can be used to tell if transaction committed when unknown_result is returned.

Merge Master

TSAN output: 3: WARNING: ThreadSanitizer: data race (pid=63) 3: Read of size 1 at 0x7b6800000970 by thread T1: 3: #0 N2::Net2::onMainThread(Promise<Void>&&, TaskPriority) /home/anoyes/workspace/foundationdb/flow/Net2.actor.cpp:1734:6 (libfdb_c.so+0xca3d9e) 3: #1 void onMainThreadVoid<ThreadSingleAssignmentVarBase::cancel()::'lambda'()>(ThreadSingleAssignmentVarBase::cancel()::'lambda'(), Error*, TaskPriority) /home/anoyes/workspace/foundationdb/flow/ThreadHelper.actor.h:47:13 (libfdb_c.so+0x3fe73a) 3: #2 ThreadSingleAssignmentVarBase::cancel() /home/anoyes/workspace/foundationdb/flow/ThreadHelper.actor.h:317:3 (libfdb_c.so+0x3fe290) 3: #3 fdb_future_cancel /home/anoyes/workspace/foundationdb/bindings/c/fdb_c.cpp:141:96 (libfdb_c.so+0x3f7896) 3: #4 DLThreadSingleAssignmentVar<long>::cancel() /home/anoyes/workspace/foundationdb/fdbclient/MultiVersionAssignmentVars.h:175:4 (libfdb_c.so+0x45bd4d) 3: #5 MapSingleAssignmentVar<long, Void>::cancel() /home/anoyes/workspace/foundationdb/fdbclient/MultiVersionAssignmentVars.h:241:20 (libfdb_c.so+0x475ba7) 3: #6 FlatMapSingleAssignmentVar<Void, Void>::cancel() /home/anoyes/workspace/foundationdb/fdbclient/MultiVersionAssignmentVars.h:301:27 (libfdb_c.so+0x4742ce) 3: #7 ThreadFuture<Void>::cancel() /home/anoyes/workspace/foundationdb/flow/ThreadHelper.actor.h:444:32 (libfdb_c.so+0x441237) 3: #8 MultiVersionDatabase::Connector::cancel() /home/anoyes/workspace/foundationdb/fdbclient/MultiVersionTransaction.actor.cpp:969:20 (libfdb_c.so+0x441237) 3: #9 MultiVersionDatabase::DatabaseState::cancelConnections()::$_22::operator()() const /home/anoyes/workspace/foundationdb/fdbclient/MultiVersionTransaction.actor.cpp:1082:11 (libfdb_c.so+0x441237) 3: #10 (anonymous namespace)::DoOnMainThreadVoidActorState<MultiVersionDatabase::DatabaseState::cancelConnections()::$_22, (anonymous namespace)::DoOnMainThreadVoidActor<MultiVersionDatabase::DatabaseState::cancelConnections()::$_22> >::a_body1cont1(Void const&, int) /home/anoyes/workspace/foundationdb/flow/ThreadHelper.actor.h:559:2 (libfdb_c.so+0x441237) 3: #11 (anonymous namespace)::DoOnMainThreadVoidActorState<MultiVersionDatabase::DatabaseState::cancelConnections()::$_22, (anonymous namespace)::DoOnMainThreadVoidActor<MultiVersionDatabase::DatabaseState::cancelConnections()::$_22> >::a_body1when1(Void const&, int) /home/anoyes/build/foundationdb-tsan/flow/ThreadHelper.actor.g.h:992:15 (libfdb_c.so+0x441237) 3: apple#12 (anonymous namespace)::DoOnMainThreadVoidActorState<MultiVersionDatabase::DatabaseState::cancelConnections()::$_22, (anonymous namespace)::DoOnMainThreadVoidActor<MultiVersionDatabase::DatabaseState::cancelConnections()::$_22> >::a_callback_fire(ActorCallback<(anonymous namespace)::DoOnMainThreadVoidActor<MultiVersionDatabase::DatabaseState::cancelConnections()::$_22>, 0, Void>*, Void const&) /home/anoyes/build/foundationdb-tsan/flow/ThreadHelper.actor.g.h:1013:4 (libfdb_c.so+0x440dff) 3: apple#13 ActorCallback<(anonymous namespace)::DoOnMainThreadVoidActor<MultiVersionDatabase::DatabaseState::cancelConnections()::$_22>, 0, Void>::fire(Void const&) /home/anoyes/workspace/foundationdb/flow/flow.h:1016:78 (libfdb_c.so+0x440dff) 3: apple#14 void SAV<Void>::send<Void>(Void&&) /home/anoyes/workspace/foundationdb/flow/flow.h:465:23 (libfdb_c.so+0x4086b2) 3: apple#15 void Promise<Void>::send<Void>(Void&&) const /home/anoyes/workspace/foundationdb/flow/flow.h:782:8 (libfdb_c.so+0xcc079e) 3: apple#16 N2::PromiseTask::operator()() /home/anoyes/workspace/foundationdb/flow/Net2.actor.cpp:1157:11 (libfdb_c.so+0xcc079e) 3: apple#17 N2::Net2::run() /home/anoyes/workspace/foundationdb/flow/Net2.actor.cpp:1493:5 (libfdb_c.so+0xca1640) 3: apple#18 runNetwork() /home/anoyes/workspace/foundationdb/fdbclient/NativeAPI.actor.cpp:1774:13 (libfdb_c.so+0x499eb4) 3: apple#19 ThreadSafeApi::runNetwork() /home/anoyes/workspace/foundationdb/fdbclient/ThreadSafeTransaction.cpp:429:3 (libfdb_c.so+0xb666cc) 3: apple#20 MultiVersionApi::runNetwork() /home/anoyes/workspace/foundationdb/fdbclient/MultiVersionTransaction.actor.cpp:1521:20 (libfdb_c.so+0x41e9dd) 3: apple#21 fdb_run_network /home/anoyes/workspace/foundationdb/bindings/c/fdb_c.cpp:129:45 (libfdb_c.so+0x3f7645) 3: apple#22 decltype(std::__1::forward<int (*)()>(fp)()) std::__1::__invoke<int (*)()>(int (*&&)()) /usr/local/bin/../include/c++/v1/type_traits:3539:173 (fdb_c_unit_tests+0x384856) 3: apple#23 void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, int (*)()>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, int (*)()>&, std::__1::__tuple_indices<>) /usr/local/bin/../include/c++/v1/thread:273:5 (fdb_c_unit_tests+0x384856) 3: apple#24 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, int (*)()> >(void*) /usr/local/bin/../include/c++/v1/thread:284:5 (fdb_c_unit_tests+0x384856) 3: 3: Previous write of size 1 at 0x7b6800000970 by thread T2: 3: #0 N2::Net2::stopImmediately() /home/anoyes/workspace/foundationdb/flow/Net2.actor.cpp:253:11 (libfdb_c.so+0xca936b) 3: #1 N2::Net2::stop()::'lambda'()::operator()() const /home/anoyes/workspace/foundationdb/flow/Net2.actor.cpp:171:36 (libfdb_c.so+0xca936b) 3: #2 (anonymous namespace)::DoOnMainThreadVoidActorState<N2::Net2::stop()::'lambda'(), (anonymous namespace)::DoOnMainThreadVoidActor<N2::Net2::stop()::'lambda'()> >::a_body1cont1(Void const&, int) /home/anoyes/workspace/foundationdb/flow/ThreadHelper.actor.h:559:2 (libfdb_c.so+0xca936b) 3: #3 (anonymous namespace)::DoOnMainThreadVoidActorState<N2::Net2::stop()::'lambda'(), (anonymous namespace)::DoOnMainThreadVoidActor<N2::Net2::stop()::'lambda'()> >::a_body1when1(Void const&, int) /home/anoyes/build/foundationdb-tsan/flow/ThreadHelper.actor.g.h:992:15 (libfdb_c.so+0xca936b) 3: #4 (anonymous namespace)::DoOnMainThreadVoidActorState<N2::Net2::stop()::'lambda'(), (anonymous namespace)::DoOnMainThreadVoidActor<N2::Net2::stop()::'lambda'()> >::a_callback_fire(ActorCallback<(anonymous namespace)::DoOnMainThreadVoidActor<N2::Net2::stop()::'lambda'()>, 0, Void>*, Void const&) /home/anoyes/build/foundationdb-tsan/flow/ThreadHelper.actor.g.h:1013:4 (libfdb_c.so+0xca9128) 3: #5 ActorCallback<(anonymous namespace)::DoOnMainThreadVoidActor<N2::Net2::stop()::'lambda'()>, 0, Void>::fire(Void const&) /home/anoyes/workspace/foundationdb/flow/flow.h:1016:78 (libfdb_c.so+0xca9128) 3: #6 void SAV<Void>::send<Void>(Void&&) /home/anoyes/workspace/foundationdb/flow/flow.h:465:23 (libfdb_c.so+0x4086b2) 3: #7 void Promise<Void>::send<Void>(Void&&) const /home/anoyes/workspace/foundationdb/flow/flow.h:782:8 (libfdb_c.so+0xcc079e) 3: #8 N2::PromiseTask::operator()() /home/anoyes/workspace/foundationdb/flow/Net2.actor.cpp:1157:11 (libfdb_c.so+0xcc079e) 3: #9 N2::Net2::run() /home/anoyes/workspace/foundationdb/flow/Net2.actor.cpp:1493:5 (libfdb_c.so+0xca1640) 3: #10 runNetwork() /home/anoyes/workspace/foundationdb/fdbclient/NativeAPI.actor.cpp:1774:13 (libfdb_c.so+0x499eb4) 3: #11 ThreadSafeApi::runNetwork() /home/anoyes/workspace/foundationdb/fdbclient/ThreadSafeTransaction.cpp:429:3 (libfdb_c.so+0xb666cc) 3: apple#12 MultiVersionApi::runNetwork() /home/anoyes/workspace/foundationdb/fdbclient/MultiVersionTransaction.actor.cpp:1521:20 (libfdb_c.so+0x41e9dd) 3: apple#13 fdb_run_network /home/anoyes/workspace/foundationdb/bindings/c/fdb_c.cpp:129:45 (libfdb_c.so+0x3f7645) 3: apple#14 DLApi::runNetwork() /home/anoyes/workspace/foundationdb/fdbclient/MultiVersionTransaction.actor.cpp:492:11 (libfdb_c.so+0x40dd18) 3: apple#15 runNetworkThread(void*) /home/anoyes/workspace/foundationdb/fdbclient/MultiVersionTransaction.actor.cpp:1495:30 (libfdb_c.so+0x41e895) 3: 3: Location is heap block of size 1304 at 0x7b6800000600 allocated by main thread: 3: #0 operator new(unsigned long) <null> (fdb_c_unit_tests+0x31694e) 3: #1 newNet2(TLSConfig const&, bool, bool) /home/anoyes/workspace/foundationdb/flow/Net2.actor.cpp:1969:16 (libfdb_c.so+0xca607d) 3: #2 setupNetwork(unsigned long, bool) /home/anoyes/workspace/foundationdb/fdbclient/NativeAPI.actor.cpp:1754:14 (libfdb_c.so+0x499c53) 3: #3 ThreadSafeApi::setupNetwork() /home/anoyes/workspace/foundationdb/fdbclient/ThreadSafeTransaction.cpp:423:2 (libfdb_c.so+0xb66675) 3: #4 MultiVersionApi::setupNetwork() /home/anoyes/workspace/foundationdb/fdbclient/MultiVersionTransaction.actor.cpp:1462:21 (libfdb_c.so+0x41dc42) 3: #5 fdb_setup_network_impl() /home/anoyes/workspace/foundationdb/bindings/c/fdb_c.cpp:116:45 (libfdb_c.so+0x3f7525) 3: #6 DLApi::setupNetwork() /home/anoyes/workspace/foundationdb/fdbclient/MultiVersionTransaction.actor.cpp:488:15 (libfdb_c.so+0x40dc88) 3: #7 MultiVersionApi::setupNetwork()::$_27::operator()(Reference<ClientInfo>) const /home/anoyes/workspace/foundationdb/fdbclient/MultiVersionTransaction.actor.cpp:1481:17 (libfdb_c.so+0x442cd4) 3: #8 decltype(std::__1::forward<MultiVersionApi::setupNetwork()::$_27&>(fp)(std::__1::forward<Reference<ClientInfo> >(fp0))) std::__1::__invoke<MultiVersionApi::setupNetwork()::$_27&, Reference<ClientInfo> >(MultiVersionApi::setupNetwork()::$_27&, Reference<ClientInfo>&&) /usr/local/bin/../include/c++/v1/type_traits:3539:173 (libfdb_c.so+0x442cd4) 3: #9 void std::__1::__invoke_void_return_wrapper<void>::__call<MultiVersionApi::setupNetwork()::$_27&, Reference<ClientInfo> >(MultiVersionApi::setupNetwork()::$_27&, Reference<ClientInfo>&&) /usr/local/bin/../include/c++/v1/__functional_base:348:9 (libfdb_c.so+0x442cd4) 3: #10 std::__1::__function::__alloc_func<MultiVersionApi::setupNetwork()::$_27, std::__1::allocator<MultiVersionApi::setupNetwork()::$_27>, void (Reference<ClientInfo>)>::operator()(Reference<ClientInfo>&&) /usr/local/bin/../include/c++/v1/functional:1540:16 (libfdb_c.so+0x442cd4) 3: #11 std::__1::__function::__func<MultiVersionApi::setupNetwork()::$_27, std::__1::allocator<MultiVersionApi::setupNetwork()::$_27>, void (Reference<ClientInfo>)>::operator()(Reference<ClientInfo>&&) /usr/local/bin/../include/c++/v1/functional:1714:12 (libfdb_c.so+0x442cd4) 3: apple#12 std::__1::__function::__value_func<void (Reference<ClientInfo>)>::operator()(Reference<ClientInfo>&&) const /usr/local/bin/../include/c++/v1/functional:1867:16 (libfdb_c.so+0x4153ed) 3: apple#13 std::__1::function<void (Reference<ClientInfo>)>::operator()(Reference<ClientInfo>) const /usr/local/bin/../include/c++/v1/functional:2473:12 (libfdb_c.so+0x4153ed) 3: apple#14 MultiVersionApi::runOnExternalClients(int, std::__1::function<void (Reference<ClientInfo>)>, bool) /home/anoyes/workspace/foundationdb/fdbclient/MultiVersionTransaction.actor.cpp:1117:5 (libfdb_c.so+0x4153ed) 3: apple#15 MultiVersionApi::runOnExternalClientsAllThreads(std::__1::function<void (Reference<ClientInfo>)>, bool) /home/anoyes/workspace/foundationdb/fdbclient/MultiVersionTransaction.actor.cpp:1102:3 (libfdb_c.so+0x41992c) 3: apple#16 MultiVersionApi::setupNetwork() /home/anoyes/workspace/foundationdb/fdbclient/MultiVersionTransaction.actor.cpp:1475:3 (libfdb_c.so+0x41dd2a) 3: apple#17 fdb_setup_network_impl() /home/anoyes/workspace/foundationdb/bindings/c/fdb_c.cpp:116:45 (libfdb_c.so+0x3f7525) 3: apple#18 main /home/anoyes/workspace/foundationdb/bindings/c/test/unit/unit_tests.cpp:2147:12 (fdb_c_unit_tests+0x35f6b2) 3: 3: Thread T1 (tid=65, running) created by main thread at: 3: #0 pthread_create <null> (fdb_c_unit_tests+0x29df32) 3: #1 std::__1::__libcpp_thread_create(unsigned long*, void* (*)(void*), void*) /usr/local/bin/../include/c++/v1/__threading_support:394:10 (fdb_c_unit_tests+0x372da6) 3: #2 std::__1::thread::thread<int (*)(), void>(int (*&&)()) /usr/local/bin/../include/c++/v1/thread:300:16 (fdb_c_unit_tests+0x372da6) 3: #3 main /home/anoyes/workspace/foundationdb/bindings/c/test/unit/unit_tests.cpp:2148:14 (fdb_c_unit_tests+0x35f6dd) 3: 3: Thread T2 (tid=66, finished) created by thread T1 at: 3: #0 pthread_create <null> (fdb_c_unit_tests+0x29df32) 3: #1 startThread(void* (*)(void*), void*, int, char const*) /home/anoyes/workspace/foundationdb/flow/Platform.actor.cpp:2745:2 (libfdb_c.so+0xd08d74) 3: #2 N2::Net2::startThread(void* (*)(void*), void*) /home/anoyes/workspace/foundationdb/flow/Net2.actor.cpp:1749:9 (libfdb_c.so+0xca42a4) 3: #3 MultiVersionApi::runNetwork()::$_28::operator()(Reference<ClientInfo>) const /home/anoyes/workspace/foundationdb/fdbclient/MultiVersionTransaction.actor.cpp:1516:34 (libfdb_c.so+0x443106) 3: #4 decltype(std::__1::forward<MultiVersionApi::runNetwork()::$_28&>(fp)(std::__1::forward<Reference<ClientInfo> >(fp0))) std::__1::__invoke<MultiVersionApi::runNetwork()::$_28&, Reference<ClientInfo> >(MultiVersionApi::runNetwork()::$_28&, Reference<ClientInfo>&&) /usr/local/bin/../include/c++/v1/type_traits:3539:173 (libfdb_c.so+0x443106) 3: #5 void std::__1::__invoke_void_return_wrapper<void>::__call<MultiVersionApi::runNetwork()::$_28&, Reference<ClientInfo> >(MultiVersionApi::runNetwork()::$_28&, Reference<ClientInfo>&&) /usr/local/bin/../include/c++/v1/__functional_base:348:9 (libfdb_c.so+0x443106) 3: #6 std::__1::__function::__alloc_func<MultiVersionApi::runNetwork()::$_28, std::__1::allocator<MultiVersionApi::runNetwork()::$_28>, void (Reference<ClientInfo>)>::operator()(Reference<ClientInfo>&&) /usr/local/bin/../include/c++/v1/functional:1540:16 (libfdb_c.so+0x443106) 3: #7 std::__1::__function::__func<MultiVersionApi::runNetwork()::$_28, std::__1::allocator<MultiVersionApi::runNetwork()::$_28>, void (Reference<ClientInfo>)>::operator()(Reference<ClientInfo>&&) /usr/local/bin/../include/c++/v1/functional:1714:12 (libfdb_c.so+0x443106) 3: #8 std::__1::__function::__value_func<void (Reference<ClientInfo>)>::operator()(Reference<ClientInfo>&&) const /usr/local/bin/../include/c++/v1/functional:1867:16 (libfdb_c.so+0x4153ed) 3: #9 std::__1::function<void (Reference<ClientInfo>)>::operator()(Reference<ClientInfo>) const /usr/local/bin/../include/c++/v1/functional:2473:12 (libfdb_c.so+0x4153ed) 3: #10 MultiVersionApi::runOnExternalClients(int, std::__1::function<void (Reference<ClientInfo>)>, bool) /home/anoyes/workspace/foundationdb/fdbclient/MultiVersionTransaction.actor.cpp:1117:5 (libfdb_c.so+0x4153ed) 3: #11 MultiVersionApi::runOnExternalClientsAllThreads(std::__1::function<void (Reference<ClientInfo>)>, bool) /home/anoyes/workspace/foundationdb/fdbclient/MultiVersionTransaction.actor.cpp:1102:3 (libfdb_c.so+0x41992c) 3: apple#12 MultiVersionApi::runNetwork() /home/anoyes/workspace/foundationdb/fdbclient/MultiVersionTransaction.actor.cpp:1514:3 (libfdb_c.so+0x41ea53) 3: apple#13 fdb_run_network /home/anoyes/workspace/foundationdb/bindings/c/fdb_c.cpp:129:45 (libfdb_c.so+0x3f7645) 3: apple#14 decltype(std::__1::forward<int (*)()>(fp)()) std::__1::__invoke<int (*)()>(int (*&&)()) /usr/local/bin/../include/c++/v1/type_traits:3539:173 (fdb_c_unit_tests+0x384856) 3: apple#15 void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, int (*)()>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, int (*)()>&, std::__1::__tuple_indices<>) /usr/local/bin/../include/c++/v1/thread:273:5 (fdb_c_unit_tests+0x384856) 3: apple#16 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, int (*)()> >(void*) /usr/local/bin/../include/c++/v1/thread:284:5 (fdb_c_unit_tests+0x384856) 3: 3: SUMMARY: ThreadSanitizer: data race /home/anoyes/workspace/foundationdb/flow/Net2.actor.cpp:1734:6 in N2::Net2::onMainThread(Promise<Void>&&, TaskPriority) 3: ================== 3: ThreadSanitizer: reported 1 warnings

Fix build

…point ParallelStream improvements

Avoid doing thread-unsafe things in the ReadYourWritesTransaction default constructor since we now will be calling it in a foreign thread. Here's the warning we're fixing: Write of size 8 at 0x7b6800000398 by thread T1: #0 N2::Net2::updateNow() /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/flow/Net2.actor.cpp:192:33 (libfdb_c.so+0xe74568) #1 N2::Net2::run() /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/flow/Net2.actor.cpp:1459:3 (libfdb_c.so+0xe74568) #2 runNetwork() /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/fdbclient/NativeAPI.actor.cpp:2036:13 (libfdb_c.so+0x575004) #3 ThreadSafeApi::runNetwork() /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/_build/../fdbclient/ThreadSafeTransaction.cpp:445:3 (libfdb_c.so+0xcda9cc) #4 MultiVersionApi::runNetwork() /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/fdbclient/MultiVersionTransaction.actor.cpp:1741:20 (libfdb_c.so+0x4eab8d) #5 fdb_run_network /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/_build/../bindings/c/fdb_c.cpp:130:2 (libfdb_c.so+0x4bb425) #6 decltype(std::__1::forward<int (*)()>(fp)()) std::__1::__invoke<int (*)()>(int (*&&)()) /usr/local/bin/../include/c++/v1/type_traits:3899:1 (fdb_c_unit_tests+0x389816) #7 void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, int (*)()>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, int (*)()>&, std::__1::__tuple_indices<>) /usr/local/bin/../include/c++/v1/thread:280:5 (fdb_c_unit_tests+0x389816) #8 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, int (*)()> >(void*) /usr/local/bin/../include/c++/v1/thread:291:5 (fdb_c_unit_tests+0x389816) Previous read of size 8 at 0x7b6800000398 by main thread: #0 N2::Net2::now() const /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/flow/Net2.actor.cpp:160:39 (libfdb_c.so+0xe7ea5d) #1 Span::Span(UID, Location, std::initializer_list<UID> const&) /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/_build/../flow/Tracing.h:38:41 (libfdb_c.so+0x5837b4) #2 Transaction::Transaction() /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/fdbclient/NativeAPI.actor.cpp:4092:64 (libfdb_c.so+0x5837b4) #3 ReadYourWritesTransaction::ReadYourWritesTransaction() /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/_build/../fdbclient/ReadYourWrites.h:134:2 (libfdb_c.so+0x7cb4a8) #4 ISingleThreadTransaction::allocateOnForeignThread(ISingleThreadTransaction::Type) /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/_build/../fdbclient/ISingleThreadTransaction.cpp:29:17 (libfdb_c.so+0xcf785f) #5 ThreadSafeTransaction::ThreadSafeTransaction(DatabaseContext*, ISingleThreadTransaction::Type) /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/_build/../fdbclient/ThreadSafeTransaction.cpp:149:23 (libfdb_c.so+0xcc9540) #6 ThreadSafeDatabase::createTransaction() /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/_build/../fdbclient/ThreadSafeTransaction.cpp:50:37 (libfdb_c.so+0xcc4b1c) #7 fdb_database_create_transaction /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/_build/../bindings/c/fdb_c.cpp:332:2 (libfdb_c.so+0x4bce1d) #8 fdb::Transaction::Transaction(FDB_database*) /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/_build/../bindings/c/test/unit/fdb_api.cpp:111:24 (fdb_c_unit_tests+0x3925d0) #9 _DOCTEST_ANON_FUNC_10() /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/_build/../bindings/c/test/unit/unit_tests.cpp:250:19 (fdb_c_unit_tests+0x32a7a8) #10 doctest::Context::run() /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/_build/doctest/src/doctest/doctest/doctest.h:6112:21 (fdb_c_unit_tests+0x3262e4) #11 main /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/_build/../bindings/c/test/unit/unit_tests.cpp:2286:20 (fdb_c_unit_tests+0x36303d)

``` Read of size 1 at 0x7f02082ac578 by main thread: #0 TraceLog::close() /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/_build/../flow/Trace.cpp:487:7 (libfdb_c.so+0xf4928c) #1 closeTraceFile() /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/_build/../flow/Trace.cpp:759:13 (libfdb_c.so+0xf3992a) #2 stopNetwork() /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/fdbclient/NativeAPI.actor.cpp:2052:2 (libfdb_c.so+0x58315c) #3 ThreadSafeApi::stopNetwork() /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/_build/../fdbclient/ThreadSafeTransaction.cpp:466:2 (libfdb_c.so+0xce8ad3) #4 MultiVersionApi::stopNetwork() /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/fdbclient/MultiVersionTransaction.actor.cpp:1756:20 (libfdb_c.so+0x4f7da3) #5 fdb_stop_network /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/_build/../bindings/c/fdb_c.cpp:134:2 (libfdb_c.so+0x4c84e5) #6 main /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/_build/../bindings/c/test/unit/trace_partial_file_suffix_test.cpp:93:12 (trace_partial_file_suffix_test+0x40b3d0) Previous write of size 1 at 0x7f02082ac578 by thread T1 (mutexes: write M100): #0 TraceLog::open(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned long, unsigned long, Optional<NetworkAddress>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/_build/../flow/Trace.cpp:344:10 (libfdb_c.so+0xf48aaa) #1 openTraceFile(NetworkAddress const&, unsigned long, unsigned long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >) /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/_build/../flow/Trace.cpp:741:13 (libfdb_c.so+0xf3828c) #2 Database::createDatabase(Reference<ClusterConnectionFile>, int, IsInternal, LocalityData const&, DatabaseContext*) /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/fdbclient/NativeAPI.actor.cpp:1695:4 (libfdb_c.so+0x57d57e) #3 ThreadSafeDatabase::ThreadSafeDatabase(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, int)::$_7::operator()() const /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/_build/../fdbclient/ThreadSafeTransaction.cpp:124:8 (libfdb_c.so+0xcf018c) #4 internal_thread_helper::DoOnMainThreadVoidActorState<ThreadSafeDatabase::ThreadSafeDatabase(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, int)::$_7, internal_thread_helper::DoOnMainThreadVoidActor<ThreadSafeDatabase::ThreadSafeDatabase(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, int)::$_7> >::a_body1cont1(Void const&, int) /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/flow/ThreadHelper.actor.h:45:4 (libfdb_c.so+0xcf018c) #5 internal_thread_helper::DoOnMainThreadVoidActorState<ThreadSafeDatabase::ThreadSafeDatabase(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, int)::$_7, internal_thread_helper::DoOnMainThreadVoidActor<ThreadSafeDatabase::ThreadSafeDatabase(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, int)::$_7> >::a_body1when1(Void const&, int) /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/_build/flow/ThreadHelper.actor.g.h:148:15 (libfdb_c.so+0xcf018c) #6 internal_thread_helper::DoOnMainThreadVoidActorState<ThreadSafeDatabase::ThreadSafeDatabase(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, int)::$_7, internal_thread_helper::DoOnMainThreadVoidActor<ThreadSafeDatabase::ThreadSafeDatabase(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, int)::$_7> >::a_callback_fire(ActorCallback<internal_thread_helper::DoOnMainThreadVoidActor<ThreadSafeDatabase::ThreadSafeDatabase(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, int)::$_7>, 0, Void>*, Void const&) /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/_build/flow/ThreadHelper.actor.g.h:169:4 (libfdb_c.so+0xcefdef) #7 ActorCallback<internal_thread_helper::DoOnMainThreadVoidActor<ThreadSafeDatabase::ThreadSafeDatabase(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, int)::$_7>, 0, Void>::fire(Void const&) /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/_build/../flow/flow.h:1074:78 (libfdb_c.so+0xcefdef) #8 void SAV<Void>::send<Void>(Void&&) /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/_build/../flow/flow.h:462:23 (libfdb_c.so+0x4dd542) #9 void Promise<Void>::send<Void>(Void&&) const /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/_build/../flow/flow.h:699:8 ```

Bg bindings result

Net2 Run loop IO optimization

Previously we did not null out "attributes" in `Span& Span::operator=(Span&& o);`, but we destroyed the arena owning memory referenced by "attributes". Fix that by nulling out "attributes", and rewrite it in a way that's (hopefully) less error-prone. ASAN diagnostic: ``` ==24==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! ================================================================= ==24==ERROR: AddressSanitizer: heap-use-after-free on address 0x60300033fa06 at pc 0x0000026eadcf bp 0x7ffca646fe50 sp 0x7ffca646f618 READ of size 9 at 0x60300033fa06 thread T0 #0 0x26eadce in __asan_memmove /tmp/llvm-project/compiler-rt/lib/asan/asan_interceptors_memintrinsics.cpp:30:3 #1 0xd43db76 in __copy<const unsigned char, unsigned char> /usr/local/bin/../include/c++/v1/__algorithm/copy.h:59:9 #2 0xd43db76 in copy<const unsigned char *, unsigned char *> /usr/local/bin/../include/c++/v1/__algorithm/copy.h:72:13 #3 0xd43db76 in write_bytes /mnt/ephemeral/anoyes/foundationdb/fdbrpc/include/fdbrpc/Msgpack.h:45:3 #4 0xd43db76 in serialize_string(unsigned char const*, int, MsgpackBuffer&) /mnt/ephemeral/anoyes/foundationdb/fdbrpc/include/fdbrpc/Msgpack.h:109:6 #5 0xd43d208 in void serialize_map<SmallVectorRef<KeyValueRef, 1> >(SmallVectorRef<KeyValueRef, 1> const&, MsgpackBuffer&) /mnt/ephemeral/anoyes/foundationdb/fdbrpc/include/fdbrpc/Msgpack.h:154:3 #6 0xd42e2a2 in (anonymous namespace)::UDPTracer::serialize_span(Span const&, MsgpackBuffer&) /mnt/ephemeral/anoyes/foundationdb/fdbclient/Tracing.actor.cpp:157:3 #7 0xd42c8f4 in (anonymous namespace)::FastUDPTracer::trace(Span const&) /mnt/ephemeral/anoyes/foundationdb/fdbclient/Tracing.actor.cpp:301:3 #8 0xd41dfe2 in Span::operator=(Span&&) /mnt/ephemeral/anoyes/foundationdb/fdbclient/Tracing.actor.cpp:355:13 #9 0xc3eb122 in (anonymous namespace)::ReadVersionBatcherActorState<(anonymous namespace)::ReadVersionBatcherActor>::a_body1loopBody1cont1(int) /mnt/ephemeral/anoyes/foundationdb/fdbclient/NativeAPI.actor.cpp:7180:9 #10 0xc3e386d in a_body1loopBody1when2 /mnt/ephemeral/anoyes/build/foundationdb.linux.clang.asan.x86_64/fdbclient/NativeAPI.actor.g.cpp:32322:15 #11 0xc3e386d in a_callback_fire /mnt/ephemeral/anoyes/build/foundationdb.linux.clang.asan.x86_64/fdbclient/NativeAPI.actor.g.cpp:32434:4 apple#12 0xc3e386d in ActorCallback<(anonymous namespace)::ReadVersionBatcherActor, 1, Void>::fire(Void const&) /mnt/ephemeral/anoyes/foundationdb/flow/include/flow/flow.h:1316:34 apple#13 0x278af3f in void SAV<Void>::send<Void>(Void&&) /mnt/ephemeral/anoyes/foundationdb/flow/include/flow/flow.h:655:23 apple#14 0xdd2dc36 in send<Void> /mnt/ephemeral/anoyes/foundationdb/flow/include/flow/flow.h:901:8 apple#15 0xdd2dc36 in Sim2::execTask(Sim2::PromiseTask&) /mnt/ephemeral/anoyes/foundationdb/fdbrpc/sim2.actor.cpp:2319:15 apple#16 0xdd2cf0e in Sim2::runLoop(Sim2*) /mnt/ephemeral/anoyes/foundationdb/fdbrpc/sim2.actor.cpp:1279:11 apple#17 0x7ebdf2e in main /mnt/ephemeral/anoyes/foundationdb/fdbserver/fdbserver.actor.cpp:2276:17 apple#18 0x7f62c7be6554 in __libc_start_main (/lib64/libc.so.6+0x22554) apple#19 0x266f028 in _start (/mnt/ephemeral/anoyes/build/foundationdb.linux.clang.asan.x86_64/bin/fdbserver+0x266f028) 0x60300033fa06 is located 6 bytes inside of 32-byte region [0x60300033fa00,0x60300033fa20) freed by thread T0 here: #0 0x26eb352 in free /tmp/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:111:3 #1 0xdd90118 in ArenaBlock::destroy() /mnt/ephemeral/anoyes/foundationdb/flow/Arena.cpp:466:6 #2 0xdd8f0b0 in delref /mnt/ephemeral/anoyes/foundationdb/flow/Arena.cpp:173:3 #3 0xdd8f0b0 in delref<ArenaBlock> /mnt/ephemeral/anoyes/foundationdb/flow/include/flow/FastRef.h:95:7 #4 0xdd8f0b0 in operator= /mnt/ephemeral/anoyes/foundationdb/flow/include/flow/FastRef.h:147:5 #5 0xdd8f0b0 in Arena::operator=(Arena&&) /mnt/ephemeral/anoyes/foundationdb/flow/Arena.cpp:119:36 #6 0xd41dfed in Span::operator=(Span&&) /mnt/ephemeral/anoyes/foundationdb/fdbclient/Tracing.actor.cpp:357:8 #7 0xc3eb122 in (anonymous namespace)::ReadVersionBatcherActorState<(anonymous namespace)::ReadVersionBatcherActor>::a_body1loopBody1cont1(int) /mnt/ephemeral/anoyes/foundationdb/fdbclient/NativeAPI.actor.cpp:7180:9 #8 0xc3e386d in a_body1loopBody1when2 /mnt/ephemeral/anoyes/build/foundationdb.linux.clang.asan.x86_64/fdbclient/NativeAPI.actor.g.cpp:32322:15 #9 0xc3e386d in a_callback_fire /mnt/ephemeral/anoyes/build/foundationdb.linux.clang.asan.x86_64/fdbclient/NativeAPI.actor.g.cpp:32434:4 #10 0xc3e386d in ActorCallback<(anonymous namespace)::ReadVersionBatcherActor, 1, Void>::fire(Void const&) /mnt/ephemeral/anoyes/foundationdb/flow/include/flow/flow.h:1316:34 #11 0x278af3f in void SAV<Void>::send<Void>(Void&&) /mnt/ephemeral/anoyes/foundationdb/flow/include/flow/flow.h:655:23 apple#12 0xdd2dc36 in send<Void> /mnt/ephemeral/anoyes/foundationdb/flow/include/flow/flow.h:901:8 apple#13 0xdd2dc36 in Sim2::execTask(Sim2::PromiseTask&) /mnt/ephemeral/anoyes/foundationdb/fdbrpc/sim2.actor.cpp:2319:15 apple#14 0xdd2cf0e in Sim2::runLoop(Sim2*) /mnt/ephemeral/anoyes/foundationdb/fdbrpc/sim2.actor.cpp:1279:11 apple#15 0x7ebdf2e in main /mnt/ephemeral/anoyes/foundationdb/fdbserver/fdbserver.actor.cpp:2276:17 apple#16 0x7f62c7be6554 in __libc_start_main (/lib64/libc.so.6+0x22554) previously allocated by thread T0 here: #0 0x26ebeb2 in aligned_alloc /tmp/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:176:3 #1 0xdd8e5e0 in ArenaBlock::create(int, Reference<ArenaBlock>&) /mnt/ephemeral/anoyes/foundationdb/flow/Arena.cpp:339:21 #2 0xdd91139 in ArenaBlock::allocate(Reference<ArenaBlock>&, int) /mnt/ephemeral/anoyes/foundationdb/flow/Arena.cpp:322:7 #3 0x2bbf49b in operator new[] /mnt/ephemeral/anoyes/foundationdb/flow/include/flow/Arena.h:206:9 #4 0x2bbf49b in StringRef /mnt/ephemeral/anoyes/foundationdb/flow/include/flow/Arena.h:446:54 #5 0x2bbf49b in Span::Span(SpanContext const&, Location const&, SpanContext const&, std::initializer_list<SpanContext> const&) /mnt/ephemeral/anoyes/foundationdb/fdbclient/include/fdbclient/Tracing.h:141:33 #6 0x2bbef25 in Span::Span(Location const&, SpanContext const&, std::initializer_list<SpanContext> const&) /mnt/ephemeral/anoyes/foundationdb/fdbclient/include/fdbclient/Tracing.h:148:6 #7 0xc1bd941 in Span /mnt/ephemeral/anoyes/foundationdb/fdbclient/include/fdbclient/Tracing.h:160:44 #8 0xc1bd941 in ReadVersionBatcherActorState /mnt/ephemeral/anoyes/foundationdb/fdbclient/NativeAPI.actor.cpp:7125:6 #9 0xc1bd941 in ReadVersionBatcherActor /mnt/ephemeral/anoyes/build/foundationdb.linux.clang.asan.x86_64/fdbclient/NativeAPI.actor.g.cpp:32621:6 #10 0xc1bd941 in readVersionBatcher(DatabaseContext* const&, FutureStream<DatabaseContext::VersionRequest> const&, TransactionPriority const&, unsigned int const&) /mnt/ephemeral/anoyes/foundationdb/fdbclient/NativeAPI.actor.cpp:7098:26 #11 0xc1b316a in Transaction::getReadVersion(unsigned int) /mnt/ephemeral/anoyes/foundationdb/fdbclient/NativeAPI.actor.cpp:7366:8 apple#12 0xc19055f in getReadVersion /mnt/ephemeral/anoyes/foundationdb/fdbclient/include/fdbclient/NativeAPI.actor.h:315:44 apple#13 0xc19055f in Transaction::get(Standalone<StringRef> const&, Snapshot) /mnt/ephemeral/anoyes/foundationdb/fdbclient/NativeAPI.actor.cpp:5314:13 apple#14 0x5eca2f1 in a_body1 /mnt/ephemeral/anoyes/foundationdb/fdbserver/MoveKeys.actor.cpp:190:55 apple#15 0x5eca2f1 in ReadMoveKeysLockActor /mnt/ephemeral/anoyes/build/foundationdb.linux.clang.asan.x86_64/fdbserver/MoveKeys.actor.g.cpp:1001:9 apple#16 0x5eca2f1 in readMoveKeysLock(Transaction* const&, MoveKeysLock* const&) /mnt/ephemeral/anoyes/foundationdb/fdbserver/MoveKeys.actor.cpp:188:26 apple#17 0x5ee590d in a_body1loopBody1 /mnt/ephemeral/anoyes/foundationdb/fdbserver/MoveKeys.actor.cpp:228:39 apple#18 0x5ee590d in (anonymous namespace)::TakeMoveKeysLockActorState<(anonymous namespace)::TakeMoveKeysLockActor>::a_body1loopHead1(int) /mnt/ephemeral/anoyes/build/foundationdb.linux.clang.asan.x86_64/fdbserver/MoveKeys.actor.g.cpp:1421:49 apple#19 0x5ecb801 in a_body1 /mnt/ephemeral/anoyes/build/foundationdb.linux.clang.asan.x86_64/fdbserver/MoveKeys.actor.g.cpp:1400:16 apple#20 0x5ecb801 in TakeMoveKeysLockActor /mnt/ephemeral/anoyes/build/foundationdb.linux.clang.asan.x86_64/fdbserver/MoveKeys.actor.g.cpp:1807:9 apple#21 0x5ecb801 in takeMoveKeysLock(Database const&, UID const&) /mnt/ephemeral/anoyes/foundationdb/fdbserver/MoveKeys.actor.cpp:216:34 apple#22 0x542a421 in DDTxnProcessor::takeMoveKeysLock(UID const&) const /mnt/ephemeral/anoyes/foundationdb/fdbserver/DDTxnProcessor.actor.cpp:620:9 apple#23 0x562aa75 in takeMoveKeysLock /mnt/ephemeral/anoyes/foundationdb/fdbserver/DataDistribution.actor.cpp:330:69 apple#24 0x562aa75 in DataDistributor::InitActorState<DataDistributor::InitActor>::a_body1loopBody1(int) /mnt/ephemeral/anoyes/foundationdb/fdbserver/DataDistribution.actor.cpp:370:44 apple#25 0x562a0ba in a_body1loopHead1 /mnt/ephemeral/anoyes/build/foundationdb.linux.clang.asan.x86_64/fdbserver/DataDistribution.actor.g.cpp:1116:49 apple#26 0x562a0ba in DataDistributor::InitActorState<DataDistributor::InitActor>::a_body1(int) /mnt/ephemeral/anoyes/build/foundationdb.linux.clang.asan.x86_64/fdbserver/DataDistribution.actor.g.cpp:1083:16 apple#27 0x55088c0 in InitActor /mnt/ephemeral/anoyes/build/foundationdb.linux.clang.asan.x86_64/fdbserver/DataDistribution.actor.g.cpp:1715:9 apple#28 0x55088c0 in init /mnt/ephemeral/anoyes/foundationdb/fdbserver/DataDistribution.actor.cpp:367:26 apple#29 0x55088c0 in a_body1loopBody1 /mnt/ephemeral/anoyes/foundationdb/fdbserver/DataDistribution.actor.cpp:611:39 apple#30 0x55088c0 in (anonymous namespace)::DataDistributionActorState<(anonymous namespace)::DataDistributionActor>::a_body1loopHead1(int) /mnt/ephemeral/anoyes/build/foundationdb.linux.clang.asan.x86_64/fdbserver/DataDistribution.actor.g.cpp:2772:49 apple#31 0x54e9cf1 in a_body1 /mnt/ephemeral/anoyes/build/foundationdb.linux.clang.asan.x86_64/fdbserver/DataDistribution.actor.g.cpp:2751:16 apple#32 0x54e9cf1 in DataDistributionActor /mnt/ephemeral/anoyes/build/foundationdb.linux.clang.asan.x86_64/fdbserver/DataDistribution.actor.g.cpp:3976:9 apple#33 0x54e9cf1 in dataDistribution(Reference<DataDistributor> const&, PromiseStream<GetMetricsListRequest> const&) /mnt/ephemeral/anoyes/foundationdb/fdbserver/DataDistribution.actor.cpp:586:26 apple#34 0x54f8758 in a_body1 /mnt/ephemeral/anoyes/foundationdb/fdbserver/DataDistribution.actor.cpp:1533:39 apple#35 0x54f8758 in DataDistributorActor /mnt/ephemeral/anoyes/build/foundationdb.linux.clang.asan.x86_64/fdbserver/DataDistribution.actor.g.cpp:11473:9 apple#36 0x54f8758 in dataDistributor(DataDistributorInterface const&, Reference<AsyncVar<ServerDBInfo> const> const&) /mnt/ephemeral/anoyes/foundationdb/fdbserver/DataDistribution.actor.cpp:1517:26 apple#37 0x88f3f88 in (anonymous namespace)::WorkerServerActorState<(anonymous namespace)::WorkerServerActor>::a_body1cont10loopBody1when6(InitializeDataDistributorRequest&&, int) /mnt/ephemeral/anoyes/foundationdb/fdbserver/worker.actor.cpp:2219:42 apple#38 0x8871c15 in a_callback_fire /mnt/ephemeral/anoyes/build/foundationdb.linux.clang.asan.x86_64/fdbserver/worker.actor.g.cpp:13169:4 apple#39 0x8871c15 in ActorSingleCallback<(anonymous namespace)::WorkerServerActor, 6, InitializeDataDistributorRequest>::fire(InitializeDataDistributorRequest&&) /mnt/ephemeral/anoyes/foundationdb/flow/include/flow/flow.h:1338:34 apple#40 0x33e13c0 in send<InitializeDataDistributorRequest> /mnt/ephemeral/anoyes/foundationdb/flow/include/flow/flow.h:1004:29 apple#41 0x33e13c0 in NetNotifiedQueue<InitializeDataDistributorRequest, false>::receive(ArenaObjectReader&) /mnt/ephemeral/anoyes/foundationdb/fdbrpc/include/fdbrpc/fdbrpc.h:702:10 apple#42 0xd91b6c6 in (anonymous namespace)::DeliverActorState<(anonymous namespace)::DeliverActor>::a_body1cont1(int) /mnt/ephemeral/anoyes/foundationdb/fdbrpc/FlowTransport.actor.cpp:1042:15 apple#43 0xd91a37d in a_body1cont2 /mnt/ephemeral/anoyes/build/foundationdb.linux.clang.asan.x86_64/fdbrpc/FlowTransport.actor.g.cpp:4333:15 apple#44 0xd91a37d in a_body1when1 /mnt/ephemeral/anoyes/build/foundationdb.linux.clang.asan.x86_64/fdbrpc/FlowTransport.actor.g.cpp:4345:15 apple#45 0xd91a37d in a_callback_fire /mnt/ephemeral/anoyes/build/foundationdb.linux.clang.asan.x86_64/fdbrpc/FlowTransport.actor.g.cpp:4366:4 apple#46 0xd91a37d in ActorCallback<(anonymous namespace)::DeliverActor, 0, Void>::fire(Void const&) /mnt/ephemeral/anoyes/foundationdb/flow/include/flow/flow.h:1316:34 apple#47 0x278af3f in void SAV<Void>::send<Void>(Void&&) /mnt/ephemeral/anoyes/foundationdb/flow/include/flow/flow.h:655:23 apple#48 0xdd2dc36 in send<Void> /mnt/ephemeral/anoyes/foundationdb/flow/include/flow/flow.h:901:8 apple#49 0xdd2dc36 in Sim2::execTask(Sim2::PromiseTask&) /mnt/ephemeral/anoyes/foundationdb/fdbrpc/sim2.actor.cpp:2319:15 apple#50 0xdd2cf0e in Sim2::runLoop(Sim2*) /mnt/ephemeral/anoyes/foundationdb/fdbrpc/sim2.actor.cpp:1279:11 apple#51 0x7ebdf2e in main /mnt/ephemeral/anoyes/foundationdb/fdbserver/fdbserver.actor.cpp:2276:17 apple#52 0x7f62c7be6554 in __libc_start_main (/lib64/libc.so.6+0x22554) SUMMARY: AddressSanitizer: heap-use-after-free /tmp/llvm-project/compiler-rt/lib/asan/asan_interceptors_memintrinsics.cpp:30:3 in __asan_memmove Shadow bytes around the buggy address: 0x0c068005fef0: fa fa fd fd fd fd fa fa fd fd fd fd fa fa fd fd 0x0c068005ff00: fd fd fa fa fd fd fd fd fa fa fd fd fd fa fa fa 0x0c068005ff10: fd fd fd fd fa fa fd fd fd fd fa fa fd fd fd fd 0x0c068005ff20: fa fa fd fd fd fa fa fa fd fd fd fd fa fa fd fd 0x0c068005ff30: fd fd fa fa fd fd fd fd fa fa fd fd fd fa fa fa =>0x0c068005ff40:[fd]fd fd fd fa fa fd fd fd fd fa fa fd fd fd fd 0x0c068005ff50: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd 0x0c068005ff60: fd fa fa fa fd fd fd fa fa fa fd fd fd fd fa fa 0x0c068005ff70: fd fd fd fd fa fa fd fd fd fa fa fa 00 00 00 00 0x0c068005ff80: fa fa fd fd fd fd fa fa 00 00 00 00 fa fa fd fd 0x0c068005ff90: fd fd fa fa 00 00 00 00 fa fa 00 00 00 00 fa fa Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb ```

Adds a --recursive option for ls to the s3client command tool. Adds test to s3client ctest. * Fix a heap-use-after-free corruption bug revealed running against s3 ================================================================= ==1988376==ERROR: AddressSanitizer: heap-use-after-free on address 0x508000012020 at pc 0x0000014b7bc2 bp 0x7ffca9e22410 sp 0x7ffca9e21bd0 WRITE of size 80 at 0x508000012020 thread T0 #0 0x0000014b7bc1 in __asan_memcpy /tmp/llvm-project/compiler-rt/lib/asan/asan_interceptors_memintrinsics.cpp:63:3 #1 0x0000043f5c83 in XXH64_reset /root/src/foundationdb/flow/include/flow/xxhash.h:1977:2 #2 0x00000337a336 in (anonymous namespace)::CopyUpFileActorState<(anonymous namespace)::CopyUpFileActor>::a_body1loopBody1(int) /root/src/foundationdb/fdbclient/S3Client.actor.cpp:429:2

jzhou77 force-pushed the lock branch from 023d4c3 to d46c4ee Compare July 17, 2020 04:06

jzhou77 force-pushed the master branch from 24a8a20 to 96af97c Compare July 17, 2020 04:09

jzhou77 force-pushed the lock branch 2 times, most recently from 3085676 to 5251513 Compare July 19, 2020 18:35

jzhou77 changed the base branch from master to backup-fix July 19, 2020 18:40

jzhou77 changed the base branch from backup-fix to master July 19, 2020 18:40

jzhou77 force-pushed the lock branch 3 times, most recently from 2e5b709 to 5dbca79 Compare July 27, 2020 21:53

jzhou77 force-pushed the lock branch from 69f926e to a2ea16d Compare August 3, 2020 04:26

jzhou77 changed the base branch from master to release-6.3 August 3, 2020 04:28

jzhou77 changed the base branch from release-6.3 to master August 3, 2020 04:28

jzhou77 force-pushed the lock branch from a2ea16d to 7c41c77 Compare September 16, 2020 20:26

jzhou77 changed the base branch from master to release-6.3 September 16, 2020 22:11

jzhou77 changed the base branch from release-6.3 to master September 16, 2020 22:12

jzhou77 force-pushed the lock branch 2 times, most recently from 67055f1 to 02b8491 Compare September 27, 2020 21:04

jzhou77 pushed a commit that referenced this pull request Oct 7, 2020

Merge pull request #2 from xumengpanda/yang/range-metadata-file

c34bdfd

Review comments and integrate extended getRestoreSet API into fast restore

jzhou77 force-pushed the lock branch 2 times, most recently from 6d1ca6f to 0e078ad Compare October 13, 2020 19:58

jzhou77 pushed a commit that referenced this pull request Oct 13, 2020

Commit pipelining attempt #2

34d1d04

jzhou77 added 9 commits October 13, 2020 13:05

Add (un)lockRange AIP for transactions

e42eca5

Update client transactions with range lock atomic ops

fcd302b

Add LockRange workload

1607fac

Add CycleAndLockRange test file

1d47372

Refactor resolver

b522231

Limit range lock APIs to a single txn

889142a

The lock/unlock range operation is stored on txn state store, thus can't be used with other ops (mutations) on normal key ranges.

Add locking with a vector of ranges

c6ff40c

Convert CycleAndLockRange test to TOML format

02574d7

jzhou77 added 11 commits October 13, 2020 13:15

Fix snapshot for RangeLockCache

d1d1a47

Passes CycleAndLockRange.toml tests

a93ec0b

Fix cmake file for test

b857716

Fix inverted range problem

a186048

Remove unused code

7ae23b0

Throw range_locks_access_denied when get snapshot timeouted

d495708

This range_locks_access_denied causes transaction retry, so there is no harm. In contrast, a timeout error would be raised and can cause client to fail.

Remove an unused function

8ead8ed

Add range locks to release notes

69faec3

Correctly handle transaction_too_old or unknown_result for lockRange

5a11907

We'll need a full reset to trigger the refresh of rangeLock caches. When getting unknown_result and retrying, short-circuit unlock request as NOOP.

jzhou77 force-pushed the lock branch from 0e078ad to 187b74e Compare October 13, 2020 20:16

Limit range lock request to normal keys

c1d7a98

jzhou77 force-pushed the lock branch from 187b74e to c1d7a98 Compare October 14, 2020 03:43

jzhou77 added 2 commits October 13, 2020 20:48

Use override instead of virtual

c314381

Add lock owner for each lockRange transaction

fee825a

This lock owner UID is stored as part of the value for keys in "lockedKeyRanges" key space so that it can be used to tell if transaction committed when unknown_result is returned.

jzhou77 closed this Oct 15, 2020

jzhou77 pushed a commit that referenced this pull request Mar 3, 2021

Merge pull request #2 from apple/master

d0c62b2

Merge Master

jzhou77 pushed a commit that referenced this pull request May 4, 2021

Merge pull request #2 from sfc-gh-ljoswiak/fixes/fix-build

712531b

Fix build

jzhou77 pushed a commit that referenced this pull request Jun 28, 2021

Merge pull request #2 from sfc-gh-tclinkenbeard/getrange-stream-split…

ae52217

…point ParallelStream improvements

jzhou77 pushed a commit that referenced this pull request Dec 21, 2021

Merge pull request #2 from sfc-gh-jslocum/bg_bindings_result

7a8fbd9

Bg bindings result

jzhou77 pushed a commit that referenced this pull request Mar 4, 2022

Merge pull request #2 from sfc-gh-jfu/flowhancements-net2run

76ff1f9

Net2 Run loop IO optimization

jzhou77 pushed a commit that referenced this pull request Jun 27, 2022

fixup! Reformat source #2

21ee76a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add (un)lockRange APIs #2

Add (un)lockRange APIs #2

Uh oh!

jzhou77 commented Jul 15, 2020 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Add (un)lockRange APIs #2

Add (un)lockRange APIs #2

Uh oh!

Conversation

jzhou77 commented Jul 15, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TODOs

High Level Ideas

Guarantees

Caveats

TODOs

Mechanism:

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jzhou77 commented Jul 15, 2020 •

edited

Loading