Java bindings for BG Purge (#6816)

* fix heap-use-after-free caused by early noErrorsActors destroy

* add comment

* move keys out of StorageMetrics

* format code

* Java bindings for BG Purge (#6816)

* Add support for returning whether a key-backed type range read has more results or not.

* Use narrow includes in key backed types; remove some unnecessary return statements.

* Fix snapshotFormatUnitTest infinite loop

Patch addresses an issue where loop generating random buffer
for the test might run in an infinite loop if KeySpace gets
exhausted but the 'targetBuffer' bytes aren't generated.

Description

Testing

* smaller function

* Remove one more include and some extraneous return statements. Add a NullCodec that passes through the arguments unchanged.

* Add a Tuple::makeTuple function to easily construct a tuple. Update Tuple to allow all types to be passed via .append() so they can be used with makeTuple.

* Add the ability for StringRef.eat to return whether a separator was found. Add a serialize function and equality operators to ClusterConnectionString. Remove unused variable.

* add unittest

* Convert existing tuple usages to use Tuple::makeTuple()

* resolve conflict upstream/main

* Remove the ability to configure a tenant subspace. Rename the prefixes used for tenant metadata.

* Fixing merge convergence bugs

* Refactoring granule flush to retry properly on granule rollback

* cleanup

* Issue 7570: Check existance of container before creation attempt

* cleanup

* Use a constexpr check for deserialization (from code review)

* Increase AllTags field length in TLogReady

* Add a comment to name the fold expression syntax in makeTuple. This should make it easier to look up for those unfamiliar with the syntax.

* Fix ASAN build (#7629)

* Fix ASAN build

Description
  -diff-2: Fix Mac build issues
  -diff-1: Address review comments

Patch addresses the issue where ASAN build failed after introducing
BlobGranule compression.

Testing

ASAN build

* only do check after bootstraping test

* Some minor changes from code review

* solve review comments

* function naming using intention rather than implementation details

* update comments

* Build fixes/improvements for sanitizers (#7657)

* Don't build fdb c shim with ubsan

This avoids duplicate symbols when linking. It doesn't really make sense
to assemble files with -fsanitize=undefined anyway, since it won't
insert instrumentation.

* Consolidate boost_asan with boost_target

* Log tlog initialization

* Allow read range on null shard (#7654)

* Fixed ShardedRocks unit test.

* Removed `UNIT` from noSim/ShardedRocksDBTest.toml.

Co-authored-by: He Liu <[email protected]>

* Validate subrange reads in simulation (#7597)

* Add extra validation to special key space reads in simulation

* Fix bugs turned up by validating subrange reads

* Change to validateSpecialSubrangeRead

It is in general not safe to expect that a read from the special key
space returns the same results if performed again, since the
transaction may be being modified concurrently.

* Add comment

* Add comment

* Add TraceEvent to the construction/destruction of DatabaseContext

This is to figure out why huge databaseLogger related TraceEvents
generated after 7.1 released.

* Use key-backed types for tenants. Add a tenant state field that will be used in upcoming work. Some other tenant related refactoring.

* Add CommitDebug event after Commit Proxy reporting committed version to master

* fixup! Update code per comments

* Ss shard management (#7340)

* Storage server shard management with physical shards.

* Cleanup.

* Resolved comments.

* Added `UnlimintedCommitBytes`.

Co-authored-by: He Liu <[email protected]>

* Proxy GlobalConfig reads through GRV proxies

Clients should avoid reading system keys unless authorized. Under global
config, each client reads from the system keyspace to check for new
global config keys. This commit moves these reads to a server role (the
GRV proxies) and sends the results back to GlobalConfig for an in-memory
update.

* Batch global config refresh requests

* Perform migration automatically on proxy boot

* Update fdbserver/GrvProxyServer.actor.cpp

Co-authored-by: A.J. Beamon <[email protected]>

* Update fdbclient/include/fdbclient/CommitProxyInterface.h

Co-authored-by: A.J. Beamon <[email protected]>

* Add migration timeout

* Reduce global configuration system key reads from proxy

Clients now poll the proxy for the latest global config for a specific
version. The proxy now periodically requests the latest global
configuration data and stores it in memory, enabling it to respond
immediately to clients with the appropriate version.

* Add delay to allow for global config sync time

* Mess with timeout values

* Simplify refresh condition

* Fix Tuple error

* Update fdbserver/GrvProxyServer.actor.cpp

Co-authored-by: A.J. Beamon <[email protected]>

* Update fdbserver/GrvProxyServer.actor.cpp

Co-authored-by: A.J. Beamon <[email protected]>

* Add comments to deleteTenantTransaction. Fetch the tenant ID from the first transaction in deleteTransaction to use on subsequent retries.

* Don't pass non-pod to variadic function (#7668)

fdbserver/storageserver.actor.cpp:654:24: error: cannot pass object of non-trivial type 'const std::string' (aka 'const basic_string<char, char_traits<char>, allocator<char>>') through variadic function; call will abort at runtime [-Wnon-pod-varargs]

Just use fmt instead

* Disable guard pages for valgrind build. Closes #7610 (#7672)

* fixup! Let clang accepts crc32_wrapper.c

* Add timeout during Status generation

Resolving hostname might be blocking.

* Reduce a DNS lookup in Status generation

* Update status json doc with fetch_storage_wiggler_stats_timeout error

* Fix WinGDI name collision by defining macro NOGDI

TenantState::ERROR collides with WinGDI's #define ERROR 0

* Move DD queue code over to using movement-reasons rather than priority (#7614)

* Use enum variables to invoke Priority checking

* add an explicit isDataMovementForReadBalancing function

* Set up RelocateShard in terms of data movement reason instead of priority

* Remove isMountainChopperPriority

* Remove isDiskRebalancePriority

* Fix formatting

* Fix misnamed DataMovementReason::TEAM_HEALTHY

Co-authored-by: Zhongxing Zhang <[email protected]>

* Error in FetchKey GRV shouldn't fail storageserver

* Add testing mechanism

* ddsketch_utility: Add utility scripts for ddsketch (#7602)

* Adds ddsketch_calc.py which implements a class for DDSketch related calculations
* Adds ddsketch_conversion.py for quickly computing a bucket index to a value or vice-versa
* Adds ddsketch_compare.py to compute how similar two ddsketch distributions are
* Adds export_graph.py to graph the ddsketch distribution outputted from mako

The arguments for ddsketch_conversion.py are:
-b, --bucket: the bucket index that we need to calculate the value from (optional)
-v, --value: the value that we need to calculate the bucket index from (optional)
-e, --error_guarantee: the error guarantee for ddsketch (optional, default is 0.005)

The arguments for ddsketch_compare.py are:
--file1: Path to first ddsketch json
--file2: Path to second ddsketch json
--txn1: The transaction type for the first file
--txn2: The transaction type for the second file
--op: The operation name (ex: GRV, GET ...)

The arguments for export_graph.py:
--file: path to ddsketch distribution
--txn, -t: Transaction type from file
--title: title for graph (optional, otherwise "Title" is used)
--savefig: Path to save the image plot (optional)
--op: Which operation to plot

Co-authored-by: Xiaoxi Wang <[email protected]>
Co-authored-by: Josh Slocum <[email protected]>
Co-authored-by: A.J. Beamon <[email protected]>
Co-authored-by: Ata E Husain Bohra <[email protected]>
Co-authored-by: Hari Bhaskaran <[email protected]>
Co-authored-by: Trevor Clinkenbeard <[email protected]>
Co-authored-by: Zhe Wu <[email protected]>
Co-authored-by: Andrew Noyes <[email protected]>
Co-authored-by: He Liu <[email protected]>
Co-authored-by: He Liu <[email protected]>
Co-authored-by: Xiaoge Su <[email protected]>
Co-authored-by: Jingyu Zhou <[email protected]>
Co-authored-by: Lukas Joswiak <[email protected]>
Co-authored-by: Junhyun Shim <[email protected]>
Co-authored-by: Bharadwaj V.R <[email protected]>
Co-authored-by: Zhongxing Zhang <[email protected]>
Co-authored-by: Markus Pilman <[email protected]>
Co-authored-by: Kevin Hoxha <[email protected]>