Use ListObjectsV2 in S3 Blobstore client #12238

noctella · 2025-07-11T16:10:12Z

Problem

The current s3 blobstore client uses the deprecated V1 list API: https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjects.html

This means that backup restoration will fail if the S3-compatible endpoint only implements the V2 endpoint.

Solution

Migrate to the (simpler) V2 API. Add integration tests in s3client_test.sh. Add support for non-Apple S3 environments when running this integration test :)

noctella · 2025-07-11T16:12:32Z

fdbclient/ClientKnobs.cpp

 	init( BLOBSTORE_READ_REQUESTS_PER_SECOND,       100 );
 	init( BLOBSTORE_DELETE_REQUESTS_PER_SECOND,     200 );

+	// Request 1000 keys at a time, the maximum allowed


Added knob here so that integration tests can test pagination without creating 1k+ files

noctella · 2025-07-11T16:13:16Z

fdbclient/S3Client_cli.actor.cpp

 	OPT_PROXY,
-	OPT_HELP
+	OPT_HELP,
+	OPT_LS_RECURSIVE


Add recursive option here so we can use it in integration testgs

noctella · 2025-07-11T16:13:32Z

fdbclient/tests/aws_fixture.sh

  local dir="${1}"
  local aws_dir
-  aws_dir=$(mktemp -d -p "${dir}" -t s3.$$.XXXX)
+  aws_dir=$(mktemp -d "${dir}/s3.$$.XXXX")


x-platform compatible

noctella · 2025-07-11T16:14:42Z

fdbclient/tests/s3client_test.sh

  exit 1
 fi
+
+blob_credentials_file=""


aws_setup currently uses (presumably) apple-internal aws configuration. Add flags so we can use a different set of params here.

foundationdb-ci · 2025-07-11T17:08:12Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: d7496cc
Duration 0:04:14
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-11T17:08:14Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: d7496cc
Duration 0:04:13
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-16T18:43:52Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: f041307
Duration 0:04:18
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-16T18:43:54Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: f041307
Duration 0:04:15
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-16T18:43:55Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: f041307
Duration 0:04:17
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-16T18:43:55Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: f041307
Duration 0:04:18
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-16T18:44:01Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: f041307
Duration 0:04:23
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2025-07-16T19:18:13Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: f041307
Duration 0:38:34
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-16T19:40:41Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: f041307
Duration 1:01:06
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

Fix a heap-use-after-free corruption bug revealed running against s3 ================================================================= ==1988376==ERROR: AddressSanitizer: heap-use-after-free on address 0x508000012020 at pc 0x0000014b7bc2 bp 0x7ffca9e22410 sp 0x7ffca9e21bd0 WRITE of size 80 at 0x508000012020 thread T0 #0 0x0000014b7bc1 in __asan_memcpy /tmp/llvm-project/compiler-rt/lib/asan/asan_interceptors_memintrinsics.cpp:63:3 apple#1 0x0000043f5c83 in XXH64_reset /root/src/foundationdb/flow/include/flow/xxhash.h:1977:2 apple#2 0x00000337a336 in (anonymous namespace)::CopyUpFileActorState<(anonymous namespace)::CopyUpFileActor>::a_body1loopBody1(int) /root/src/foundationdb/fdbclient/S3Client.actor.cpp:429:2 * fdbclient/tests/s3client_test.sh When run w/ bash -x, script was picking up the '++' output. When going against s3, need to do secure_connection=1. Fix the check for empty when looking for empty response. Fixed an odd one where we had an errant ' on end of the url.

saintstack · 2025-07-17T07:24:09Z

Pushed a change that does 'clang-format' on the changed .cpp files. Ran tests against s3 and ran into a few items: 1. a use-after-free in s3client (revealed by this PR but not caused by it); 2. need secure_connection=1 when against s3 ... was '0'; and 3. check for empty response is a little tricky.

foundationdb-ci · 2025-07-17T07:44:33Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 5cda991
Duration 0:24:24
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-17T08:00:28Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 5cda991
Duration 0:40:19
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-17T08:07:27Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 5cda991
Duration 0:47:17
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-17T08:23:16Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: 5cda991
Duration 1:03:06
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-17T08:24:10Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 5cda991
Duration 1:04:01
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2025-07-17T08:25:35Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 5cda991
Duration 1:05:28
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-17T08:32:39Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 5cda991
Duration 1:12:30
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

saintstack · 2025-07-17T15:17:12Z

Joshua reports a failing test:

[Container] 2025/07/17 08:19:22.348689 Running command python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID}
  20250717-075109-pr12238-clang-5cda9916-1233-f048e437e8434b95 compressed=True data_size=41297603 duration=623119 ended=10000 fail=1 fail_fast=10 max_runs=10000 pass=9999 priority=100 remaining=0 runtime=0:28:13 sanity=False started=10000 stopped=20250717-081922 submitted=20250717-075109 timeout=5400 username=pr12238-clang-5cda9916-12335

InternalError: FailedAssertion="res.empty()" 
File: /foundationdb/fdbserver/workloads/RandomRangeLock.actor.cpp
Line: 195

Looks like a bug, but not because of this PR.

foundationdb-ci · 2025-07-17T15:42:58Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 5cda991
Duration 0:25:27
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-17T15:56:14Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 5cda991
Duration 0:38:48
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-17T16:05:27Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 5cda991
Duration 0:47:58
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-17T16:18:14Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 5cda991
Duration 1:00:46
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2025-07-17T16:18:33Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: 5cda991
Duration 1:01:04
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-17T18:09:11Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 5cda991
Duration 2:51:46
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-17T18:09:15Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 5cda991
Duration 2:51:45
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

saintstack · 2025-07-13T03:30:29Z

fdbclient/S3BlobStore.actor.cpp

-	// Request 1000 keys at a time, the maximum allowed
-	state std::string resource = bstore->constructResourcePath(bucket, "");
+	state std::string resource = bstore->constructResourcePath(bucket, "");										
+	resource.append("?list-type=2&max-keys=").append(std::to_string(CLIENT_KNOBS->BLOBSTORE_LIST_MAX_KEYS_PER_PAGE));


saintstack · 2025-07-17T18:49:03Z

Thanks for the contribution @noctella

noctella added 19 commits July 9, 2025 11:56

Prevent double-encoding on s3 retries

cd383f5

move things around

f072ce6

working

43f89a3

working

18b319e

quick review

8bab20f

quick review

9136b2e

quick review

02a9b6c

quick review

b22e70e

quick review

a64918c

quick review

a6bd8e9

quick review

74987a1

quick review

95ddadb

quick review

18917b8

quick review

11e51de

quick review

52d5849

quick review

e8813fc

quick review

3402e33

quick review

ff1a61f

quick review

d7496cc

noctella changed the title ~~Jillianc/list apiv2~~ Use ListObjectsV2 in S3 Blobstore client Jul 11, 2025

noctella closed this Jul 11, 2025

noctella reopened this Jul 11, 2025

noctella commented Jul 11, 2025

View reviewed changes

jzhou77 requested a review from saintstack July 11, 2025 17:02

jzhou77 closed this Jul 11, 2025

jzhou77 reopened this Jul 11, 2025

saintstack reopened this Jul 16, 2025

saintstack closed this Jul 17, 2025

saintstack reopened this Jul 17, 2025

saintstack approved these changes Jul 17, 2025

View reviewed changes

saintstack merged commit 15dd76a into apple:main Jul 17, 2025
7 checks passed

Use ListObjectsV2 in S3 Blobstore client #12238

Use ListObjectsV2 in S3 Blobstore client #12238

Uh oh!

Conversation

noctella commented Jul 11, 2025 • edited by johscheuer Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Solution

Uh oh!

noctella Jul 11, 2025

Choose a reason for hiding this comment

Uh oh!

noctella Jul 11, 2025

Choose a reason for hiding this comment

Uh oh!

noctella Jul 11, 2025

Choose a reason for hiding this comment

Uh oh!

noctella Jul 11, 2025

Choose a reason for hiding this comment

Uh oh!

foundationdb-ci commented Jul 11, 2025

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jul 11, 2025

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jul 16, 2025

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jul 16, 2025

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jul 16, 2025

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Jul 16, 2025

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jul 16, 2025

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jul 16, 2025

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Jul 16, 2025

Result of foundationdb-pr-macos on macOS Ventura 13.x

Uh oh!

saintstack commented Jul 17, 2025

Uh oh!

foundationdb-ci commented Jul 17, 2025

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jul 17, 2025

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Jul 17, 2025

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Jul 17, 2025

Result of foundationdb-pr-macos on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Jul 17, 2025

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jul 17, 2025

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jul 17, 2025

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

saintstack commented Jul 17, 2025

Uh oh!

foundationdb-ci commented Jul 17, 2025

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jul 17, 2025

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Jul 17, 2025

noctella commented Jul 11, 2025 •

edited by johscheuer

Loading