use sni in the tls handshake by evkuzin · Pull Request #12374 · apple/foundationdb

evkuzin · 2025-09-16T09:08:50Z

Original PR is in #12366

Problem
FoundationDB's network layer was not properly setting Server Name Indication (SNI) when making TLS connections to hostnames that require SNI for proper certificate selection. This caused failures when connecting to services like Google Cloud Storage, which returns a dummy self-signed certificate when SNI is not provided.

Symptoms:

Backup agent fails with "VerifyError": "self signed certificate" when connecting to storage.googleapis.com
TLS handshake errors: "certificate verify failed"
Network captures showed TLS Client Hello without SNI extension

Root Cause: The connection flow resolved hostnames to IP addresses early in the process but lost the original hostname string needed for SNI. When INetworkConnections::connect(host, service, isTLS) was called:
- resolveTCPEndpoint() resolved hostname to IP addresses
- One IP was selected and addr.fromHostname = true was set
- The original hostname string was discarded
- connectExternal(addr) was called with only the IP address
- During TLS handshake, no SNI was set because the hostname was lost

Solution
Modified the network connection chain to preserve hostname information for SNI

Testing
Verification Methods:

Network capture: tshark -Y "tls.handshake.extensions_server_name" confirms SNI is now sent in TLS Client Hello
Trace logs: Added SSLConnectionWithHostname and SSLSetSNI events for debugging
Real-world test: Backup agent connecting to storage.googleapis.com now sends proper SNI

[x] The PR has a description, explaining both the problem and the solution.
[x] The description mentions which forms of testing were done and the testing seems reasonable.
[x] Every function/class/actor that was touched is reasonably well documented.

foundationdb-ci · 2025-09-16T15:51:43Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 3d045ff
Duration 0:04:13
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-09-16T15:52:02Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 3d045ff
Duration 0:04:32
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-09-16T15:52:03Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 3d045ff
Duration 0:04:30
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-09-16T15:52:04Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 3d045ff
Duration 0:04:33
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-09-16T15:52:06Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 3d045ff
Duration 0:04:33
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2025-09-16T16:27:25Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 3d045ff
Duration 0:39:56
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-09-16T16:44:54Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: b62a3cb
Duration 0:43:13
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-09-16T16:46:33Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: cb4f651
Duration 0:42:09
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-09-16T16:53:38Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: 3d045ff
Duration 1:06:05
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-09-16T16:53:50Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: cb4f651
Duration 0:49:21
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-09-16T17:14:25Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: b62a3cb
Duration 1:12:44
Result: ❌ FAILED
Error: Error while executing command: ctest -j ${NPROC} --no-compress-output -T test --output-on-failure. Reason: exit status 8
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-09-16T17:15:29Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: b62a3cb
Duration 1:13:44
Result: ❌ FAILED
Error: Error while executing command: ctest -j ${NPROC} --no-compress-output -T test --output-on-failure. Reason: exit status 8
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-09-16T17:16:06Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: b62a3cb
Duration 1:14:22
Result: ❌ FAILED
Error: Error while executing command: ctest -j ${NPROC} --no-compress-output -T test --output-on-failure. Reason: exit status 8
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

saintstack · 2025-09-16T17:16:21Z

These failed:

The following tests FAILED:
3 - bulkload_test (Failed)
12 - s3_backup_tests (Timeout)
Errors while running CTest

I have a fix coming for these transient failures... but not ready yet. Meantime, let me retry.

foundationdb-ci · 2025-09-16T17:26:59Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: b62a3cb
Duration 1:25:16
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2025-09-16T17:29:56Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: cb4f651
Duration 1:24:38
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2025-09-16T17:49:28Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: cb4f651
Duration 0:32:53
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-09-16T18:04:53Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: cb4f651
Duration 0:48:19
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-09-17T05:18:04Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: ef89d47
Duration 0:04:26
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2025-09-17T05:18:08Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: ef89d47
Duration 0:04:29
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-09-17T05:18:09Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: ef89d47
Duration 0:04:29
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-09-17T05:53:52Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: ef89d47
Duration 0:40:16
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-09-17T06:18:34Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: ef89d47
Duration 1:04:58
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

jzhou77

looks like a clang-format is needed.

evkuzin · 2025-09-17T21:07:54Z

@jzhou77 thanks for the hint :)

foundationdb-ci · 2025-09-18T04:09:14Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: b54d8cf
Duration 0:23:13
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-09-18T04:33:33Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: b54d8cf
Duration 0:47:28
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-09-18T04:50:10Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: b54d8cf
Duration 1:04:04
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-09-18T04:59:11Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: b54d8cf
Duration 1:13:09
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2025-09-18T04:59:31Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: b54d8cf
Duration 1:13:30
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-09-18T05:03:30Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: b54d8cf
Duration 1:17:28
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

jzhou77 · 2025-09-18T15:57:01Z

20250918-155625-jzhou-dab0cc3c453fcfef compressed=True data_size=40477684 duration=6080157 ended=100000 fail_fast=10 max_runs=100000 pass=100000 priority=100 remaining=0 runtime=1:02:14 sanity=False started=100000 stopped=20250918-165839 submitted=20250918-155625 timeout=5400 username=jzhou

* use sni in the tls handshake * Ran clang-format -i * Remove accidental overcommit * drop too spamy logs * clang-format -style=file -i flow/Net2.actor.cpp --------- Co-authored-by: stack <[email protected]>

* use sni in the tls handshake * Ran clang-format -i * Remove accidental overcommit * drop too spamy logs * clang-format -style=file -i flow/Net2.actor.cpp --------- Co-authored-by: Evgeny <[email protected]> Co-authored-by: stack <[email protected]>

* use sni in the tls handshake * Ran clang-format -i * Remove accidental overcommit * drop too spamy logs * clang-format -style=file -i flow/Net2.actor.cpp --------- Co-authored-by: stack <[email protected]>

* use sni in the tls handshake * Ran clang-format -i * Remove accidental overcommit * drop too spamy logs * clang-format -style=file -i flow/Net2.actor.cpp --------- Co-authored-by: Evgeny <[email protected]> Co-authored-by: stack <[email protected]>

* use sni in the tls handshake * Ran clang-format -i * Remove accidental overcommit * drop too spamy logs * clang-format -style=file -i flow/Net2.actor.cpp --------- Co-authored-by: stack <[email protected]> (cherry picked from commit 947276f)

* use sni in the tls handshake * Ran clang-format -i * Remove accidental overcommit * drop too spamy logs * clang-format -style=file -i flow/Net2.actor.cpp --------- (cherry picked from commit 947276f) Co-authored-by: Evgeny <[email protected]> Co-authored-by: stack <[email protected]>

FoundationDB sends TLS SNI only from 7.3.78 and 7.4.6 (apple/foundationdb#12374). The provisioned bucket endpoint is an ingress that routes by SNI and aborts the handshake without it, so the 7.3.63 binaries the operator ships, and the 7.3.63 backup_agent of a default cluster, fail with "Network connection failed" before any S3 request. Bump the operator's init containers, which supply the fdbbackup and fdbrestore binaries picked by the cluster's major.minor, to 7.3.79 and 7.4.8. Add minimumAgentVersions to the FoundationDB strategy template: a cluster on an older patch of a listed major.minor runs its backup_agent at the listed version instead. Patch releases of one minor are wire-compatible, so this needs no cluster upgrade. The default strategy lists the same two versions as the operator. Assisted-by: LLM Signed-off-by: Andrey Kolkov <[email protected]>

use sni in the tls handshake

3d045ff

evkuzin mentioned this pull request Sep 16, 2025

Fix TLS SNI (Server Name Indication) support for hostname-based connections #12366

Closed

3 tasks done

jzhou77 closed this Sep 16, 2025

jzhou77 reopened this Sep 16, 2025

jzhou77 requested a review from kakaiu September 16, 2025 15:47

stack added 2 commits September 16, 2025 09:00

Ran clang-format -i

b62a3cb

Remove accidental overcommit

cb4f651

saintstack approved these changes Sep 16, 2025

View reviewed changes

saintstack closed this Sep 16, 2025

saintstack reopened this Sep 16, 2025

kakaiu approved these changes Sep 16, 2025

View reviewed changes

jzhou77 reviewed Sep 16, 2025

View reviewed changes

Comment thread flow/Net2.actor.cpp Resolved

Comment thread flow/Net2.actor.cpp Resolved

Comment thread flow/Net2.actor.cpp Resolved

jzhou77 reviewed Sep 17, 2025

View reviewed changes

clang-format -style=file -i flow/Net2.actor.cpp

b54d8cf

jzhou77 closed this Sep 18, 2025

jzhou77 reopened this Sep 18, 2025

jzhou77 merged commit 947276f into apple:main Sep 18, 2025

jzhou77 mentioned this pull request Sep 22, 2025

use sni in the tls handshake [release-7.4] #12385

Merged

5 tasks

arnav-ag mentioned this pull request Apr 27, 2026

Backport #12374 to release-7.3 #13089

Merged

androndo mentioned this pull request Oct 14, 2026

fix(backups): make cozy-default FoundationDB backup and restore reach S3 cozystack/cozystack#4901

Open

12 tasks

Conversation

evkuzin commented Sep 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

foundationdb-ci commented Sep 16, 2025

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Sep 16, 2025

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Sep 16, 2025

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented Sep 16, 2025

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

foundationdb-ci commented Sep 16, 2025

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

foundationdb-ci commented Sep 16, 2025

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Sep 16, 2025

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Sep 16, 2025

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Sep 16, 2025

Result of foundationdb-pr-macos on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Sep 16, 2025

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Sep 16, 2025

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented Sep 16, 2025

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Sep 16, 2025

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

saintstack commented Sep 16, 2025

Uh oh!

foundationdb-ci commented Sep 16, 2025

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

foundationdb-ci commented Sep 16, 2025

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

foundationdb-ci commented Sep 16, 2025

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Sep 16, 2025

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

Uh oh!

Uh oh!

Uh oh!

foundationdb-ci commented Sep 17, 2025

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

foundationdb-ci commented Sep 17, 2025

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented Sep 17, 2025

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Sep 17, 2025

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Sep 17, 2025

Result of foundationdb-pr-macos on macOS Ventura 13.x

Uh oh!

jzhou77 left a comment

Choose a reason for hiding this comment

Uh oh!

evkuzin commented Sep 17, 2025

Uh oh!

evkuzin commented Sep 16, 2025 •

edited

Loading

jzhou77 commented Sep 18, 2025 •

edited

Loading