Skip to content

Commit c901a5e

Browse files
Andrey Kolkovandrondo
authored andcommitted
fix(kafka): sign S3 host without non-default port in backup strategy
The Strimzi image ships curl 7.76.1, whose --aws-sigv4 omits a non-default port from the SigV4 canonical Host while still sending it in the Host header (a bug fixed in curl 7.87.0). seaweedfs recomputes the signature from the received host:port, so every PUT/GET to the :8333 in-cluster S3 Service the e2e harness targets was rejected with HTTP 403, failing kafka-2-backup-roundtrip at the backup upload. It passed on dev7 only because that run used a :443 ingress, where the port is absent from the Host anyway. Strip the port from the signed and sent URL and redirect the connection to the real port with --connect-to, so the signed, sent and server-side Host all agree on the bare host. Verified against the image's curl 7.76.1: the emitted SigV4 signature matches a portless canonical Host, and the request still reaches the :8333 listener. Correct on curl >= 7.87 too, which drops the default port from the canonical Host as well. Signed-off-by: Andrey Kolkov <[email protected]>
1 parent b92c438 commit c901a5e

1 file changed

Lines changed: 26 additions & 3 deletions

File tree

‎examples/backups/kafka/01-create-strategy.sh‎

Lines changed: 26 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -113,15 +113,38 @@ spec:
113113
*) BASE="https://\${S3_ENDPOINT}" ;;
114114
esac
115115
116+
# Split the endpoint into scheme/host/port. The Strimzi image ships
117+
# curl 7.76.1, whose --aws-sigv4 has a bug fixed only in 7.87.0: it
118+
# omits a non-default port from the SigV4 canonical Host even though
119+
# it sends the port in the Host header. seaweedfs recomputes the
120+
# signature from the received "host:port" and so rejects the PUT
121+
# with 403 whenever the endpoint carries a non-default port (the
122+
# e2e harness points S3_ENDPOINT at the :8333 in-cluster Service).
123+
# Strip the port from the signed/sent URL and redirect the
124+
# connection to the real port with --connect-to, so the signed,
125+
# sent and server-side Host all agree on the bare host. This is
126+
# also correct on curl >= 7.87, which drops the default port too.
127+
SCHEME="\${BASE%%://*}"
128+
HOSTPORT="\${BASE#*://}"; HOSTPORT="\${HOSTPORT%%/*}"
129+
HOST="\${HOSTPORT%%:*}"
130+
PORT="\${HOSTPORT##*:}"
131+
if [ "\${PORT}" = "\${HOSTPORT}" ]; then PORT=""; fi
132+
if [ "\${SCHEME}" = http ]; then DEFPORT=80; else DEFPORT=443; fi
133+
CONNECT_TO=""
134+
if [ -n "\${PORT}" ] && [ "\${PORT}" != "\${DEFPORT}" ]; then
135+
CONNECT_TO="--connect-to \${HOST}:\${DEFPORT}:\${HOST}:\${PORT}"
136+
fi
137+
116138
if [ "\${MODE}" = backup ]; then SRC="\${SRC_BACKUP}"; else SRC="\${SRC_RESTORE}"; fi
117139
KEY="\${SRC}/kafka-topics.tar"
118-
OBJ_URL="\${BASE}/\${S3_BUCKET}/\${KEY}"
140+
OBJ_URL="\${SCHEME}://\${HOST}/\${S3_BUCKET}/\${KEY}"
119141
120142
# curl --aws-sigv4 signs the request (SigV4) so no separate S3
121143
# client image is needed. -k accepts seaweedfs's internal
122144
# self-signed cert; a production strategy would mount the tenant
123-
# CA and drop -k.
124-
s3() { curl -fsS -k --aws-sigv4 "aws:amz:\${S3_REGION}:s3" --user "\${AWS_ACCESS_KEY_ID}:\${AWS_SECRET_ACCESS_KEY}" "\$@"; }
145+
# CA and drop -k. \${CONNECT_TO} is deliberately unquoted so an
146+
# empty value expands to no argument.
147+
s3() { curl -fsS -k \${CONNECT_TO} --aws-sigv4 "aws:amz:\${S3_REGION}:s3" --user "\${AWS_ACCESS_KEY_ID}:\${AWS_SECRET_ACCESS_KEY}" "\$@"; }
125148
126149
WORK=/tmp/kafka-dump
127150
rm -rf "\${WORK}"; mkdir -p "\${WORK}"

0 commit comments

Comments
 (0)