Skip to content

Commit 9379729

Browse files
androndoclaude
andcommitted
fix(clickhouse): address review notes on endpointCA and the CA copy
Review notes from IvanHunters on #3385 (LGTM with non-blocking findings). 1. The backup-s3-ca volume carries no `optional: true` while the adjacent useSystemBucket credential refs deliberately do, and the asymmetry was undocumented — an operator naming a Secret that does not exist gets the whole ClickHouse Pod stuck in ContainerCreating, not just the sidecar. Kept the fail-closed behaviour and wrote down why the two cases differ: credentials are projected by the platform after install, so a strict ref would wedge every fresh release until the first BackupJob, whereas this CA is named by the tenant, must already exist, and is trust material — optional would mount an empty dir, the Pod would look healthy, and the first backup would fail deep inside clickhouse-backup with an x509 error naming neither the Secret nor the chart. The comment also states the blast radius plainly. 2. The CA probe swallowed stderr, so an RBAC denial on the cross-namespace read was reported as "CA not found" followed by advice (set S3_CA_SECRET) that cannot fix a permissions problem. It now separates present / absent / not-allowed-to-look and says so, and the README no longer labels step 03 simply "tenant": copying the CA reads a Secret in tenant-root, which is a platform capability. Added a section spelling out the three ways to satisfy that (admin credentials, a one-off admin copy, or skipping the CA entirely on a publicly-trusted endpoint). 3. Caveat, and a usability regression this branch introduced: on a cluster with a publicly-trusted endpoint and no seaweedfs, step 03 hard-exited on a default the user never chose. It now distinguishes an explicitly named Secret (still an error when missing — a typo should not be guessed past) from the unset default, where it warns and continues with backup.endpointCA unset. Verified all four paths against a fake kubectl: Forbidden, absent+default, absent+explicit, and present. 4. Caveat: endpointCA reaches the clickhouse-backup sidecar only. The legacy `schedule` CronJob writes to S3 through restic and does not consume it, so a private-CA endpoint combined with `schedule` still fails TLS. Documented on the field itself, so it lands in values.schema.json, the README and the cozyrds openAPISchema. The PR body's claim about select-e2e.sh/select-install.sh mappings was stale after the restructure that removed them; corrected there rather than in code. Assisted-By: Claude <[email protected]> Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Signed-off-by: Andrey Kolkov <[email protected]>
1 parent 0c14d6c commit 9379729

9 files changed

Lines changed: 96 additions & 32 deletions

File tree

‎api/apps/v1alpha1/clickhouse/types.go‎

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎examples/backups/clickhouse/00-helpers.sh‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,13 @@ export RESTOREJOB_TOCOPY_NAME="${RESTOREJOB_TOCOPY_NAME:-clickhouse-restore-to-c
3838
# silently break the copy. On a cluster whose S3 endpoint is signed by a
3939
# publicly-trusted CA, set S3_CA_SECRET="" to skip the copy and leave
4040
# backup.endpointCA unset.
41+
# Whether the CA Secret name is the user's choice or just this default. Step 03
42+
# treats the two differently: a name the user typed and that does not exist is an
43+
# error worth stopping on, while the default failing to resolve on a cluster with
44+
# no seaweedfs most likely means a publicly-trusted S3 endpoint — a legal input,
45+
# so it warns and leaves backup.endpointCA unset instead of failing.
46+
export S3_CA_SECRET_EXPLICIT="${S3_CA_SECRET:+1}"
47+
export S3_CA_SECRET_EXPLICIT="${S3_CA_SECRET_EXPLICIT:-0}"
4148
export S3_CA_SECRET="${S3_CA_SECRET:-seaweedfs-ca-cert}"
4249
# Independent of NAMESPACE: the shared seaweedfs and its CA live in tenant-root
4350
# even when the demo runs elsewhere.

‎examples/backups/clickhouse/03-create-bucket.sh‎

Lines changed: 27 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -64,7 +64,19 @@ done
6464
# secretName. Leave S3_CA_SECRET empty to skip the copy on a public-CA endpoint.
6565
CH_BACKUP_CA_SECRET=""
6666
if [[ -n "$S3_CA_SECRET" ]]; then
67-
if ! kubectl -n "$S3_CA_NAMESPACE" get secret "$S3_CA_SECRET" >/dev/null 2>&1; then
67+
# Reading the platform's CA lives in S3_CA_NAMESPACE (tenant-root by
68+
# default), which is an admin/platform capability, not a tenant one — see
69+
# the note on step 03 in README.md. So the three outcomes have to be kept
70+
# apart: present, absent, and not-allowed-to-look. Swallowing stderr here
71+
# would report an RBAC denial as "not found" and then advise setting
72+
# S3_CA_SECRET, which cannot fix a permissions problem.
73+
ca_probe_err=""
74+
if ! ca_probe_err=$(kubectl -n "$S3_CA_NAMESPACE" get secret "$S3_CA_SECRET" -o name 2>&1 >/dev/null); then
75+
if echo "$ca_probe_err" | grep -qiE 'forbidden|cannot (get|list)|Unauthorized'; then
76+
log_error "Cannot read ${S3_CA_NAMESPACE}/${S3_CA_SECRET}: ${ca_probe_err}"
77+
log_error "This is a permissions problem, not a missing Secret: copying the S3 endpoint CA needs read access to ${S3_CA_NAMESPACE}. Ask an admin to copy it into ${NAMESPACE}/${CH_CA_SECRET_NAME}, then re-run with S3_CA_SECRET=\"\" to skip the copy."
78+
exit 1
79+
fi
6880
log_warning "S3 CA secret ${S3_CA_NAMESPACE}/${S3_CA_SECRET} not found; discovering the seaweedfs CA Certificate..."
6981
# List every seaweedfs Certificate as "<isCA> <secretName>" and pick the
7082
# CA one in shell — avoids kubectl jsonpath's finicky boolean-literal
@@ -76,11 +88,23 @@ if [[ -n "$S3_CA_SECRET" ]]; then
7688
if [[ -n "$DISCOVERED_CA" ]]; then
7789
log_success "Discovered seaweedfs CA secret ${S3_CA_NAMESPACE}/${DISCOVERED_CA}"
7890
S3_CA_SECRET="$DISCOVERED_CA"
79-
else
80-
log_error "No seaweedfs CA Certificate found in ${S3_CA_NAMESPACE}; set S3_CA_SECRET explicitly (or empty for a public-CA endpoint)."
91+
elif [[ "${S3_CA_SECRET_EXPLICIT}" == "1" ]]; then
92+
# The user named a specific Secret and it is not there: that is a
93+
# typo or a missing prerequisite, and guessing past it would hide it.
94+
log_error "S3 CA secret ${S3_CA_NAMESPACE}/${S3_CA_SECRET} does not exist and no seaweedfs CA Certificate was found in ${S3_CA_NAMESPACE}. Fix the name, or set S3_CA_SECRET=\"\" for a publicly-trusted endpoint."
8195
exit 1
96+
else
97+
# Nothing was named and there is no seaweedfs here, so this is most
98+
# likely a cluster whose S3 endpoint is publicly trusted — a legal
99+
# input for this demo. Warn and continue with endpointCA unset
100+
# rather than failing on a default the user never chose.
101+
log_warning "No seaweedfs CA found in ${S3_CA_NAMESPACE} and S3_CA_SECRET was not set explicitly."
102+
log_warning "Continuing with backup.endpointCA unset — correct for a publicly-trusted S3 endpoint. If your endpoint uses a private CA, set S3_CA_SECRET to the Secret holding its bundle."
103+
S3_CA_SECRET=""
82104
fi
83105
fi
106+
fi
107+
if [[ -n "$S3_CA_SECRET" ]]; then
84108
log_substep "Copying S3 CA ${S3_CA_NAMESPACE}/${S3_CA_SECRET}[${S3_CA_KEY}] -> ${CH_CA_SECRET_NAME}..."
85109
# The dot in the default key has to be escaped for kubectl jsonpath.
86110
ca_pem=$(kubectl -n "$S3_CA_NAMESPACE" get secret "$S3_CA_SECRET" \

‎examples/backups/clickhouse/README.md‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@ that POSTs to the sidecar and polls the action log.
2727
| `00-helpers.sh` | Shared bash helpers and env defaults; sourced by every step. | n/a |
2828
| `01-create-strategy.sh` | Creates the cluster-scoped `Altinity` strategy that wraps `clickhouse-backup`. | admin |
2929
| `02-create-backupclass.sh` | Maps `apps.cozystack.io/ClickHouse` to that strategy. | admin |
30-
| `03-create-bucket.sh` | Provisions a `Bucket`, caches its S3 coordinates into `.bucket-info.env` (chmod 600; raw access keys), and copies the S3 endpoint CA into a per-release Secret so the sidecar can verify a self-signed endpoint. `cleanup.sh` removes both. | tenant |
30+
| `03-create-bucket.sh` | Provisions a `Bucket`, caches its S3 coordinates into `.bucket-info.env` (chmod 600; raw access keys), and copies the S3 endpoint CA into a per-release Secret so the sidecar can verify a self-signed endpoint. `cleanup.sh` removes both. | tenant, **plus admin** for the CA copy — see below |
3131
| `04-create-clickhouse.sh` | Provisions a `ClickHouse` instance with `backup.enabled=true` (chart emits the backup-s3 Secret + sidecar) and writes a sentinel row. | tenant |
3232
| `05-create-backupjob.sh` | Submits a `BackupJob` and waits for Succeeded. | tenant |
3333
| `06-restore-in-place.sh` | Drops the sentinel and restores into the same instance via `RestoreJob`. | tenant |
@@ -64,4 +64,14 @@ All variables come from `00-helpers.sh`:
6464
- Cozystack cluster with the backup-controller and backupstrategy-controller installed.
6565
- `kubectl`, `jq`, and (for `04`+) the ClickHouse operator deployed by the chart.
6666

67+
### Who can run step 03
68+
69+
Step 03 is otherwise a tenant action, but copying the S3 endpoint CA reads a Secret in `S3_CA_NAMESPACE` (`tenant-root` by default), which is a platform/admin capability rather than a tenant one. Three ways out, in order of preference:
70+
71+
- run step 03 with credentials that can read that namespace (the admin path this demo assumes);
72+
- have an admin copy the CA into the application namespace once, then run the demo with `S3_CA_SECRET=""` and set `backup.endpointCA` to that Secret by hand;
73+
- skip it entirely with `S3_CA_SECRET=""` when the S3 endpoint's certificate is signed by a publicly-trusted CA — nothing needs a CA bundle then.
74+
75+
The script distinguishes the three outcomes so a permissions problem does not present as a missing Secret: it stops with an explicit permissions message on `Forbidden`, and on a genuinely absent default (no `seaweedfs-ca-cert`, no seaweedfs `Certificate` to discover, and `S3_CA_SECRET` not set explicitly) it warns and continues with `backup.endpointCA` unset instead of failing — which is the right outcome on a cluster whose endpoint is publicly trusted. Naming a Secret that does not exist is still an error.
76+
6777
[altinity]: https://github.com/Altinity/clickhouse-backup

0 commit comments

Comments
 (0)