Skip to content

Commit 14eab3c

Browse files
authored
fix(backups): restrict the kafka-metadata CLI Pod and correct demo personas (cozystack#4695)
## What this PR does This fixes two problems in the Kafka and NATS backup demos under `examples/backups/`. The Kafka metadata demo ran its CLI Pod without any securityContext, so a namespace that enforces the restricted Pod Security level would reject it. The Pod now gets the same restricted context as the CLI Pod in the topic-data demo: `runAsNonRoot`, the `RuntimeDefault` seccomp profile, no privilege escalation, all capabilities dropped. The new bats case reads the overrides passed to `kubectl run` and fails if any of those fields is missing or the patch is not sent as `--override-type=json`. It is red on main, and it goes red on each of seven single-field mutations of the patch while the other five cases stay green. One known limit: `runAsNonRoot` needs an image that runs as a non-root user. The default Strimzi image runs as UID 1001 and is fine. If `KAFKA_IMAGE` is overridden with an image that runs as root, the kubelet refuses to start the container, and the helper fails only when its 5 minute `kubectl wait` times out. The kafka and nats walkthroughs told a tenant to run scripts that the tenant roles forbid. Steps 03, 04, 06, 07 and `cleanup.sh` were in the tenant column, and the scenario docs and several script comments had the tenant create the backup Secret. Those rows now say admin and name the part a tenant admin can do on its own, and the scenario docs say the same. After this change the persona column in both tables matches the tenant roles. kafka-metadata has no persona table, so I didn't change anything there. These are the tenant rules on pods and backup objects, plus one example of the per-app secret rules. Each app chart's dashboard Role allows `get`, `list` and `watch` on a few named Secrets and no other verb on secrets. Nothing aggregated into the tenant roles grants `pods/exec`, `pods/log`, `helmreleases`, `objectstorage.k8s.io`, `statefulsets` or `kafkas.kafka.strimzi.io`. ```yaml # packages/system/cozystack-basics/templates/clusterroles.yaml:100-112 (cozy:tenant:view:base, every tenant level) - apiGroups: - "" resources: - pods - services - persistentvolumes - endpoints - events - resourcequotas verbs: - get - list - watch # packages/system/cozystack-basics/templates/clusterroles.yaml:238-246 (cozy:tenant:admin:base) - apiGroups: [""] resources: - pods - services - persistentvolumes - endpoints - events verbs: - delete # packages/system/backup-controller/templates/tenant-clusterroles.yaml:35-44 (cozy:backups:admin) - apiGroups: ["backups.cozystack.io"] resources: - plans - backupjobs - restorejobs verbs: - create - update - patch - delete # packages/apps/bucket/templates/dashboard-resourcemap.yaml:6-14 (bound at the tenant "use" level) - apiGroups: - "" resources: - secrets resourceNames: {{- range $name, $user := .Values.users }} - {{ $.Release.Name }}-{{ $name }}-credentials {{- end }} verbs: ["get", "list", "watch"] ``` How each changed row runs into those rules: | Row | What the script needs | Why a tenant can't | | --- | --- | --- | | 03 (kafka, nats) | waits on the `HelmRelease` and COSI objects, `get` on the `bucket-<name>-backup` Secret, kafka also reads the CA Secret in `tenant-root` | no `helmreleases` or `objectstorage.k8s.io` rule; the bucket Role covers only `<release>-<user>-credentials` | | 04 (kafka, nats) | waits on the `HelmRelease` and the Strimzi `Kafka` or the `StatefulSet`, `create` on `secrets`, `create` on `pods`, `pods/exec` | pods are `get`/`list`/`watch`/`delete` only, and nothing grants `create` on secrets | | 06 (kafka, nats) | `create` on `pods` and `pods/exec` for the CLI Pod, kafka also `get` on `pods/log` for the refused restore | same pod rules, and nothing grants `pods/log` | | 07 (kafka, nats) | same as 04 | same as 04 | | `cleanup.sh` (kafka, nats) | `delete` on `secrets`, on `backups`, and on the cluster-scoped `BackupClass` and strategy | `cozy:backups:admin` deletes only plans, backupjobs and restorejobs, and no tenant rule covers secrets or cluster-scoped objects | Step 05 stays a tenant step. Its `BackupJob` comes from `cozy:backups:admin`, so in practice that means a tenant admin. The nats demo's CLI Pod has the same missing securityContext as the kafka-metadata one, and I left it for a separate issue. Closes cozystack#4625 Closes cozystack#4636 ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> Not a UI change. ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(backups): the kafka-metadata backup demo now runs its CLI Pod with the restricted Pod Security context, and the kafka and nats backup demos now list steps 03, 04, 06, 07 and cleanup as admin steps, because they need Pod, exec, Secret and cluster-scoped permissions that tenant roles do not grant. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Security** * Kafka metadata backup CLI Pods are recreated with restricted security settings when they are not running or lack the required non-root configuration. * **Documentation** * Kafka and NATS guides clarify which setup, restore, and cleanup steps require administrator access and which job actions tenants can perform. * Updated examples clarify that the demo provides required storage Secrets and certificates. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 1621c47 + 534f8f1 commit 14eab3c

16 files changed

Lines changed: 93 additions & 51 deletions

‎examples/backups/kafka-metadata/00-helpers.sh‎

Lines changed: 16 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -150,23 +150,35 @@ kafka_wait_ready() {
150150
# Ensure the long-lived kafka-cli Pod exists and is Ready, so kafka_run can exec
151151
# into it. Idempotent: a Ready Pod this demo owns is reused across calls and
152152
# across the numbered demo scripts; a leftover in a terminal phase (Succeeded /
153-
# Failed) is replaced rather than waited on; and a same-named Pod this demo does
154-
# not own is refused rather than hijacked or deleted. cleanup.sh removes it.
153+
# Failed) is replaced rather than waited on, and so is one started before the
154+
# Pod ran under the restricted profile, since a Pod's securityContext cannot be
155+
# changed in place; and a same-named Pod this demo does not own is refused
156+
# rather than hijacked or deleted. cleanup.sh removes it.
155157
# Every step returns on failure explicitly: this runs as the left side of
156158
# `|| return 1`, and some callers also wrap it in $(...), so errexit never
157159
# applies inside it and a bare failure would carry on to the next step.
158160
kafka_cli_pod() {
159-
local phase owner
161+
local phase owner nonroot overrides
160162
phase=$(kubectl -n "$NAMESPACE" get pod "$KAFKA_CLI_POD" --ignore-not-found -o jsonpath='{.status.phase}') || return 1
161163
owner=$(kubectl -n "$NAMESPACE" get pod "$KAFKA_CLI_POD" --ignore-not-found -o jsonpath='{.metadata.labels.cozystack\.io/backup-demo}') || return 1
162164
if [ -n "$phase" ] && [ "$owner" != "kafka-metadata" ]; then
163165
log_error "Pod $NAMESPACE/$KAFKA_CLI_POD exists but this demo does not own it; refusing to use or delete it"
164166
return 1
165167
fi
166-
if [ "$phase" != "Running" ] && [ "$phase" != "Pending" ]; then
168+
# Only this helper creates the Pod, and it sets every restricted field at
169+
# once, so one of them tells a current Pod from an older one.
170+
nonroot=$(kubectl -n "$NAMESPACE" get pod "$KAFKA_CLI_POD" --ignore-not-found -o jsonpath='{.spec.securityContext.runAsNonRoot}') || return 1
171+
if { [ "$phase" != "Running" ] && [ "$phase" != "Pending" ]; } || [ "$nonroot" != "true" ]; then
172+
# The Strimzi image runs as non-root UID 1001, so runAsNonRoot and
173+
# seccomp at the Pod level plus allowPrivilegeEscalation=false and
174+
# drop-ALL on the container satisfy PSA "restricted" without pinning a
175+
# runAsUser specific to this image tag. The add-only JSON patch leaves
176+
# the run-generated Pod's image, command and args untouched.
177+
overrides='[{"op":"add","path":"/spec/securityContext","value":{"runAsNonRoot":true,"seccompProfile":{"type":"RuntimeDefault"}}},{"op":"add","path":"/spec/containers/0/securityContext","value":{"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]}}}]'
167178
kubectl -n "$NAMESPACE" delete pod "$KAFKA_CLI_POD" --grace-period=1 --ignore-not-found >/dev/null || return 1
168179
kubectl -n "$NAMESPACE" run "$KAFKA_CLI_POD" --image="$KAFKA_IMAGE" \
169180
--labels=cozystack.io/backup-demo=kafka-metadata \
181+
--override-type=json --overrides="$overrides" \
170182
--restart=Never --command -- sleep infinity >/dev/null || return 1
171183
fi
172184
kubectl -n "$NAMESPACE" wait --for=condition=Ready "pod/$KAFKA_CLI_POD" \

‎examples/backups/kafka/00-helpers.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -309,7 +309,7 @@ topic_replication_factor() {
309309
# Create the "<app>-backup-s3" Secret the Job strategy Pod consumes, from the
310310
# bucket coordinates cached by 03-create-bucket.sh. The generic Job strategy -
311311
# unlike the app-specific drivers - has no chart support to emit this Secret,
312-
# so the tenant provides it. Called for the source app (step 04) and the
312+
# so the demo provides it. Called for the source app (step 04) and the
313313
# restore target (step 07). The S3 endpoint's CA, when 03-create-bucket.sh
314314
# cached one, rides along as ca.crt: the strategy Pod mounts that key and
315315
# verifies the endpoint against it instead of disabling verification.

‎examples/backups/kafka/01-create-strategy.sh‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -82,7 +82,7 @@ spec:
8282
value: "{{ .Release.Namespace }}"
8383
- name: MODE
8484
value: "{{ .Mode }}"
85-
# S3 coordinates from the tenant-provided <release>-backup-s3 Secret
85+
# S3 coordinates from the demo-provided <release>-backup-s3 Secret
8686
# (created by create_s3_secret in 00-helpers.sh from the Bucket).
8787
- name: AWS_ACCESS_KEY_ID
8888
valueFrom:
@@ -169,7 +169,7 @@ spec:
169169
170170
# curl --aws-sigv4 signs the request (SigV4) so no separate S3
171171
# client image is needed. The certificate is verified: against the
172-
# CA projected from the Secret when the tenant supplied one, else
172+
# CA projected from the Secret when it carries one, else
173173
# against the image's trust store. \${CA_OPT} and \${CONNECT_TO}
174174
# are deliberately unquoted so an empty value expands to no
175175
# argument.

‎examples/backups/kafka/03-create-bucket.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
#!/bin/bash
22
# Step 03: Provision an in-cluster Bucket and cache its S3 coordinates. The
33
# generic Job strategy has no chart support to emit a backup Secret, so the
4-
# tenant materialises the "<app>-backup-s3" Secret itself. This step only
4+
# demo materialises the "<app>-backup-s3" Secret itself. This step only
55
# caches the coordinates; create_s3_secret (00-helpers.sh) turns them into
66
# per-app Secrets in steps 04 and 07.
77
set -euo pipefail

‎examples/backups/kafka/90-scenario-admin-prepare.md‎

Lines changed: 2 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,14 @@
11
# Scenario: Cluster admin prepares Kafka topic-data backups
22

3-
A cluster administrator performs the cluster-level preparation once. Tenants
4-
will then be able to back up and restore the topic data of their Kafka
5-
applications by referencing the BackupClass created here, with no further
6-
admin action.
3+
A cluster administrator performs the cluster-level preparation once. Tenant admins can then back up and restore the topic data of their Kafka applications with a `BackupJob` or `RestoreJob` that references the BackupClass created here. The demo scripts around those jobs still need admin access, because they create Secrets and work through a CLI Pod; the README's step table says which part of each step a tenant admin can do.
74

85
## Prerequisites
96

107
- A running Cozystack cluster with the backup-controller and
118
backupstrategy-controller installed.
129
- `kubectl` access with permissions to create cluster-scoped
1310
`Job.strategy.backups.cozystack.io` and `BackupClass.backups.cozystack.io`.
14-
- Reachable S3-compatible storage. The demo uses the in-cluster `Bucket` app
15-
(step 03); the tenant turns its coordinates and the endpoint's CA into a
16-
`<app>-backup-s3` Secret that the strategy Pod reads (step 04 / 07).
11+
- Reachable S3-compatible storage. The demo uses the in-cluster `Bucket` app (step 03), and steps 04 and 07 turn its coordinates and the endpoint's CA into a `<app>-backup-s3` Secret that the strategy Pod reads. Creating that Secret needs admin access, since no tenant role grants `create` on `secrets`.
1712

1813
## Steps
1914

‎examples/backups/kafka/91-scenario-user-backup.md‎

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2,14 +2,13 @@
22

33
This scenario assumes the admin steps (`01-...`, `02-...`) have already run.
44

5+
The user's own action here, at the tenant admin level, is the `BackupJob` in step 05, plus creating the `Bucket` and the `Kafka`. Steps 03 and 04 as scripted need admin access: they read and create Secrets, wait on HelmReleases and operator resources, and seed the topic through a CLI Pod, which no tenant role allows. The README's step table has the details.
6+
57
## Prerequisites
68

79
- Tenant namespace (default: `tenant-root`, the tenant whose Pods can reach
810
the in-cluster seaweedfs; see the README).
9-
- A Cozystack `Bucket` (step 03) and the `<app>-backup-s3` Secret derived from
10-
it (step 04). The generic `Job` strategy has no chart support to emit this
11-
Secret, so the tenant creates it from the bucket's `BucketInfo` plus the S3
12-
endpoint's CA (see `create_s3_secret` in `00-helpers.sh`).
11+
- A Cozystack `Bucket` (step 03) and the `<app>-backup-s3` Secret derived from it (step 04). The generic `Job` strategy has no chart support to emit this Secret, so step 04 creates it from the bucket's `BucketInfo` plus the S3 endpoint's CA (see `create_s3_secret` in `00-helpers.sh`). Creating it needs admin access.
1312
- A `Kafka` application with at least one plain (neither compacted nor
1413
transactional) topic holding data.
1514

‎examples/backups/kafka/92-scenario-user-restore.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,8 @@ Two restore variants are supported. Both append: the strategy recreates a
44
topic only if it is absent and then replays every record, so it refuses a
55
target topic that already holds data.
66

7+
The user's own action in both variants, at the tenant admin level, is the `RestoreJob`, plus the second `Kafka` in variant B. The scripts as written need admin access: variant A deletes and checks the topic through a CLI Pod and reads the refused restore's Pod log, and variant B checks the topic the same way, creates the target's Secret and waits on its `HelmRelease` and Strimzi `Kafka`, none of which a tenant role allows. The README's step table has the details.
8+
79
## A. In-place restore
810

911
Restore back into the same Kafka application.

‎examples/backups/kafka/README.md‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -36,15 +36,15 @@ For a faithful, offset-preserving Kafka backup, prefer a volume-snapshot strateg
3636
| `00-helpers.sh` | Shared bash helpers, env defaults, and the kafka-CLI / S3-secret helpers; sourced by every step. | n/a |
3737
| `01-create-strategy.sh` | Creates the cluster-scoped `Job` strategy (the Kafka backup/restore `PodTemplateSpec`). | admin |
3838
| `02-create-backupclass.sh` | Maps `apps.cozystack.io/Kafka` to that strategy, with the `topics` parameter (and the optional `replicationFactor` override). | admin |
39-
| `03-create-bucket.sh` | Provisions a `Bucket` and caches its S3 coordinates plus the endpoint's CA into `.bucket-info.env` (chmod 600; raw access keys). `cleanup.sh` removes this file. | tenant, **plus admin** for the CA copy: it reads a Secret in `tenant-root`, which a tenant persona is not granted (see the TLS bullet above) |
40-
| `04-create-kafka.sh` | Provisions a two-broker `Kafka`, creates the `<app>-backup-s3` Secret, seeds a topic (replication factor 2) with sentinel messages. | tenant |
39+
| `03-create-bucket.sh` | Provisions a `Bucket` and caches its S3 coordinates plus the endpoint's CA into `.bucket-info.env` (chmod 600; raw access keys). `cleanup.sh` removes this file. | **admin**. A tenant admin can create the `Bucket` itself, but the script also waits on its `HelmRelease` and COSI objects, reads the `bucket-<name>-backup` Secret, and copies the CA from a Secret in `tenant-root` (see the TLS bullet above); no tenant role grants access to `helmreleases` or `objectstorage.k8s.io`, or `get` on those Secrets |
40+
| `04-create-kafka.sh` | Provisions a two-broker `Kafka`, creates the `<app>-backup-s3` Secret, seeds a topic (replication factor 2) with sentinel messages. | **admin**. A tenant admin can create the `Kafka` itself, but the script also waits on its `HelmRelease` and Strimzi `Kafka`, creates the Secret, and seeds the topic through a CLI Pod it execs into; no tenant role grants access to `helmreleases` or `kafkas.kafka.strimzi.io`, `create` on `secrets` or `pods`, or `pods/exec` |
4141
| `05-create-backupjob.sh` | Submits a `BackupJob`, waits for Succeeded and caches the resolved `Backup` name into `.backup-name.env`. | tenant |
42-
| `06-restore-in-place.sh` | Deletes the topic and restores it into the same instance via `RestoreJob`, then submits a second `RestoreJob` that must be refused because the topic is no longer empty. | tenant |
43-
| `07-restore-to-copy.sh` | Provisions a second two-broker `Kafka` and restores into it via `RestoreJob.spec.targetApplicationRef`. | tenant |
44-
| `cleanup.sh` | Removes everything created by the demo. | admin or tenant |
42+
| `06-restore-in-place.sh` | Deletes the topic and restores it into the same instance via `RestoreJob`, then submits a second `RestoreJob` that must be refused because the topic is no longer empty. | **admin**. A tenant admin can submit the two `RestoreJob`s, but the script deletes and checks the topic through the CLI Pod and reads the refusal from the restore Pod's log; no tenant role grants `create` on `pods` or `pods/exec`, or `get` on `pods/log` |
43+
| `07-restore-to-copy.sh` | Provisions a second two-broker `Kafka` and restores into it via `RestoreJob.spec.targetApplicationRef`. | **admin**, for the same reasons as step 04. The tenant admin's part is the second `Kafka` and the `RestoreJob` |
44+
| `cleanup.sh` | Removes everything created by the demo. | **admin**: it deletes Secrets, the `Backup`, and the cluster-scoped `BackupClass` and strategy, and no tenant role grants `delete` on any of them |
4545
| `run-all.sh` | Convenience runner that executes 01..07 in order. | demo |
4646

47-
The scenario walk-throughs `90-scenario-admin-prepare.md`, `91-scenario-user-backup.md` and `92-scenario-user-restore.md` describe the same steps from the admin's and the tenant's side.
47+
The scenario walk-throughs `90-scenario-admin-prepare.md`, `91-scenario-user-backup.md` and `92-scenario-user-restore.md` describe the same steps from the admin's and the tenant's side, and say which parts of the tenant's side need admin access.
4848

4949
## Running
5050

‎examples/backups/nats/00-helpers.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -131,7 +131,7 @@ stream_message_count() {
131131
# Create the "<app>-backup-s3" Secret the Job strategy Pod consumes, from the
132132
# bucket coordinates cached by 03-create-bucket.sh. The generic Job strategy -
133133
# unlike the app-specific drivers - has no chart support to emit this Secret,
134-
# so the tenant provides it. Called for the source app (step 04) and the
134+
# so the demo provides it. Called for the source app (step 04) and the
135135
# restore target (step 07).
136136
create_s3_secret() {
137137
local app="$1"

‎examples/backups/nats/01-create-strategy.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -64,7 +64,7 @@ spec:
6464
value: "{{ if .Backup }}{{ .Backup.ApplicationRef.Name }}{{ end }}"
6565
- name: MODE
6666
value: "{{ .Mode }}"
67-
# S3 coordinates from the tenant-provided <release>-backup-s3 Secret
67+
# S3 coordinates from the demo-provided <release>-backup-s3 Secret
6868
# (created by create_s3_secret in 00-helpers.sh from the Bucket).
6969
- name: AWS_ACCESS_KEY_ID
7070
valueFrom:

0 commit comments

Comments
 (0)