Skip to content

Commit 72c4eb9

Browse files
authored
feat(backups): implement Job backup strategy (#1721)
## What Implements the `strategy.backups.cozystack.io/Job` backup strategy — a generic, application-agnostic driver that runs a user-supplied Kubernetes Job to create (and restore) a backup. ## Background This PR began as a draft written against an earlier shape of the backups API. The API has since moved to a `BackupClass`-based dispatch model, which broke the original draft on three counts: - `BackupJob.Spec.StrategyRef` and `BackupJob.Spec.StorageRef` were removed; the strategy and its parameters are now resolved from a `BackupClass` into a `ResolvedBackupConfig`. - Strategy handling moved from standalone per-strategy reconcilers to a dispatch model: `BackupJobReconciler.Reconcile` / `RestoreJobReconciler.Reconcile` resolve the strategy and switch into per-strategy methods (`reconcileVelero`, `reconcileCNPG`, `reconcileAltinity`, …). - The draft's template context (`.Values`, `.Storage`) diverged from the convention the other strategies adopted, and its `.Storage` half targeted a storage API that does not exist. ## What changed - **`reconcileJob` / `reconcileJobRestore`** (`internal/backupcontroller/jobstrategy_controller.go`): replaces the standalone `BackupJobStrategyReconciler` with the two dispatch methods the `BackupJob` / `RestoreJob` reconcilers call. The Job strategy is the generic counterpart of the Altinity driver — same lifecycle (render `PodTemplateSpec` → run a one-shot `batch/v1.Job` → poll its terminal condition → emit a `Backup`), minus the ClickHouse-specific `applicationRef` gate, so it runs for any application. Shared package helpers (`renderJobTemplate`, `jobConditionState`, `getApplicationUnstructured`, status helpers, deterministic Job naming, controller refs) are reused. - **Template context** follows the in-tree convention used by the other drivers: `.Application` (live application object), `.Release.Name` / `.Release.Namespace`, `.Mode` (`backup` / `restore`), `.Parameters` (resolved `BackupClass` parameters), and `.Backup` (restore runs only). Strategy parameters round-trip through the `Backup`'s `driverMetadata` so a later restore re-renders against the same values. - **Restore** re-renders the same template with `.Mode == "restore"`. Cross-namespace restore is unsupported (the RestoreJob target reference is namespace-local), matching the Altinity driver. - **`JobSpec.Template` doc + regenerated CRD**: the field doc previously advertised `.Values`; it now documents the real context, and `strategy.backups.cozystack.io_jobs.yaml` is regenerated to match (the change is limited to that field's description). - **Tests** (`jobstrategy_controller_test.go`): backup-Job creation / args / labels / status, completion → `Backup` artifact with parameter round-trip, Job-failure surfacing, missing-strategy terminal failure, restore into the target namespace, restore completion, and the parameter-prefix unit. A deliberately generic application kind pins that the driver never gates on `applicationRef.kind`. No RBAC changes are required: the controller already holds `strategy.backups.cozystack.io/*`, `batch/jobs`, `apps.cozystack.io/*`, and `backups` permissions. ## Notes for reviewers - Backup and restore share one template, so a strategy author's `PodTemplateSpec` must branch on `.Mode` to do anything restore-specific. This matches the existing convention; worth confirming it's the intended contract for the Job strategy's docs/examples. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Implemented full backup/restore job strategy with deterministic Job creation, idempotent execution, and status/condition updates; added demo example scripts and reusable shell helpers for a NATS JetStream backup/restore workflow. * **Documentation** * Clarified Job template rendering context and listed supported template variables; added README and step-by-step scenario docs for the NATS example. * **Tests** * Added comprehensive unit tests covering job creation, completion/failure handling, parameter extraction, and unmappable-target failure cases. <!-- review_stack_entry_start --> [![Review Change Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/1721?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents b630706 + 727343d commit 72c4eb9

18 files changed

Lines changed: 2043 additions & 18 deletions

‎api/backups/strategy/v1alpha1/job_types.go‎

Lines changed: 12 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -49,12 +49,18 @@ type JobList struct {
4949

5050
// JobSpec specifies the desired behavior of a backup job.
5151
type JobSpec struct {
52-
// Template holds a PodTemplateSpec with the right shape to
53-
// run a single pod to completion and create a tarball with
54-
// a given apps data. Helm-like Go templates are supported.
55-
// The values of the source application are available under
56-
// `.Values`. `.Release.Name` and `.Release.Namespace` are
57-
// also exported.
52+
// Template holds a PodTemplateSpec with the right shape to run a single
53+
// pod to completion and create (or restore) a backup of a given app's
54+
// data. Helm-like Go templates are supported in every string field and
55+
// are rendered against the following context:
56+
// - `.Application` — the live application object (e.g.
57+
// `.Application.spec.replicas`).
58+
// - `.Release.Name` and `.Release.Namespace` — the application's name
59+
// and namespace.
60+
// - `.Mode` — `"backup"` or `"restore"`, identifying the run.
61+
// - `.Parameters` — the resolved BackupClass strategy parameters.
62+
// - `.Backup` — only on restore runs: `.Backup.Name`, `.Backup.Namespace`
63+
// and `.Backup.ApplicationRef` describing the source backup.
5864
Template corev1.PodTemplateSpec `json:"template"`
5965
}
6066

Lines changed: 132 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,132 @@
1+
#!/bin/bash
2+
# Shared helpers for the NATS JetStream backup/restore demo.
3+
# Source this file in other scripts: source "$(dirname "$0")/00-helpers.sh"
4+
5+
export RED='\033[0;31m'
6+
export GREEN='\033[0;32m'
7+
export YELLOW='\033[1;33m'
8+
export BLUE='\033[0;34m'
9+
export MAGENTA='\033[0;35m'
10+
export CYAN='\033[0;36m'
11+
export WHITE='\033[1;37m'
12+
export NC='\033[0m'
13+
export BOLD='\033[1m'
14+
15+
# Default settings (override via environment).
16+
export NAMESPACE="${NAMESPACE:-tenant-test}"
17+
export NATS_NAME="${NATS_NAME:-nats-test}"
18+
export NATS_RESTORE_NAME="${NATS_RESTORE_NAME:-nats-restore}"
19+
export NATS_USER="${NATS_USER:-backup}"
20+
export NATS_PASSWORD="${NATS_PASSWORD:-jetstream-demo-pw}"
21+
export STREAM_NAME="${STREAM_NAME:-ORDERS}"
22+
export MESSAGE_COUNT="${MESSAGE_COUNT:-10}"
23+
export BUCKET_NAME="${BUCKET_NAME:-nats-backups}"
24+
export BACKUPCLASS_NAME="${BACKUPCLASS_NAME:-nats-backup}"
25+
export STRATEGY_NAME="${STRATEGY_NAME:-nats-job}"
26+
export BACKUPJOB_NAME="${BACKUPJOB_NAME:-nats-backup-job}"
27+
export RESTOREJOB_INPLACE_NAME="${RESTOREJOB_INPLACE_NAME:-nats-restore-inplace}"
28+
export RESTOREJOB_TOCOPY_NAME="${RESTOREJOB_TOCOPY_NAME:-nats-restore-to-copy}"
29+
# natsio/nats-box ships the `nats` CLI plus curl + tar + sh - everything the
30+
# generic Job strategy needs, with no purpose-built backup image. The seed /
31+
# verify helpers below run it as a throwaway Pod via `kubectl run`.
32+
export NATS_BOX_IMAGE="${NATS_BOX_IMAGE:-natsio/nats-box:0.14.5}"
33+
34+
log_info() { echo -e "${BLUE}i${NC} $*" >&2; }
35+
log_success() { echo -e "${GREEN}OK${NC} $*" >&2; }
36+
log_warning() { echo -e "${YELLOW}!${NC} $*" >&2; }
37+
log_error() { echo -e "${RED}x${NC} $*" >&2; }
38+
log_step() { echo -e "\n${MAGENTA}${BOLD}> $*${NC}" >&2; }
39+
log_substep() { echo -e "${CYAN} -> $*${NC}" >&2; }
40+
log_command() { echo -e "${WHITE} $ $*${NC}" >&2; }
41+
42+
separator() {
43+
echo -e "\n${CYAN}------------------------------------------------------------${NC}\n" >&2
44+
}
45+
46+
print_header() {
47+
local title="$1"
48+
echo -e "\n${MAGENTA}${BOLD}== $title ==${NC}\n" >&2
49+
}
50+
51+
# Wait until a JSONPath value on a resource matches the desired string.
52+
wait_for_field() {
53+
local resource_type="$1"
54+
local resource_name="$2"
55+
local jsonpath="$3"
56+
local desired="$4"
57+
local namespace="${5:-}"
58+
local timeout="${6:-300}"
59+
60+
log_substep "Waiting for $resource_type/$resource_name $jsonpath to become '$desired'..."
61+
local elapsed=0
62+
local ns_flag=()
63+
[[ -n "$namespace" ]] && ns_flag=(-n "$namespace")
64+
65+
while true; do
66+
local current
67+
current=$(kubectl get "$resource_type" "$resource_name" "${ns_flag[@]}" -o jsonpath="$jsonpath" 2>/dev/null || true)
68+
if [[ "$current" == "$desired" ]]; then
69+
log_success "$resource_type/$resource_name reached '$desired'"
70+
return 0
71+
fi
72+
if [[ $elapsed -ge $timeout ]]; then
73+
log_error "Timeout waiting for $resource_type/$resource_name (current: '$current', expected: '$desired')"
74+
return 1
75+
fi
76+
sleep 5
77+
elapsed=$((elapsed + 5))
78+
done
79+
}
80+
81+
# In-cluster NATS client URL for the given application instance. The NATS app
82+
# names its client Service after the application (fullnameOverride=<name>) and
83+
# stores per-user passwords in the "<name>-credentials" Secret. The demo sets a
84+
# fixed password (NATS_PASSWORD) on both the source and restore-target apps, so
85+
# a single URL shape works everywhere.
86+
nats_url() {
87+
local app="$1"
88+
echo "nats://${NATS_USER}:${NATS_PASSWORD}@${app}.${NAMESPACE}.svc:4222"
89+
}
90+
91+
# Run a `nats` CLI invocation against an application instance from a throwaway
92+
# nats-box Pod. Args after the app name are passed verbatim to `nats`.
93+
# Example: nats_cli "$NATS_NAME" stream ls
94+
nats_cli() {
95+
local app="$1"; shift
96+
kubectl -n "$NAMESPACE" run "nats-cli-$RANDOM" \
97+
--image="$NATS_BOX_IMAGE" --restart=Never --rm -i --quiet \
98+
--command -- nats --server "$(nats_url "$app")" "$@"
99+
}
100+
101+
# Number of messages currently stored in a JetStream stream, or "" if the
102+
# stream does not exist.
103+
stream_message_count() {
104+
local app="$1"
105+
local stream="$2"
106+
nats_cli "$app" stream info "$stream" --json 2>/dev/null \
107+
| jq -r '.state.messages // empty' 2>/dev/null | tr -d '[:space:]'
108+
}
109+
110+
# Create the "<app>-backup-s3" Secret the Job strategy Pod consumes, from the
111+
# bucket coordinates cached by 03-create-bucket.sh. The generic Job strategy -
112+
# unlike the app-specific drivers - has no chart support to emit this Secret,
113+
# so the tenant provides it. Called for the source app (step 04) and the
114+
# restore target (step 07).
115+
create_s3_secret() {
116+
local app="$1"
117+
local SCRIPT_DIR
118+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
119+
[[ -f "$SCRIPT_DIR/.bucket-info.env" ]] || { log_error "missing $SCRIPT_DIR/.bucket-info.env; run 03-create-bucket.sh first"; return 1; }
120+
# shellcheck disable=SC1091
121+
source "$SCRIPT_DIR/.bucket-info.env"
122+
for v in S3_ACCESS_KEY S3_SECRET_KEY S3_ENDPOINT S3_REGION S3_BUCKET; do
123+
[[ -n "${!v:-}" ]] || { log_error "required variable is missing or empty: ${v}"; return 1; }
124+
done
125+
kubectl -n "$NAMESPACE" create secret generic "${app}-backup-s3" \
126+
--from-literal=accessKey="$S3_ACCESS_KEY" \
127+
--from-literal=secretKey="$S3_SECRET_KEY" \
128+
--from-literal=endpoint="$S3_ENDPOINT" \
129+
--from-literal=region="$S3_REGION" \
130+
--from-literal=bucket="$S3_BUCKET" \
131+
--dry-run=client -o yaml | kubectl apply -f -
132+
}
Lines changed: 150 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,150 @@
1+
#!/bin/bash
2+
# Step 01: Create the cluster-scoped generic Job backup strategy for NATS
3+
# JetStream. Unlike the app-specific drivers (Altinity, CNPG, ...), the Job
4+
# strategy runs an arbitrary Pod the operator supplies. Here that Pod is a
5+
# stock natsio/nats-box image (which carries the `nats` CLI plus curl and tar)
6+
# running a single shell script that branches on `.Mode`:
7+
# backup : `nats stream backup` -> tar -> PUT the tarball to S3
8+
# restore: GET the tarball from S3 -> untar -> `nats stream restore`
9+
#
10+
# Everything the template injects is funnelled through container env vars so
11+
# the script body stays plain shell. Note how the same PodTemplateSpec serves
12+
# both directions: the engine templates each string leaf independently (it
13+
# cannot add/remove containers per mode), so the *one* image branches at
14+
# runtime on $MODE.
15+
set -euo pipefail
16+
17+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
18+
source "$SCRIPT_DIR/00-helpers.sh"
19+
20+
print_header "Step 01: Create Job backup strategy '${STRATEGY_NAME}'"
21+
log_command "kubectl apply -f - (Job strategy: $STRATEGY_NAME)"
22+
23+
kubectl apply -f - <<EOF
24+
apiVersion: strategy.backups.cozystack.io/v1alpha1
25+
kind: Job
26+
metadata:
27+
name: ${STRATEGY_NAME}
28+
spec:
29+
template:
30+
spec:
31+
restartPolicy: Never
32+
containers:
33+
- name: nats-backup
34+
image: ${NATS_BOX_IMAGE}
35+
imagePullPolicy: IfNotPresent
36+
env:
37+
# .Parameters carry the static knobs from the BackupClass: which
38+
# JetStream stream to back up and which NATS user to authenticate
39+
# as. On restore they are read back from the Backup's
40+
# driverMetadata, so the same values apply round-trip.
41+
- name: STREAM
42+
value: '{{ default "ORDERS" (index .Parameters "stream") }}'
43+
- name: NATS_USER
44+
value: '{{ default "backup" (index .Parameters "natsUser") }}'
45+
# .Release is the application being acted on: the source on backup,
46+
# the restore target on restore (in-place=source, to-copy=the new
47+
# app). The NATS app exposes its client Service as <name>.<ns>.svc
48+
# and stores per-user passwords in <name>-credentials.
49+
- name: NATS_HOST
50+
value: "{{ .Release.Name }}.{{ .Release.Namespace }}.svc"
51+
- name: NATS_PASSWORD
52+
valueFrom:
53+
secretKeyRef:
54+
name: "{{ .Release.Name }}-credentials"
55+
key: '{{ default "backup" (index .Parameters "natsUser") }}'
56+
# S3 object key is scoped by the SOURCE app name so a to-copy
57+
# restore reads what the source wrote. On backup the source is
58+
# .Release.Name; on restore it is .Backup.ApplicationRef.Name
59+
# (.Backup is only set on restore - the guard keeps backup mode
60+
# from rendering "<no value>" into an unused var).
61+
- name: SRC_BACKUP
62+
value: "{{ .Release.Name }}"
63+
- name: SRC_RESTORE
64+
value: "{{ if .Backup }}{{ .Backup.ApplicationRef.Name }}{{ end }}"
65+
- name: MODE
66+
value: "{{ .Mode }}"
67+
# S3 coordinates from the tenant-provided <release>-backup-s3 Secret
68+
# (created by create_s3_secret in 00-helpers.sh from the Bucket).
69+
- name: AWS_ACCESS_KEY_ID
70+
valueFrom:
71+
secretKeyRef:
72+
name: "{{ .Release.Name }}-backup-s3"
73+
key: accessKey
74+
- name: AWS_SECRET_ACCESS_KEY
75+
valueFrom:
76+
secretKeyRef:
77+
name: "{{ .Release.Name }}-backup-s3"
78+
key: secretKey
79+
- name: S3_ENDPOINT
80+
valueFrom:
81+
secretKeyRef:
82+
name: "{{ .Release.Name }}-backup-s3"
83+
key: endpoint
84+
- name: S3_REGION
85+
valueFrom:
86+
secretKeyRef:
87+
name: "{{ .Release.Name }}-backup-s3"
88+
key: region
89+
- name: S3_BUCKET
90+
valueFrom:
91+
secretKeyRef:
92+
name: "{{ .Release.Name }}-backup-s3"
93+
key: bucket
94+
command: ["/bin/sh", "-c"]
95+
args:
96+
- |
97+
set -eu
98+
NATS_URL="nats://\${NATS_USER}:\${NATS_PASSWORD}@\${NATS_HOST}:4222"
99+
100+
# BucketInfo endpoints are bare host:port; COSI/seaweedfs serves
101+
# S3 over HTTPS, so default to https:// when no scheme is given.
102+
case "\${S3_ENDPOINT}" in
103+
http://*|https://*) BASE="\${S3_ENDPOINT}" ;;
104+
*) BASE="https://\${S3_ENDPOINT}" ;;
105+
esac
106+
107+
if [ "\${MODE}" = backup ]; then SRC="\${SRC_BACKUP}"; else SRC="\${SRC_RESTORE}"; fi
108+
KEY="\${SRC}/\${STREAM}.tar"
109+
OBJ_URL="\${BASE}/\${S3_BUCKET}/\${KEY}"
110+
111+
# curl --aws-sigv4 signs the request (SigV4) so no separate S3
112+
# client image is needed. -k accepts seaweedfs's internal
113+
# self-signed cert; a production strategy would mount the tenant
114+
# CA and drop -k.
115+
s3() { curl -fsS -k --aws-sigv4 "aws:amz:\${S3_REGION}:s3" --user "\${AWS_ACCESS_KEY_ID}:\${AWS_SECRET_ACCESS_KEY}" "\$@"; }
116+
117+
rm -rf /tmp/bk && mkdir -p /tmp/bk
118+
if [ "\${MODE}" = backup ]; then
119+
echo "backing up JetStream stream \${STREAM} from \${NATS_HOST}"
120+
nats --server "\${NATS_URL}" stream backup "\${STREAM}" /tmp/bk
121+
tar -C /tmp/bk -cf /tmp/backup.tar .
122+
s3 -X PUT --upload-file /tmp/backup.tar "\${OBJ_URL}"
123+
echo "uploaded s3://\${S3_BUCKET}/\${KEY}"
124+
else
125+
echo "restoring JetStream stream \${STREAM} into \${NATS_HOST} from s3://\${S3_BUCKET}/\${KEY}"
126+
s3 -o /tmp/backup.tar "\${OBJ_URL}"
127+
tar -C /tmp/bk -xf /tmp/backup.tar
128+
nats --server "\${NATS_URL}" stream restore /tmp/bk
129+
echo "restore complete"
130+
fi
131+
resources:
132+
requests:
133+
cpu: 10m
134+
memory: 32Mi
135+
limits:
136+
cpu: 200m
137+
memory: 128Mi
138+
securityContext:
139+
# Minimal hardening, matching the ClickHouse example. nats-box runs
140+
# the script as its default user; tenants enforcing PSA "restricted"
141+
# should add runAsNonRoot/runAsUser for the image they pin.
142+
allowPrivilegeEscalation: false
143+
capabilities:
144+
drop: ["ALL"]
145+
seccompProfile:
146+
type: RuntimeDefault
147+
EOF
148+
149+
log_success "Job strategy '${STRATEGY_NAME}' created."
150+
echo -e "\n${GREEN}${BOLD}Next:${NC} ./02-create-backupclass.sh"
Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
#!/bin/bash
2+
# Step 02: Map the NATS application kind to the Job strategy from step 01. The
3+
# strategy parameters (which stream to back up, which NATS user to connect as)
4+
# travel through the BackupClass and are exposed to the strategy template as
5+
# `.Parameters`. They are also snapshotted onto the resulting Backup's
6+
# driverMetadata, so a later restore re-renders with the same values.
7+
set -euo pipefail
8+
9+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
10+
source "$SCRIPT_DIR/00-helpers.sh"
11+
12+
print_header "Step 02: Create BackupClass '${BACKUPCLASS_NAME}'"
13+
log_command "kubectl apply -f - (BackupClass: $BACKUPCLASS_NAME -> $STRATEGY_NAME)"
14+
15+
kubectl apply -f - <<EOF
16+
apiVersion: backups.cozystack.io/v1alpha1
17+
kind: BackupClass
18+
metadata:
19+
name: ${BACKUPCLASS_NAME}
20+
spec:
21+
strategies:
22+
- application:
23+
apiGroup: apps.cozystack.io
24+
kind: NATS
25+
strategyRef:
26+
apiGroup: strategy.backups.cozystack.io
27+
kind: Job
28+
name: ${STRATEGY_NAME}
29+
parameters:
30+
stream: "${STREAM_NAME}"
31+
natsUser: "${NATS_USER}"
32+
EOF
33+
34+
log_success "BackupClass '${BACKUPCLASS_NAME}' created."
35+
echo -e "\n${GREEN}${BOLD}Next:${NC} ./03-create-bucket.sh"

0 commit comments

Comments
 (0)