|
| 1 | +--- |
| 2 | +title: "Application Backup and Recovery" |
| 3 | +linkTitle: "Backup and Recovery" |
| 4 | +description: "Back up and restore managed databases (Postgres, MariaDB, ClickHouse, FoundationDB) with BackupJob, Plan, and RestoreJob." |
| 5 | +weight: 4 |
| 6 | +--- |
| 7 | + |
| 8 | +This guide covers backing up and restoring **Cozystack-managed databases** — Postgres, MariaDB, ClickHouse, and FoundationDB — as a tenant user: running one-off and scheduled backups, checking status, and restoring from a backup either in place or into a separate target instance. |
| 9 | + |
| 10 | +{{% alert color="warning" %}} |
| 11 | +**These backups are data-only.** Each strategy snapshots the database contents through the operator's native mechanism (CloudNativePG barman, mariadb-operator dumps, Altinity `clickhouse-backup`, FoundationDB `backup_agent`). They do **not** capture the `apps.cozystack.io/*` CR, its `HelmRelease`, chart values, or operator-managed Secrets. |
| 12 | + |
| 13 | +To restore you must either: |
| 14 | +- keep the source application alive and restore in place (each driver re-bootstraps data into the existing operator-managed cluster), **or** |
| 15 | +- pre-provision an empty target application of the same Kind, then restore into it. |
| 16 | + |
| 17 | +For backups that include the application's Helm release, CRs, and PVC snapshots (used for VMInstance / VMDisk), see [Backup and Recovery (VMs)]({{% ref "/docs/next/virtualization/backup-and-recovery" %}}). |
| 18 | +{{% /alert %}} |
| 19 | + |
| 20 | +## Prerequisites |
| 21 | + |
| 22 | +- A `BackupClass` exists in the cluster for the application Kind you want to back up. Run `kubectl get backupclasses` to confirm; if none is present, ask your administrator to follow the [Managed Application Backup Configuration]({{% ref "/docs/next/operations/services/managed-app-backup-configuration" %}}) guide. |
| 23 | +- S3-compatible storage. Either provision an in-cluster `Bucket` (shown below) or use external S3 coordinates supplied by your administrator. |
| 24 | +- `kubectl` and kubeconfig for the management cluster. |
| 25 | + |
| 26 | +## List available BackupClasses |
| 27 | + |
| 28 | +`BackupClass` resources are cluster-scoped and tell you which application Kinds can be backed up and which driver handles each: |
| 29 | + |
| 30 | +```bash |
| 31 | +kubectl get backupclasses |
| 32 | +``` |
| 33 | + |
| 34 | +Example output: |
| 35 | + |
| 36 | +``` |
| 37 | +NAME AGE |
| 38 | +postgres-data-backup 14m |
| 39 | +mariadb-data-backup 14m |
| 40 | +clickhouse-data-backup 14m |
| 41 | +foundationdb-data-backup 14m |
| 42 | +velero 1d |
| 43 | +``` |
| 44 | + |
| 45 | +Use the `BackupClass` name when creating a `BackupJob` or `Plan`. The examples below assume `tenant-user` for the tenant namespace; substitute your own. |
| 46 | + |
| 47 | +## Provision the storage Bucket |
| 48 | + |
| 49 | +If your administrator has not pre-configured external S3, provision an in-cluster `Bucket` in the tenant namespace: |
| 50 | + |
| 51 | +```yaml |
| 52 | +apiVersion: apps.cozystack.io/v1alpha1 |
| 53 | +kind: Bucket |
| 54 | +metadata: |
| 55 | + name: db-backups |
| 56 | + namespace: tenant-user |
| 57 | +spec: |
| 58 | + users: |
| 59 | + backup: |
| 60 | + readonly: false |
| 61 | +``` |
| 62 | +
|
| 63 | +```bash |
| 64 | +kubectl apply -f bucket.yaml |
| 65 | +kubectl -n tenant-user wait hr/bucket-db-backups --for=condition=ready --timeout=300s |
| 66 | +``` |
| 67 | + |
| 68 | +The `Bucket` controller materialises a `bucket-<name>-backup` Secret in the namespace carrying a `BucketInfo` JSON blob. The S3 endpoint, bucket name, and access keys come from there. |
| 69 | + |
| 70 | +## Create per-application backup credentials |
| 71 | + |
| 72 | +Each driver expects per-application credential Secrets in the application namespace — the strategy templates reference them by name. The snippets below assume a single shell session: first read the bucket credentials, then run only the per-driver block for the application Kind you are setting up. |
| 73 | + |
| 74 | +### Read the bucket credentials |
| 75 | + |
| 76 | +Run this once per shell session. Every per-driver block below reuses `$ACCESS_KEY`, `$SECRET_KEY`, and `/tmp/bucket.json`: |
| 77 | + |
| 78 | +```bash |
| 79 | +kubectl -n tenant-user get secret bucket-db-backups-backup \ |
| 80 | + -o jsonpath='{.data.BucketInfo}' | base64 -d > /tmp/bucket.json |
| 81 | +ACCESS_KEY=$(jq -r .spec.secretS3.accessKeyID /tmp/bucket.json) |
| 82 | +SECRET_KEY=$(jq -r .spec.secretS3.accessSecretKey /tmp/bucket.json) |
| 83 | +``` |
| 84 | + |
| 85 | +If you start a new shell, re-run that snippet before continuing. |
| 86 | + |
| 87 | +### Postgres |
| 88 | + |
| 89 | +Project the credentials in the keys CNPG's barman client expects: |
| 90 | + |
| 91 | +```bash |
| 92 | +kubectl -n tenant-user create secret generic my-postgres-cnpg-backup-creds \ |
| 93 | + --from-literal=ACCESS_KEY_ID="$ACCESS_KEY" \ |
| 94 | + --from-literal=ACCESS_SECRET_KEY="$SECRET_KEY" |
| 95 | +``` |
| 96 | + |
| 97 | +When the S3 endpoint uses a self-signed certificate (the SeaweedFS default), also create a CA Secret: |
| 98 | + |
| 99 | +```bash |
| 100 | +kubectl -n tenant-user create secret generic my-postgres-cnpg-backup-ca \ |
| 101 | + --from-file=ca.crt=/path/to/ca.crt |
| 102 | +``` |
| 103 | + |
| 104 | +### MariaDB |
| 105 | + |
| 106 | +```bash |
| 107 | +kubectl -n tenant-user create secret generic my-mariadb-mariadb-backup-creds \ |
| 108 | + --from-literal=AWS_ACCESS_KEY_ID="$ACCESS_KEY" \ |
| 109 | + --from-literal=AWS_SECRET_ACCESS_KEY="$SECRET_KEY" |
| 110 | +``` |
| 111 | + |
| 112 | +For self-signed endpoints, add `my-mariadb-mariadb-backup-ca` carrying `ca.crt` the same way. |
| 113 | + |
| 114 | +### ClickHouse |
| 115 | + |
| 116 | +ClickHouse backups read S3 credentials from the chart-emitted `<release>-backup-s3` Secret directly. Set `backup.enabled: true` on the ClickHouse application and fill in `backup.*` with the bucket coordinates — no extra Secret is needed for the BackupClass flow. See the [ClickHouse application reference]({{% ref "/docs/next/applications/clickhouse" %}}) for the `backup.*` values. |
| 117 | + |
| 118 | +### FoundationDB |
| 119 | + |
| 120 | +FoundationDB's `backup_agent` requires a `blob_credentials.json` payload in a specific shape. This block reads the bucket endpoint from `/tmp/bucket.json` (created by [Read the bucket credentials](#read-the-bucket-credentials) above) and reuses `$ACCESS_KEY` / `$SECRET_KEY` from the same step: |
| 121 | + |
| 122 | +```bash |
| 123 | +ENDPOINT_FULL=$(jq -r .spec.secretS3.endpoint /tmp/bucket.json) |
| 124 | +ENDPOINT_HOSTPORT=${ENDPOINT_FULL#http://} |
| 125 | +ENDPOINT_HOSTPORT=${ENDPOINT_HOSTPORT#https://} |
| 126 | +ACCOUNT_NAME="${ACCESS_KEY}@${ENDPOINT_HOSTPORT}" |
| 127 | + |
| 128 | +jq -nc \ |
| 129 | + --arg account "$ACCOUNT_NAME" \ |
| 130 | + --arg key "$ACCESS_KEY" \ |
| 131 | + --arg secret "$SECRET_KEY" \ |
| 132 | + '{accounts: {($account): {api_key: $key, secret: $secret}}}' \ |
| 133 | + > /tmp/blob_credentials.json |
| 134 | + |
| 135 | +kubectl -n tenant-user create secret generic my-fdb-fdb-backup-creds \ |
| 136 | + --from-file=blob_credentials.json=/tmp/blob_credentials.json |
| 137 | +``` |
| 138 | + |
| 139 | +Your administrator must also patch the FoundationDB `BackupClass` parameters with the resolved `accountName`, `bucket`, `region`, and `secureConnection` values before the first backup runs. Otherwise the first `BackupJob` fails fast with a validation error (`accountName is required`) — this is the intentional fail-loud behaviour for a half-configured tenant. |
| 140 | + |
| 141 | +## Run a backup |
| 142 | + |
| 143 | +### One-off backup |
| 144 | + |
| 145 | +Use a `BackupJob` for an ad-hoc backup (for example, before a risky change): |
| 146 | + |
| 147 | +```yaml |
| 148 | +apiVersion: backups.cozystack.io/v1alpha1 |
| 149 | +kind: BackupJob |
| 150 | +metadata: |
| 151 | + name: my-postgres-adhoc |
| 152 | + namespace: tenant-user |
| 153 | +spec: |
| 154 | + applicationRef: |
| 155 | + apiGroup: apps.cozystack.io |
| 156 | + kind: Postgres |
| 157 | + name: my-postgres |
| 158 | + backupClassName: postgres-data-backup |
| 159 | +``` |
| 160 | +
|
| 161 | +```bash |
| 162 | +kubectl apply -f backupjob.yaml |
| 163 | +kubectl -n tenant-user get backupjobs |
| 164 | +kubectl -n tenant-user describe backupjob my-postgres-adhoc |
| 165 | +``` |
| 166 | + |
| 167 | +When the `BackupJob` reaches `phase: Succeeded`, the driver creates a `Backup` object with the same name. That name is what you reference when restoring. |
| 168 | + |
| 169 | +Replace `Postgres` / `postgres-data-backup` with `MariaDB` / `mariadb-data-backup`, `ClickHouse` / `clickhouse-data-backup`, or `FoundationDB` / `foundationdb-data-backup` for the other drivers. |
| 170 | + |
| 171 | +### Scheduled backup |
| 172 | + |
| 173 | +Use a `Plan` for cron-driven recurring backups: |
| 174 | + |
| 175 | +```yaml |
| 176 | +apiVersion: backups.cozystack.io/v1alpha1 |
| 177 | +kind: Plan |
| 178 | +metadata: |
| 179 | + name: my-postgres-daily |
| 180 | + namespace: tenant-user |
| 181 | +spec: |
| 182 | + applicationRef: |
| 183 | + apiGroup: apps.cozystack.io |
| 184 | + kind: Postgres |
| 185 | + name: my-postgres |
| 186 | + backupClassName: postgres-data-backup |
| 187 | + schedule: |
| 188 | + type: cron |
| 189 | + cron: "0 */6 * * *" # every 6 hours |
| 190 | +``` |
| 191 | +
|
| 192 | +Each scheduled run creates a `BackupJob` (and, on success, a `Backup`) named after the `Plan` with a timestamp suffix. |
| 193 | + |
| 194 | +```bash |
| 195 | +kubectl apply -f plan.yaml |
| 196 | +kubectl -n tenant-user get plans |
| 197 | +kubectl -n tenant-user get backupjobs -l backups.cozystack.io/plan=my-postgres-daily |
| 198 | +``` |
| 199 | + |
| 200 | +## Check backup status |
| 201 | + |
| 202 | +List `BackupJob` and `Backup` resources in the namespace: |
| 203 | + |
| 204 | +```bash |
| 205 | +kubectl -n tenant-user get backupjobs |
| 206 | +kubectl -n tenant-user get backups |
| 207 | +``` |
| 208 | + |
| 209 | +Inspect a failed run: |
| 210 | + |
| 211 | +```bash |
| 212 | +kubectl -n tenant-user get backupjob my-postgres-adhoc -o jsonpath='{.status.message}' |
| 213 | +kubectl -n tenant-user describe backupjob my-postgres-adhoc |
| 214 | +``` |
| 215 | + |
| 216 | +For driver-side detail, inspect the operator-native CR each driver materialises (one of `cnpg.io/Backup`, `k8s.mariadb.com/Backup`, `apps.foundationdb.org/FoundationDBBackup`, or the ClickHouse strategy `Pod`). |
| 217 | + |
| 218 | +## Restore in place |
| 219 | + |
| 220 | +An **in-place restore** replays the backup into the **same** application. Use this to roll back accidental deletion or corruption on a live database you intend to keep using under the same name. |
| 221 | + |
| 222 | +{{% alert color="warning" %}} |
| 223 | +In-place restore is **destructive**. Each driver wipes or replaces existing data on the source application; any writes since the backup point are lost. If you cannot afford to lose recent writes, use [Restore to a copy](#restore-to-a-copy) instead. |
| 224 | +{{% /alert %}} |
| 225 | + |
| 226 | +```yaml |
| 227 | +apiVersion: backups.cozystack.io/v1alpha1 |
| 228 | +kind: RestoreJob |
| 229 | +metadata: |
| 230 | + name: my-postgres-restore-inplace |
| 231 | + namespace: tenant-user |
| 232 | +spec: |
| 233 | + backupRef: |
| 234 | + name: my-postgres-adhoc |
| 235 | + # targetApplicationRef omitted: driver restores into Backup.spec.applicationRef. |
| 236 | + # options: |
| 237 | + # recoveryTime: "2026-05-01T12:00:00Z" # Postgres only; RFC3339 PITR |
| 238 | +``` |
| 239 | + |
| 240 | +```bash |
| 241 | +kubectl apply -f restorejob.yaml |
| 242 | +kubectl -n tenant-user get restorejobs |
| 243 | +kubectl -n tenant-user describe restorejob my-postgres-restore-inplace |
| 244 | +``` |
| 245 | + |
| 246 | +### Per-driver caveats |
| 247 | + |
| 248 | +- **Postgres (CNPG)** — the driver deletes the live `cnpg.io/Cluster` and its PVCs, then re-bootstraps from the Barman archive. Connections drop for the duration. `spec.options.recoveryTime` (RFC3339) is supported for point-in-time recovery; omit it to restore to the latest WAL. |
| 249 | +- **MariaDB** — the operator replays the logical dump into the live `MariaDB` via `mariadb-import`. Pre-existing tables will collide; pre-truncate the relevant schemas if your dump does not include `DROP TABLE`. |
| 250 | +- **ClickHouse** — the Altinity strategy does **not** pass `clickhouse-backup --rm`. You are responsible for dropping conflicting tables on the source before submitting the `RestoreJob`; otherwise the operation fails with a duplicate-table error. |
| 251 | +- **FoundationDB** — the operator pauses the FoundationDB cluster, clears the keyspace, and replays the backup via `fdbrestore`. Any data written after the snapshot is lost. Only one `FoundationDBBackup` directory may exist per cluster at a time — the driver stops any prior backup before starting a new one. |
| 252 | + |
| 253 | +## Restore to a copy |
| 254 | + |
| 255 | +A **to-copy restore** replays the backup into a **different**, freshly-provisioned application of the same Kind. Use this for disaster-recovery drills, side-by-side validation, branch databases, or migrating to a new version of the upstream operator. |
| 256 | + |
| 257 | +First, provision an empty target application with the same Kind. For example, an empty `Postgres`: |
| 258 | + |
| 259 | +```yaml |
| 260 | +apiVersion: apps.cozystack.io/v1alpha1 |
| 261 | +kind: Postgres |
| 262 | +metadata: |
| 263 | + name: my-postgres-restored |
| 264 | + namespace: tenant-user |
| 265 | +spec: |
| 266 | + # ...same shape as the source, no bootstrap data required... |
| 267 | +``` |
| 268 | + |
| 269 | +Wait for the target to become Ready, then submit a `RestoreJob` that points at it: |
| 270 | + |
| 271 | +```yaml |
| 272 | +apiVersion: backups.cozystack.io/v1alpha1 |
| 273 | +kind: RestoreJob |
| 274 | +metadata: |
| 275 | + name: my-postgres-restore-to-copy |
| 276 | + namespace: tenant-user |
| 277 | +spec: |
| 278 | + backupRef: |
| 279 | + name: my-postgres-adhoc |
| 280 | + targetApplicationRef: |
| 281 | + apiGroup: apps.cozystack.io |
| 282 | + kind: Postgres |
| 283 | + name: my-postgres-restored |
| 284 | +``` |
| 285 | + |
| 286 | +The source application stays untouched. Cross-namespace restores are **not** supported — `targetApplicationRef` is a local reference; the target must live in the same namespace as the `RestoreJob`. |
| 287 | + |
| 288 | +## Limitations and lifecycle |
| 289 | + |
| 290 | +- **Data-only scope.** Application CRs, HelmReleases, chart values, and operator-managed Secrets (e.g. `cnpg.io` superuser secret, `clickhouse-installation` users) are not captured. Pre-provision the target application before a to-copy restore. |
| 291 | +- **Archive retention is driver-owned.** Deleting a Cozystack `Backup` CR removes the artefact reference but leaves the actual S3 object intact. Each driver enforces its own retention: |
| 292 | + - CNPG: `retentionPolicy` on the strategy (`30d` default in the admin example). |
| 293 | + - MariaDB: configure `cleanupStrategy` on the operator-side `Backup` CR or rotate at the bucket level. |
| 294 | + - ClickHouse: governed by the in-pod sidecar's retention configuration. Tenants who need to purge an archive call `DELETE /backup/<name>/remote` on the sidecar. |
| 295 | + - FoundationDB: each `BackupJob` owns a discrete blob-store directory; clean up at the bucket level. |
| 296 | +- **One running backup per FoundationDB cluster.** The driver enforces this by stopping any prior `FoundationDBBackup` on the same cluster before starting a new one. |
| 297 | +- **ClickHouse depends on the in-chart sidecar.** The Altinity strategy is a thin HTTP client; the backup itself runs inside each `chi-*` Pod via `clickhouse-backup`. Disabling `backup.enabled` on the application also disables the BackupClass flow. |
| 298 | + |
| 299 | +## Troubleshooting |
| 300 | + |
| 301 | +If a `BackupJob` or `RestoreJob` ends in `phase: Failed`, check the message field: |
| 302 | + |
| 303 | +```bash |
| 304 | +kubectl -n tenant-user get backupjob my-postgres-adhoc -o jsonpath='{.status.message}' |
| 305 | +kubectl -n tenant-user get restorejob my-postgres-restore-inplace -o jsonpath='{.status.message}' |
| 306 | +``` |
| 307 | + |
| 308 | +Then look at the operator-native CR the driver created: |
| 309 | + |
| 310 | +```bash |
| 311 | +# Postgres |
| 312 | +kubectl -n tenant-user get backups.cnpg.io |
| 313 | +# MariaDB |
| 314 | +kubectl -n tenant-user get backups.k8s.mariadb.com,restores.k8s.mariadb.com |
| 315 | +# ClickHouse |
| 316 | +kubectl -n tenant-user logs -l backups.cozystack.io/owned-by.BackupJobName=my-clickhouse-adhoc |
| 317 | +# FoundationDB |
| 318 | +kubectl -n tenant-user get foundationdbbackups.apps.foundationdb.org,foundationdbrestores.apps.foundationdb.org \ |
| 319 | + -l backups.cozystack.io/owned-by.BackupJobName=my-fdb-adhoc |
| 320 | +``` |
| 321 | + |
| 322 | +## See also |
| 323 | + |
| 324 | +- [Managed Application Backup Configuration]({{% ref "/docs/next/operations/services/managed-app-backup-configuration" %}}) — how administrators define strategies and `BackupClass` resources. |
| 325 | +- [Backup and Recovery (VMs)]({{% ref "/docs/next/virtualization/backup-and-recovery" %}}) — the parallel guide for VMInstance / VMDisk backups (HelmRelease + CRs + PVC snapshots). |
| 326 | +- [Velero Backup Configuration]({{% ref "/docs/next/operations/services/velero-backup-configuration" %}}) — administrator setup for the Velero-driven VM backups. |
0 commit comments