Skip to content

Commit 4efa0e7

Browse files
committed
docs(platform): add guides related managed apps backups
Signed-off-by: Andrey Kolkov <[email protected]>
1 parent f466d34 commit 4efa0e7

8 files changed

Lines changed: 693 additions & 0 deletions

File tree

Lines changed: 326 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,326 @@
1+
---
2+
title: "Application Backup and Recovery"
3+
linkTitle: "Backup and Recovery"
4+
description: "Back up and restore managed databases (Postgres, MariaDB, ClickHouse, FoundationDB) with BackupJob, Plan, and RestoreJob."
5+
weight: 4
6+
---
7+
8+
This guide covers backing up and restoring **Cozystack-managed databases** — Postgres, MariaDB, ClickHouse, and FoundationDB — as a tenant user: running one-off and scheduled backups, checking status, and restoring from a backup either in place or into a separate target instance.
9+
10+
{{% alert color="warning" %}}
11+
**These backups are data-only.** Each strategy snapshots the database contents through the operator's native mechanism (CloudNativePG barman, mariadb-operator dumps, Altinity `clickhouse-backup`, FoundationDB `backup_agent`). They do **not** capture the `apps.cozystack.io/*` CR, its `HelmRelease`, chart values, or operator-managed Secrets.
12+
13+
To restore you must either:
14+
- keep the source application alive and restore in place (each driver re-bootstraps data into the existing operator-managed cluster), **or**
15+
- pre-provision an empty target application of the same Kind, then restore into it.
16+
17+
For backups that include the application's Helm release, CRs, and PVC snapshots (used for VMInstance / VMDisk), see [Backup and Recovery (VMs)]({{% ref "/docs/next/virtualization/backup-and-recovery" %}}).
18+
{{% /alert %}}
19+
20+
## Prerequisites
21+
22+
- A `BackupClass` exists in the cluster for the application Kind you want to back up. Run `kubectl get backupclasses` to confirm; if none is present, ask your administrator to follow the [Managed Application Backup Configuration]({{% ref "/docs/next/operations/services/managed-app-backup-configuration" %}}) guide.
23+
- S3-compatible storage. Either provision an in-cluster `Bucket` (shown below) or use external S3 coordinates supplied by your administrator.
24+
- `kubectl` and kubeconfig for the management cluster.
25+
26+
## List available BackupClasses
27+
28+
`BackupClass` resources are cluster-scoped and tell you which application Kinds can be backed up and which driver handles each:
29+
30+
```bash
31+
kubectl get backupclasses
32+
```
33+
34+
Example output:
35+
36+
```
37+
NAME AGE
38+
postgres-data-backup 14m
39+
mariadb-data-backup 14m
40+
clickhouse-data-backup 14m
41+
foundationdb-data-backup 14m
42+
velero 1d
43+
```
44+
45+
Use the `BackupClass` name when creating a `BackupJob` or `Plan`. The examples below assume `tenant-user` for the tenant namespace; substitute your own.
46+
47+
## Provision the storage Bucket
48+
49+
If your administrator has not pre-configured external S3, provision an in-cluster `Bucket` in the tenant namespace:
50+
51+
```yaml
52+
apiVersion: apps.cozystack.io/v1alpha1
53+
kind: Bucket
54+
metadata:
55+
name: db-backups
56+
namespace: tenant-user
57+
spec:
58+
users:
59+
backup:
60+
readonly: false
61+
```
62+
63+
```bash
64+
kubectl apply -f bucket.yaml
65+
kubectl -n tenant-user wait hr/bucket-db-backups --for=condition=ready --timeout=300s
66+
```
67+
68+
The `Bucket` controller materialises a `bucket-<name>-backup` Secret in the namespace carrying a `BucketInfo` JSON blob. The S3 endpoint, bucket name, and access keys come from there.
69+
70+
## Create per-application backup credentials
71+
72+
Each driver expects per-application credential Secrets in the application namespace — the strategy templates reference them by name. The snippets below assume a single shell session: first read the bucket credentials, then run only the per-driver block for the application Kind you are setting up.
73+
74+
### Read the bucket credentials
75+
76+
Run this once per shell session. Every per-driver block below reuses `$ACCESS_KEY`, `$SECRET_KEY`, and `/tmp/bucket.json`:
77+
78+
```bash
79+
kubectl -n tenant-user get secret bucket-db-backups-backup \
80+
-o jsonpath='{.data.BucketInfo}' | base64 -d > /tmp/bucket.json
81+
ACCESS_KEY=$(jq -r .spec.secretS3.accessKeyID /tmp/bucket.json)
82+
SECRET_KEY=$(jq -r .spec.secretS3.accessSecretKey /tmp/bucket.json)
83+
```
84+
85+
If you start a new shell, re-run that snippet before continuing.
86+
87+
### Postgres
88+
89+
Project the credentials in the keys CNPG's barman client expects:
90+
91+
```bash
92+
kubectl -n tenant-user create secret generic my-postgres-cnpg-backup-creds \
93+
--from-literal=ACCESS_KEY_ID="$ACCESS_KEY" \
94+
--from-literal=ACCESS_SECRET_KEY="$SECRET_KEY"
95+
```
96+
97+
When the S3 endpoint uses a self-signed certificate (the SeaweedFS default), also create a CA Secret:
98+
99+
```bash
100+
kubectl -n tenant-user create secret generic my-postgres-cnpg-backup-ca \
101+
--from-file=ca.crt=/path/to/ca.crt
102+
```
103+
104+
### MariaDB
105+
106+
```bash
107+
kubectl -n tenant-user create secret generic my-mariadb-mariadb-backup-creds \
108+
--from-literal=AWS_ACCESS_KEY_ID="$ACCESS_KEY" \
109+
--from-literal=AWS_SECRET_ACCESS_KEY="$SECRET_KEY"
110+
```
111+
112+
For self-signed endpoints, add `my-mariadb-mariadb-backup-ca` carrying `ca.crt` the same way.
113+
114+
### ClickHouse
115+
116+
ClickHouse backups read S3 credentials from the chart-emitted `<release>-backup-s3` Secret directly. Set `backup.enabled: true` on the ClickHouse application and fill in `backup.*` with the bucket coordinates — no extra Secret is needed for the BackupClass flow. See the [ClickHouse application reference]({{% ref "/docs/next/applications/clickhouse" %}}) for the `backup.*` values.
117+
118+
### FoundationDB
119+
120+
FoundationDB's `backup_agent` requires a `blob_credentials.json` payload in a specific shape. This block reads the bucket endpoint from `/tmp/bucket.json` (created by [Read the bucket credentials](#read-the-bucket-credentials) above) and reuses `$ACCESS_KEY` / `$SECRET_KEY` from the same step:
121+
122+
```bash
123+
ENDPOINT_FULL=$(jq -r .spec.secretS3.endpoint /tmp/bucket.json)
124+
ENDPOINT_HOSTPORT=${ENDPOINT_FULL#http://}
125+
ENDPOINT_HOSTPORT=${ENDPOINT_HOSTPORT#https://}
126+
ACCOUNT_NAME="${ACCESS_KEY}@${ENDPOINT_HOSTPORT}"
127+
128+
jq -nc \
129+
--arg account "$ACCOUNT_NAME" \
130+
--arg key "$ACCESS_KEY" \
131+
--arg secret "$SECRET_KEY" \
132+
'{accounts: {($account): {api_key: $key, secret: $secret}}}' \
133+
> /tmp/blob_credentials.json
134+
135+
kubectl -n tenant-user create secret generic my-fdb-fdb-backup-creds \
136+
--from-file=blob_credentials.json=/tmp/blob_credentials.json
137+
```
138+
139+
Your administrator must also patch the FoundationDB `BackupClass` parameters with the resolved `accountName`, `bucket`, `region`, and `secureConnection` values before the first backup runs. Otherwise the first `BackupJob` fails fast with a validation error (`accountName is required`) — this is the intentional fail-loud behaviour for a half-configured tenant.
140+
141+
## Run a backup
142+
143+
### One-off backup
144+
145+
Use a `BackupJob` for an ad-hoc backup (for example, before a risky change):
146+
147+
```yaml
148+
apiVersion: backups.cozystack.io/v1alpha1
149+
kind: BackupJob
150+
metadata:
151+
name: my-postgres-adhoc
152+
namespace: tenant-user
153+
spec:
154+
applicationRef:
155+
apiGroup: apps.cozystack.io
156+
kind: Postgres
157+
name: my-postgres
158+
backupClassName: postgres-data-backup
159+
```
160+
161+
```bash
162+
kubectl apply -f backupjob.yaml
163+
kubectl -n tenant-user get backupjobs
164+
kubectl -n tenant-user describe backupjob my-postgres-adhoc
165+
```
166+
167+
When the `BackupJob` reaches `phase: Succeeded`, the driver creates a `Backup` object with the same name. That name is what you reference when restoring.
168+
169+
Replace `Postgres` / `postgres-data-backup` with `MariaDB` / `mariadb-data-backup`, `ClickHouse` / `clickhouse-data-backup`, or `FoundationDB` / `foundationdb-data-backup` for the other drivers.
170+
171+
### Scheduled backup
172+
173+
Use a `Plan` for cron-driven recurring backups:
174+
175+
```yaml
176+
apiVersion: backups.cozystack.io/v1alpha1
177+
kind: Plan
178+
metadata:
179+
name: my-postgres-daily
180+
namespace: tenant-user
181+
spec:
182+
applicationRef:
183+
apiGroup: apps.cozystack.io
184+
kind: Postgres
185+
name: my-postgres
186+
backupClassName: postgres-data-backup
187+
schedule:
188+
type: cron
189+
cron: "0 */6 * * *" # every 6 hours
190+
```
191+
192+
Each scheduled run creates a `BackupJob` (and, on success, a `Backup`) named after the `Plan` with a timestamp suffix.
193+
194+
```bash
195+
kubectl apply -f plan.yaml
196+
kubectl -n tenant-user get plans
197+
kubectl -n tenant-user get backupjobs -l backups.cozystack.io/plan=my-postgres-daily
198+
```
199+
200+
## Check backup status
201+
202+
List `BackupJob` and `Backup` resources in the namespace:
203+
204+
```bash
205+
kubectl -n tenant-user get backupjobs
206+
kubectl -n tenant-user get backups
207+
```
208+
209+
Inspect a failed run:
210+
211+
```bash
212+
kubectl -n tenant-user get backupjob my-postgres-adhoc -o jsonpath='{.status.message}'
213+
kubectl -n tenant-user describe backupjob my-postgres-adhoc
214+
```
215+
216+
For driver-side detail, inspect the operator-native CR each driver materialises (one of `cnpg.io/Backup`, `k8s.mariadb.com/Backup`, `apps.foundationdb.org/FoundationDBBackup`, or the ClickHouse strategy `Pod`).
217+
218+
## Restore in place
219+
220+
An **in-place restore** replays the backup into the **same** application. Use this to roll back accidental deletion or corruption on a live database you intend to keep using under the same name.
221+
222+
{{% alert color="warning" %}}
223+
In-place restore is **destructive**. Each driver wipes or replaces existing data on the source application; any writes since the backup point are lost. If you cannot afford to lose recent writes, use [Restore to a copy](#restore-to-a-copy) instead.
224+
{{% /alert %}}
225+
226+
```yaml
227+
apiVersion: backups.cozystack.io/v1alpha1
228+
kind: RestoreJob
229+
metadata:
230+
name: my-postgres-restore-inplace
231+
namespace: tenant-user
232+
spec:
233+
backupRef:
234+
name: my-postgres-adhoc
235+
# targetApplicationRef omitted: driver restores into Backup.spec.applicationRef.
236+
# options:
237+
# recoveryTime: "2026-05-01T12:00:00Z" # Postgres only; RFC3339 PITR
238+
```
239+
240+
```bash
241+
kubectl apply -f restorejob.yaml
242+
kubectl -n tenant-user get restorejobs
243+
kubectl -n tenant-user describe restorejob my-postgres-restore-inplace
244+
```
245+
246+
### Per-driver caveats
247+
248+
- **Postgres (CNPG)** — the driver deletes the live `cnpg.io/Cluster` and its PVCs, then re-bootstraps from the Barman archive. Connections drop for the duration. `spec.options.recoveryTime` (RFC3339) is supported for point-in-time recovery; omit it to restore to the latest WAL.
249+
- **MariaDB** — the operator replays the logical dump into the live `MariaDB` via `mariadb-import`. Pre-existing tables will collide; pre-truncate the relevant schemas if your dump does not include `DROP TABLE`.
250+
- **ClickHouse** — the Altinity strategy does **not** pass `clickhouse-backup --rm`. You are responsible for dropping conflicting tables on the source before submitting the `RestoreJob`; otherwise the operation fails with a duplicate-table error.
251+
- **FoundationDB** — the operator pauses the FoundationDB cluster, clears the keyspace, and replays the backup via `fdbrestore`. Any data written after the snapshot is lost. Only one `FoundationDBBackup` directory may exist per cluster at a time — the driver stops any prior backup before starting a new one.
252+
253+
## Restore to a copy
254+
255+
A **to-copy restore** replays the backup into a **different**, freshly-provisioned application of the same Kind. Use this for disaster-recovery drills, side-by-side validation, branch databases, or migrating to a new version of the upstream operator.
256+
257+
First, provision an empty target application with the same Kind. For example, an empty `Postgres`:
258+
259+
```yaml
260+
apiVersion: apps.cozystack.io/v1alpha1
261+
kind: Postgres
262+
metadata:
263+
name: my-postgres-restored
264+
namespace: tenant-user
265+
spec:
266+
# ...same shape as the source, no bootstrap data required...
267+
```
268+
269+
Wait for the target to become Ready, then submit a `RestoreJob` that points at it:
270+
271+
```yaml
272+
apiVersion: backups.cozystack.io/v1alpha1
273+
kind: RestoreJob
274+
metadata:
275+
name: my-postgres-restore-to-copy
276+
namespace: tenant-user
277+
spec:
278+
backupRef:
279+
name: my-postgres-adhoc
280+
targetApplicationRef:
281+
apiGroup: apps.cozystack.io
282+
kind: Postgres
283+
name: my-postgres-restored
284+
```
285+
286+
The source application stays untouched. Cross-namespace restores are **not** supported — `targetApplicationRef` is a local reference; the target must live in the same namespace as the `RestoreJob`.
287+
288+
## Limitations and lifecycle
289+
290+
- **Data-only scope.** Application CRs, HelmReleases, chart values, and operator-managed Secrets (e.g. `cnpg.io` superuser secret, `clickhouse-installation` users) are not captured. Pre-provision the target application before a to-copy restore.
291+
- **Archive retention is driver-owned.** Deleting a Cozystack `Backup` CR removes the artefact reference but leaves the actual S3 object intact. Each driver enforces its own retention:
292+
- CNPG: `retentionPolicy` on the strategy (`30d` default in the admin example).
293+
- MariaDB: configure `cleanupStrategy` on the operator-side `Backup` CR or rotate at the bucket level.
294+
- ClickHouse: governed by the in-pod sidecar's retention configuration. Tenants who need to purge an archive call `DELETE /backup/<name>/remote` on the sidecar.
295+
- FoundationDB: each `BackupJob` owns a discrete blob-store directory; clean up at the bucket level.
296+
- **One running backup per FoundationDB cluster.** The driver enforces this by stopping any prior `FoundationDBBackup` on the same cluster before starting a new one.
297+
- **ClickHouse depends on the in-chart sidecar.** The Altinity strategy is a thin HTTP client; the backup itself runs inside each `chi-*` Pod via `clickhouse-backup`. Disabling `backup.enabled` on the application also disables the BackupClass flow.
298+
299+
## Troubleshooting
300+
301+
If a `BackupJob` or `RestoreJob` ends in `phase: Failed`, check the message field:
302+
303+
```bash
304+
kubectl -n tenant-user get backupjob my-postgres-adhoc -o jsonpath='{.status.message}'
305+
kubectl -n tenant-user get restorejob my-postgres-restore-inplace -o jsonpath='{.status.message}'
306+
```
307+
308+
Then look at the operator-native CR the driver created:
309+
310+
```bash
311+
# Postgres
312+
kubectl -n tenant-user get backups.cnpg.io
313+
# MariaDB
314+
kubectl -n tenant-user get backups.k8s.mariadb.com,restores.k8s.mariadb.com
315+
# ClickHouse
316+
kubectl -n tenant-user logs -l backups.cozystack.io/owned-by.BackupJobName=my-clickhouse-adhoc
317+
# FoundationDB
318+
kubectl -n tenant-user get foundationdbbackups.apps.foundationdb.org,foundationdbrestores.apps.foundationdb.org \
319+
-l backups.cozystack.io/owned-by.BackupJobName=my-fdb-adhoc
320+
```
321+
322+
## See also
323+
324+
- [Managed Application Backup Configuration]({{% ref "/docs/next/operations/services/managed-app-backup-configuration" %}}) — how administrators define strategies and `BackupClass` resources.
325+
- [Backup and Recovery (VMs)]({{% ref "/docs/next/virtualization/backup-and-recovery" %}}) — the parallel guide for VMInstance / VMDisk backups (HelmRelease + CRs + PVC snapshots).
326+
- [Velero Backup Configuration]({{% ref "/docs/next/operations/services/velero-backup-configuration" %}}) — administrator setup for the Velero-driven VM backups.

‎content/en/docs/next/applications/clickhouse.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,10 @@ It is used for online analytical processing (OLAP).
2020

2121
### How to restore backup from S3
2222

23+
{{% alert color="warning" %}}
24+
**Backups: prefer the `BackupClass` flow.** `backup.enabled` and the S3 fields (`s3Region`, `s3Bucket`, `endpoint`, `s3PathOverride`, `s3AccessKey`/`s3SecretKey` or `s3CredentialsSecret`) are still required — they materialise the in-pod `clickhouse-backup` sidecar that the Altinity backup strategy talks to. However, `backup.schedule`, `backup.cleanupStrategy`, and `backup.resticPassword` (which drive the legacy chart-managed CronJob doing dump + restic, and the matching restic restore flow documented below) are **superseded** by the Cozystack backups framework: define a `BackupClass` + `Altinity` strategy once, then drive scheduled backups via `Plan` and restores via `RestoreJob`. See [Application Backup and Recovery]({{% ref "/docs/next/applications/backup-and-recovery" %}}) (tenant guide) and [Managed Application Backup Configuration]({{% ref "/docs/next/operations/services/managed-app-backup-configuration" %}}) (admin setup).
25+
{{% /alert %}}
26+
2327
1. Find the snapshot:
2428

2529
```bash

‎content/en/docs/next/applications/foundationdb.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,11 @@ resources:
6666
6767
### Backup (Optional)
6868
69+
{{% alert color="warning" %}}
70+
**The chart-level `backup.*` values documented below are deprecated.** The in-chart `FoundationDBBackup` wiring is superseded by the Cozystack backups framework: define a `BackupClass` + `FoundationDB` strategy once, then drive backups via `BackupJob` / `Plan` and restores via `RestoreJob`. Existing tenants with `backup.enabled=true` continue to render the legacy `FoundationDBBackup` CR unchanged. See [Application Backup and Recovery]({{% ref "/docs/next/applications/backup-and-recovery" %}}) (tenant guide) and [Managed Application Backup Configuration]({{% ref "/docs/next/operations/services/managed-app-backup-configuration" %}}) (admin setup).
71+
{{% /alert %}}
72+
73+
6974
```yaml
7075
backup:
7176
enabled: true

‎content/en/docs/next/applications/mariadb.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -110,6 +110,9 @@ more details:
110110

111111

112112
### Backup parameters
113+
{{% alert color="warning" %}}
114+
**The chart-level `backup.*` values documented below are deprecated.** The legacy `mariadb-dump` + `restic` flow is superseded by the Cozystack backups framework: define a `BackupClass` + `MariaDB` strategy once, then drive backups via `BackupJob` / `Plan` and restores via `RestoreJob`. Existing tenants with `backup.enabled=true` continue to render the legacy resources unchanged. See [Application Backup and Recovery]({{% ref "/docs/next/applications/backup-and-recovery" %}}) (tenant guide) and [Managed Application Backup Configuration]({{% ref "/docs/next/operations/services/managed-app-backup-configuration" %}}) (admin setup).
115+
{{% /alert %}}
113116

114117
| Name | Description | Type | Value |
115118
| ------------------------ | ----------------------------------------------- | -------- | ------------------------------------------------------ |

‎content/en/docs/next/applications/postgres.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,10 @@ This managed service is controlled by the CloudNativePG operator, ensuring effic
2727

2828
## Operations
2929

30+
{{% alert color="warning" %}}
31+
**Backups: prefer the `BackupClass` flow.** The chart-level `backup.*` values documented below still configure the Barman object store and S3 credentials that backups read from, but the chart-emitted `ScheduledBackup` and the `bootstrap`-based recovery flow have been **superseded** by the Cozystack backups framework: define a `BackupClass` + `CNPG` strategy once, then drive scheduled backups via `Plan` and restores via `RestoreJob`. See [Application Backup and Recovery]({{% ref "/docs/next/applications/backup-and-recovery" %}}) (tenant guide) and [Managed Application Backup Configuration]({{% ref "/docs/next/operations/services/managed-app-backup-configuration" %}}) (admin setup).
32+
{{% /alert %}}
33+
3034
### How to enable backups
3135

3236
To back up a PostgreSQL application, an external S3-compatible storage is required.

0 commit comments

Comments
 (0)