Skip to content

Commit e94fcf8

Browse files
authored
feat(backups): RabbitMQ definitions backup via a dedicated strategy driver (#3990)
## What this PR does Adds backup/restore support for the managed `RabbitMQ` application. RabbitMQ's cluster operator ships no backup CRD and RabbitMQ has no consistent online export of message payloads, so this backs up broker **definitions** (vhosts, users, permissions, queues, exchanges, bindings, policies, parameters) over the management HTTP API (`GET`/`POST /api/definitions`) and stores them in the system bucket. Message payloads and point-in-time recovery are intentionally out of scope; a full-data backup would be a Velero volume snapshot, which is a separate strategy. It is delivered as a **first-class `strategy.backups.cozystack.io/Rabbitmq` driver**, following the Altinity pattern (a dedicated driver that renders a one-shot Job for an HTTP-API engine): - `reconcileRabbitmq` / `reconcileRabbitmqRestore` render the strategy's `PodTemplateSpec` — `curl` `GET`/`POST /api/definitions`, pushing/pulling the JSON to S3 with `curl --aws-sigv4` — as a `batch/v1.Job`, covering both restore flows: in-place (into the source broker) and to-copy (into a separate, differently-named RabbitMQ in the same namespace, authenticating with the target's own default-user credentials and reading the source's object key). - A **Go-read precondition** gates each run on the `RabbitmqCluster` reporting `AllReplicasReady` (read through a small `rabbitmqtypes` partial type), so a not-yet-ready broker surfaces a precise `Ready=False` wait on the BackupJob/RestoreJob rather than a Job that starts and fails against an unreachable management API. - The produced `Backup` records the stored object's location on `status.artifact.uri`, rendered from the strategy's `spec.artifactURITemplate`. The `Backup` CRD has no status subresource, so this persists on Create, mirroring the Etcd driver. Dispatch is wired in both controllers (plus `supportedBackupStrategyKinds`), `rabbitmqtypes` is registered in the scheme, and RBAC grants read on `rabbitmqclusters.rabbitmq.com` (the Job itself is already covered by the existing `batch/jobs` grant). It ships `cozy-default-rabbitmq` (a `$bucketName`-gated `Rabbitmq` strategy) and routes `apps.cozystack.io/RabbitMQ` to it in the `cozy-default` BackupClass, which also enrols it in the `DefaultObjectsGate`. The RabbitMQ application chart is left untouched — it had no prior backup wiring, and the strategy carries the S3 coordinates itself. Unit tests cover the driver's pure helpers (applicationRef validation, cluster-name derivation, the readiness-message builder, artifact-URI rendering, parameter round-trip) and the reconcile precondition gate (defers with `Ready=False` and no Job until the cluster is ready, then proceeds). A runnable example under `examples/backups/rabbitmq/` and an honest-roundtrip Chainsaw e2e under `hack/e2e-chainsaw/rabbitmq/` seed a sentinel vhost, back it up, drop it, and assert it round-trips through object storage on both restore flows. The Job pod reuses the existing `platform-migrations` image (curl + ca-certificates) under a new `backupStrategyController.rabbitmqBackupClientImage` key, stamped in lockstep with `chBackupClientImage` in `packages/core/platform/Makefile` so the repository stays pinned at a single digest (`hack/image-pin-consistency.bats`). A dedicated slim image built by this package is a follow-up. **Validation:** `go build`/`go vet`/`gofmt`/`go test ./internal/backupcontroller/... ./internal/template/...`, deepcopy + Rabbitmq CRD generation, `helm template` proving the CRD-ordering gate both on and off, `helm lint`, and `helm unittest` (18/18). The backup + in-place + to-copy roundtrip was verified end-to-end on a dev cluster (a real management-API export and a real S3 `--aws-sigv4` upload/download against SeaweedFS), so the driver's core contract is exercised on a real cluster; the in-CI Chainsaw run is the remaining automated evidence. The object-cleanup, terminating-namespace teardown and readiness-deadline paths added during review are unit-tested (including `Reconcile`-level finalizer retention and the terminating-namespace release) but not yet cluster-exercised. (Note found while testing: RabbitMQ 4.2 is OOMKilled on the `nano` preset, so the example/e2e use `c1.small`.) ### Screenshots Not applicable — no UI changes. ### Downstream repositories Walked the trigger map against the diff. This is an additive backup strategy plus a new strategy CRD; no downstream repository is mechanically forced by the change. A documentation page for RabbitMQ backup on [cozystack/website](https://github.com/cozystack/website) would be a reasonable follow-up, but it is not required by this diff — leaving the boxes unticked for a human to decide rather than ticking "no downstream" to clear the checklist. ### Release note ```release-note feat(backups): add RabbitMQ definitions backup/restore via a dedicated strategy.backups.cozystack.io/Rabbitmq driver; it gates each run on the RabbitmqCluster being ready and records the stored object on Backup.status.artifact.uri. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added RabbitMQ backup and restore support for definitions, including vhosts, users, queues, exchanges, bindings, policies, and related settings. * Added in-place and restore-to-copy workflows with readiness checks. * Added scheduled and on-demand backup examples. * Integrated RabbitMQ backups with the default backup configuration. * Backups store definitions only; message payloads and point-in-time recovery are not included. * **Documentation** * Added a complete RabbitMQ backup and restore demonstration guide. * **Tests** * Added end-to-end coverage for readiness and backup/restore round trips. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 88bae1f + 6eba4d9 commit e94fcf8

37 files changed

Lines changed: 3547 additions & 27 deletions
Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
// SPDX-License-Identifier: Apache-2.0
2+
// Package v1alpha1 defines strategy.backups.cozystack.io API types.
3+
//
4+
// Group: strategy.backups.cozystack.io
5+
// Version: v1alpha1
6+
package v1alpha1
7+
8+
import (
9+
corev1 "k8s.io/api/core/v1"
10+
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
11+
"k8s.io/apimachinery/pkg/runtime"
12+
)
13+
14+
func init() {
15+
SchemeBuilder.Register(func(s *runtime.Scheme) error {
16+
s.AddKnownTypes(GroupVersion,
17+
&Rabbitmq{},
18+
&RabbitmqList{},
19+
)
20+
return nil
21+
})
22+
}
23+
24+
const (
25+
RabbitmqStrategyKind = "Rabbitmq"
26+
)
27+
28+
// +kubebuilder:object:root=true
29+
// +kubebuilder:subresource:status
30+
// +kubebuilder:resource:scope=Cluster
31+
32+
// Rabbitmq defines a backup strategy that exports and imports RabbitMQ broker
33+
// DEFINITIONS (vhosts, users, permissions, queues, exchanges, bindings,
34+
// policies, parameters) via the management HTTP API, run as a one-shot
35+
// batch/v1.Job per BackupJob / RestoreJob. Message payloads are out of scope -
36+
// RabbitMQ has no consistent online export of them, so a full-data backup is a
37+
// Velero volume snapshot, not this strategy. The driver gates on the target
38+
// RabbitmqCluster being ready, wraps the templated PodTemplateSpec in a Job,
39+
// and records the stored object on the Cozystack Backup artifact.
40+
//
41+
// Restore is a MERGE, not a reset: it POSTs the definitions to /api/definitions,
42+
// which creates or updates what is in the export and never deletes anything, so
43+
// objects created after the backup survive the restore and the broker is not
44+
// returned to its exact backup-time state. A to-copy restore also imports the
45+
// source's users (with password hashes) into the target.
46+
type Rabbitmq struct {
47+
metav1.TypeMeta `json:",inline"`
48+
metav1.ObjectMeta `json:"metadata,omitempty"`
49+
50+
Spec RabbitmqSpec `json:"spec,omitempty"`
51+
Status RabbitmqStatus `json:"status,omitempty"`
52+
}
53+
54+
// +kubebuilder:object:root=true
55+
56+
// RabbitmqList contains a list of Rabbitmq backup strategies.
57+
type RabbitmqList struct {
58+
metav1.TypeMeta `json:",inline"`
59+
metav1.ListMeta `json:"metadata,omitempty"`
60+
Items []Rabbitmq `json:"items"`
61+
}
62+
63+
// RabbitmqSpec specifies the desired Rabbitmq-driven backup strategy.
64+
type RabbitmqSpec struct {
65+
// Template is the PodTemplateSpec the driver wraps in a batch/v1.Job for
66+
// both backup and restore runs. Helm-style Go templates are supported in
67+
// every string field. The available context is:
68+
// .Application - the application object (apps.cozystack.io/RabbitMQ)
69+
// .Release.Name, .Release.Namespace - shorthand for the application
70+
// name/namespace
71+
// .Mode - "backup" or "restore"
72+
// .BackupName - the per-run identity used to scope the object key so
73+
// distinct backups of one application do not overwrite
74+
// each other: the BackupJob name on backup, the source
75+
// Backup name on restore.
76+
// .ArtifactURI - the stored object's location and single source of truth:
77+
// the rendered ArtifactURITemplate on backup, and the value
78+
// recorded on the source Backup's status.artifact.uri on
79+
// restore (so restore reads the exact object backup wrote).
80+
// .Parameters - map[string]string from the matched BackupClassStrategy
81+
// .Backup - the source Backup metadata (only set on restore);
82+
// exposes .Name, .Namespace, and
83+
// .ApplicationRef.{APIGroup,Kind,Name} so a to-copy
84+
// restore can address the source release's object key.
85+
//
86+
// The template is stored schemaless (its content is preserved but not
87+
// validated by the apiserver): the driver renders and validates the pod
88+
// spec, and generating the full PodTemplateSpec OpenAPI schema here would
89+
// add ~590 KB to this CRD. The backupstrategy-controller chart inlines
90+
// every definitions/*.yaml into one Helm release, whose state Secret is
91+
// capped at 1 MiB - a fourth full PodTemplateSpec schema (after Altinity,
92+
// Job and FoundationDB) pushes that Secret over the cap and fails install.
93+
//
94+
// +kubebuilder:validation:Schemaless
95+
// +kubebuilder:pruning:PreserveUnknownFields
96+
Template corev1.PodTemplateSpec `json:"template"`
97+
98+
// ArtifactURITemplate is the object location the backup writes to, rendered
99+
// against the run context and recorded on the produced Backup's
100+
// status.artifact.uri. It is the single source of truth for where the
101+
// export lives: the backup Job writes exactly this object, and restore reads
102+
// the recorded value back rather than reconstructing a key. It is REQUIRED
103+
// for backups - a BackupJob whose resolved strategy leaves it empty fails,
104+
// because nothing would record where the export was stored. Ignored on
105+
// restore (the recorded value is used instead).
106+
// +optional
107+
ArtifactURITemplate string `json:"artifactURITemplate,omitempty"`
108+
}
109+
110+
// RabbitmqStatus reports observed state for the strategy CR.
111+
type RabbitmqStatus struct {
112+
// Conditions holds the latest available observations.
113+
// +optional
114+
// +listType=map
115+
// +listMapKey=type
116+
Conditions []metav1.Condition `json:"conditions,omitempty"`
117+
}

‎api/backups/strategy/v1alpha1/zz_generated.deepcopy.go‎

Lines changed: 97 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎cmd/backupstrategy-controller/main.go‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,7 @@ import (
5252
"github.com/cozystack/cozystack/internal/backupcontroller/mongodbapp"
5353
"github.com/cozystack/cozystack/internal/backupcontroller/postgresapp"
5454
"github.com/cozystack/cozystack/internal/backupcontroller/psmdbtypes"
55+
"github.com/cozystack/cozystack/internal/backupcontroller/rabbitmqtypes"
5556
velerov1 "github.com/vmware-tanzu/velero/pkg/apis/velero/v1"
5657
// +kubebuilder:scaffold:imports
5758
)
@@ -77,6 +78,7 @@ func init() {
7778
utilruntime.Must(foundationdbapp.AddToScheme(scheme))
7879
utilruntime.Must(etcdtypes.AddToScheme(scheme))
7980
utilruntime.Must(etcdapp.AddToScheme(scheme))
81+
utilruntime.Must(rabbitmqtypes.AddToScheme(scheme))
8082
// +kubebuilder:scaffold:scheme
8183
}
8284

@@ -262,9 +264,10 @@ func main() {
262264
}
263265

264266
if err = (&backupcontroller.BackupReconciler{
265-
Client: mgr.GetClient(),
266-
Scheme: mgr.GetScheme(),
267-
Recorder: mgr.GetEventRecorderFor("backup-controller"),
267+
Client: mgr.GetClient(),
268+
Scheme: mgr.GetScheme(),
269+
Recorder: mgr.GetEventRecorderFor("backup-controller"),
270+
CredentialsConfig: credentialsConfig,
268271
}).SetupWithManager(mgr); err != nil {
269272
setupLog.Error(err, "unable to create controller", "controller", "Backup")
270273
os.Exit(1)

‎docs/operations/backup-classes.md‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ Tenants reference `cozy-default` from `BackupJob`, `Plan`, and `RestoreJob` reso
1515
| `apps.cozystack.io/ClickHouse` | Altinity `clickhouse-backup` sidecar | `strategy.backups.cozystack.io/Altinity` `cozy-default-altinity` |
1616
| `apps.cozystack.io/MongoDB` | Percona psmdb operator (pbm) dump | `strategy.backups.cozystack.io/MongoDB` `cozy-default-mongodb` |
1717
| `apps.cozystack.io/Etcd` | etcd-operator snapshot | `strategy.backups.cozystack.io/Etcd` `cozy-default-etcd` |
18+
| `apps.cozystack.io/RabbitMQ` | RabbitMQ definitions (management API) | `strategy.backups.cozystack.io/Rabbitmq` `cozy-default-rabbitmq` |
1819
| `apps.cozystack.io/VMInstance` | Velero + kubevirt-velero-plugin | `strategy.backups.cozystack.io/Velero` `cozy-default-velero-vminstance` |
1920
| `apps.cozystack.io/VMDisk` | Velero | `strategy.backups.cozystack.io/Velero` `cozy-default-velero-vmdisk` |
2021

@@ -119,7 +120,7 @@ The bucket lives in `tenant-root` and is provisioned through the `apps.cozystack
119120

120121
| Key | Consumer |
121122
|-----------------------------------------------|-------------------------------------------|
122-
| `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` | CNPG, MariaDB, Etcd |
123+
| `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` | CNPG, MariaDB, Etcd, RabbitMQ |
123124
| `accessKey` / `secretKey` (plus `bucketName`, `endpoint`, `region`) | ClickHouse sidecar |
124125
| `cloud` | Velero (AWS credentials file format) |
125126
| `blob_credentials.json` | FoundationDB backup_agent |
@@ -196,7 +197,7 @@ The default-objects gate emits three more:
196197

197198
## Admin overrides for `cozy-default`
198199

199-
`cozy-default` is rendered by the `backupstrategy-controller` chart and owned by Flux's helm-controller. **Direct `kubectl edit backupclass cozy-default` is overwritten on the next helm reconcile** — the same applies to its companion `strategy.backups.cozystack.io/*` CRs (`cozy-default-cnpg`, `cozy-default-etcd`, `cozy-default-mariadb`, `cozy-default-altinity`, `cozy-default-mongodb`, `cozy-default-foundationdb`, the two `cozy-default-velero-*`). The supported override path is the `backupStorage` block on the **`platform` component** of the `cozystack.cozystack-platform` Package CR:
200+
`cozy-default` is rendered by the `backupstrategy-controller` chart and owned by Flux's helm-controller. **Direct `kubectl edit backupclass cozy-default` is overwritten on the next helm reconcile** — the same applies to its companion `strategy.backups.cozystack.io/*` CRs (`cozy-default-cnpg`, `cozy-default-etcd`, `cozy-default-mariadb`, `cozy-default-altinity`, `cozy-default-mongodb`, `cozy-default-foundationdb`, `cozy-default-rabbitmq`, the two `cozy-default-velero-*`). The supported override path is the `backupStorage` block on the **`platform` component** of the `cozystack.cozystack-platform` Package CR:
200201

201202
```yaml
202203
apiVersion: cozystack.io/v1alpha1
@@ -243,6 +244,7 @@ The defaults aim at a reasonable middle (30-day retention, gzip compression wher
243244
- **MongoDB strategy**: `storageName` (which `spec.backup.storages` entry on the psmdb cluster to use), `type` (`logical`), `compressionType` / `compressionLevel`. The S3 target itself is tuned via `backup.*` values on the MongoDB release (see [MongoDB: the application owns the backup storage](#mongodb-the-application-owns-the-backup-storage)).
244245
- **Altinity strategy**: tune the `clickhouse-backup` sidecar via `backup.*` values on the ClickHouse release; the strategy Pod is a thin HTTP client. When the S3 endpoint's certificate is signed by a private CA rather than a publicly-trusted one — SeaweedFS's in-cluster `:8333` being the case in point — point `backup.endpointCA` at a Secret holding that CA bundle; the chart mounts it into the sidecar and adds it to the trust store via `SSL_CERT_DIR`, which supplements the system CA set rather than replacing it.
245246
- **FoundationDB strategy**: `snapshotPeriodSeconds`, `agentCount`, `urlParameters[]`.
247+
- **Rabbitmq strategy**: `artifactURITemplate` (the `<namespace>/<application>/<backup-name>/definitions.json` object-key layout) and the pod `template` (image, resources). The backup is a definitions export over the management API — vhosts, users, permissions, queues, exchanges, bindings, policies, parameters — so message payloads are out of scope and there is no point-in-time recovery; a full message-data backup is a Velero volume snapshot instead. **Restore is a merge, not a reset**: importing definitions (`POST /api/definitions`) creates or updates what the export contains and never deletes, so anything created after the backup survives the restore and the broker is not returned to its exact backup-time state. A to-copy restore imports the source's users (with password hashes) into the target, so those source credentials become valid logins there. Unlike the operator-backed strategies (whose engine owns archive retention), the Rabbitmq driver owns its object outright and deletes it from the bucket when its `Backup` is deleted — via a one-shot Job the Backup's removal waits on — so a retention-pruned `Plan` does not normally accumulate objects. The delete is **best-effort**: when it cannot run from here the Backup is released without deleting the object (with a `Warning` Event naming what was left behind), so the Backup — and any namespace being torn down — never wedges. The named give-up conditions are: the namespace is terminating (`CREATE` is forbidden there), the strategy CR or the projected credentials are unavailable, the rendered Job name is invalid, or the skip annotation below is set. Otherwise a genuinely failing delete (object storage unreachable) keeps the `Backup` in `Terminating` (a visible signal to act on) and retries. To release such a stuck `Backup`, annotate it `backups.cozystack.io/skip-artifact-cleanup: "true"` — cleanup then skips the delete and lets the `Backup` go, leaving the object in the bucket to be reclaimed manually or by a bucket lifecycle policy (`kubectl annotate backup <name> -n <namespace> backups.cozystack.io/skip-artifact-cleanup=true`).
246248
- **Velero strategy (VMInstance / VMDisk)**: `ttl`, `includedResources[]`, `excludedResources[]`.
247249
- **Etcd strategy**: today the strategy is path-only; combine with `Plan.spec.retentionPolicy` for trim cadence.
248250

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
# Cozystack-managed S3 bucket for storing this demo's RabbitMQ definitions.
2+
# The Bucket controller provisions a Secret named "bucket-<bucket>-<user>"
3+
# per user (the bucket app carries the "bucket-" release prefix); run-all.sh
4+
# reads the "backup" user's BucketInfo and projects its access keys into the
5+
# rabbitmq-backup-creds Secret the strategy below references.
6+
#
7+
# The platform ships a cozy-default-rabbitmq strategy that writes to the shared
8+
# cozy-backups system bucket; this demo instead provisions its own Bucket and a
9+
# matching strategy so the round-trip is self-contained and its objects are
10+
# torn down with the demo (see cleanup.sh).
11+
---
12+
apiVersion: apps.cozystack.io/v1alpha1
13+
kind: Bucket
14+
metadata:
15+
name: rabbitmq-backups
16+
spec:
17+
users:
18+
backup:
19+
readonly: false

0 commit comments

Comments
 (0)