Skip to content

Commit e05c565

Browse files
authored
fix(backups): prepare standalone VMDisk restore targets and add VM backup/restore e2e round-trips (#3427)
## What this PR does Closes the uncovered VM backup path from #3236. `hack/e2e-chainsaw/backup/` held only a disabled Chainsaw port of the old `backup.bats`, written against the retired API (`kind: VirtualMachine`, `BackupJob.storageRef`/`strategyRef`, `velero-strategy-default`), so it guarded nothing. This removes it, fixes the in-place restore of a standalone VMDisk in the backup controller, and adds two real round-trips in the `vminstance` suite that drive the `examples/backups/{vmi,vmdisk}` scripts, so the documented demo and the automated test stay in step. ### Controller: standalone VMDisk in-place restore `collectUnderlyingResources` in `internal/backupcontroller/velerostrategy_controller.go` returned nothing for any kind but VMInstance, so `prepareForRestore` did nothing for a standalone VMDisk: no HelmRelease suspend, no DataVolume delete, no `keepOriginalPVC` rename. An in-place VMDisk restore then raced Flux and CDI over the PVC name the data mover was repopulating. A standalone VMDisk now reports its own disk as its underlying resource, so the restore preparation already used for VMInstance disks suspends the `vm-disk-<name>` HelmRelease, deletes its DataVolume and renames the live PVC to `<name>-orig-<hash>` before Velero recreates it. Unit tests cover both the VMDisk and the VMInstance branch. A VMInstance is halted before its disks are touched, but a VMDisk target is not, and every VMInstance disk is a VMDisk. Renaming the PVC of a disk that a running VM mounts would leave the original `Terminating` behind `pvc-protection`; Velero's PVC restore action skips a PVC that still exists, so the RestoreJob would report `Succeeded` having moved nothing, and the VM would lose its disk on the next restart. An in-place VMDisk restore therefore fails before changing anything when a Pod that has not terminated mounts the disk's PVC. The message names the Pod and tells the user to stop the VM or restore the VMInstance instead. `TestPrepareForRestore_VMDiskInUse` covers a running mounting Pod (refused, PVC not renamed, HelmRelease not suspended), a finished Pod, a Pod on another disk, and a VMInstance target whose own launcher Pod must not refuse it. A VMDisk Backup taken before the disk was recorded as its underlying resource has none, so the disk is derived from the application name (`vm-disk-<name>`): such a restore is prepared and checked for a mounting Pod too. The per-disk `orLabelSelectors` stay a VMInstance concern: a VMDisk strategy already selects its disk, and Velero rejects a Backup or Restore carrying both `labelSelector` and `orLabelSelectors`, so appending them for a VMDisk would break a strategy written with `labelSelector`. When an in-place restore fails after preparation, nothing is rolled back. Resuming the disk's HelmRelease while its PVC is renamed and its DataVolume deleted would have CDI import a fresh disk under the original name, which looks healthy and hides that the data sits in the `-orig-` PVC. The failure message and a Warning event instead name the HelmReleases preparation suspended and the `-orig-` PVCs holding the original data, read back from the cluster so that a failure before a step ran does not name what it would have left, and say that a HelmRelease Velero already restored is resumed, so the disk has to be checked; `examples/backups/vmdisk/README.md` describes the recovery. Tests cover both selector paths, the pre-upgrade Backup, a failed Pod list (the restore requeues and suspends nothing), and all three failures after preparation: a `PartiallyFailed` Velero Restore, several Restores owning the job, and a Restore that cannot be created. Because preparation takes the target apart before Velero validates the Restore, the RestoreJob is refused before preparation, with nothing changed, when the Velero Backup is gone (expired or deleted) or not `Completed`. This is the same single check #4672 added, now run on every pass behind the preparation failure path, so a Backup that expires after preparation started names the leftovers too. A Velero Restore that ends in `FailedValidation` fails the RestoreJob like `Failed`, with Velero's validation errors and the leftovers in the message, instead of polling forever. A HelmRelease suspend or a VM halt that does not happen now stops preparation (requeue) rather than letting the rename and DataVolume delete run under a live release or VM, and a VMInstance's disks are renamed only once no Pod mounts them. A VMDisk held only by a Pod already being deleted is waited for rather than refused, for up to 30 minutes after the RestoreJob started, so a Pod stuck terminating on a lost node fails the job instead of keeping it `Running`. Each of these is pinned by a test that turns red when the guard is removed. Between renaming a kept disk's PVC and deleting its DataVolume, CDI can recreate an empty PVC under the original name for the still-present DataVolume. Velero then skips that PVC as already present ("PVC already exists. Skip restore this PVC."), the RestoreJob reports `Succeeded`, and the VM boots a blank disk. The round-trip's new DataDownload check caught this on a VMInstance restore, where the uid, label and `-orig-` checks had all passed. Once the DataVolume is gone (the wait #4672 added), preparation now deletes a PVC found under the original name (read uncached) before the Velero Restore is created; `TestPrepareForRestore_ClearsPVCRecreatedAfterRename` reproduces the recreation inside the DataVolume delete. Preparation spans several reconciles, so the RestoreJob records a `PreparationStarted` condition before it first changes the target; if recording it fails, preparation retries without changing anything, so a later failure cannot report leftovers that were never made. A refusal on a later pass (the Velero Backup expired, a VM started on the disk) then names what preparation left behind instead of saying nothing was changed, and a preparation still waiting 30 minutes after it started fails the same way instead of polling forever. The vmdisk script starts its consumer Pod only on a PVC that carries the restore label, so it cannot pin a PVC CDI recreated in the rename window. Now that a VMDisk target is prepared, `keepOriginalPVC: false` reaches it as well. Its DataVolume is deleted with orphan propagation, and the bound PVC is deleted and waited for before the Restore (the path #4672 added for VMInstance disks). Left in place, Velero would skip the PVC and the CDI webhook would refuse the DataVolume the resumed HelmRelease creates, leaving the disk's HelmRelease stuck. `TestPrepareForRestore_VMDiskWithoutKeepDeletesOriginalPVC` pins it. ### `vminstance-2-backup-roundtrip` (un-gated in CI) A tiny cirros guest (`u1.nano`, 2Gi, in-place restore only) runs the current API end to end: Velero strategy, BackupClass, a booted VMInstance, an ad-hoc `BackupJob` and an in-place `RestoreJob`, each waited to `Succeeded` and failing fast on `Failed`. The VM must come back `Running`, and its disk PVC must have been created by the restore: a new `metadata.uid`, Velero's `velero.io/restore-name` label, and the `vm-disk-<name>-orig-<hash>` PVC that the rename leaves behind. The label alone does not prove the data moved, because with `existingResourcePolicy: update` Velero also stamps it on a PVC it only patched, and the patch path never reaches the data mover. An in-guest signal is not available on this platform: the KubeVirt CR disables serial-console logging cluster-wide and cirros ships no qemu-guest-agent. ### `vmdisk-2-backup-roundtrip` (un-gated in CI) The disk-only companion: import a 5Gi Ubuntu disk, back it up, restore it in place relying on the controller fix above, and require the same three signals (new uid, restore label, `-orig-` PVC), a `Completed` DataDownload for the restore that wrote bytes, and a Ready HelmRelease. On a `WaitForFirstConsumer` StorageClass, such as the `local` class the CI lane uses, the Velero data mover waits until a scheduled Pod has selected a node for the restored PVC (`WaitPVCConsumed` in Velero's generic restore exposer). A VMInstance gets that from its virt-launcher Pod; a detached disk has none, so its `RestoreJob` stays `Running` until the 30m data-mover prepare timeout fails it. The script therefore starts a restricted-profile `pause` Pod on the new PVC once Velero has created it, standing in for the VM that would attach the disk, and removes it after the restore. `examples/backups/vmdisk/README.md` documents this for operators restoring a detached disk. ### Harness details - The disk `storageClass` is an override (default `replicated`). The Chainsaw steps pass the lane's `COZY_E2E_STORAGE_CLASS` (`local` on the DRBD-off PR gate). - Each harness stands up a throwaway in-cluster BSL (`cozy-e2e-vmi`/`cozy-e2e-vmdisk`) at `seaweedfs-s3.tenant-root:8333` with `insecureSkipTLSVerify` and `checksumAlgorithm: ""` (the #3236 fixes), reusing the platform bucket and `cozy-backups-creds` under a distinct prefix, because the platform `cozy-default` BSL derives the external S3 ingress endpoint, unroutable in the CI cluster. - The script waits are bounded so that their worst case stays under the 40m op timeout (38m for the VMDisk step), and a hung wait trips the script's own named timeout rather than a bare `signal: killed`. - On failure, `catch` also describes the Velero `Restore`, `DataDownload` and `DataUpload` objects in `cozy-velero`. - `skipDelete` plus a `finally` that resumes the suspended HelmReleases, reclaims the `-orig-` PVC and its LINSTOR volume (the PV is flipped back to `Delete` first), and drains the Velero backups on the throwaway BSL, branching on kubectl's exit status, before deleting it. When the drain is not confirmed, the BSL is kept and the step fails, so the Velero Backups are not orphaned behind a green step. - HelmRelease waits use the shared `wait_hr_ready`, which fails fast on `Stalled`. After the VMDisk restore the script first waits for `observedGeneration` to reach the restored spec's `generation`, because the `Ready=True` from before the suspend would otherwise satisfy the wait at once. The `vmi` example is CI-drivable: `run-all.sh`/`cleanup.sh` are non-interactive, the VM is waited to boot before the backup, `volumeSnapshotLocations` is dropped from the demo strategies (`snapshotMoveData` routes data through the BSL), and `BACKUP_STORAGE_LOCATION` defaults to `cozy-default`. `select-e2e.sh` maps `examples/backups/{vmi,vmdisk}` edits to the `vminstance` suite, except `07-restore-to-copy.sh`, which the suite skips (`SKIP_RESTORE_TO_COPY=1`) and so stays docs-only. A `packages/system/velero` change selects the `vminstance` suite as well, since these are the only Tests that run a Velero backup and restore. ### Screenshots N/A — no UI changes. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff (backup controller restore preparation for VMDisk, shell scripts under `examples/backups/`, Chainsaw tests, `hack/select-e2e.sh` routing, a deleted stale suite, one doc line): no `packages/apps|extra` add/rename, no `values.schema.json`/enum/default changes, no CRD/ApplicationDefinition/label/annotation/namespace/metric changes. The controller change alters what a VMDisk `RestoreJob` does in the cluster, not any API surface a downstream repository consumes. `cozystack/examples` holds Terraform for bare nodes, unrelated to the in-repo `examples/` tree. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(backups): an in-place RestoreJob for a standalone VMDisk now suspends the disk's HelmRelease, deletes its DataVolume and (with keepOriginalPVC) renames the original PVC before Velero restores it, instead of racing Flux and CDI over the same PVC; without keepOriginalPVC the original PVC is deleted so Velero recreates it from the backup. Such a restore fails before changing anything when a running Pod, such as the launcher of a VM that uses the disk, mounts the disk's PVC: stop the VM or restore the VMInstance instead. If such a restore fails after the disk was prepared, its message and a Warning event name the HelmRelease preparation suspended and the PVC holding the original data. On a WaitForFirstConsumer StorageClass the data mover still starts only once a Pod consumes the restored PVC, for example when the disk is attached to a VMInstance. An in-place restore with keepOriginalPVC no longer reports Succeeded over an empty PVC that CDI recreated under the disk's name while the original was being kept aside. Adds Chainsaw backup/restore round-trips for VMInstance and standalone VMDisk that run a real Velero backup and restore in CI. ```
2 parents 588d829 + 58b4343 commit e05c565

28 files changed

Lines changed: 2544 additions & 397 deletions

‎docs/agents/e2e-testing.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -70,7 +70,7 @@ Trap-based cleanup was the single biggest source of false failures in the BATS s
7070

7171
### 4. Do not mask cleanup or teardown failures
7272

73-
- Resources a test creates that Chainsaw cannot reclaim — because a controller, not the test, owns them (e.g. the Velero `Backup`, `BackupStorageLocation`, and credentials secret in `cozy-velero` in the disabled `hack/e2e-chainsaw/backup/` suite) — must be pruned explicitly. Do not `|| true` over a stuck delete and leave stale state for the next run.
73+
- Resources a test creates that Chainsaw cannot reclaim — because a controller, not the test, owns them (e.g. the Velero `Backup`/`Restore` in `cozy-velero`, which the `vminstance` backup round-trips prune by deleting the cozystack `Backup` and `RestoreJob` CRs, whose finalizers then clean up the Velero objects asynchronously — the `BackupJob` has no finalizer and its `Backup` carries no owner reference, so deleting a `BackupJob` reclaims nothing — plus an explicit delete of the throwaway `BackupStorageLocation`, sequenced after the Velero `Backup` is gone so it does not race the finalizer) — must be pruned explicitly. Do not `|| true` over a stuck delete and leave stale state for the next run.
7474
- For nested tenants, tear down **child → parent with a hard wait for deletion between** each. Deleting the parent while a child is still uninstalling wedges the parent's cleanup Job on the child namespace, and both stuck uninstalls occupy helm-controller workers past the end of the test — starving whichever suite runs next. The kubernetes suites encode this ordering in `hack/e2e-chainsaw/_lib/run-kubernetes.sh`.
7575
- A test that creates a tenant `Kubernetes` cluster waits for that release to become **Ready** before the assertions it ends on — not before every read of the cluster, which is a different rule and a false one. The chart's post-install hook (`packages/apps/kubernetes/templates/oidc-rbac-job.yaml`) polls for the Kamaji-issued admin kubeconfig in the modes that have one, and that Secret is what the test needs to reach the tenant API, so bringing up a worker or waiting on a child release happens while the parent is still installing. Reading mid-install is fine. What the rule is about is the COMPLETION path: a run that asserts, passes and ends while the parent is still installing turns a green result into a red one in teardown. A failure path that diagnoses and exits mid-install does the same to the install, but that run is already red, so the second red costs noise rather than a wrong verdict — and the script has a number of such exits between the CR and the wait, which the guard does not and should not treat as violations.
7676
- A Helm release cannot be uninstalled while its install action is running, and for this kind the action can outlive the fast assertions — by how much depends on the mode, per the bullet below: `release.cozystack.io/helm-install-disable-wait` on the `ApplicationDefinition` drops the wait on the main resources, but the chart's blocking post-install hook still holds the install. A test that asserts and ends therefore hands the remainder of the install to teardown and is reported red on a step that tests nothing.
Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
#!/bin/bash
2+
# Helper functions and variables for the VMDisk backup/restore demo.
3+
# Source this file in other scripts: source "$(dirname "$0")/00-helpers.sh"
4+
5+
# ANSI color codes
6+
export RED='\033[0;31m'
7+
export GREEN='\033[0;32m'
8+
export YELLOW='\033[1;33m'
9+
export MAGENTA='\033[0;35m'
10+
export CYAN='\033[0;36m'
11+
export WHITE='\033[1;37m'
12+
export NC='\033[0m'
13+
export BOLD='\033[1m'
14+
15+
# Default settings
16+
export NAMESPACE="${NAMESPACE:-tenant-root}"
17+
# The platform ships the SeaweedFS-backed BackupStorageLocation as cozy-default
18+
# (packages/system/backupstrategy-controller). Point the demo/e2e strategy at it
19+
# by default so the flow works out of the box on a backups-enabled cluster;
20+
# override for an external S3 whose BSL carries a different name.
21+
export BACKUP_STORAGE_LOCATION="${BACKUP_STORAGE_LOCATION:-cozy-default}"
22+
23+
# Logging functions (output to stderr to avoid polluting captured output)
24+
log_info() { echo -e "${CYAN}ℹ${NC} $*" >&2; }
25+
log_success() { echo -e "${GREEN}✔${NC} $*" >&2; }
26+
log_warning() { echo -e "${YELLOW}⚠${NC} $*" >&2; }
27+
log_error() { echo -e "${RED}✖${NC} $*" >&2; }
28+
log_step() { echo -e "\n${MAGENTA}${BOLD}▶ $*${NC}" >&2; }
29+
log_command() { echo -e "${WHITE} \$ $*${NC}" >&2; }
30+
log_substep() { echo -e "${CYAN} → $*${NC}" >&2; }
31+
32+
separator() {
33+
echo -e "\n${CYAN}────────────────────────────────────────────────────────────${NC}\n" >&2
34+
}
35+
36+
print_header() {
37+
local title="$1"
38+
echo -e "\n${MAGENTA}${BOLD}== ${title} ==${NC}\n" >&2
39+
}
40+
41+
# wait_for_field, wait_hr_ready and wait_deleted live in one file shared by
42+
# every backup walkthrough, so a fix to one reaches all of them.
43+
# shellcheck source-path=SCRIPTDIR source=../_lib/wait-helpers.sh
44+
source "$(dirname "${BASH_SOURCE[0]}")/../_lib/wait-helpers.sh"
Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
#!/bin/bash
2+
# Step 01: Create the Velero backup/restore strategy for a standalone VMDisk.
3+
set -euo pipefail
4+
5+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
6+
source "$SCRIPT_DIR/00-helpers.sh"
7+
8+
print_header "Step 1: Create VMDisk Velero strategy"
9+
10+
log_step "Creating Velero strategy 'vmdisk-backup-strategy'..."
11+
log_command "kubectl apply -f - (Velero strategy: vmdisk-backup-strategy)"
12+
13+
# A standalone VMDisk is just an HelmRelease + a DataVolume-backed PVC (+ its
14+
# configmaps), all labelled app.kubernetes.io/instance: vm-disk-<name>. Unlike a
15+
# VMInstance there is no VirtualMachine / VMI / launcher pod to capture. With
16+
# snapshotMoveData the CSI data mover streams the volume through the
17+
# BackupStorageLocation, so no VolumeSnapshotLocation is required (matching the
18+
# platform's own cozy-default-velero-vmdisk strategy).
19+
kubectl apply -f - <<'EOF'
20+
apiVersion: strategy.backups.cozystack.io/v1alpha1
21+
kind: Velero
22+
metadata:
23+
name: vmdisk-backup-strategy
24+
spec:
25+
template:
26+
restoreSpec:
27+
existingResourcePolicy: update
28+
includedNamespaces:
29+
- '{{ .Application.metadata.namespace }}'
30+
orLabelSelectors:
31+
- matchLabels:
32+
app.kubernetes.io/instance: 'vm-disk-{{ .Application.metadata.name }}'
33+
- matchLabels:
34+
apps.cozystack.io/application.kind: '{{ .Application.kind }}'
35+
apps.cozystack.io/application.name: '{{ .Application.metadata.name }}'
36+
includedResources:
37+
- helmreleases.helm.toolkit.fluxcd.io
38+
- persistentvolumeclaims
39+
- configmaps
40+
includeClusterResources: false
41+
spec:
42+
includedNamespaces:
43+
- '{{ .Application.metadata.namespace }}'
44+
orLabelSelectors:
45+
- matchLabels:
46+
app.kubernetes.io/instance: 'vm-disk-{{ .Application.metadata.name }}'
47+
- matchLabels:
48+
apps.cozystack.io/application.kind: '{{ .Application.kind }}'
49+
apps.cozystack.io/application.name: '{{ .Application.metadata.name }}'
50+
includedResources:
51+
- helmreleases.helm.toolkit.fluxcd.io
52+
- persistentvolumeclaims
53+
- configmaps
54+
includeClusterResources: false
55+
storageLocation: '{{ .Parameters.backupStorageLocationName }}'
56+
snapshotVolumes: true
57+
snapshotMoveData: true
58+
ttl: 720h0m0s
59+
itemOperationTimeout: 24h0m0s
60+
EOF
61+
62+
log_success "Velero strategy created"
63+
echo -e "\n${GREEN}${BOLD}Next step:${NC} ./02-create-backupclass.sh" >&2
Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
#!/bin/bash
2+
# Step 02: Create a BackupClass binding VMDisk to the Velero strategy.
3+
set -euo pipefail
4+
5+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
6+
source "$SCRIPT_DIR/00-helpers.sh"
7+
8+
print_header "Step 2: Create BackupClass"
9+
10+
log_step "Creating BackupClass 'vmdisk-velero'..."
11+
log_command "kubectl apply -f - (BackupClass: vmdisk-velero)"
12+
13+
# BackupClass is cluster-scoped; a single instance covers every namespace.
14+
kubectl apply -f - <<EOF
15+
apiVersion: backups.cozystack.io/v1alpha1
16+
kind: BackupClass
17+
metadata:
18+
name: vmdisk-velero
19+
spec:
20+
strategies:
21+
- application:
22+
apiGroup: apps.cozystack.io
23+
kind: VMDisk
24+
strategyRef:
25+
apiGroup: strategy.backups.cozystack.io
26+
kind: Velero
27+
name: vmdisk-backup-strategy
28+
parameters:
29+
backupStorageLocationName: ${BACKUP_STORAGE_LOCATION}
30+
EOF
31+
32+
log_success "BackupClass created"
33+
echo -e "\n${GREEN}${BOLD}Next step:${NC} ./03-create-vmdisk.sh" >&2
Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
#!/bin/bash
2+
# Step 03: Create a standalone VMDisk and wait for it to be ready to back up.
3+
set -euo pipefail
4+
5+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
6+
source "$SCRIPT_DIR/00-helpers.sh"
7+
8+
print_header "Step 3: Create VMDisk 'backup-src'"
9+
10+
log_step "Creating VMDisk 'backup-src' in namespace $NAMESPACE..."
11+
log_info "Imports the Ubuntu Noble cloud image into a 5Gi ${VMI_DISK_STORAGE_CLASS:-replicated} disk"
12+
log_command "kubectl apply -f - (VMDisk: backup-src)"
13+
14+
kubectl apply -f - <<EOF
15+
apiVersion: apps.cozystack.io/v1alpha1
16+
kind: VMDisk
17+
metadata:
18+
name: backup-src
19+
namespace: ${NAMESPACE}
20+
spec:
21+
optical: false
22+
source:
23+
http:
24+
url: https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img
25+
storage: 5Gi
26+
# Default 'replicated' for a real DRBD cluster; overridable so the CI lane can
27+
# pass its own class (the PR gate runs DRBD-off and only renders 'local', so a
28+
# hardcoded 'replicated' never binds a PVC and the import hangs).
29+
storageClass: ${VMI_DISK_STORAGE_CLASS:-replicated}
30+
EOF
31+
32+
log_success "VMDisk created"
33+
34+
# Back up only a fully imported disk: the DataVolume import must finish and the
35+
# PVC bind before the backup runs, or Velero captures an empty/partial volume.
36+
log_step "Waiting for the VMDisk to import (HelmRelease Ready, DataVolume Ready, PVC Bound)..."
37+
wait_hr_ready vm-disk-backup-src "${IMPORT_WAIT:-600}"
38+
wait_for_field datavolume.cdi.kubevirt.io vm-disk-backup-src \
39+
'{.status.phase}' Succeeded "$NAMESPACE" "${IMPORT_WAIT:-600}"
40+
wait_for_field pvc vm-disk-backup-src '{.status.phase}' Bound "$NAMESPACE" 300
41+
42+
log_success "VMDisk 'backup-src' is imported and ready to back up"
43+
echo -e "\n${GREEN}${BOLD}Next step:${NC} ./04-create-backupjob.sh" >&2
Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
#!/bin/bash
2+
# Step 04: Back up the standalone VMDisk via a BackupJob.
3+
set -euo pipefail
4+
5+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
6+
source "$SCRIPT_DIR/00-helpers.sh"
7+
8+
print_header "Step 4: Create BackupJob"
9+
10+
log_step "Creating BackupJob 'vmdisk-backup' in namespace $NAMESPACE..."
11+
log_command "kubectl apply -f - (BackupJob: vmdisk-backup)"
12+
13+
kubectl apply -f - <<EOF
14+
apiVersion: backups.cozystack.io/v1alpha1
15+
kind: BackupJob
16+
metadata:
17+
name: vmdisk-backup
18+
namespace: ${NAMESPACE}
19+
spec:
20+
applicationRef:
21+
apiGroup: apps.cozystack.io
22+
kind: VMDisk
23+
name: backup-src
24+
backupClassName: vmdisk-velero
25+
EOF
26+
27+
log_success "BackupJob created"
28+
29+
log_step "Waiting for BackupJob to complete..."
30+
# Default 1800s: the Velero CSI data mover copies the whole disk (5Gi) to S3.
31+
# Overridable (BACKUP_WAIT) so the CI harness can bound it under the Chainsaw op
32+
# timeout. Fail fast if the job flips to Failed instead of polling out the budget.
33+
wait_for_field backupjob vmdisk-backup '{.status.phase}' Succeeded "$NAMESPACE" "${BACKUP_WAIT:-1800}" Failed
34+
35+
BACKUP_NAME=$(kubectl -n "$NAMESPACE" get backupjob vmdisk-backup \
36+
-o jsonpath='{.status.backupRef.name}')
37+
[[ -n "$BACKUP_NAME" ]] || { log_error "BackupJob succeeded but reported no backupRef"; exit 1; }
38+
log_success "BackupJob completed; backup artefact: ${BACKUP_NAME}"
39+
40+
kubectl get backups -n "$NAMESPACE" >&2 || true
41+
echo -e "\n${GREEN}${BOLD}Next step:${NC} ./05-restore-in-place.sh" >&2
Lines changed: 159 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,159 @@
1+
#!/bin/bash
2+
# Step 05: Restore the standalone VMDisk in-place from its backup.
3+
set -euo pipefail
4+
5+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
6+
source "$SCRIPT_DIR/00-helpers.sh"
7+
8+
print_header "Step 5: Restore VMDisk In-Place"
9+
10+
PVC=vm-disk-backup-src
11+
CONSUMER=vm-disk-backup-src-restore-consumer
12+
13+
orig_uid=$(kubectl -n "$NAMESPACE" get pvc "$PVC" -o jsonpath='{.metadata.uid}')
14+
[[ -n "$orig_uid" ]] || { log_error "PVC $PVC has no uid before the restore"; exit 1; }
15+
log_info "PVC $PVC before restore: uid $orig_uid"
16+
17+
log_step "Creating RestoreJob 'vmdisk-restore-in-place' in namespace $NAMESPACE..."
18+
log_info "In-place restore: the original PVC is renamed and the disk is restored from backup"
19+
log_command "kubectl apply -f - (RestoreJob: vmdisk-restore-in-place)"
20+
21+
# The controller suspends the vm-disk-<name> HelmRelease, deletes its DataVolume
22+
# and (keepOriginalPVC) renames the live PVC to <name>-orig-<hash>; Velero then
23+
# creates a fresh PVC under the original name for the data mover to populate.
24+
kubectl apply -f - <<EOF
25+
apiVersion: backups.cozystack.io/v1alpha1
26+
kind: RestoreJob
27+
metadata:
28+
name: vmdisk-restore-in-place
29+
namespace: ${NAMESPACE}
30+
spec:
31+
backupRef:
32+
name: vmdisk-backup
33+
targetApplicationRef:
34+
apiGroup: apps.cozystack.io
35+
kind: VMDisk
36+
name: backup-src
37+
options:
38+
keepOriginalPVC: true # rename the original PVC to <name>-orig-<hash> before restoring
39+
EOF
40+
41+
log_success "RestoreJob created"
42+
43+
# On a WaitForFirstConsumer StorageClass the Velero data mover does not start
44+
# until a scheduled Pod has picked a node for the restored PVC (Velero's
45+
# WaitPVCConsumed). A VMInstance restore gets that from the virt-launcher Pod; a
46+
# detached disk has no consumer, so stand in for the VM that would attach it.
47+
# Only once the new PVC exists: a Pod mounting the original would pin it with
48+
# pvc-protection and block the rename.
49+
start_consumer() {
50+
local mode volume mount
51+
mode=$(kubectl -n "$NAMESPACE" get pvc "$PVC" -o jsonpath='{.spec.volumeMode}')
52+
if [[ "$mode" == "Block" ]]; then
53+
mount='volumeDevices: [{name: disk, devicePath: /dev/disk0}]'
54+
else
55+
mount='volumeMounts: [{name: disk, mountPath: /disk}]'
56+
fi
57+
volume="{name: disk, persistentVolumeClaim: {claimName: $PVC}}"
58+
log_info "Starting consumer Pod $CONSUMER for the restored PVC ($mode)"
59+
kubectl apply -f - <<EOF
60+
apiVersion: v1
61+
kind: Pod
62+
metadata:
63+
name: ${CONSUMER}
64+
namespace: ${NAMESPACE}
65+
spec:
66+
securityContext:
67+
runAsNonRoot: true
68+
runAsUser: 65532
69+
seccompProfile:
70+
type: RuntimeDefault
71+
containers:
72+
- name: pause
73+
image: registry.k8s.io/pause:3.9
74+
securityContext:
75+
allowPrivilegeEscalation: false
76+
readOnlyRootFilesystem: true
77+
capabilities:
78+
drop: ["ALL"]
79+
${mount}
80+
volumes:
81+
- ${volume}
82+
EOF
83+
}
84+
85+
log_step "Waiting for RestoreJob to complete..."
86+
log_command "waiting for restorejob/vmdisk-restore-in-place {.status.phase} == 'Succeeded'"
87+
consumer_started=false
88+
elapsed=0
89+
timeout="${RESTORE_WAIT:-1800}"
90+
while true; do
91+
phase=$(kubectl -n "$NAMESPACE" get restorejob vmdisk-restore-in-place -o jsonpath='{.status.phase}' || true)
92+
[[ "$phase" == "Succeeded" ]] && break
93+
if [[ "$phase" == "Failed" ]]; then
94+
log_error "restorejob/vmdisk-restore-in-place reached terminal failure state 'Failed'"
95+
exit 1
96+
fi
97+
if [[ "$consumer_started" == false ]]; then
98+
# A PVC CDI recreated in the rename window has a new uid too but no
99+
# restore label; mounting it would pin it while the controller clears it.
100+
read -r uid restored < <(kubectl -n "$NAMESPACE" get pvc "$PVC" --ignore-not-found \
101+
-o jsonpath='{.metadata.uid} {.metadata.labels.velero\.io/restore-name}' || true) || true
102+
if [[ -n "${uid:-}" && "$uid" != "$orig_uid" && -n "${restored:-}" ]]; then
103+
start_consumer
104+
consumer_started=true
105+
fi
106+
fi
107+
if [[ $elapsed -ge $timeout ]]; then
108+
log_error "Timeout waiting for restorejob/vmdisk-restore-in-place (current: '$phase', expected: 'Succeeded', consumer started: $consumer_started)"
109+
exit 1
110+
fi
111+
sleep 5
112+
elapsed=$((elapsed + 5))
113+
echo -n "." >&2
114+
done
115+
log_success "restorejob/vmdisk-restore-in-place reached 'Succeeded'"
116+
117+
kubectl -n "$NAMESPACE" delete pod "$CONSUMER" --ignore-not-found --timeout=120s
118+
119+
# Neither the restore-name label nor a new uid proves the data moved: Velero's
120+
# existingResourcePolicy: update stamps the label on a PVC it only patched, and
121+
# CDI recreating the disk would also give it a new uid. The DataDownload check
122+
# below does; the uid and the -orig- PVC show the original was kept aside.
123+
log_step "Verifying the PVC was recreated from the backup..."
124+
wait_for_field pvc "$PVC" '{.status.phase}' Bound "$NAMESPACE" 120
125+
new_uid=$(kubectl -n "$NAMESPACE" get pvc "$PVC" -o jsonpath='{.metadata.uid}')
126+
[[ "$new_uid" != "$orig_uid" ]] || {
127+
log_error "PVC $PVC still has its pre-restore uid $orig_uid — the restore did not recreate it"
128+
exit 1
129+
}
130+
restore_label=$(kubectl -n "$NAMESPACE" get pvc "$PVC" \
131+
-o "jsonpath={.metadata.labels.velero\.io/restore-name}")
132+
[[ -n "$restore_label" ]] || {
133+
log_error "PVC $PVC carries no velero.io/restore-name label — Velero did not restore it"
134+
exit 1
135+
}
136+
all_pvcs=$(kubectl -n "$NAMESPACE" get pvc -o name)
137+
[[ "$all_pvcs" == *"persistentvolumeclaim/${PVC}-orig-"* ]] || {
138+
log_error "no ${PVC}-orig-* PVC — keepOriginalPVC did not rename the original disk"
139+
exit 1
140+
}
141+
# The DataDownload carries the same restore-name label as the PVC; one that
142+
# Completed having written bytes is what shows the data mover filled the disk.
143+
downloads=$(kubectl -n cozy-velero get datadownloads.velero.io -l "velero.io/restore-name=${restore_label}" \
144+
-o jsonpath='{range .items[*]}{.status.phase} {.status.progress.bytesDone}{"\n"}{end}')
145+
grep -Eq '^Completed [1-9][0-9]*$' <<<"$downloads" || {
146+
log_error "no Completed DataDownload with data for restore ${restore_label} — the data mover did not restore $PVC (got: ${downloads:-none})"
147+
exit 1
148+
}
149+
150+
log_step "Waiting for the restored VMDisk to become Ready again..."
151+
# The HelmRelease was Ready before the controller suspended it, and that
152+
# condition outlives the restore; only once Flux has reconciled the restored
153+
# spec does Ready describe the restored release.
154+
hr_generation=$(kubectl -n "$NAMESPACE" get hr vm-disk-backup-src -o jsonpath='{.metadata.generation}')
155+
wait_for_field hr vm-disk-backup-src '{.status.observedGeneration}' "$hr_generation" "$NAMESPACE" 120
156+
wait_hr_ready vm-disk-backup-src 180
157+
158+
kubectl get restorejob vmdisk-restore-in-place -n "$NAMESPACE" -o wide >&2 || true
159+
log_success "Restore completed; VMDisk 'backup-src' recreated from backup (${restore_label}, uid ${new_uid}), Bound and Ready"

0 commit comments

Comments
 (0)