Skip to content

Commit 9cc3a9e

Browse files
androndoandrondo
committed
test(vmi): add VMInstance and standalone VMDisk backup/restore e2e round-trips
The disabled hack/e2e-chainsaw/backup/ suite was a Chainsaw port of the retired backup.bats, written against the old backup API, so the VMInstance and standalone VMDisk Velero paths had no e2e coverage. Replace it with two round-trips in the vminstance suite that drive the validated examples/backups/{vmi,vmdisk} scripts, so the documented demo and the automated test stay in step. Both run un-gated in CI: - vminstance-2-backup-roundtrip boots a tiny cirros VM (u1.nano, 2Gi), backs it up via BackupClass + BackupJob, restores in place, and asserts the VM returns Running off a disk PVC the restore created: a new uid, Velero's restore-name label and the -orig- PVC keepOriginalPVC leaves behind. The label alone is not enough, because existingResourcePolicy: update also stamps it on a PVC Velero only patched, which moves no data. A guest-side boot proof is unavailable: the KubeVirt CR disables serial-console logging and cirros ships no guest-agent. - vmdisk-2-backup-roundtrip imports a 5Gi disk, backs it up, and restores it in place with the same three assertions. The controller prepares the target (suspend HelmRelease, delete DataVolume, rename PVC). On a WaitForFirstConsumer class the Velero data mover waits for a Pod to select a node for the restored PVC, which a detached disk never gets, so the script starts a pause Pod on the new PVC, standing in for the VM that would attach it. The disk storageClass is an override (default replicated; the CI steps pass the lane's COZY_E2E_STORAGE_CLASS, 'local' on the DRBD-off gate) and the script waits are bounded so their worst case stays under the 40m op timeout, sized to keep the whole parallel:1 suite inside the 215m job cap. On failure the catch also dumps the Velero Restore, DataDownload and DataUpload objects. Each harness stands up a throwaway in-cluster BSL (the platform cozy-default BSL derives the external S3 endpoint, unroutable in kind); its finally resumes the suspended HelmReleases, reclaims the keepOriginalPVC orig PVC and its LINSTOR volume, and drains the Velero backups before deleting the BSL. The example scripts are non-interactive so the harness can drive them. Assisted-by: LLM Signed-off-by: Andrey Kolkov <[email protected]>
1 parent 9af1c0f commit 9cc3a9e

21 files changed

Lines changed: 1071 additions & 357 deletions

‎examples/backups/vmdisk/00-helpers.sh‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
#!/bin/bash
2+
# Helper functions and variables for the VMDisk backup/restore demo.
3+
# Source this file in other scripts: source "$(dirname "$0")/00-helpers.sh"
4+
5+
# ANSI color codes
6+
export RED='\033[0;31m'
7+
export GREEN='\033[0;32m'
8+
export YELLOW='\033[1;33m'
9+
export MAGENTA='\033[0;35m'
10+
export CYAN='\033[0;36m'
11+
export WHITE='\033[1;37m'
12+
export NC='\033[0m'
13+
export BOLD='\033[1m'
14+
15+
# Default settings
16+
export NAMESPACE="${NAMESPACE:-tenant-root}"
17+
# The platform ships the SeaweedFS-backed BackupStorageLocation as cozy-default
18+
# (packages/system/backupstrategy-controller). Point the demo/e2e strategy at it
19+
# by default so the flow works out of the box on a backups-enabled cluster;
20+
# override for an external S3 whose BSL carries a different name.
21+
export BACKUP_STORAGE_LOCATION="${BACKUP_STORAGE_LOCATION:-cozy-default}"
22+
23+
# Logging functions (output to stderr to avoid polluting captured output)
24+
log_info() { echo -e "${CYAN}ℹ${NC} $*" >&2; }
25+
log_success() { echo -e "${GREEN}✔${NC} $*" >&2; }
26+
log_warning() { echo -e "${YELLOW}⚠${NC} $*" >&2; }
27+
log_error() { echo -e "${RED}✖${NC} $*" >&2; }
28+
log_step() { echo -e "\n${MAGENTA}${BOLD}▶ $*${NC}" >&2; }
29+
log_command() { echo -e "${WHITE} \$ $*${NC}" >&2; }
30+
log_substep() { echo -e "${CYAN} → $*${NC}" >&2; }
31+
32+
separator() {
33+
echo -e "\n${CYAN}────────────────────────────────────────────────────────────${NC}\n" >&2
34+
}
35+
36+
print_header() {
37+
local title="$1"
38+
echo -e "\n${MAGENTA}${BOLD}== ${title} ==${NC}\n" >&2
39+
}
40+
41+
# wait_for_field, wait_hr_ready and wait_deleted live in one file shared by
42+
# every backup walkthrough, so a fix to one reaches all of them.
43+
# shellcheck source-path=SCRIPTDIR source=../_lib/wait-helpers.sh
44+
source "$(dirname "${BASH_SOURCE[0]}")/../_lib/wait-helpers.sh"

‎examples/backups/vmdisk/01-create-strategy.sh‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
#!/bin/bash
2+
# Step 01: Create the Velero backup/restore strategy for a standalone VMDisk.
3+
set -euo pipefail
4+
5+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
6+
source "$SCRIPT_DIR/00-helpers.sh"
7+
8+
print_header "Step 1: Create VMDisk Velero strategy"
9+
10+
log_step "Creating Velero strategy 'vmdisk-backup-strategy'..."
11+
log_command "kubectl apply -f - (Velero strategy: vmdisk-backup-strategy)"
12+
13+
# A standalone VMDisk is just an HelmRelease + a DataVolume-backed PVC (+ its
14+
# configmaps), all labelled app.kubernetes.io/instance: vm-disk-<name>. Unlike a
15+
# VMInstance there is no VirtualMachine / VMI / launcher pod to capture. With
16+
# snapshotMoveData the CSI data mover streams the volume through the
17+
# BackupStorageLocation, so no VolumeSnapshotLocation is required (matching the
18+
# platform's own cozy-default-velero-vmdisk strategy).
19+
kubectl apply -f - <<'EOF'
20+
apiVersion: strategy.backups.cozystack.io/v1alpha1
21+
kind: Velero
22+
metadata:
23+
name: vmdisk-backup-strategy
24+
spec:
25+
template:
26+
restoreSpec:
27+
existingResourcePolicy: update
28+
includedNamespaces:
29+
- '{{ .Application.metadata.namespace }}'
30+
orLabelSelectors:
31+
- matchLabels:
32+
app.kubernetes.io/instance: 'vm-disk-{{ .Application.metadata.name }}'
33+
- matchLabels:
34+
apps.cozystack.io/application.kind: '{{ .Application.kind }}'
35+
apps.cozystack.io/application.name: '{{ .Application.metadata.name }}'
36+
includedResources:
37+
- helmreleases.helm.toolkit.fluxcd.io
38+
- persistentvolumeclaims
39+
- configmaps
40+
includeClusterResources: false
41+
spec:
42+
includedNamespaces:
43+
- '{{ .Application.metadata.namespace }}'
44+
orLabelSelectors:
45+
- matchLabels:
46+
app.kubernetes.io/instance: 'vm-disk-{{ .Application.metadata.name }}'
47+
- matchLabels:
48+
apps.cozystack.io/application.kind: '{{ .Application.kind }}'
49+
apps.cozystack.io/application.name: '{{ .Application.metadata.name }}'
50+
includedResources:
51+
- helmreleases.helm.toolkit.fluxcd.io
52+
- persistentvolumeclaims
53+
- configmaps
54+
includeClusterResources: false
55+
storageLocation: '{{ .Parameters.backupStorageLocationName }}'
56+
snapshotVolumes: true
57+
snapshotMoveData: true
58+
ttl: 720h0m0s
59+
itemOperationTimeout: 24h0m0s
60+
EOF
61+
62+
log_success "Velero strategy created"
63+
echo -e "\n${GREEN}${BOLD}Next step:${NC} ./02-create-backupclass.sh" >&2

‎examples/backups/vmdisk/02-create-backupclass.sh‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
#!/bin/bash
2+
# Step 02: Create a BackupClass binding VMDisk to the Velero strategy.
3+
set -euo pipefail
4+
5+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
6+
source "$SCRIPT_DIR/00-helpers.sh"
7+
8+
print_header "Step 2: Create BackupClass"
9+
10+
log_step "Creating BackupClass 'vmdisk-velero'..."
11+
log_command "kubectl apply -f - (BackupClass: vmdisk-velero)"
12+
13+
# BackupClass is cluster-scoped; a single instance covers every namespace.
14+
kubectl apply -f - <<EOF
15+
apiVersion: backups.cozystack.io/v1alpha1
16+
kind: BackupClass
17+
metadata:
18+
name: vmdisk-velero
19+
spec:
20+
strategies:
21+
- application:
22+
apiGroup: apps.cozystack.io
23+
kind: VMDisk
24+
strategyRef:
25+
apiGroup: strategy.backups.cozystack.io
26+
kind: Velero
27+
name: vmdisk-backup-strategy
28+
parameters:
29+
backupStorageLocationName: ${BACKUP_STORAGE_LOCATION}
30+
EOF
31+
32+
log_success "BackupClass created"
33+
echo -e "\n${GREEN}${BOLD}Next step:${NC} ./03-create-vmdisk.sh" >&2

‎examples/backups/vmdisk/03-create-vmdisk.sh‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
#!/bin/bash
2+
# Step 03: Create a standalone VMDisk and wait for it to be ready to back up.
3+
set -euo pipefail
4+
5+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
6+
source "$SCRIPT_DIR/00-helpers.sh"
7+
8+
print_header "Step 3: Create VMDisk 'backup-src'"
9+
10+
log_step "Creating VMDisk 'backup-src' in namespace $NAMESPACE..."
11+
log_info "Imports the Ubuntu Noble cloud image into a 5Gi ${VMI_DISK_STORAGE_CLASS:-replicated} disk"
12+
log_command "kubectl apply -f - (VMDisk: backup-src)"
13+
14+
kubectl apply -f - <<EOF
15+
apiVersion: apps.cozystack.io/v1alpha1
16+
kind: VMDisk
17+
metadata:
18+
name: backup-src
19+
namespace: ${NAMESPACE}
20+
spec:
21+
optical: false
22+
source:
23+
http:
24+
url: https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img
25+
storage: 5Gi
26+
# Default 'replicated' for a real DRBD cluster; overridable so the CI lane can
27+
# pass its own class (the PR gate runs DRBD-off and only renders 'local', so a
28+
# hardcoded 'replicated' never binds a PVC and the import hangs).
29+
storageClass: ${VMI_DISK_STORAGE_CLASS:-replicated}
30+
EOF
31+
32+
log_success "VMDisk created"
33+
34+
# Back up only a fully imported disk: the DataVolume import must finish and the
35+
# PVC bind before the backup runs, or Velero captures an empty/partial volume.
36+
log_step "Waiting for the VMDisk to import (HelmRelease Ready, DataVolume Ready, PVC Bound)..."
37+
wait_hr_ready vm-disk-backup-src "${IMPORT_WAIT:-600}"
38+
wait_for_field datavolume.cdi.kubevirt.io vm-disk-backup-src \
39+
'{.status.phase}' Succeeded "$NAMESPACE" "${IMPORT_WAIT:-600}"
40+
wait_for_field pvc vm-disk-backup-src '{.status.phase}' Bound "$NAMESPACE" 300
41+
42+
log_success "VMDisk 'backup-src' is imported and ready to back up"
43+
echo -e "\n${GREEN}${BOLD}Next step:${NC} ./04-create-backupjob.sh" >&2

‎examples/backups/vmdisk/04-create-backupjob.sh‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
#!/bin/bash
2+
# Step 04: Back up the standalone VMDisk via a BackupJob.
3+
set -euo pipefail
4+
5+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
6+
source "$SCRIPT_DIR/00-helpers.sh"
7+
8+
print_header "Step 4: Create BackupJob"
9+
10+
log_step "Creating BackupJob 'vmdisk-backup' in namespace $NAMESPACE..."
11+
log_command "kubectl apply -f - (BackupJob: vmdisk-backup)"
12+
13+
kubectl apply -f - <<EOF
14+
apiVersion: backups.cozystack.io/v1alpha1
15+
kind: BackupJob
16+
metadata:
17+
name: vmdisk-backup
18+
namespace: ${NAMESPACE}
19+
spec:
20+
applicationRef:
21+
apiGroup: apps.cozystack.io
22+
kind: VMDisk
23+
name: backup-src
24+
backupClassName: vmdisk-velero
25+
EOF
26+
27+
log_success "BackupJob created"
28+
29+
log_step "Waiting for BackupJob to complete..."
30+
# Default 1800s: the Velero CSI data mover copies the whole disk (5Gi) to S3.
31+
# Overridable (BACKUP_WAIT) so the CI harness can bound it under the Chainsaw op
32+
# timeout. Fail fast if the job flips to Failed instead of polling out the budget.
33+
wait_for_field backupjob vmdisk-backup '{.status.phase}' Succeeded "$NAMESPACE" "${BACKUP_WAIT:-1800}" Failed
34+
35+
BACKUP_NAME=$(kubectl -n "$NAMESPACE" get backupjob vmdisk-backup \
36+
-o jsonpath='{.status.backupRef.name}')
37+
[[ -n "$BACKUP_NAME" ]] || { log_error "BackupJob succeeded but reported no backupRef"; exit 1; }
38+
log_success "BackupJob completed; backup artefact: ${BACKUP_NAME}"
39+
40+
kubectl get backups -n "$NAMESPACE" >&2 || true
41+
echo -e "\n${GREEN}${BOLD}Next step:${NC} ./05-restore-in-place.sh" >&2

‎examples/backups/vmdisk/05-restore-in-place.sh‎

Lines changed: 159 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,159 @@
1+
#!/bin/bash
2+
# Step 05: Restore the standalone VMDisk in-place from its backup.
3+
set -euo pipefail
4+
5+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
6+
source "$SCRIPT_DIR/00-helpers.sh"
7+
8+
print_header "Step 5: Restore VMDisk In-Place"
9+
10+
PVC=vm-disk-backup-src
11+
CONSUMER=vm-disk-backup-src-restore-consumer
12+
13+
orig_uid=$(kubectl -n "$NAMESPACE" get pvc "$PVC" -o jsonpath='{.metadata.uid}')
14+
[[ -n "$orig_uid" ]] || { log_error "PVC $PVC has no uid before the restore"; exit 1; }
15+
log_info "PVC $PVC before restore: uid $orig_uid"
16+
17+
log_step "Creating RestoreJob 'vmdisk-restore-in-place' in namespace $NAMESPACE..."
18+
log_info "In-place restore: the original PVC is renamed and the disk is restored from backup"
19+
log_command "kubectl apply -f - (RestoreJob: vmdisk-restore-in-place)"
20+
21+
# The controller suspends the vm-disk-<name> HelmRelease, deletes its DataVolume
22+
# and (keepOriginalPVC) renames the live PVC to <name>-orig-<hash>; Velero then
23+
# creates a fresh PVC under the original name for the data mover to populate.
24+
kubectl apply -f - <<EOF
25+
apiVersion: backups.cozystack.io/v1alpha1
26+
kind: RestoreJob
27+
metadata:
28+
name: vmdisk-restore-in-place
29+
namespace: ${NAMESPACE}
30+
spec:
31+
backupRef:
32+
name: vmdisk-backup
33+
targetApplicationRef:
34+
apiGroup: apps.cozystack.io
35+
kind: VMDisk
36+
name: backup-src
37+
options:
38+
keepOriginalPVC: true # rename the original PVC to <name>-orig-<hash> before restoring
39+
EOF
40+
41+
log_success "RestoreJob created"
42+
43+
# On a WaitForFirstConsumer StorageClass the Velero data mover does not start
44+
# until a scheduled Pod has picked a node for the restored PVC (Velero's
45+
# WaitPVCConsumed). A VMInstance restore gets that from the virt-launcher Pod; a
46+
# detached disk has no consumer, so stand in for the VM that would attach it.
47+
# Only once the new PVC exists: a Pod mounting the original would pin it with
48+
# pvc-protection and block the rename.
49+
start_consumer() {
50+
local mode volume mount
51+
mode=$(kubectl -n "$NAMESPACE" get pvc "$PVC" -o jsonpath='{.spec.volumeMode}')
52+
if [[ "$mode" == "Block" ]]; then
53+
mount='volumeDevices: [{name: disk, devicePath: /dev/disk0}]'
54+
else
55+
mount='volumeMounts: [{name: disk, mountPath: /disk}]'
56+
fi
57+
volume="{name: disk, persistentVolumeClaim: {claimName: $PVC}}"
58+
log_info "Starting consumer Pod $CONSUMER for the restored PVC ($mode)"
59+
kubectl apply -f - <<EOF
60+
apiVersion: v1
61+
kind: Pod
62+
metadata:
63+
name: ${CONSUMER}
64+
namespace: ${NAMESPACE}
65+
spec:
66+
securityContext:
67+
runAsNonRoot: true
68+
runAsUser: 65532
69+
seccompProfile:
70+
type: RuntimeDefault
71+
containers:
72+
- name: pause
73+
image: registry.k8s.io/pause:3.9
74+
securityContext:
75+
allowPrivilegeEscalation: false
76+
readOnlyRootFilesystem: true
77+
capabilities:
78+
drop: ["ALL"]
79+
${mount}
80+
volumes:
81+
- ${volume}
82+
EOF
83+
}
84+
85+
log_step "Waiting for RestoreJob to complete..."
86+
log_command "waiting for restorejob/vmdisk-restore-in-place {.status.phase} == 'Succeeded'"
87+
consumer_started=false
88+
elapsed=0
89+
timeout="${RESTORE_WAIT:-1800}"
90+
while true; do
91+
phase=$(kubectl -n "$NAMESPACE" get restorejob vmdisk-restore-in-place -o jsonpath='{.status.phase}' || true)
92+
[[ "$phase" == "Succeeded" ]] && break
93+
if [[ "$phase" == "Failed" ]]; then
94+
log_error "restorejob/vmdisk-restore-in-place reached terminal failure state 'Failed'"
95+
exit 1
96+
fi
97+
if [[ "$consumer_started" == false ]]; then
98+
# A PVC CDI recreated in the rename window has a new uid too but no
99+
# restore label; mounting it would pin it while the controller clears it.
100+
read -r uid restored < <(kubectl -n "$NAMESPACE" get pvc "$PVC" --ignore-not-found \
101+
-o jsonpath='{.metadata.uid} {.metadata.labels.velero\.io/restore-name}' || true) || true
102+
if [[ -n "${uid:-}" && "$uid" != "$orig_uid" && -n "${restored:-}" ]]; then
103+
start_consumer
104+
consumer_started=true
105+
fi
106+
fi
107+
if [[ $elapsed -ge $timeout ]]; then
108+
log_error "Timeout waiting for restorejob/vmdisk-restore-in-place (current: '$phase', expected: 'Succeeded', consumer started: $consumer_started)"
109+
exit 1
110+
fi
111+
sleep 5
112+
elapsed=$((elapsed + 5))
113+
echo -n "." >&2
114+
done
115+
log_success "restorejob/vmdisk-restore-in-place reached 'Succeeded'"
116+
117+
kubectl -n "$NAMESPACE" delete pod "$CONSUMER" --ignore-not-found --timeout=120s
118+
119+
# Neither the restore-name label nor a new uid proves the data moved: Velero's
120+
# existingResourcePolicy: update stamps the label on a PVC it only patched, and
121+
# CDI recreating the disk would also give it a new uid. The DataDownload check
122+
# below does; the uid and the -orig- PVC show the original was kept aside.
123+
log_step "Verifying the PVC was recreated from the backup..."
124+
wait_for_field pvc "$PVC" '{.status.phase}' Bound "$NAMESPACE" 120
125+
new_uid=$(kubectl -n "$NAMESPACE" get pvc "$PVC" -o jsonpath='{.metadata.uid}')
126+
[[ "$new_uid" != "$orig_uid" ]] || {
127+
log_error "PVC $PVC still has its pre-restore uid $orig_uid — the restore did not recreate it"
128+
exit 1
129+
}
130+
restore_label=$(kubectl -n "$NAMESPACE" get pvc "$PVC" \
131+
-o "jsonpath={.metadata.labels.velero\.io/restore-name}")
132+
[[ -n "$restore_label" ]] || {
133+
log_error "PVC $PVC carries no velero.io/restore-name label — Velero did not restore it"
134+
exit 1
135+
}
136+
all_pvcs=$(kubectl -n "$NAMESPACE" get pvc -o name)
137+
[[ "$all_pvcs" == *"persistentvolumeclaim/${PVC}-orig-"* ]] || {
138+
log_error "no ${PVC}-orig-* PVC — keepOriginalPVC did not rename the original disk"
139+
exit 1
140+
}
141+
# The DataDownload carries the same restore-name label as the PVC; one that
142+
# Completed having written bytes is what shows the data mover filled the disk.
143+
downloads=$(kubectl -n cozy-velero get datadownloads.velero.io -l "velero.io/restore-name=${restore_label}" \
144+
-o jsonpath='{range .items[*]}{.status.phase} {.status.progress.bytesDone}{"\n"}{end}')
145+
grep -Eq '^Completed [1-9][0-9]*$' <<<"$downloads" || {
146+
log_error "no Completed DataDownload with data for restore ${restore_label} — the data mover did not restore $PVC (got: ${downloads:-none})"
147+
exit 1
148+
}
149+
150+
log_step "Waiting for the restored VMDisk to become Ready again..."
151+
# The HelmRelease was Ready before the controller suspended it, and that
152+
# condition outlives the restore; only once Flux has reconciled the restored
153+
# spec does Ready describe the restored release.
154+
hr_generation=$(kubectl -n "$NAMESPACE" get hr vm-disk-backup-src -o jsonpath='{.metadata.generation}')
155+
wait_for_field hr vm-disk-backup-src '{.status.observedGeneration}' "$hr_generation" "$NAMESPACE" 120
156+
wait_hr_ready vm-disk-backup-src 180
157+
158+
kubectl get restorejob vmdisk-restore-in-place -n "$NAMESPACE" -o wide >&2 || true
159+
log_success "Restore completed; VMDisk 'backup-src' recreated from backup (${restore_label}, uid ${new_uid}), Bound and Ready"

0 commit comments

Comments
 (0)