Skip to content

Commit 549b32e

Browse files
kjellyclaude
andcommitted
fix(skills): never rm -rf the shared VM image directory
Four vm-target-spec-testing passages told agents to run `sudo rm -rf /var/lib/libvirt/images/pilot/<name>` after `vm-target down`, and the clean-room contract's first teardown step removed every VM "belonging to the runbook" before saying not to touch VMs you did not create. On this host every clone and session shares the pilot data dir and that image directory (`vm-target list` here shows other sessions' VMs, and the directory holds other runs' leftovers and the base images), so both could delete another session's VM. The rm -rf was also never needed: `down` already removes the per-target directory (teardown -> os.RemoveAll(t.Dir)), and nothing is shared between host and VM, so a Docker bind mount inside the VM lives in the overlay that down deletes. The "bind-mount survives vm-target down" trap is rewritten as what it is: state that survives a re-run on the same VM, cleared by `vm-target reset` or `down`. - vm-target-spec-testing SKILL.md §8, vm-target-basics.md, container-in- vmtesting.md Trap 3, case-study-freeipa.md: drop the rm -rf; down only VMs this session created; a directory surviving down is a teardown bug to report. - clean-room contract §4 step 1: record VMs, remove only this run's own or user-named disposable ones, use fresh names for the rest. - TestDown_RemovesState now asserts the directory is gone (fails if teardown stops removing it). Checked on this host: `vm-target up`, a file written inside the VM, `vm-target down` -> the per-target directory, libvirt domain and state record are all gone. Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent e5949e8 commit 549b32e

6 files changed

Lines changed: 63 additions & 27 deletions

File tree

‎.agents/skills/_shared/clean-room-contract.md‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -93,8 +93,15 @@ source, that is a **stop condition** — see §8.
9393

9494
In order, and recorded:
9595

96-
1. Record the existing VM inventory, then remove every VM belonging to the
97-
runbook under test.
96+
1. Record the existing VM inventory (`pilot vm-target list`, `virsh list --all`).
97+
Remove only VMs this run created in an earlier attempt, or that the user
98+
explicitly named as disposable for this run, each with `pilot vm-target down`
99+
(`topology down` for a topology this run brought up). Every other VM stays,
100+
even one whose name matches the runbook's example: sibling clones and other
101+
sessions share the pilot data dir and the libvirt host, so `vm-target list`
102+
shows their VMs too. Use fresh run-scoped names instead (below). Never delete
103+
files under `/var/lib/libvirt/images/pilot` by hand; `down` removes a
104+
target's directory itself.
98105
2. Remove the entire disposable workspace.
99106
3. **Prove prohibited stale state is absent** — don't assume removal worked.
100107
Check for surviving VMs, workspace directories, generated inventory, vault

‎.agents/skills/vm-target-spec-testing/SKILL.md‎

Lines changed: 17 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -305,22 +305,30 @@ go run ./cmd/pilot vm-target down --name <client-vm> # if multi-VM
305305
go run ./cmd/pilot vm-target down --name <server-vm>
306306
```
307307

308-
**Always down after the run**, even on failure. The VM overlay is a
309-
qcow2 that grows with every mutation; leaving it running clutters the
310-
host's `virsh list`.
308+
**Always down after the run**, even on failure, but **only the VMs this
309+
session created** (or that the user explicitly named as disposable). The VM
310+
overlay is a qcow2 that grows with every mutation; leaving it running clutters
311+
the host's `virsh list`. `vm-target list` is not "your" list: sibling clones and
312+
other sessions share the same pilot data dir (`~/.local/share/pilot`) and the
313+
same image directory, so it shows their VMs too. Record the names you `up` and
314+
tear down only those.
311315
312316
**Iterating on a playbook that isn't green yet?** Don't `down`+`up`
313317
between attempts -- `pilot vm-target reset --name <name>` reverts to the
314318
pristine post-`up` snapshot in seconds. Fix the playbook, `reset`, `run`
315319
again. Save `down` for when the spec is actually green (§9) or you're
316320
switching to a different base image/disk size.
317321

318-
**If the apply playbook wrote shared state outside the VM's qcow2** (e.g.
319-
a bind-mounted `--data-dir`), remove it so the next run starts clean:
320-
321-
```bash
322-
sudo rm -rf /var/lib/libvirt/images/pilot/<name>
323-
```
322+
`down` already deletes the per-target directory
323+
`/var/lib/libvirt/images/pilot/<name>/` (overlay, seed ISO, SSH key, run
324+
transcripts), the static DHCP reservation, the SSH control directory and the
325+
cached Ansible facts, so the next `up` of that name starts from a fresh overlay
326+
and key. Everything a playbook writes inside the VM, Docker bind mounts
327+
included, lives in that overlay; nothing is shared between host and VM.
328+
**Never `rm -rf` anything under `/var/lib/libvirt/images/pilot` by hand**: other
329+
sessions' VMs live in the same directory and `images/` holds the shared base
330+
images. If a directory survives a successful `down`, that is a teardown bug
331+
(AGENTS.md §5.14): report it with `ls -la` output, don't delete it.
324332

325333
## 9. After a green run: promote the spec
326334

‎.agents/skills/vm-target-spec-testing/references/case-study-freeipa.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -121,9 +121,11 @@ go run ./cmd/pilot vm-target exec --name freeipa-client -- getent passwd pil
121121
```bash
122122
go run ./cmd/pilot vm-target down --name freeipa-client
123123
go run ./cmd/pilot vm-target down --name freeipa-server
124-
sudo rm -rf /var/lib/libvirt/images/pilot/freeipa-server
125124
```
126125

126+
`down` deletes each VM's directory under `/var/lib/libvirt/images/pilot`;
127+
nothing else needs removing.
128+
127129
## Lessons extracted for future multi-VM specs
128130

129131
1. **Always ping cross-VM before applying any playbook.** The `default`

‎.agents/skills/vm-target-spec-testing/references/container-in-vmtesting.md‎

Lines changed: 14 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -56,21 +56,27 @@ docker exec <container> /usr/sbin/ipactl status
5656
docker exec <container> bash -lc 'ipactl status'
5757
```
5858

59-
## Trap 3: bind-mount survives vm-target down
59+
## Trap 3: bind-mounted state survives a re-run
6060

6161
Playbooks that use `volumes: ["/host/path:/container/data:Z"]` write
62-
state **outside** the VM's qcow2. A subsequent `vm-target up` reuses
63-
the same host directory. This is great for idempotency (re-applying
64-
skips already-installed services) and terrible for debugging (a
65-
half-baked install silently masks the real failure).
62+
state to `/host/path` **on the VM**, outside the container. Recreating the
63+
container or re-running the playbook on the same VM reuses it. This is
64+
great for idempotency (re-applying skips already-installed services) and
65+
terrible for debugging (a half-baked install silently masks the real
66+
failure).
6667

67-
**Pattern**: after a failed run, delete the per-target dir:
68+
**Pattern**: after a failed run, put the VM back to its post-`up` state
69+
before retrying:
6870

6971
```bash
70-
pilot vm-target down --name <name>
71-
sudo rm -rf /var/lib/libvirt/images/pilot/<name>
72+
pilot vm-target reset --name <name> # revert to the pristine post-up snapshot
7273
```
7374

75+
`pilot vm-target down` also clears it: the VM's whole filesystem, bind-mounted
76+
paths included, lives in the per-target overlay that `down` deletes. Do not
77+
`rm -rf` under `/var/lib/libvirt/images/pilot` yourself; other sessions' VMs
78+
live there.
79+
7480
## Trap 4: community.docker.docker_container detach:false does not surface exit codes
7581

7682
When `detach: false` (the container runs in the foreground and ansible

‎.agents/skills/vm-target-spec-testing/references/vm-target-basics.md‎

Lines changed: 15 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -314,17 +314,25 @@ Key behaviors:
314314

315315
## Clean slate
316316

317-
When you need a truly clean VM:
317+
When you need a truly clean VM of your own (one this session created):
318318

319319
```bash
320-
pilot vm-target down --name <name> # destroy + undefine
321-
sudo rm -rf /var/lib/libvirt/images/pilot/<name> # delete qcow2 + seed + key
320+
pilot vm-target down --name <name> # destroy, undefine, delete /var/lib/libvirt/images/pilot/<name>/
321+
pilot vm-target up --name <name> ...
322322
```
323323

324-
Deleting the per-target dir ensures the next `up` gets a fresh overlay
325-
and a new SSH key -- necessary when the playbook wrote persistent state
326-
outside the VM's filesystem (e.g. bind-mounted data dirs, docker
327-
volumes on the host).
324+
`down` removes the per-target directory (overlay, seed ISO, SSH key, run
325+
transcripts), the static DHCP reservation, the SSH control directory and the
326+
cached facts, so the next `up` gets a fresh overlay and a new SSH key.
327+
Everything the VM wrote, Docker bind mounts and volumes included, is inside
328+
that overlay: no host directory is shared into the VM. To go back to the
329+
post-`up` state without re-provisioning, `pilot vm-target reset --name <name>`
330+
is faster.
331+
332+
Never `rm -rf` under `/var/lib/libvirt/images/pilot` yourself. Every clone and
333+
session on the host shares that directory (and `images/` holds the base
334+
images); a hand delete can remove another session's running VM disk. A
335+
directory left behind after a successful `down` is a teardown bug to report.
328336

329337
## Useful diagnostics
330338

‎internal/vmtarget/vmtarget_test.go‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -250,6 +250,11 @@ func TestDown_RemovesState(t *testing.T) {
250250
if len(all) != 0 {
251251
t.Fatalf("List after Down: %+v", all)
252252
}
253+
// The vm-target skills tell agents that down deletes the per-target dir and
254+
// never to rm -rf under the shared image directory themselves.
255+
if _, err := os.Stat(filepath.Join(m.vmDir, "doomed")); !os.IsNotExist(err) {
256+
t.Fatalf("Down left the target dir behind: %v", err)
257+
}
253258
}
254259

255260
func TestDown_UnknownNameIsIdempotent(t *testing.T) {

0 commit comments

Comments
 (0)