Repository navigation
Workshop refresh fails on zfs destroy if the dataset is busy, then subsequent tries fail with Instance "stash-$WORKSHOP_NAME-21952d26" already exists #903
Description
Initial attempt at refreshing a local workshop failed with:
STATUS DURATION SUMMARY
Done 58.614s Download "[email protected]" base image
Done 35ms Retrieve "system" SDK
Undone 13ms Create SDK state storage
Done 177ms Run hook "save-state" for "system" SDK
Done 135ms Run hook "save-state" for "omp" SDK
Done 31ms Disconnect interfaces of "omp" SDK
Done 21ms Disconnect interfaces of "system" SDK
Undone 2.768s Stop "dev" workshop
Undone 112ms Uninstall "omp" SDK
Undone 95ms Uninstall "system" SDK
Error 322ms Stash previous "dev" workshop
Error 10.257s Rebuild "dev" workshop
Hold - Start "dev" workshop
Hold - Install "system" SDK
Hold - Run hook "setup-base" for "system" SDK
Hold - Snapshot "system" SDK installation
Hold - Install "omp" SDK
Hold - Run hook "setup-base" for "omp" SDK
Hold - Snapshot "omp" SDK installation
Hold - Configure "dev" workshop timezone
Hold - Mount project directory "/home/ubuntu/Documents/workshop-omp"
Hold - Resolve relations between interfaces of "dev" workshop
Hold - Auto-connect interfaces of "system" SDK
Hold - Auto-connect interfaces of "omp" SDK
Hold - Run hook "setup-project" for "system" SDK
Hold - Run hook "setup-project" for "omp" SDK
Hold - Run hook "restore-state" for "system" SDK
Hold - Run hook "restore-state" for "omp" SDK
Hold - Run hook "check-health" for "system" SDK
Hold - Run hook "check-health" for "omp" SDK
Hold - Remove SDK state storage
Hold - Remove "dev" workshop from stash
Undone 1.017s Disconnect "dev/omp:omp-home" from "dev/system:mount"
Done 82ms Remove "omp" SDK profile
Done 91ms Remove "system" SDK profile
......................................................................
Stash previous "dev" workshop
2026-06-29T14:15:54+07:00 ERROR Failed refreshing instance: Failed running: zfs destroy -r workshop/containers/workshop.1044716967_dev-21952d26: exit status 1 (cannot destroy 'workshop/containers/workshop.1044716967_dev-21952d26': dataset is busy)
......................................................................
Rebuild "dev" workshop
2026-06-29T14:15:43+07:00 ERROR Failed rebuilding instance from image: Error deleting storage volume: Failed running: zfs destroy -r workshop/containers/workshop.1044716967_dev-21952d26: exit status 1 (cannot destroy 'workshop/containers/workshop.1044716967_dev-21952d26': dataset is busy)
The zpool isn't full, and I'm confused as to why this failed. Regardless though, it seems that once this fails once, subsequent attempts are blocked on the instance record:
STATUS DURATION SUMMARY
Done 36ms Retrieve "system" SDK
Undone 17ms Create SDK state storage
Done 205ms Run hook "save-state" for "system" SDK
Done 143ms Run hook "save-state" for "omp" SDK
Done 30ms Disconnect interfaces of "omp" SDK
Done 47ms Disconnect interfaces of "system" SDK
Undone 3.965s Stop "dev" workshop
Undone 115ms Uninstall "omp" SDK
Undone 108ms Uninstall "system" SDK
Error 53ms Stash previous "dev" workshop
Hold - Restore "dev" workshop from "system" snapshot
Hold - Install "system" SDK
Hold - Start "dev" workshop
Hold - Install "omp" SDK
Hold - Run hook "setup-base" for "omp" SDK
Hold - Snapshot "omp" SDK installation
Hold - Configure "dev" workshop timezone
Hold - Mount project directory "/home/ubuntu/Documents/workshop-omp"
Hold - Resolve relations between interfaces of "dev" workshop
Hold - Auto-connect interfaces of "system" SDK
Hold - Auto-connect interfaces of "omp" SDK
Hold - Run hook "setup-project" for "system" SDK
Hold - Run hook "setup-project" for "omp" SDK
Hold - Run hook "restore-state" for "system" SDK
Hold - Run hook "restore-state" for "omp" SDK
Hold - Run hook "check-health" for "system" SDK
Hold - Run hook "check-health" for "omp" SDK
Hold - Remove SDK state storage
Hold - Remove "dev" workshop from stash
Undone 707ms Disconnect "dev/omp:omp-home" from "dev/system:mount"
Done 84ms Remove "omp" SDK profile
Done 101ms Remove "system" SDK profile
......................................................................
Stash previous "dev" workshop
2026-06-29T14:35:57+07:00 ERROR Failed creating instance record: Instance "stash-dev-21952d26" already exists
Activity
jonathan-conder commentedon Jun 29, 2026
Thanks for the report. This is a nasty one because the rollback didn't work, so I don't think there's an easy way to recover from it.
It may be related to canonical/lxd#18362, which was fixed in LXD 6.9. Any chance you were still on 6.8 when this happened?
As for what to do in this scenario: it seems to me like the original workshop is still intact, so you should be able to shell into it and back up any important files (hopefully these are stored in the project folder rather than inside the workshop anyway). Then you can manually delete the stash using:
# lxc delete --project workshop-snapshots.104471696 stash-dev-21952d26It's likely a rare occurrence but if you manage to reproduce the issue reliably I'd be very keen to know how.
barryprice commentedon Jun 29, 2026
I'm tracking
6/stablefor the lxd snap - it looks like it was last refreshed yesterday to 6.9, but I ran into this today, so very recently upgraded but not actually running 6.8:Deleting that
stash-devsnapshot seems to have unblocked therefreshthough, it's all working again now.I will definitely see if I can trigger it again and will update here if so, thanks!
jonathan-conder commentedon Jun 29, 2026
Looks like a similar issue, but not the same one. It's probably going to be tricky to track down. Could you please share a copy of the workshop definition you were using?
jonathan-conder commentedon Jun 29, 2026
Actually now that I think about it, we only fixed one specific case of that issue. It probably is the same bug, just manifested differently. I'll follow up with LXD
jonathan-conder commentedon Jun 30, 2026
I can reproduce the likely cause of this reliably, see canonical/lxd#18629. Our next release should fix it by changing the way we do certain filesystem operations.
barryprice commentedon Jun 30, 2026
Not sure you still need this from the folllow-ups, but fine to share - it's an extremely simple one, but uses a
try-SDK as the version in the store is a way behind upstream - so I rebuild regularly locally to get new features in the meantime. just in case that's relevant - in its entirety:dmitry-lyfar commentedon Jul 7, 2026
Closing this one as workshop has moved away from using sftp which caused the issue in the first place. @barryprice the latest snap is stable contains this update, please ping us if you come across the issue again.