remote build: stale git repositories on launchpad #1205

Bug Description

Hi, I've found ~20k stale Launchpad git repositories that were used by a remote build but failed to be cleaned.

The only deletion for these repositories is RemoteBuildService.cleanup(), which runs in two places in commands/remote.py : normal completion, and Ctrl-C followed by "Cancel builds?" = yes. These paths leak:

• Failed start. After the repository is pushed, a failure in recipe creation, in the build request, or on a timeout leaves the repository behind. A failed push leaves it behind too.
• Failed recipe deletion. cleanup() aborted before deleting the repository when recipe.delete()  raised.
• Detached builds. Declining "Cancel builds?" or timing out intentionally keeps resources for  --recover. If the user never recovers, they stay forever. The repository name embeds a hash of the project, so any change to the project also orphans the old repository.

I'd suggest to check the craft-remote-build project each time a user triggers a remote build to clean past stale git repositories. We can make it best-effort and have limits to the number of repositories that are cleaned each time.

I've checked #249 that suggest user manual cleanup while this is fully automatic.

That would fix the cause and the backlog while only adding some lightweight best-effort API calls that can make the remote build fail.

To Reproduce

The easiest way to reproduce is the Detached builds leak from the above list. You can just create a remote build and detach from it. If the user never runs --recover the Launchpad git repository will be there forever.

part yaml

Relevant log output

N/A