I've got a machine with thousands of layers on it (and it's actually at a low right now because I recently nuked /var/lib/docker; in my experience with enterprise clients this volume of images isn't really unusual, especially if they've been doing a lot of docker builds, etc) that I finally upgraded to latest master today, and the migration process took a full hour and 23 minutes. Even a really dramatic increase in speed is not going to be enough to make the amount of downtime palpable, so we need to come up with other options.

$ docker info
Containers: 4
Images: 3842

What I'd personally love to see (if it's plausible) is to have some kind of async migration queue, and have images get pushed to the top of the queue as they're necessary/requested, but I don't know whether that's even feasible.

At the bare minimum, we'll need to communicate this issue very widely with the 1.10 release, and hopefully have some kind of online migration tool that can be run before restarting. I think we'll also need to include some way to go the other direction, if we can, to ensure users can downgrade if they run into other related or unrelated issues with the 1.10 release.

cc @tonistiigi @vbatts @LK4D4