Follow-up from: #4415

The WCOW Snapshotter's Commit step needs to restream the entire layer's data out and back, unlike every other Snapshotter for which Commit is mostly a local metadata update.

This is problematic both because it's done inline with other operations, rather than concurrently, and because it's holding a boltdb transaction lock.

Time taken will be proportional to layer size (not image size, thankfully), and this doesn't apply to imported layers, which are already in the correct on-disk format after Apply completes.

It's possible this could be made more efficient: The tar-streaming exposed by hcsshim's ociwclayer package is actually wrapping some other lower-level public APIs, which potentially could be used directly. However, it will still likely be proportional to the size of the image.

So finding a way to parallelise/background this work, or at least do it without the BoltDB lock held, would be a great improvement.

We also lack a good way to clean up the partially-complete restreaming if the Commit is aborted, without hard-coding some low-level details of the on-disk layer format. Again, no other Snapshotter currently has this problem, as they are all able to synchronously update the metadata, except devicemapper, which blocks on a background kernel IO flush which has probably already completed by the time of Commit.

I would suggest not starting to address this until #4419 lands, or we otherwise gain test coverage of the WCOW snapshotter.

See thread starting from post by @fuweid in #4415 (comment)