What is the problem you're trying to solve

The current ContainerD fetches layers in parallel, but unpacks them in a single thread layer by layer sequentially.

Describe the solution you'd like

Proposal:

We propose a config below in ContainerD (/etc/containerd/config.toml) to support unpacking images in parallel.

[plugins."io.containerd.grpc.v1.cri".containerd.overlayfs]
  unpacking_mode = "parallel"

Key Changes: ContainerD Handlers + Content

This option reuses the existing FetchHandler and Content to support pre-decompression during the fetch phase of each layer. The actual decompression happens directly following the fetch process as the graph below. Then the unpack or snapshot handling becomes a light operation by just renaming the pre-decompression folder to the desired overlay path.

• Fetch

  • When ContainerD config (/etc/containerd/config.toml) unpacking_mode = "parallel" is set, replace FetchHandler in pull.go with FetchUnzipHandler created below.
  • Create a new FetchUnzipHandler similar to FetchHandler, which enables the unzip option in Fetch.
  • Add unzip option to Fetch function; Update Content to support pre-decompress buckets as part of its content.Store APIs for content lifecycle management, i.e., A fetched gzip file and corresponding pre-decompressed fs folder will have the identical lifecycle, for example, create and delete.

• Unpack and Snapshot processing

  • Update apply function to support an option to apply with the pre-decompression folder.
  • Update s.store.ReaderAt implementation, the content store knows if there is a pre-decompress bucket for a layer, if yes, it will return a list of file paths as byte[]. Alternatively, maybe we will add a new method s.store.FolderAt to Provider.

Additional context

Who should enable this feature?

• Those who use disks with high in-parallel IO support. For example, PD or LocalSSD is designed for a deeper IO queue to get better throughput by in-parallel IO operations.
• Those who use large containers and are sensitive to slow pod cold start. For example, containers with GPU libraries and frameworks (> 4GB). All GPU workloads can be put into this category. In contrast, previously top containers without using GPU were significantly smaller (< 500MB).

Who should NOT enable this feature?

HDD with high seek time for random read and write.

Potential Benefit

If user can improve disk performance, the image pull latency by this proposal can be reduced significantly. For example, Tao was able to achieve 3X faster image pull (120 seconds -> 40 seconds) for pulling a popular container: gcr.io/deeplearning-platform-release/base-cu113:m106 (5.4 GB) with a common setup of a deep learning node (2500 GB PD-SSD, 32 vCPUs).