Skip to content

Parallelise layer downloads #9922

@azr

Description

@azr

What is the problem you're trying to solve

If I'm being correct, currently, each layer is downloaded in one goroutine within newHTTPReadSeeker.
I work with quite big AI related images (drivers, models and deps are big), and sometimes, a layer can be really big, it is not rare at all to see ~10 GB layers.

I think it could benefit this use case to allow to parallelise the download of a single layer.

Describe the solution you'd like

A way to configure the parallelisation of a layer would be nice to have.

I know that the amazon-ecr-containerd-resolver uses parallelisation with htcat: https://github.com/awslabs/amazon-ecr-containerd-resolver/blob/dc8a230a63b583bda4e0601a8945b4fdf9166084/ecr/fetcher.go#L125

If that setting is set, htcat tries to do range requests on big enough files and if that works, downloads things in parallel.

Additional context

happy to work on this.

Related: #9138

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions