Pull Request Check List

• Added tests for changed code.
• Updated documentation for changed code.

With #5509, this is only relevant for indexes that do not support PEP 658 - thus, probably for most indexes except for PyPI - or old wheels without PEP 658 metadata.

This PR adds a config option solver.lazy-wheel. If active and a server supports HTTP range requests, we do not have to download entire wheels to fetch metadata but can just download the METADATA file from the remote wheels with a couple of range requests. Especially with slow network connections this setting can speed up dependency resolution significantly. (If the cache has already been filled or the server does not support HTTP range requests, this setting makes no difference.)

This PR is based on pypa/pip#12208, which I learned about at PackagingCon, so all credits to the authors of this pip PR. 👏 I just had to make the interface fit for Poetry, write some tests and fix some special cases I encountered.

I measured the time of poetry lock of a largish project, once from a machine with a slow network connection (1.5 MB/s, 40 ms ping) and once from a machine with a fast network connection (95 MB/s, 1 ms ping) to the package index.

What lazy-wheel basically does is:

• try a range request with a negative offset in order to fetch the central directory of the wheel zip file
• if negative range requests are not supported, send a HEAD request to get the size of the wheel and afterwards send a normal range request to fetch the last bytes of the wheel
• if the central directory is quite large we may need an additional range request to fetch the rest of the central directory
• look for the position of the METADATA file in the central directory and download it in one range request

In order to not try range requests again and again in vain, we keep track if a domain supports range requests at all and especially with negative offsets.

In contrast to the pip PR, we only fetch the METADATA file instead of the entire .dist-info directory. On the machine with a slow network connection this made a difference of 190 s to 330 s.

Further, I added some handling for special cases I encountered while trying range requests with different servers:

¹ "Devpi server 1" hosts several indexes including a PyPI mirror. I noticed that range requests were supported for wheels that have been downloaded before, but were not supported for wheels that have not been downloaded before. In other words, a server might supports range requests for some but not all wheels. (This is handled in HTTPRepository.)

The behavior of the different servers can be interpreted as follows:

• The internal GitLab, piwheels.org and download.pytorch.org behave as you would expect from a server that supports range requests with negative offsets.
• PyPI behaves as you would expect from a server that supports range requests but not negative offsets.
• The internal devpi server 2 behaves as you would expect from a server that does not support range requests.
• The internal devpi server 1 has a quirk when the negative offset is greater than the entire wheel. It does return 200 OK instead of 206 Partial content, so you have to inspect the Accept-Ranges header to determine if it supports range requests or not.
• The internal Artifactory has the most unexpected behavior. It does not support negative offsets but interprets it as positive. E.g. -100 is handled as 0-100.

In case you were wondering at the beginning why we should introduce a config option and not just always use range requests if possible, it's probably clearer now: Although various servers were tested and special cases were implemented, it's not unlikely that there are still servers with unexpected behavior that is not handled well.