Lazy wheel broken in 1.8.0 and 1.8.1 when there are redirects

### Description

I'm trying to package up a nice repro, but the gist of it is in a custom simple mirror, we use this convention:

```
/simple/foo/ includes a url like /packages/<n>/foo-0.1-py3-none-any.whl
/packages/<n>/foo-0.1-py3-none-any.whl gives a 307 to a pythonhosted url
```

It fails with the following formatted traceback (the wheel url doesn't matter; I had to clear caches a couple of times during testing to be sure it would trigger)

```
% time poetry lock --no-update
Updating dependencies
Resolving dependencies... (28.2s)
Resolving dependencies... (28.3s)
Resolving dependencies... (28.4s)
Resolving dependencies... (37.9s)
Resolving dependencies... (38.0s)
Resolving dependencies... (38.1s)
Resolving dependencies... (52.7s)

  ValueError

  Package('opentelemetry-instrumentation-celery', '0.25b0') is not in list

  at <redacted>/lib/python3.10/site-packages/poetry/repositories/legacy_repository.py:66 in package
       62│         Note that this will be cached so the subsequent operations
       63│         should be much faster.
       64│         """
       65│         try:
    →  66│             index = self._packages.index(Package(name, version))
       67│ 
       68│             return self._packages[index]
       69│         except ValueError:
       70│             package = super().package(name, version, extras)

The following error occurred when trying to handle this error:


  HTTPError

  501 Server Error: Unsupported client range for url: https://files.pythonhosted.org/packages/67/1b/8920801a4361dd069cc7edf9106ceb308fa9b8adc703f3fcd516195f4a4e/opentelemetry_instrumentation_celery-0.25b0-py3-none-any.whl#sha256=1d9b88e8aa861d1fe0b4bd1fb3b9e5032e7a5b65c5419fcb6977b778e52cf621

  at <redacted>/lib/python3.10/site-packages/requests/models.py:1021 in raise_for_status
      1017│                 f"{self.status_code} Server Error: {reason} for url: {self.url}"
      1018│             )
      1019│ 
      1020│         if http_error_msg:
    → 1021│             raise HTTPError(http_error_msg, response=self)
      1022│ 
      1023│     def close(self):
      1024│         """Releases the connection back to the pool. Once this method has been
      1025│         called the underlying ``raw`` object must not be accessed again.

The following error occurred when trying to handle this error:


  AssertionError

  

  at <redacted>/lib/python3.10/site-packages/poetry/inspection/lazy_wheel.py:388 in _content_length_from_head
      384│                                              for "bytes" ranges."""
      385│         self._request_count += 1
      386│         head = self._session.head(self._url, headers=self._uncached_headers())
      387│         head.raise_for_status()
    → 388│         assert head.status_code == codes.ok
      389│         accepted_range = head.headers.get("Accept-Ranges", None)
      390│         if accepted_range != "bytes":
      391│             raise HTTPRangeRequestUnsupported(
      392│                 f"server does not support byte ranges: header was '{accepted_range}'"
poetry lock --no-update  7.50s user 0.94s system 15% cpu 54.780 total
```


### Workarounds

```Markdown
Downgrade to 1.7.1 if you use redirects in your simple mirror
```


### Poetry Installation Method

pip

### Operating System

OS X (does not seem to be OS-dependent)

### Poetry Version

Poetry (version 1.8.1)

### Poetry Configuration

```bash session
cache-dir = "/Users/timhatch/Library/Caches/pypoetry"
experimental.system-git-client = false
installer.max-workers = null
installer.modern-installation = true
installer.no-binary = null
installer.parallel = true
repositories.<redacted>.url = "https://<redact>/simple/"
virtualenvs.create = true
virtualenvs.in-project = null
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.no-setuptools = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = "{cache-dir}/virtualenvs"  # /Users/timhatch/Library/Caches/pypoetry/virtualenvs
virtualenvs.prefer-active-python = false
virtualenvs.prompt = "{project_name}-py{python_version}"
warnings.export = true
```


### Python Sysconfig

```bash session
I don't think this is relevant
```


### Example pyproject.toml

```TOML
[[tool.poetry.source]]
name = "<redact>"
url = "https://<redacted>/simple/"
priority = "primary"

[[tool.poetry.source]]
name = "PyPI"
priority = "supplemental"

...plus some normal deps...
```


### Poetry Runtime Logs

```bash session
[urllib3:urllib3.connectionpool] https://<redacted>:443 "GET /packages/8515241215/responses-0.25.0-py3-none-any.whl HTTP/1.1" 307 0
[urllib3:urllib3.connectionpool] Starting new HTTPS connection (1): files.pythonhosted.org:443
[urllib3:urllib3.connectionpool] https://files.pythonhosted.org:443 "GET /packages/30/0b/bff1e6a5b646e6ff770deb6a292a96bd844ea13fb523ccbd9209fc4b90b8/responses-0.25.0-py3-none-any.whl HTTP/1.1" 501 463
Retrying HTTP request in 0.0 seconds.
[urllib3:urllib3.connectionpool] https://<redacted>:443 "GET /packages/8515241215/responses-0.25.0-py3-none-any.whl HTTP/1.1" 307 0
[urllib3:urllib3.connectionpool] Starting new HTTPS connection (2): files.pythonhosted.org:443
[urllib3:urllib3.connectionpool] https://files.pythonhosted.org:443 "GET /packages/30/0b/bff1e6a5b646e6ff770deb6a292a96bd844ea13fb523ccbd9209fc4b90b8/responses-0.25.0-py3-none-any.whl HTTP/1.1" 501 463
Retrying HTTP request in 0.0 seconds.
[urllib3:urllib3.connectionpool] https://<redacted>:443 "GET /packages/8515241215/responses-0.25.0-py3-none-any.whl HTTP/1.1" 307 0
[urllib3:urllib3.connectionpool] Starting new HTTPS connection (3): files.pythonhosted.org:443
[urllib3:urllib3.connectionpool] https://files.pythonhosted.org:443 "GET /packages/30/0b/bff1e6a5b646e6ff770deb6a292a96bd844ea13fb523ccbd9209fc4b90b8/responses-0.25.0-py3-none-any.whl HTTP/1.1" 501 465
Retrying HTTP request in 0.0 seconds.
[urllib3:urllib3.connectionpool] https://<redacted>:443 "GET /packages/8515241215/responses-0.25.0-py3-none-any.whl HTTP/1.1" 307 0
[urllib3:urllib3.connectionpool] Starting new HTTPS connection (4): files.pythonhosted.org:443
[urllib3:urllib3.connectionpool] https://files.pythonhosted.org:443 "GET /packages/30/0b/bff1e6a5b646e6ff770deb6a292a96bd844ea13fb523ccbd9209fc4b90b8/responses-0.25.0-py3-none-any.whl HTTP/1.1" 501 462
Retrying HTTP request in 0.0 seconds.
[urllib3:urllib3.connectionpool] https://<redacted>:443 "GET /packages/8515241215/responses-0.25.0-py3-none-any.whl HTTP/1.1" 307 0
[urllib3:urllib3.connectionpool] Starting new HTTPS connection (5): files.pythonhosted.org:443
[urllib3:urllib3.connectionpool] https://files.pythonhosted.org:443 "GET /packages/30/0b/bff1e6a5b646e6ff770deb6a292a96bd844ea13fb523ccbd9209fc4b90b8/responses-0.25.0-py3-none-any.whl HTTP/1.1" 501 465
Retrying HTTP request in 0.0 seconds.
[urllib3:urllib3.connectionpool] https://<redacted>:443 "GET /packages/8515241215/responses-0.25.0-py3-none-any.whl HTTP/1.1" 307 0
[urllib3:urllib3.connectionpool] Starting new HTTPS connection (6): files.pythonhosted.org:443
[urllib3:urllib3.connectionpool] https://files.pythonhosted.org:443 "GET /packages/30/0b/bff1e6a5b646e6ff770deb6a292a96bd844ea13fb523ccbd9209fc4b90b8/responses-0.25.0-py3-none-any.whl HTTP/1.1" 501 463
Negative byte range not supported for domain '<redacted>': using HEAD request before lazy wheel from now on (code: 501)
[urllib3:urllib3.connectionpool] https://<redacted>:443 "HEAD /packages/8515241215/responses-0.25.0-py3-none-any.whl HTTP/1.1" 307 0
   1: Version solving took 14.982 seconds.
   1: Tried 1 solutions.

  ValueError

  Package('responses', '0.25.0') is not in list

  at .venv/lib/python3.11/site-packages/poetry/repositories/legacy_repository.py:66 in package
       62│         Note that this will be cached so the subsequent operations
       63│         should be much faster.
       64│         """
       65│         try:
    →  66│             index = self._packages.index(Package(name, version))
       67│ 
       68│             return self._packages[index]
       69│         except ValueError:
       70│             package = super().package(name, version, extras)

The following error occurred when trying to handle this error:


  Stack trace:

  4  .venv/lib/python3.11/site-packages/poetry/inspection/lazy_wheel.py:609 in _extract_content_length
      607│         try:
      608│             # Initial range request for just the end of the file.
    → 609│             file_length, tail = self._try_initial_chunk_request(initial_chunk_size)
      610│         except HTTPError as e:
      611│             # Our initial request using a negative byte range was not supported.

  3  .venv/lib/python3.11/site-packages/poetry/inspection/lazy_wheel.py:572 in _try_initial_chunk_request
      570│ 
      571│         self._request_count += 1
    → 572│         tail = self._session.get(self._url, headers=headers, stream=True)
      573│         tail.raise_for_status()
      574│ 

  2  .venv/lib/python3.11/site-packages/poetry/utils/authenticator.py:267 in get
      265│ 
      266│     def get(self, url: str, **kwargs: Any) -> requests.Response:
    → 267│         return self.request("get", url, **kwargs)
      268│ 
      269│     def head(self, url: str, **kwargs: Any) -> requests.Response:

  1  .venv/lib/python3.11/site-packages/poetry/utils/authenticator.py:245 in request
      243│                 if resp.status_code not in STATUS_FORCELIST or is_last_attempt:
      244│                     if raise_for_status:
    → 245│                         resp.raise_for_status()
      246│                     return resp
      247│ 

  HTTPError

  501 Server Error: Unsupported client range for url: https://files.pythonhosted.org/packages/30/0b/bff1e6a5b646e6ff770deb6a292a96bd844ea13fb523ccbd9209fc4b90b8/responses-0.25.0-py3-none-any.whl#sha256=2f0b9c2b6437db4b528619a77e5d565e4ec2a9532162ac1a131a83529db7be1a

  at .venv/lib/python3.11/site-packages/requests/models.py:1021 in raise_for_status
      1017│                 f"{self.status_code} Server Error: {reason} for url: {self.url}"
      1018│             )
      1019│ 
      1020│         if http_error_msg:
    → 1021│             raise HTTPError(http_error_msg, response=self)
      1022│ 
      1023│     def close(self):
      1024│         """Releases the connection back to the pool. Once this method has been
      1025│         called the underlying ``raw`` object must not be accessed again.

The following error occurred when trying to handle this error:


  Stack trace:

  31  .venv/lib/python3.11/site-packages/cleo/application.py:327 in run
       325│ 
       326│             try:
     → 327│                 exit_code = self._run(io)
       328│             except BrokenPipeError:
       329│                 # If we are piped to another process, it may close early and send a

  30  .venv/lib/python3.11/site-packages/poetry/console/application.py:190 in _run
       188│         self._load_plugins(io)
       189│ 
     → 190│         exit_code: int = super()._run(io)
       191│         return exit_code
       192│ 

  29  .venv/lib/python3.11/site-packages/cleo/application.py:431 in _run
       429│             io.input.interactive(interactive)
       430│ 
     → 431│         exit_code = self._run_command(command, io)
       432│         self._running_command = None
       433│ 

  28  .venv/lib/python3.11/site-packages/cleo/application.py:473 in _run_command
       471│ 
       472│         if error is not None:
     → 473│             raise error
       474│ 
       475│         return terminate_event.exit_code

  27  .venv/lib/python3.11/site-packages/cleo/application.py:457 in _run_command
       455│ 
       456│             if command_event.command_should_run():
     → 457│                 exit_code = command.run(io)
       458│             else:
       459│                 exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED

  26  .venv/lib/python3.11/site-packages/cleo/commands/base_command.py:117 in run
       115│         io.input.validate()
       116│ 
     → 117│         return self.execute(io) or 0
       118│ 
       119│     def merge_application_definition(self, merge_args: bool = True) -> None:

  25  .venv/lib/python3.11/site-packages/cleo/commands/command.py:61 in execute
        59│ 
        60│         try:
     →  61│             return self.handle()
        62│         except KeyboardInterrupt:
        63│             return 1

  24  .venv/lib/python3.11/site-packages/poetry/console/commands/lock.py:55 in handle
        53│         self.installer.lock(update=not self.option("no-update"))
        54│ 
     →  55│         return self.installer.run()
        56│ 

  23  .venv/lib/python3.11/site-packages/poetry/installation/installer.py:104 in run
       102│             self.verbose(True)
       103│ 
     → 104│         return self._do_install()
       105│ 
       106│     def dry_run(self, dry_run: bool = True) -> Installer:

  22  .venv/lib/python3.11/site-packages/poetry/installation/installer.py:241 in _do_install
       239│                 source_root=self._env.path.joinpath("src")
       240│             ):
     → 241│                 ops = solver.solve(use_latest=self._whitelist).calculate_operations()
       242│         else:
       243│             self._io.write_line("Installing dependencies from lock file")

  21  .venv/lib/python3.11/site-packages/poetry/puzzle/solver.py:71 in solve
        69│         with self._progress(), self._provider.use_latest_for(use_latest or []):
        70│             start = time.time()
     →  71│             packages, depths = self._solve()
        72│             end = time.time()
        73│ 

  20  .venv/lib/python3.11/site-packages/poetry/puzzle/solver.py:154 in _solve
       152│ 
       153│         try:
     → 154│             result = resolve_version(self._package, self._provider)
       155│ 
       156│             packages = result.packages

  19  .venv/lib/python3.11/site-packages/poetry/mixology/__init__.py:18 in resolve_version
        16│     solver = VersionSolver(root, provider)
        17│ 
     →  18│     return solver.solve()
        19│ 

  18  .venv/lib/python3.11/site-packages/poetry/mixology/version_solver.py:175 in solve
       173│             while next is not None:
       174│                 self._propagate(next)
     → 175│                 next = self._choose_package_version()
       176│ 
       177│             return self._result()

  17  .venv/lib/python3.11/site-packages/poetry/mixology/version_solver.py:514 in _choose_package_version
       512│             package = locked
       513│ 
     → 514│         package = self._provider.complete_package(package)
       515│ 
       516│         conflict = False

  16  .venv/lib/python3.11/site-packages/poetry/puzzle/provider.py:489 in complete_package
       487│                 dependency_package = DependencyPackage(
       488│                     dependency,
     → 489│                     self._pool.package(
       490│                         package.pretty_name,
       491│                         package.version,

  15  .venv/lib/python3.11/site-packages/poetry/repositories/repository_pool.py:204 in package
       202│         for repo in self.repositories:
       203│             try:
     → 204│                 return repo.package(name, version, extras=extras)
       205│             except PackageNotFound:
       206│                 continue

  14  .venv/lib/python3.11/site-packages/poetry/repositories/legacy_repository.py:70 in package
        68│             return self._packages[index]
        69│         except ValueError:
     →  70│             package = super().package(name, version, extras)
        71│             package._source_type = "legacy"
        72│             package._source_url = self._url

  13  .venv/lib/python3.11/site-packages/poetry/repositories/cached_repository.py:75 in package
        73│         extras: list[str] | None = None,
        74│     ) -> Package:
     →  75│         return self.get_release_info(canonicalize_name(name), version).to_package(
        76│             name=name, extras=extras
        77│         )

  12  .venv/lib/python3.11/site-packages/poetry/repositories/cached_repository.py:52 in get_release_info
        50│             return PackageInfo.load(self._get_release_info(name, version))
        51│ 
     →  52│         cached = self._release_cache.remember(
        53│             f"{name}:{version}", lambda: self._get_release_info(name, version)
        54│         )

  11  .venv/lib/python3.11/site-packages/poetry/utils/cache.py:147 in remember
       145│         value = self.get(key)
       146│         if value is None:
     → 147│             value = callback() if callable(callback) else callback
       148│             self.put(key, value, minutes)
       149│         return value

  10  .venv/lib/python3.11/site-packages/poetry/repositories/cached_repository.py:53 in <lambda>
        51│ 
        52│         cached = self._release_cache.remember(
     →  53│             f"{name}:{version}", lambda: self._get_release_info(name, version)
        54│         )
        55│ 

   9  .venv/lib/python3.11/site-packages/poetry/repositories/legacy_repository.py:123 in _get_release_info
       121│         yanked = page.yanked(name, version)
       122│ 
     → 123│         return self._links_to_data(
       124│             links,
       125│             PackageInfo(

   8  .venv/lib/python3.11/site-packages/poetry/repositories/http_repository.py:360 in _links_to_data
       358│ 
       359│         # drop yanked files unless the entire release is yanked
     → 360│         info = self._get_info_from_links(links, ignore_yanked=not data.yanked)
       361│ 
       362│         data.summary = info.summary

   7  .venv/lib/python3.11/site-packages/poetry/repositories/http_repository.py:310 in _get_info_from_links
       308│                 return self._get_info_from_metadata(
       309│                     universal_python3_wheel
     → 310│                 ) or self._get_info_from_wheel(universal_python3_wheel)
       311│ 
       312│             if universal_python2_wheel:

   6  .venv/lib/python3.11/site-packages/poetry/repositories/http_repository.py:123 in _get_info_from_wheel
       121│             try:
       122│                 package_info = PackageInfo.from_metadata(
     → 123│                     metadata_from_wheel_url(link.filename, link.url, self.session)
       124│                 )
       125│             except HTTPRangeRequestUnsupported:

   5  .venv/lib/python3.11/site-packages/poetry/inspection/lazy_wheel.py:77 in metadata_from_wheel_url
        75│         # After context manager exit, wheel.name will point to a deleted file path.
        76│         # Add `delete_backing_file=False` to disable this for debugging.
     →  77│         with LazyWheelOverHTTP(url, session) as lazy_file:
        78│             metadata_bytes = lazy_file.read_metadata(name)
        79│ 

   4  .venv/lib/python3.11/site-packages/poetry/inspection/lazy_wheel.py:289 in __enter__
       287│     def __enter__(self: U) -> U:
       288│         super().__enter__()
     → 289│         self._setup_content()
       290│         return self
       291│ 

   3  .venv/lib/python3.11/site-packages/poetry/inspection/lazy_wheel.py:357 in _setup_content
       355│         if self._length is None:
       356│             logger.debug("begin fetching content length")
     → 357│             self._length = self._fetch_content_length()
       358│             logger.debug("done fetching content length (is: %d)", self._length)
       359│             # Enable us to seek and write anywhere in the backing file up to this

   2  .venv/lib/python3.11/site-packages/poetry/inspection/lazy_wheel.py:508 in _fetch_content_length
       506│         """
       507│         initial_chunk_size = self._initial_chunk_length()
     → 508│         ret_length, tail = self._extract_content_length(initial_chunk_size)
       509│ 
       510│         # Need to explicitly truncate here in order to perform the write and seek

   1  .venv/lib/python3.11/site-packages/poetry/inspection/lazy_wheel.py:638 in _extract_content_length
       636│             self._domains_without_negative_range.add(domain)
       637│             # Apply a HEAD request to get the real size, and nothing else for now.
     → 638│             return self._content_length_from_head(), None
       639│ 
       640│         # Some servers that do not support negative offsets,

  AssertionError

  

  at .venv/lib/python3.11/site-packages/poetry/inspection/lazy_wheel.py:388 in _content_length_from_head
      384│                                              for "bytes" ranges."""
      385│         self._request_count += 1
      386│         head = self._session.head(self._url, headers=self._uncached_headers())
      387│         head.raise_for_status()
    → 388│         assert head.status_code == codes.ok
      389│         accepted_range = head.headers.get("Accept-Ranges", None)
      390│         if accepted_range != "bytes":
      391│             raise HTTPRangeRequestUnsupported(
      392│                 f"server does not support byte ranges: header was '{accepted_range}'"
```


Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Lazy wheel broken in 1.8.0 and 1.8.1 when there are redirects #9039

Description

Workarounds

Poetry Installation Method

Operating System

Poetry Version

Poetry Configuration

Python Sysconfig

Example pyproject.toml

Poetry Runtime Logs

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Lazy wheel broken in 1.8.0 and 1.8.1 when there are redirects #9039

Description

Description

Workarounds

Poetry Installation Method

Operating System

Poetry Version

Poetry Configuration

Python Sysconfig

Example pyproject.toml

Poetry Runtime Logs

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions