get sdist requires from egg-info#88
Conversation
|
Tests found at least one case where this errors out. I'll look into it. |
|
Should I delete the files afterward or does it not matter as code meant for the CI? (which I assume just cleans itself up) |
| for info in file_in: | ||
| folder_name = info.name | ||
| # Assume a single folder | ||
| break |
There was a problem hiding this comment.
I would add a check here, whether folder_name was actually set, or maybe even check that there really is exactly one folder to account for broken eggs. But for the former case, the following should be simpler and have that required error check:
| for info in file_in: | |
| folder_name = info.name | |
| # Assume a single folder | |
| break | |
| # Assume a single folder | |
| folder_name = next(iter(file_in)).name |
You can then also drop the initialization of folder_name.
Or just use * instead of folder_name in the glob below.
There was a problem hiding this comment.
Or just use * instead of folder_name in the glob below.
New code looks much cleaner like this, and makes no assumption. Especially since it's extracting to a temp folder now.
| ) | ||
|
|
||
|
|
||
| def get_sdist_requires( |
There was a problem hiding this comment.
Nit: "get" makes this function sound simpler than what it actually does. Maybe "extract"?
Also I think it would be safer to extract the sdist into an empty temporary directory than to where ever we currently are.
use glob to not have to assume match count
|
At least some of the test failures will be fixed by #95 |
|
I was thinking integration tests might be too fragile, but I can just pin the version I'm testing against. I'm sure you'd like me to add tests for this? Quick testing script I've been using: from __future__ import annotations
import requests
from packaging.requirements import Requirement
from stub_uploader.metadata import extract_sdist_requires, validate_response
for req_name in ("pycocotools", "click-default-group", "numpy", "mypy", "pywin32"): # Pin these
req = Requirement(req_name)
resp = requests.get(f"https://pypi.org/pypi/{req.name}/json")
validate_response(resp, req)
data = resp.json()
sdist_data = next(
(
url_data
for url_data in reversed(data["urls"])
if url_data["packagetype"] == "sdist"
),
None,
)
if sdist_data is None:
print(f"{req_name}: sdist_data is None!")
continue
requirements = extract_sdist_requires(sdist_data, req)
print(f"{req_name}: {list(requirements)}")Results: pycocotools: [<Requirement('matplotlib>=2.1.0')>, <Requirement('numpy')>]
click-default-group: [<Requirement('click')>]
numpy: []
mypy: [<Requirement('typing_extensions>=3.10')>, <Requirement('mypy_extensions>=1.0.0')>, <Requirement('tomli>=1.1.0')>, <Requirement('typed_ast<2,>=1.4.0')>, <Requirement('psutil>=4.0')>, <Requirement('pip')>, <Requirement('typed_ast<2,>=1.4.0')>, <Requirement('lxml')>]
pywin32: sdist_data is None! |
3 participants
CC @AlexWaygood This was the idea I mentioned.
When the requirements can't be found in
requires_dist, fallback to searching in *.egg-info. All without executing any arbitrary code.I move the
f"Expected dependency {req} to be present in the allowlist"error higher so it can fail earlier and leave the network requests last.It's my first time playing around with downloading and manipulating archives in python. Let me know what I can do better here.
I tested the idea with
pycocotools(best case scenario)click-default-group(on my windows machine with 7zip there's an extra folder between the gz and tar, still worked),numpy(no *.egg-info folder),mypy(has empty lines and [extras]) andpywin32(no Source Distribution)