Update to python 3.9.1

Python 3.9 is out. It would be good to upgrade. (In particular I have my eye on several convenient extensions to the C API.)
https://docs.python.org/release/3.9.1/whatsnew/3.9.html#porting-to-python-3-9

It would be good to update, but we need to be sure that all packages we use made a Python 3.9 compatible release. They should be mostly compatible, but it would still be good to avoid issues such as #877 . Also Cython is not forward compatible for packages that include the Cythonized sources in .tar.gz

Can you make another checklist for this? Probably Python 3.9 is still too newly released. It was released in October, so maybe give six months for packages to add support for it and try to update in April?

Something we can look into for 0.18.0. The constraint is not so much that packages won't support 3.9 or 3.10 (they usually would mostly without changes) but that for packages released some time ago there might not be binary wheels on PyPi for those python versions.
Which is important if you want to share objects as pickles between native python and Pyodide. So we should probably make an automated summary of packages that have wheels for the the python version we ship.

It would be good to add a list of instructions for migrating between versions of Python to the development docs when we do this, in interest of explicitly documenting maintenance procedures.

Yes, maybe as a subsection of development/updating-packages.html where we should also move https://pyodide.org/en/latest/development/contributing.html#migrating-patches

FYI, here's a quick n'dirty snippet to see what the current status is, using pipx & Python 3.10:

from pathlib import Path
import subprocess

packages = [str(p.parent) for p in Path(".").glob("*/meta.yaml")]


def check_pkg(package: str) -> dict[str, bool]:
    res = subprocess.run(
        ["pipx", "run", "pypi-command-line", "wheels", package],
        capture_output=True,
        text=True,
    ).stdout
    return {
        "found": "Package Type:" in res,
        "universal": "none-any.whl" in res,
        "py39": "cp39" in res,
        "py310": "cp310" in res,
    }

ans = {package: check_pkg(package) for package in packages}

for name, pkg in ans.items():
    match pkg:
        case {"found": False}:
            print(name, "not found")
for name, pkg in ans.items():
    match pkg:
        case {"found": True, "py39": True, "py310": False}:
            print(name, "only has Python 3.9 wheels")
for name, pkg in ans.items():
    match pkg:
        case {"found": True, "py39": True, "py310": True}:
            print(name, "is ready!")

libiconv not found
pyb2d not found
libxml not found
libxslt not found
cffi_example not found
fpcast-test not found
libyaml not found
sharedlib-test not found
zlib not found
CLAPACK not found
micropip not found
sharedlib-test-py not found
nlopt only has Python 3.9 wheels
pyodide-interrupts only has Python 3.9 wheels
numcodecs only has Python 3.9 wheels
msgpack is ready!
python-sat is ready!
MarkupSafe is ready!
pyrsistent is ready!
traits is ready!
pyerfa is ready!
iminuit is ready!
cffi is ready!
biopython is ready!
sqlalchemy is ready!
regex is ready!
wrapt is ready!
numpy is ready!
yt is ready!
scikit-image is ready!
pillow is ready!
kiwisolver is ready!
matplotlib is ready!
pyyaml is ready!
pywavelets is ready!
pydantic is ready!
statsmodels is ready!
astropy is ready!
lxml is ready!
scipy is ready!
scikit-learn is ready!
pandas is ready!
swiglpk is ready!

~~More fun~~, uh, I mean faster:

from pathlib import Path
import asyncio
import shutil

pipx = shutil.which("pipx")
packages = [str(p.parent) for p in Path(".").glob("*/meta.yaml")]


async def check_pkg(package: str) -> dict[str, bool]:
    proc = await asyncio.subprocess.create_subprocess_exec(
            pipx, "run", "pypi-command-line", "wheels", package,
            stdout=asyncio.subprocess.PIPE
    )
    stdout = await proc.stdout.read()
    res = stdout.decode()

    return {
        "found": "Package Type:" in res,
        "universal": "none-any.whl" in res,
        "py39": "cp39" in res,
        "py310": "cp310" in res,
    }

async def run_all(packages: list[str]) -> list[str]:
    res = await asyncio.gather(*(check_pkg(package) for package in packages))
    return res

packages_out = asyncio.run(run_all(packages))

ans = dict(sorted(zip(packages, packages_out)))

for name, pkg in ans.items():
    match pkg:
        case {"found": False}:
            print(name, "not found")
for name, pkg in ans.items():
    match pkg:
        case {"found": True, "py39": True, "py310": False}:
            print(name, "only has Python 3.9 wheels")
for name, pkg in ans.items():
    match pkg:
        case {"found": True, "py39": True, "py310": True}:
            print(name, "is ready!")

Still irritated by the boilerplate required to use asyncio in a normal file, vs. pyodide or iPython shells, or python -m asyncio.

@henryiii

Thanks for the investigation, @henryiii! It looks like most of the packages in Pyodide supports Python 3.10. I think we could start a discussion of uprading our Python version before long.

pyodide-interrupts we can just remove, we have a better version of that functionality with pyodide.setInterruptBuffer.

Ideally we should open a new issue about updating to Python 3.10 rather than discussing it here though...

Opened #2190 about updating to Py3.10.

Navigation Menu

Repository navigation

Update to python 3.9.1 #978

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Update to python 3.9.1 #978

Description

Activity

rth commented on Dec 31, 2020

hoodmane commented on Dec 31, 2020

rth commented on Apr 18, 2021

hoodmane commented on May 12, 2021

rth commented on May 12, 2021

henryiii commented on Feb 17, 2022

henryiii commented on Feb 17, 2022

ryanking13 commented on Feb 18, 2022

hoodmane commented on Feb 18, 2022

hoodmane commented on Feb 18, 2022

rth commented on Feb 19, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions