Skip to content

chore(python): Test free-threaded in ci - #21914

Draft
bschoenmaeckers wants to merge 6 commits into
pola-rs:mainfrom
bschoenmaeckers:test-free-threaded
Draft

bschoenmaeckers wants to merge 6 commits into
pola-rs:mainfrom
bschoenmaeckers:test-free-threaded

Conversation

@bschoenmaeckers

Copy link
Copy Markdown
Contributor

No description provided.

@github-actions github-actions Bot added internal An internal refactor or improvement python Related to Python Polars labels Mar 24, 2025
@bschoenmaeckers
bschoenmaeckers force-pushed the test-free-threaded branch 11 times, most recently from ee5fd23 to 02e489e Compare March 24, 2025 22:08
@bschoenmaeckers
bschoenmaeckers force-pushed the test-free-threaded branch 7 times, most recently from 0af294d to 25f50aa Compare April 5, 2025 10:32
@bschoenmaeckers
bschoenmaeckers force-pushed the test-free-threaded branch 2 times, most recently from 82eaada to ea0c7b0 Compare April 28, 2025 08:07
@bschoenmaeckers
bschoenmaeckers force-pushed the test-free-threaded branch 5 times, most recently from 71c2245 to b80d35d Compare June 19, 2025 10:30
@bschoenmaeckers
bschoenmaeckers force-pushed the test-free-threaded branch 3 times, most recently from 4c73b32 to 2ce2608 Compare September 17, 2025 13:54
@bschoenmaeckers

bschoenmaeckers commented Sep 17, 2025 •

Copy link
Copy Markdown
Contributor Author

I ran the CI again but ran into issues again. Some can be compiled from source, others have have compiler errors.

This is a non-exhaustive list of packages without precompiled 3.14(t) wheels.

  • torch
  • pyiceberg-core (needs pyo3 update)
  • pyroaring
  • pandas
  • pyarrow
  • connectorx (no sdist)
  • kuzu
  • fastexcel
  • deltalake
  • zstandard
  • gevent
  • markupsafe
  • pyyaml
  • grpcio
  • pydantic-core (beta release available)

I would like for most of these packages to have a wheel available so the CI does not have to compile everything and of course to avoid compilation errors. Which also makes debugging the this easier. I will try again when the 3.14 release is near, and most of the packages are pre-compiled.

@paultiq

paultiq commented Nov 2, 2025 •

Copy link
Copy Markdown

FYI:

Updates:

  • kuzu: "This repository was archived by the owner on Oct 10, 2025. It is now read-only.".
  • pyarrow 22.0 has 3.14t wheels now
  • markupsafe: has 3.14t wheels now
  • pyyaml: has 3.14t wheels now
  • pydantic-core: has 3.14t wheels now
  • (edit: 2.9.0 released) torch 2.9.0 has 3.14t wheels

Pandas:

  • Pandas 2.3.3 shipped a GIL-enabled 3.14t wheel, which isn't what you'd want. For free threading testing, you want the nightly 3.0 build:
    uv pip install --pre --upgrade --extra-index-url https://pypi.anaconda.org/scientific-python-nightly-wheels/simple pandas

@lukapeschke

Copy link
Copy Markdown
Contributor

FYI I've started working on a free-threaded version for fastexcel. Linked issue: ToucanToco/fastexcel#416

@harshil21

harshil21 commented Dec 8, 2025 •

Copy link
Copy Markdown

For those who want to use free threading, and don't want to compile polars from source, I've compiled 2 wheels myself for version 1.36.0. This is all you need to do in your pyproject.toml:

[project]
dependencies = [
    "polars==1.36.0",
    "polars-runtime-32"
]

[tool.uv.sources]
polars-runtime-32 = [
{ url = "https://github.com/harshil21/polars-runtime-32-ft/raw/refs/heads/main/wheels/polars_runtime_32-1.36.0-cp314-cp314t-manylinux_2_39_aarch64.whl", marker = "platform_machine == 'aarch64'" },
{ url = "https://github.com/harshil21/polars-runtime-32-ft/raw/refs/heads/main/wheels/polars_runtime_32-1.36.0-cp314-cp314t-manylinux_2_39_x86_64.whl", marker = "platform_machine == 'x86_64'" }
]

You can find the whole repository here

@Rogdham

Rogdham commented Jan 16, 2026

Copy link
Copy Markdown

3.14 also ships with zstandard bindings in the standard library so that might also be a path forward for whatever library is bringing python-zstandard into the dependency tree.

python-zstandard seems to only be used for tests. I opened #26144 to remove it.

@bschoenmaeckers

bschoenmaeckers commented Jun 23, 2026 •

Copy link
Copy Markdown
Contributor Author

I've checked again what the current status is of the test dependencies. It still looks like where are blocked by the following packages.

@ngoldbaum

Copy link
Copy Markdown

For zstandard, is there a reason why the standard lib zstd module can't be used instead?

Thanks for summarizing!

@bschoenmaeckers

bschoenmaeckers commented Jun 23, 2026 •

Copy link
Copy Markdown
Contributor Author

For zstandard, is there a reason why the standard lib zstd module can't be used instead?

Thanks for summarizing!

This has been proposed in the past: #21914 (comment) #26144

@ngoldbaum

Copy link
Copy Markdown

OK, I don't see any objections in that PR to adding it as a fallback if zstandard isn't available. Maybe we could try that instead. That said, the lack of an environment marker for free-threading might be annoying to deal with.

I've tried to get free-threaded wheels built for zstandard for a while but it's hard to get maintainer attention there.

Kumar already did a pass on connector-x and we're waiting for maintainer feedback there.

I think the next project to look at is deltalake.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

internal An internal refactor or improvement python Related to Python Polars

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants