Skip to content

GH-55586: [Packaging][Python] Use pyodide venv for Emscripten/Pyodide testing infrastructure and unskip several tests - #55588

Open
agriyakhetarpal wants to merge 6 commits into
apache:mainfrom
agriyakhetarpal:feat/pyodide-venv-tests
Open

GH-55586: [Packaging][Python] Use pyodide venv for Emscripten/Pyodide testing infrastructure and unskip several tests#55588
agriyakhetarpal wants to merge 6 commits into
apache:mainfrom
agriyakhetarpal:feat/pyodide-venv-tests

Conversation

@agriyakhetarpal

@agriyakhetarpal agriyakhetarpal commented Nov 13, 2026 •
edited by github-actions Bot
Loading

Copy link
Copy Markdown
Contributor

Rationale for this change

Closes #55586.

This PR removes the NodeDriver part of the Emscripten/Pyodide testing infrastructure to replace it with a native replacement that we offer, called pyodide venv. Here is our documentation: https://pyodide-build.readthedocs.io/en/stable/how-to/testing.html

Since the Pyodide Python interpreter included in this virtual environment can access the file system, the timezone database directory can now be read, which enables the tests for the ORC reader.

What changes are included in this PR?

This PR includes the following changes:

  • Removes the NodeDriver testing class and runtime=node option from scripts/run_emscripten_tests.py, and replaces them with a pyodide venv invocation, where testing dependencies (either pure Python or WASM wheels) can be pip-installed, akin to a conventional Python virtual environment. The documentation has been updated with a short section to reflect this.
  • Additional dependencies have been enabled for testing: cffi, cloudpickle, fsspec, and scipy, which allow for 380 or so extra tests on Node, and 346 more in Chrome, to be run and pass. The remaining skips are due to features not built for the Emscripten platform, such as S3, Flight, and threading.
  • The ORC tests have been enabled, since /usr/share/zoneinfo can now be accessed.
    • Two tests out of those, test_timezone_database_absent and test_timezone_absent, rely on subprocesses and are skipped accordingly.
  • The Substrait expression serialisation test that was noted to crash Pyodide at some point no longer fails on my machine, so I've unskipped it. :)

Are these changes tested?

Yes, I have built a WASM build for Arrow, and a pyemscripten wheel for PyArrow, and tested the changes locally in a Pyodide virtual environment and with Chrome installed on my macOS machine/

Are there any user-facing changes?

No, there are no user-facing changes as part of this PR.

Was AI used for this PR?

In accordance to the AI generation guidelines, please disclose below whether and how AI was used in this PR.

PR code and description written by:

  • Human
  • AI

Reviewed before submission by:

  • Human
  • AI
  • Not reviewed

I have not used any AI tools for this PR.

Copilot AI balanced review requested due to automatic review settings November 13, 2026 12:49
@agriyakhetarpal agriyakhetarpal changed the title GH-55586: Use pyodide venv for Emscripten/Pyodide testing infrastructure and unskip several tests GH-55586: [Packaging][Python] Use pyodide venv for Emscripten/Pyodide testing infrastructure and unskip several tests Nov 13, 2026
@github-actions

github-actions Bot commented Nov 13, 2026

Copy link
Copy Markdown

⚠️ GitHub issue #55586 has been automatically assigned in GitHub to PR creator.

@github-actions

github-actions Bot commented Nov 13, 2026

Copy link
Copy Markdown

⚠️ GitHub issue #55586 has no components, please add labels for components.

Copilot AI reviewed Nov 13, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues require further changes.

No open issues 🎉

What changed in this PR

Replaces the custom Node.js test driver with pyodide venv, simplifying PyArrow’s Emscripten testing infrastructure and expanding test coverage.

Changes:

  • Runs Node.js tests in a Pyodide virtual environment with additional testing dependencies.
  • Expands browser test dependencies, restores Substrait test execution, and marks subprocess-dependent ORC tests.
  • Documents the virtual-environment workflow and browser testing option.
File Description
python/​scripts/​run_emscripten_tests.py Removes the Node driver, adds browser dependencies, and removes the Substrait exclusion.
python/​pyarrow/​tests/​test_orc.py Marks two timezone tests as requiring subprocess support.
docs/​source/​developers/​cpp/​emscripten.rst Documents testing wheels with pyodide venv and browsers.
ci/​scripts/​python_test_emscripten.sh Runs Node.js tests through pyodide venv while retaining Chrome testing.

Review effort: Balanced

@agriyakhetarpal

agriyakhetarpal commented Nov 13, 2026

Copy link
Copy Markdown
Contributor Author

Hi @tadeja! I hope it's alright for me to ping you here, continuing on from #47330. :) Could you be so kind as to submit a crossbow job for Emscripten?

@tadeja

tadeja commented Nov 13, 2026

Copy link
Copy Markdown
Member

@github-actions crossbow submit emscripten

@github-actions

github-actions Bot commented Nov 13, 2026

Copy link
Copy Markdown

Revision: 470540e

Submitted crossbow builds: ursacomputing/crossbow @ actions-8bf8f13948

Task Status
test-conda-python-emscripten GitHub Actions
test-ubuntu-22.04-cpp-emscripten GitHub Actions

@agriyakhetarpal

agriyakhetarpal commented Nov 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

For two, yet another follow-up work/PRs after this:

  1. I was exploring PyArrow in WASM a bit more, having not used it as much, and found that we can actually easily enable the bz2 (ARROW_WITH_BZ2) feature and enable some more testing here – and we can adopt the same for our PyArrow recipe in https://github.com/pyodide/pyodide-recipes. It will add less than 100 KiB to our wheel, which is 11.3 MiB, so the change is small.

  2. Another issue I found is that the pyodide-unix-timezones package we ship as a PyArrow dependency to provide zoneinfo data doesn't work at all, because it's a stub and the wheel is empty. What a surprise :D No one raised this either, as far as I can tell. When I tried this with assume_timezone(arr, "Europe/Paris") in our REPL online, it failed with discover_tz_dir failed to find zoneinfo.

    Initially, in added unix timezone package pyodide/pyodide#4889, the fat timezones were being added, and later in the PR, the package was moved to PyPI but the data was not added there, there's only an __init__.py file 😄 After that, post Add datafiles package type pyodide/pyodide#4892, we have supported data files, so we can probably add the fat timezones in our pyodide-unix-timezones recipe.

    That package only exists because of these two issues: [C++][Python] Date and time errors with slim zoneinfo files #42157 and GH-49351: [C++] Check TZDIR environment variable in vendored date library #49353. If those get implemented, or if we could use std::chrono on Emscripten too1 (beyond Windows as noted in GH-49351: [C++] Check TZDIR environment variable in vendored date library #49353 (comment)), then PyArrow on Pyodide can simply rely on slim files and use tzdata, which is already a dependency because of pandas.

Footnotes

  1. That would also need Emscripten's libc++ to add a timezone database, of course, but I haven't investigated what it would take to make that happen. ↩

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@AlenkaF AlenkaF Awaiting requested review from AlenkaF AlenkaF is a code owner
@raulcd raulcd Awaiting requested review from raulcd raulcd is a code owner
@rok rok Awaiting requested review from rok rok is a code owner
@assignUser assignUser Awaiting requested review from assignUser assignUser is a code owner
@jonkeane jonkeane Awaiting requested review from jonkeane jonkeane is a code owner
@kou kou Awaiting requested review from kou kou is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Use pyodide venv for testing PyArrow pyemscripten builds

3 participants

@agriyakhetarpal @tadeja