Skip to content

Add projects' interactive documentation #19

Open

Description

@trallard

📝 Summary

Expand the availability of interactive documentation examples and tutorials to at least five projects.

🚀 Tasks / Deliverables

See the "Status" section below

📅 Estimated completion

24 months milestone

📋 Additional information

Status

Interactive documentation websites for projects: insights

This table is best complemented by the one in #18 which contains further linked PRs about out-of-tree Pyodide/WASM CI builds (first column here) and around PRs linked towards the availability of nightly wheels (third column here). The nightly wheels for projects have been hosted on the Anaconda.org PyPI index at https://anaconda.org/scientific-python-nightly-wheels. For the lack of CORS headers around the installation of nightly wheels, please refer to pyodide/pyodide#4898. For discussions predating this choice where thoughts pertaining to an ideal location for nightly wheels were being exchanged, please refer to pyodide/pyodide#3049.

Package name 🧱 Builds for WASM (either out-of-tree CI builds, in-tree within the Pyodide distribution, or both) 🚀 Interactive documentation deployments 🌃 Use of nightly WASM wheels in deployments
NumPy ✅ Both 🚀 Docstring-based API examples (where applicable) merged at numpy/numpy#26745 and available on the "latest" documentation at https://numpy.org/devdocs/reference/. They will be available in the stable release with NumPy 2.3.0: https://numpy.org/devdocs/release/2.3.0-notes.html#interactive-examples-in-the-numpy-documentation; NumPy 2.3 maintenance branch cut hasn't happened yet. The "User Guide" section is being converted and awaits design decisions. The NumPy tutorials website is not ready yet. Nightly WASM wheels are available but have not yet been utilised.
PyWavelets ✅ Both Examples under "API Reference" were enabled for interactivity via PyWavelets/pywt#728. PyWavelets/pywt#741 aims to convert the "Usage examples" section with long-form notebook-based content. There are some discussions around switching to GitHub Pages (to be finalised) at PyWavelets/pywt#777. Planned, and nightly wheels available but not used yet
pandas ✅ Both In progress. pandas-dev/pandas#60758 was merged, and is being followed with pandas-dev/pandas#61061 Planned, nightly wheels available but not used yet
awkward and awkward-cpp CI builds for WASM in out-of-tree are enabled with a docs-based workflow, though not fully tested via the test suite. In-tree builds available with Pyodide releases An interactive shell is available at https://awkward-array.org/doc/main/_static/try-it.html which preloads Awkward Array for importing – does not use jupyterlite-sphinx yet Extra wheels left in with Pyodide distribution using JupyterLite. Nightly wheels are not hosted on Anaconda.org or used
scikit-learn ✅ Both ✅ Yes, deployed using Sphinx-Gallery's JupyterLite integration already ✅ scikit-learn/scikit-learn#31085
scikit-image ✅ Both scikit-image/scikit-image#7644 attempts to add interactive examples Awaiting a review in scikit-image/scikit-image#7440
statsmodels ✅ Both Uses sphinx-immaterial – styling differences exist with PST and aesthetic changes will be required for the buttons. In progress at statsmodels/statsmodels#9536 Planned, nightly wheels available but not used yet
Zarr WASM builds in progress. Updated versions of all dependencies are now available in Pyodide in-tree with Pyodide 0.27 Planned Planned
numcodecs WASM builds completed via zarr-developers/numcodecs#529, awaiting a review ❌ Not planned, since numcodecs is not particularly user-facing and is not a major dependency for Zarr's codecs ❌ Not planned
SciPy In-tree builds up-to-date in Pyodide 0.27 alpha releases. Out-of-tree CI job not implemented yet because of magnitude of FORTRAN 77 patches. Possibility to use flang via LLVM 19 via pyodide/pyodide-recipes#238 to cross-compile F77/F90 code to WASM directly instead of f2c, needs experimentation Docstring-based API examples deployed scipy/scipy#20019, later disabled and need to be re-enabled – tracked in scipy/scipy#22241 and being looked into. The scipy.stats notebooks were deployed for a start in in scipy/scipy#20303 and scipy/scipy#21042 in a similar fashion to PyWavelets's above – via Jupytext for documentation build-time conversion from reST to MyST Markdown notebooks Planned
SymPy ✅ Both Documentation is built with Furo theme instead of PST, some aesthetic changes to jupyterlite-sphinx's style applied via sympy/sympy#27419, awaiting merge. Additional context: interactive doctest-based examples used to exist with a server-side Google App Engine deployment and a Sphinx extension, but have been long deprecated. Nightly wheels are available, and are planned for use in SymPy's live shell first before proceeding to SymPy tutorials. Update as of 03/01/2025: via sympy/live#22, https://live.sympy.org is now at the latest available stable version at the time of writing. Update as of 21/01/2025: nightly wheels will be used post merge of SymPy docs interactivity PR noted above.
Matplotlib Out-of-tree builds tracked in matplotlib/matplotlib#27870, being tracked in matplotlib/matplotlib#29093. Builds available in-tree up to version 3.8.4 via pyodide/pyodide#4510, and patched WebAgg planned for further use (see pyodide/pyodide#5398 and matplotlib/matplotlib#29568) In progress at matplotlib/matplotlib#29506 which adds a shell and enables the gallery, Sphinx-Gallery's JupyterLite integration planned next Planned for addition in matplotlib/matplotlib#29093
h5py and libhdf5 h5py/h5py#2397 Planned, decision needed Planned
PyTables Planned, decision needed Planned, decision needed Planned

Miscellaneous items of note

Installation of nightly wheels for projects' "dev"/"latest" docs websites

Communication and adoption

See pyodide/pyodide#17 for interactive docs SPEC – first draft by @agriyakhetarpal ready as of 04/03/2025 and undergoing internal review(s)

Notebook styling improvements for docs

See pyodide/pyodide#142 for the major round-up around design decisions for notebook-based docs.

Activity

  1. added this to the 24 months milestone on Dec 16, 2022
  2. rgommers commented on Jan 10, 2024

    @rgommers
    Member

    Year 1 goal of the first interactive docs for 1 project (SciPy) almost met, see pyodide/pyodide#16.

    @Carreau is now joining in to work on this larger year 2 goal.

  3. moved this to 🏗 In progress in czi-scientific-python 🐍on Jan 10, 2024
  4. rgommers commented on May 22, 2024

    @rgommers
    Member

    As a quick update:

  5. rgommers commented on May 22, 2024

    @rgommers
    Member

    Also related: a fair amount of work is going into making binary wheels smaller (e.g. the machinery to strip off tests and data with a single build flag), reducing the number of build variants in compiled code, etc. Small binaries will be quite important to make the user experience good, because downloading many MBs of packages yields high latency and can be costly when a user doesn't have a connection with unlimited bandwidth.

  6. rgommers commented on Nov 15, 2024

    @rgommers
    Member

    @agriyakhetarpal it may be useful to have a Status table in the issue description here (just like in pyodide/pyodide#18), linking to deployed interactive docs and/or open PRs. Would you be able to create one?

  7. agriyakhetarpal commented on Nov 15, 2024

    @agriyakhetarpal
    Member

    Yes, I'll do this in a moment, @rgommers. I expect that my interactive docs work will increase in magnitude in the coming months.

  8. Carreau commented on Jan 31, 2025

    @Carreau
    Contributor

    With respect to installing nightly wheels when running jupyter-lite:

    • My own opinion is that one should install package when starting the kernel, cf https://github.com/jupyterlite/pyodide-kernel/pull/158 (backticks for not crosslinking). I see installing those packages not as arbitrary code execution, as IMHO jupyter lite goal is to have a environment ready for end user to install.
    • There are some concern about load on Pypi and other indexes (despite CDN), and breaking pyodide itself.
    • As Agriya also pointed out there is --piplite-wheel, which allows embedding wheel into jupyter-lite, it raises the questions of dependencies (usually this is used for building a PR, but deps are pulled from indexes). My understanding is that even if we had --piplite-wheel-and-all-deps, they would still not allow install at kernel startup.
      • There is a current technical limitation that piplite/micropip cannot install only from the --piplite-wheel, but that's just a technical limitation.

    Alternatives to a patch in pyodide-kernel are:

    • A fork.
    • An Jupyterlite Extension
      • A full extension – not sure there are the right hooks ?
      • A shallow extension pyodide-kernel-autoinstall a shallow extension that just overrwite the startup method to install packages – but not sure how hard.
    • Auto inject a await piplite.install([..packages..], index_urls=['proxy?']) (potentially from a "file") to avoid a big cell.
    • Patch in IPython if in emscripten to install things.

    See also https://github.com/jupyterlite/jupyterlite/issues/508, https://github.com/jupyterlite/pyodide-kernel/issues/60

  9. juntyr commented on Feb 26, 2025

    @juntyr

    I've been working on a similar problem and am now using the following solution (not quite for micropip installs since I need locked environments):

  10. rootsmusic commented on Apr 29, 2025

    @rootsmusic

    @trallard Should Scientific Python Lectures be added to your status table?

  11. agriyakhetarpal commented on Apr 29, 2025

    @agriyakhetarpal
    Member

    Hi @rootsmusic, I cannot answer for @trallard here of course, but I would believe that porting the Scientific Python Lectures to be interactive would be considered low-priority for now. Some of my thoughts are as follows:

    • Those materials were set up with another Sphinx theme in mind, and swaying away from the PyData Sphinx Theme (which is one of the nicer ones in the avenue in terms of design, features, and accessibility) means that we need to create new styles for the jupyterlite-sphinx components to comply – unless they can switch themes. I don't immediately see a proposal to do so in their issue tracker.
    • It's relatively easy to copy the notebooks and embed them in a JupyterLite gallery that's separate from the documentation. At the moment, you may even build/host your own site to do this with the notebooks downloaded from the repository. From a cursory look, I'm not entirely sure about how many notebooks will work seamlessly within a WASM environment and whether they have been updated to work with latest versions of Scientific Python libraries, but it could be worth a shot!
    • Lastly, it would be nice if there were progress on Start conversion to markdown scipy-lectures/scientific-python-lectures#781. It would help avoid redundant efforts around making them work well with interactive documentation and styling changes being discussed as a part of MAINT: Rename python2js_int to _python2js pyodide/pyodide#142. See also: Plan for Scientific Python lecture notes? scientific-python/summit-2025#5, where it would be nice to have consensus on the improvements to be made to these notes and their scope before putting Pyodide/JupyterLite forward as a pedagogical driver.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Type

No type

Projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions