There seems to be a bug where I cannot correctly build the "API reference" section of the documentation on a local build – I have just the skimage index and not the rest of the top-level modules; they aren't being generated fully. The sphinx-gallery examples work fine, though. I shall debug with the CircleCI previews for now...

I noticed another bug with sphinx-gallery's notebook paths – I will submit a patch there.

I'm sorry, @lesteve – I should have checked that you've had an open PR to do this via #6974 before I started working on this initiative further. How would you want me to proceed? I am happy to either pull in the relevant commits from that branch I haven't implemented here and to grant you authorship as appropriate, or step back and let you finish that one. Now that scikit-image's WASM wheels are being built via #7350 (though not being tested temporarily because of #7576), my further plans for this PR were/are to work on #6958 to complete the full deployment (#6972 (comment)).

@agriyakhetarpal no worries and thanks for all the work you are doing on the Pyodide/interactive documentation front!

Feel free to do whatever is simpler for you! I am more than happy for #6974 to be closed and you taking over this work.

There seems to be a bug where I cannot correctly build the "API reference" section of the documentation on a local build – I have just the skimage index and not the rest of the top-level modules; they aren't being generated fully.

@agriyakhetarpal, I've seen that locally when I installed skimage in editable mode with spin install. I don't think your PR or related to JupyterLite. Sorry for the confusion on your part. I'll see if I can come up with a quick fix. It's very probably something with how our doc/tools/apigen.py not dealing well with the editable hook.

I proposed a fix in #7647 to make apigen.py work with editable hooks.

@lesteve, I was able to move the CORS proxy handling to a skimage.data.patch_cors() function, continuing from #6911. This makes it have a much nicer interface, as we can replace the initialisation code that was exposed to the users via the notebooks previously, like the following

import re
import skimage.data._registry
new_registry_urls = {
    k: re.sub(
        r'https://gitlab.com/(.+)/-/raw(.+)',
        r'https://cdn.statically.io/gl/\1\2',
        url
    )
    for k, url in skimage.data._registry.registry_urls.items()
}
skimage.data._registry.registry_urls = new_registry_urls
    """.splitlines()
        )

with smaller code, as

import skimage.data
skimage.data.patch_cors()

It seems to work quite well. You can try it out here: "Segment human cells (in mitosis)" example — skimage 0.25.3rc0.dev0 documentation

Of course, we can't realistically test all example notebooks, as we discussed. However, I don't think there's a reason why this shouldn't work for any other cases besides this one. The PR is ready for another look!

@agriyakhetarpal Let me know if you want me to take a look at this. I wasn't sure of its current status.

@agriyakhetarpal Let me know if you want me to take a look at this. I wasn't sure of its current status.

@stefanv, thanks! It's ready for review.

Hi @stefanv, I hope it is okay to ping you again here – this PR stands ready for review. @lesteve recently noted in my contemporary PRs to scikit-learn at scikit-learn/scikit-learn#29791 and scikit-learn/scikit-learn#31078 that the approach I was using here wasn't the best one as it would end up bloating the documentation repository at https://github.com/scikit-image/docs/tree/gh-pages with binary WASM wheels, which I think we should avoid.

I'm now applying mostly the same code as scikit-learn/scikit-learn#31085, which requires #7440 to be merged so that the nightly WASM wheels for scikit-image can show up on https://anaconda.org/scientific-python-nightly-wheels (i.e., when you're able to get to this PR, please review that PR first). The recent commits I've pushed today are mostly in accordance with this change.

In particular, this PR also applies a patch to scikit-image's registry to use the https://statically.io/ CDN as a CORS proxy for data files that are hosted on GitLab. This means that examples requiring such functionality in ski.data won't work, but will after this PR makes it to the nightly wheels. Please let me know if you'd like me to split up the changes I've made to skimage/data/_fetchers.py to a new PR that should be reviewed/merged before this PR (I probably should).

Thank you for your time!

I'm also happy to hop on to another one of scikit-image's community calls to interact with you and the rest of the scikit-image core developers, similar to our previous discussions on #7470 (I'm waiting to get this change addressed before I proceed to work on that!).

I only had a quick scan, but it's not 100% clear to me how we match the pip install version to the docs version in the non-dev gallery. Otherwise, I see no major hurdles.

Thanks for the review, @stefanv!

This PR was addressing this point and the one you made in #7644 (comment) up to the time when I was bundling the wheel along with the docs, which I removed in later commits as it would increase the size of the https://github.com/scikit-image/docs repository. At this moment, this PR shouldn't close #6958 as it adds a nightly installation of scikit-image for the dev docs when it gets merged; it doesn't sync up the versions for older docs.

I'd say that it is acceptable to add binary files to a repository like that and work with only a shallow clone if we need to modify the repository manually, but there's also a security angle towards adding them, which I cannot ignore. We'll need to figure out a solution for this. I have a few suggestions at hand:

• We enable interactive docs only for the stable and the dev versions (partial solution + on a best-effort basis, and the current solution in scikit-learn via DOC Use nightly WASM wheels for JupyterLite in the dev documentation scikit-learn/scikit-learn#31085 that was just merged)
  • The dev docs will use the nightly wheels (through this PR and Use cibuildwheel to build WASM/Pyodide wheels for scikit-image, push nightlies to Anaconda.org #7440)
  • The stable docs use as up-to-date a version of scikit-image as possible that's available in Pyodide (already the case)
• Or, we use the Xeus kernel and its corresponding emscripten-forge instead of the Pyodide kernel. I could work on getting an updated version of scikit-image in emscripten-forge beyond 0.19 (I've noted this in the PR description). However, the Xeus kernel doesn't allow %pip-installs for packages in the browser yet, so I'll need to change the approach a bit (https://jupyterlite.readthedocs.io/en/latest/howto/configure/kernels.html#adding-a-python-kernel).
• Yet another way is to use the https://jupyterlite-pyodide-lock.readthedocs.io/ extension, but I don't know how stable it is as it's only on v0.1.2 so far. It does seem to work well, though.

I'm also curious if Loïc would have any suggestions, as this problem is one of the unsolved ones we have. :)

My feeling right now for scikit-learn: the JupyterLite/Pyodide setup is experimental. We had zero reports about people being confused because the scikit-learn version on JupyterLite did not match what they expected 😅, probably because not that many people click on the JupyterLite link and/or they don't know where to complain. In a similar fashion, the Binder scikit-learn had been broken for at least 2 years and there had been zero report about it 😅.

Personally, I sum this up as "it is completely fine to be less conservative than usual, let's try to get more people to use the JupyterLite setup, and let's improve it as we go". Of course I fully understand if other projects decide to be more conservative than scikit-learn.

What I envision medium-term for scikit-learn (a bit hacky but 🤷) is to upload Pyodide wheels to anaconda.org scientific-python-nightly-wheel for scikit-learn released versions, so that await micropip.install(f'scikit-learn=={version}', index_urls='https://pypi.anaconda.org/scientific-python-nightly-wheels/simple') will work as well for scikit-learn released versions. This would mean that the 1.6 doc uses scikit-learn 1.6.1 in its JupyterLite and the dev doc uses scikit-learn dev in its JupyterLite.

I think building the Pyodide wheel during the build and shipping it within JupyterLite (see doc) is a fine option. Amongst other things, this should make it easier to keep the same version in the JupyterLite environment and the example gallery. For scikit-learn, I was a bit worried about the impact on the scikit-learn.github.io repo and on complicating the doc build slightly but I may change my mind about this in the future, we will see ...

I haven't tried emscripten-forge recently but my understanding is that you do the equivalent of importing all the installed packages at kernel start-up time, which in my opinion doesn't lead to a super nice user experience ¹. Having said this, I completely get it that opinions may differ about the trade-offs involved.

This is the first time I hear about pyodide-jupyter-lock so I am not going to comment on it 😅 ...

@lesteve, @lagru, and I are here in person at the Scientific Python summit. Our goal is to have this PR merged within the next few days. On my radar:

• update to the newer version of the Pyodide kernel, and bring the latest Pyodide; and
• sync up on whether we add just nightly wheels to this PR, or if we bundle the wheel within the GH Pages branch on https://github.com/scikit-image/docs/tree/gh-pages; and
• re-enable Pyodide wheel pushes to the Anaconda index.

I'm marking this as a draft until #7931 is merged and a nightly Pyodide wheel is rereleased for usage.