Add pyodide resources to Iodide build by mdboom · Pull Request #1122 · iodide-project/iodide

mdboom · 2018-11-02T15:06:40Z

@wlach: I have no idea if this is the right approach.

pyodide-demo (which should probably be renamed to pyodide-deploy at some point) is repo where the pyodide built resources get uploaded to by CI.

This works with the local make up server, and I can load a language plugin as follows:

{
  "languageId": "py",
  "displayName": "Python",
  "codeMirrorMode": "python",
  "keybinding": "p",
  "url": "/pyodide/pyodide.js",
  "module": "pyodide",
  "evaluator": "runPython",
  "asyncEvaluator": "runPythonAsync",
  "pluginType": "language"
}

EDIT:

To add some context for this change. @rth's work to get Scipy working includes a Scipy package that is larger than 100MB, exceeding the maximum file size limit for Github pages. I'd like to merge that and start making that publicly available, even though the long term plan is to do some "hard things" to get that size down.

By deploying to Heroku we can get around this limit. We don't have to include it as part of Iodide's deployment (it could be its own static Heroku project, or use some other hosting), but it seems like it saves a little effort for now to do it that way. This also gives you a local copy of pyodide as part of npm run build, which could be handy for doing local/offline demos, but again not critical.

mdboom · 2018-11-02T16:05:20Z

As discussed elsewhere, this is going to use Github releases rather than a git checkout.

rth · 2018-11-05T11:29:01Z

To get the dev version, for files larger than 100 MB (e.g. scipy), that we can't store in a github repo, another possibility could be to try to download the Circle CI artifacts that are stored for each successful build on master. Internally CircleCI stores them in S3 I think, but there is no documented way of accessing those.

There is a related discussion in https://discuss.circleci.com/t/referencing-latest-build-artifacts/931/24 and circlecipy now exposes a way to list latest artifacts e.g. on master branch (though this would fail if latest build failed on master).

mdboom · 2018-11-07T21:55:55Z

@rth: Now that CircleCI is deploying the artifacts on tags to Github releases, I think we can use that for now. Plus, it's nice for other reasons to have those in Github releases. And I think pinning this to a release (and getting in the habit of tagging releases) is probably a better approach than pulling from master all the time.

@wlach: Would you mind having a second look at this?

wlach

This is looking much better! I have some suggestions on how to make this a bit easier to handle from an administration perspective.

wlach · 2018-11-07T22:10:12Z

bin/post_compile

 npm install && npm run build

+# Pull in the Pyodide static resources from a Github release
+export PYODIDE_VERSION=0.1.10


I wonder, should we default to getting the latest release on github? A little more involved, but would save us from having to constantly update this file whenever a new pyodide release goes out.

https://developer.github.com/v3/repos/releases/

We should probably also let the admin select a specific release by specifying a PYODIDE_VERSION environment variable.

I actually think we should be explicit about the version here, or at least pin to a major semver or something. There have been problems with mismatching iodide and pyodide in the past (like when changes to the local variables pane broke things, or the recent asyncEvaluator changes). Ideally, we should probably do some sort of integration test...

Ok, in that case let's just use the heroku-specified environment variable, and error out if it's not set?

I was actually thinking that the version of Pyodide should be tied to the version of Iodide (not as in the same number, but that as features are changed in one or the other, we'll need to make sure the version of Pyodide installed in Iodide is compatible). So that actually leads me to think I'm going about this wrong and maybe this should be done as part of webpack. The advantage there is that things would work for local development as they do on Heroku. Does that make sense?

I think it does! Having pyodide easily available for the local dev use case would be a huge win.

wlach · 2018-11-07T22:11:22Z

bin/post_compile

 # since WhiteNoise's Django storage backend only gzips assets handled by
 # collectstatic, and so does not affect files in the `dist/` directory.
-python -m whitenoise.compress prod
+python -m whitenoise.compress build


I am curious, have we verified that this works recently? :) I imagine compressing the pyodide files correctly/incorrectly would have a large impact on performance. I can take a look at this if you like.

I suspect this hasn't been working since the change from prod to build as output of npm run build. Should help with Iodide itself.

The Pyodide data files are LZ4 compressed so that they can be partially block-decompressed in the client. Gzip should probably be able to compress them further, but not a ton. None of the .js files are compressed, though, just minimized. We should confirm all of this works and helps. If not, maybe there's a way to tell whitenoise to just compress build/* but not build/pyodide/*.

Looks like the potential space savings are quite considerable with gzip, even moreso with brotli compression (supported by basically all major browsers at this point):

(venv)Hammersmith:pyodide wlach$ python -m whitenoise.compress . Brotli compressed ./py.data (336K -> 200K) Gzip compressed ./py.data (336K -> 226K) Brotli compressed ./kiwisolver.js (6K -> 2K) Gzip compressed ./kiwisolver.js (6K -> 2K) Brotli compressed ./setuptools.data (1742K -> 790K) Gzip compressed ./setuptools.data (1742K -> 1205K) Brotli compressed ./python.html (6K -> 2K) Gzip compressed ./python.html (6K -> 2K) Brotli compressed ./pluggy.data (51K -> 29K) Gzip compressed ./pluggy.data (51K -> 31K) Brotli compressed ./pyodide.asm.js (3035K -> 283K) Gzip compressed ./pyodide.asm.js (3035K -> 372K) Brotli compressed ./atomicwrites.data (15K -> 7K) Gzip compressed ./atomicwrites.data (15K -> 8K) Brotli compressed ./pytz.data (653K -> 215K) Gzip compressed ./pytz.data (653K -> 330K) Brotli compressed ./pyparsing.data (268K -> 151K) Gzip compressed ./pyparsing.data (268K -> 176K) Brotli compressed ./pandas.data (21820K -> 10135K) Gzip compressed ./pandas.data (21820K -> 13061K) Brotli compressed ./test.data (8067K -> 4424K) Gzip compressed ./test.data (8067K -> 5223K) Brotli compressed ./test.js (229K -> 54K) Gzip compressed ./test.js (229K -> 64K) Brotli compressed ./numpy.js (206K -> 50K) Gzip compressed ./numpy.js (206K -> 59K) Brotli compressed ./matplotlib.html (6K -> 2K) Gzip compressed ./matplotlib.html (6K -> 2K) Brotli compressed ./more-itertools.data (239K -> 134K) Gzip compressed ./more-itertools.data (239K -> 156K) Brotli compressed ./nose.data (585K -> 328K) Gzip compressed ./nose.data (585K -> 390K) Brotli compressed ./numpy.data (9671K -> 4866K) Gzip compressed ./numpy.data (9671K -> 6248K) Brotli compressed ./more-itertools.js (9K -> 2K) Gzip compressed ./more-itertools.js (9K -> 3K) Brotli compressed ./setuptools.js (46K -> 9K) Gzip compressed ./setuptools.js (46K -> 12K) Brotli compressed ./pyodide.asm.wasm (10018K -> 1978K) Gzip compressed ./pyodide.asm.wasm (10018K -> 2747K) Brotli compressed ./pandas.js (509K -> 123K) Gzip compressed ./pandas.js (509K -> 141K) Brotli compressed ./pluggy.js (7K -> 1K) Gzip compressed ./pluggy.js (7K -> 2K) Brotli compressed ./atomicwrites.js (5K -> 1K) Gzip compressed ./atomicwrites.js (5K -> 1K) Brotli compressed ./attrs.data (104K -> 59K) Gzip compressed ./attrs.data (104K -> 68K) Brotli compressed ./matplotlib.js (253K -> 48K) Gzip compressed ./matplotlib.js (253K -> 73K) Brotli compressed ./kiwisolver.data (77K -> 43K) Gzip compressed ./kiwisolver.data (77K -> 48K) Brotli compressed ./matplotlib-sideload.html (2K -> 0K) Gzip compressed ./matplotlib-sideload.html (2K -> 0K) Brotli compressed ./renderedhtml.css (3K -> 0K) Gzip compressed ./renderedhtml.css (3K -> 0K) Brotli compressed ./attrs.js (8K -> 2K) Gzip compressed ./attrs.js (8K -> 2K) Brotli compressed ./nose.js (22K -> 4K) Gzip compressed ./nose.js (22K -> 6K) Brotli compressed ./test.html (0K -> 0K) Gzip compressed ./test.html (0K -> 0K) Brotli compressed ./joblib.data (480K -> 272K) Gzip compressed ./joblib.data (480K -> 317K) Brotli compressed ./pyodide_dev.js (11K -> 2K) Gzip compressed ./pyodide_dev.js (11K -> 3K) Brotli compressed ./pytest.data (791K -> 459K) Gzip compressed ./pytest.data (791K -> 531K) Brotli compressed ./pyparsing.js (8K -> 2K) Gzip compressed ./pyparsing.js (8K -> 3K) Brotli compressed ./pyodide.js (11K -> 2K) Gzip compressed ./pyodide.js (11K -> 3K) Brotli compressed ./packages.json (1K -> 0K) Gzip compressed ./packages.json (1K -> 0K) Brotli compressed ./py.js (17K -> 3K) Gzip compressed ./py.js (17K -> 4K) Brotli compressed ./python-dateutil.js (14K -> 3K) Gzip compressed ./python-dateutil.js (14K -> 4K) Brotli compressed ./pytest.js (25K -> 5K) Gzip compressed ./pytest.js (25K -> 7K) Brotli compressed ./pytz.js (75K -> 10K) Gzip compressed ./pytz.js (75K -> 13K) Brotli compressed ./pyodide.asm.data.js (136K -> 35K) Gzip compressed ./pyodide.asm.data.js (136K -> 42K) Brotli compressed ./python_dev.html (1K -> 0K) Gzip compressed ./python_dev.html (1K -> 0K) Brotli compressed ./pyodide.asm.data (6579K -> 3631K) Gzip compressed ./pyodide.asm.data (6579K -> 4446K) Brotli compressed ./python-dateutil.data (409K -> 286K) Gzip compressed ./python-dateutil.data (409K -> 317K) Brotli compressed ./cycler.js (5K -> 1K) Gzip compressed ./cycler.js (5K -> 2K) Brotli compressed ./cycler.data (24K -> 12K) Gzip compressed ./cycler.data (24K -> 13K) Brotli compressed ./matplotlib.data (15500K -> 8808K) Gzip compressed ./matplotlib.data (15500K -> 11356K) Brotli compressed ./joblib.js (27K -> 4K) Gzip compressed ./joblib.js (27K -> 6K)

Brotli compressed ./numpy.data (9671K -> 4866K)
Gzip compressed ./numpy.data (9671K -> 6248K)

Wow, Brotli compression seems to work really well! Particularly considering that it's already lz4 compressed pyodide/pyodide#235

mdboom · 2018-11-12T19:26:28Z

@wlach: I've reworked this to install Pyodide into build/pyodide by webpack. This means we'll get it for local testing as well as when deploying to Heroku. Would you mind having another look?

bcolloran · 2018-11-12T19:43:17Z

i don't know what WebpackShellPlugin does -- does including that pyodide reference in the webpack make pyodide a dependency for iodide? If so, perhaps we should avoid that situation by putting that bit behind an env var?

mdboom · 2018-11-12T19:51:20Z

WebpackShellPlugin just allows running shell commands.

This doesn't make pyodide a dependency of iodide -- the whole point of doing it in this unusual way is to just stick the files alongside, not to compile them together.

It might not be a bad idea to put this behind an env var just so you could proceed with npm run build without an internet connection, though.

wlach

It seems like this won't autoupgrade pyodide if we change versions, since the directory is already there. Any chance you could add some smarts to make that happen only if necessary? Other than that this is looking good.

bin/install_pyodide

bin/post_compile

Fix unit tests broken by #1122

Add pyodide resources to Heroku static files

3fe6b7d

mdboom requested a review from wlach November 2, 2018 15:06

mdboom mentioned this pull request Nov 2, 2018

USe github releases for deployment pyodide/pyodide#239

Merged

wlach suggested changes Nov 7, 2018

View reviewed changes

rth mentioned this pull request Nov 12, 2018

Port scikit-image pyodide/pyodide#253

Closed

Install pyodide into Webpack

512b177

mdboom force-pushed the add-pyodide-resources branch from 9385b93 to 512b177 Compare November 12, 2018 19:24

mdboom changed the title ~~Add pyodide resources to Heroku static files~~ Add pyodide resources to Iodide build Nov 12, 2018

wlach suggested changes Nov 12, 2018

View reviewed changes

bin/install_pyodide Outdated Show resolved Hide resolved

bin/post_compile Outdated Show resolved Hide resolved

mdboom added 3 commits November 12, 2018 17:37

Remove Heroku-specific version

28f9d4b

Set variable only if it doesn't exist

d05450e

Upgrade pyodide if necessary

9f266a1

wlach approved these changes Nov 14, 2018

View reviewed changes

mdboom merged commit 862611f into iodide-project:master Nov 14, 2018

mdboom added a commit to mdboom/iodide that referenced this pull request Nov 14, 2018

Fix unit tests broken by iodide-project#1122

6f51ec4

mdboom added a commit that referenced this pull request Nov 14, 2018

Merge pull request #1173 from mdboom/fix-unit-tests

aab8950

Fix unit tests broken by #1122

mdboom mentioned this pull request Nov 14, 2018

Make deploying Pyodide alongside Iodide work #1178

Merged

rth mentioned this pull request Nov 15, 2018

Scipy packaging pyodide/pyodide#211

Merged

wlach pushed a commit to wlach/iodide that referenced this pull request Nov 16, 2018

Fix unit tests broken by iodide-project#1122

1eac676

Comments

Conversation

mdboom commented Nov 2, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mdboom commented Nov 2, 2018

Uh oh!

rth commented Nov 5, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mdboom commented Nov 7, 2018

Uh oh!

wlach left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rth Nov 8, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mdboom commented Nov 12, 2018

Uh oh!

bcolloran commented Nov 12, 2018

Uh oh!

mdboom commented Nov 12, 2018

Uh oh!

wlach left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mdboom commented Nov 2, 2018 •

edited

Loading

rth commented Nov 5, 2018 •

edited

Loading

rth Nov 8, 2018 •

edited

Loading