Skip to content

Support partial builds#645

Merged
phorward merged 9 commits intopyodide:masterfrom
rth:partial-build
May 12, 2020
Merged

Support partial builds#645
phorward merged 9 commits intopyodide:masterfrom
rth:partial-build

Conversation

@rth
Copy link
Copy Markdown
Member

@rth rth commented May 8, 2020 •
edited
Loading

Following discussion in #553 this adds the support of partial builds. From the added documentation,

To build a subset of available packages in pyodide, set the environment
variable PYODIDE_PACKAGES to a comma separated list of packages. For
instance,

PYODIDE_PACKAGES="toolz,attrs" make

Note that this environment variable must contain both the packages and their
dependencies. The package names must match the folder names in packages/
exactly; in particular they are case sensitive.

To build a minimal version of pyodide, set PYODIDE_PACKAGES="micropip". The
micropip package is generally always included for any non empty value of
PYODIDE_PACKAGES.

Of course one can always build individual packages with pyodide_build/buildpkg.py, but the goal of this is to get a consistent but smaller pyodide distribution with a valid packages.json.

This also adds a minimal build with only micropip included. It should be fairly fast, and not make CI slower too much, since it would run in parallel with the main build job. I find it helpful to add this extra CI job, to see if we can optimize the size of files in this minimal distribution further.

cc @phorward

@rth rth added this to the 0.15.0 milestone May 8, 2020
@phorward
Copy link
Copy Markdown
Member

phorward commented May 9, 2020

Thanks, that's cool!

Quite now, I'm using a service script which downloads the release tarball and "crops" the Pyodide-installation to a micropip-only version. The packages.json is built manually. This is usually not a good way, but a working one with minimal effort to get a Pyodide for our web-app development quickly.

@rth rth mentioned this pull request May 9, 2020
Open
6 tasks
@rth
Copy link
Copy Markdown
Member Author

rth commented May 9, 2020 •
edited
Loading

Good to know thanks @phorward ! By having a way to optionally build a minimal pyodide distribution, there is quite a few things we could optimize for the size (and load time) to the resulting pyodide package see #646

@rth
Copy link
Copy Markdown
Member Author

rth commented May 9, 2020 •
edited
Loading

So with this option, the minimal build in CI takes 13min as opposed to 35min for the full pyodide build (assuming emsdk is already built).

@rth rth mentioned this pull request May 9, 2020
Closed
@rth rth requested a review from phorward May 11, 2020 20:15
phorward
phorward requested changes May 11, 2020
View reviewed changes
Copy link
Copy Markdown
Member

@phorward phorward left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @rth this looks good to me, only two minor notes, we might ignore that.

@rth
Copy link
Copy Markdown
Member Author

rth commented May 12, 2020

Thanks for the review @phorward ! I have addressed you comments.

(The chrome CI failure in scipy and mne can be ignored.)

@phorward phorward merged commit e8d2d82 into pyodide:master May 12, 2020
@rth rth deleted the partial-build branch May 12, 2020 12:54
@rth
Copy link
Copy Markdown
Member Author

rth commented May 12, 2020

One thing I forgot to mention @phorward is that we should click on "squash and merge" button when merging, to get a single commit on master.

@phorward
Copy link
Copy Markdown
Member

phorward commented May 12, 2020

@rth oops okay I'm sorry ;-) next time I will do so! In the repository settings, direct merge can also be prohibited...
image

@rth
Copy link
Copy Markdown
Member Author

rth commented May 12, 2020 •
edited
Loading

No worries. Once you do it that way, I think it should automatically switch to it by default in your Github UI (or we can indeed disable it in setting).

@wlach
Copy link
Copy Markdown
Contributor

wlach commented May 12, 2020

No worries. Once you do it that way, I think it should automatically switch to it by default in your Github UI (or we can indeed disable it in setting).

👍 on disabling merge commits, I'm honestly not sure why that option is still there...

@rth rth mentioned this pull request Jul 8, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@phorward phorward phorward requested changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

0.15.0

Development

Successfully merging this pull request may close these issues.

3 participants

@rth @phorward @wlach