Skip to content

MAINT Split CI build-packages step in two#2017

Merged
hoodmane merged 7 commits intopyodide:mainfrom
hoodmane:split-build-packages
Dec 5, 2021
Merged

MAINT Split CI build-packages step in two#2017
hoodmane merged 7 commits intopyodide:mainfrom
hoodmane:split-build-packages

Conversation

@hoodmane
Copy link
Copy Markdown
Member

@hoodmane hoodmane commented Dec 5, 2021 •
edited by rth
Loading

Split build-packages into two stages. Hopefully this will fix the CI timeouts when the cache is not around.

Closes #2013

rth
rth reviewed Dec 5, 2021
View reviewed changes
Copy link
Copy Markdown
Member

@rth rth left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks ! This would resolve #2013 . The ideal case would have been to run those two jobs in parallel then merge the results. But that looks not very straightforward so we can leave it for later.

There are a few test failures in test-python otherwise LGTM.

@hoodmane
Copy link
Copy Markdown
Member Author

hoodmane commented Dec 5, 2021

It seems to be building a bunch of the packages twice, just the second time the ccache is already warmed up. Why doesn't it think the packages are already built the second time?

@hoodmane
Copy link
Copy Markdown
Member Author

hoodmane commented Dec 5, 2021

The ideal case would have been to run those two jobs in parallel then merge the results. But that looks not very straightforward so we can leave it for later.

I don't think it would make any difference to do that. The first stage is 100% dependent on time to build numpy (it finishes last even if it starts first), and all of the packages in the second stage are dependencies of numpy.

@hoodmane
Copy link
Copy Markdown
Member Author

hoodmane commented Dec 5, 2021 •
edited
Loading

It would be pretty natural to split it again after building scipy, so that it would be:

  1. numpy + packages that don't depend on numpy
  2. scipy + packages that don't depend on scipy
  3. scipy-dependent packages

@hoodmane hoodmane changed the title split build packages MAINT Split CI build-packages step in two Dec 5, 2021
@hoodmane hoodmane merged commit 418813d into pyodide:main Dec 5, 2021
@hoodmane hoodmane deleted the split-build-packages branch December 5, 2021 20:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rth rth rth left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

CI build-packages timeouts without ccache

2 participants

@hoodmane @rth