Skip to content

Speed up builds#585

Merged
mshinwell merged 3 commits intooxcaml:mainfrom
stedolan:improve-build
May 23, 2022
Merged

Speed up builds#585
mshinwell merged 3 commits intooxcaml:mainfrom
stedolan:improve-build

Conversation

@stedolan
Copy link
Contributor

@stedolan stedolan commented Mar 21, 2022 •
edited
Loading

This patch started life as an attempt to remove some of the redundancy in the flambda-backend build, which currently builds most targets three times. It grew to the point where I ended up rewriting the Makefile completely, with the following properties:

  • Build and test is now ~3x faster locally (~2x faster on github CI, I think because of fewer available cores)
  • Requires installed OCaml 4.12 on PATH (stock OPAM or flambda-backend builds will do)
  • Merlin support restored, using stock merlin
  • Uses dune workspaces, so dune can be invoked directly without setting up env vars
  • Configures ocaml/ only once
  • Upstream testsuite is faster
  • Github CI requires slightly fewer workers, because check_all_arches is integrated with normal build

The major change is that the build no longer does the stage0 / stage1 / stage2 process of building the compiler and supporting tools three times. Instead, a pre-installed OCaml 4.12 compiler is used instead of stage0, and then the build proceeds using three dune workspaces:

  • boot: builds a minimal compiler (ocamlopt, ocamlc, ocamlmklib and ocamldep) using stock OCaml
  • runtime_stdlib: builds the runtime and stdlib using the boot compiler
  • main: builds the compiler using the boot compiler

The installed artifacts are those build as part of runtime_stdlib and main. Note that the runtime and stdlib are only built once, and the bytecode is never used (instead, we use a .opt compiler from stock OCaml).

There are three commits in this patch:

  1. Speed up the build, as described above
  2. Speed up the upstream testsuite by lowering some overly-conservative timeouts and iteration counts
  3. Update the github actions workflow

The major changes are to Makefile.in. There are also identical small changes to every dune file, as it is no longer necessary to explicitly link against stdlib.

Still left to do:

  • check OS X build
  • ensure make install installs the same stuff
  • update HACKING

@mshinwell
Copy link
Collaborator

mshinwell commented Mar 21, 2022

Looks good in outline, but I will leave some review comments in due course. I've had a lot of trouble with my ocamllsp setup involving the need to use a new merlin with the current system, so I will try running with this patch for a few days.

@lukemaurer Perhaps you could try rebasing your coverage PR onto this one.

@lukemaurer
Copy link
Contributor

lukemaurer commented Mar 22, 2022

Looks good in outline, but I will leave some review comments in due course. I've had a lot of trouble with my ocamllsp setup involving the need to use a new merlin with the current system, so I will try running with this patch for a few days.

@lukemaurer Perhaps you could try rebasing your coverage PR onto this one.

Probably best to wait until this is in; #352 is going to have to change significantly in the rebase, and anything that changes here is likely to have fiddly ripple effects.

I can say that this supersedes #361 (by a wide margin), since it never runs ./configure on top of an already-configured source tree, and it uses workspaces in a similar way (which #352 was depending on).

@Gbury
Copy link
Contributor

Gbury commented Mar 22, 2022

With this PR, what version of dune is required ? is it a specific version like what's needed currently ? and if so, is it the same version or another specific revision ?

@stedolan
Copy link
Contributor Author

stedolan commented Mar 23, 2022

With this PR, what version of dune is required ? is it a specific version like what's needed currently ? and if so, is it the same version or another specific revision ?

Currently, it's the same version as needed currently. (I'd quite like to be able to build with standard Dune, though. I'll have a look and see whether the patches are still needed)

@mshinwell mshinwell marked this pull request as draft April 14, 2022 07:58
lukemaurer
lukemaurer reviewed Apr 25, 2022
View reviewed changes
lukemaurer
lukemaurer reviewed Apr 29, 2022
View reviewed changes
@lukemaurer lukemaurer mentioned this pull request May 4, 2022
Merged
@stedolan stedolan force-pushed the improve-build branch 3 times, most recently from 4649799 to ddd238e Compare May 10, 2022 16:35
@stedolan stedolan marked this pull request as ready for review May 18, 2022 13:02
@stedolan
Copy link
Contributor Author

stedolan commented May 18, 2022

I still have some hacking to do on the macos CI run, but this is ready to go once CI passes.

I verified that it installs the same files as the current build, in the same places. The only differences are:

  • bin/ocamlopt is now a symlink to rather than a copy of bin/ocamlopt.opt
  • files that include build paths have changed slightly since these differ (_build vs _build[012])
  • the installed Makefile.config differs in a couple of fields (it now reports the CONFIGURE_ARGS used to configure flambda-backend, rather than one of the sub-configures)

@stedolan stedolan force-pushed the improve-build branch 2 times, most recently from b049032 to 213d953 Compare May 23, 2022 10:12
@mshinwell mshinwell merged commit 960ceb5 into oxcaml:main May 23, 2022
This was referenced May 23, 2022
Merged
Merged
mshinwell added a commit that referenced this pull request May 24, 2022
@mshinwell
Squashed 'ocaml/' changes from da6ff04..454150b
0cc4b0a 
454150b flambda-backend: Speed up testsuite (#658)
8362f9e flambda-backend: Speed up builds (#585)
a527cab flambda-backend: Update backends for changes from ocaml-jst
ce88833 Merge flambda-backend changes
b7506bb Revert "Cherry-pick of ocaml/ocaml 1eeb0e7fe595f5f9e1ea1edbdf785ff3b49feeeb (#12)"
183f688 Add config option to enable/disable stack allocation (#22)
ee7c849 If both the type and mode of an ident are wrong, complain about the type. (#19)
44bade0 Allow submoding during module inclusion checks (#21)
de3bec9 Add subtyping between arrows of related modes (#20)
fe8a98b flambda-backend: Save Mach as Cfg after Selection (#624)
2b205d8 flambda-backend: Clean up algorithms (#611)
93d8615 Enable the local keywords even when the local extension is off (#18)
524f0b4 flambda-backend: Initial refactoring of To_cmm (#619)
81dd85e Documentation for local allocations
b05519f Fix a GC bug in local stack scanning (#17)
9f879de Fix __FUNCTION__ (#15)
0bf75de flambda-backend: Refactor and correct the "is pure" and "can raise" (port upstream PR#10354 and PR#10387) (#555)
d234bfd flambda-backend: Cpp mangling is now a configuration option (#614)
20fc614 flambda-backend: Check that stack frames are not too large (#10085) (#561)
5fc2e95 flambda-backend: Allow CSE of immutable loads across stores (port upstream PR#9562) (#562)
2a650de flambda-backend: Backport commit fc95347 from trunk (#584)
a78975e Optimise "include struct ... end" in more cases (ocaml/ocaml#11134)
b819c66 Cherry-pick of ocaml/ocaml 1eeb0e7fe595f5f9e1ea1edbdf785ff3b49feeeb (#12)
bb363d4 Optimise the allocation of optional arguments (#11)
31651b8 flambda-backend: Improved ARM64 code generation (port upstream PR#9937) (#556)
f0b6d68 flambda-backend: Simplify processing and remove dead code (error paths) in asmlink (port upstream PR#9943) (#557)
90c6746 flambda-backend: Improve code-generation for inlined comparisons (port upstream PR#10228) (#563)

git-subtree-dir: ocaml
git-subtree-split: 454150b
mshinwell pushed a commit that referenced this pull request May 24, 2022
@stedolan @mshinwell
Speed up builds (#585)
5f02d64
@stedolan stedolan mentioned this pull request May 25, 2022
Merged
@xclerc xclerc mentioned this pull request Jun 1, 2022
Closed
@lthls lthls mentioned this pull request Jun 21, 2022
Open
@lukemaurer lukemaurer mentioned this pull request Jun 28, 2022
Closed
lpw25 added a commit to lpw25/flambda-backend that referenced this pull request Jul 4, 2022
@lpw25
Squashed 'ocaml/' changes from ce88833..454150b
5452394 
454150b flambda-backend: Speed up testsuite (oxcaml#658)
8362f9e flambda-backend: Speed up builds (oxcaml#585)
a527cab flambda-backend: Update backends for changes from ocaml-jst

git-subtree-dir: ocaml
git-subtree-split: 454150b
lpw25 added a commit that referenced this pull request Jul 4, 2022
@lpw25
Squashed 'ocaml/' changes from ce88833..454150b
f6ea85c 
454150b flambda-backend: Speed up testsuite (#658)
8362f9e flambda-backend: Speed up builds (#585)
a527cab flambda-backend: Update backends for changes from ocaml-jst

git-subtree-dir: ocaml
git-subtree-split: 454150b
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lukemaurer lukemaurer lukemaurer approved these changes

@mshinwell mshinwell mshinwell approved these changes

@gretay-js gretay-js Awaiting requested review from gretay-js

@lthls lthls Awaiting requested review from lthls

@chambart chambart Awaiting requested review from chambart

@Gbury Gbury Awaiting requested review from Gbury

+1 more reviewer

@poechsel poechsel poechsel approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

build system

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@stedolan @mshinwell @lukemaurer @Gbury @poechsel