Skip to content

Clobber-free build#361

Closed
lukemaurer wants to merge 7 commits intooxcaml:mainfrom
lukemaurer:clobber-free
Closed

Clobber-free build#361
lukemaurer wants to merge 7 commits intooxcaml:mainfrom
lukemaurer:clobber-free

Conversation

@lukemaurer
Copy link
Contributor

@lukemaurer lukemaurer commented Oct 29, 2021

NOTE: Requires updating your special Dune to the lukemaurer:special-dune-2.9 branch.

The current system relies on modifying the ocaml/ subtree in place. This is fragile, and is particularly a problem now that we have stage1 and stage2 as build contexts sharing a _build directory (and hence sharing saved build state): stage1 and stage2 can clobber each other's inputs, triggering unnecessary rebuilds.

We can avoid ever modifying a file in ocaml/ by exploiting two
non-obvious properties of Autoconf:

  • a configure script can be run from any directory and will only create files where it's run

  • the config.status script that configure generates can generate individual files on demand, also from any directory

For stage1 and stage2, then, we can just use configure to generate a config.status, and then Dune can take care of running config.status for itself - which naturally happens in the stage's own _build subdirectory. There's some faff required to use a config.status as a Dune action, since it normally creates files in subdirectories, which Dune doesn't support. Also, Dune wants to know exactly what files config.status will create, so we end up tied more closely to the upstream configure.ac than we might like. Finally, we have several Dune actions that run Make and thus depend on Makefile.config, which has to be declared explicitly now because Makefile.config is not in the source tree. But in return, the build should be considerably more robust, and we continue moving work from Make to Dune (make stage1 and make stage2 barely do anything more than call Dune).

Copy link
Contributor

@Gbury Gbury left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't really looked at how the build process works (and I guess that as long as CI (and us) manage to build the tree, these changes should be fine), however you should also update the documentation about the version of dune required in the README.
Better yet, you could push your branch to the ocaml-flambda fork of dune (and overwrite the current special_dune branch with your branch), so that we only need to update our local dune ?

@mshinwell
Copy link
Collaborator

Yeah I will sort out the update of special_dune.

NOTE: This commit requires updating your special Dune to the
[lukemaurer:special-dune-2.9][] branch.

[lukemaurer:special-dune-2.9]: http://github.com/lukemaurer/dune/tree/special-dune-2.9

The current system relies on modifying the `ocaml/` subtree in place.
This is fragile, and is particularly a problem now that we have stage1
and stage2 as build contexts sharing a `_build` directory (and hence
sharing saved build state): stage1 and stage2 can clobber each other's
inputs, triggering unnecessary rebuilds.

We can avoid ever modifying a file in `ocaml/` by exploiting two
non-obvious properties of Autoconf:

- a `configure` script can be run from any directory and will only
  create files where it's run

- the `config.status` script that `configure` generates can generate
  individual files on demand, also from any directory

For stage1 and stage2, then, we can just use `configure` to generate
a `config.status`, and then Dune can take care of running
`config.status` for itself - which naturally happens in the stage's
own `_build` subdirectory. There's some faff required to use a
`config.status` as a Dune action, since it normally creates files in
subdirectories, which Dune doesn't support. Also, Dune wants to know
exactly what files `config.status` will create, so we end up tied more
closely to the upstream `configure.ac` than we might like. Finally, we
have several Dune actions that run Make and thus depend on
`Makefile.config`, which has to be declared explicitly now because
`Makefile.config` is not in the source tree. But in return, the build
should be considerably more robust, and we continue moving work from
Make to Dune (`make stage1` and `make stage2` barely do anything more
than call Dune).
Also added a comment to make the new stage0 config code a little
clearer.
@lukemaurer lukemaurer mentioned this pull request Mar 22, 2022
3 tasks
@lukemaurer
Copy link
Contributor Author

Superseded by #585.

@lukemaurer lukemaurer closed this Jun 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants