Skip to content

Separate concretization of build dependencies (iterative)#35739

Closed
alalazo wants to merge 3 commits intospack:developfrom
alalazo:features/separate_concretization_iterative
Closed

Separate concretization of build dependencies (iterative)#35739
alalazo wants to merge 3 commits intospack:developfrom
alalazo:features/separate_concretization_iterative

Conversation

@alalazo
Copy link
Copy Markdown
Member

@alalazo alalazo commented Feb 28, 2023

This draft PR is a tentative implementation of the following iterative procedure to get separate concretization of build dependencies:

iterative_procedure

The basic idea is to decouple the problem and:

  1. Solve first the link/run subDAG of the root specs, and keep track of their build requirements
  2. Perform another solve unifying on every dependency type (like we do now) and concretize the build requirements
  3. Compose the initial spec from the two solutions

Step two is performed as a single "build environment" concretized with unify: when_possible and can have different requirements from the specs concretized at step one. In this draft, for instance, we use the default compiler for build dependencies. This allow us to concretize:

$ spack graph -d --color py-shapely %oneapi

and obtain the following graph (build dependencies in red):
shapely

In case we have a conflict during the solve at 2 the idea is to go back at point one with additional integrity constraints, which would prevent the offending build requirements to be in a valid solution.

Additional notes:

  • During step 1 we have to take particular care to concretize build dependencies for extensions (like py-setuptools) correctly. This is because those build dependencies need to share the same compiler etc. than the underlying interpreter. See py-setuptools for an example of that in the graph above
  • The feedback cycle is not implemented here, so any conflict when computing build dependencies currently leads to a failure
  • The algorithm can be extended to have more than one "build environment" levels. This fits cases like having to build an intermediate compiler when the system provides an ancient one and we want e.g. to bootstrap the latest.

@spackbot-app spackbot-app bot added core PR affects Spack core functionality dependencies python update-package labels Feb 28, 2023
@alalazo alalazo requested review from becker33, haampie, scheibelp, tgamblin and trws and removed request for adamjstewart, pradyunsg and rgommers February 28, 2023 17:22
@adamjstewart
Copy link
Copy Markdown
Member

During step 1 we have to take particular care to concretize build dependencies for extensions (like py-setuptools) correctly. This is because those build dependencies need to share the same compiler etc. than the underlying interpreter. See py-setuptools for an example of that in the graph above

setuptools is a pure-Python library that doesn't depend on a compiler at all (although our current Spack model has every package depend on a compiler).

Does this PR support multiple versions of build deps like setuptools in the DAG, or is that a follow-up PR?

@alalazo
Copy link
Copy Markdown
Member Author

alalazo commented Feb 28, 2023

Does this PR support multiple versions of build deps like setuptools in the DAG, or is that a follow-up PR?

Look closely at the small red cluster of 4 nodes in the DAG 😉 (Spoiler: there are 2 py-setuptools nodes v59.X and v65.X)

@alalazo
Copy link
Copy Markdown
Member Author

alalazo commented Feb 28, 2023

@adamjstewart Note that this approach is currently a draft, and there are competing ones that - if they get performant enough to be usable - would be preferable on the model side.

@adamjstewart
Copy link
Copy Markdown
Member

Awesome! Happy to test or benchmark any of these PRs if needed, although I can't offer much help on the modeling side. py-jupyter-packaging is an even better example than py-setuptools where we literally had to split the package into multiple version-specific packages. Excited to undo that once one of these proposals gets merged!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core PR affects Spack core functionality dependencies python update-package

Projects

Development

Successfully merging this pull request may close these issues.

2 participants