Skip to content

support pkgconfig for mkl#33382

Merged
adamjstewart merged 9 commits intospack:developfrom
rscohn2:dev/mkl-pkgconfig
Oct 27, 2022
Merged

support pkgconfig for mkl#33382
adamjstewart merged 9 commits intospack:developfrom
rscohn2:dev/mkl-pkgconfig

Conversation

@rscohn2
Copy link
Copy Markdown
Member

@rscohn2 rscohn2 commented Oct 17, 2022

Support pkgconfig by adding the mkl directories to PKG_CONFIG_PATH. Some of the pc files depend on pc files provided by the compiler, so try to add them as well.

@haampie
@rgommers
@adamjstewart

@rscohn2 rscohn2 requested a review from haampie October 17, 2022 21:24
@spackbot-app spackbot-app bot added compilers core PR affects Spack core functionality intel update-package labels Oct 17, 2022
@rscohn2 rscohn2 mentioned this pull request Oct 17, 2022
@adamjstewart
Copy link
Copy Markdown
Member

Not sure who would be best to review this, maybe @becker33 or @alalazo?

@rscohn2
Copy link
Copy Markdown
Member Author

rscohn2 commented Oct 17, 2022

It doesn't work because it is using the compiler for mkl, and not the compiler for the dependent package. I want to do something like this:

    def setup_dependent_build_environment(self, env, dependent_spec):
        env.set("MKLROOT", self.component_prefix)
        env.append_path("PKG_CONFIG_PATH", self.component_prefix.lib.pkgconfig)
        # some of the pc files have dependencies on omp from compiler
        if dependent_spec.satisfies("%oneapi"):
            env.append_path("PKG_CONFIG_PATH", join_path(dependent_spec.compiler.prefix, "lib", "pkgconfig"))

But dependent.spec.compiler is a compiler spec and not a compiler. Is it possible to get the compiler from the dependent spec?

@rscohn2
Copy link
Copy Markdown
Member Author

rscohn2 commented Oct 18, 2022

I got rid of the code that tries to automatically add the openmp pkcgconfig. If you are using icx installed manually, then you already need to source the vars.sh for full functionality. If you are using icx from intel-oneapi-compilers, then you need load the package/module.

adamjstewart
adamjstewart previously approved these changes Oct 18, 2022
@rscohn2
Copy link
Copy Markdown
Member Author

rscohn2 commented Oct 20, 2022

@adamjstewart: I see you did a force push. I guess that was a rebase? It failed again. I don't think the problem is related to my change. I looked through the logs and see something suspicious below. The build cache emits a lot of messages that look like errors but are not really errors--patchelf failing, 404, ... so I am never sure what the real error is. Do you think below is normal? Should I wait a while, do a rebase and try again?

==> [2022-10-20-03:42:45.335131] Failure reading URL: Download failed: HTTP Error 404: Not Found
==> [2022-10-20-03:42:56.172639] Error: No binary for mpich-4.0.2-l5klnet3lqifc7p47ho2wzrmvr3vzp54 found when cache-only specified
==> [2022-10-20-03:42:56.245101] Flagging mpich-4.0.2-l5klnet3lqifc7p47ho2wzrmvr3vzp54 as failed: 1
==> [2022-10-20-03:42:56.290009] Warning: Skipping build of openpmd-api-0.14.5-aogwdcrvyva7zfgejpmlkx2lffnubw6s since mpich-4.0.2-l5klnet3lqifc7p47ho2wzrmvr3vzp54 failed
==> [2022-10-20-03:42:56.290370] Flagging openpmd-api-0.14.5-aogwdcrvyva7zfgejpmlkx2lffnubw6s as failed
==> [2022-10-20-03:42:56.343457] Warning: Skipping build of warpx-22.08-xedw6g7liwwkfgh7sf462glcruoijanv since openpmd-api-0.14.5-aogwdcrvyva7zfgejpmlkx2lffnubw6s failed
==> [2022-10-20-03:42:56.343885] Flagging warpx-22.08-xedw6g7liwwkfgh7sf462glcruoijanv as failed
==> [2022-10-20-03:42:56.400319] Warning: Skipping build of py-warpx-22.08-oknb7tlqiotlrh47gwibxyzgkrvhkyj5 since warpx-22.08-xedw6g7liwwkfgh7sf462glcruoijanv failed
==> [2022-10-20-03:42:56.400662] Flagging py-warpx-22.08-oknb7tlqiotlrh47gwibxyzgkrvhkyj5 as failed
==> [2022-10-20-03:42:56.483234] Removing build task for py-warpx-22.08-oknb7tlqiotlrh47gwibxyzgkrvhkyj5 from list
==> [2022-10-20-03:42:56.483614] Removing build task for warpx-22.08-xedw6g7liwwkfgh7sf462glcruoijanv from list
==> [2022-10-20-03:42:56.484036] Removing build task for openpmd-api-0.14.5-aogwdcrvyva7zfgejpmlkx2lffnubw6s from list
==> [2022-10-20-03:42:56.484357] Warning: Skipping build of py-mpi4py-3.1.2-4stbvulpshylsrfqe5iyw5hzmt7madh4 since mpich-4.0.2-l5klnet3lqifc7p47ho2wzrmvr3vzp54 failed
==> [2022-10-20-03:42:56.484930] Flagging py-mpi4py-3.1.2-4stbvulpshylsrfqe5iyw5hzmt7madh4 as failed
==> [2022-10-20-03:42:56.529441] No build task for py-warpx-22.08-oknb7tlqiotlrh47gwibxyzgkrvhkyj5 to skip since py-mpi4py-3.1.2-4stbvulpshylsrfqe5iyw5hzmt7madh4 failed
==> [2022-10-20-03:42:56.529857] Removing build task for py-mpi4py-3.1.2-4stbvulpshylsrfqe5iyw5hzmt7madh4 from list
==> [2022-10-20-03:42:56.530186] Warning: Skipping build of fftw-3.3.10-e6vdw4tjqg572p4y76cktedsxh6zyu63 since mpich-4.0.2-l5klnet3lqifc7p47ho2wzrmvr3vzp54 failed
==> [2022-10-20-03:42:56.530474] Flagging fftw-3.3.10-e6vdw4tjqg572p4y76cktedsxh6zyu63 as failed
==> [2022-10-20-03:42:56.570972] No build task for warpx-22.08-xedw6g7liwwkfgh7sf462glcruoijanv to skip since fftw-3.3.10-e6vdw4tjqg572p4y76cktedsxh6zyu63 failed
==> [2022-10-20-03:42:56.571315] Removing build task for fftw-3.3.10-e6vdw4tjqg572p4y76cktedsxh6zyu63 from list
==> [2022-10-20-03:42:56.571620] Warning: Skipping build of hdf5-1.12.2-rfzkmwlbwme635nezoadne2qxh63raj3 since mpich-4.0.2-l5klnet3lqifc7p47ho2wzrmvr3vzp54 failed
==> [2022-10-20-03:42:56.571958] Flagging hdf5-1.12.2-rfzkmwlbwme635nezoadne2qxh63raj3 as failed
==> [2022-10-20-03:42:56.614849] No build task for openpmd-api-0.14.5-aogwdcrvyva7zfgejpmlkx2lffnubw6s to skip since hdf5-1.12.2-rfzkmwlbwme635nezoadne2qxh63raj3 failed
==> [2022-10-20-03:42:56.615200] Removing build task for hdf5-1.12.2-rfzkmwlbwme635nezoadne2qxh63raj3 from list
==> [2022-10-20-03:42:56.615496] Warning: Skipping build of adios2-2.8.3-lwe4ksr7wqjzymmvzq7p4k7vnyb5ceyp since mpich-4.0.2-l5klnet3lqifc7p47ho2wzrmvr3vzp54 failed
==> [2022-10-20-03:42:56.615818] Flagging adios2-2.8.3-lwe4ksr7wqjzymmvzq7p4k7vnyb5ceyp as failed
==> [2022-10-20-03:42:56.664746] No build task for openpmd-api-0.14.5-aogwdcrvyva7zfgejpmlkx2lffnubw6s to skip since adios2-2.8.3-lwe4ksr7wqjzymmvzq7p4k7vnyb5ceyp failed
==> [2022-10-20-03:42:56.665109] Removing build task for adios2-2.8.3-lwe4ksr7wqjzymmvzq7p4k7vnyb5ceyp from list
==> [2022-10-20-03:42:56.665406] No build task for warpx-22.08-xedw6g7liwwkfgh7sf462glcruoijanv to skip since mpich-4.0.2-l5klnet3lqifc7p47ho2wzrmvr3vzp54 failed
==> [2022-10-20-03:42:56.665673] No build task for libnrm-0.1.0-ttjs7boaf34je6by3g57wck7b3bsaksz to skip since mpich-4.0.2-l5klnet3lqifc7p47ho2wzrmvr3vzp54 failed
==> [2022-10-20-03:42:56.666142] Error: Failed to install mpich due to SystemExit: 1

@rscohn2 rscohn2 disabled auto-merge October 20, 2022 12:25
@adamjstewart
Copy link
Copy Markdown
Member

@scottwittenburg @zackgalbreath can you take a look?

@adamjstewart
Copy link
Copy Markdown
Member

@spackbot run pipeline

@spackbot-app
Copy link
Copy Markdown

spackbot-app bot commented Oct 24, 2022

I've started that pipeline for you!

@adamjstewart
Copy link
Copy Markdown
Member

@spackbot run pipeline

@spackbot-app
Copy link
Copy Markdown

spackbot-app bot commented Oct 24, 2022

I've started that pipeline for you!

@adamjstewart
Copy link
Copy Markdown
Member

@spackbot run pipeline

@spackbot-app
Copy link
Copy Markdown

spackbot-app bot commented Oct 25, 2022

I've started that pipeline for you!


def setup_dependent_build_environment(self, env, dependent_spec):
env.set("MKLROOT", self.component_prefix)
# env.append_path("PKG_CONFIG_PATH", self.component_prefix.lib.pkgconfig)
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did something happen during a rebase? This PR seems just to add a commented out line?

Copy link
Copy Markdown
Member

@adamjstewart adamjstewart Oct 27, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this most recent commit was designed to test whether or not the feature added in this PR was the reason that CI fails 99% of the time.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rscohn2 can you uncomment this? It seems like CI is working again.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It passed when I commented out the line because it did not run any of the tests. The logic that decides when to run tests must be very smart... or broken.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it compute a hash that is not affected by comments?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it is very smart. I believe @tgamblin wrote all the logic that goes into computing the hash, he might be able to point to the logic. At a very minimum, we convert to an AST (so comments are stripped), and we manually remove things like docstrings that won't affect the build process.

Copy link
Copy Markdown
Member

@tgamblin tgamblin Oct 27, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can see the source that affects the hash with:

spack pkg source --canonical intel-oneapi-mkl

and the hash with

spack pkg hash intel-oneapi-mkl

it removes directives (like versions that only add metadata), docstrings, comments, and also @when-annotated methods if we can figure out statically that they're irrelevant for a given spec.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@adamjstewart adamjstewart enabled auto-merge (squash) October 27, 2022 18:28
@adamjstewart adamjstewart merged commit 3e966f2 into spack:develop Oct 27, 2022
@rscohn2 rscohn2 deleted the dev/mkl-pkgconfig branch October 28, 2022 00:58
becker33 pushed a commit to RikkiButler20/spack that referenced this pull request Nov 2, 2022
charmoniumQ pushed a commit to charmoniumQ/spack that referenced this pull request Nov 19, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

compilers core PR affects Spack core functionality intel update-package

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants