Skip to content

Windows rpath support#31930

Merged
scheibelp merged 10 commits intospack:developfrom
johnwparent:windows-rpath-support
Sep 13, 2022
Merged

Windows rpath support#31930
scheibelp merged 10 commits intospack:developfrom
johnwparent:windows-rpath-support

Conversation

@johnwparent
Copy link
Copy Markdown
Contributor

This PR vendors initial pass Windows rpath analog support. Windows does not understand the concept of an RPATH, so Spack must implement similar functionality and perform such actions manually. This is accomplished via symlinking shared libraries, later to be replaced with a more sophisticated method. This PR is intended as a temporary stop gap support for the concept of RPATHs on Windows.

PR is currently in progress and pending review.

@spackbot-app

This comment was marked as off-topic.

@johnwparent johnwparent requested a review from scheibelp August 4, 2022 18:14
@scheibelp scheibelp self-assigned this Aug 5, 2022
Copy link
Copy Markdown
Member

@scheibelp scheibelp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Preliminary review. Feel free to ignore lower-level criticisms (only 2-3 of the comments are high-level).

@johnwparent johnwparent force-pushed the windows-rpath-support branch from 2f7bf60 to b75ec51 Compare August 16, 2022 19:27
@johnwparent johnwparent marked this pull request as ready for review August 23, 2022 22:23
Copy link
Copy Markdown
Member

@scheibelp scheibelp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional questions/requests.

Copy link
Copy Markdown
Member

@scheibelp scheibelp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This follow up is mostly questions/responses (although depending on the answers, some changes may be useful)

@johnwparent johnwparent force-pushed the windows-rpath-support branch from 3a5d002 to 4e1337f Compare September 6, 2022 14:09
Copy link
Copy Markdown
Member

@scheibelp scheibelp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional requests, all minor (hopefully - let me know if you disagree).

@johnwparent johnwparent force-pushed the windows-rpath-support branch 2 times, most recently from dbd44a7 to a21c54e Compare September 7, 2022 18:37
@johnwparent johnwparent force-pushed the windows-rpath-support branch from 577694c to fd6c431 Compare September 8, 2022 18:23
@scheibelp

This comment was marked as outdated.

@spackbot-app

This comment was marked as outdated.

@scheibelp

This comment was marked as outdated.

@spackbot-app

This comment was marked as outdated.

Copy link
Copy Markdown
Member

@scheibelp scheibelp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a couple of notes for future refactoring but I think this is in a good state to merge as is.

for path in self.pkg.rpath:
dependent_libs.extend(list(find_all_shared_libraries(path, recursive=True)))
for extra_path in self._addl_rpaths:
dependent_libs.extend(list(find_all_shared_libraries(extra_path, recursive=True)))
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[4] Ultimately I think it will be most idiomatic to iterate through dependencies (transitively) and ask for .libs: as it is, this essentially replicates that but excludes special cases (e.g. a specific package might know where to find libraries and def libs accordingly, but this would miss that). That can be refactored later.

"""

dependent_libs = []
for path in self.pkg.rpath:
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We spoke about this needing to be transitive at https://github.com/spack/spack/pull/31930/files#r957678049 - I should correct myself and say that Package.rpath only includes direct dependencies (not that this needs to change now given that this would likely change as part of [4]).

@scheibelp
Copy link
Copy Markdown
Member

@spackbot run pipeline

@spackbot-app
Copy link
Copy Markdown

spackbot-app bot commented Sep 11, 2022

I've started that pipeline for you!

@scheibelp scheibelp merged commit 53a7b49 into spack:develop Sep 13, 2022
Comment on lines +2057 to +2063
if is_windows:
static = "lib"
shared = "dll"
else:
# Used on both Linux and macOS
static = "a"
shared = "so"
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mwkrentel @scheibelp @johnwparent That's the error. Here the shared argument is overridden, so we'll always hit the True condition in the conditional below.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh just found my way here looking into the armadillo build failure in the tutorial stack from today. It can't find libraries in the relocated superlu binary. I guess this is the source of the problem?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zackgalbreath I think this ☝️ could the problem we've been looking into.

Copy link
Copy Markdown
Contributor Author

@johnwparent johnwparent Sep 14, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alalazo Noticed this a bit ago, I have a solution for this if you haven't started.

#32653

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No I didn't work on a solution, just getting to the point of figuring out the cause.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

4 participants