Skip to content

Fix "spack install" when using Python 3.8 on macos#18205

Merged
tgamblin merged 110 commits intodevelopfrom
bugfix/fork-function
Nov 12, 2020
Merged

Fix "spack install" when using Python 3.8 on macos#18205
tgamblin merged 110 commits intodevelopfrom
bugfix/fork-function

Conversation

@scheibelp
Copy link
Copy Markdown
Member

@scheibelp scheibelp commented Aug 20, 2020

See: #14187

This is the same as #14187 but created as a branch of the main Spack repository to run additional CI checks.

Fixes #14102

…aturely. Saving them in __init__ prevents this although I don't know why
…ferent entities attempt to close the same FD)
… the requirement to make Spec and all associated objects pickle-able)
…at the spec is passed as a dictionary rather than an object
…e applying updates related to installing the package (running the _writer_daemon process requires the environment that was used to start Spack itself)
@scheibelp scheibelp force-pushed the bugfix/fork-function branch from df2d556 to 6bbb7de Compare November 11, 2020 22:44
@scheibelp
Copy link
Copy Markdown
Member Author

@becker33 @tgamblin discussed changes are now complete:

  • Reverted changes that use spawn start method everywhere (for now)
  • Renamed test_state module to subprocess_context (and added docstring)
  • Added logic to revert environment changes when starting _writer_daemon: 6bbb7de

@tldahlgren #18205 (comment) is now addressed.

Copy link
Copy Markdown
Member

@tgamblin tgamblin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@scheibelp: I have a few very minor change requests.

@scheibelp
Copy link
Copy Markdown
Member Author

@tgamblin all review comments are now addressed.

I left #18205 (comment) unresolved because I only handled the part about changing the comment to clarify how things work on Windows. I did not do

Can Python 2 be made to spawn as well so that we do not have a separate code path for 2 and 3 here?

because I think we already spoke about skipping this for now (I got the impression this was an old review comment because it was out of date before I made any edits).

@scheibelp
Copy link
Copy Markdown
Member Author

@adamjstewart #18205 (review) is now addressed

@becker33
Copy link
Copy Markdown
Member

Closing and reopening to kick off the tests that's wedged.

@becker33 becker33 closed this Nov 12, 2020
@becker33 becker33 reopened this Nov 12, 2020
@tgamblin tgamblin merged commit bb42470 into develop Nov 12, 2020
@tgamblin tgamblin deleted the bugfix/fork-function branch November 12, 2020 20:26
scheibelp added a commit that referenced this pull request Nov 12, 2020
As of #18205, all packages must be pickle-able to be installed by
Spack.

This adds a test to check that each package can be pickled. If any
package fails to pickle, the test keeps going and collects the names
of all failed packages; it then takes the first one that failed and
attempts to re-pickle it, generating the full stack trace for the
failed pickle attempt.
scheibelp pushed a commit that referenced this pull request Dec 11, 2020
This fixes a logging error observed on macOS 11.0.1 (Big Sur).
When performing a Spack install in debugging mode (e.g.
`spack -d install py-scipy`) Spack is supposed to write a log of
compiler wrapper command line invocations to the current working
directory.

Due to a regression error introduced by #18205, these files were
no-longer generated, and Spack was printing errors such as
"No such file or directory: None/." This is because the log file
directory gets set from `spack.main.spack_working_dir`, but that
variable is not set in the spawned process.

This PR ensures that the working directory (at the time of the
"spack install" invocation) is persisted to the subprocess.
bollig pushed a commit to bollig/spack that referenced this pull request Jan 12, 2021
This fixes a logging error observed on macOS 11.0.1 (Big Sur).
When performing a Spack install in debugging mode (e.g.
`spack -d install py-scipy`) Spack is supposed to write a log of
compiler wrapper command line invocations to the current working
directory.

Due to a regression error introduced by spack#18205, these files were
no-longer generated, and Spack was printing errors such as
"No such file or directory: None/." This is because the log file
directory gets set from `spack.main.spack_working_dir`, but that
variable is not set in the spawned process.

This PR ensures that the working directory (at the time of the
"spack install" invocation) is persisted to the subprocess.
alalazo added a commit to alalazo/spack that referenced this pull request Aug 26, 2021
This object was introduced in spack#18124 which
was later superseded by spack#18205 and removed
any use if the object.
scheibelp pushed a commit that referenced this pull request Aug 26, 2021
This object was introduced in #18124, and was later superseded by
#18205 and removed any use if the object.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MacOS Mojave, Python 3.8: "AttributeError: Can't pickle local object 'fork.<locals>.child_process'"

6 participants