Skip to content

Mirrors: add option to exclude packages from "mirror create"#14154

Merged
becker33 merged 11 commits intospack:developfrom
scheibelp:bugfix/mirror-exclude-failing-packages
Jun 4, 2020
Merged

Mirrors: add option to exclude packages from "mirror create"#14154
becker33 merged 11 commits intospack:developfrom
scheibelp:bugfix/mirror-exclude-failing-packages

Conversation

@scheibelp
Copy link
Copy Markdown
Member

@scheibelp scheibelp commented Dec 13, 2019

See: #13604
See: #13925

Add an --exclude-file option to 'spack mirror create' which allows a user to specify a file of specs to exclude when creating a mirror. this is anticipated to be useful especially when using the '--all' option.

The option can be used as follows:

spack mirror create --exclude-file path/to/mirror-exclude-specs.txt ...

Where the exclude file looks like

py-torch
rna-seqc

#13604 points out there are cases where spack mirror create --all stalls because of packages that require interaction or otherwise hang. #13881 adds a connection timeout for URL fetchers, which handles a subset of these cases. #13925 adds a universal process timeout that is available for all fetch actions, which is intended as a catch-all solution, but requires additional work and special-case handling of git resources (which are the most common non-URL resources).

This also adds a file to share/spack/cmd-config/mirror-exclude-specs.txt

This file is added because it is anticipated that users will commonly need to filter the same set of packages. This is not added as config because modifying the mirror configuration to support blacklisting would require significant adjustments, and also because this option should only be required while #13925 is incomplete.

New updates (as of May 28):

  • When mirroring all specs for an environment, include dependency specs (fixes Mirror All Dependencies in Environment #16808). This does not require specifying --dependencies
  • You can now specify --versions-per-spec when using the --all option. For example spack mirror create --all --versions-per-spec 1 would download the latest version of every package.

…user to specify a file of specs to exclude when creating a mirror. this is anticipated to be useful especially when using the '--all' option
@scheibelp

This comment has been minimized.

Copy link
Copy Markdown
Member

@becker33 becker33 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly LGTM. One API request

@becker33 becker33 merged commit 2477569 into spack:develop Jun 4, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Mirror All Dependencies in Environment

3 participants