fix doubly shell quoting args to spack spec#29282
Conversation
ed6ecca to
3c34a6b
Compare
spack specspack spec
|
Can you check the failing tests @cosmicexplorer? To me this change looks sensible, we shouldn't rely on the shell splitting arguments, but just pass a string with whitespaces to the parser. |
lib/spack/spack/test/cmd/spec.py
Outdated
| def test_spec_parse_arg_quoting(): | ||
| """Assert that certain spec constructions work when provided as arguments.""" | ||
| # Verify that we can provide multiple key=value variants to multiple separate | ||
| # packages within a spec string. | ||
| output = spec('multivalue-variant fee=barbaz ^ a foobar=baz') | ||
|
|
||
| assert 'fee=barbaz' in output | ||
| assert 'foobar=baz' in output | ||
|
|
There was a problem hiding this comment.
I believe that this test succeeds even without the changes in this PR, can you double-check on that?
There was a problem hiding this comment.
You have truly fantastic timing @cosmicexplorer. Would you mind testing:
spack spec ‘[email protected] cflags=“-Os -pipe” cxxflags=“-Os -pipe”spack spec ‘[email protected] cflags=-Os -pipe cxxflags=-Os -pipe
and see what comes out?
This should definitely help, but I think we may still have an over-eager glob on options with a VAL token.
There was a problem hiding this comment.
I believe that this test succeeds even without the changes in this PR, can you double-check on that?
I orchestrated the commits so that the test is added before the fix, and this test indeed fails without the fix:
spack.variant.InvalidVariantValueError: invalid values for variant "fee" in package "multivalue-variant": ['barbaz ^ a foobar=baz']
> spack spec '[email protected] cflags="-Os -pipe" cxxflags="-Os -pipe"'
...
[email protected]%[email protected] cflags="-Os -pipe" cxxflags="-Os -pipe" ~binutils+bootstrap~graphite~nvptx~piclibs~strip languages=c,c++,fortran patches=0689c26,27e9829,7112abb,f5643d9,f73de87 arch=linux-alpine3-zen3
...
> spack spec '[email protected] cflags=-Os -pipe cxxflags=-Os -pipe'
==> Error: Cannot specify variant "pipe" twiceRegarding the second command line with cflags=-Os -pipe: that one is currently accepted by spack on develop, so while I think this change should still be made, it may break some users. How should we notify users about this?
Additionally, we can verify the issue from #29282 (comment) is fixed with the --yaml output:
> spack spec --yaml 'gh cflags="-Os -pipe" cxxflags="-Os -pipe"' | yq
...
cflags:
- -Os
- -pipe
...
> git checkout develop
> spack spec --yaml 'gh cflags="-Os -pipe" cxxflags="-Os -pipe"' | yq
...
cflags:
- '"-Os'
- -pipe" cxxflags="-Os
- -pipe"
...
So it looks like that would absolutely be a good one to include as a new test.
|
Absolutely fabulous: This would be a good one to include as a test, because it was most definitely broken before this. |
The test I see breaking looks very much like a broken test to me. It relies on |
1ac8f9b to
333cf7e
Compare
I fixed
Since this change is actually necessary in order to e.g. write EDIT: I just realized we should be able to actually shlex |
|
Ok, just generated a really nice error report, which also has a test added for it in af00a25. It looks like: For multiple such cases (probably unlikely, but necessary in order to ensure that the arguments causing the error are represented in the error report): |
d79ca03 to
af00a25
Compare
|
That looks really good to me. |
af00a25 to
dfdecb8
Compare
|
gitlab ci had a spurious failure to start the docker daemon and said to try again (https://gitlab.spack.io/spack/spack/-/jobs/2481179), so just force pushed to kick it off again. |
|
Looks like some github action unit test issues also? |
|
Ok, the unit test failure on 3.5 appears to be because subscripting |
c6a442b to
f807d35
Compare
|
codecov/project failed again, so I pulled |
f807d35 to
5a4d9c2
Compare
|
If this set passes with only the codecov failure, we can merge. Codecov is no longer set as required because of spurious failure issues. |
|
Looks like it passed! |
5a4d9c2 to
8971312
Compare
| elif normalize: | ||
| spec.normalize(tests=tests) | ||
| sargs = ' '.join(args) | ||
| unquoted_flags = _UnquotedFlags.extract(sargs) |
There was a problem hiding this comment.
I'd move this inside the except block (just before if unquoted_flags:), as it's not needed except when there's an error.
* add test to verify fix works * fix spec cflags/variants parsing test (breaking change) * fix `spack spec` arg quoting issue * add error report for deprecated cflags coalescing * use .group(n) vs subscript regex group extraction for 3.5 compat * add random test for untested functionality to pass codecov * fix new test failure since rebase
* add test to verify fix works * fix spec cflags/variants parsing test (breaking change) * fix `spack spec` arg quoting issue * add error report for deprecated cflags coalescing * use .group(n) vs subscript regex group extraction for 3.5 compat * add random test for untested functionality to pass codecov * fix new test failure since rebase
* add test to verify fix works * fix spec cflags/variants parsing test (breaking change) * fix `spack spec` arg quoting issue * add error report for deprecated cflags coalescing * use .group(n) vs subscript regex group extraction for 3.5 compat * add random test for untested functionality to pass codecov * fix new test failure since rebase
This PR does three things:
- [x] Allow any character to appear in the quoted values of variants and flags.
- [x] Allow easier passing of quoted flags on the command line, e.g. `cflags="-O2 -g"`
- [x] Handle quoting better in spec output, using single quotes around double
quotes and vice versa.
This PR is motivated by ORNL's [tips for launching at scale on
Frontier](https://docs.olcf.ornl.gov/systems/frontier_user_guide.html#tips-for-launching-at-scale).
ORNL recommends using `sbcast --send-libs` to broadcast executables and their libraries
to compute nodes when running large jobs (e.g., 80k ranks). For an executable named
`exe`, `sbcast --send-libs` stores the needed libraries in a directory alongside the
executable called `exe_libs`. ORNL recommends pointing `LD_LIBRARY_PATH` at that
directory so that `exe` will find the local libraries and not overwhelm the filesystem.
Spack's `RPATHs` break this because they're considered with higher precedence than
`LD_LIBRARY_PATH`. So Spack applications will still end up hitting the shared filesystem
when searching for libraries. We can avoid this by injecting some `ldflags` in to the
build, e.g., if were were going to launch, say, `LAMMPS` at scale, we could add another
`RPATH` specifically for use with `sbcast`:
spack install lammps ldflags='-Wl,-rpath=$ORIGIN/lmp_libs'
This will put the `lmp_libs` directory alongside `LAMMPS`'s `lmp` executable first in
the `RPATH`, so it will be searched before any directories on the shared filesystem.
Before this PR, the command above would've errored out for two reasons:
1. `$` wasn't an allowed character in our spec parser.
2. You would've had to double quote the flags to get them to pass through correctly:
spack install lammps ldflags='"-Wl,-rpath=$ORIGIN/lmp_libs"'
This is ugly and I don't think many users will easily figure it out. The behavior was
added in #29282, and it improved parsing of specs passed as a single string, e.g.:
spack install 'lammps ldflags="-Wl,-rpath=$ORIGIN/lmp_libs"'
but a lot of users are naturally going to try to quote arguments *directly* on the
command line, without quoting their entire spec.
needed to quote them, but the error message could be confusing to users. In particular,
if you wrote `cflags="-O2 -g"` on the command line, it would break the flags up, warn,
and tell you that you could fix the issue by writing `cflags="-O2 -g"` even though you
just wrote that. It's telling you to *quote* that value, but the user has to know to
double quote.
There are only two places where we allow arbitrary quoted strings in specs: flags and
variant values, so this PR adds a simpler heuristic to the CLI parser: if an argument in
`sys.argv` starts with `name=...`, then we assume the whole argument is quoted.
This means you can write:
spack install bzip2 cflags="-O2 -g"
directly on the command line, without multiple levels of quoting. This also works:
spack install 'bzip2 cflags="-O2 -g"'
The only place where this heuristic runs into ambiguity is if you attempt to pass
anonymous specs that start with `name=...` as one large string. e.g., this will be
interpreted as one large flag value:
spack find 'cflags="-O2 -g" ~bar +baz'
This sets `cflags` to `"-O2 -g" ~bar +baz`, which is likely not what you wanted. You can
fix this easily by either removing the quotes:
spack find cflags="-O2 -g" ~bar +baz
Or by adding a space at the start, which has the same effect:
spack find ' cflags="-O2 -g" ~bar +baz'
You may wonder why we don't just look for quotes inside of flag arguments, and the
reason is that you *might* want them there. If you are passing arguments like:
spack install zlib cppflags="-D DEBUG_MSG1='quick fox' -D DEBUG_MSG2='lazy dog'"
You *need* the quotes there. So we've opted for one potentially confusing, but easily
fixed outcome vs. limiting what you can put in your quoted strings.
In addition to being more lenient about characters accepted in quoted strings, this PR
fixes up spec formatting a bit. If a variant or flag value has double quotes in it,
we'll print it with single quotes, and vice versa. If a variant or flag value has both
types of quotes in it, we'll surround it with single quotes and escape internal single
quotes. Examples:
zlib cflags='-D FOO="bar"'
zlib cflags="-D FOO='bar'"
zlib cflags='-D FOO=\'bar\' BAR="baz"'
This PR does three things:
- [x] Allow any character to appear in the quoted values of variants and flags.
- [x] Allow easier passing of quoted flags on the command line, e.g. `cflags="-O2 -g"`
- [x] Handle quoting better in spec output, using single quotes around double
quotes and vice versa.
This PR is motivated by ORNL's [tips for launching at scale on
Frontier](https://docs.olcf.ornl.gov/systems/frontier_user_guide.html#tips-for-launching-at-scale).
ORNL recommends using `sbcast --send-libs` to broadcast executables and their libraries
to compute nodes when running large jobs (e.g., 80k ranks). For an executable named
`exe`, `sbcast --send-libs` stores the needed libraries in a directory alongside the
executable called `exe_libs`. ORNL recommends pointing `LD_LIBRARY_PATH` at that
directory so that `exe` will find the local libraries and not overwhelm the filesystem.
Spack's `RPATHs` break this because they're considered with higher precedence than
`LD_LIBRARY_PATH`. So Spack applications will still end up hitting the shared filesystem
when searching for libraries. We can avoid this by injecting some `ldflags` in to the
build, e.g., if were were going to launch, say, `LAMMPS` at scale, we could add another
`RPATH` specifically for use with `sbcast`:
spack install lammps ldflags='-Wl,-rpath=$ORIGIN/lmp_libs'
This will put the `lmp_libs` directory alongside `LAMMPS`'s `lmp` executable first in
the `RPATH`, so it will be searched before any directories on the shared filesystem.
Before this PR, the command above would've errored out for two reasons:
1. `$` wasn't an allowed character in our spec parser.
2. You would've had to double quote the flags to get them to pass through correctly:
spack install lammps ldflags='"-Wl,-rpath=$ORIGIN/lmp_libs"'
This is ugly and I don't think many users will easily figure it out. The behavior was
added in #29282, and it improved parsing of specs passed as a single string, e.g.:
spack install 'lammps ldflags="-Wl,-rpath=$ORIGIN/lmp_libs"'
but a lot of users are naturally going to try to quote arguments *directly* on the
command line, without quoting their entire spec.
needed to quote them, but the error message could be confusing to users. In particular,
if you wrote `cflags="-O2 -g"` on the command line, it would break the flags up, warn,
and tell you that you could fix the issue by writing `cflags="-O2 -g"` even though you
just wrote that. It's telling you to *quote* that value, but the user has to know to
double quote.
There are only two places where we allow arbitrary quoted strings in specs: flags and
variant values, so this PR adds a simpler heuristic to the CLI parser: if an argument in
`sys.argv` starts with `name=...`, then we assume the whole argument is quoted.
This means you can write:
spack install bzip2 cflags="-O2 -g"
directly on the command line, without multiple levels of quoting. This also works:
spack install 'bzip2 cflags="-O2 -g"'
The only place where this heuristic runs into ambiguity is if you attempt to pass
anonymous specs that start with `name=...` as one large string. e.g., this will be
interpreted as one large flag value:
spack find 'cflags="-O2 -g" ~bar +baz'
This sets `cflags` to `"-O2 -g" ~bar +baz`, which is likely not what you wanted. You can
fix this easily by either removing the quotes:
spack find cflags="-O2 -g" ~bar +baz
Or by adding a space at the start, which has the same effect:
spack find ' cflags="-O2 -g" ~bar +baz'
You may wonder why we don't just look for quotes inside of flag arguments, and the
reason is that you *might* want them there. If you are passing arguments like:
spack install zlib cppflags="-D DEBUG_MSG1='quick fox' -D DEBUG_MSG2='lazy dog'"
You *need* the quotes there. So we've opted for one potentially confusing, but easily
fixed outcome vs. limiting what you can put in your quoted strings.
In addition to being more lenient about characters accepted in quoted strings, this PR
fixes up spec formatting a bit. If a variant or flag value has double quotes in it,
we'll print it with single quotes, and vice versa. If a variant or flag value has both
types of quotes in it, we'll surround it with single quotes and escape internal single
quotes. Examples:
zlib cflags='-D FOO="bar"'
zlib cflags="-D FOO='bar'"
zlib cflags='-D FOO=\'bar\' BAR="baz"'
This PR does three things:
- [x] Allow any character to appear in the quoted values of variants and flags.
- [x] Allow easier passing of quoted flags on the command line, e.g. `cflags="-O2 -g"`
- [x] Handle quoting better in spec output, using single quotes around double
quotes and vice versa.
This PR is motivated by ORNL's [tips for launching at scale on
Frontier](https://docs.olcf.ornl.gov/systems/frontier_user_guide.html#tips-for-launching-at-scale).
ORNL recommends using `sbcast --send-libs` to broadcast executables and their libraries
to compute nodes when running large jobs (e.g., 80k ranks). For an executable named
`exe`, `sbcast --send-libs` stores the needed libraries in a directory alongside the
executable called `exe_libs`. ORNL recommends pointing `LD_LIBRARY_PATH` at that
directory so that `exe` will find the local libraries and not overwhelm the filesystem.
Spack's `RPATHs` break this because they're considered with higher precedence than
`LD_LIBRARY_PATH`. So Spack applications will still end up hitting the shared filesystem
when searching for libraries. We can avoid this by injecting some `ldflags` in to the
build, e.g., if were were going to launch, say, `LAMMPS` at scale, we could add another
`RPATH` specifically for use with `sbcast`:
spack install lammps ldflags='-Wl,-rpath=$ORIGIN/lmp_libs'
This will put the `lmp_libs` directory alongside `LAMMPS`'s `lmp` executable first in
the `RPATH`, so it will be searched before any directories on the shared filesystem.
Before this PR, the command above would've errored out for two reasons:
1. `$` wasn't an allowed character in our spec parser.
2. You would've had to double quote the flags to get them to pass through correctly:
spack install lammps ldflags='"-Wl,-rpath=$ORIGIN/lmp_libs"'
This is ugly and I don't think many users will easily figure it out. The behavior was
added in #29282, and it improved parsing of specs passed as a single string, e.g.:
spack install 'lammps ldflags="-Wl,-rpath=$ORIGIN/lmp_libs"'
but a lot of users are naturally going to try to quote arguments *directly* on the
command line, without quoting their entire spec.
needed to quote them, but the error message could be confusing to users. In particular,
if you wrote `cflags="-O2 -g"` on the command line, it would break the flags up, warn,
and tell you that you could fix the issue by writing `cflags="-O2 -g"` even though you
just wrote that. It's telling you to *quote* that value, but the user has to know to
double quote.
There are only two places where we allow arbitrary quoted strings in specs: flags and
variant values, so this PR adds a simpler heuristic to the CLI parser: if an argument in
`sys.argv` starts with `name=...`, then we assume the whole argument is quoted.
This means you can write:
spack install bzip2 cflags="-O2 -g"
directly on the command line, without multiple levels of quoting. This also works:
spack install 'bzip2 cflags="-O2 -g"'
The only place where this heuristic runs into ambiguity is if you attempt to pass
anonymous specs that start with `name=...` as one large string. e.g., this will be
interpreted as one large flag value:
spack find 'cflags="-O2 -g" ~bar +baz'
This sets `cflags` to `"-O2 -g" ~bar +baz`, which is likely not what you wanted. You can
fix this easily by either removing the quotes:
spack find cflags="-O2 -g" ~bar +baz
Or by adding a space at the start, which has the same effect:
spack find ' cflags="-O2 -g" ~bar +baz'
You may wonder why we don't just look for quotes inside of flag arguments, and the
reason is that you *might* want them there. If you are passing arguments like:
spack install zlib cppflags="-D DEBUG_MSG1='quick fox' -D DEBUG_MSG2='lazy dog'"
You *need* the quotes there. So we've opted for one potentially confusing, but easily
fixed outcome vs. limiting what you can put in your quoted strings.
In addition to being more lenient about characters accepted in quoted strings, this PR
fixes up spec formatting a bit. If a variant or flag value has double quotes in it,
we'll print it with single quotes, and vice versa. If a variant or flag value has both
types of quotes in it, we'll surround it with single quotes and escape internal single
quotes. Examples:
zlib cflags='-D FOO="bar"'
zlib cflags="-D FOO='bar'"
zlib cflags='-D FOO=\'bar\' BAR="baz"'
This PR does three things:
- [x] Allow any character to appear in the quoted values of variants and flags.
- [x] Allow easier passing of quoted flags on the command line, e.g. `cflags="-O2 -g"`
- [x] Handle quoting better in spec output, using single quotes around double
quotes and vice versa.
This PR is motivated by ORNL's [tips for launching at scale on
Frontier](https://docs.olcf.ornl.gov/systems/frontier_user_guide.html#tips-for-launching-at-scale).
ORNL recommends using `sbcast --send-libs` to broadcast executables and their libraries
to compute nodes when running large jobs (e.g., 80k ranks). For an executable named
`exe`, `sbcast --send-libs` stores the needed libraries in a directory alongside the
executable called `exe_libs`. ORNL recommends pointing `LD_LIBRARY_PATH` at that
directory so that `exe` will find the local libraries and not overwhelm the filesystem.
Spack's `RPATHs` break this because they're considered with higher precedence than
`LD_LIBRARY_PATH`. So Spack applications will still end up hitting the shared filesystem
when searching for libraries. We can avoid this by injecting some `ldflags` in to the
build, e.g., if were were going to launch, say, `LAMMPS` at scale, we could add another
`RPATH` specifically for use with `sbcast`:
spack install lammps ldflags='-Wl,-rpath=$ORIGIN/lmp_libs'
This will put the `lmp_libs` directory alongside `LAMMPS`'s `lmp` executable first in
the `RPATH`, so it will be searched before any directories on the shared filesystem.
Before this PR, the command above would've errored out for two reasons:
1. `$` wasn't an allowed character in our spec parser.
2. You would've had to double quote the flags to get them to pass through correctly:
spack install lammps ldflags='"-Wl,-rpath=$ORIGIN/lmp_libs"'
This is ugly and I don't think many users will easily figure it out. The behavior was
added in #29282, and it improved parsing of specs passed as a single string, e.g.:
spack install 'lammps ldflags="-Wl,-rpath=$ORIGIN/lmp_libs"'
but a lot of users are naturally going to try to quote arguments *directly* on the
command line, without quoting their entire spec.
needed to quote them, but the error message could be confusing to users. In particular,
if you wrote `cflags="-O2 -g"` on the command line, it would break the flags up, warn,
and tell you that you could fix the issue by writing `cflags="-O2 -g"` even though you
just wrote that. It's telling you to *quote* that value, but the user has to know to
double quote.
There are only two places where we allow arbitrary quoted strings in specs: flags and
variant values, so this PR adds a simpler heuristic to the CLI parser: if an argument in
`sys.argv` starts with `name=...`, then we assume the whole argument is quoted.
This means you can write:
spack install bzip2 cflags="-O2 -g"
directly on the command line, without multiple levels of quoting. This also works:
spack install 'bzip2 cflags="-O2 -g"'
The only place where this heuristic runs into ambiguity is if you attempt to pass
anonymous specs that start with `name=...` as one large string. e.g., this will be
interpreted as one large flag value:
spack find 'cflags="-O2 -g" ~bar +baz'
This sets `cflags` to `"-O2 -g" ~bar +baz`, which is likely not what you wanted. You can
fix this easily by either removing the quotes:
spack find cflags="-O2 -g" ~bar +baz
Or by adding a space at the start, which has the same effect:
spack find ' cflags="-O2 -g" ~bar +baz'
You may wonder why we don't just look for quotes inside of flag arguments, and the
reason is that you *might* want them there. If you are passing arguments like:
spack install zlib cppflags="-D DEBUG_MSG1='quick fox' -D DEBUG_MSG2='lazy dog'"
You *need* the quotes there. So we've opted for one potentially confusing, but easily
fixed outcome vs. limiting what you can put in your quoted strings.
In addition to being more lenient about characters accepted in quoted strings, this PR
fixes up spec formatting a bit. If a variant or flag value has double quotes in it,
we'll print it with single quotes, and vice versa. If a variant or flag value has both
types of quotes in it, we'll surround it with single quotes and escape internal single
quotes. Examples:
zlib cflags='-D FOO="bar"'
zlib cflags="-D FOO='bar'"
zlib cflags='-D FOO=\'bar\' BAR="baz"'
This PR does several things:
- [x] Allow any character to appear in the quoted values of variants and flags.
- [x] Allow easier passing of quoted flags on the command line, e.g. `cflags="-O2 -g"`.
- [x] Handle quoting better in spec output, using single quotes around double
quotes and vice versa.
- [x] Disallow spaces around `=` and `==` when parsing variants and flags.
## Motivation
This PR is motivated by the issues above and by ORNL's
[tips for launching at scale on Frontier](https://docs.olcf.ornl.gov/systems/frontier_user_guide.html#tips-for-launching-at-scale).
ORNL recommends using `sbcast --send-libs` to broadcast executables and their
libraries to compute nodes when running large jobs (e.g., 80k ranks). For an
executable named `exe`, `sbcast --send-libs` stores the needed libraries in a
directory alongside the executable called `exe_libs`. ORNL recommends pointing
`LD_LIBRARY_PATH` at that directory so that `exe` will find the local libraries and
not overwhelm the filesystem.
There are other ways to mitigate this problem:
* You could build with `RUNPATH` using `spack config add config:shared_linking:type:runpath`,
which would make `LD_LIBRARY_PATH` take precedence over Spack's `RUNPATHs`.
I don't recommend this one because `RUNPATH` can cause many other things to go wrong.
* You could use `spack config add config:shared_linking:bind:true`, added in #31948, which
will greatly reduce the filesystem load for large jobs by pointing `DT_NEEDED` entries in
ELF *directly* at the needed `.so` files instead of relying on `RPATH` search via soname.
I have not experimented with this at 80,000 ranks, but it should help quite a bit.
* You could use [Spindle](https://github.com/hpc/Spindle) (as LLNL does on its machines)
which should transparently fix this without any changes to your executable and without
any need to use `sbcast` or other tools.
But we want to support the `sbcast` use case as well.
## `sbcast` and Spack
Spack's `RPATHs` break the `sbcast` fix because they're considered with higher precedence
than `LD_LIBRARY_PATH`. So Spack applications will still end up hitting the shared filesystem
when searching for libraries. We can avoid this by injecting some `ldflags` in to the build, e.g.,
if were were going to launch, say, `LAMMPS` at scale, we could add another `RPATH`
specifically for use with `sbcast`:
spack install lammps ldflags='-Wl,-rpath=$ORIGIN/lmp_libs'
This will put the `lmp_libs` directory alongside `LAMMPS`'s `lmp` executable first in the
`RPATH`, so it will be searched before any directories on the shared filesystem.
## Issues with quoting
Before this PR, the command above would've errored out for two reasons:
1. `$` wasn't an allowed character in our spec parser.
2. You would've had to double quote the flags to get them to pass through correctly:
spack install lammps ldflags='"-Wl,-rpath=$ORIGIN/lmp_libs"'
This is ugly and I don't think many users will easily figure it out. The behavior was added in
#29282, and it improved parsing of specs passed as a single string, e.g.:
spack install 'lammps ldflags="-Wl,-rpath=$ORIGIN/lmp_libs"'
but a lot of users are naturally going to try to quote arguments *directly* on the command
line, without quoting their entire spec. #29282 used a heuristic to detect unquoted flags
and warn the user, but the warning could be confusing. In particular, if you wrote
`cflags="-O2 -g"` on the command line, it would break the flags up, warn, and tell you
that you could fix the issue by writing `cflags="-O2 -g"` even though you just wrote
that. It's telling you to *quote* that value, but the user has to know to double quote.
## New heuristic for quoted arguments from the CLI
There are only two places where we allow arbitrary quoted strings in specs: flags and
variant values, so this PR adds a simpler heuristic to the CLI parser: if an argument in
`sys.argv` starts with `name=...`, then we assume the whole argument is quoted.
This means you can write:
spack install bzip2 cflags="-O2 -g"
directly on the command line, without multiple levels of quoting. This also works:
spack install 'bzip2 cflags="-O2 -g"'
The only place where this heuristic runs into ambiguity is if you attempt to pass
anonymous specs that start with `name=...` as one large string. e.g., this will be
interpreted as one large flag value:
spack find 'cflags="-O2 -g" ~bar +baz'
This sets `cflags` to `"-O2 -g" ~bar +baz`, which is likely not what you wanted. You
can fix this easily by either removing the quotes:
spack find cflags="-O2 -g" ~bar +baz
Or by adding a space at the start, which has the same effect:
spack find ' cflags="-O2 -g" ~bar +baz'
You may wonder why we don't just look for quotes inside of flag arguments, and the
reason is that you *might* want them there. If you are passing arguments like:
spack install zlib cppflags="-D DEBUG_MSG1='quick fox' -D DEBUG_MSG2='lazy dog'"
You *need* the quotes there. So we've opted for one potentially confusing, but easily
fixed outcome vs. limiting what you can put in your quoted strings.
## Quotes in formatted spec output
In addition to being more lenient about characters accepted in quoted strings, this PR fixes
up spec formatting a bit. We now format quoted strings in specs with single quotes, unless
the string has a single quote in it, in which case we JSON-escape the string (i.e., we add
`\` before `"` and `\`).
zlib cflags='-D FOO="bar"'
zlib cflags="-D FOO='bar'"
zlib cflags="-D FOO='bar' BAR=\"baz\""
This PR does several things:
- [x] Allow any character to appear in the quoted values of variants and flags.
- [x] Allow easier passing of quoted flags on the command line, e.g. `cflags="-O2 -g"`.
- [x] Handle quoting better in spec output, using single quotes around double
quotes and vice versa.
- [x] Disallow spaces around `=` and `==` when parsing variants and flags.
## Motivation
This PR is motivated by the issues above and by ORNL's
[tips for launching at scale on Frontier](https://docs.olcf.ornl.gov/systems/frontier_user_guide.html#tips-for-launching-at-scale).
ORNL recommends using `sbcast --send-libs` to broadcast executables and their
libraries to compute nodes when running large jobs (e.g., 80k ranks). For an
executable named `exe`, `sbcast --send-libs` stores the needed libraries in a
directory alongside the executable called `exe_libs`. ORNL recommends pointing
`LD_LIBRARY_PATH` at that directory so that `exe` will find the local libraries and
not overwhelm the filesystem.
There are other ways to mitigate this problem:
* You could build with `RUNPATH` using `spack config add config:shared_linking:type:runpath`,
which would make `LD_LIBRARY_PATH` take precedence over Spack's `RUNPATHs`.
I don't recommend this one because `RUNPATH` can cause many other things to go wrong.
* You could use `spack config add config:shared_linking:bind:true`, added in spack#31948, which
will greatly reduce the filesystem load for large jobs by pointing `DT_NEEDED` entries in
ELF *directly* at the needed `.so` files instead of relying on `RPATH` search via soname.
I have not experimented with this at 80,000 ranks, but it should help quite a bit.
* You could use [Spindle](https://github.com/hpc/Spindle) (as LLNL does on its machines)
which should transparently fix this without any changes to your executable and without
any need to use `sbcast` or other tools.
But we want to support the `sbcast` use case as well.
## `sbcast` and Spack
Spack's `RPATHs` break the `sbcast` fix because they're considered with higher precedence
than `LD_LIBRARY_PATH`. So Spack applications will still end up hitting the shared filesystem
when searching for libraries. We can avoid this by injecting some `ldflags` in to the build, e.g.,
if were were going to launch, say, `LAMMPS` at scale, we could add another `RPATH`
specifically for use with `sbcast`:
spack install lammps ldflags='-Wl,-rpath=$ORIGIN/lmp_libs'
This will put the `lmp_libs` directory alongside `LAMMPS`'s `lmp` executable first in the
`RPATH`, so it will be searched before any directories on the shared filesystem.
## Issues with quoting
Before this PR, the command above would've errored out for two reasons:
1. `$` wasn't an allowed character in our spec parser.
2. You would've had to double quote the flags to get them to pass through correctly:
spack install lammps ldflags='"-Wl,-rpath=$ORIGIN/lmp_libs"'
This is ugly and I don't think many users will easily figure it out. The behavior was added in
spack#29282, and it improved parsing of specs passed as a single string, e.g.:
spack install 'lammps ldflags="-Wl,-rpath=$ORIGIN/lmp_libs"'
but a lot of users are naturally going to try to quote arguments *directly* on the command
line, without quoting their entire spec. spack#29282 used a heuristic to detect unquoted flags
and warn the user, but the warning could be confusing. In particular, if you wrote
`cflags="-O2 -g"` on the command line, it would break the flags up, warn, and tell you
that you could fix the issue by writing `cflags="-O2 -g"` even though you just wrote
that. It's telling you to *quote* that value, but the user has to know to double quote.
## New heuristic for quoted arguments from the CLI
There are only two places where we allow arbitrary quoted strings in specs: flags and
variant values, so this PR adds a simpler heuristic to the CLI parser: if an argument in
`sys.argv` starts with `name=...`, then we assume the whole argument is quoted.
This means you can write:
spack install bzip2 cflags="-O2 -g"
directly on the command line, without multiple levels of quoting. This also works:
spack install 'bzip2 cflags="-O2 -g"'
The only place where this heuristic runs into ambiguity is if you attempt to pass
anonymous specs that start with `name=...` as one large string. e.g., this will be
interpreted as one large flag value:
spack find 'cflags="-O2 -g" ~bar +baz'
This sets `cflags` to `"-O2 -g" ~bar +baz`, which is likely not what you wanted. You
can fix this easily by either removing the quotes:
spack find cflags="-O2 -g" ~bar +baz
Or by adding a space at the start, which has the same effect:
spack find ' cflags="-O2 -g" ~bar +baz'
You may wonder why we don't just look for quotes inside of flag arguments, and the
reason is that you *might* want them there. If you are passing arguments like:
spack install zlib cppflags="-D DEBUG_MSG1='quick fox' -D DEBUG_MSG2='lazy dog'"
You *need* the quotes there. So we've opted for one potentially confusing, but easily
fixed outcome vs. limiting what you can put in your quoted strings.
## Quotes in formatted spec output
In addition to being more lenient about characters accepted in quoted strings, this PR fixes
up spec formatting a bit. We now format quoted strings in specs with single quotes, unless
the string has a single quote in it, in which case we JSON-escape the string (i.e., we add
`\` before `"` and `\`).
zlib cflags='-D FOO="bar"'
zlib cflags="-D FOO='bar'"
zlib cflags="-D FOO='bar' BAR=\"baz\""
This PR does several things:
- [x] Allow any character to appear in the quoted values of variants and flags.
- [x] Allow easier passing of quoted flags on the command line, e.g. `cflags="-O2 -g"`.
- [x] Handle quoting better in spec output, using single quotes around double
quotes and vice versa.
- [x] Disallow spaces around `=` and `==` when parsing variants and flags.
## Motivation
This PR is motivated by the issues above and by ORNL's
[tips for launching at scale on Frontier](https://docs.olcf.ornl.gov/systems/frontier_user_guide.html#tips-for-launching-at-scale).
ORNL recommends using `sbcast --send-libs` to broadcast executables and their
libraries to compute nodes when running large jobs (e.g., 80k ranks). For an
executable named `exe`, `sbcast --send-libs` stores the needed libraries in a
directory alongside the executable called `exe_libs`. ORNL recommends pointing
`LD_LIBRARY_PATH` at that directory so that `exe` will find the local libraries and
not overwhelm the filesystem.
There are other ways to mitigate this problem:
* You could build with `RUNPATH` using `spack config add config:shared_linking:type:runpath`,
which would make `LD_LIBRARY_PATH` take precedence over Spack's `RUNPATHs`.
I don't recommend this one because `RUNPATH` can cause many other things to go wrong.
* You could use `spack config add config:shared_linking:bind:true`, added in spack#31948, which
will greatly reduce the filesystem load for large jobs by pointing `DT_NEEDED` entries in
ELF *directly* at the needed `.so` files instead of relying on `RPATH` search via soname.
I have not experimented with this at 80,000 ranks, but it should help quite a bit.
* You could use [Spindle](https://github.com/hpc/Spindle) (as LLNL does on its machines)
which should transparently fix this without any changes to your executable and without
any need to use `sbcast` or other tools.
But we want to support the `sbcast` use case as well.
## `sbcast` and Spack
Spack's `RPATHs` break the `sbcast` fix because they're considered with higher precedence
than `LD_LIBRARY_PATH`. So Spack applications will still end up hitting the shared filesystem
when searching for libraries. We can avoid this by injecting some `ldflags` in to the build, e.g.,
if were were going to launch, say, `LAMMPS` at scale, we could add another `RPATH`
specifically for use with `sbcast`:
spack install lammps ldflags='-Wl,-rpath=$ORIGIN/lmp_libs'
This will put the `lmp_libs` directory alongside `LAMMPS`'s `lmp` executable first in the
`RPATH`, so it will be searched before any directories on the shared filesystem.
## Issues with quoting
Before this PR, the command above would've errored out for two reasons:
1. `$` wasn't an allowed character in our spec parser.
2. You would've had to double quote the flags to get them to pass through correctly:
spack install lammps ldflags='"-Wl,-rpath=$ORIGIN/lmp_libs"'
This is ugly and I don't think many users will easily figure it out. The behavior was added in
spack#29282, and it improved parsing of specs passed as a single string, e.g.:
spack install 'lammps ldflags="-Wl,-rpath=$ORIGIN/lmp_libs"'
but a lot of users are naturally going to try to quote arguments *directly* on the command
line, without quoting their entire spec. spack#29282 used a heuristic to detect unquoted flags
and warn the user, but the warning could be confusing. In particular, if you wrote
`cflags="-O2 -g"` on the command line, it would break the flags up, warn, and tell you
that you could fix the issue by writing `cflags="-O2 -g"` even though you just wrote
that. It's telling you to *quote* that value, but the user has to know to double quote.
## New heuristic for quoted arguments from the CLI
There are only two places where we allow arbitrary quoted strings in specs: flags and
variant values, so this PR adds a simpler heuristic to the CLI parser: if an argument in
`sys.argv` starts with `name=...`, then we assume the whole argument is quoted.
This means you can write:
spack install bzip2 cflags="-O2 -g"
directly on the command line, without multiple levels of quoting. This also works:
spack install 'bzip2 cflags="-O2 -g"'
The only place where this heuristic runs into ambiguity is if you attempt to pass
anonymous specs that start with `name=...` as one large string. e.g., this will be
interpreted as one large flag value:
spack find 'cflags="-O2 -g" ~bar +baz'
This sets `cflags` to `"-O2 -g" ~bar +baz`, which is likely not what you wanted. You
can fix this easily by either removing the quotes:
spack find cflags="-O2 -g" ~bar +baz
Or by adding a space at the start, which has the same effect:
spack find ' cflags="-O2 -g" ~bar +baz'
You may wonder why we don't just look for quotes inside of flag arguments, and the
reason is that you *might* want them there. If you are passing arguments like:
spack install zlib cppflags="-D DEBUG_MSG1='quick fox' -D DEBUG_MSG2='lazy dog'"
You *need* the quotes there. So we've opted for one potentially confusing, but easily
fixed outcome vs. limiting what you can put in your quoted strings.
## Quotes in formatted spec output
In addition to being more lenient about characters accepted in quoted strings, this PR fixes
up spec formatting a bit. We now format quoted strings in specs with single quotes, unless
the string has a single quote in it, in which case we JSON-escape the string (i.e., we add
`\` before `"` and `\`).
zlib cflags='-D FOO="bar"'
zlib cflags="-D FOO='bar'"
zlib cflags="-D FOO='bar' BAR=\"baz\""
* add test to verify fix works * fix spec cflags/variants parsing test (breaking change) * fix `spack spec` arg quoting issue * add error report for deprecated cflags coalescing * use .group(n) vs subscript regex group extraction for 3.5 compat * add random test for untested functionality to pass codecov * fix new test failure since rebase
Problem
The following command fails:
But this appears to be a valid spec when parsed directly:
Solution
spack specmultiple times before joining them.Result