spack ci rebuild fails to install all cached dependencies. #17544
Description
Steps to reproduce the issue
I'm trying to narrow down an issue in spack ci rebuild for which #17536 should be a solution, but it seems to not entirely fix everything.
I believe the issue is when you install a package (e.g. curl) which has dependencies in common with a compiler (e.g. for gcc it is perl, zlib) it will only install one version of perl, say perl%[email protected], but not the spec compiled with the new compiler perl%[email protected].
It seems the spack installcommand will fail the first time, but will not fail if I repeat it once more
If you want to try reproduce this, please use the stabbles/spack-fix docker image, since it contains the fixes from #17536 and more, and do e.g. docker run -it --rm --entrypoint="" -v `pwd`:/workspace stabbles/spack-fix /bin/bash and run the following:
$ cd /workspace
$ ls
config.yaml curl.yaml run.sh spack.asc
$ cat config.yaml
config:
install_missing_compilers: True
$ cat run.sh
#!/bin/bash
unset MODULEPATH
export AWS_ACCESS_KEY_ID=xxxx
export AWS_SECRET_ACCESS_KEY=xxxx
export S3_ENDPOINT_URL=http://148.187.98.133:9000
cd /workspace
spack gpg trust spack.asc
spack mirror add minio s3://spack
cp /workspace/config.yaml /opt/spack/etc/spack/config.yaml
'/opt/spack/bin/spack' '-d' 'install' '--keep-stage' '--cache-only' '--only' 'dependencies' './curl.yaml'
$ cat spack.asc
-----BEGIN PGP PUBLIC KEY BLOCK-----
mQGNBF8GYB4BDADAoHHjv6uObSgXgVmTDsa9uobekL8zKhNLPakj2Oa1l7x4CyLZ
Ac+xRa4WfN0YoEaOaM71XS3g5svTKkDq+h/OfNq1iKBs0uA6E4H5SmfwVK9mV9L5
4HeyWhI6PvFI8ek6kUm0G+UAh/M8kUZbMwHlarQ23J4LYofRUYb/LyVXft55+sYi
VWhf/1stydzJMRdSCjFu/0GUMUhXCtifCk8MFkolZ6hT8j8qbCkzpKbP96FMyHFO
BzQInIV15AoJxOOMjDLqyhoKujuS5GzA31CgiSsLX03B/uk4rAm1dMPj9q5hs3pl
xWX5opj96OS9yPQrnD+zcxLV9So/mrrlDSPLCkUYmTKYAYkqDVfcn/Kb66f5TDZY
TnWQcPbAdSDBp3fuVdjo1WedIiyKoTPXO+vXEbC72CsuSEGJJTEoygKdFACSgC9r
oUE1OGW/uzENXgemXyfAvvkiJhXOJbkdTH9v7Ibtoh9GXUMIp9DZmqg5AwmXyzpa
TtmUPOATNKZOhSkAEQEAAbRESGFybWVuIFN0b3BwZWxzIChHaXRsYWIgU3BhY2sg
c2lnbmluZyBrZXkpIDxoYXJtZW4uc3RvcHBlbHNAY3Njcy5jaD6JAc4EEwEKADgW
IQTF1j3z8gwfZR07C4SeH2D7uK8ulQUCXwZgHgIbAwULCQgHAgYVCgkICwIEFgID
AQIeAQIXgAAKCRCeH2D7uK8ulZfuC/9U3PbmdLnyBNfyru2i985O2pDfqED9n0Ee
+zJ+T+oMlxSxJvfLaetJG3IBIPb6fZOZJbJpN26/9g/2PEQeWDcKcUHgylaomFot
CoVTAbP81YTaqtykmBnEduuxLB3V1nbpKckXJPNzFxTqAyZsx32ri0j1B7uA1LFB
/CS+0eZ07IVcdMzLpkByC4Z+9EyKBP4dGXWsiXjOr57qGa2u2LC5HWTBilGpkP3g
WEYsia+yVtrX8B80sT/Ah6iRScfdAKULodOgp3H/+ylcCB5FJuhJA3kWC62uo+hJ
Me4oOSm9kZcu3MzGHB/fS47SNJJT2AzVjXnZ7yB651gdWQ0u9b6UaHDhJbINBESl
0wepzHhOnvKaYGf9QaPtt45KNOoLLNyAxV9rgib/5U7DitLp4MES5jRPpWirtEj8
kPZUnuZonuqUUtcUzm/CJYHuQ+t+kaHPLqpJmlYkY/R4nzjmuavGIneKV3B7rQgx
FQMI2m/vtDgYo4rfy6Nv+5tU1hJCRxS5AY0EXwZgHgEMAMS5xJJrhEuqKSCNLKYE
eWbUsRtV3DF3wP0Edd8quNX54IdXd62BjDjX1uHTU/Y/+WCT+u12HlMjRdDaq4a/
QtzauhC9o+u9/5gbNtqDowB4gs9/2QTF92s3sKD/DiuhQeQE7871RCpC9g9UlJOf
RlSIgigaBVfUXftHeHAK0QTFlHSxQiXX4KqVGp+n/G5hnwsDl1JyZzHRrSV/FxTj
kskJkbO2CavJMDrVDiVYI4WxKpY7LfebGbluWGpkMtLMA9eDuHfwkuaih/Oj+AJd
4I7WxwEyo58unXv2fTA1kVxIHWxIua7Se94FPKbkERL4daTFqPfc0QKjpC19ETMt
N0Nwv0v66BZ3CXtjoZ2nbBvWl5gG440slSOhLb5q/Bzn5wtz3ixNdYFXtTz7pZpE
0LjzU7ljtjVLT0OtfzAsrgaC19cqO0LFLdWpJ9WLVAKgyF4gZsnR5rZcFHsxNNcV
aFjJQ3mMskJ1Sjueh5WKHVlH/xv0hNnS3MFrNzNRS4jPBQARAQABiQG2BBgBCgAg
FiEExdY98/IMH2UdOwuEnh9g+7ivLpUFAl8GYB4CGwwACgkQnh9g+7ivLpUc2gv7
BMwPbATk8Gkkaj8co35CtS/jMPw/7IOUaMdKBQiP6KPGL0TOFblYaUreL3SV9qEP
tIzwJiixfwwaE3Vz7MsvuiuWzqTNgVhV/tfgGFjBDVgeZ5Zw+w5kt5wQ0KXxnqR/
3xzI+8cxI+2V4w0PWDK6TIOaXFmmhoBBmLj9c8b1yECsFLPwzexJvyEb3golJ0AK
gq7pk6E2GjMoiFUt5mCZRxJls5wMQeZRdPfgb2jRjb8dBfF0dr5W2CDmn2q6Dq7N
ZYDpePYtJQ+NGJ0+MZgy4/iLfoTWKGZHqEbY17sC8oITHPtgRuMKJAJ9XDcBL1zS
2DkJ5ooh85dA2XODjieC+qRRu1bux2GIH9sGdwIIw+9cfAw9uNCoJULReJ/m6M6o
OnfRqT82RtvBJ3npJKbo8fwGY/svp6iR05rCQWuawfrMg592JC5xdLHCc0mgU2qW
QzI6Ifk92xFmikxfpQNbAKuZ7FSH3t2eeJfhZYJ72jXJv16ff+dt0kzQibzDyhG2
=sqpd
-----END PGP PUBLIC KEY BLOCK-----
The curl.yaml file can be downloaded here: https://gitlab.com/cscs-ci/ci-testing/spack-ci/-/jobs/639550384/artifacts/raw/jobs_scratch_dir/specs/curl.yaml
Error Message
The first time I run ./run.sh it will error:
$ ./run.sh
...
==> [2020-07-15-13:12:09.000894, 393] Error: Detected uninstalled dependencies for openssl: {'perl', 'zlib', 'gcc'}
==> [2020-07-15-13:12:09.000981, 393] InstallError: Cannot proceed with openssl: 3 uninstalled dependencies: perl,zlib,gcc
==> [2020-07-15-13:12:09.001019, 393] Error: Cannot proceed with openssl: 3 uninstalled dependencies: perl,zlib,gcc
Traceback (most recent call last):
File "/opt/spack/lib/spack/spack/main.py", line 761, in main
return _invoke_command(command, parser, args, unknown)
File "/opt/spack/lib/spack/spack/main.py", line 489, in _invoke_command
return_val = command(parser, args)
File "/opt/spack/lib/spack/spack/cmd/install.py", line 397, in install
install_spec(args, kwargs, abstract, concrete)
File "/opt/spack/lib/spack/spack/cmd/install.py", line 243, in install_spec
spec.package.do_install(**kwargs)
File "/opt/spack/lib/spack/spack/package.py", line 1476, in do_install
builder.install(**kwargs)
File "/opt/spack/lib/spack/spack/installer.py", line 1456, in install
','.join(task.uninstalled_deps)))
spack.installer.InstallError: Cannot proceed with openssl: 3 uninstalled dependencies: perl,zlib,gcc
The second time it is fine and installs the missing specs:
$ ./run.sh
...
$ spack find
==> 28 installed packages
-- linux-ubuntu18.04-broadwell / [email protected] ----------------------
[email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected]
-- linux-ubuntu18.04-broadwell / [email protected] ---------------------
[email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected]
Additional information
- I have run
spack debug reportand reported the version of Spack/Python/Platform - I have searched the issues of this repo and believe this is not a duplicate
- I have run the failing commands in debug mode and reported the output