spack load results in duplicate paths in env vars #25306
Description
Steps to reproduce
When I load a package with spack load, I end up with duplicate paths in the environment variables that get set. For even a moderately sized set of packages (10+), this quickly exceeds the maximum length for environment variables on the system:
-bash: /usr/bin/dirname: Argument list too long
To demonstrate this, take the py-matplotlib package:
$ spack install py-matplotlib
$ spack load --first py-matplotlibError message
When I check PYTHONPATH, I see that dependencies like setuptools appear 3 times, and six appears 4 times.
$ echo $PYTHONPATH | tr : '\n' | sort
/home/t-astewart/spack/opt/spack/linux-ubuntu20.04-skylake/gcc-9.3.0/py-cycler-0.10.0-5jodfhlupwm2e3adp3hemefhj34hffky/lib/python3.8/site-packages
/home/t-astewart/spack/opt/spack/linux-ubuntu20.04-skylake/gcc-9.3.0/py-cycler-0.10.0-5jodfhlupwm2e3adp3hemefhj34hffky/lib/python3.8/site-packages
/home/t-astewart/spack/opt/spack/linux-ubuntu20.04-skylake/gcc-9.3.0/py-kiwisolver-1.1.0-zikb73vv6sdlbyg67xpjhnmaaict5pkl/lib/python3.8/site-packages
/home/t-astewart/spack/opt/spack/linux-ubuntu20.04-skylake/gcc-9.3.0/py-kiwisolver-1.1.0-zikb73vv6sdlbyg67xpjhnmaaict5pkl/lib/python3.8/site-packages
/home/t-astewart/spack/opt/spack/linux-ubuntu20.04-skylake/gcc-9.3.0/py-matplotlib-3.4.2-i747rnk3jjwylwzfh6y3t6n4nw5qb7l4/lib/python3.8/site-packages
/home/t-astewart/spack/opt/spack/linux-ubuntu20.04-skylake/gcc-9.3.0/py-numpy-1.21.1-4gd5peaoz6hufkh4asjmkttgqijx3gp2/lib/python3.8/site-packages
/home/t-astewart/spack/opt/spack/linux-ubuntu20.04-skylake/gcc-9.3.0/py-numpy-1.21.1-4gd5peaoz6hufkh4asjmkttgqijx3gp2/lib/python3.8/site-packages
/home/t-astewart/spack/opt/spack/linux-ubuntu20.04-skylake/gcc-9.3.0/py-pillow-simd-7.0.0.post3-nntq4ynahh6t4eulvisz67dzf5dajdl7/lib/python3.8/site-packages
/home/t-astewart/spack/opt/spack/linux-ubuntu20.04-skylake/gcc-9.3.0/py-pillow-simd-7.0.0.post3-nntq4ynahh6t4eulvisz67dzf5dajdl7/lib/python3.8/site-packages
/home/t-astewart/spack/opt/spack/linux-ubuntu20.04-skylake/gcc-9.3.0/py-pyparsing-2.4.7-t7pbxymwj64kxkyuxp72up4d4f7vajlz/lib/python3.8/site-packages
/home/t-astewart/spack/opt/spack/linux-ubuntu20.04-skylake/gcc-9.3.0/py-pyparsing-2.4.7-t7pbxymwj64kxkyuxp72up4d4f7vajlz/lib/python3.8/site-packages
/home/t-astewart/spack/opt/spack/linux-ubuntu20.04-skylake/gcc-9.3.0/py-python-dateutil-2.8.1-buwbm3rju5exkjbuui4rcwvhjabt25x7/lib/python3.8/site-packages
/home/t-astewart/spack/opt/spack/linux-ubuntu20.04-skylake/gcc-9.3.0/py-python-dateutil-2.8.1-buwbm3rju5exkjbuui4rcwvhjabt25x7/lib/python3.8/site-packages
/home/t-astewart/spack/opt/spack/linux-ubuntu20.04-skylake/gcc-9.3.0/py-setuptools-50.3.2-hazhr4mmmfcq3a2n7gyeav3jtuhp7u53/lib/python3.8/site-packages
/home/t-astewart/spack/opt/spack/linux-ubuntu20.04-skylake/gcc-9.3.0/py-setuptools-50.3.2-hazhr4mmmfcq3a2n7gyeav3jtuhp7u53/lib/python3.8/site-packages
/home/t-astewart/spack/opt/spack/linux-ubuntu20.04-skylake/gcc-9.3.0/py-setuptools-50.3.2-hazhr4mmmfcq3a2n7gyeav3jtuhp7u53/lib/python3.8/site-packages
/home/t-astewart/spack/opt/spack/linux-ubuntu20.04-skylake/gcc-9.3.0/py-six-1.15.0-tsg7v3onyxhkobvgo6xlxj6sulaw4djl/lib/python3.8/site-packages
/home/t-astewart/spack/opt/spack/linux-ubuntu20.04-skylake/gcc-9.3.0/py-six-1.15.0-tsg7v3onyxhkobvgo6xlxj6sulaw4djl/lib/python3.8/site-packages
/home/t-astewart/spack/opt/spack/linux-ubuntu20.04-skylake/gcc-9.3.0/py-six-1.15.0-tsg7v3onyxhkobvgo6xlxj6sulaw4djl/lib/python3.8/site-packages
/home/t-astewart/spack/opt/spack/linux-ubuntu20.04-skylake/gcc-9.3.0/py-six-1.15.0-tsg7v3onyxhkobvgo6xlxj6sulaw4djl/lib/python3.8/site-packagesInformation on your system
- Spack: 0.16.2-3828-cfbefee0fa
- Python: 3.8.10
- Platform: linux-ubuntu20.04-skylake
- Concretizer: original
General information
- I have run
spack debug reportand reported the version of Spack/Python/Platform - I have searched the issues of this repo and believe this is not a duplicate
- I have run the failing commands in debug mode and reported the output