Spack more often than not runs into write lock timeouts during parallel builds #31387
Description
Steps to reproduce
I'm using spack env depfile and make -j128 which spins up 14 spack install processes on some environment.
10 of these processes simply error out saying they timed out for a write lock.
==> Warning: krb5-1.19.3-g43xlroaaqdkssako26cl5k2kfp6kiwl failed to install
==> Warning: AssertionError exception when releasing read lock for krb5-1.19.3-g43xlroaaqdkssako26cl5k2kfp6kiwl:
==> Error: krb5-1.19.3-g43xlroaaqdkssako26cl5k2kfp6kiwl: Package was not installed
==> Error: LockTimeoutError: Timed out waiting for a write lock.
/dev/shm/spack/lib/spack/spack/package_base.py:1562, in do_fetch:
1559 self.stage.check()
1560
1561 self.stage.cache_local()
>> 1562
1563 for patch in self.spec.patches:
1564 patch.fetch()
1565 if patch.stage:
==> Error: LockTimeoutError: Timed out waiting for a write lock.
Warning: AssertionError exception when releasing read lock for krb5-1.19.3-g43xlroaaqdkssako26cl5k2kfp6kiwl
@tldahlgren any clue?
Error message
.
Information on your system
.
General information
- I have run
spack debug reportand reported the version of Spack/Python/Platform - I have searched the issues of this repo and believe this is not a duplicate
- I have run the failing commands in debug mode and reported the output