Binary cache issue: "reason: full_hash was missing from remote spec.yaml" happens all the time #19128
Description
Spack keeps pushing existing packages to the binary cache
Steps to reproduce the issue
$ spack find -p /2ocj3q2
==> 1 installed package
-- linux-ubuntu18.04-x86_64 / [email protected] -------------------------
[email protected] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/python-3.8.5-2ocj3q2hl5vz5qhpeli6y73z63mk23vn
$ spack buildcache check --rebuild-on-error --mirror-url https://spack.dev:9000/spack -s /2ocj3q2
==> Rebuilding [email protected]%[email protected]+bz2+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4+uuid+zlib patches=0d98e93189bc278fbc37a50ed7f183bd8aaf249a8e1670a465f0db6bb4f8cf87 arch=linux-ubuntu18.04-x86_64/2ocj3q2, reason: full_hash was missing from remote spec.yamlbut the remote has full hashes for sure:
$ curl -s https://spack.dev:9000/spack/build_cache/linux-ubuntu18.04-x86_64-gcc-7.5.0-python-3.8.5-2ocj3q2hl5vz5qhpeli6y73z63mk23vn.spec.yaml | grep --color=always full_hash
full_hash: ytltjej6ln3tst4f7birdiundhwpswev
full_hash: t57kskuxst7gbcgzjjrj3fruhl2z2vq6
full_hash: e7kp4ragnx2k7fz7gfckrhtuqgaufvc5
full_hash: cu3gas6whofmoemp36g4qj4xr5hpzzfr
full_hash: xlwj7aqluirzplat4koc3x4ihbs25vlb
full_hash: qfppn3u4swovlkdgydwsslcfbflqfrv7
full_hash: al4gdgbj7ma4aukg4a2j37mdz4pvnaik
full_hash: rcic4xkdpndjv2lja3y2oqmefjbgqek6
full_hash: ozozfu57iuxi6hzdbddqdva5c5fvjxcu
full_hash: j5iefqodml65n32mymuau2pd3ngstsmi
full_hash: pusjal4kweom63v4wcg6d6lb2m7dctrh
full_hash: tzsur4woxnjlrdz3urdcfwg3a3c4qsjp
full_hash: nlz25m53qas3gdkerbbnmn7el2pn2end
full_hash: dfsabpon5ujg2g2ts6o5w7j77jgsgqg2
full_hash: 3kzo4e3zoaguf6zatxpy36fyhbrxr4fe
full_hash: efht5eh5tp7a7lgp74nimrcj6vvxxxv2
full_hash: ee4jlhplhlrbyycu5t5mfmqfmldnvocl
Ping @scottwittenburg
Information on your system
- Spack: 7f2f2a0
- Python: 3.6.9
- Platform: linux-ubuntu18.04-zen2