How does this work if the cache has 22,000+ entries? Does adding this many entries create a problem?

I'm using this branch to run spack buildcache list -l on our cache of 23k+ packages right now. The command has been running for more than 2 hours.... I will post final runtime and size information after it completes.

@eugeneswalker we could improve the performance of that command by doing a single write transaction with all of the specs to add, I'm more worried about the performance of spack find after you finish that command. I think that's where we'll really learn whether this is feasible, since that's just a single read, see if the database is too big for good performance.

I don't think the DB will necessarily be too big after this -- we already read the buildcache after buildcache list and that operation is pretty fast. I'm guessing (maybe many) seconds for a spack find if we do one write transaction.

spack buildcache list -l finished in 172 minutes

I ran it with debug output turned on. This was part of the final debug output:

$> time spack buildcache list -l
...
==> [2020-10-14-15:58:36.762000] 0 cached builds.
==> [2020-10-14-15:58:36.762128] You can query all available architectures with:
  spack buildcache list --allarch

real	172m39.813s
user	168m31.895s
sys	1m51.269s

I feel like something is off. 0 cached builds ??? What do you think?

$> time spack find
==> 15 installed packages
-- linux-rhel7-sandybridge / [email protected] --------------------------
[email protected]  [email protected]   [email protected]          [email protected]  [email protected]             [email protected]  [email protected]
[email protected]   [email protected]  [email protected]  [email protected]   [email protected]  rdma-core@20          [email protected]

-- linux-ubuntu18.04-ppc64le / [email protected] ------------------------
[email protected]

real	0m9.906s
user	0m7.470s
sys	0m0.475s

@eugeneswalker were any of the builds in the cache for the same architecture as the machine you ran it on? You could use the --allarch option to see them all, that would be my guess.

I've also pushed a commit that should improve the performance some. Can you run a time spack buildcache list -l again and see how long that takes? Or can you slack me the url for the mirror you're using and I can try this out? I'll also want to compare to timing on develop, so it might be easier if I don't have to bother you every time I change something.

Closing: we decided to solve this problem with a hash cache rather than adding to the database, because of scalability concerns with the json database.