The reason that we're not using WAL / small transactions is that (at least at the time) there was significant overhead to using them while populating the cache, especially for something like Nixpkgs.

Can you benchmark the performance of something like rm -rf ~/.cache/nix/eval-cache*; nix search nixpkgs hello with and without this change?

@edolstra

Yes, that is right indeed. While WAL does help a lot, transaction do appear to have some overhead:

Benchmark 1: ./baseline/bin/nix --extra-experimental-features 'nix-command flakes' search nixpkgs hello
  Time (mean ± σ):     19.053 s ±  0.300 s    [User: 17.738 s, System: 1.286 s]
  Range (min … max):   18.745 s … 19.667 s    10 runs
 
Benchmark 2: ./wal/bin/nix --extra-experimental-features 'nix-command flakes' search nixpkgs hello
  Time (mean ± σ):     30.813 s ±  2.847 s    [User: 24.298 s, System: 4.352 s]
  Range (min … max):   26.250 s … 35.306 s    10 runs
 
Benchmark 3: ./no-wal/bin/nix --extra-experimental-features 'nix-command flakes' search nixpkgs hello
  Time (mean ± σ):     40.453 s ±  0.851 s    [User: 26.327 s, System: 13.886 s]
  Range (min … max):   39.018 s … 41.679 s    10 runs
 
Summary
  ./baseline/bin/nix --extra-experimental-features 'nix-command flakes' search nixpkgs hello ran
    1.62 ± 0.15 times faster than ./wal/bin/nix --extra-experimental-features 'nix-command flakes' search nixpkgs hello
    2.12 ± 0.06 times faster than ./no-wal/bin/nix --extra-experimental-features 'nix-command flakes' search nixpkgs hello

But then, we cannot reasonably keep a write lock on the database for the whole duration of the nix search command.

Would you consider a patch that performs only reads, and then writes all the computed data at once, when the cursors are destroyed ? It would still be efficient, and more friendly to concurrent accesses to the database.

@layus Yeah that sounds good.

An alternative might be (if possible) to start with a read lock and only upgrade to a write lock if we actually need to make changes to the database. Or to disable the eval cache if we can't acquire a write lock. Then it would be slower but at least it wouldn't be blocked.