_chunker.c seems to be allocating a fixed buffer of 10MB no matter the size of the input file. I believe this is the cause on one of my machines for some significant overhead / slowdown as any new pages allocated from the OS will get blanked first (I believe - I was just reading up on it). I'm still testing this though - as I haven't studied this code in detail, im unsure if the buf_size relates to anything else.

In many cases when the file cache is on, it's mostly files under 4096 bytes that are being processed (and changed files), and allocating a 10MB buffer for each one does not seem efficient - unless I have missed something ?

[edit] this could all be a red herring - I think the problem I'm having is related to the fact the cache folder is on the same drive, and the mmap'd file is causing a lot of hd seeking, which is slowing down the directory traversal.