Have you tested that this actually works? I've been playing for a few hours now with DB_LOG_REMOVE, DB_LOG_AUTO_REMOVE, closing all db files occasionally, checkpoinging, flushing, stopping and restarting DbEnv, ... but all I've seen are a lot of unexplainable segfauls, and not a single deleted logfile.

EDIT: apparently it does. I'm baffled.

Nice! I'm baffled, too.

I don't think this should be a command-line option, though: I think it should just happen, I can't think of any reason to keep gigabytes of logs around.

@sipa: I tested it a lot before making the pull request, yes. It seems to work well in db5.1.25. I don't have db4.x.

@gavinandresen: I think the logs are so you can do database recovery in the event of a crash.

It seems that in the event of a crash we usually manage to clean up the logs before exiting however :)

I'll update my patch so it's not a command line option.

I'm testing a bit more extensive patch that counts the number of transactions updated or time since last flush as condition for flushing (instead of 5000 blocks), plus a few other changes. I'm benchmarking now.

Thanks for showing this was possible at all - I had almost given up (seems there was a bug in my load-blocks-from-file patch that didn't call txn_checkpoint at all, which prevented log cleaning).

No problem. I don't care what triggers the deletion so long as it's done occasionally.

To arrange for rapid blockchain downloading to test this I've been using a second computer on the same LAN with a complete copy of the blockchain and connecting only to it.

Ok, one full load of the blockchain up to 172000 (from a local file) completed in 35 minutes. That's a few minutes slower than a previous benchmark, but it never needed more than 12 log files (of 10 MiB again). This did a log flush every 200k transactions (which was every 0.5-2 minutes). I'll post the code tomorrow.

How much of that was spent loading the blocks 168000 to 172000? The last checkpoint being at 168k makes the last 4k blocks so much slower to load.

How about adding a new checkpoint before the release of 0.6.0?

grrr...

So after WASTING time testing dbenv.set_flags(DB_LOG_AUTO_REMOVE, 1) this afternoon, I discover the correct call is actually log_set_config:

dbenv.log_set_config(DB_LOG_AUTO_REMOVE, 1);

Testing with a 10MB logfile size and the logs are automatically being cleaned up as the blockchain is downloading.

(the set_flags call didn't trigger an error or warning because LOG_AUTO_REMOVE and AUTO_COMMIT share the same #define)

So there are three options now: a) remove logs at given points b) flush + remove logs at given points c) use auto remove.

I wonder if there is any danger in using auto remove (meaning logs will be removed while transactions/databases are open). I suppose not.

Gavin, how many log files were present at most at one given time during the block chain download?

The most log files present at one given time with LOG_AUTO_REMOVE and 10M logfiles was 5.

Typical is 2.

Raw data (ls -l in the database/ dir run every 2 minutes): https://gist.github.com/2236954

Thanks for the help Chris-- without this pull I think we would have given up hope of finding a good solution. But I'm going to close, because using AUTO_REMOVE (properly) is fewer lines of code.