Slow responses and out of memory issues #277
Description
Just updated to the latest master of lbrycrd a couple days ago, but have not been able to make it stable. Daemon will shutdown after a couple rpc calls to getclaimtrie and/or getblocktemplate, and all calls are generally very slow to execute, espescially getclaimtrie which takes from 60000-80000ms.
Daemon shuts down for apparently no good reason, but every now and then i see Error: Out of memory. Terminating. as last entry in log, sometimes its just nothing indicating that the daemon have shut down in the logs.
2019-05-19T13:16:38Z addSupportToQueues: nValidAtHeight: 570138
2019-05-19T13:16:38Z UpdateTip: new best=d8cf51c89206d1ff70066c4b05074ac2174486bd0c65a0362a796aa8f91ad5ce height=570138 version=0x20000000 log2_work=72.799238 tx=4053388 date='2019-05-19T13:15:24Z' progress=0.989165 cache=120.5MiB(550771txo)
2019-05-19T13:16:38Z [default wallet] AddToWallet 922459c27b7a3cd0ab02a7fd6c943a873ab03b020154fb4368409f0d1f4fd485 update
2019-05-19T13:16:42Z Error: Out of memory. Terminating.
Sometimes getblocktemplate response is error -1: createnewblock: testblockvalidity failed: bad-claim-merkle-hash (code 16) which usually is gone within a couple seconds.
build: LBRYcrd Core version v0.17.1.0-55f5f2049 (release build)
os: Ubuntu 18.04.1 LTS
Available RAM is above 40GB, Server load is less than 1.
2019-05-19T13:20:29Z LBRYcrd Core version v0.17.1.0-55f5f2049 (release build)
2019-05-19T13:20:29Z InitParameterInteraction: parameter interaction: -whitelistforcerelay=1 -> setting -whitelistrelay=1
2019-05-19T13:20:29Z Assuming ancestors of block a6bbb48f5343eb9b0287c22f3ea8b29f36cf10794a37f8a925a894d6f4519913 have valid signatures.
2019-05-19T13:20:29Z Setting nMinimumChainWork=000000000000000000000000000000000000000000000000607ca7e806c4c1e9
2019-05-19T13:20:29Z Using the 'sse4(1way),sse41(4way),avx2(8way)' SHA256 implementation
2019-05-19T13:20:29Z Using RdRand as an additional entropy source
2019-05-19T13:20:29Z Default data directory /home/lbry/.lbrycrd
2019-05-19T13:20:29Z Using data directory /home/lbry/.lbrycrd
2019-05-19T13:20:29Z Using config file /home/lbry/.lbrycrd/lbrycrd.conf
2019-05-19T13:20:29Z Using at most 125 automatic connections (1024 file descriptors available)
2019-05-19T13:20:29Z Using 16 MiB out of 32/2 requested for signature cache, able to store 524288 elements
2019-05-19T13:20:29Z Using 16 MiB out of 32/2 requested for script execution cache, able to store 524288 elements
2019-05-19T13:20:29Z Using 12 threads for script verification
2019-05-19T13:20:29Z scheduler thread start
2019-05-19T13:20:29Z Binding RPC on address 0.0.0.0 port 23301 failed.
2019-05-19T13:20:29Z HTTP: creating work queue of depth 16
2019-05-19T13:20:29Z Config options rpcuser and rpcpassword will soon be deprecated. Locally-run instances may remove rpcuser to use cookie-based auth, or may be replaced with rpcauth. Please see share/rpcauth for rpcauth auth generation.
2019-05-19T13:20:29Z HTTP: starting 12 worker threads
2019-05-19T13:20:29Z Using wallet directory /home/lbry/.lbrycrd
2019-05-19T13:20:29Z init message: Verifying wallet(s)...
2019-05-19T13:20:29Z Using BerkeleyDB version Berkeley DB 4.8.30: (April 9, 2010)
2019-05-19T13:20:29Z Using wallet wallet.dat
2019-05-19T13:20:29Z BerkeleyEnvironment::Open: LogDir=/home/lbry/.lbrycrd/database ErrorFile=/home/lbry/.lbrycrd/db.log
For now i have switched back to release v0.12.4.0 which is stable.