Skip to content

Conversation

@sipa
Copy link
Member

@sipa sipa commented Jan 11, 2014

Apparently (on 64-bit systems), an average CBlock takes up about 8 times as much space as its serialized form. This results in high memory usage for all orphan blocks that are kept. Change this to retain them in serialized form instead.

Tested by doing a (partial) sync under valgrind, which involved processing orphans.

@BitcoinPullTester
Copy link

Automatic sanity-testing: PASSED, see http://jenkins.bluematt.me/pull-tester/da0fecffa788a0a74121d554d3d76936ab96b39e for binaries and test log.
This test script verifies pulls every time they are updated. It, however, dies sometimes and fails to test properly. If you are waiting on a test, please check timestamps to verify that the test.log is moving at http://jenkins.bluematt.me/pull-tester/current/
Contact BlueMatt on freenode if something looks broken.

@laanwj
Copy link
Member

laanwj commented Jan 12, 2014

ACK on code changes
(haven't tested memory usage, but looks like a good idea)
Currently trying a full sync...

@sipa
Copy link
Member Author

sipa commented Jan 12, 2014

I have a branch (serorphans) which is a merge of #3514 with this; it should provide some stronger memory usage guarantees, as it also limits the maximum number of orphans in memory.

@laanwj
Copy link
Member

laanwj commented Jan 12, 2014

My client appears to be stuck at block 202978 on the mainnet chain
For some reason I'm receiving the entire chain starting from 0000000000000062e33609a6653b870f76f3f61e6f0aa3d9100ad12d78926a79 (#241041) to #241869 as ORPHAN BLOCK
Could of course be unrelated to this pull and simply a case of bad peers.

Oh... and it's already syncing again :)

(still lots and lots of orphans though, if you're interested in the debug.log, it's here http://download.visucore.com/bitcoin/debug-3516.log.bz2)

@sipa
Copy link
Member Author

sipa commented Jan 12, 2014

No, receiving orphans is very expected and one of the reason why I'm working on headers-first.

The problem is that any peer that announces some blocks, we ask for it, and if it's something whose parent we don't know, we ask for the parents - starting a pretty much full sync process from this peer, even if we were already syncing from another before. The problem is just that if you're syncing from two peers at once, some ranges of blocks are bound to be downloaded in incorrect order.

It's certainly possible to improve this, and try to prevent starting a second sync from another peer if another is ongoing, but this sounds hard to do right and guarantee to deal well with all weird edge cases that can arise.

@laanwj
Copy link
Member

laanwj commented Jan 12, 2014

Indeed. The amount of received orphans shouldn't be affected by these changes. So all the orphans are good for testing, in this case.

@laanwj
Copy link
Member

laanwj commented Jan 13, 2014

I did a succesful full sync with this patch.
The orphans picked up around #202978 were successfully written when it actually came to block #241041.
ACK

@gavinandresen
Copy link
Contributor

ACK (reviewed changes and compiled/sanity tested on OSX).

gavinandresen added a commit that referenced this pull request Jan 13, 2014
Store orphan blocks in serialized form
@gavinandresen gavinandresen merged commit 266921e into bitcoin:master Jan 13, 2014
@bitcoin bitcoin locked as resolved and limited conversation to collaborators Sep 8, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants