[WIP] rest: Stream entire utxo set #7759
Conversation
|
Looks like |
Weird. What good is a chunk function if you have no clue if the data was sent. I'll take a look. |
laanwj
force-pushed
the
2016_03_utxo_streaming
branch
from
b307590 to
073948d
Compare
|
It seems the last stable release of libevent (2.0.22) was 2.5 years, though the master branch is being updated still. Do we want for libevent 2.1 for this, or find another way? |
Yes, libevent version management is like that, unfortunately. I'd tend to include a newer libevent in depends, then disable the functionality when building with older libevent. This will be too late for 0.13. Let's hope there will be a stable libevent release again before 0.14 which includes this. Not holding my breath though. |
This builds on bitcoin#7756 and - Adds a streaming API to the HTTP server. This allows streaming data to the client chunk by chunk, which is useful when not the entire data is available at once or it is huge and wouldn't fit (efficiently) in memory. - Allows downloading the entire UTXO set through `/rest/utxoset`. This is a raw dump of all outputs, the state normally hashed by `gettxoutsetinfo`. The dump is performed in the background by making use of leveldb snapshotting, so without keeping cs_main locked. - This can be useful for analysis purposes if you don't want to mess with bitcoin core's database - Filename (via content-disposition) is `utxoset-<height>-<bestblockhash>.dat`. Also a custom `X-Best-Block` and `X-Block-Height` header is added.
laanwj
force-pushed
the
2016_03_utxo_streaming
branch
from
073948d to
2498324
Compare
|
Rebased, updated for boost-removal from |
| // LogPrintf("set_http_chunk_cb\n"); | ||
| { | ||
| std::unique_lock<std::mutex> lock(cs); | ||
| evhttp_send_reply_chunk_with_cb(req, databuf, &http_chunk_cb, this); |
There was a problem hiding this comment.
I guess this requires libevent2.1 (depends package is still on 2.0.x IIRC)
There was a problem hiding this comment.
Sorry.. that was already discussed.
There was a problem hiding this comment.
Yes, I need to find exactly what version this was introduce in and guard the streaming stuff with #if LIBEVENT_VERSION_NUMBER >= 0x0201XXXX. I think it can only be supported for newer libevent .
There was a problem hiding this comment.
For reference, the commit that introduced evhttp_send_reply_chunk_with_cb libevent/libevent@8d8decf , first appearing in version 0x02010401 is from 2009, and that's still the beta branch. I'm getting a bit concerned about libevent's release process.
There was a problem hiding this comment.
RE libevent release process -- several projects are feeling the limits of libevent http support, and moving to https://github.com/ellzey/libevhtp
I had to do that in one project, in order to support streaming chunked http downloads.
libevent's http was really written for simple app servers with small replies.
There was a problem hiding this comment.
I know of that project, but I'd prefer to avoid adding another dependency. It can be considered if there is really no other way out, but it seems to me that chunked downloads can be done with that function.
This needs a better interface so that HTTPServer's users (such as rest) can query capabilities.
laanwj
force-pushed
the
2016_03_utxo_streaming
branch
from
1bc34c2 to
5cd59bb
Compare
|
Closing this. I think it was a nice experiement but I don't expect to get around to it again in the near future. |
|
:(
|
|
For anyone thinking about picking this up: The good news here is that libevent 2.1 is out of alpha, and is stable as of 2.1.7 at this moment. The bad news is that it might be impossible to stream reliably using libevent's http server. At least it's mentioned as one of the motivations for the libevhtp replacement:
So I'm not sure "The timeout actually runs while downloading, causing it to break off after downloading." mentioned in the OP is solvable. It might be with some hack. |
|
I will be happy to assist with any migration issues to libevhtp if this project decides to go this route. Cheers! |
8 participants
This is by no means ready, but anyhow this builds on #7756 and
/rest/utxoset. This is a raw dump of all outputs, the state normally hashed bygettxoutsetinfo. The dump is performed in the background by making use of leveldb snapshotting, so without keeping cs_main locked.utxoset-<height>-<bestblockhash>.dat. Also a customX-Best-BlockandX-Block-Heightheader is added.It matches:
$ src/bitcoin-cli -datadir=/store/tmp/testbtc gettxoutsetinfo { ... "hash_serialized": "5017f82bbb82a8199ae0fbaa9e5881a0c82575db89e6edd5b39414b35299363b", ... } $ wget --content-disposition http://127.0.0.1:8332/rest/utxoset 2016-03-28 22:58:32 (44.3 MB/s) - ‘utxoset-404681-0000000000000000034854f5a3ab27cfbc220a42c75061dd13d2067cda71191d.dat’ saved [1291578967] $ ~/bin/dsha256 utxoset-404681-0000000000000000034854f5a3ab27cfbc220a42c75061dd13d2067cda71191d.dat 5017f82bbb82a8199ae0fbaa9e5881a0c82575db89e6edd5b39414b35299363b utxosetTODO
Rebase after Add cursor to iterate over utxo set, use this ingettxoutsetinfo#7756 merged-rpcservertimeout=6000or such.UTXO set dump doesn't contain keys (?) I'm not sure this format is actually useful this way (see Merkleize the utxo set dump (was:(fixed in Divergence between 32- and 64-bit when hashing >4GB affectsgettxoutsetinfoserializedhash doesn't contain the keys?) #7758)gettxoutsetinfo#7848)Note that the HTTP streaming API could in principle also be used for other large data (say, wallet backups), or even for websocket-like event notification.