Adds publishing blocks and transactions over ZMQ #4594

jmcorgan · 2014-07-27T21:05:59Z

Block and Transaction Broadcasting With ZeroMQ

ZeroMQ is a lightweight wrapper around TCP
connections, inter-process communications, and shared-memory,
providing various message-oriented semantics such as publish/subcribe,
request/reply, and push/pull.

The Bitcoin Core daemon can be configured to act as a trusted "border
router", implementing the bitcoin wire protocol and relay, making
consensus decisions, maintaining the local blockchain database,
broadcasting locally generated transactions into the network, and
providing a queryable RPC interface to interact on a polled basis for
requesting blockchain related data. However, there exists only a
limited service to notify external software of events like the arrival
of new blocks or transactions.

The ZeroMQ facility binds a "publish" port that broadcasts all newly
arrived and validated blocks and transactions to one or more
connected subscribers. This read-only facility requires only the
connection of a corresponding ZeroMQ subscriber port in receiving
software; it is not authenticated nor is there any two-way protocol
involvement.

In this fashion, external software can use a trusted Bitcoin Core
daemon to do the heavy lifting of communicating with the P2P network,
while still receiving network information asynchronously in real
time. As transactions and blocks arrive via the P2P network, Bitcoin
Core will apply all the validation and standardization rules to this
data, only passing along through ZeroMQ those items that pass.

ZeroMQ sockets are self-connecting and self-healing; that is, connects
made between two endpoints will be automatically restored after an
outage, and either end may be freely started or stopped in any order.

Because ZeroMQ is message oriented, subscribers receive transactions
and blocks all-at-once and do not need to implement any sort of
buffering or reassembly.

Prerequisites

The ZeroMQ feature in Bitcoin Core uses only a very small part of the
ZeroMQ C API, and is thus compatible with any version of ZeroMQ
from 2.1 onward, including all versions in the 3.x and 4.x release
series. Typically, it is packaged by distributions as something like
libzmq-dev.

The C++ wrapper for ZeroMQ is not needed.

Enabling

By default, the ZeroMQ port functionality is disabled. Two steps are
required to enable--compiling in the ZeroMQ code, and configuring
runtime operation on the command-line or configuration file.

$ ./configure --enable-zmq (other options)

This will produce a binary that is capable of providing the ZeroMQ
facility, but will not do so until also configured properly.

Configuration

Currently, the ZeroMQ facility only needs to have the ZeroMQ endpoint
specified:

$ bitcoind -zmqpub=tcp://127.0.0.1/28332

This will cause bitcoind to establish a PUB listening socket at the
specified host address and port number. The endpoint specifier may
also be provided as the equivalent line item in bitcoin.conf.

ZeroMQ endpoint specifiers for TCP (and others) are documented in the
ZeroMQ API.

Operation

ZeroMQ publish sockets prepend each data item with an arbitrary topic
prefix that allows subscriber clients to request only those items with
a matching prefix. When publishing, bitcoind will prepend the topic
"TXN" (no quotes, no null terminator) to the binary, serialized form
of a published transaction, and "BLK" to the binary, serialized form
of a published block.

Client side, then, the ZeroMQ subscriber socket must have the ZMQ_SUBSCRIBE option set to one or either of these prefixes; without doing so will result in no messages arriving.

Here is a small example, in the Python language, using the python-zmq wrapper:

import zmq
import binascii

port = 28332
topic1 = "BLK"
topic2 = "TXN"
topic_len = len(topic1)

zmqContext = zmq.Context()
zmqSubSocket = zmqContext.socket(zmq.SUB)
zmqSubSocket.setsockopt(zmq.SUBSCRIBE, topic1)
zmqSubSocket.setsockopt(zmq.SUBSCRIBE, topic2)
zmqSubSocket.connect("tcp://127.0.0.1:%i" % port)

def handleBLK(blk):
    print "-BLKHDR-"
    print binascii.hexlify(blk[:80])

def handleTX(tx):
    print "-TX-"
    print binascii.hexlify(tx)

try:
    while True:
        msg = zmqSubSocket.recv()
        msg_topic = msg[:topic_len]
        msg_data  = msg[topic_len:]

        if msg_topic == "TXN":
            handleTX(msg_data)
        elif msg_topic == "BLK":
            handleBLK(msg_data)

except KeyboardInterrupt:
    zmqContext.destroy()

This example is provided in the contrib/zmq directory of the source code.

Security Considerations

From the perspective of bitcoind, the ZeroMQ socket is write-only; PUB sockets don't even have a read function. Thus, there is no state introduced into bitcoind directly. Furthermore, no information is broadcast that wasn't already received from the public P2P network.

No authentication or authorization is done on connecting clients; it is assumed that the ZeroMQ port is exposed only to trusted entities, using other means such as firewalling.

Transactions and blocks are broadcast in their serialized form directly as received and validated by bitcoind. External software may assume that these have passed all validity/consensus/standard checks, but of course is free to perform these functions again in part or in whole.

jmcorgan · 2014-07-27T21:23:58Z

Pull tester unhappy, but merge, default configure, compile, and make check all pass ok here.

laanwj · 2014-07-28T07:13:19Z

src/main.cpp

No whitespace changes please, they're going to be solved in one go at some point with clang-format (see #4498).

Ok. My emacs is configured to automatically remove trailing whitespace, so this was unintended. I'll figure out a way to make a clean diff.

laanwj · 2014-07-28T07:16:42Z

The ZeroMQ feature in Bitcoin Core uses only a very small part of the
ZeroMQ C API, and is thus compatible with any version of ZeroMQ
from 2.1 onward

Thanks for that. For obelisk I had to find so many wrappers, high level wrappers, c++ wrappers for high-level wrappers (only some of them with an ubuntu package, which ofc was too old) that at some point I gave up.

laanwj · 2014-07-28T07:24:32Z

src/zmqports.cpp

Shouldn't this be its own category, zmq?

I'll change this.

laanwj · 2014-07-28T07:30:45Z

Looks good to me apart from the nits -- async notification is superior to polling. Only minimal impact on the current code and is a no-op if --enable-zmq isn't passed to configure.

jmcorgan · 2014-07-28T15:06:47Z

Per feedback, updated and rebased commit:

Removed unintended whitespace changes
Moved contrib/zmq/README.md to doc/zmq.md
New logging category 'zmq' created
Cleaned up logging

jgarzik · 2014-07-28T15:20:31Z

Concept ACK. Supporting ZeroMQ in this fashion (publish public data) is a useful feature.

One of the problems the Linux kernel ran into early on was a profusion of #ifdef'd features. Bitcoin Core does not have this problem, but it is creeping in that direction.

As mentioned in another github issue days ago, conditionally compiled code is evil. Such code is only compiled by a few, rather than all. As a result, the code quickly breaks because it is not used by the primary developers.

The solution is to create wrappers for the #ifndef MY_FEATURE branch in your my_feature.h, which creates an API that becomes a no-op at compile time. That way, zeromq code is always built, without ifdefs in the main codebase. As such, zeromq calls are built by all, and less likely to be broken.

jmcorgan · 2014-07-28T15:28:27Z

I intentionally made this such that you have to explicitly --enable-zmq in order to compile it in, for testing purposes. If this gets merged, at some point we'll decide to enable it for everyone. At that point, I'll rework things so that there are no ifdef's--instead, if libzmq-dev is not detected at configure time, the function calls become NOPs internally.

jgarzik · 2014-07-28T15:30:24Z

There should be no need to disable zmq by default at compile time. Disabled at runtime is sufficient.

Typical pattern is for configure to find a library and enable the building of that feature automatically, without --enable-foo support. You don't want to make the user work harder, just to get new features.

Also, I'm in general strongly against doesn't-compile-by-default code. It's a recipe for bitrot.

jmcorgan · 2014-07-28T15:31:56Z

I'll go ahead then and rework this to eliminate the --enable-zmq and disable the feature at runtime if libzmq-dev is not detected.

jmcorgan · 2014-07-28T15:43:22Z

Looks like I missed a logging simplification; I'll get this when I make the above changes.

laanwj · 2014-07-28T15:43:32Z

I strongly disagree with @jgarzik here. I don't want a fixed dependency on zeromq.

jmcorgan · 2014-07-28T15:45:15Z

To be clear--my plan is to make it such that if ZeroMQ is missing, the feature is disabled, so there won't be a dependency. Did I misunderstand?

laanwj · 2014-07-28T15:46:41Z

That's fine with me. Defaulting to having it enabled when the library is installed sounds sane. And it should still be possible to override the autodetection with --with[out]-zeromq, IMO.

sipa · 2014-07-28T15:48:26Z

@jgarzik dislikes conditionally compiled code, and wants to have the ZeroMQ code always compiled (thus always having the dependency).

@laanwj dislikes dependencies, and thus prefers the conditional compilation you're doing now.

I agree with @laanwj here. I really want to minimize dependencies. Another way of doing it is exposing the signals interface for such notifications, and register them from an optionally compiled object file (the same could be done fo -blocknotify, etc). That avoid #ifdefs entirely.

jmcorgan · 2014-07-28T22:13:43Z

Ok, this has been reworked as follows:

ZMQ features are automatically implemented if libzmq-dev is found, and a warning issued if not found
One can disable the ZMQ features with --disable-zmq
All conditional code is isolated to zmqports.cpp; code becomes NOP if zmq is disabled.
A new RPC call, 'getzmqurl', is provided to return the configuration option -zmqpub text.

jmcorgan · 2014-07-31T05:22:05Z

Minor rebase to accommodate 'rawtx' merge.

jtimon · 2014-07-31T13:00:20Z

I didn't test it but I like what it does.

jmcorgan · 2014-08-12T08:55:18Z

Minor rebase to accommodate RPC help categorization.

randy-waterhouse · 2014-08-17T23:12:51Z

Suggest minor change to add check for libzmq version >=2.1 requirement.

PKG_CHECK_MODULES([ZMQ],[libzmq >= 2.1],
[AC_DEFINE([ENABLE_ZMQ],[1],[Define to 1 to enable ZMQ functions])],
[AC_DEFINE([ENABLE_ZMQ],[0],[Define to 1 to enable ZMQ functions])
AC_MSG_WARN([libzmq >=2.1 not found, disabling])

ZMQ-based features are encapsulated in zmqports.{cpp,h} and disabled if libzmq-dev is not found, or configure is invoked with --disable-zmq. Use -zmqpub=<endpoint> to use feature. See contrib/zmq/README.md for details.

jmcorgan · 2014-08-18T13:50:43Z

Rebase and add zmq version check as requested.

BitcoinPullTester · 2014-08-18T18:48:32Z

Automatic sanity-testing: FAILED BUILD/TEST, see http://jenkins.bluematt.me/pull-tester/p4594_3c9d0c12ac2476fc8fb5de9293edbf27081b1d4e/ for binaries and test log.

This could happen for one of several reasons:

It chanages paths in makefile.linux-mingw or otherwise changes build scripts in a way that made them incompatible with the automated testing scripts (please tweak those patches in qa/pull-tester)
It adds/modifies tests which test network rules (thanks for doing that), which conflicts with a patch applied at test time
It does not build on either Linux i386 or Win32 (via MinGW cross compile)
The test suite fails on either Linux i386 or Win32
The block test-cases failed (lookup the first bNN identifier which failed in https://github.com/TheBlueMatt/test-scripts/blob/master/FullBlockTestGenerator.java)

If you believe this to be in error, please ping BlueMatt on freenode or TheBlueMatt here.

This test script verifies pulls every time they are updated. It, however, dies sometimes and fails to test properly. If you are waiting on a test, please check timestamps to verify that the test.log is moving at http://jenkins.bluematt.me/pull-tester/current/
Contact BlueMatt on freenode if something looks broken.

jgarzik · 2014-08-30T00:18:23Z

PR #4599 (commit c7b6117) which was just merged illustrates a useful plugin approach. Stick one or more signals in the needed locations, and then ZMQ or other features add their own hooks.

jgarzik · 2014-09-15T02:22:13Z

To make the last comment more explicit, I do not like seeing "zmq" in main.cpp or net.cpp.

This pull request has my ACK iff the ZMQPublish*() calls are turned into ui_interface.h signals, as illustrated in c7b6117. No objections other than that.

jmcorgan · 2014-09-18T13:07:26Z

Thanks for the clarification. It will be a week or two until I can turn my attention back to this, but I'll look over the signaling mechanism and see how I can refit this patch to use that.

btcdrak · 2014-11-07T16:42:32Z

needs rebased

jgarzik · 2014-11-07T17:12:20Z

I'm tempted to take this patch, add signals, and re-PR. ZMQ is conceptually acceptable AFAIK.

btcdrak · 2014-11-07T17:17:24Z

@jgarzik do it!

theuni · 2014-11-07T17:54:11Z

@jgarzik If you do, please take theuni@dd8d00b and theuni@25844d3 (feel free to squash) so that travis can build. Those are quite old though, let me know if there are merge problems and I'll rebase.

jmcorgan · 2014-11-07T18:23:08Z

@jgarzik Go for it. I haven't had much time to spend on bitcoin development these last few months, but I'd still like to see this get integrated.

laanwj · 2014-11-19T11:32:56Z

Closing in favor of #5303.

laanwj reviewed Jul 28, 2014
View reviewed changes

jmcorgan mentioned this pull request Jul 30, 2014

Adds transaction index by address (updates #2802 to current master) #3652

Closed

laanwj added the RPC label Jul 31, 2014

jmcorgan added 2 commits August 18, 2014 06:42

Adds publishing blocks and transactions over ZMQ

3658e93

ZMQ-based features are encapsulated in zmqports.{cpp,h} and disabled if libzmq-dev is not found, or configure is invoked with --disable-zmq. Use -zmqpub=<endpoint> to use feature. See contrib/zmq/README.md for details.

zmq: require version >= 2.1

3c9d0c1

jgarzik mentioned this pull request Nov 18, 2014

Add ZMQ support. Report blocks and transactions via ZMQ. #5303

Closed

laanwj closed this Jan 8, 2015

promag mentioned this pull request May 4, 2015

Add ZeroMQ notifications #6103

Merged

jmcorgan deleted the zmq branch September 19, 2015 06:10

bitcoin locked as resolved and limited conversation to collaborators Sep 8, 2021

Adds publishing blocks and transactions over ZMQ #4594

Adds publishing blocks and transactions over ZMQ #4594

Uh oh!

Conversation

jmcorgan commented Jul 27, 2014

Block and Transaction Broadcasting With ZeroMQ

Prerequisites

Enabling

Configuration

Operation

Security Considerations

Uh oh!

jmcorgan commented Jul 27, 2014

Uh oh!

laanwj Jul 28, 2014

Choose a reason for hiding this comment

Uh oh!

jmcorgan Jul 28, 2014

Choose a reason for hiding this comment

Uh oh!

laanwj commented Jul 28, 2014

Uh oh!

laanwj Jul 28, 2014

Choose a reason for hiding this comment

Uh oh!

jmcorgan Jul 28, 2014

Choose a reason for hiding this comment

Uh oh!

laanwj commented Jul 28, 2014

Uh oh!

jmcorgan commented Jul 28, 2014

Uh oh!

jgarzik commented Jul 28, 2014

Uh oh!

jmcorgan commented Jul 28, 2014

Uh oh!

jgarzik commented Jul 28, 2014

Uh oh!

jmcorgan commented Jul 28, 2014

Uh oh!

jmcorgan commented Jul 28, 2014

Uh oh!

laanwj commented Jul 28, 2014

Uh oh!

jmcorgan commented Jul 28, 2014

Uh oh!

laanwj commented Jul 28, 2014

Uh oh!

sipa commented Jul 28, 2014

Uh oh!

jmcorgan commented Jul 28, 2014

Uh oh!

jmcorgan commented Jul 31, 2014

Uh oh!

jtimon commented Jul 31, 2014

Uh oh!

jmcorgan commented Aug 12, 2014

Uh oh!

randy-waterhouse commented Aug 17, 2014

Uh oh!

jmcorgan commented Aug 18, 2014

Uh oh!

BitcoinPullTester commented Aug 18, 2014

Uh oh!

jgarzik commented Aug 30, 2014

Uh oh!

jgarzik commented Sep 15, 2014

Uh oh!

jmcorgan commented Sep 18, 2014

Uh oh!

btcdrak commented Nov 7, 2014

Uh oh!

jgarzik commented Nov 7, 2014

Uh oh!

btcdrak commented Nov 7, 2014

Uh oh!

theuni commented Nov 7, 2014

Uh oh!

jmcorgan commented Nov 7, 2014

Uh oh!