bigchaindb version: v2.0.0a2
tendermint version: 0.12
setup: 4 node network with docker-compose

Description

Tendermint polls to bigchaindb pyabci interface only on startup. So if at any point bigchaindb service crashes and restarts tendermint never tries to re-establish connection with bigchaindb, and bigchaindb in absence of a connection request form tendermint crashes again.

This tight coupling between tendermint and bigchaindb services requires them to be packaged together as a single unit to improve the crash recovery mechanism.

Solution

• package tendermint binary as part of bigchaindb python package. This binary will be installed as part of bigchaindb package. (This needs to be scoped yet)
• Add supervisord configs for spawning bigchaindb and tendermint services together and restart both in the expected order if one crashes. Supervisord will allow similar behavior across platforms and can be packaged with un-privileged docker containers as compared to other process managers like systemd which are distribution specific and can only work with privileged containers.