Problem: BigchainDB and Tendermint tightly coupled and need some supervision by muawiakh · Pull Request #2410 · bigchaindb/bigchaindb

muawiakh · 2018-07-25T15:28:46Z

Problem

BigchainDB and Tendermint are very tightly coupled which implies that if the connection(ABCI) between BigchainDB and Tendermint breaks, the system halts i.e. Tendermint does not re-iniate the connection and both processes keep running without being productive.

Solution

Supervise both the services/processes with a process manager. For now using Monit. Monit will launch both BigchainDB and Tendermint and will monitor the lifecycle of both services.

Workflow

When BigchainDB is installed e.g.:

$ git clone https://github.com/bigchaindb/bigchaindb.git
$ cd bigchaindb/
# might need sudo or python3, details not important here
$ python setup.py install

OR
$ pip install bigchaindb

It will install a script bigchaindb-monit-config, which is placed under /usr/local/bin and can be intiated using:

$ bigchaindb-monit-config

For more options:

$ bigchaindb-monit-config -h

    Usage: bigchaindb-monit-config [-h]

    Configure Monit for BigchainDB and Tendermint process management.

    ENV[MONIT_PID_PATH] || --monit-pid-path PATH

    Absolute path to directory where the the program's pid-file will reside.
    The pid-file contains the ID(s) of the process(es).

    ENV[MONIT_SCRIPT_PATH] || --monit-script-path PATH

    Absolute path to the directory where the executable program or
    script is present.

    ENV[MONIT_LOG_PATH] || --monit-log-path PATH

    Absolute path to the directory where all the logs for processes
    monitored by Monit are stored.

    ENV[MONIT_EXEC_PATH] || --monit-exec-path PATH

    Absolute path to the directory to run the script form.

    -h|--help
        Show this help and exit.

Once the above script is executed successfully with the message BigchainDB process manager configured!. User can use monit to monitor both processes at the same time i.e. if one of them crashes or the connection breaks, monit will restart them.

$ monit -d 1

# For more detailed info about monit

$ monit --help

Issues Resolved

Resolves #2238

Dependencies

Monit

Remaining TODOs

The default path for .monitrc Monit control file is $HOME/.monitrc, we don't want to override it if someone is already using it with monit. So change the current script to become more interactive about .monitrc file. Something similar to ssh-keygen
We send monit logs to $HOME/.bigchaindb-monit/logs/<bigchaindb-logs> but at the same time the BigchainDB process, generates log files at $HOME/<bigchaindb-logs>(default path). There should only be one location for this.
In help, print the default values
Cleanup some TBD items in the script.

- Fix config utils log info, previously misleading even if .bigchaindb file not present.

codecov-io · 2018-07-25T15:40:33Z

Codecov Report

Merging #2410 into master will decrease coverage by 0.06%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #2410      +/-   ##
==========================================
- Coverage    86.9%   86.83%   -0.07%     
==========================================
  Files          38       38              
  Lines        2168     2173       +5     
==========================================
+ Hits         1884     1887       +3     
- Misses        284      286       +2

kansi

The issue description has nice documentation about how process monitoring is achieved. It would be really helpful if the same could be included in the docs.

ldmberman · 2018-07-26T08:55:37Z

It would be really helpful if the same could be included in the docs.

@kansi could you, please, suggest a section to update? I think, it would not be equally useful everywhere. Also, depending on the status quo it might be difficult to introduce a new section properly without a good idea of what else will go there.

ttmc · 2018-07-26T09:23:02Z

Some docs to update or add:

http://docs.bigchaindb.com/projects/server/en/master/simple-network-setup.html
http://docs.bigchaindb.com/projects/contributing/en/latest/dev-setup-coding-and-contribution-process/run-node-as-processes.html
Maybe add a new page with more details in the Appendices of the BigchainDB Server docs.

Note: There's an open PR (#2388) to delete the old page in the Appendices about installing OS-level dependencies.

ldmberman · 2018-07-26T09:37:13Z

@ttmc I plan to update http://docs.bigchaindb.com/projects/server/en/master/simple-network-setup.html in the following PR when we ship a new release since the instructions depend on the binary that is to come with the new version.

Thank you for other hints though!

- Settling on default path of control file: $HOME/.monitrc - Handling overwriting of control file interactively - Save BigchainDB logs in the current directory i.e. from which BigchainDB is launched. - Print default values in help - Cleanup TBD items

kansi · 2018-07-26T12:26:38Z

bigchaindb/log.py



-DEFAULT_LOG_DIR = expanduser('~')
+DEFAULT_LOG_DIR = os.getcwd()


What will be the value of DEFAULT_LOG_DIR incase BigchainDB is installed via pip and when using it via docker?

It depends on where you run the command from, not on how and where you install it.

It will be derived from the directory where you are running bigchaindb start from and in Docker we set the WORKDIR to /usr/src/app, the logs will be generated there.

It would be good to make a note and mention such details in the doc, in this PR or a future PR whenever the related docs are added.

I will note down where the logs are stored in the Monit setup section.

ttmc · 2018-07-26T12:47:44Z

I will defer review until the docs explaining how to use this are written in a future pull request.

ldmberman · 2018-07-26T12:51:23Z

@ttmc giving it a second thought, what if we add the docs now, although the script they depend on will come a little later? It's only about a single section in the end..

ldmberman · 2018-07-26T14:04:00Z

pkg/scripts/bigchaindb-monit-config

+
+    ENV[MONIT_EXEC_PATH] || --monit-exec-path PATH
+
+    Absolute path to the directory to run the script form. (default: ${monit_exec_path})


This information is outdated. It is now "the path to the Monit control file". I would also use a different variable name for it, like monitrc_path.

Solution: Document how to manage processes using Monit.

ldmberman · 2018-07-26T14:56:15Z

@ttmc I checked http://docs.bigchaindb.com/projects/contributing/en/latest/dev-setup-coding-and-contribution-process/run-node-as-processes.html - the Monit setup does not really belong there because:

The following doc describes how to run a local node for developing BigchainDB Tendermint version.

I will just create a Monit setup section in "Appendices".

ldmberman · 2018-07-26T15:09:49Z

I will just create a Monit setup section in "Appendices".

Actually, I would prefer to only keep it inside the network setup guide to avoid maintaining it in two different places.

kansi

LGTM, 👍 for the docs.

ttmc · 2018-07-27T11:53:28Z

I will test the new docs soon, but I already have some basic questions about how this works. (I looked at the Monit setup code but I didn't find the answers there.)

What does Monit monitor/check, and what triggers it to restart BigchainDB and/or Tendermint? Will it always restart both?

ttmc · 2018-07-27T12:25:53Z

docs/server/source/simple-network-setup.md


-```
 # Change 2.0.0b3 to the latest version as explained above:
+```


When the line "# Change 2.0.0b3 to the latest version as explained above:" moved outside the code block, it changed from a comment into a top-level heading. Now it renders with big bold text and shows up in the overall top-level table of contents for the BigchainDB Server Docs:

Suggestion: Move that line back inside the code block.

ttmc · 2018-07-27T14:35:55Z

docs/server/source/simple-network-setup.md

+
+This section describes how to manage the BigchainDB and Tendermint processes using [Monit][monit] - a small open-source utility for managing and monitoring Unix processes.
+
+This section assumes that you followed the guide down to the [start MongoDB section](member-start-mongodb) inclusive.


The hyperlink on this line isn't working..

Forgot to test it, thank you for spotting it!

ttmc · 2018-07-27T14:45:02Z

docs/server/source/simple-network-setup.md

+
+Check the status by running `monit status` or `monit summary`.
+
+By default, it will collect program logs into the `~/.bigchaindb-monit/logs` folder.


I looked in some of those logs and was surprised to find that some of the "main" BigchainDB logs were not in bigchaindb.out.log but in bigchaind-benchmark.log, e.g.

~/.bigchaindb-monit/logs$ cat bigchaindb.out.log ======== Running on http://localhost:9985 ======== (Press CTRL+C to quit) ~/.bigchaindb-monit/logs$ cat bigchaindb-benchmark.log 2018-07-27 14:40:03, INFO, BigchainDB Version 2.0.0b3 2018-07-27 14:40:03, INFO, Initializing database 2018-07-27 14:40:03, INFO, Create database `bigchain`. 2018-07-27 14:40:03, INFO, Create `transactions` table. 2018-07-27 14:40:03, INFO, Create `utxos` table. 2018-07-27 14:40:03, INFO, Create `assets` table. 2018-07-27 14:40:03, INFO, Create `blocks` table. 2018-07-27 14:40:03, INFO, Create `metadata` table. 2018-07-27 14:40:03, INFO, Create `validators` table. 2018-07-27 14:40:03, INFO, Create `pre_commit` table. 2018-07-27 14:40:03, INFO, Create `transactions` secondary index. 2018-07-27 14:40:03, INFO, Create `assets` secondary index. 2018-07-27 14:40:03, INFO, Create `assets` secondary index. 2018-07-27 14:40:03, INFO, Create `utxos` secondary index. 2018-07-27 14:40:03, INFO, Create `pre_commit` secondary index. 2018-07-27 14:40:04, INFO, Create `validators` secondary index. 2018-07-27 14:40:04, INFO, Starting BigchainDB main process. 2018-07-27 14:40:04, INFO, **************************************************************************** * * * ┏┓ ╻┏━╸┏━╸╻ ╻┏━┓╻┏┓╻╺┳┓┏┓ ┏━┓ ┏━┓ ╺┳┓┏━╸╻ ╻ * * ┣┻┓┃┃╺┓┃ ┣━┫┣━┫┃┃┗┫ ┃┃┣┻┓ ┏━┛ ┃┃┃ ┃┃┣╸ ┃┏┛ * * ┗━┛╹┗━┛┗━╸╹ ╹╹ ╹╹╹ ╹╺┻┛┗━┛ ┗━╸╹┗━┛╹╺┻┛┗━╸┗┛ * * codename "fluffy cat" * * Initialization complete. BigchainDB Server is ready and waiting. * * * * You can send HTTP requests via the HTTP API documented in the * * BigchainDB Server docs at: * * https://bigchaindb.com/http-api * * * * Listening to client connections on: 0.0.0.0:9984 * * * **************************************************************************** 2018-07-27 14:40:04, INFO, ABCIServer started on port: 26658 2018-07-27 14:40:04, WARNING, WebSocket connection failed with exception Cannot connect to host localhost:26657 ssl:False [Connect call failed ('127.0.0.1', 26657)] 2018-07-27 14:40:06, INFO, ... connection from Tendermint: 127.0.0.1:60950 ... 2018-07-27 14:40:06, INFO, ... connection from Tendermint: 127.0.0.1:60952 ... 2018-07-27 14:40:06, INFO, ... connection from Tendermint: 127.0.0.1:60954 ... 2018-07-27 14:40:07, INFO, Connected to tendermint ws server

We did not change anything in this area. We do need to tidy the logging up a little in the future.

I will I create a new issue that links to this review comment and then we can consider this comment taken-care-of for this PR.

ldmberman · 2018-07-27T15:03:01Z

What does Monit monitor/check, and what triggers it to restart BigchainDB and/or Tendermint? Will it always restart both?

A crash of the BigchainDB root process will trigger a restart of both. A crash of the Tendermint process will only trigger a restart of Tendermint, what should be fine. Also, it does not try to bring Tendermint up while BigchainDB is not running.

muawiakh added 2 commits July 25, 2018 13:20

Adding supervisor(monit) for BigchainDB and Tendermint

755795b

Set default directories to $HOME/.bigchaindb-monit

bd6e6b7

- Fix config utils log info, previously misleading even if .bigchaindb file not present.

muawiakh added ⚠️ WIP ⚠️ DO NOT MERGE labels Jul 25, 2018

muawiakh requested a review from ldmberman July 25, 2018 15:28

remove some leftover debug statements

4b74e23

kansi suggested changes Jul 26, 2018

View reviewed changes

ldmberman requested review from shahbazn and ttmc July 26, 2018 09:05

muawiakh added 2 commits July 26, 2018 12:34

Remove some unwanted/pasted* code

3a05a92

muawiakh removed DO NOT MERGE ⚠️ WIP ⚠️ labels Jul 26, 2018

kansi reviewed Jul 26, 2018

View reviewed changes

Remove more redundant code

36877de

ttmc removed their request for review July 26, 2018 12:47

ldmberman reviewed Jul 26, 2018

View reviewed changes

ldmberman and others added 3 commits July 26, 2018 16:31

Problem: The setup guide offers a cumbersome way to manage processes.

0faba2e

Solution: Document how to manage processes using Monit.

change monit_exec_path to monitrc_path

c3e1745

fix some leftover changes from monit_exec_path -> monitrc_path

95f3c2d

kansi approved these changes Jul 27, 2018

View reviewed changes

ttmc suggested changes Jul 27, 2018

View reviewed changes

ttmc mentioned this pull request Jul 27, 2018

Problem: Example priv_validator.json is out-of-date in docs #2416

Merged

Fix some docs issues, hyperlinks and headings

d7034c8

ttmc mentioned this pull request Jul 27, 2018

Problem: "Main" BigchainDB logs being stored in bigchaindb-benchmark.log rather than bigchaindb.out.log? #2417

Closed

ttmc approved these changes Jul 27, 2018

View reviewed changes

ldmberman merged commit c3f5e2a into bigchaindb:master Jul 27, 2018

ldmberman mentioned this pull request Jul 30, 2018

[shutdown] Implement a stop function to stop all running processes #1397

Open



		DEFAULT_LOG_DIR = expanduser('~')
		DEFAULT_LOG_DIR = os.getcwd()


		ENV[MONIT_EXEC_PATH] \|\| --monit-exec-path PATH

		Absolute path to the directory to run the script form. (default: ${monit_exec_path})


		This section describes how to manage the BigchainDB and Tendermint processes using [Monit][monit] - a small open-source utility for managing and monitoring Unix processes.

		This section assumes that you followed the guide down to the [start MongoDB section](member-start-mongodb) inclusive.


		Check the status by running `monit status` or `monit summary`.

		By default, it will collect program logs into the `~/.bigchaindb-monit/logs` folder.

Conversation

muawiakh commented Jul 25, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Solution

Workflow

Issues Resolved

Dependencies

Remaining TODOs

Uh oh!

codecov-io commented Jul 25, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

kansi left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ldmberman commented Jul 26, 2018

Uh oh!

ttmc commented Jul 26, 2018

Uh oh!

ldmberman commented Jul 26, 2018

Uh oh!

kansi Jul 26, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kansi Jul 26, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ttmc commented Jul 26, 2018

Uh oh!

ldmberman commented Jul 26, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ldmberman commented Jul 26, 2018

Uh oh!

ldmberman commented Jul 26, 2018

Uh oh!

kansi left a comment

Choose a reason for hiding this comment

Uh oh!

ttmc commented Jul 27, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ldmberman commented Jul 27, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

muawiakh commented Jul 25, 2018 •

edited

Loading

codecov-io commented Jul 25, 2018 •

edited

Loading

kansi left a comment •

edited

Loading

kansi Jul 26, 2018 •

edited

Loading

kansi Jul 26, 2018 •

edited

Loading