Problem: No log rotation when BigchainDB and Tendermint started with Monit by muawiakh · Pull Request #2528 · bigchaindb/bigchaindb

muawiakh · 2018-09-10T08:37:04Z

Solution

Currently, BigchainDB server, takes care of log rotation using logging module.
This enforces log rotation if the file size exceeds 209715200 i.e. for bigchaindb-benchmark.log, bigchaindb.log, bigchaindb-errors.log.

Remove the extra bigchaindb.err.log and bigchaindb.out.log redirection, because BigchainDB also stores logs and has log rotation for the relevant files instead redirect to /dev/null
Introduce logrotate script.
Fix a minor log message.
Add tendermint log rotation for tendermint.err.log and tendermint.out.log if the size exceeds 200 MB.
Add docs for log rotation

codecov-io · 2018-09-10T08:46:42Z

Codecov Report

Merging #2528 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #2528   +/-   ##
=======================================
  Coverage   93.53%   93.53%           
=======================================
  Files          42       42           
  Lines        2551     2551           
=======================================
  Hits         2386     2386           
  Misses        165      165

ttmc · 2018-09-10T10:53:10Z

pkg/scripts/bigchaindb-monit-config


    pushd \$4
+
      nohup tendermint node --consensus.create_empty_blocks=false >> \$3/tendermint.out.log 2>> \$3/tendermint.err.log &


I'm not sure we need the --consensus.create_empty_blocks=false in the command to start Tendermint, because that's ensured by the settings we tell them to put in config.toml, i.e. see http://docs.bigchaindb.com/projects/server/en/master/simple-deployment-template/network-setup.html#member-connect-to-the-other-members

shahbazn · 2018-09-10T13:09:58Z

pkg/scripts/bigchaindb-monit-config


    pushd \$4
-      nohup bigchaindb -l DEBUG start >> \$3/bigchaindb.out.log 2>> \$3/bigchaindb.err.log &
+      nohup bigchaindb -l DEBUG start > /dev/null 2>&1 &


Why set logging level to debug? I think its better to leave logging flag out of the command here and let it pick up logging configs from the config file.

It is a test network, but I can change if that is necessary.

Some people might use our Monit script for their production node.

shahbazn · 2018-09-10T13:14:38Z

pkg/scripts/bigchaindb-monit-config

    with pidfile ${monit_pid_path}/tendermint.pid
    start program "${monit_script_path} start_tendermint ${monit_pid_path}/tendermint.pid ${monit_log_path} ${monit_log_path}"
-    restart program "${monit_script_path} start_bigchaindb ${monit_pid_path}/bigchaindb.pid ${monit_log_path} ${monit_log_path}"
+    restart program "${monit_script_path} start_tendermint ${monit_pid_path}/tendermint.pid ${monit_log_path} ${monit_log_path}"


does restart here stops and then starts again

Not necessarily, restart just happens when the service is not running.

Because already taken care of in the config file

ttmc · 2018-09-10T15:18:12Z

docs/server/source/appendices/log-rotation.md

@@ -0,0 +1,46 @@
+Copyright BigchainDB GmbH and BigchainDB contributors


The block at the top should be an HTML comment (so it doesn't render on ReadTheDocs). To do that, you will have to add <!-- or <!--- before it, maybe as a new line 1. (All HTML markup is valid Markdown markup.)

ttmc · 2018-09-10T15:20:56Z

docs/server/source/appendices/log-rotation.md

+
+Default location for these log files is: `$HOME/.bigchaindb-monit/logs`
+
+The logs for BigchainDB server are rotated when any of the above mentioned file exceeds `209715200 bytes`.


I wonder why that number of bytes. I guess that is the default inherited from the logging module?

Its already part of the BigchainDB code. https://github.com/bigchaindb/bigchaindb/blob/master/bigchaindb/log.py#L49

Not entirely sure why.

Maybe write "209715200 bytes (i.e. approximately 209 MB)." since 209 MB is easier to understand for humans.

ldmberman · 2018-09-10T15:56:20Z

pkg/scripts/bigchaindb-monit-config

+    /bin/mv \$2 \$2.\$(date +%y-%m-%d)
+    /bin/tar -cvf \$2.\$(date +%Y%m%d_%H%M%S).tar.gz \$2.\$(date +%y-%m-%d)
+    /bin/rm \$2.\$(date +%y-%m-%d)
+    /bin/kill -2 \$(cat \$3)


I think, you can clear the file without removing it, then Tendermint does not have to be restarted. One way of clearing it is cp /dev/null \$2.

ldmberman · 2018-09-10T16:05:49Z

docs/server/source/appendices/log-rotation.md

+
+## Log rotation for BigchainDB
+
+Log rotation is baked into BigchainDB server using the `logging` module. If you followed the [How to Set Up a BigchainDB Network](../simple-deployment-template/network-setup.md) guide. BigchainDB processes are monitored using [Monit]( https://www.mmonit.com/monit). BigchainDB server logs information into the following files:


Should we mention Monit in the BigchainDB section?

Also, makes sense to add that bigchaindb.log and other log files are by default created in the directory from where bigchaindb is run.

I rephrased the sentence, i think it makes sense now for a monit based user as well as a normal user.

ldmberman · 2018-09-10T16:09:48Z

docs/server/source/appendices/log-rotation.md

+
+## Log rotation for Tendermint
+
+Log rotation for Tendermint is handled by [Monit]( https://www.mmonit.com/monit). When we start BigchainDB and Tendermint using [How to Set Up a BigchainDB Network](../simple-deployment-template/network-setup.md) guide.


Log rotation for Tendermint is handled by Monit.

It kinda suggests that Monit is always there. I would say something like "you need to use Monit setup to use the log rotation scripts prepared by us".

ttmc · 2018-09-11T07:35:09Z

docs/server/source/appendices/log-rotation.md

+Code is Apache-2.0 and docs are CC-BY-4.0
+--->
+
+# Log rotation for a BigchainDB node


This is a new docs page so it must be added to an index.rst file otherwise Sphinx will say, for example:

Warning, treated as error: /home/troy/repos/bigchaindb/docs/server/source/appendices/log-rotation.md:document isn't included in any toctree

It's in the appendices directory, so that's the index.rst file that should be updated, i.e.

docs/server/source/appendices/index.rst

forgot to add it will add in next commit.

- Rephrase log rotation to imply monit is not default - Add statement to explain where bigchaindb log files are created - Fix HTML comment - Add log rotation to index

Log rotation for tendermint and BigchainDB using Monit

d8f5d72

muawiakh requested review from ldmberman, shahbazn and ttmc September 10, 2018 08:37

Change file size from 100KB used for testing to 200 MB for test envs

1d15e45

ttmc reviewed Sep 10, 2018

View reviewed changes

shahbazn reviewed Sep 10, 2018

View reviewed changes

muawiakh added 3 commits September 10, 2018 15:54

Add docs for log rotation

450974d

remove --consensus.create_empty_blocks from monit start

b0f0a03

Because already taken care of in the config file

do not run bigchaindb with debug flag instead use config file

ac24801

ttmc suggested changes Sep 10, 2018

View reviewed changes

ldmberman reviewed Sep 10, 2018

View reviewed changes

ttmc suggested changes Sep 11, 2018

View reviewed changes

Address comments I

439d0c4

- Rephrase log rotation to imply monit is not default - Add statement to explain where bigchaindb log files are created - Fix HTML comment - Add log rotation to index

ttmc approved these changes Sep 11, 2018

View reviewed changes

Address comments II

25d6c32

ldmberman approved these changes Sep 11, 2018

View reviewed changes

ttmc merged commit 5a44084 into bigchaindb:master Sep 12, 2018


		pushd \$4

		nohup tendermint node --consensus.create_empty_blocks=false >> \$3/tendermint.out.log 2>> \$3/tendermint.err.log &

		@@ -0,0 +1,46 @@
		Copyright BigchainDB GmbH and BigchainDB contributors


		Default location for these log files is: `$HOME/.bigchaindb-monit/logs`

		The logs for BigchainDB server are rotated when any of the above mentioned file exceeds `209715200 bytes`.


		## Log rotation for BigchainDB

		Log rotation is baked into BigchainDB server using the `logging` module. If you followed the [How to Set Up a BigchainDB Network](../simple-deployment-template/network-setup.md) guide. BigchainDB processes are monitored using [Monit]( https://www.mmonit.com/monit). BigchainDB server logs information into the following files:


		## Log rotation for Tendermint

		Log rotation for Tendermint is handled by [Monit]( https://www.mmonit.com/monit). When we start BigchainDB and Tendermint using [How to Set Up a BigchainDB Network](../simple-deployment-template/network-setup.md) guide.

Conversation

muawiakh commented Sep 10, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Solution

Uh oh!

codecov-io commented Sep 10, 2018

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shahbazn Sep 10, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

muawiakh commented Sep 10, 2018 •

edited

Loading

shahbazn Sep 10, 2018 •

edited

Loading