docker0<\/code> and docker_gwbridge<\/code> bridges<\/li>\n- Up and down Swarm nodes<\/li>\n
- Centralize logs<\/li>\n<\/ol>\n
The goal of this post is to demonstrate the use of Elasticsearch<\/a> + Kibana<\/a> + cAdvisor<\/a> as tools to analyze and gather metrics and visualize dashboards for Docker containers.<\/p>\nLater on in this post, you can find a dashboard trying to address a few points from the previous list. There are also points that can\u2019t be addressed by simply using cAdvisor, like the status of Swarm Mode nodes.<\/p>\n
Also, if you have specific needs that aren\u2019t covered by cAdvisor or another tool, I encourage you to write your own data collector and data shipper (e.g.<\/em>, Beats<\/a>). Note that I won\u2019t be showing you how to centralize Docker containers log on Elasticsearch.<\/p>\nWhy Do We Need to Monitor Containers?<\/h2>\n
Imagine yourself in the classic situation of managing a virtual machine, either just one or several. You are a tmux hero<\/strong>, so you have your sessions preconfigured to do basically everything, monitoring included. There\u2019s a problem in production? You just do a top<\/code>, htop<\/code>, iotop<\/code>, jnettop<\/code>, whatevertop<\/code> on all your machines, and you\u2019re ready for troubleshooting!<\/p>\nNow imagine that you have the same three nodes but split into 50 containers. You need some history displayed nicely in a single place where you can perform queries to know what happened instead of just risking your life in front of those ncurses tools.<\/p>\n
What Is the Elastic Stack?<\/h2>\n
The Elastic Stack is a set of tools composed of:<\/p>\n
\n- Elasticsearch<\/li>\n
- Kibana<\/li>\n
- Logstash<\/li>\n
- Beats<\/li>\n<\/ul>\n
We\u2019re going to use a few open-source tools from the Elastic Stack, such as Elasticsearch for the JSON-based analytics engine and Kibana to visualize data and create dashboards.<\/p>\n
Another important piece of the Elastic Stack is Beats<\/a>, but in this post, we\u2019re focused on containers. There\u2019s no official Beat for Docker, so we\u2019ll just use cAdvisor<\/strong> that can natively talk with Elasticsearch.<\/p>\ncAdvisor<\/a> is a tool that collects, aggregates, and exports metrics about running containers. In our case, those metrics are being exported to an Elasticsearch storage.<\/p>\nTwo cool facts about cAdvisor are:<\/p>\n
\n- It\u2019s not limited to Docker containers.<\/li>\n
- It has its own webserver with a simple dashboard to visualize gathered metrics for the current node.<\/li>\n<\/ul>\n
Set Up a Test Cluster or BYOI<\/h2>\n
As I did in my previous<\/a> posts<\/a>, my habit is to provide a small script to allow the reader to set up a test environment on which to try out my project\u2019s steps in no time. So you can use the following not-for-production-use<\/strong> script to set up a little Swarm Mode cluster with Elasticsearch running as a container.<\/p>\nIf you have enough time\/experience, you can BYOI<\/strong> (Bring Your Own Infrastructure).<\/p>\nTo follow this post, you\u2019ll just need:<\/p>\n
\n- One or more nodes running the Docker daemon >= 1.12<\/li>\n
- At least a stand-alone Elasticsearch node 2.4.X<\/li>\n<\/ul>\n
Again, note that this post is not<\/strong> about setting up a production-ready Elasticsearch cluster. A single node cluster is not recommended for production. So if you\u2019re planning a production installation, please refer to Elastic guidelines<\/a>.<\/p>\nA friendly note for early adopters<\/h3>\n
I\u2019m usually an early adopter (and I\u2019m already using the latest alpha version in production, of course). But for this post, I chose not to use the latest Elasticsearch 5.0.0 alpha. Their roadmap is not perfectly clear to me, and I don\u2019t want be the root cause of your problems!<\/p>\n
So the Elasticsearch reference version for this post is the latest stable version, 2.4.0<\/a> at the moment of writing.<\/p>\nTest cluster setup script<\/h3>\n
As said, I wanted to provide this script for everyone who would like to follow the blog without having to figure out how to create a Swarm cluster and install an Elasticsearch. Of course, you can skip this if you choose to use your own Swarm Mode engines and your own Elasticsearch nodes.<\/p>\n
To execute the setup script, you\u2019ll need:<\/p>\n
\n- Docker Machine \u2013 latest version<\/a><\/strong>: to provision Docker engines on DigitalOcean<\/li>\n
- DigitalOcean API Token<\/a><\/strong>: to allow
docker-machine<\/code> to start nodes on your behalf<\/li>\n<\/ul>\nCreate Cluster Script<\/h2>\n
Now that you have everything we need, you can copy the following script in a file named create-cluster.sh<\/code>:<\/p>\n#!\/usr\/bin\/env bash\r\n#\r\n# Create a Swarm Mode cluster with a single master and a configurable number of workers\r\n\r\nworkers=${WORKERS:-\"worker1 worker2\"}\r\n\r\n#######################################\r\n# Creates a machine on Digital Ocean\r\n# Globals:\r\n# DO_ACCESS_TOKEN The token needed to access DigitalOcean's API\r\n# Arguments:\r\n# $1 the actual name to give to the machine\r\n#######################################\r\ncreate_machine() {\r\n docker-machine create \\\r\n -d digitalocean \\\r\n --digitalocean-access-token=$DO_ACCESS_TOKEN \\\r\n --digitalocean-size 2gb \\\r\n $1\r\n}\r\n\r\n#######################################\r\n# Executes a command on the specified machine\r\n# Arguments:\r\n# $1 The machine on which to run the command\r\n# $2..$n The command to execute on that machine\r\n#######################################\r\nmachine_do() {\r\n docker-machine ssh $@\r\n}\r\n\r\nmain() {\r\n\r\n if [ -z \"$DO_ACCESS_TOKEN\" ]; then\r\n echo \"Please export a DigitalOcean Access token: https:\/\/cloud.digitalocean.com\/settings\/api\/tokens\/new\"\r\n echo \"export DO_ACCESS_TOKEN=<yourtokenhere>\"\r\n exit 1\r\n fi\r\n\r\n if [ -z \"$WORKERS\" ]; then\r\n echo \"You haven't provided your workers by setting the \\$WORKERS environment variable, using the default ones: $workers\"\r\n fi\r\n\r\n # Create the first and only master\r\n echo \"Creating the master\"\r\n\r\n create_machine master1\r\n\r\n master_ip=$(docker-machine ip master1)\r\n\r\n # Initialize the swarm mode on it\r\n echo \"Initializing the swarm mode\"\r\n machine_do master1 docker swarm init --advertise-addr $master_ip\r\n\r\n # Obtain the token to allow workers to join\r\n worker_tkn=$(machine_do master1 docker swarm join-token -q worker)\r\n echo \"Worker token: ${worker_tkn}\"\r\n\r\n # Create and join the workers\r\n for worker in $workers; do\r\n echo \"Creating worker ${worker}\"\r\n create_machine $worker\r\n machine_do $worker docker swarm join --token $worker_tkn $master_ip:2377\r\n done\r\n}\r\n\r\nmain $@<\/pre>\nAnd make it executable:<\/p>\n
chmod +x create-cluster.sh<\/pre>\nCreate the cluster<\/h3>\n
As the name suggests, we\u2019ll use the script to create the cluster. By default, the script will create a cluster with a single master and two workers. If you want to configure the number of workers, you can do that by setting the WORKERS<\/code> environment variable.<\/p>\nNow, let\u2019s create that cluster!<\/p>\n
.\/create-cluster.sh<\/pre>\nOk, now you can go out for a coffee. This will take a while.<\/p>\n
Finally the cluster is ready!<\/p>\n
![\"cluster\"](\"http:\/\/www.webcodegeeks.com\/wp-content\/uploads\/2016\/09\/cluster.png\")
<\/a><\/p>\nNow, to verify that our Swarm Mode cluster is up and running, we can SSH into the master:<\/p>\n
docker-machine ssh master1<\/code><\/p>\nand we can list the nodes in the cluster:<\/p>\n
docker node ls<\/pre>\nID HOSTNAME STATUS AVAILABILITY MANAGER STATUS\r\n26fi3wiqr8lsidkjy69k031w2 * master1 Ready Active Leader\r\ndyluxpq8sztj7kmwlzs51u4id worker2 Ready Active\r\nepglndegvixag0jztarn2lte8 worker1 Ready Active<\/pre>\nInstall Elasticsearch and Kibana<\/h3>\n
Note that from now on, all the commands are intended to be executed against a master node (master1).<\/p>\n
In a production environment, you\u2019ll probably want to install Elasticsearch and Kibana on a separate set of instances with proper sizing<\/a>. In this case, we\u2019re just going to install them as containers in the same Swarm Mode cluster.<\/p>\nIn order to connect Elasticsearch with Kibana and cAdvisor, we need to create a custom network. Since we\u2019re in a cluster and the containers may be in different nodes, our network will use the overlay<\/a> driver.<\/p>\nYou might be asking yourself, \u201cWhy do I need a network? Aren\u2019t links enough?\u201d Please consider that since the introduction of user defined networks<\/em>, links are considered a legacy feature<\/strong>.<\/p>\nFrom the documentation<\/a>:<\/p>\nBefore the Docker networks feature, you could use the Docker link feature to allow containers to discover each other and securely transfer information about one container to another container. With the introduction of the Docker networks feature, you can still create links, but they behave differently between default bridge network and user defined networks.<\/p>\n
So let\u2019s create an overlay network named monitoring<\/code>:<\/p>\ndocker network create monitoring -d overlay<\/pre>\nElasticsearch container<\/h2>\ndocker service create --network=monitoring \\\r\n --mount type=volume,target=\/usr\/share\/elasticsearch\/data \\\r\n --constraint node.hostname==worker1 \\\r\n --name elasticsearch elasticsearch:2.4.0<\/pre>\nNote that the Elasticsearch container is constrained to the worker1<\/code> node. This is because it depends on the host where it\u2019s running for mounted volumes.<\/p>\nKibana container<\/h2>\ndocker service create --network=monitoring --name kibana -e ELASTICSEARCH_URL=\"http:\/\/elasticsearch:9200\" -p 5601:5601 kibana:4.6.0<\/pre>\nAs you can see, we started both containers telling them to join the monitoring<\/code> network so the elasticsearch<\/code> container can be accessed by its name from the services in the same network like Kibana.<\/p>\nNow, thanks to the routing mesh<\/a>, we can point our browser to any IP address of one of our machines and see the Kibana dashboard status page.<\/p>\nTo get the public IP of the master1<\/code> instance:<\/p>\ndocker-machine ip master1<\/pre>\nOpen your favorite browser and point it to: http:\/\/[master1-ip-address]:5601\/status<\/code><\/p>\nEverything should be green, like this:<\/p>\n
![\"kibana-screenshot\"](\"http:\/\/www.webcodegeeks.com\/wp-content\/uploads\/2016\/09\/kibana-screenshot-1024x576.png\")
<\/a><\/p>\nLet\u2019s go on and start collecting telemetries!<\/p>\n
Collect Metrics on Running Containers<\/h2>\n
To start gathering metrics on running containers, we\u2019ll create a service running cAdvisor in global mode<\/strong> so that the scheduler can place one task on each available node.<\/p>\nThis service has to be started in the same network of Elasticsearch so that it can push metrics to it.<\/p>\n
docker service create --network=monitoring --mode global --name cadvisor \\\r\n --mount type=bind,source=\/,target=\/rootfs,readonly=true \\\r\n --mount type=bind,source=\/var\/run,target=\/var\/run,readonly=false \\\r\n --mount type=bind,source=\/sys,target=\/sys,readonly=true \\\r\n --mount type=bind,source=\/var\/lib\/docker\/,target=\/var\/lib\/docker,readonly=true \\\r\n google\/cadvisor:latest \\\r\n -storage_driver=elasticsearch \\\r\n -storage_driver_es_host=\"http:\/\/elasticsearch:9200\"<\/pre>\nNote: If you want to configure cAdvisor options, here\u2019s the reference<\/a>.<\/p>\nOkay, now that cAdvisor is sending metrics to Elasticsearch, we can define how we want to explore that data in Kibana by declaring an index pattern<\/strong>. We can do this in one of two ways: via Kibana OR via API. We\u2019ll do that the API way of course!<\/p>\nYou have to execute the index creation command in a running container attached to the monitoring<\/code> network. You can get a shell in the cAdvisor<\/code> container. Unfortunately Swarm Mode appends a unique id to the name of containers when started as services, so you have to figure out by yourself the name of the cadvisor<\/code> container.<\/p>\nGet a shell:<\/p>\n
docker exec -ti <cadvisor-container-name> sh<\/pre>\nCreate the index:<\/p>\n
curl -XPUT http:\/\/elasticsearch:9200\/.kibana\/index-pattern\/cadvisor -d '{\"title\" : \"cadvisor*\", \"timeFieldName\": \"container_stats.timestamp\"}'<\/pre>\nAnd if you\u2019re too lazy, here\u2019s a one-liner:<\/p>\n
docker exec $(docker ps | grep cadvisor | awk '{print $1}' | head -1) curl -XPUT http:\/\/elasticsearch:9200\/.kibana\/index-pattern\/cadvisor -d '{\"title\" : \"cadvisor*\", \"timeFieldName\": \"container_stats.timestamp\"}'<\/pre>\nPutting The Pieces Together in a Dashboard<\/h2>\n
Cool! Now you can go and play with Kibana to create nice visualizations of the gathered metrics. But don\u2019t worry. If you\u2019re on the fast lane, I created a dashboard and a few visualizations for you as a starting point.<\/p>\n
![\"dashboard\"](\"http:\/\/www.webcodegeeks.com\/wp-content\/uploads\/2016\/09\/dashboard-1024x565.png\")
<\/a><\/p>\nTo import my settings, visit your Kibana dashboard and go to Settings => Objects => Import<\/code>, then select the file where you saved the following JSON:<\/p>\n[\r\n {\r\n \"_id\": \"cAdvisor\",\r\n \"_type\": \"dashboard\",\r\n \"_source\": {\r\n \"title\": \"cAdvisor\",\r\n \"hits\": 0,\r\n \"description\": \"\",\r\n \"panelsJSON\": \"[{\\\"id\\\":\\\"Filesystem-usage\\\",\\\"type\\\":\\\"visualization\\\",\\\"panelIndex\\\":1,\\\"size_x\\\":6,\\\"size_y\\\":3,\\\"col\\\":1,\\\"row\\\":1},{\\\"id\\\":\\\"Memory-[Node-equal->Container]\\\",\\\"type\\\":\\\"visualization\\\",\\\"panelIndex\\\":2,\\\"size_x\\\":6,\\\"size_y\\\":4,\\\"col\\\":7,\\\"row\\\":4},{\\\"id\\\":\\\"memory-usage-by-machine\\\",\\\"type\\\":\\\"visualization\\\",\\\"panelIndex\\\":3,\\\"size_x\\\":6,\\\"size_y\\\":6,\\\"col\\\":1,\\\"row\\\":4},{\\\"id\\\":\\\"CPU-Total-Usage\\\",\\\"type\\\":\\\"visualization\\\",\\\"panelIndex\\\":4,\\\"size_x\\\":6,\\\"size_y\\\":5,\\\"col\\\":7,\\\"row\\\":8},{\\\"id\\\":\\\"Network-RX-TX\\\",\\\"type\\\":\\\"visualization\\\",\\\"panelIndex\\\":5,\\\"size_x\\\":6,\\\"size_y\\\":3,\\\"col\\\":7,\\\"row\\\":1}]\",\r\n \"optionsJSON\": \"{\\\"darkTheme\\\":false}\",\r\n \"uiStateJSON\": \"{}\",\r\n \"version\": 1,\r\n \"timeRestore\": false,\r\n \"kibanaSavedObjectMeta\": {\r\n \"searchSourceJSON\": \"{\\\"filter\\\":[{\\\"query\\\":{\\\"query_string\\\":{\\\"query\\\":\\\"*\\\",\\\"analyze_wildcard\\\":true}}}]}\"\r\n }\r\n }\r\n },\r\n {\r\n \"_id\": \"Network\",\r\n \"_type\": \"search\",\r\n \"_source\": {\r\n \"title\": \"Network\",\r\n \"description\": \"\",\r\n \"hits\": 0,\r\n \"columns\": [\r\n \"machine_name\",\r\n \"container_Name\",\r\n \"container_stats.network.name\",\r\n \"container_stats.network.interfaces\",\r\n \"container_stats.network.rx_bytes\",\r\n \"container_stats.network.rx_packets\",\r\n \"container_stats.network.rx_dropped\",\r\n \"container_stats.network.rx_errors\",\r\n \"container_stats.network.tx_packets\",\r\n \"container_stats.network.tx_bytes\",\r\n \"container_stats.network.tx_dropped\",\r\n \"container_stats.network.tx_errors\"\r\n ],\r\n \"sort\": [\r\n \"container_stats.timestamp\",\r\n \"desc\"\r\n ],\r\n \"version\": 1,\r\n \"kibanaSavedObjectMeta\": {\r\n \"searchSourceJSON\": \"{\\\"index\\\":\\\"cadvisor*\\\",\\\"query\\\":{\\\"query_string\\\":{\\\"analyze_wildcard\\\":true,\\\"query\\\":\\\"*\\\"}},\\\"highlight\\\":{\\\"pre_tags\\\":[\\\"@kibana-highlighted-field@\\\"],\\\"post_tags\\\":[\\\"@\/kibana-highlighted-field@\\\"],\\\"fields\\\":{\\\"*\\\":{}},\\\"fragment_size\\\":2147483647},\\\"filter\\\":[]}\"\r\n }\r\n }\r\n },\r\n {\r\n \"_id\": \"Filesystem-usage\",\r\n \"_type\": \"visualization\",\r\n \"_source\": {\r\n \"title\": \"Filesystem usage\",\r\n \"visState\": \"{\\\"title\\\":\\\"Filesystem usage\\\",\\\"type\\\":\\\"histogram\\\",\\\"params\\\":{\\\"addLegend\\\":true,\\\"addTimeMarker\\\":false,\\\"addTooltip\\\":true,\\\"defaultYExtents\\\":false,\\\"mode\\\":\\\"stacked\\\",\\\"scale\\\":\\\"linear\\\",\\\"setYExtents\\\":false,\\\"shareYAxis\\\":true,\\\"times\\\":[],\\\"yAxis\\\":{}},\\\"aggs\\\":[{\\\"id\\\":\\\"1\\\",\\\"type\\\":\\\"avg\\\",\\\"schema\\\":\\\"metric\\\",\\\"params\\\":{\\\"field\\\":\\\"container_stats.filesystem.usage\\\",\\\"customLabel\\\":\\\"USED\\\"}},{\\\"id\\\":\\\"2\\\",\\\"type\\\":\\\"terms\\\",\\\"schema\\\":\\\"split\\\",\\\"params\\\":{\\\"field\\\":\\\"machine_name\\\",\\\"size\\\":5,\\\"order\\\":\\\"desc\\\",\\\"orderBy\\\":\\\"1\\\",\\\"row\\\":false}},{\\\"id\\\":\\\"3\\\",\\\"type\\\":\\\"avg\\\",\\\"schema\\\":\\\"metric\\\",\\\"params\\\":{\\\"field\\\":\\\"container_stats.filesystem.capacity\\\",\\\"customLabel\\\":\\\"AVAIL\\\"}},{\\\"id\\\":\\\"4\\\",\\\"type\\\":\\\"terms\\\",\\\"schema\\\":\\\"segment\\\",\\\"params\\\":{\\\"field\\\":\\\"container_stats.filesystem.device\\\",\\\"size\\\":5,\\\"order\\\":\\\"desc\\\",\\\"orderBy\\\":\\\"1\\\"}}],\\\"listeners\\\":{}}\",\r\n \"uiStateJSON\": \"{\\\"vis\\\":{\\\"colors\\\":{\\\"Average container_stats.filesystem.available\\\":\\\"#E24D42\\\",\\\"Average container_stats.filesystem.base_usage\\\":\\\"#890F02\\\",\\\"Average container_stats.filesystem.capacity\\\":\\\"#3F6833\\\",\\\"Average container_stats.filesystem.usage\\\":\\\"#E24D42\\\",\\\"USED\\\":\\\"#BF1B00\\\",\\\"AVAIL\\\":\\\"#508642\\\"}}}\",\r\n \"description\": \"\",\r\n \"version\": 1,\r\n \"kibanaSavedObjectMeta\": {\r\n \"searchSourceJSON\": \"{\\\"index\\\":\\\"cadvisor*\\\",\\\"query\\\":{\\\"query_string\\\":{\\\"analyze_wildcard\\\":true,\\\"query\\\":\\\"*\\\"}},\\\"filter\\\":[]}\"\r\n }\r\n }\r\n },\r\n {\r\n \"_id\": \"CPU-Total-Usage\",\r\n \"_type\": \"visualization\",\r\n \"_source\": {\r\n \"title\": \"CPU Total Usage\",\r\n \"visState\": \"{\\\"title\\\":\\\"CPU Total Usage\\\",\\\"type\\\":\\\"area\\\",\\\"params\\\":{\\\"shareYAxis\\\":true,\\\"addTooltip\\\":true,\\\"addLegend\\\":true,\\\"smoothLines\\\":false,\\\"scale\\\":\\\"linear\\\",\\\"interpolate\\\":\\\"linear\\\",\\\"mode\\\":\\\"stacked\\\",\\\"times\\\":[],\\\"addTimeMarker\\\":false,\\\"defaultYExtents\\\":false,\\\"setYExtents\\\":false,\\\"yAxis\\\":{}},\\\"aggs\\\":[{\\\"id\\\":\\\"1\\\",\\\"type\\\":\\\"avg\\\",\\\"schema\\\":\\\"metric\\\",\\\"params\\\":{\\\"field\\\":\\\"container_stats.cpu.usage.total\\\"}},{\\\"id\\\":\\\"2\\\",\\\"type\\\":\\\"date_histogram\\\",\\\"schema\\\":\\\"segment\\\",\\\"params\\\":{\\\"field\\\":\\\"container_stats.timestamp\\\",\\\"interval\\\":\\\"auto\\\",\\\"customInterval\\\":\\\"2h\\\",\\\"min_doc_count\\\":1,\\\"extended_bounds\\\":{}}},{\\\"id\\\":\\\"3\\\",\\\"type\\\":\\\"terms\\\",\\\"schema\\\":\\\"group\\\",\\\"params\\\":{\\\"field\\\":\\\"container_Name\\\",\\\"size\\\":5,\\\"order\\\":\\\"desc\\\",\\\"orderBy\\\":\\\"1\\\"}},{\\\"id\\\":\\\"4\\\",\\\"type\\\":\\\"terms\\\",\\\"schema\\\":\\\"split\\\",\\\"params\\\":{\\\"field\\\":\\\"machine_name\\\",\\\"size\\\":5,\\\"order\\\":\\\"desc\\\",\\\"orderBy\\\":\\\"1\\\",\\\"row\\\":true}}],\\\"listeners\\\":{}}\",\r\n \"uiStateJSON\": \"{}\",\r\n \"description\": \"\",\r\n \"version\": 1,\r\n \"kibanaSavedObjectMeta\": {\r\n \"searchSourceJSON\": \"{\\\"index\\\":\\\"cadvisor*\\\",\\\"query\\\":{\\\"query_string\\\":{\\\"query\\\":\\\"*\\\",\\\"analyze_wildcard\\\":true}},\\\"filter\\\":[]}\"\r\n }\r\n }\r\n },\r\n {\r\n \"_id\": \"memory-usage-by-machine\",\r\n \"_type\": \"visualization\",\r\n \"_source\": {\r\n \"title\": \"Memory [Node]\",\r\n \"visState\": \"{\\\"title\\\":\\\"Memory [Node]\\\",\\\"type\\\":\\\"area\\\",\\\"params\\\":{\\\"shareYAxis\\\":true,\\\"addTooltip\\\":true,\\\"addLegend\\\":true,\\\"smoothLines\\\":false,\\\"scale\\\":\\\"linear\\\",\\\"interpolate\\\":\\\"linear\\\",\\\"mode\\\":\\\"stacked\\\",\\\"times\\\":[],\\\"addTimeMarker\\\":false,\\\"defaultYExtents\\\":false,\\\"setYExtents\\\":false,\\\"yAxis\\\":{}},\\\"aggs\\\":[{\\\"id\\\":\\\"1\\\",\\\"type\\\":\\\"avg\\\",\\\"schema\\\":\\\"metric\\\",\\\"params\\\":{\\\"field\\\":\\\"container_stats.memory.usage\\\"}},{\\\"id\\\":\\\"2\\\",\\\"type\\\":\\\"date_histogram\\\",\\\"schema\\\":\\\"segment\\\",\\\"params\\\":{\\\"field\\\":\\\"container_stats.timestamp\\\",\\\"interval\\\":\\\"auto\\\",\\\"customInterval\\\":\\\"2h\\\",\\\"min_doc_count\\\":1,\\\"extended_bounds\\\":{}}},{\\\"id\\\":\\\"3\\\",\\\"type\\\":\\\"terms\\\",\\\"schema\\\":\\\"group\\\",\\\"params\\\":{\\\"field\\\":\\\"machine_name\\\",\\\"size\\\":5,\\\"order\\\":\\\"desc\\\",\\\"orderBy\\\":\\\"1\\\"}}],\\\"listeners\\\":{}}\",\r\n \"uiStateJSON\": \"{}\",\r\n \"description\": \"\",\r\n \"version\": 1,\r\n \"kibanaSavedObjectMeta\": {\r\n \"searchSourceJSON\": \"{\\\"index\\\":\\\"cadvisor*\\\",\\\"query\\\":{\\\"query_string\\\":{\\\"query\\\":\\\"*\\\",\\\"analyze_wildcard\\\":true}},\\\"filter\\\":[]}\"\r\n }\r\n }\r\n },\r\n {\r\n \"_id\": \"Network-RX-TX\",\r\n \"_type\": \"visualization\",\r\n \"_source\": {\r\n \"title\": \"Network RX TX\",\r\n \"visState\": \"{\\\"title\\\":\\\"Network RX TX\\\",\\\"type\\\":\\\"histogram\\\",\\\"params\\\":{\\\"addLegend\\\":true,\\\"addTimeMarker\\\":true,\\\"addTooltip\\\":true,\\\"defaultYExtents\\\":false,\\\"mode\\\":\\\"stacked\\\",\\\"scale\\\":\\\"linear\\\",\\\"setYExtents\\\":false,\\\"shareYAxis\\\":true,\\\"times\\\":[],\\\"yAxis\\\":{}},\\\"aggs\\\":[{\\\"id\\\":\\\"1\\\",\\\"type\\\":\\\"avg\\\",\\\"schema\\\":\\\"metric\\\",\\\"params\\\":{\\\"field\\\":\\\"container_stats.network.rx_bytes\\\",\\\"customLabel\\\":\\\"RX\\\"}},{\\\"id\\\":\\\"2\\\",\\\"type\\\":\\\"date_histogram\\\",\\\"schema\\\":\\\"segment\\\",\\\"params\\\":{\\\"field\\\":\\\"container_stats.timestamp\\\",\\\"interval\\\":\\\"s\\\",\\\"customInterval\\\":\\\"2h\\\",\\\"min_doc_count\\\":1,\\\"extended_bounds\\\":{}}},{\\\"id\\\":\\\"3\\\",\\\"type\\\":\\\"avg\\\",\\\"schema\\\":\\\"metric\\\",\\\"params\\\":{\\\"field\\\":\\\"container_stats.network.tx_bytes\\\",\\\"customLabel\\\":\\\"TX\\\"}}],\\\"listeners\\\":{}}\",\r\n \"uiStateJSON\": \"{\\\"vis\\\":{\\\"colors\\\":{\\\"RX\\\":\\\"#EAB839\\\",\\\"TX\\\":\\\"#BF1B00\\\"}}}\",\r\n \"description\": \"\",\r\n \"savedSearchId\": \"Network\",\r\n \"version\": 1,\r\n \"kibanaSavedObjectMeta\": {\r\n \"searchSourceJSON\": \"{\\\"filter\\\":[]}\"\r\n }\r\n }\r\n },\r\n {\r\n \"_id\": \"Memory-[Node-equal->Container]\",\r\n \"_type\": \"visualization\",\r\n \"_source\": {\r\n \"title\": \"Memory [Node=>Container]\",\r\n \"visState\": \"{\\\"title\\\":\\\"Memory [Node=>Container]\\\",\\\"type\\\":\\\"area\\\",\\\"params\\\":{\\\"shareYAxis\\\":true,\\\"addTooltip\\\":true,\\\"addLegend\\\":true,\\\"smoothLines\\\":false,\\\"scale\\\":\\\"linear\\\",\\\"interpolate\\\":\\\"linear\\\",\\\"mode\\\":\\\"stacked\\\",\\\"times\\\":[],\\\"addTimeMarker\\\":false,\\\"defaultYExtents\\\":false,\\\"setYExtents\\\":false,\\\"yAxis\\\":{}},\\\"aggs\\\":[{\\\"id\\\":\\\"1\\\",\\\"type\\\":\\\"avg\\\",\\\"schema\\\":\\\"metric\\\",\\\"params\\\":{\\\"field\\\":\\\"container_stats.memory.usage\\\"}},{\\\"id\\\":\\\"2\\\",\\\"type\\\":\\\"date_histogram\\\",\\\"schema\\\":\\\"segment\\\",\\\"params\\\":{\\\"field\\\":\\\"container_stats.timestamp\\\",\\\"interval\\\":\\\"auto\\\",\\\"customInterval\\\":\\\"2h\\\",\\\"min_doc_count\\\":1,\\\"extended_bounds\\\":{}}},{\\\"id\\\":\\\"3\\\",\\\"type\\\":\\\"terms\\\",\\\"schema\\\":\\\"group\\\",\\\"params\\\":{\\\"field\\\":\\\"container_Name\\\",\\\"size\\\":5,\\\"order\\\":\\\"desc\\\",\\\"orderBy\\\":\\\"1\\\"}},{\\\"id\\\":\\\"4\\\",\\\"type\\\":\\\"terms\\\",\\\"schema\\\":\\\"split\\\",\\\"params\\\":{\\\"field\\\":\\\"machine_name\\\",\\\"size\\\":5,\\\"order\\\":\\\"desc\\\",\\\"orderBy\\\":\\\"1\\\",\\\"row\\\":true}}],\\\"listeners\\\":{}}\",\r\n \"uiStateJSON\": \"{}\",\r\n \"description\": \"\",\r\n \"version\": 1,\r\n \"kibanaSavedObjectMeta\": {\r\n \"searchSourceJSON\": \"{\\\"index\\\":\\\"cadvisor*\\\",\\\"query\\\":{\\\"query_string\\\":{\\\"query\\\":\\\"* NOT container_Name.raw: \\\\\\\\\\\\\\\"\/\\\\\\\\\\\\\\\" AND NOT container_Name.raw: \\\\\\\\\\\\\\\"\/docker\\\\\\\\\\\\\\\"\\\",\\\"analyze_wildcard\\\":true}},\\\"filter\\\":[]}\"\r\n }\r\n }\r\n }\r\n]<\/pre>\nThere are a lot of fields to play with, so you may want to customize the dashboard to add Memory Page faulting insights or RX\/TX dropped packets for example. Also it would be nice if you can address the missing points from my previous list.<\/p>\n
Conclusion<\/h2>\n
Monitoring the right way requires a lot of time and effort. Monitoring CPU, memory, IO, network, and disk of containers is just a drop in the sea.<\/p>\n
I don\u2019t know where you are personally in the process, but next steps might be:<\/p>\n
\n- Aggregating logs from running containers<\/li>\n
- Aggregating logs from applications<\/li>\n
- Application performance monitoring<\/li>\n
- Alerting<\/li>\n
- State\/Health check monitoring<\/li>\n<\/ul>\n
If you have hints or advice, please share in the comments! Good luck with your setup!<\/p>\n
And now, shut down your test infrastructure:<\/p>\n
docker-machine rm master1 worker{1,2}<\/pre>\n\n