{"id":118170,"date":"2022-07-09T21:21:15","date_gmt":"2022-07-09T18:21:15","guid":{"rendered":"https:\/\/computingforgeeks.com\/?p=118170"},"modified":"2023-10-07T10:48:07","modified_gmt":"2023-10-07T07:48:07","slug":"run-elastic-stack-elk-on-docker","status":"publish","type":"post","link":"https:\/\/computingforgeeks.com\/run-elastic-stack-elk-on-docker\/","title":{"rendered":"Run Elastic stack (ELK) on Docker Containers with Docker Compose"},"content":{"rendered":"\n<p>The <strong><em>Elastic stack<\/em><\/strong> (ELK) is made up of 3 open source components that work together to realize logs collection, analysis, and visualization. The 3 main components are:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Elasticsearch<\/strong> &#8211; which is the core of the Elastic software. This is a search and analytics engine. Its task in the Elastic stack is to store incoming logs from Logstash and offer the ability to search the logs in real-time<\/li>\n\n\n\n<li><strong>Logstash<\/strong> &#8211; It is used to collect data, transform logs incoming from multiple sources simultaneously, and sends them to storage.<\/li>\n\n\n\n<li><strong>Kibana<\/strong> &#8211; This is a graphical tool that offers data visualization. In the Elastic stack, it is used to generate charts and graphs to make sense of the raw data in your database.<\/li>\n<\/ul>\n\n\n\n<p>The Elastic stack can as well be used with <strong>Beats<\/strong>. These are lightweight data shippers that allow multiple data sources\/indices, and send them to Elasticsearch or Logstash. There are several Beats, each with a distinct role.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Filebeat<\/strong> &#8211; Its purpose is to forward files and centralize logs usually in either<strong><em> .log<\/em><\/strong>  or <strong><em>.json<\/em><\/strong> format.<\/li>\n\n\n\n<li><strong>Metricbeat<\/strong> &#8211; It collects metrics from systems and services including CPU, memory usage, and load, as well as other data statistics from network data and process data, before being shipped to either Logstash or Elasticsearch directly.<\/li>\n\n\n\n<li><strong>Packetbeat<\/strong> &#8211; It supports a collection of network protocols from the application and lower-level protocols, databases, and key-value stores, including HTTP, DNS, Flows, DHCPv4, MySQL, and TLS. It helps identify suspicious network activities.<\/li>\n\n\n\n<li><strong>Auditbeat<\/strong> &#8211; It is used to collect Linux audit framework data and monitor file integrity, before being shipped to either Logstash or Elasticsearch directly.<\/li>\n\n\n\n<li><strong>Heartbeat<\/strong> &#8211; It is used for active probing to determine whether services are available.<\/li>\n<\/ul>\n\n\n\n<p>This guide offers a deep illustration of how to run the Elastic stack (ELK) on Docker Containers using Docker Compose.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Setup Requirements<\/h3>\n\n\n\n<p>For this guide, you need the following.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Memory<\/strong> &#8211; 1.5 GB and above<\/li>\n\n\n\n<li><strong>Docker Engine<\/strong> &#8211; version 18.06.0 or newer<\/li>\n\n\n\n<li><strong>Docker Compose<\/strong> &#8211; version 1.26.0 or newer<\/li>\n<\/ul>\n\n\n\n<p>Install the required packages below:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code><em><mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-vivid-purple-color\">## On Debian\/Ubuntu\n<\/mark><\/em>sudo apt update &amp;&amp; sudo apt upgrade\nsudo apt install curl vim git\n\n<em><mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-vivid-purple-color\">## On RHEL\/CentOS\/RockyLinux 8\n<\/mark><\/em>sudo yum -y update\nsudo yum -y install curl vim git\n\n<em><mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-vivid-purple-color\">## On Fedora\n<\/mark><\/em>sudo dnf update\nsudo dnf -y install curl vim git<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Step 1 &#8211; Install Docker and Docker Compose<\/h2>\n\n\n\n<p>Use the dedicated guide below to install the Docker Engine on your system.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><a href=\"https:\/\/computingforgeeks.com\/solve-error-package-docker-ce-stable-requires-container-selinux\/\" target=\"_blank\" rel=\"noreferrer noopener\">How To Install Docker CE on Linux Systems<\/a><\/li>\n<\/ul>\n\n\n\n<p>Add your system user to the docker group.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo usermod -aG docker $USER\nnewgrp docker<\/code><\/pre>\n\n\n\n<p>Start and enable the Docker service.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo systemctl start docker &amp;&amp; sudo systemctl enable docker<\/code><\/pre>\n\n\n\n<p>Now proceed and install Docker Compose with the aid of the below guide:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><a href=\"https:\/\/computingforgeeks.com\/how-to-install-latest-docker-compose-on-linux\/\" target=\"_blank\" rel=\"noreferrer noopener\">How To Install Docker Compose on Linux<\/a><\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Step 2 &#8211; Provision the Elastic stack (ELK) Containers.<\/h2>\n\n\n\n<p>We will begin by cloning the file from Github as below<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>git clone https:\/\/github.com\/deviantony\/docker-elk.git\ncd docker-elk<\/code><\/pre>\n\n\n\n<p>Open the deployment file for editing:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>vim docker-compose.yml<\/code><\/pre>\n\n\n\n<p>The Elastic stack deployment file consists of 3 main parts.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Elasticsearch<\/strong> &#8211; with ports:\n<ul class=\"wp-block-list\">\n<li>9200: Elasticsearch HTTP<\/li>\n\n\n\n<li>9300: Elasticsearch TCP transport<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Logstash<\/strong> &#8211; with ports:\n<ul class=\"wp-block-list\">\n<li>5044: Logstash Beats input<\/li>\n\n\n\n<li>5000: Logstash TCP input<\/li>\n\n\n\n<li>9600: Logstash monitoring API<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Kibana<\/strong> &#8211; with port 5601<\/li>\n<\/ul>\n\n\n\n<p>In the opened file, you can make the below adjustments:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Configure Elasticsearch<\/strong><\/li>\n<\/ul>\n\n\n\n<p>The configuration file for Elasticsearch is stored in the <strong><em>elasticsearch\/config\/elasticsearch.yml<\/em><\/strong> file. So you can configure the environment by setting the cluster name, network host, and licensing as below<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>elasticsearch:\n  environment:\n    cluster.name:<em> <mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-vivid-purple-color\">my-cluster<\/mark><\/em>\n    xpack.license.self_generated.type: <mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-vivid-purple-color\">basic<\/mark><\/code><\/pre>\n\n\n\n<p>To disable paid features, you need to change the<em> xpack.license.self_generated.type<\/em> setting from <strong>trial<\/strong>(the self-generated license gives access only to all the features of an x-pack for 30 days) to <strong>basic<\/strong>. <\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Configure Kibana<\/strong><\/li>\n<\/ul>\n\n\n\n<p>The configuration file is stored in the <strong><em>kibana\/config\/kibana.yml<\/em><\/strong> file. Here you can specify the environment variables as below.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kibana:\n  environment:\n    SERVER_NAME: <em><mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-vivid-purple-color\">kibana.example.com<\/mark><\/em><\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>JVM tuning<\/strong><\/li>\n<\/ul>\n\n\n\n<p>Normally, both Elasticsearch and Logstash start with <em><strong>1\/4 <\/strong><\/em>of the total host memory allocated to the JVM Heap Size. You can adjust the memory by setting the below options.<\/p>\n\n\n\n<p><strong>For Logstash<\/strong>(An example with increased memory to 1GB)<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>logstash:\n  environment:\n    LS_JAVA_OPTS: <mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-vivid-purple-color\">-Xm1g -Xms1g<\/mark><\/code><\/pre>\n\n\n\n<p>For Elasticsearch(An example with increased memory to 1GB)<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>elasticsearch:\n  environment:\n    ES_JAVA_OPTS: <mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-vivid-purple-color\">-Xm1g -Xms1g<\/mark><\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Configure the Usernames and Passwords.<\/h3>\n\n\n\n<p>To configure the usernames, passwords, and version, edit the .env file.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>vim .env<\/code><\/pre>\n\n\n\n<p>Make desired changes for the version, usernames, and passwords.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ELASTIC_VERSION=<em><mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-vivid-purple-color\">&lt;VERSION&gt;<\/mark><\/em>\n\n## Passwords for stack users\n#\n\n# User 'elastic' (built-in)\n#\n# Superuser role, full access to cluster management and data indices.\n# https:\/\/www.elastic.co\/guide\/en\/elasticsearch\/reference\/current\/built-in-users.html\nELASTIC_PASSWORD='<em><mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-luminous-vivid-orange-color\">StrongPassw0rd1<\/mark><\/em>'\n\n# User 'logstash_internal' (custom)\n#\n# The user Logstash uses to connect and send data to Elasticsearch.\n# https:\/\/www.elastic.co\/guide\/en\/logstash\/current\/ls-security.html\nLOGSTASH_INTERNAL_PASSWORD='<em><em><mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-luminous-vivid-orange-color\">StrongPassw0rd1<\/mark><\/em><\/em>'\n\n# User 'kibana_system' (built-in)\n#\n# The user Kibana uses to connect and communicate with Elasticsearch.\n# https:\/\/www.elastic.co\/guide\/en\/elasticsearch\/reference\/current\/built-in-users.html\nKIBANA_SYSTEM_PASSWORD='<em><em><mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-luminous-vivid-orange-color\">StrongPassw0rd1<\/mark><\/em><\/em>'<\/code><\/pre>\n\n\n\n<p>Source environment:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>source .env<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Step 3 &#8211; Configure Persistent Volumes.<\/h2>\n\n\n\n<p>For the Elastic stack to persist data, we need to map the volumes correctly. In the YAML file, we have several volumes to be mapped. In this guide, I will configure a secondary disk attached to my device.<\/p>\n\n\n\n<p>Identify the disk.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>$ <mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-pale-pink-color\">lsblk<\/mark>\nNAME        MAJ:MIN RM SIZE RO TYPE MOUNTPOINT\nsda           8:0    0  40G  0 disk \n\u251c\u2500sda1        8:1    0   1G  0 part \/boot\n\u2514\u2500sda2        8:2    0  39G  0 part \n  \u251c\u2500rl-root 253:0    0  35G  0 lvm  \/\n  \u2514\u2500rl-swap 253:1    0   4G  0 lvm  &#91;SWAP]\n<mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-luminous-vivid-amber-color\">sdb           8:16   0  10G  0 disk <\/mark>\n<em>\u2514\u2500sdb1        8:17   0  10G  0 part <\/em><\/code><\/pre>\n\n\n\n<p>Format the disk and create an XFS file system to it.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo parted --script \/dev\/sdb \"mklabel gpt\"\nsudo parted --script \/dev\/sdb \"mkpart primary 0% 100%\"\nsudo mkfs.xfs <mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-luminous-vivid-amber-color\">\/dev\/sdb1<\/mark><\/code><\/pre>\n\n\n\n<p>Mount the disk to your desired path.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo mkdir <mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-luminous-vivid-amber-color\">\/mnt\/datastore<\/mark>\nsudo mount <em><mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-pale-cyan-blue-color\">\/dev\/sdb1<\/mark><\/em> <mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-luminous-vivid-amber-color\">\/mnt\/datastore<\/mark><\/code><\/pre>\n\n\n\n<p>Update <code>\/etc\/fstab<\/code> file for persistent mounting.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>$ <mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-pale-pink-color\">sudo vim \/etc\/fstab<\/mark>\n\/dev\/sdb1 \/mnt\/datastore xfs defaults 0 0<\/code><\/pre>\n\n\n\n<p>Verify if the disk has been mounted.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>$ <mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-pale-pink-color\">sudo mount | grep \/dev\/sdb1<\/mark>\n\/dev\/sdb1 on \/mnt\/datastore type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)<\/code><\/pre>\n\n\n\n<p>Create the persistent volumes in the disk.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo mkdir \/mnt\/datastore\/setup\nsudo mkdir \/mnt\/datastore\/elasticsearch<\/code><\/pre>\n\n\n\n<p>Set the right permissions.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo chmod 775 -R \/mnt\/datastore\nsudo chown -R $USER:docker \/mnt\/datastore<\/code><\/pre>\n\n\n\n<p>On Rhel-based systems, configure SELinux as below.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo setenforce 0\nsudo sed -i 's\/^SELINUX=.*\/SELINUX=permissive\/g' \/etc\/selinux\/config<\/code><\/pre>\n\n\n\n<p>Create the external volumes:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>For Elasticsearch<\/strong><\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>docker volume create --driver local \\\n     --opt type=none \\\n     --opt device=<mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-luminous-vivid-amber-color\">\/mnt\/datastore\/elasticsearch<\/mark> \\\n     --opt o=bind elasticsearch<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>For setup<\/strong><\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>docker volume create --driver local \\\n     --opt type=none \\\n     --opt device=<mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-luminous-vivid-amber-color\">\/mnt\/datastore\/setup<\/mark> \\\n     --opt o=bind setup<\/code><\/pre>\n\n\n\n<p>Verify if the volumes have been created.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>$ <mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-pale-pink-color\">docker volume list<\/mark>\nDRIVER    VOLUME NAME\n<strong>local     elasticsearch\nlocal     setup<\/strong><\/code><\/pre>\n\n\n\n<p>View more details about the volume.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>$ <mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-pale-pink-color\">docker volume inspect setup<\/mark>\n&#91;\n    {\n        \"CreatedAt\": \"2022-05-06T13:19:33Z\",\n        \"Driver\": \"local\",\n        \"Labels\": {},\n        \"Mountpoint\": \"\/var\/lib\/docker\/volumes\/setup\/_data\",\n        \"Name\": \"setup\",\n        \"Options\": {\n            \"device\": \"\/mnt\/datastore\/setup\",\n            \"o\": \"bind\",\n            \"type\": \"none\"\n        },\n        \"Scope\": \"local\"\n    }\n]<\/code><\/pre>\n\n\n\n<p>Go back to the YAML file and add these lines <em><mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-luminous-vivid-orange-color\">at the end of the file<\/mark><\/em>.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>$ <mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-pale-pink-color\">vim docker-compose.yml<\/mark>\n.......\nvolumes:\n  setup:\n    external: true\n  elasticsearch:\n    external: true<\/code><\/pre>\n\n\n\n<p>Now you should have the YAML file with changes made in the below areas:<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"522\" height=\"932\" src=\"https:\/\/computingforgeeks.com\/wp-content\/uploads\/2022\/07\/Run-Elastic-stack-ELK-on-Docker-Containers.png\" alt=\"\" class=\"wp-image-120698\" title=\"\" srcset=\"https:\/\/computingforgeeks.com\/wp-content\/uploads\/2022\/07\/Run-Elastic-stack-ELK-on-Docker-Containers.png 522w, https:\/\/computingforgeeks.com\/wp-content\/uploads\/2022\/07\/Run-Elastic-stack-ELK-on-Docker-Containers-168x300.png 168w, https:\/\/computingforgeeks.com\/wp-content\/uploads\/2022\/07\/Run-Elastic-stack-ELK-on-Docker-Containers-235x420.png 235w\" sizes=\"auto, (max-width: 522px) 100vw, 522px\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">Step 4 &#8211; Bringing up the Elastic stack<\/h2>\n\n\n\n<p>After the desired changes have been made, bring up the Elastic stack with the command:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>docker-compose up -d<\/code><\/pre>\n\n\n\n<p>Execution output:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;+] Building 6.4s (12\/17)                                                                                                                   \n =&gt; &#91;docker-elk_setup internal] load build definition from Dockerfile                                                                  0.3s\n =&gt; =&gt; transferring dockerfile: 389B                                                                                                   0.0s\n =&gt; &#91;docker-elk_setup internal] load .dockerignore                                                                                     0.5s\n =&gt; =&gt; transferring context: 250B                                                                                                      0.0s\n =&gt; &#91;docker-elk_logstash internal] load build definition from Dockerfile                                                               0.6s\n =&gt; =&gt; transferring dockerfile: 312B                                                                                                   0.0s\n =&gt; &#91;docker-elk_elasticsearch internal] load build definition from Dockerfile                                                          0.6s\n =&gt; =&gt; transferring dockerfile: 324B                                                                                                   0.0s\n =&gt; &#91;docker-elk_logstash internal] load .dockerignore                                                                                  0.7s\n =&gt; =&gt; transferring context: 188B                                                 \n........<\/code><\/pre>\n\n\n\n<p>Once complete, check if the containers are running:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>$ <mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-pale-pink-color\">docker ps<\/mark>\nCONTAINER ID   IMAGE                      COMMAND                  CREATED          STATUS         PORTS                                                                                                                                                                        NAMES\n096ddc76c6b9   docker-elk_logstash        \"\/usr\/local\/bin\/dock\u2026\"   9 seconds ago    Up 5 seconds   0.0.0.0:5000-&gt;5000\/tcp, :::5000-&gt;5000\/tcp, 0.0.0.0:5044-&gt;5044\/tcp, :::5044-&gt;5044\/tcp, 0.0.0.0:9600-&gt;9600\/tcp, 0.0.0.0:5000-&gt;5000\/udp, :::9600-&gt;9600\/tcp, :::5000-&gt;5000\/udp   docker-elk-logstash-1\nec3aab33a213   docker-elk_kibana          \"\/bin\/tini -- \/usr\/l\u2026\"   9 seconds ago    Up 5 seconds   0.0.0.0:5601-&gt;5601\/tcp, :::5601-&gt;5601\/tcp                                                                                                                                    docker-elk-kibana-1\nb365f809d9f8   docker-elk_setup           \"\/entrypoint.sh\"         10 seconds ago   Up 7 seconds   9200\/tcp, 9300\/tcp                                                                                                                                                           docker-elk-setup-1\n45f6ba48a89f   docker-elk_elasticsearch   \"\/bin\/tini -- \/usr\/l\u2026\"   10 seconds ago   Up 7 seconds   0.0.0.0:9200-&gt;9200\/tcp, :::9200-&gt;9200\/tcp, 0.0.0.0:9300-&gt;9300\/tcp, :::9300-&gt;9300\/tcp                                                                                         docker-elk-elasticsearch-1<\/code><\/pre>\n\n\n\n<p>Verify if Elastic search is running:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>$ <mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-pale-pink-color\">curl http:\/\/localhost:9200 -u elastic:StrongPassw0rd1<\/mark>\n{\n  \"name\" : \"45f6ba48a89f\",\n  \"cluster_name\" : \"my-cluster\",\n  \"cluster_uuid\" : \"hGyChEAVQD682yVAx--iEQ\",\n  \"version\" : {\n    \"number\" : \"8.1.3\",\n    \"build_flavor\" : \"default\",\n    \"build_type\" : \"docker\",\n    \"build_hash\" : \"39afaa3c0fe7db4869a161985e240bd7182d7a07\",\n    \"build_date\" : \"2022-04-19T08:13:25.444693396Z\",\n    \"build_snapshot\" : false,\n    \"lucene_version\" : \"9.0.0\",\n    \"minimum_wire_compatibility_version\" : \"7.17.0\",\n    \"minimum_index_compatibility_version\" : \"7.0.0\"\n  },\n  \"tagline\" : \"You Know, for Search\"\n}<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Step 5 &#8211; Access the Kibana Dashboard.<\/h2>\n\n\n\n<p>At this point, you can proceed and access the Kibana dashboard running on port <strong>5601<\/strong>. But first, allow the required ports through the firewall.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code><em><mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-vivid-purple-color\">##For Firewalld\n<\/mark><\/em>sudo firewall-cmd --add-port=5601\/tcp --permanent\nsudo firewall-cmd --add-port=5044\/tcp --permanent\nsudo firewall-cmd --reload\n\n<em><mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-vivid-purple-color\">##For UFW\n<\/mark><\/em>sudo ufw allow 5601\/tcp\nsudo ufw allow 5044\/tcp<\/code><\/pre>\n\n\n\n<p>Now proceed and access the Kibana dashboard with the URL <a href=\"http:\/\/IP_Address:5601\" target=\"_blank\" rel=\"noreferrer noopener\">http:\/\/IP_Address:5601<\/a> or <a href=\"http:\/\/Domain_name:5601\" target=\"_blank\" rel=\"noreferrer noopener\">http:\/\/Domain_name:5601<\/a>.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"519\" height=\"461\" src=\"https:\/\/computingforgeeks.com\/wp-content\/uploads\/2022\/05\/Run-Elastic-stack-ELK-on-Docker-Containers-using-Docker-Compose.png\" alt=\"\" class=\"wp-image-118206\" title=\"\" srcset=\"https:\/\/computingforgeeks.com\/wp-content\/uploads\/2022\/05\/Run-Elastic-stack-ELK-on-Docker-Containers-using-Docker-Compose.png 519w, https:\/\/computingforgeeks.com\/wp-content\/uploads\/2022\/05\/Run-Elastic-stack-ELK-on-Docker-Containers-using-Docker-Compose-300x266.png 300w, https:\/\/computingforgeeks.com\/wp-content\/uploads\/2022\/05\/Run-Elastic-stack-ELK-on-Docker-Containers-using-Docker-Compose-473x420.png 473w\" sizes=\"auto, (max-width: 519px) 100vw, 519px\" \/><\/figure>\n\n\n\n<p>Login using the credentials set for the Elasticsearch user:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Username: <em><mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-pale-cyan-blue-color\">elastic<\/mark><\/em>\nPassword: <em><mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-vivid-purple-color\">StrongPassw0rd1<\/mark><\/em><\/code><\/pre>\n\n\n\n<p>On successful authentication, you should see the dashboard.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"737\" src=\"https:\/\/computingforgeeks.com\/wp-content\/uploads\/2022\/05\/Run-Elastic-stack-ELK-on-Docker-Containers-using-Docker-Compose-1-1024x737.png\" alt=\"\" class=\"wp-image-118207\" title=\"\" srcset=\"https:\/\/computingforgeeks.com\/wp-content\/uploads\/2022\/05\/Run-Elastic-stack-ELK-on-Docker-Containers-using-Docker-Compose-1-1024x737.png 1024w, https:\/\/computingforgeeks.com\/wp-content\/uploads\/2022\/05\/Run-Elastic-stack-ELK-on-Docker-Containers-using-Docker-Compose-1-300x216.png 300w, https:\/\/computingforgeeks.com\/wp-content\/uploads\/2022\/05\/Run-Elastic-stack-ELK-on-Docker-Containers-using-Docker-Compose-1-768x552.png 768w, https:\/\/computingforgeeks.com\/wp-content\/uploads\/2022\/05\/Run-Elastic-stack-ELK-on-Docker-Containers-using-Docker-Compose-1-696x501.png 696w, https:\/\/computingforgeeks.com\/wp-content\/uploads\/2022\/05\/Run-Elastic-stack-ELK-on-Docker-Containers-using-Docker-Compose-1-1068x768.png 1068w, https:\/\/computingforgeeks.com\/wp-content\/uploads\/2022\/05\/Run-Elastic-stack-ELK-on-Docker-Containers-using-Docker-Compose-1-584x420.png 584w, https:\/\/computingforgeeks.com\/wp-content\/uploads\/2022\/05\/Run-Elastic-stack-ELK-on-Docker-Containers-using-Docker-Compose-1.png 1261w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>Now to prove that the ELK stack is running as desired. We will inject some data\/log entries. Logstash here allows us to send content via TCP as below.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code><em><strong># Using BSD netcat (Debian, Ubuntu, MacOS system, ...)\n<\/strong><\/em>cat \/path\/to\/logfile.log | nc -q0 localhost 5000<\/code><\/pre>\n\n\n\n<p>For example:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>cat \/var\/log\/syslog | nc -q0 localhost 5000<\/code><\/pre>\n\n\n\n<p>Once the logs have been loaded, proceed and view them under the <strong>Observability<\/strong> tab.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"648\" src=\"https:\/\/computingforgeeks.com\/wp-content\/uploads\/2022\/05\/Run-Elastic-stack-ELK-on-Docker-Containers-using-Docker-Compose-2-1024x648.png\" alt=\"\" class=\"wp-image-118208\" title=\"\" srcset=\"https:\/\/computingforgeeks.com\/wp-content\/uploads\/2022\/05\/Run-Elastic-stack-ELK-on-Docker-Containers-using-Docker-Compose-2-1024x648.png 1024w, https:\/\/computingforgeeks.com\/wp-content\/uploads\/2022\/05\/Run-Elastic-stack-ELK-on-Docker-Containers-using-Docker-Compose-2-300x190.png 300w, https:\/\/computingforgeeks.com\/wp-content\/uploads\/2022\/05\/Run-Elastic-stack-ELK-on-Docker-Containers-using-Docker-Compose-2-768x486.png 768w, https:\/\/computingforgeeks.com\/wp-content\/uploads\/2022\/05\/Run-Elastic-stack-ELK-on-Docker-Containers-using-Docker-Compose-2-696x440.png 696w, https:\/\/computingforgeeks.com\/wp-content\/uploads\/2022\/05\/Run-Elastic-stack-ELK-on-Docker-Containers-using-Docker-Compose-2-1068x675.png 1068w, https:\/\/computingforgeeks.com\/wp-content\/uploads\/2022\/05\/Run-Elastic-stack-ELK-on-Docker-Containers-using-Docker-Compose-2-664x420.png 664w, https:\/\/computingforgeeks.com\/wp-content\/uploads\/2022\/05\/Run-Elastic-stack-ELK-on-Docker-Containers-using-Docker-Compose-2.png 1262w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>That is it! You have your Elastic stack (ELK) running perfectly.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Step 6 &#8211; Cleanup the setup<\/h2>\n\n\n\n<p>In case you completely want to remove the Elastic stack (ELK) and all the persistent data, use the command:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>$<mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-pale-pink-color\"> docker-compose down -v<\/mark>\n&#91;+] Running 5\/4\n \u283f Container docker-elk-kibana-1         Removed                                                                                      10.5s\n \u283f Container docker-elk-setup-1          Removed                                                                                       0.1s\n \u283f Container docker-elk-logstash-1       Removed                                                                                       9.9s\n \u283f Container docker-elk-elasticsearch-1  Removed                                                                                       3.0s\n \u283f Network docker-elk_elk                Removed                                                                                       0.1s<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Closing Thoughts.<\/h2>\n\n\n\n<p>We have successfully walked through how to run Elastic stack (ELK) on Docker Containers using Docker Compose. Futhermore, we have learned how to create an external persistent volume for Docker containers. I hope this was significant.<\/p>\n\n\n\n<p>Related posts:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><a href=\"https:\/\/computingforgeeks.com\/forward-kubernetes-logs-to-elasticsearch-using-fluentbit\/\" target=\"_blank\" rel=\"noreferrer noopener\">Forward Kubernetes Logs to Elasticsearch (ELK) using Fluentbit<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/computingforgeeks.com\/forward-server-logs-and-metrics-using-beats\/\" target=\"_blank\" rel=\"noreferrer noopener\">Forward Server logs and metrics to Elasticsearch using Beats<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/computingforgeeks.com\/stream-logs-in-aws-from-cloudwatch-to-elasticsearch\/\" target=\"_blank\" rel=\"noreferrer noopener\">Stream Logs in AWS from CloudWatch to ElasticSearch<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>The Elastic stack (ELK) is made up of 3 open source components that work together to realize logs collection, analysis, and visualization. The 3 main components are: The Elastic stack can as well be used with Beats. These are lightweight data shippers that allow multiple data sources\/indices, and send them to Elasticsearch or Logstash. There &#8230; <a title=\"Run Elastic stack (ELK) on Docker Containers with Docker Compose\" class=\"read-more\" href=\"https:\/\/computingforgeeks.com\/run-elastic-stack-elk-on-docker\/\" aria-label=\"Read more about Run Elastic stack (ELK) on Docker Containers with Docker Compose\">Read more<\/a><\/p>\n","protected":false},"author":21,"featured_media":81695,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[316,27,396,299,50],"tags":[37582,37583],"class_list":["post-118170","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-containers","category-docker","category-elasticseach","category-how-to","category-linux-tutorials","tag-elastic-stack-elk-on-docker","tag-elastic-stack-docker-containers"],"_links":{"self":[{"href":"https:\/\/computingforgeeks.com\/wp-json\/wp\/v2\/posts\/118170","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/computingforgeeks.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/computingforgeeks.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/computingforgeeks.com\/wp-json\/wp\/v2\/users\/21"}],"replies":[{"embeddable":true,"href":"https:\/\/computingforgeeks.com\/wp-json\/wp\/v2\/comments?post=118170"}],"version-history":[{"count":0,"href":"https:\/\/computingforgeeks.com\/wp-json\/wp\/v2\/posts\/118170\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/computingforgeeks.com\/wp-json\/wp\/v2\/media\/81695"}],"wp:attachment":[{"href":"https:\/\/computingforgeeks.com\/wp-json\/wp\/v2\/media?parent=118170"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/computingforgeeks.com\/wp-json\/wp\/v2\/categories?post=118170"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/computingforgeeks.com\/wp-json\/wp\/v2\/tags?post=118170"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}