INSTALL ELASTICSEARCH, FLUENTD AND KIBANA (EFK)
LOGGING STACK ON WINDOWS
Introduction
Log monitoring and analysis is an essential part of server or container infrastructure
and is useful when handling complex applications. One of the popular logging
solutions is the Elasticsearch, Fluentd, and Kibana (EFK) stack.
Elasticsearch is a real-time, distributed, and scalable search engine that allows for full-
text search and analytics. It is used to index and search through large amounts of data.
It is commonly deployed alongside Kibana, a powerful data visualization dashboard
for Elasticsearch. Kibana allows you to explore the Elasticsearch log data and build
dashboards and queries to gain insight into your application. Fluentd collects,
transforms, and ships the log data to the Elasticsearch backend.
Prerequisites
- Win10 machine installed EFK (IP: 10.0.2.15)
- 2 monitoring machines installed Fluentd are Win10 machine (IP: 10.0.2.4) and Win
server2016 (IP: 10.0.2.4)
1. Install Elasticsearch with .zip on Windows
- Download the .zip archive for Elasticsearch 8.15.0 from:
https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.15.0-windows-
x86_64.zip
- Unzip it with your favorite unzip tool. This will create a folder called elasticsearch-
8.15.0, which we will refer to as %ES_HOME%. In a terminal window, cd to the
%ES_HOME% directory, for instance:
cd C:\elasticsearch-8.15.0
- Run Elasticsearch from the command line
Run the following command to start Elasticsearch from the command line:
.\bin\elasticsearch.bat
When starting Elasticsearch for the first time, security features are enabled and
configured by default. The following security configuration occurs automatically:
Authentication and authorization are enabled, and a password is generated for
the elastic built-in superuser.
Certificates and keys for TLS are generated for the transport and HTTP layer,
and TLS is enabled and configured with these keys and certificates.
An enrollment token is generated for Kibana, which is valid for 30 minutes.
The password for the elastic user and the enrollment token for Kibana are output to
your terminal. Example:
Elasticsearch security features have been automatically configured!
✅ Authentication is enabled and cluster connections are encrypted.
ℹ️ Password for the elastic user (reset with `bin/elasticsearch-reset-password -u
elastic`):
B4MMJq8BlRFCdKf0P3Vz
ℹ️ HTTP CA certificate SHA-256 fingerprint:
63556e1341f969e924c18c2d15d4686d6112604581d40fc44a9634d734602e44
ℹ️ Configure Kibana to use this cluster:
• Run Kibana and click the configuration link in the terminal when Kibana starts.
• Copy the following enrollment token and paste it into Kibana in your browser
(valid for the next 30 minutes):
eyJ2ZXIiOiI4LjE0LjAiLCJhZHIiOlsiMTcyLjE4LjAuMjo5MjAwIl0sImZnciI6IjY
zNTU2ZTEzNDFmOTY5ZTkyNGMxOGMyZDE1ZDQ2ODZkNjExMjYwNDU4
MWQ0MGZjNDRhOTYzNGQ3MzQ2MDJlNDQiLCJrZXkiOiJaT3JXZVpFQjRPc
0dLbG9HdGdKdjpBWXZzUVVXOVJkQ1V6b1BiYnF1LU5RIn0=
ℹ️Configure other nodes to join this cluster:
• Copy the following enrollment token and start new Elasticsearch nodes with
`bin/elasticsearch --enrollment-token <token>` (valid for the next 30 minutes):
eyJ2ZXIiOiI4LjE0LjAiLCJhZHIiOlsiMTcyLjE4LjAuMjo5MjAwIl0sImZnciI6IjY
zNTU2ZTEzNDFmOTY5ZTkyNGMxOGMyZDE1ZDQ2ODZkNjExMjYwNDU4
MWQ0MGZjNDRhOTYzNGQ3MzQ2MDJlNDQiLCJrZXkiOiJZdXJXZVpFQjRP
c0dLbG9HdGdJaTppUzk2TTU0b1RKLUx5TVAzQng5LVdRIn0=
If you're running in Docker, copy the enrollment token and run:
`docker run -e "ENROLLMENT_TOKEN=<token>"
docker.elastic.co/elasticsearch/elasticsearch:8.15.0`
- Check that Elasticsearch is running
You can test that your Elasticsearch node is running by sending an HTTPS request to
port 9200 on localhost:
Open your web browser and go to:
http://10.0.2.15:9200
You should see a JSON response with information about your Elasticsearch instance,
confirming that it's running.
{
"name" : "Cp8oag6",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "AT69_T_DTp-1qgIJlatQqA",
"version" : {
"number" : "8.15.0",
"build_type" : "tar",
"build_hash" : "f27399d",
"build_flavor" : "default",
"build_date" : "2016-03-30T09:51:41.449Z",
"build_snapshot" : false,
"lucene_version" : "9.11.1",
"minimum_wire_compatibility_version" : "1.2.3",
"minimum_index_compatibility_version" : "1.2.3"
},
"tagline" : "You Know, for Search"
}
- Install Elasticsearch as a Windows Service
If you want Elasticsearch to run as a Windows service, open the Command Prompt as
Administrator and navigate to the bin directory.
Run the following command to install the service:
elasticsearch-service.bat install
You can start the service with:
elasticsearch-service.bat start
Stop the service with:
elasticsearch-service.bat stop
2. Install Kibana on Windows
- Download and install the .zip package
Download the .zip windows archive for Kibana v8.15.0 from
https://artifacts.elastic.co/downloads/kibana/kibana-8.15.0-windows-x86_64.zip
- Extract the Package
After downloading, extract the .zip file to a directory of your choice (e.g., C:\kibana).
- Configure Kibana
Navigate to the config directory inside the extracted folder.
Open the kibana.yml file using a text editor.
Configure basic settings if needed. At a minimum, you should specify the
Elasticsearch host. For example:
elasticsearch.hosts: ["http://localhost:9200"]
Other optional configurations include setting the server host and port, enabling SSL,
and more.
- Run Kibana
Open a Command Prompt (cmd) window.
Navigate to the bin directory inside your Kibana folder.
Run the following command to start Kibana:
.\bin\ kibana.bat
Kibana will start, and you should see logs in the command prompt indicating that the
server is running.
Access Kibana. Open your web browser and go to:
http://10.0.2.15:5601
You should see the Kibana dashboard, confirming that it is running.
- Install Kibana as a Windows Service
If you want Kibana to run as a Windows service, you can use third-party tools like
NSSM (Non-Sucking Service Manager) to install it as a service.
Download and install NSSM, then use the following command to install Kibana as a
service:
nssm install Kibana
In the dialog that appears, configure the path to kibana.bat in the Application tab.
3. Install fluentd on 2 machines win 10 and win server 2016
- Install fluent-package
Download the latest MSI installer
Set up fluentd.conf
# Thu thập log từ Windows Event Log
<source>
@type windows_eventlog2
@id windows_eventlog2
channels system,application,security
read_existing_events true
tag winevt.raw
rate_limit 200
<storage>
@type local
persistent true
path C:\opt\fluent\winlog.json
</storage>
</source>
# Thu thập log từ file Nginx access log
<source>
@type tail
path /var/log/nginx/access.log
pos_file /var/log/td-agent/nginx-access.pos
tag nginx.access
format nginx
</source>
<source>
@type tail
path /var/log/nginx/error.log
pos_file /var/log/td-agent/nginx-error.pos
tag nginx.error
format nginx
</source>
<source>
@type tail
path C:/xampp/apache/logs/access.log
pos_file C:/xampp/apache/logs/access.log.pos
tag xampp.apache.access
<parse>
@type apache2
</parse>
</source>
<source>
@type tail
path C:/xampp/apache/logs/error.log
pos_file C:/xampp/apache/logs/error.log.pos
tag xampp.apache.error
<parse>
@type apache2
</parse>
</source>
# Thu thập log qua HTTP
<source>
@type http
port 8888
bind 0.0.0.0
body_size_limit 32m
keepalive_timeout 10s
</source>
# Gửi log từ Windows Event Log tới Elasticsearch
<match winevt.raw>
@type elasticsearch
host 10.0.2.15
port 9200
user elastic
password q+IB32YvHIpR+P56-fNb
logstash_format true
logstash_prefix fluentd-DESKTOP-3N7FSOP
include_tag_key true
include_timestamp true
flush_interval 5s
</match>
# Gửi log từ Nginx access log tới Elasticsearch
<match nginx.access>
@type elasticsearch
host 10.0.2.15
port 9200
user elastic
password q+IB32YvHIpR+P56-fNb
logstash_format true
logstash_prefix fluentd-nginx-access
include_tag_key true
include_timestamp true
flush_interval 5s
</match>
# Gửi log từ HTTP tới Elasticsearch
<match http.logs>
@type elasticsearch
host 10.0.2.15
port 9200
user elastic
password q+IB32YvHIpR+P56-fNb
logstash_format true
logstash_prefix fluentd-http
include_tag_key true
include_timestamp true
flush_interval 5s
</match>
<match xampp.apache.access>
@type elasticsearch
host 10.0.2.15
port 9200
logstash_format true
logstash_prefix xampp-apache-access
include_tag_key true
include_timestamp true
flush_interval 5s
</match>
<match nginx.error>
@type elasticsearch
host 10.0.2.15
port 9200
user elastic
password q+IB32YvHIpR+P56-fNb
logstash_format true
logstash_prefix fluentd-nginx-error
include_tag_key true
include_timestamp true
flush_interval 5s
</match>
<match xampp.apache.error>
@type elasticsearch
host 10.0.2.15
port 9200
logstash_format true
logstash_prefix xampp-apache-error
include_tag_key true
include_timestamp true
flush_interval 5s
</match>
- Launch Fluent Package Command Prompt with Administrator privilege
Open Windows Start menu, and search Fluent Package Command Prompt. In most
environments, the program will be found right under the "Recently Added" section or
"Best match" section.
Fluent Package Command Prompt is basically cmd.exe, with a few PATH tweaks for
Fluentd programs. Use this program whenever you need to interact with Fluentd.
- Run fluentd
Type the following command into Fluent Package Command Prompt with
Administrator privilege:
C:\opt\fluent> fluentd
Now fluentd starts listening to Windows Eventlog, and will print records to stdout as
they occur.
- Run fluentd as Windows service
Fluentd is registered as a Windows service permanently by the msi installer. Since
version 5.0.0, the service does not automatically start after installed. You must
manually start it.
Choose one of your preferred way:
Using GUI
Please guide yourself to Control Panel -> System and Security -> Administrative
Tools -> Services, and you'll see Fluentd Windows Service is listed.
Please double click Fluentd Window Service, and click Start button. Then the process
will be executed as Windows Service.
Using net.ext
> net start fluentdwinsvc
The Fluentd Windows Service service is starting..
The Fluentd Windows Service service was started successfully.
- Install Plugins
Open Fluent Package Command Prompt and use fluent-gem command:
C:\opt\fluent> fluent-gem install fluent-plugin-xyz --version=1.2.3
- Create a data view to receive logs sent from Fluentd
- See logs in “Analytics -> Discover”
- Create a Dashboard in “Analytics -> Dashboards”
SQL Injection Attack Scenario Using EFK to Collect Logs from DVWA
1. Architecture:
Monitoring machine: Install the EFK stack to collect logs from the
DVWA machine.
Target machine: Install DVWA and Fluentd Agent to send logs to the
monitoring machine.
Attacker: Performs SQL Injection attack on DVWA.
2. Setup:
Target machine: Install DVWA, configure Fluentd Agent to send
Apache logs (access/error logs) to the monitoring machine via Fluentd.
Monitoring machine: Install EFK, configure Fluentd to receive logs
from the target machine and forward them to Elasticsearch.
3. Attack:
The attacker inputs a SQL Injection payload (' OR 1=1 --) into
DVWA.
Fluentd collects Apache logs and sends them to Elasticsearch.
4. Monitoring and Detection:
Kibana monitors logs, creating dashboards to detect SQL Injection
patterns in HTTP requests.
Once an attack is detected, set up automatic alerts via email or
notifications.
5. Example log:
192.168.1.100 - - [19/Sep/2024:10:30:15 +0000] "GET
/dvwa/vulnerabilities/sqli/?id=1' OR '1'='1 HTTP/1.1" 200
3456
6. Response:
After detection, the admin can block the attacker's IP and conduct a
security review to strengthen the system.
Brute Force Attack Scenario Using EFK to Collect Logs from DVWA
1. Architecture:
Monitoring machine: EFK stack installed to collect logs from the
DVWA machine.
Target machine: DVWA and Fluentd Agent installed to send logs to
the monitoring machine.
Attacker: Performs a brute force attack on the DVWA login page.
2. Setup:
Target machine: Install DVWA and configure Fluentd Agent to send
Apache access/error logs to the monitoring machine.
Monitoring machine: Install EFK, configure Fluentd to receive logs
from the target machine, and forward them to Elasticsearch.
3. Attack:
The attacker uses a brute force tool (e.g., Hydra) to guess the login
credentials for DVWA by trying multiple username-password
combinations.
Fluentd collects login attempt logs from Apache and sends them to
Elasticsearch.
4. Monitoring and Detection:
Kibana monitors the logs and creates a dashboard to track multiple
failed login attempts from the same IP within a short timeframe, which
is an indication of a brute force attack.
Set up an automatic alert when a certain threshold of failed login
attempts is reached.
5. Example log:
192.168.1.100 - - [19/Sep/2024:10:35:22 +0000] "POST
/dvwa/login.php HTTP/1.1" 200 1234
192.168.1.100 - - [19/Sep/2024:10:35:25 +0000] "POST
/dvwa/login.php HTTP/1.1" 200 1234
192.168.1.100 - - [19/Sep/2024:10:35:27 +0000] "POST
/dvwa/login.php HTTP/1.1" 200 1234
Repeated login attempts from the same IP address in a short time indicate
a brute force attack.
6. Response:
After detecting the brute force attack, the admin can block the attacker's
IP or set rate-limiting rules to prevent further login attempts, and
conduct a review to enhance security.
