Security Event Generation and Collection using SIEM

1
Bilal Ahmed
ELK

2
Bilal Ahmed
 ELK
• ELK Stack is a powerful tool for collecting, analyzing, and visualizing log data. It consists of
three open-source components:
• Elasticsearch
• Logstash
• Kibana.
• Open-source tools built by Elastic.
• We will have an overview of each component and how they work together to help you
manage your logs more effectively.

3
Bilal Ahmed
 ELK
• Elasticsearch is a search engine that stores and indexes data
• Logstash is a data processing pipeline that collects, enriches, and transports data
• Kibana is a visualization and analytics platform that allows you to interact with and explore
your data.
• Together, these three components form a powerful tool for managing log data.
• Now that we have a global overview of ELK Stack, let’s go deep into each of its specific
components.

4
Bilal Ahmed
• Elasticsearch is a flexible document database engine that specializes in the ability to write
interesting queries to get insight into your data. It is a common tool used by many
applications to facilitate search capabilities.
• It also exposes an HTTP API and has a default port of 9200.
• As part of the ELK stack, it is used to store your logs for long-term retention.

5
Bilal Ahmed
• It acts like a super-fast search engine, capable of handling all kinds of data, such as text,
numbers, and logs. Imagine it as a smart librarian who can instantly find any information you
need, even from massive collections of data.
• One of the key features of Elasticsearch is its speed; it can process millions of records in
seconds, making it ideal for tasks that require quick results.
• Elasticsearch is not just about searching; it also helps you understand your data by analyzing
trends, patterns, and behaviors, providing valuable insights.
• Elasticsearch is scalable, capable of handling small projects for individuals or massive
datasets for large businesses without any trouble.

6
Bilal Ahmed
• Common use cases for Elasticsearch include:
• Powering search functions on websites.
• Monitoring systems for errors.
• Analyzing data for business decisions.
• Elasticsearch is versatile and user-friendly, making it useful for businesses, data analysts, and
IT teams alike. Its ability to process and analyze data in real-time makes it perfect for
applications like:
• Monitoring website performance.
• Detecting errors in systems.
• Powering advanced search engines.
7
Bilal Ahmed
• Logstash is a tool for consuming and transforming log data provided in many different
formats into a clean and consistent format.
• Its functionality encompasses tasks such as monitoring for JSON sent via UDP, analyzing
Nginx or Apache log files, and parsing syslog logs.
• It even enables advanced data transformations that necessitate external data, like converting
IP addresses into geolocations. Additionally, there are numerous open-source libraries
available in various programming languages for delivering data to Logstash. Among the most
widely used libraries in the Node.js environment is winston-logstash.
8
Bilal Ahmed
• It acts as a pipeline that collects, processes, and sends data to locations like Elasticsearch or
databases for storage or analysis.
• Logstash excels in environments requiring the management of large amounts of data from
multiple sources, such as web servers, applications, or devices.

9
Bilal Ahmed
 10
Bilal Ahmed
• Kibana is a data visualization and exploration tool that is part of the ELK stack.
• It provides users with web interface that enables them to create interactive dashboards,
reports, and visualizations that allow them to gain insights into their data.
• Kibana has a wide range of features, including advanced charting options, data filtering and
aggregation, and geospatial analysis.
• Allow users to create custom charts and graphs that help them to identify trends and patterns
in their data, as well as to detect anomalies or outliers.
• Collaboration features allow users to share their visualizations and dashboards
11
Bilal Ahmed
 12
Bilal Ahmed
 13
Bilal Ahmed
• Beats is a lightweight data shipper in the ELK Stack ecosystem.
• It works like a small helper that collects and sends data from your systems to Elasticsearch or
Logstash.
• Each Beat is designed to handle specific types of data, making it a modular and efficient way
to gather information from different parts of your system.

14
Bilal Ahmed
These Beats are installed on your servers or devices to act as agents that send data to the
ELK Stack for analysis and visualization.

15
Bilal Ahmed
• One of the key advantages of Beats is its simplicity and efficiency. Since
each Beat is lightweight, it doesn’t use a lot of system resources, ensuring
your devices or servers can run smoothly while collecting data.
• This makes it ideal for large-scale setups where multiple systems are being
monitored simultaneously.
• Beats also integrates seamlessly with Logstash, where you can preprocess
and transform the data before sending it to Elasticsearch.
• This adds flexibility, allowing you to clean and format the data as needed
for better insights.

16
Bilal Ahmed
• Kafka is a powerful tool used alongside the ELK stack. Kafka comes into play as a messenger
between different parts of a system. Imagine it as a postal service that delivers data from one
place to another.
• In the context of the ELK stack, Kafka sits between the data sources and Logstash. It collects
data from various places, like applications or servers, and streams it in real-time to where it
needs to go.

17
Bilal Ahmed
• Kafka adds flexibility to the ELK stack.
• Since it decouples data producers from consumers, you can add or change data sources
without disrupting the whole system.
• This makes it easier to scale and adapt as your data needs grow.

18
Bilal Ahmed
HOW ELK STACK WORKS

19
Bilal Ahmed
 HOW ELK STACK WORKS

• ELK stack uses Beats, Kafka, Logstash, Elasticsearch, and Kibana in a collaborative
workflow.
• First, Beats acts as lightweight agents installed on servers or systems. These agents
collect log data, metrics, or other system events and forward it to Kafka. Kafka is a
message broker that ensures all data flows smoothly and can handle large volumes
by queuing data before sending it further down the pipeline.
• This step helps manage scalability and reliability.
20
Bilal Ahmed
 HOW ELK STACK WORKS

• The collected data from Kafka is passed to Logstash. Logstash processes this data,
performing filtering, enrichment, and transformation to make it suitable for analysis.
• It structures the data in a way that Elasticsearch can index and search efficiently.
21
Bilal Ahmed
 HOW ELK STACK WORKS

• Next, Elasticsearch stores the data in a distributed and scalable manner. It is a search
engine capable of indexing and querying large amounts of data at high speed. The
structured and indexed data is now ready for exploration and analysis.
22
Bilal Ahmed
 HOW ELK STACK WORKS

• Finally, Kibana is used to visualize and analyze this data interactively. It connects with
Elasticsearch to create dashboards, graphs, and reports that help users make sense of
the information. This end-to-end workflow provides insights into system performance
and potential issues, empowering organizations to monitor and act effectively.
23
Bilal Ahmed