Splunk and Sysmon

Summary
Terminology
Syslog
Sysmon
Installing Sysmon for Linux
Register Microsoft Key & Feed
Registering the Microsoft Key
Adding the Microsoft Package Feed
Install Sysmon for Linux
Accept EULA & Run Installer With Default Config
Sysmon Configuration File
Create the Configuration File
Rebuild Sysmon With New Config
Configure Logging for Sysmon
Why Give syslog Ownership of sysmon.log?
Ownership and Permissions
Why Configure syslog for Sysmon?
Centralized Log Management
Steps for Setting Up syslog for Sysmon
Summary
Forward Sysmon Logs to Splunk
Setting up a New Index & Source Type
What is a Splunk Index?
How to Create a Splunk Index
What is a Splunk Sourcetype?
Why Install an Add-on/App Instead of Creating a Sourcetype Manually?
How to Install the Sysmon for Linux App
Restart Splunk
Configure the Splunk Forwarder
Restart the Splunk Forwarder
Splunk Time!

Summary

Splunk and Sysmon 1

 In this guide, we will be setting up a Splunk container on our Ubuntu 22.04 VPS,
which is configured to listen for logs on port 9997. The Splunk Forwarder is
already installed on the VPS and is configured to ingest several types of logs,
including Nginx access & error logs, MySQL error logs, Fail2ban, UFW, and
authentication logs. We will also install and configure Sysmon on the VPS. Finally,
we'll modify our inputs.conf file so that the Splunk Forwarder can feed these
additional logs into our Splunk container.

Terminology
Syslog
Overview:

Syslog is a standard protocol used for message logging. It facilitates the

separation of the software that generates messages, the system that stores these
messages, and the software that reports and analyzes them. This separation
enables a centralized approach to log management, allowing diverse systems and
applications to send their log data to a central repository.
Key Features:

1. Network Protocol:
Syslog operates over a network protocol, typically using UDP on port 514,
although it can also use TCP for more reliable transmission. This allows log
messages to be sent from remote systems to a central log server.

2. Versatility:
Syslog can collect log data from a wide variety of devices including routers,
switches, firewalls, servers, and applications. This versatility makes it a
valuable tool for network and system administrators who need to monitor the
health and security of their infrastructure.

3. Message Structure:
A syslog message consists of a priority value, a timestamp, the hostname or IP
address of the source device, and the actual log message. The priority value
combines the facility and severity level of the message, helping to categorize
and prioritize logs.

Splunk and Sysmon 2

 4. Centralized Log Management:
By centralizing log data, syslog simplifies the monitoring and analysis of
system events, security incidents, and performance metrics. It enables the
correlation of events across multiple systems, aiding in troubleshooting and
forensic investigations.

Typical Use Cases:

System Monitoring:
Collecting system logs from various servers to monitor performance and
detect issues.

Security Auditing:
Gathering logs from firewalls, intrusion detection systems, and other security
devices to audit security events.

Compliance:
Centralizing logs to meet regulatory requirements for data retention and audit
trails.

Sysmon
Overview:

Sysmon (System Monitor) is a Windows system service and device driver that
remains resident across system reboots to monitor and log system activity to the
Windows event log. It is designed to provide detailed information about system
activities, which is particularly useful for security monitoring and forensic analysis.

Key Features:

1. Detailed Logging:
Sysmon provides comprehensive logging of various system activities,
including process creation, network connections, and changes to file creation
times. These logs offer deep insights into the behavior of the system and its
applications.

2. Persistence Across Reboots:

Unlike some logging mechanisms that only run when the system is active,
Sysmon is designed to remain operational across system reboots. This

Splunk and Sysmon 3

 ensures continuous monitoring and logging, capturing events that occur
during system startup and shutdown.

3. Attacker's Perspective:
Sysmon is developed with an understanding of the tactics, techniques, and
procedures used by attackers. It captures details that are often leveraged in
advanced attacks, such as process command lines, hashes of executables,
and parent-child process relationships.

4. Windows Event Log Integration:

Sysmon logs events directly into the Windows event log. This integration
makes it easy to collect and analyze Sysmon logs using standard tools like the
Windows Event Viewer, or central logging solutions like Splunk and Azure
Sentinel.

Typical Use Cases:

Incident Response:
Providing detailed logs that help in identifying and responding to security
incidents.

Malware Analysis:
Capturing the behavior of malicious software, including the processes it
spawns and the network connections it initiates.

Threat Hunting:
Enabling security analysts to proactively search for signs of compromise by
analyzing detailed system activity logs.

Installing Sysmon for Linux

Register Microsoft Key & Feed
Registering the Microsoft key and feed is a necessary step to enable your Linux
system to download and install packages from Microsoft repositories. This
process involves two main actions:

Registering the Microsoft Key

Splunk and Sysmon 4

 Linux package management systems (like apt on Ubuntu) use GPG keys to verify
the integrity and authenticity of the packages. By registering the Microsoft key,
you ensure that packages downloaded from the Microsoft repository can be
verified against this key, confirming they haven't been tampered with.

Adding the Microsoft Package Feed

This action involves adding Microsoft's package repository to your system's list of
package sources. The package feed (repository) is where the system will look to
find packages for installation and updates. Adding this feed ensures that your
system can find and install the Sysmon package and other Microsoft-provided
packages.

wget -q https://packages.microsoft.com/config/ubuntu/$(lsb_release -rs)/packag

sudo dpkg -i packages-microsoft-prod.deb

Install Sysmon for Linux

sudo apt update && sudo apt install -y sysmonforlinux

💡 While the command technically “installs” the Sysmon binary, the process
& service haven’t been executed yet, so Sysmon is not yet running on
our system.

Accept EULA & Run Installer With Default Config

We need to accept the End User License Agreement (EULA) and run the installer
so that it creates required directories and files. Keep in mind that this will have
everything running, but Sysmon will not be configured to monitor anything yet.
We’ll take care of that in a bit.

sysmon -accepteula -i

Splunk and Sysmon 5

 Sysmon Configuration File
Now that we’ve accepted the EULA and run the installer, we should have a new
directory created. This is where Sysmon will live and operate from ( /opt/sysmon/ ).
We can now create a configuration file and rebuild Sysmon with it.

Create the Configuration File

Use Nano to create the file /opt/sysmon/sysmon-config.xml and paste the following
contents in it.

nano /opt/sysmon/sysmon-config.xml

Toggle Config File

<Sysmon schemaversion="4.81">
<EventFiltering>
<!-- Capture all process creation events -->
<ProcessCreate onmatch="include">
<Rule name="ProcessCreation">
<Image condition="contains">/bin/</Image>
<Image condition="contains">/usr/bin/</Image>
<Image condition="contains">/sbin/</Image>
<Image condition="contains">/usr/sbin/</Image>
</Rule>
</ProcessCreate>

<!-- Capture process termination events -->

<ProcessTerminate onmatch="include">
<Rule name="ProcessTermination">
<Image condition="contains">/bin/</Image>
<Image condition="contains">/usr/bin/</Image>
<Image condition="contains">/sbin/</Image>
<Image condition="contains">/usr/sbin/</Image>
</Rule>
</ProcessTerminate>

Splunk and Sysmon 6

 <!-- Capture file creation events -->
<FileCreate onmatch="include">
<Rule name="FileCreation">
<Image condition="contains">/bin/</Image>
<Image condition="contains">/usr/bin/</Image>
<Image condition="contains">/sbin/</Image>
<Image condition="contains">/usr/sbin/</Image>
</Rule>
</FileCreate>

<!-- Capture network connection events -->

<NetworkConnect onmatch="include">
<Rule name="NetworkConnections">
<Image condition="contains">/bin/</Image>
<Image condition="contains">/usr/bin/</Image>
<Image condition="contains">/sbin/</Image>
<Image condition="contains">/usr/sbin/</Image>
</Rule>
</NetworkConnect>

<!-- Capture raw read access events (indicative of low-level system acces
<RawAccessRead onmatch="include">
<Rule name="RawAccessRead">
<Image condition="contains">/bin/</Image>
<Image condition="contains">/usr/bin/</Image>
<Image condition="contains">/sbin/</Image>
<Image condition="contains">/usr/sbin/</Image>
</Rule>
</RawAccessRead>

<!-- Capture process access events -->

<ProcessAccess onmatch="include">
<Rule name="ProcessAccess">
<SourceImage condition="contains">/bin/</SourceImage>
<SourceImage condition="contains">/usr/bin/</SourceImage>
<SourceImage condition="contains">/sbin/</SourceImage>

Splunk and Sysmon 7

 <SourceImage condition="contains">/usr/sbin/</SourceImage>
</Rule>
</ProcessAccess>

<!-- Capture file delete events -->

<FileDelete onmatch="include">
<Rule name="FileDeletion">
<Image condition="contains">/bin/</Image>
<Image condition="contains">/usr/bin/</Image>
<Image condition="contains">/sbin/</Image>
<Image condition="contains">/usr/sbin/</Image>
</Rule>
</FileDelete>
</EventFiltering>
</Sysmon>

Rebuild Sysmon With New Config

sysmon -accepteula -c /opt/sysmon/sysmon-config.xml

Configure Logging for Sysmon

Why Give syslog Ownership of sysmon.log?
Ownership and Permissions
1. syslog's Role in Logging:

syslog is a system service responsible for collecting and storing log

messages generated by various applications and services on a Linux
system.

It operates with its own user and group ( syslog ) to manage log files
securely and consistently.

2. File Ownership:

Splunk and Sysmon 8

 When we set up a new log file for Sysmon, such as /var/log/sysmon/sysmon.log ,
we need to ensure that syslog can write to it.

By setting the ownership of the log file to syslog:syslog , we ensure that the
syslog service has the necessary permissions to write log entries to this
file.

3. Security and Access Control:

Changing ownership to syslog:syslog prevents unauthorized users from

tampering with the log files.

It ensures that only the syslog service, which is trusted and runs with
appropriate privileges, can write to these logs.

Why Configure syslog for Sysmon?

Centralized Log Management
1. Separation of Concerns:

Sysmon (System Monitor) is responsible for generating detailed system

activity logs, such as process creation, network connections, and file
changes.

syslog, on the other hand, is responsible for collecting, storing, and

managing these logs in a centralized location.

2. Log Routing:

Configuring syslog to handle Sysmon logs ensures that all log entries
generated by Sysmon are collected and stored consistently with other
system logs.

This makes it easier to manage and analyze logs from different sources in
a unified manner.

3. System Compatibility:

By using syslog, which is a standard logging system on Linux, we can

leverage existing infrastructure for log management, analysis, and
forwarding to other systems (e.g., Splunk).

Splunk and Sysmon 9

 It avoids the need for custom solutions or additional services to handle the
log files.

Steps for Setting Up syslog for Sysmon

1. Create the Log Directory and File:
Ensure the log directory exists and create the log file:

💡 The -p flag in the mkdir command stands for "parents". It allows

the creation of nested directories in a single command, even if
some or all of the parent directories don't exist yet. If the
directories already exist, mkdir -p will not return an error, making it
a useful option for scripts.

sudo mkdir -p /var/log/sysmon

sudo touch /var/log/sysmon/sysmon.log

2. Change Ownership:
Change the ownership of the log file to syslog:

💡 The -R option in the chown command stands for "recursive". It is

used when you want to change the owner of a directory and all
the files and subdirectories within it. Without the -R option, the
chown command would only change the ownership of the

directory itself, leaving the ownership of files and subdirectories

within it unchanged.

sudo chown -R syslog:syslog /var/log/sysmon

3. Set Permissions:
Set appropriate permissions to allow syslog to write to the log file:

Splunk and Sysmon 10

 sudo chmod 644 /var/log/sysmon/sysmon.log

4. Configure syslog:

Create a new syslog configuration file to direct Sysmon logs to the correct file:

sudo nano /etc/rsyslog.d/30-sysmon.conf

💡 The configuration file, /etc/rsyslog.d/30-sysmon.conf , is created to direct

the Sysmon logs to the correct file. The syslog service uses this
configuration file to determine where Sysmon logs should be
stored.

The number 30 in the filename is used to set the order in which

the configuration files are loaded. Files in the /etc/rsyslog.d/
directory are loaded by rsyslog in ASCII order. Therefore, by
prefixing the filename with 30, we can control the load order to
ensure that the Sysmon configuration is loaded at the appropriate
time in relation to other configuration files. If there are
dependencies or specific load order requirements among multiple
configuration files, the numbering scheme helps manage this
order.

Add the following lines to ensure syslog captures Sysmon logs:

if $programname == 'sysmon' then /var/log/sysmon/sysmon.log

& stop

Splunk and Sysmon 11

 💡 This is a configuration line written in Rsyslog config language.
Here's a breakdown of what it does:

: This is a condition that checks if the

if $programname == 'sysmon'

name of the program that generated the log message is

'sysmon'. The $programname is a property in Rsyslog that
contains the name of the program/app that originated the log
message. If the program name is indeed 'sysmon', then the
subsequent action is carried out.

: If the condition is true (i.e., the log

then /var/log/sysmon/sysmon.log

message came from 'sysmon'), then the log message is

written into the /var/log/sysmon/sysmon.log file. This is the action
that is executed when the condition is met.

& stop : The '&' character here is a continuation marker that

links it to the previous line, meaning that if the previous
condition was met, it should also perform this action. The
'stop' action means that once the log message has been
written into the 'sysmon.log' file, Rsyslog should stop
processing this message. No further rules will be processed,
and the message will not be written to any other logs. This is
useful for preventing duplicate log entries in different files.

So, in summary, this line in the configuration tells Rsyslog to write

all log messages from 'sysmon' into a specific log file
( /var/log/sysmon/sysmon.log ), and then stop processing the message.

5. Restart syslog:
Restart the syslog service to apply the changes:

sudo systemctl restart rsyslog

Summary

Splunk and Sysmon 12

 By giving syslog ownership of the sysmon.log directory and file, we ensure that the
syslog service can properly manage and write to the log files. Configuring syslog
to handle Sysmon logs integrates Sysmon’s detailed system activity logs into the
existing centralized logging infrastructure. This setup improves security,
consistency, and ease of log management, making it easier to analyze and
respond to system events and potential security incidents.

Forward Sysmon Logs to Splunk

Setting up a New Index & Source Type
What is a Splunk Index?
In Splunk, an index is a repository of data, or a collection where Splunk stores
event data. The indexing process is the process by which Splunk captures raw
data, breaks it down into individual events, and then stores it in an index. The
index allows you to correlate events, set up alerts, create visualizations, and much
more.
Each index in Splunk has its own set of configuration files, which control the
behavior of the data in that index. These configurations define how long the data
is stored, how much disk space it can use, and where the data resides, among
other settings.

How to Create a Splunk Index

Below are the steps to create a new index in Splunk:

1. Access Splunk’s Indexes Page:

From your Splunk instance, go to
Settings > Indexes . This will open the Indexes page, which shows all existing

indexes.

2. Add a New Index:

Click on
New Index . This will open the Add New page.

3. Fill in the Name:

Enter the

Splunk and Sysmon 13

 . This should be something descriptive and relevant to the data you'll
Index Name

be storing in this index. The rest of the settings can be left on default.

4. Save the New Index:

Click
Save . Your new index should now appear in the list of indexes on the Indexes

page.

Remember, the index name will be used in the inputs configuration on your Splunk
forwarder, so make a note of the exact name you've used.

What is a Splunk Sourcetype?

In Splunk, a "sourcetype" is a field that identifies the data's format, allowing
Splunk to apply the correct formatting and analysis during indexing. It is
essentially a category that you assign to your data, which helps Splunk to parse
the data correctly and optimize the search performance.
By identifying data with a particular sourcetype, you can maintain consistency in
the way the data is indexed and searched. This enables you to extract fields,
assign timestamps, and apply event breaking rules to your data in a structured
manner.

Why Install an Add-on/App Instead of Creating a Sourcetype

Manually?
While it is possible to create a sourcetype manually, it can be a complex and time-
consuming process, especially when dealing with complicated datasets like those
produced by Sysmon.
Installing an add-on or app, like Sysmon for Linux, simplifies this process by
creating the sourcetype and field extractions for us. This preconfigured
sourcetype ensures that the data is indexed and formatted correctly, allowing you
to get the most out of your Sysmon logs in Splunk.

Apps like Sysmon for Linux are made by experts who understand the data
structure and how to get the most out of it in Splunk. By using these add-ons, you
can leverage their expertise and reduce the risk of misconfiguration.

How to Install the Sysmon for Linux App

Splunk and Sysmon 14

 You can download the Sysmon for Linux app from Splunkbase and upload it
manually to your Splunk instance. Here are the steps:

1. Download the App:

Visit the
Sysmon for Linux app page on Splunkbase and download the app.

2. Access Splunk’s Apps Page:

From your Splunk instance, go to
Apps > Manage Apps . This will open the Apps page, which shows all existing apps

and add-ons.

3. Install the App from File:

Click on
Install app from file . This will open a dialog box.

4. Choose File:
Click
Choose File and select the Sysmon for Linux app ( .spl file) you downloaded from

Splunkbase.

5. Upload and Install:

Click
Upload to install the app. Your new app should now appear in the list of apps

and add-ons.

💡 Installing this add-on has not given you a brand new source type called
sysmon:linux

Restart Splunk
To ensure that everything is working as expected, it’s best practice to restart
Splunk after installing an app. Since you’re operating in a Docker container, you
can simply SSH into your server and issue the command:

docker restart splunk

Splunk and Sysmon 15

 Wait a few minutes until Splunk is back up and running. In the meantime, we can
configure our forwarder to send the new Sysmon logs to this new index and its
associated source type.

Configure the Splunk Forwarder

💡 The file /opt/splunkforwarder/etc/system/local/inputs.conf is a crucial configuration file

used by the Splunk Forwarder.
Splunk Forwarder is a component of Splunk that collects logs from
different sources and forwards them to a Splunk indexer for
consolidation and analysis.
The inputs.conf file specifically is responsible for specifying the data
inputs that you want the forwarder to monitor. This can include log files,
network events, scripts, or other types of data sources.
In this file, you define the specifics of what data to collect, such as the
file path of a log file, a network port, or a script command. Moreover, you
can specify settings for each data input, such as the index it should be
sent to, the source type it should be assigned, the host value, and other
metadata.
By editing and managing this file, you control what data the Splunk
Forwarder collects and how it handles that data before forwarding it to
the indexer.

sudo nano /opt/splunkforwarder/etc/system/local/inputs.conf

Then add the following content at the bottom of the file:

# Monitor the Sysmon log file.

[monitor:///var/log/sysmon]
disabled = false

Splunk and Sysmon 16

 index = sysmon
sourcetype = sysmon:linux

Restart the Splunk Forwarder

Since we modified a configuration, we’ll need to restart the associated service.

sudo /opt/splunkforwarder/bin/splunk restart

Splunk Time!
With the configuration completed, we should now be able to see logs in Splunk
from Sysmon. Here’s a fancy SPL query that filters for interesting commands
executed on the VPS and observed by Sysmon:

index="sysmon" sourcetype="sysmon:linux"
(
CommandLine="*/etch/passwd*" OR
CommandLine="*/etc/shadow*" OR
CommandLine="*john*" OR
CommandLine="*/etc/sudoers*" OR
CommandLine="*/etc/ssh/ssh_config*" OR
CommandLine="*/etc/ssh/sshd_config*" OR
CommandLine="*hydra*" OR
CommandLine="*nmap*" OR
CommandLine="*nikto*" OR
CommandLine="*sqlmap*" OR
CommandLine="*msfconsole*" OR
CommandLine="*/var/log/messages*" OR
CommandLine="*/etc/issue*" OR
CommandLine="*/etc/motd*" OR
CommandLine="*tcpdump*" OR
CommandLine="*wireshark*" OR
CommandLine="*iptables*" OR
CommandLine="*ufw*" OR

Splunk and Sysmon 17

 CommandLine="*/etc/rc.local*" OR
CommandLine="*crontab*" OR
CommandLine="*/etc/cron.d*" OR
CommandLine="*/etc/cron.daily*" OR
CommandLine="*/etc/cron.hourly*" OR
CommandLine="*/etc/cron.weekly*" OR
CommandLine="*/etc/cron.monthly*" OR
CommandLine="*/etc/init.d*" OR
CommandLine="*chmod 777*" OR
CommandLine="*service ssh*" OR
CommandLine="*curl*" AND NOT CommandLine="curl --max-time 30 --fail --
CommandLine="*scp*" OR
CommandLine="*sftp*" OR
CommandLine="*base64*" OR
CommandLine="*openssl*" AND NOT CommandLine="/opt/splunkforwarder/b
CommandLine="*/root/.ssh/authorized_keys*" OR
CommandLine="*/home/*/.ssh/authorized_keys*"
)
| regex CommandLine!="(?i).*docker.*"
| regex CommandLine!="(?i).*ufw-user-.*"
| regex CommandLine!="(?i).*run-parts --report /etc/cron.hourly.*"
| regex CommandLine!="(?i).*/bin/sh -c.*run-parts --report /etc/cron.hourly.*"
| regex CommandLine!="(?i).*run-parts --report /etc/cron.daily.*"
| regex CommandLine!="(?i).*test -x /usr/sbin/anacron || ( cd / && run-parts --rep
| regex CommandLine!="(?i).*--color=auto.*"
| table CommandLine
| dedup CommandLine

Splunk and Sysmon 18