Long Live Linux Forensics!

The Linux Forensics Team

$ whoami $ whoami $ whoami

Ali Hadi Brendan Brown Victor Griswold

Professor @ Champlain Alum @ Champlain College Senior @ Champlain College
College {CDF & Cybersecurity} {Digital Forensics and
{Computer and Digital Reverse Engineer - InvictusIC Cybersecurity}
Forensics, Cybersecurity}
@br_endian @vicgriswold
@binaryz0ne
2
 Scenario(s)
- Learning Linux Forensics Through Case Studies -

#1 #2 #3
Linux User Artifacts Compromised Web More Hidden
(GNOME/XFCE) Server... Processes...

3
 Case #1: Linux User Artifacts...
✘ Looking at some test cases for the following User Activity artifacts:
○ Thumbnails
○ Trash
○ Recently Used Documents
✘ Performed on Ubuntu 18.04.1 LTS (GNOME) and Kali 2020.1b (Xfce4)
✘ Comparing results between the two test environments
✘ Comparing Freedesktop (XDG) Standards to findings

4
 Thumbnails
What we Expect...

✘ Thumbnails will appear in ~/.cache/thumbnails/[normal | large | fail]

✘ Thumbnails will be in PNG format
✘ Thumbnails will contain at least the Thumb::URI and Thumb::MTime
PNG tEXt keys
✘ Thumbnails file name will be the MD5 hash of the URI found in
Thumb::URI with .png appended

5
Xfce4 Thumbnails

Only 3 thumbnailed
exiftool -S -ThumbURI -ThumbMTime . | grep -C1 [dir_name]

11 Images
Test dir = ~/home/Desktop/Cats/
6
GNOME Thumbnails

Same 11 Images

All 11 thumbnailed!

Test dir = ~/home/Desktop/Cats/

7
GNOME v. Xfce4 Thumbnail Tags

8
GNOME + Xfce4 Thumbnail Timestamps

Last Viewed

First Viewed

sudo debugfs -R 'stat [<inode_num>]' /dev/...

9
 Trash Folder
What we Expect...

✘ Expect to find a folder called ‘Trash’ in each user’s ~/.local/share

✘ Expect to find at least 2 directories in “Trash”: files and info
○ ‘files’ directory will contain all the files and directories that have
been moved to trash
○ ‘info’ directory will contain *.trashinfo files related to files in ‘files’
directory
Note: Files only moved to trash through GUI interaction (not through ‘rm’
commands)

10
GNOME and Xfce4 Trash Folders

11
GNOME and Xfce4 Trash Folders (cont.)

Trash directory is created

when first file is trashed

“expunged” directory is
created and timestamps
update on trash empty
12
Trash Folders Timestamps

Before Trashed

Trashed files try to maintain the

original files atime and mtime.
Files ctime will indicate when it was
deleted and should match the
corresponding .trashinfo files
.trashinfo
crtime.
13
 Recently Used
What we Expect...

✘ An XML file with similar format Example from freedesktop standards:

to this <?xml version="1.0"?>
<RecentFiles>
✘ Found in ~/.recently-used <RecentItem>
<URI>[Link]
<Mime-Type>text/plain</Mime-Type>
<Timestamp>1028181153</Timestamp>
This is not exactly what we find... <Private/>
<Groups>
<Group>Recent File Test</Group>
</Groups>
</RecentItem>
</RecentFiles>

14
What we find recently used GNOME
File found in: ~/.local/share/[Link]

15
What we find recently used Xfce4
File found in: ~/.local/share/[Link]

16
Recently Used Breakdown

17
 #2: Compromised Web Server...

✘ You might be wondering that you’ve already seen this case

before?
○ Web Server Environment (Apache)
○ Unusual network activity was noticed

✘ Previous cases our team showed how to track down a threat

actor using different system artifacts, especially logs!

✘ Today we are dealing with a different scenario!

18
 Case Constraints!!!

✘ No Web Server Logs...

✘ Dealing with Hidden Processes...
✘ You can’t acquire system memory...
✘ You can’t do memory forensics...

✘ This is where this talk comes in...

19
 Apache Backdoor Module!!!
✘ But does that mean there is no one using Apache2?

[1] RGDoor found by Unit42 at Palo Alto Networks

[2] Apache Backdoor found by Welivesecurity at ESET
[3] Backdoors in XAMP by Juan Fernandez at Tarlogic
[4] Apache PoC Module by Vlad Rico @RicoVlad
20
 How it Works!!!
✘ Sending reverse shell using the malicious Apache2 module

✘ Threat actor now has root access...

21
 Normal Network Connections??!!

✘ Start by checking the network services...

✘ Everything looks normal here; right?

22
 Normal Network Connections??!!

✘ Think again; check the “avahi-daemon”

✘ Why is avahi*:
○ Not bound to UDP port 5353?
○ Communicating with TCP port 443?!

23
 Wait / What... BASH not Avahi*??!!

✘ Listing Open TCP connections using lsof…

Findings: PID = 2106 | user = root | FD = 0,1,2,12 | Device = 32914

Protocol = TCP | Dest. IP Address = [Link]
24
 Checking Open Files!!!
✘ UNIX Domain Socket used for process communication...
✘ STREAM 32194 and 32273

25
✘ PPID? !! Process Context??!!

✘ Process = Systemd ✘ Let’s check the FDs,

✘ User = root sockets, and pipes used...

✘ We can see the socket (s) file in /tmp

26
 FDs, Sockets, and Pipes!!!

✘ Process file descriptors and how they are mapped to the sockets
and pipes

✘ We can map them with what we’ve previously seen...

27
 Apache Instances!!!
✘ We have two Apache2
instances; why?!

✘ We can also see the running

Bash

28
 CGroups & Loaded Modules!!!

✘ CGROUPS for the Apache2 unit…

✘ Checking the loaded apache modules…

we can see the authz_net string!
○ Turned out to be a module!
29
 Apache2 Instance!!!
✘ PPID = 2034
✘ This is the apache2 instance
with the number of threads…
✘ Suspicious module also
loaded here

30
Unix Socket Streams!!!
✘ PPID = 2103
✘ The weird module
/tmp/mod_authz_net
we’ve seen before!
✘ This is the malicious
Apache2 instance which is
channeled with the
reverse shell using bash...

31
Static Checks!!! Network
Functions
✘ If we check the strings and/or the
imported symbols, we can see some
interesting features related to
network, system, and process activity
Backdoor
functions
✘ We can also see features related to the
backdoor itself...

System
Functions
Note: output is filtered
Process
32 Functions
 Shout(s)-Out!!!...

✘ Thanks to all those out there that keep reminding the community of
not to KILL a process, but dump it from memory first, especially if it
does not exist on disk anymore!

✘ Craig H. Rowland, @CraigHRowland

○ [Link]

33
 Anonymous Processes

- They exist...

- Process spawned from chunk of memory

- Never on filesystem -> no FS artifacts

- Creating & writing directly to FD

34
 What we’re examining

NC reverse shell

- Leveraging memfd_create()

- Non-native ELF pushed in

remotely and written to mem
Process Behavior

Fork -> Execute -> Die

- Why do we care?

- Parent data in /procfs

- Children? Commands?

36
 Monitoring
Execsnoop - leverages ftrace to record exec() calls

Pkg: perf-tools-unstable

Forkstat - records the following events:

Pkg: forkstat

IR Toolkit !∞= forensically sound

*Tsurugi - DFIR Linux

37
 Monitoring - execsnoop
execsnoop-perf -a 16 -r > [Link]
Shell creation:

- Sshd called
- Motd
- Patch state
- Update check
- Mounting FS

Process executions:

38
 Monitoring - forkstat
forkstat -e all -x -S > [Link]
How is forkstat
different?

- Datapoints:
1 vs. 12
- No exec()?
- Process Ancestry
- Procmon

39
 Monitoring
- Forkstat -

- UID info

- ssh without pseudo

terminal

- Verbose binary
(command) calls

- Witness the
fork > execute > exit cycle

40
 Session ID

Consistent session ID

Parent
->
Child

41
 Environ + Others
Most Relevant Procfs artifacts:

/proc/$$/environ
/proc/$$/cmdline
/proc/$$/task/$$/children
/proc/$$/status
/proc/$$/io

- Process data transfer

- Arguments passed to process

- Parent-child relationship & proc

name

42
 Environ + Others

Most Relevant Procfs artifacts: (/bin/dash)

- SSH host / client address + ports
/proc/$$/environ - Language
/proc/$$/cmdline - Session ID
/proc/$$/task/$$/children - User
/proc/$$/status - UID
/proc/$$/io - PWD
- Homedir
- Shell
- Underscore variable *
(binary / script last executed as arguments)
43
 Summation

- How to find data from process with no FS attribution

- How processes behave in memory
- Why they can be hard to track normally
- Tools to help you follow the chain of execution
- Artifacts in procfs found to be most relevant

44
 thanks!

Any questions?
You can find us
@binaryz0ne | @br_endian | @vicgriswold

45
 Credits & References...
Special thanks to all the people who made and released these awesome
resources for free:
✘ Presentation template by SlidesCarnival and Photographs by Unsplash
✘ Credits to Vlad Rico @RicoVlad for the Malicious Apache2 Module
○ Module here: [Link]
✘ Craig Rowland @CraigHRowland, for his awesome Linux Forensics work.
✘ Stuart @MagisterQuis, for his write ups on anonymous processes
✘ Lynx for their process forking image.
✘ Sorry if we missed someone!

46