Computer System Administration

(ECC 4209)

Lecture 7
(Emergency Tools)

[email protected]
1
 Contents
1. Recovering broken Linux systems
2. Controlling resources with Linux live-boot drives
3. Recovering data from damaged storage media
4. Manipulating an inaccessible file system

2
 Introduction
• This lecture introduces key tools from the Linux recovery suite
• Learn how to use a live-boot device to load a fresh copy of
Linux mount the drive that’s giving trouble by either
– update corrupted configuration files so can boot normally again
– rescue whatever data can be recovered before repurposing or
destroying the damaged drive
• See how the files of a nonfunctional system can be brought to
life and run within their own virtual environment, can do things
like changing a user’s forgotten password
• There’s a lot that can go wrong when hardware and software
working in a single box and expect them to play nicely together
– Murphy’s Law: “Anything that can go wrong will go wrong”

3
 Examples of Catastrophic Events
• Your computer boots, the hard drive is working, but Linux
doesn’t load or start
• Your computer boots, but not quite sure whether the hard
drive is fully functional
• Everything works, but a software problem or lost password
prevents from logging in to Linux
• The specific problem will determine the plan of action to get
right back in business
• Figure 6.1 illustrates some diagnostic and recovery options,
most of which will be discussed later in this lecture

4
Linux won’t load

Select older kernel version in GRUB

Enter rescue mode

Edit boot parameters in GRUB

Boot using Live USB and edit file

system

Drive won’t mount

Boot using Live USB

Test system memory
Can’t log in
Test partitions
Log in to alternate (sudo)
account Recover data from
and recover password
Boot using Live USB
partitions
and use chroot

Figure 6.1: Linux Diagnosis and Recovery Options

 Working in recovery/rescue mode
• Cannot log in to Linux normally?
– Perhaps the boot process unexpectedly stops before displaying the
login screen
• But wait: if Linux won’t load, how to launch those tools?
– even if Linux won’t load all the way to a normal command prompt, often
it’ll get to the GRUB menu
• From there (Figure 6.2), use the up and down arrow keys and
then Enter to select a Linux kernel running in recovery mode
• Before taking full advantage of those tools, however, need to
understand what GRUB is and how it works

6
Working in recovery/rescue mode
(continue)

Figure 6.2 The GRUB Advanced Options menu of an Ubuntu installation showing links
7
to
both current and older kernel versions, along with options for launching in recovery
 The GRUB bootloader
• GRUB is the GNU GRand System power up
Unified Bootloader
• Bootloader is the code an BIOS or UEFI
Identifies hardware environment
OS uses to bring itself to life
when it’s powered on as Mounts MBR
Master Boot Record partition
illustrated in Figure 6.3
– When a computer powers up,
GRUB displays menu
firmware instructions and executes an image kernel
embedded in the basic system
hardware identify the network, Kernel
storage, and memory resources Mounts root partition
that are available
– This was done through the Hand-off
to init or systemd
BIOS system on older
computers and, more recently, Figure 6.3: The Linux Boot Sequence
using UEFI
8
 The GRUB bootloader
• Once the system finds a partition containing a Master Boot
Record (MBR), it loads the contents into active memory
• On Linux systems, the MBR partition contains a number of
files that, when run, present one or more loadable kernel
image boot configurations
– choose to load any of those configurations from the GRUB bootloader
menu
• Please note that usually GRUB will load its default image
automatically without asking, unless a previous session fails to
load
• To force the GRUB menu to appear, press the right Shift key
as your computer boots

9
 Using recovery mode on Ubuntu
• Once Ubuntu is loaded in recovery mode, menu of tools is
shown that address common boot-time problems (Figure 6.4).
– It’s worth trying each in turn, it might address the root problem
• The clean option, for instance, removes unused files if suspect
the trouble stems from a full disk
• The dpkg option attempts to fix any broken APT-based
software packages (might need networking)
• The root option opens a root command-line shell session
• In general, using a simple shell session for recovery rather
than a full GUI desktop makes a lot of sense.
– That’s because running less complicated services, the more likely to fix
• Once manage to get a working command prompt, can start
poking around to identify and fix the problem
10
Using recovery mode on Ubuntu

Figure 6.4: The Ubuntu Recovery menu with links to some basic diagnostic
and repairtools, along with the option of opening a shell session as root

11
 Using recovery mode on Red Hat
• The GRUB menu for Red Hat offers a rescue kernel at boot
time rather than a recovery mode
• This kernel doesn’t include a tool menu like Ubuntu’s, but it will
similarly drop into a single-user shell session as root
• Figure 6.5 shows the rescue boot option in Red Hat GRUB
• Once the rescue selection on your Red Hat machine is chosen,
gets into single user root sessions with some useful tools
• But first, why not take a page from the Ubuntu playbook and
manually apply some of the tools from the automated Recovery
menu

12
Using recovery mode on Red Hat

Figure 6.5: CentOS Linux offers a rescue kernel for booting directly to a single-user
shell session for troubleshooting a damaged system

13
Finding command-line rescue tools
• If there’s code running the menu, it must already exist
somewhere within the Ubuntu file system
• Use locate to find them:

• If navigate over to the /lib/recovery-mode/ directory, the

recovery-menu file is the script that displays the menu
interface in Figure 6.3
• The options/ directory contains files for executing each of the
menu items. 14
– fsck will check and fix, if possible, any broken file systems
Finding command-line rescue tools
(continue)
• Can take a look at each of the scripts in the options/ directory
to see how they work
• Here are the contents of the fsck:
– Note the way the script is nicely documented (using the # character) to
explain what’s going on:

15
Finding command-line rescue tools
(continue)
• Can try this at home
– Manually run the clean script on a Ubuntu machine.
– Then try carefully editing the recovery-menu script (make a backup
copy first) /lib/recoverymode/
– Change something simple, like the menu title or one of the script
descriptions
– Then reboot your machine and, from the GRUB menu, go into
Recovery Mode to see what the recovery environment looks like
• With some variations and exceptions, should be able to put
those scripts to good use elsewhere, including on Red Hat

16
 Building a live-boot recovery drive
• ISO OS images employed for your VirtualBox VMs back in
Lecture 3 can also be written to a CD or USB drive and used
to boot a live session of the OS
– can load fully functioning Linux sessions without installation
– to confirm that a particular Linux distribution will run happily on their
hardware before trying to install it
– others will run live sessions as a secure way to maintain their privacy
while engaged in sensitive activities like online banking
• It turns out that those live boot drives are also a fantastic tool
for system rescue and recovery
– Remember our second disaster scenario from earlier?
• Your computer boots but not quite sure whether the hard drive
is fully functional
– Plugging a live-boot drive into a troubled computer and launching Linux
with all of its administration tools can help 17
 System rescue images
• If already have a DVD or USB drive with a full Linux system (e.g
Ubuntu), that’ll the simplest solution because most of the
software that are needed will come preinstalled
– assuming there’s a network connection, can always install other
packages during a live session
• Alternatively, here some specialty images for system rescue
• Boot-Repair can be installed on already-launched live session
– https://help.ubuntu.com/community/Boot-Repair
• GParted Live image brings the full-featured and powerful
GParted partition editor to a live CD or USB
– GParted can also be installed and used from any normal Linux session.
• SystemRescueCd, is a lightweight image built on the Gentoo
Linux distribution
– As it comes with loads of useful system recovery tools,
SystemRescueCd makes a great USB to carry around 18
 Writing live-boot images to USB
drives
• To make sure that the ISOs are not being tampered (with trojan
horse,etc) check their provided hashes values against the hash
verification of the ISO itself.
• Sample of Ubuntu 17.04 is provided in Figure 6.7
• This example generates the SHA256 hash for the
SystemRescueCd image from the same directory to which it
was downloaded:

19
 Ubuntu 17.04 Image Hashes (SHA256)
(from: http://releases.ubuntu.com/17.04/SHA256SUMS)

B718c7fb1066589af52f4ba191775b0fb514c5fb6fa7d91367043e1db06d8a0b *ubuntu-17.04-desktop-amd64.iso
dd201dc338480d1f6ad52e4c40abbc9bfbf12eba71aeac8a87858a94951b002a *ubuntu-17.04-desktop-i386.iso
ca5d9a8438e2434b9a3ac2be67b5c5fa2c1f8e3e40b954519462935195464034 *ubuntu-17.04-server-amd64.img
ca5d9a8438e2434b9a3ac2be67b5c5fa2c1f8e3e40b954519462935195464034 *ubuntu-17.04-server-amd64.iso
dd7879be4e2f9f31672f6e7681ecafeccfac294afd8ca1b04b78bc37fb44291c *ubuntu-17.04-server-i386.img
dd7879be4e2f9f31672f6e7681ecafeccfac294afd8ca1b04b78bc37fb44291c *ubuntu-17.04-server-i386.iso

Release Image type

Hash Version
Architecture
Environment

Figure 6.7: SHA256 hashes for the various images of Ubuntu 17.04
 Writing live-boot images to USB
drives (Debian, Ubuntu)
• If creating live-boot USB that will run Debian, Ubuntu, or a
derivative, use the Ubuntu Startup Disk Creator
• The Creator tool is available from the regular GUI menus
(Figure 6.8)
• Select an .ISO from somewhere on your hard drive and a target
USB (or CD) where the image will be written

Figure 6.8: Ubuntu’s Startup Disk

Creator
with an Ubuntu Server 16.04 image
and
target USB drive selected

21
 Writing live-boot images to USB
drives (Other Distro)
• For building live-boot device built from a different distro (e.g.
Red Hat derivatives) need to use dd

• If the image will be written to a USB drive and working on an

Ubuntu host, modify the image by adding an MBR to the .ISO
– So BIOS and UEFI firmware will know what to do with it (see Figure 6.9)
• On Ubuntu hosts, use isohybrid for that image modification
– apt package containing isohybrid is called syslinux-utils
• Once that’s installed, move to the directory containing the
downloaded image and run isohybrid using the image
filename as the sole argument:
– # apt update
– # apt install syslinux-utils
– $ cd ~/Downloads
22
– $ isohybrid systemrescuecd-x86-5.0.2.iso
 3a. Write image to
drive using
1. Download Creator Tool
ISO image (if for Ubuntu)
A2abdaf5750b09886cedcc
5233d91ad3d1083e10380e
555c7ca50849befbf487

2. Confirm file HASH is correct

M
B
R

3b. Add Master Boot 4. Write image to

Record to ISO image drive using dd
(all distros)

Figure 6.9 The steps necessary for writing a working live-boot USB image
 Writing live-boot images to USB
drives (Other Distro)
• The next steps should work for any Linux distributions
• Identify the system designation for your target device:
– df lists all the recognized file systems along with their designations

24
 Writing live-boot images to USB
drives (Other Distro)
• In this example, there’s a file system mounted on the
Kingston USB device called /dev/sdb1
– The device itself is known as /dev/sdb
• If want to write the image to an optical drive like a CD or
DVD, then get its designation through lsblk, which stands
for list block devices and the drive itself must be writable
– Here DVD drive is known as sr0:

25
 Writing live-boot images to USB
drives (Other Distro)
• First unmount the drive itself so dd can get full access and
then write the archive
• In this example, the systemrescuecd-x86-5.0.2.iso image and
wrote it to the drive at /dev/sdb
• Extra careful, yping sda instead of sdb (in this particular case)
will irretrievably overwrite the host file system!
• Also make sure there’s nothing important on the USB drive,
as that will definitely disappear:

26
 Writing live-boot images to USB
drives (Other Distro)
• It takes some time for dd to finish writing the image to USB
device, but, when it’s done, plug the drive into a computer,
power it up, and enter a live session
• This is assuming that your computer is configured to boot
from a USB drive
• If it isn’t, can force it to boot to a selected device this one time
by entering the Boot menu during the boot process
• Each PC manufacturer designates its own keys for such
things (often displaying a list of keystroke options)
– pressing one of F1, F9, or F12 early in the boot sequence if unsure
• Could also enter the setup utility (either BIOS or UEFI) to
permanently set the order used to boot devices.
• Accessing the setup utility might also happen through a range
27
of keys either F2, F10, Del or Enter
 Putting live-boot drive to work
• Can do a lot of things with this Linux-in-a-pocket drive that
have been created for examples
– Testing system memory
– Repair damaged partitions
– Recovering files from a damaged file system

28
 Testing system memory
• If experienced sudden and unexpected system crashes, a
likely culprit is physical memory (RAM)
– Like all hardware, RAM will eventually fail
• The problem is that cannot properly test RAM for errors while
it’s being used by a running OS
– instead need catch it before the OS loads
• As Figure 6.10 shows, one of the menu items that will be
displayed after booting to an Ubuntu drive is Test Memory

29
 Testing system memory
• Selecting Test Memory takes to the Memtest86+ program
(Figure 6.11).
– That program runs multiple scans of your RAM, displaying any errors
it finds
– But if it returns no errors after running for at least a few full passes, the
memory is probably not the cause of your trouble

Figure 6.11:The Memtest86+ tool shows the location and kind of any 30

errors in
 Repair damaged partitions
• A partition is really metadata pointing to the location on a
physical disk occupied by a file system.
– If disk data is somehow corrupted and the exact addresses of a
partition’s start and end points are changed or lost, then the file
system on the partition will become unreachable
– And if a file system is unreachable, then the data on the file system is
as good as gone
– Boot up SystemRescue drive if cannot access partition
• The default boot option will open a special commandline shell
(Figure 6.13)
• SystemRescue provides some useful orientation information
along with the command line, including networking and
text-editing basics (Figure 6.14).
– both Vim and Nano are available
– typing startx will load a fast and light GUI desktop 31
 Repair damaged partitions
(continue)

Figure 6.13 The Boot Options menu displayed during SystemRescue

startup. 32
Note the detailed explanations for each option at the bottom of the
 Repair damaged partitions
(continue)

Figure 6.14 The SystemRescue shell. Note the text editors available by
default
(including Nano) and also that typing startx launches a graphical 33
desktop
 Repair damaged partitions
•
(continue)
If need network access to download or install more tools,
perhaps, or to transfer data,
– net-setup
– select the correct interface, wired or wireless
• If it’s wireless, enter WiFi router’s SSID and its encryption key
– In most cases, let DHCP auto-detect the network config
• With a damaged partition to worry about, the primary concern
right now is recovery
• If there’s any chance that the physical disk itself might be
failing, then the first task must be to secure its data.
• For that, use dd at the SystemRescue command prompt to
create a perfect copy of the partition in its current state and
save it to a healthy disk of equal or greater capacity
34
 Repair damaged partitions
(continue)
• After running lsblk to confirm the designations of both partitions,
(failing disk is /dev/sda and empty drive is /dev/sdc1):
– #dd if=/dev/sda of=/dev/sdc1
• After that check whether can save original copy with TestDisk
– #testdisk
– need to specify session events logged, which disk want to recover, and
which partition type to expect to find
• Use TestDisk to analyze the disk, looking for existing partitions
(see Figure 6.15)
• After discovering and appropriately marked the damaged
partitions
– write the changes to disk
– should be able to successfully boot once again
35
 Repair damaged partitions

Figure 6.15: The TestDisk Partition Repair page, where partitions

discovered
Through Analyze can then be edited and recovered using the other
tools 36
 Recovering files from a damaged
file system
• If cannot fully rescue your disk, perhaps can revive your
partitions just enough to allow access to the files but not
enough to reliably boot from the disk
– then the new priority is to save as many important files as possible
• The simplest approach is to use the regular Linux file system
management tools from any kind of live-boot session
• If not sure about the partition’s designation, run lsblk
• It’s possible that your partition won’t yet be accessible as a
file system because it hasn’t been mounted
• If, in fact, it doesn’t show up among the results of a df
command, then have to mount it, meaning assign it to a
location within a file system where it can be accessed
37
 Recovering files from a damaged
file system
• Can fix that by creating a new directory where partition will be
mounted and then using the mount command
• Lets’ say temporary mount point directory in /run/, but any
unused location will do (like /media/ or /mnt/)
• Let’s assume, as before, that the partition is called /dev/sdc1:
– # mkdir /run/temp-directory
– # mount /dev/sdc1 /run/temp-directory
• Any healthy files from the stricken file system will be available
to copy or edit from within temp-directory
• They’ll probably also automatically show up in any GUI
desktop file management tool

38
 Recovery Using DDRESCUE
• Didn’t work? Time to pull out the heavy guns the
data-recovery tool, ddrescue
– copies files between file systems
– it also analyzes your files and attempts to repair any that are broken
• If it’s not already installed on the live boot, ddrescue comes
as part of SystemRescue by default
– identify the troubled partition (/dev/sdc1 in this example)
– the partition where the image will be saved
– the name and location of a log file where a record of operation events
can be written
• Having a log file allows ddrescue to continue a stopped
operation rather than starting again from scratch

39
 Recovery Using DDRESCUE
• The example uses an external drive (greater capacity than
the source drive) mounted to a directory /run/usb-mount:

• Test the rescue by using dd to write the backup image to a

new, empty drive (/dev/sdd/) and then booting the
system to the new drive:
– # dd if=backup.img of=/dev/sdd
• Even if still can’t boot the partition, it may now be possible to
use it to reliably access important individual files

40
 File Recovery Using Photorec
• PhotoRec is another tool that can help to grab files off a
damaged drive
– # photorec
• Once file system that needs rescuing has been identified,
which file types need to be included, and the location to of the
files to be saved, the files are saved to numbered directories
using the recup_dir.? prefix:
– $ ls recup_dir.12
– f1092680.elf
– f0668624.png
– f0853304.xml
– f0859464.xml
– f0867192.txt
– f0977016.gz
– f0982184.xml
– [...] 41
 File Recovery Using Photorec
(continue)
• Something to keep in mind about PhotoRec is that the files
are all given new numbered filenames while retaining their
original file extensions
• This can sometimes make finding individual files difficult, but
much better than losing them altogether

42
 Password recovery:
mounting a file system using chroot
• The problem of keeping track of sufficiently complex
passwords can be largely solved by using a good password
vault like KeePass2 or LastPass
• The problem of overusing passwords can be solved by
implementing a single sign-on solution such as Kerberos
• What happens to the to the sysadmin that every now and
then they forget a password
• That won’t be a problem if there’s another admin with sudo
power who can log in to the server and run passwd to create
a new password for the user:

43
Password recovery: Mounting a file
system using chroot
• If the forgetful user was the only sysadmin with an account on
that machine, use chroot
• Use a live-boot drive to power up the server that cannot get
into, run lsblk to determine the designation of your main
root partition on the server’s hard disk
• Mount the root partition to a temporary directory:
– #mkdir /run/mountdir/
– #mount /dev/sdb1 /run/mountdir/
– #chroot /run/mountdir/
root@ubuntu:/#
• Run commands as though working on a running version of
the physical hard drive
• Use passwd to reset the old password
– after typing exit to shut down your chroot session, reboot the machine
44
(without the live-boot USB)
 Summary
• Linux recovery modes provide access to administration tools
useful for repairing systems that won’t boot normally.
• Live-boot drives allow booting of Linux distros independently of
the file systems on a computer’s physical drives
• Purpose-build distros like SystemRescueCD are lightweight
versions of Linux that come preinstalled with a full range of
rescue tools.
• Damaged partitions can sometimes be restored using tools like
TestDisk.
• Data from damaged partitions can sometimes be recovered
using tools like ddrescue and PhotoRec.
• File systems can be mounted and administered using a virtual
process called chroot.
45
 Key Terms
• GRUB is a bootloader that manages the images to be used in
the Linux boot process.
• A hash (checksum) is a cryptographically generated value that
can be checked against a master copy to confirm an image’s
authenticity.
• A partition’s Master Boot Record (MBR) for a CD/DVD will be
different than for a USB, requiring special attention for creating
live-boot USBs.
• The tool chroot opens virtual root shells within mounted file
systems.

46
 Command-line Review
• sha256sum systemrescuecd-x86-5.0.2.iso calculates the SHA256
checksum of a .ISO file.
• isohybrid systemrescuecd-x86-5.0.2.iso adds a USB-friendly
MBR to a liveboot image.
• dd bs=4M if=systemrescuecd-x86-5.0.2.iso of=/dev/sdb &&
sync writes alive-boot image to an empty drive.
• mount /dev/sdc1 /run/temp-directory mounts a partition to a
directory on the live file system.
• ddrescue -d /dev/sdc1 /run/usb-mount/sdc1-backup.img
/run/usb-mount/ sdc1-backup.logfile saves files on a damaged
partition to an image named sdc1-backup.img and writes events to a log file.
• chroot /run/mountdir/ opens a root shell on a file system.

47
 References
• Linux in Action, David Clinton:
– https://www.manning.com/books/linux-in-action
• Learning Modern Linux, Michael Hausenblas:
– https://www.oreilly.com/library/view/learning-modern-linux/978109810893
9/
• Linux Administration Best Practices, Scott Alan Miller:
– https://www.packtpub.com/product/linux-administration-best-practices/97
81800568792
• Linux Cookbook: Essential Skills for Linux Users and System &
Network Administrators (2nd Edition)
– https://www.oreilly.com/library/view/linux-cookbook-2nd/9781492087151/
• AWS Cookbook by John Culkin and Mike Zazon
– https://www.oreilly.com/library/view/aws-cookbook/9781492092599/
• Hands-on Booting: Learn the Boot Process of Linux, Windows,
and Unix
48
– https://link.springer.com/book/10.1007/978-1-4842-5890-3
 References
• Linux Service Management Made Easy with systemd, Donald A.
Tevault:
– https://www.packtpub.com/product/linux-service-management-made-easy
-with-systemd/9781801811644
• TestDisk Documentation version 7.1:
– https://www.cgsecurity.org/testdisk.pdf
• Sudo Mastery, 2nd Edition:
– https://mwl.io/nonfiction/tools#sudo2

49