Linux Scripting with AWK, Perl, Sed, and Other
Tools
This document provides multiple examples of Linux scripting with various tools such as
AWK, Perl, Sed, Python, and more. Each section will cover key concepts and usage
examples, demonstrating the power and flexibility of these tools in scripting and
automation tasks.
1. AWK - Pattern Scanning and Processing Language
AWK is a versatile tool for text processing, especially useful for working with structured
data like CSV files, log files, or tabular data.
Example 1: Print Specific Columns from a File
Let's say we have a file data.txt with the following content:
Alice 30 Engineer
Bob 25 Developer
Charlie 35 Designer
You can use AWK to print the first and second columns (Name and Age):
awk '{ print "Name:", $1, ", Age:", $2 }' data.txt
Output:
Name: Alice , Age: 30
Name: Bob , Age: 25
Name: Charlie , Age: 35
Example 2: Filtering Rows Based on a Condition
You can filter data based on a condition. For instance, if you want to print lines where the
age (second column) is greater than 30:
awk '$2 > 30 { print $0 }' data.txt
Output:
Name: Charlie , Age: 35
Example 3: Using AWK with File Input
AWK can be used in scripts as well. Here is an AWK script that processes a file and prints a
custom message for each entry:
#!/bin/awk -f
{
if ($2 > 30) {
print $1 " is older than 30."
} else {
print $1 " is younger than or equal to 30."
}
}
You can save this in a file called process.awk and run it with:
awk -f process.awk data.txt
2. Perl - Text Manipulation and Regular Expressions
Perl is known for its powerful text processing and regular expression capabilities. Below are
some practical examples of how to use Perl for Linux scripting.
Example 1: Simple Perl Script
Here’s a simple Perl script that prints "Hello, World!":
#!/usr/bin/perl
# Print Hello, World!
print "Hello, World!\n";
Example 2: Read a File Line by Line
This script opens a file input.txt and reads it line by line, printing each line.
#!/usr/bin/perl
# Open file for reading
open my $file, '<', 'input.txt' or die "Could not open file: $!";
while (my $line = <$file>) {
chomp $line; # Remove newline character
print "$line\n";
}
close $file;
Example 3: Using Regular Expressions in Perl
Perl is particularly strong in handling regular expressions. Here's an example of extracting
email addresses from a string:
#!/usr/bin/perl
[email protected].";
# Regex to find email addresses
while ($text =~ /([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]
{2,})/g) {
print "Found email: $1\n";
}
Output:
Found email:
[email protected] 3. Sed - Stream Editor
sed is a powerful stream editor, used to modify text in files or output streams in an efficient
way. It's particularly useful for simple search and replace tasks.
Example 1: Replace Text in a File
The following example shows how to replace all occurrences of the word "apple" with
"orange" in a file:
sed 's/apple/orange/g' input.txt > output.txt
s/apple/orange/g is the substitution command.
g ensures all occurrences are replaced in each line.
Example 2: Delete Lines Matching a Pattern
This example demonstrates how to delete lines containing the word "error":
sed '/error/d' input.txt > output.txt
/error/ is the pattern to match.
d deletes the matching lines.
Example 3: In-place Editing
You can modify a file in place without creating an output file:
sed -i 's/oldword/newword/g' input.txt
The -i option tells sed to edit the file directly.
4. Python - High-Level Scripting Language
Python is widely used for scripting and automation due to its simplicity and readability. It
also has extensive support for file handling and text processing.
Example 1: Read and Print File Content
This Python script reads the contents of data.txt and prints each line:
#!/usr/bin/python3
with open("data.txt", "r") as file:
for line in file:
print(line.strip()) # Remove newline characters
Example 2: Using Regular Expressions in Python
You can use the re module in Python for regular expression matching. Below is an example
that extracts email addresses from a text:
#!/usr/bin/python3
import re
emails = re.findall(r'[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]
{2,}', text)
for email in emails:
print(f"Found email: {email}")
Output:
Found email:
[email protected]Example 3: Working with CSV Files in Python
Python’s csv module makes it easy to read and write CSV files. Here's an example that
reads a CSV file and prints each row:
#!/usr/bin/python3
import csv
with open('data.csv', mode='r') as file:
csv_reader = csv.reader(file)
for row in csv_reader:
print(row)
5. Using Bash and Other Command-Line Tools
Linux provides powerful command-line utilities that can be used in combination with shell
scripts for automation.
Example 1: Find Files and Perform Operations
Use the find command to locate all .txt files in the current directory and then search for
the word "error" within them using xargs:
find . -name "*.txt" | xargs grep "error"
Example 2: Count Lines in Files Matching a Pattern
This example shows how to count the number of lines that contain the word "error" across
multiple .log files:
grep -r "error" *.log | wc -l
grep -r "error" *.log searches recursively in .log files for the word
"error".
wc -l counts the number of lines returned by grep.
6. jq - JSON Processor
jq is a lightweight and flexible command-line JSON processor, useful for parsing and
manipulating JSON data in shell scripts.
Example 1: Parsing JSON
Here's an example that extracts specific data from a JSON file:
#!/bin/bash
# Parse a JSON file and extract the 'name' field
cat data.json | jq '.user.name'
If the data.json contains the following:
{
"user": {
"name": "John Doe",
"age": 30
}
}
Output:
"John Doe"
Example 2: Filter JSON Based on a Condition
You can also filter data based on a condition. Here’s an example that prints users with an
age greater than 25:
cat data.json | jq '.users[] | select(.age > 25)'
7. Shell Script with Command-line Arguments
Shell scripts can accept arguments from the command line, making them more flexible and
interactive. This feature is useful when you want to pass values to the script without
hardcoding them.
Example 1: Using Positional Parameters
This script accepts two command-line arguments: a name and an age, and prints a greeting
message:
#!/bin/bash
# Check if two arguments are provided
if [ $# -ne 2 ]; then
echo "Usage: $0 <name> <age>"
exit 1
fi
# Assign arguments to variables
name=$1
age=$2
# Print the greeting message
echo "Hello, $name! You are $age years old."
To run:
./greet.sh Alice 30
Output:
Hello, Alice! You are 30 years old.
Example 2: Handling Multiple Command-line Arguments
This script takes any number of command-line arguments and prints each one:
#!/bin/bash
# Print all arguments
for arg in "$@"; do
echo "Argument: $arg"
done
To run:
./args.sh first second third
Output:
Argument: first
Argument: second
Argument: third
8. Combining Multiple Commands in One Script
Sometimes, you need to execute a series of commands in a specific order, or based on
conditions. This can be achieved through sequencing commands in a script.
Example 1: Execute Multiple Commands Sequentially
In this script, multiple commands are executed sequentially:
#!/bin/bash
echo "This is the first command."
echo "This is the second command."
echo "This is the third command."
Example 2: Conditional Command Execution
You can also combine commands based on success or failure:
#!/bin/bash
# Run command1, if successful, run command2
command1 && command2
# Run command1, if it fails, run command3
command1 || command3
9. Creating and Using Functions in Shell Scripts
Functions allow you to organize your scripts into reusable pieces of code. This makes your
scripts more modular and maintainable.
Example 1: Defining a Simple Function
This script defines a simple function to print a greeting message:
#!/bin/bash
# Define the function
greet_user() {
echo "Hello, $1!"
}
# Call the function with an argument
greet_user "Alice"
Output:
Hello, Alice!
Example 2: Function with Return Values
In shell scripting, you can use return values or echo to pass information back from a
function.
#!/bin/bash
# Define the function
add_numbers() {
sum=$(($1 + $2))
echo $sum
}
# Call the function and store the result
result=$(add_numbers 5 7)
echo "The sum is: $result"
Output:
The sum is: 12
10. Handling Errors in Shell Scripts
Error handling is an important part of writing robust scripts. By checking exit codes and
validating input, you can ensure your scripts behave as expected.
Example 1: Using Exit Codes for Error Handling
This script checks if a file exists before proceeding:
#!/bin/bash
# Check if file exists
if [ ! -f "data.txt" ]; then
echo "Error: data.txt not found!"
exit 1
else
echo "File found, proceeding with the task."
fi
Example 2: Handling Command Failures with set -e
You can use set -e to make the script exit immediately if any command fails:
#!/bin/bash
set -e # Exit on any error
echo "This is a test."
non_existent_command # This will cause the script to exit
echo "This will not be printed."
Example 3: Capturing Error Output
Capture both standard output and error output in a variable:
#!/bin/bash
output=$(ls /nonexistent_folder 2>&1) # Capture both stdout and
stderr
echo "Command output: $output"
11. Scheduling Scripts with Cron Jobs
Cron jobs allow you to run scripts on a scheduled basis. This is useful for tasks such as
backups, system maintenance, or periodic reports.
Example 1: Basic Cron Job Syntax
Open the crontab editor with:
crontab -e
Then, add a line to run a script every day at 6 AM:
0 6 * * * /path/to/script.sh
This line means:
0 minutes past
6 AM
Every day (*)
Every month (*)
Every day of the week (*)
Example 2: Running Scripts at Specific Intervals
You can use cron to run scripts at custom intervals. For example, to run a script every 15
minutes:
*/15 * * * * /path/to/script.sh
This line will run the script every 15 minutes.
12. Using Loops in Shell Scripting
Loops allow you to repeat a block of code multiple times. This can be very useful for
iterating over files, directories, or other data.
Example 1: for Loop
This script uses a for loop to iterate over a list of files and prints their names:
#!/bin/bash
for file in *.txt; do
echo "Processing file: $file"
done
Example 2: while Loop
This script uses a while loop to repeatedly ask for user input until they type "exit":
#!/bin/bash
while true; do
echo "Enter a command (or 'exit' to quit):"
read input
if [ "$input" == "exit" ]; then
echo "Goodbye!"
break
else
echo "You typed: $input"
fi
done
Example 3: Nested Loops
You can use nested loops to handle more complex iterations, such as processing multiple
files:
#!/bin/bash
for dir in */; do
echo "Processing directory: $dir"
for file in "$dir"*.txt; do
echo "File: $file"
done
done
13. Using grep for Searching Text
grep is one of the most widely used tools for searching through files or output for specific
patterns.
Example 1: Simple Text Search
This script uses grep to search for the word "error" in a log file:
#!/bin/bash
grep "error" logfile.log
Example 2: Search for a Pattern in Multiple Files
You can search for a pattern across multiple files with grep:
#!/bin/bash
grep "pattern" *.log
Example 3: Using Regular Expressions with grep
grep supports regular expressions. For instance, to search for lines that contain a valid
email address:
#!/bin/bash
grep -E "[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}"
input.txt
14. Combining Multiple Commands with && and ||
You can combine commands in various ways depending on whether the previous command
succeeded or failed.
Example 1: Run Command 2 Only if Command 1 Succeeds
command1 && command2
Command 2 will run only if Command 1 succeeds (exit status 0).
Example 2: Run Command 3 Only if Command 1 Fails
command1 || command3
Command 3 will run only if Command 1 fails (exit status non-zero).
