Advance shell commands
-----------------------------------------------------------------------------------
---------------------------
## 1. `find` — Search Files Powerfully
### What is it?
`find` is used to search files & directories in Linux based on conditions like
name, size, time, type, permission etc.
### When to use it?
- When you need to search for files across directory trees.
- Cleanup unwanted files (temp, logs).
- Automate file finding in scripts.
### Why is it important?
Manual searching is impossible in large directories (like `/var/log`, `/etc`,
`/home/`).
`find` is fast and customizable.
### How to use?
Syntax:
find [path] [condition] [action]
#### Example 1 — Find all `.log` files in `/var/log`
find /var/log -name "*.log"
#### Example 2 — Find files modified in last 1 day
find /home/ubuntu -mtime -1
#### Example 3 — Find and delete empty files
find . -type f -empty -delete
-----------------------------------------------------------------------------------
-----------------------------------------------------------------------------------
--------
## 2. `xargs` — Execute Commands on Output
### What is it?
`xargs` takes input from another command's output and runs another command on each
line.
### When to use it?
- When you have a list of files/names from `find` or `cat` or `grep`.
- To avoid writing manual loops.
### Why is it powerful?
Many commands don't directly support batch operations — `xargs` bridges this gap.
### How to use?
Syntax:
command1 | xargs command2
#### Example 1 — Find large files and delete them
find . -size +100M | xargs rm -i
`-i` = Interactive confirm before delete.
#### Example 2 — Compress multiple files from a list
cat filelist.txt | xargs tar -czvf archive.tar.gz
-----------------------------------------------------------------------------------
-----------------------------------------------------------------------------------
--------
## 3. `awk` — Text Processing Tool
### What is it?
`awk` is a pattern scanning & processing tool. It processes text line by line and
operates on columns (fields).
### When to use it?
- Summarize data from log files.
- Extract specific columns.
- Generate reports.
---
### Why is it useful?
Linux files are usually column-based (`ps`, `df`, `ls`, CSV).
`awk` makes it easy to parse and extract fields.
### How to use?
Syntax:
awk '{ action }' filename
#### Example 1 — Print first and third column from a file
awk '{ print $1, $3 }' data.txt
#### Example 2 — Calculate sum of column 2
awk '{ sum += $2 } END { print sum }' sales.csv
-----------------------------------------------------------------------------------
-----------------------------------------------------------------------------------
--------
## 4. `sed` — Stream Editor
### What is it?
`sed` is used to search, find & replace, insert, or delete lines in files — non-
interactively.
### When to use it?
- Automating configuration changes.
- Bulk replace text in files.
- Remove unwanted lines.
### Why is it needed?
It's faster than manually opening files (especially when used in scripts or
cronjobs).
### How to use?
Syntax:
sed 's/search/replace/g' filename
#### Example 1 — Replace "http" with "https" in config.txt
```bash
sed -i 's/http/https/g' config.txt
```
`-i` = Edit in-place.
#### Example 2 — Delete lines matching "DEBUG"
sed -i '/DEBUG/d' logfile.log
-----------------------------------------------------------------------------------
-----------------------------------------------------------------------------------
--------
## 5. `lsof` — List Open Files
### What is it?
`lsof` shows open files and the processes using them.
### When to use it?
- Troubleshoot port or file lock issues.
- Find which process is using a file.
---
### Why is it critical?
Everything in Linux is a file — even ports, sockets, pipes.
`lsof` helps identify problems quickly.
### How to use?
#### Example 1 — Who is using port 8080?
lsof -i :8080
#### Example 2 — Find open files by a process
lsof -p 1234
-----------------------------------------------------------------------------------
-----------------------------------------------------------------------------------
--------
# 6. `grep` — Search Text Patterns
## What is it?
`grep` searches for patterns (strings or regex) in a file or output.
## When to use?
- Searching error logs.
- Filtering specific info from big outputs.
- Debugging configs.
## Why is it important?
Linux generates lots of logs/text — manual search is slow.
`grep` is fast & customizable with options.
## How to use?
Syntax:
grep [options] "pattern" filename
## Example 1 — Search for "ERROR" in logs
grep "ERROR" /var/log/syslog
## Example 2 — Show line numbers
grep -n "Failed" /var/log/auth.log
## Example 3 — Search recursively in folders
grep -r "listen" /etc/nginx/
## Example 4 — Show 2 lines before & after match
grep -A 2 -B 2 "critical" /var/log/messages
-----------------------------------------------------------------------------------
-----------------------------------------------------------------------------------
--------
# 7. `tar` — Archive & Compress Files
## What is it?
`tar` creates or extracts archive files (.tar, .gz, .bz2).
## When to use?
- Backup files/directories.
- Move data between servers.
- Packaging multiple files.
## Why is it needed?
Multiple files → Single archive → Easy to move/backup.
## How to use?
### Syntax:
tar [options] [archive_name] [files]
## Example 1 — Create tar.gz archive
tar -czvf backup.tar.gz /etc/nginx/
Options:
- `c` = create
- `z` = gzip compression
- `v` = verbose (show progress)
- `f` = filename
## Example 2 — Extract archive
tar -xzvf backup.tar.gz
-----------------------------------------------------------------------------------
-----------------------------------------------------------------------------------
--------
# 8. `nc` (netcat) — Network Tool
## What is it?
`nc` (netcat) is a versatile tool for network communication & debugging.
---
## When to use?
- Check if port is open.
- Test network connectivity.
- Transfer files.
- Create simple chat between systems.
---
## Why is it powerful?
It's like a "Swiss Army Knife" for network troubleshooting.
---
## How to use?
### Syntax:
```bash
nc [options] [target_ip] [port]
```
---
## Example 1 — Check if port is open
```bash
nc -zv google.com 443
```
Options:
- `-z` = scan only (no data)
- `-v` = verbose
---
## Example 2 — Send file over TCP
#### On Receiver:
```bash
nc -l 1234 > received_file.txt
```
#### On Sender:
```bash
nc target_ip 1234 < file_to_send.txt
```
-----------------------------------------------------------------------------------
-----------------------------------------------------------------------------------
--------
# 9. `rsync` — Remote Sync Tool
## What is it?
`rsync` synchronizes files/directories between local & remote systems.
---
## When to use?
- Backup servers.
- Sync data to remote machine.
- Transfer big files efficiently.
---
## Why is it better than `scp`?
- Faster (only sync changes).
- Compression supported.
- Resume broken transfers.
---
## How to use?
Syntax:
rsync [options] source destination
```
---
## Example 1 — Sync local to remote
```bash
rsync -avz /var/www/ user@remote_ip:/backup/
```
Options:
- `a` = archive (preserves permissions, timestamps)
- `v` = verbose
- `z` = compress during transfer
---
## Example 2 — Sync remote to local
```bash
rsync -avz user@remote_ip:/etc/nginx/ /local/path/
```
---
## Example 3 — Exclude files
```bash
rsync -avz --exclude '*.log' /data/ user@remote:/backup/
```
## Summary Table:
| Command | What | When | Why
| Example |
|---------|-------------------|-----------------------------|----------------------
-----|----------------------------------------|
| find | Search files | File management, cleanup | Complex search
filters | `find /var/log -name "*.log"` |
| xargs | Execute on output | Bulk operations | Automate loops
| `find . -name "*.log" | xargs rm -i` |
| awk | Process columns | Reporting, summarizing | Field-based
extraction | `awk '{print $1, $3}' file.txt` |
| sed | Edit files | Text replacement | Auto config change
| `sed -i 's/old/new/g' config.txt` |
| lsof | List open files | Debugging ports | Find resource locks
| `lsof -i :8080` |
| `grep` | Search text | Logs filtering | Fast search
| `grep "ERROR" /var/log/syslog` |
| `tar` | Archive files | Backup | Single package
| `tar -czvf backup.tar.gz /data/` |
| `nc` | Netcat tool | Port testing, transfer | Versatile
troubleshooting | `nc -zv google.com 443` |
| `rsync` | Sync files | Backups, transfers | Fast, efficient
| `rsync -avz /data/ user@remote:/backup/`|
