After writing more and more bash scripts for a client, I\u2019ve decided to write down my thoughts about it.<\/p>\n
This assumes you have some knowledge about bash, as it is not intended as a beginner\u2019s tutorial.<\/p>\n
This is useful for \u2018alpine\u2019 docker images, as some alpine do not include bash.<\/p>\n
The more general you need your script to be, the more you should prefer sh rather than bash. A source of generality can be making your script public (publish it), executing it under multiple environments, making it the installer for other tools, etc.<\/p>\n
Do not assume that the current directory is the place for writing temporary files (or any file, for that matter).<\/p>\n
For temporary files, use (remember to cleanup resources when your script exists – maybe use exit traps)<\/p>\n There are some resources that you need to remove \/ cleanup \/ close at the end of your script. Both when things go well as when they don\u2019t. Think of it as a (java) try-with-resources or try..catch..finally.<\/p>\n Bash offers Taken from here<\/a><\/p>\n An example:<\/p>\n More information, and this example from here<\/a><\/p>\n This is more common with perl than with bash, as most bash installs are placed at You can use Usage:<\/p>\n add these options:<\/p>\n These can be added anywhere, but I usually add them after the shebang (the beginning of the script)<\/p>\n Reference: The set built-in<\/a><\/p>\n Another reference: the inspiration for these options comes from here<\/a><\/p>\n a brief note:<\/p>\n If you want to use a try\u2026catch pattern, disable I usually make my bash scripts as simple as possible (see Limitations), but even then, they fail often while building them.<\/p>\n For that reason, you can enable the \u2018debug\u2019 option permanently:<\/p>\n Or just for one invocation:<\/p>\n Note: your script will get the parameters in the same fashion as if executing A common pattern I use while building scripts is to prepare the command but do not execute it yet:<\/p>\n When I am sure that this is the desired command, usually after trying it manually on the console, I can remove the You can use the previous pattern but as a feature of your script:<\/p>\n When some scripts grow in size and are not a script but an application, being more or less verbose is useful.<\/p>\n See Same with quiet mode, a mode to reduce verbosity.<\/p>\n Same with \u2018raw\u2019 mode, a mode to only print the raw output, maybe for consumption from another script.<\/p>\n Imagine a script that prints the first, second, and third received parameter, then all of them:<\/p>\n The normal invocation:<\/p>\n (everything works as expected)<\/p>\n now let\u2019s try strings (with spaces)<\/p>\n Ok, bash uses spaces to delimit words. Now that we know this, lets be careful.<\/p>\n We want to process some files (with spaces):<\/p>\n A defect appeared: I want \u201cfile 1.txt\u201d to be a parameter, not two.<\/p>\n Let\u2019s imagine a script checking whether a file exists:<\/p>\n Let\u2019s add quotes to the test to make it work with spaces:<\/p>\n In general, be careful with spaces, as they mark the end of the string \/ parameter. Be proactive with quoting. From the google bash guide<\/a>:<\/p>\n Also:<\/p>\n If your script is a one-off thing, or will not suffer churn\/modification, then feel free to discard this tip. On the other hand, if this script will be part of a critical path (e.g., deploying) or will be modified in the future, try to apply the SOLID principles that we apply for other pieces of software.<\/p>\n Especially the SRP (below)<\/p>\n I like to design my scripts by separating concerns or responsibilities.<\/p>\n One typical example: process many files at once:<\/p>\n The main benefit is that iterating the files is something that usually does not fail (just copy paste the script), while the main work is done in Its execution:<\/p>\n For more information on return values and functions in bash, see this article<\/a><\/p>\n Files in bash are read every time you invoke them. So if you separate the This is common knowledge, but it can happen to any of us.<\/p>\n Removing files is a sharp-edged tool, such as Some operating systems now protect To avoid the above mistake,<\/p>\n This will only delete files from the current directory down ( Shell files can also be analyzed statically, (i.e., lint<\/a>). A tool for that is ShellCheck<\/a>.<\/p>\n Shellcheck helps you locate possible errors, bugs, stylistic errors and suspicious constructs in your scripts.<\/p>\n The tool is large enough to warrant another article, but the basic usage is straightforward: run the linter with the shell script as input.<\/p>\n Some example run:<\/p>\n Note: I use the tool with docker (see here<\/a>, official docker image<\/a>)<\/p>\n Usually, I design my scripts:<\/p>\n This is a full example with code to plumb the candidate to the function.<\/p>\n I want to remove all the existing files in a directory that are greater in size than 30 KB. (I know this can be done with First, on the REPL, find all the files:<\/p>\n Find files greater than the desired size:<\/p>\n now, only need to delete the file:<\/p>\n Note: #1 – Notice the First, I make sure that the plumbing code is all correct before executing commands with side effects (e.g., rm). If you are working with delicate data, you can consider working in a docker container.<\/p>\n Then, remove the \u201ctemporary dry-run mode\u201d:<\/p>\n The full script:<\/p>\n This is a full example with a manual invocation to plumb the candidate to the function.:<\/p>\n First, on the REPL, find all the files:<\/p>\n Find files greater than the desired size:<\/p>\n Then, open vim to review, as a way of checking the valid candidates. This is the same process that I realize that the file Now,<\/p>\n then I prefer to edit the file manually than to create a script. Remember, this is a one-off effort. And programs need to be maintained. One-off scripts are to be thrown away, so no maintenance effort.<\/p>\n Hint: the vim command Now, just execute this file:<\/p>\n And your files are processed. Gone, in this case.<\/p>\n Every tool (and metaphor) has its limits. Know when to use a tool and when to change tools.<\/p>\n Small scripts, simple invocations, etc.<\/p>\n One-off tasks are perfect for bash: write code, review effects, throw it away. Don\u2019t plan on reusing it. Although you can keep a collection of snippets for iterating, dealing with spaces, etc.<\/p>\n More than 50-100 bash lines (a rough approximation), I consider a small program already. Maybe start thinking on building a better foundation around it.<\/p>\n With my current knowledge of bash, I feel that some jobs are not appropriate for bash. For example, when dealing with spaces in strings, arrays, complex functions, etc.<\/p>\n For that, I prefer a more powerful language, ideally scripting (so I can get a quick feedback cycle.) I\u2019ve been playing with Perl lately (works very well), Ruby in the past. I\u2019ve heard good things about typescript and go as well.<\/p>\n Perl works well for powerful scripts that don\u2019t need to be tested.<\/p>\n Ruby works well for programs (no longer scripts) that need to be tested.<\/p>\n For my build scripts, I enjoy hitting Now, I can mktemp<\/code>, and for directories mktemp -d<\/code><\/p>\n$ man mktemp\r\nMKTEMP(1) BSD General Commands Manual MKTEMP(1)\r\n\r\nNAME\r\nmktemp -- make temporary file name (unique)\r\n\r\nDESCRIPTION\r\nThe mktemp utility takes each of the given file name templates and \r\nover-writes a portion of it to create a file name. This file name is\r\nunique and suitable for use by the application.\r\n<\/pre>\n
(Exit) Traps<\/h3>\n
trap<\/code><\/a> to perform this task:<\/p>\ntrap arg signal\r\ntrap command signal\r\n<\/pre>\n
function finish {\r\n # Your cleanup code here\r\n}\r\ntrap finish EXIT\r\ntrap finish SIGQUIT\r\n<\/pre>\nDo not hardcode the shell location<\/h3>\n
\/bin\/bash<\/code>.<\/p>\n\/usr\/bin\/env bash<\/code> \/ \/usr\/bin\/env sh<\/code> to spawn a shell.<\/p>\n#!\/usr\/bin\/env bash\r\n\r\n#rest of commands\r\n<\/pre>\n
Options for executing \/ Header<\/h3>\n
set -euxo pipefail\r\n<\/pre>\n
\n
set -e<\/code> stops the execution if a command fails (this is the default behavior in make<\/code>)<\/li>\nset -u<\/code>: Treat unset variables and parameters other than the special parameters \u2018@\u2019 or \u2018*\u2019 as an error when performing parameter expansion. An error message will be written to the standard error, and a non-interactive shell will exit.<\/li>\nset -x<\/code>: debug. Trace the commands on the console<\/li>\nset -o pipefail<\/code>: make the pipe command fail if any of the commands in the pipe fail.\n\n
a|b|c<\/code> when a<\/code> fails, b will execute, the return value will be the one of b<\/code><\/li>\na|b|c<\/code> when a<\/code> fails, b<\/code> will not execute, the return value will be the one of a<\/code><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n-e<\/code> temporarily:<\/p>\nset +e # 1\r\nls NON_EXISTING_FILE # 2\r\nset -e # 3\r\n<\/pre>\n
\n
Debugging<\/h2>\n
Enable tracing \/ debugging mode<\/h3>\n
# Inside the script\r\nset -x\r\n<\/pre>\n
# When invoking the script\r\nbash -x myscript.sh\r\n<\/pre>\n
.\/myscript.sh<\/code>:<\/p>\n$ cat myscript.sh\r\necho $1\r\n$ .\/myscript.sh 1\r\n1\r\n$ bash -x myscript.sh 1\r\n+ echo 1\r\n1\r\n<\/pre>\n
Dry-run while building the script<\/h3>\n
...\r\n# prepare options, decide what to do\r\necho COMMAND_WITH_SIDE_EFFECTS\r\n<\/pre>\n
echo<\/code>:<\/p>\n...\r\n# prepare options, decide what to do\r\nCOMMAND_WITH_SIDE_EFFECTS\r\n<\/pre>\n
Dry-run as another switch<\/h3>\n
\n
echo<\/code> to your final command<\/li>\n<\/ul>\nCOMMAND=\"rm -rf .\/.git\"\r\nif [ $DRY_RUN ]; then\r\n COMMAND=\"echo $COMMAND\"\r\nfi\r\n\r\n$COMMAND\r\n<\/pre>\n
Verbosity levels and other modes<\/h3>\n
curl<\/code> as an example:<\/p>\n$ curl localhost:8080\r\ncurl: (7) Failed to connect to localhost port 8080: Connection refused\r\n$ curl -vvv localhost:8080\r\n* Rebuilt URL to: localhost:8080\/\r\n* Trying ::1...\r\n* connect to ::1 port 8080 failed: Connection refused\r\n* Trying fe80::1...\r\n* connect to fe80::1 port 8080 failed: Connection refused\r\n* Trying 127.0.0.1...\r\n* connect to 127.0.0.1 port 8080 failed: Connection refused\r\n* Failed to connect to localhost port 8080: Connection refused\r\n* Closing connection 0\r\n<\/pre>\n
Using quotes<\/h3>\n
$ cat myscript.sh\r\necho \"first=$1 second=$2 third=$3; all=$@\"\r\n<\/pre>\n
$ .\/myscript.sh 1 2 3\r\nfirst=1 second=2 third=3; all=1 2 3\r\n<\/pre>\n
$ .\/myscript.sh hello world\r\nfirst=hello second=world third=; all=hello world\r\n<\/pre>\n
$ ls file*\r\nfile 1.txt file 2.txt\r\n$ .\/myscript.sh $(ls file*)\r\nfirst=file second=1.txt third=file; all=file 1.txt file 2.txt\r\n<\/pre>\n
$ cat file_exists.sh\r\nif [ -e $1 ]; then # -e is for file exists; see `man test`\r\n echo \"file $1 exists\"\r\nelse\r\n echo \"file $1 does not exist\"\r\nfi\r\n<\/pre>\n
$ ls file*\r\nfile 1.txt file 2.txt file_exists.sh\r\n$ .\/file_exists.sh \"file 1.txt\"\r\n.\/file_exists.sh: line 1: [: file: binary operator expected\r\nfile file 1.txt does not exist\r\n<\/pre>\n
$ cat file_exists.sh\r\nif [ -e \"$1\" ]; then # note the quotes\r\n echo \"file $1 exists\"\r\nelse\r\n echo \"file $1 does not exist\"\r\nfi\r\n$ .\/file_exists.sh \"file 1.txt\"\r\nfile file 1.txt exists\r\n<\/pre>\n
\n
\n
'$PATH' is literally $PATH<\/code><\/li>\n\"$PATH\" is the contents of the variable $PATH<\/code><\/li>\nSOLID<\/h3>\n
Single Responsibility Principle (SRP)<\/h3>\n
$ cat s1.sh\r\n#!\/usr\/bin\/env bash\r\n\r\nfunction find_files {\r\n while IFS= read -r -d '' file; do\r\n files+=( \"$file\" )\r\n done < <(find . -maxdepth 1 -type f -iname \"file*.txt\" -print0)\r\n}\r\n\r\nfunction process_file {\r\n file=\"$1\"\r\n echo \"Will write to file $file\"\r\n}\r\n\r\nfunction main {\r\n declare -a files # this is a global variable inside the script\r\n find_files\r\n for file in \"${files[@]}\"; do\r\n process_file \"$file\"\r\n done\r\n}\r\n\r\nmain\r\n<\/pre>\nprocess_file<\/code>. The two functions have different pace of change, therefore two responsibilities. The latter, I can test manually (on the REPL) until it works, then copy-paste the script (see \u2018How I write my scripts\u2019).<\/p>\n$ ls file*\r\nfile1.txt file2.txt\r\n$ .\/s1.sh\r\nWill write to file .\/file1.txt\r\nWill write to file .\/file2.txt\r\n<\/pre>\n
Hot-swap \/ reload<\/h3>\n
process_file<\/code> function to another file, you can change the contents of it while the long-running main script is working.<\/p>\nBe extra careful with rm<\/h3>\n
DELETE<\/code> in SQL. This is why we SELECT<\/code> the same data set before deleting. Why we ls<\/code> files before rm<\/code>ing them.<\/p>\n#rm -rf \/<\/code> with another flag, but the mistake of #rm -rf $VARIABLE\/*<\/code> where $VARIABLE<\/code> is empty is common enough.<\/p>\n#!\/usr\/env\/bin bash\r\nset -euxo pipefail\r\ncd $VARIABLE #this will fail if $VARIABLE is unbound\r\nrm -rf .\/* # notice the dot (.) before the star\r\ncd - #go back to the previous folder\r\n<\/pre>\n
.\/<\/code>), yet another level of protection.<\/p>\nStatic code analysis<\/h3>\n
$ shellcheck sh1.sh\r\nIn sh1.sh line 22:\r\n destination=${date}-$(basename $file)\r\n ^-- SC2086: Double quote to prevent globbing and word splitting.\r\n\r\nIn sh1.sh line 25:\r\n git add $file\r\n ^-- SC2086: Double quote to prevent globbing and word splitting.\r\n\r\n\r\nIn sh1.sh line 34:\r\n if [[ -z $(which imagemagick) ]]; then\r\n ^-- SC2230: which is non-standard. Use builtin 'command -v' instead.\r\n<\/pre>\nHow I write my scripts<\/h2>\n
\n
process_file<\/code> to receive a single element (i.e., the function passed to map<\/code> \/ iterate). This is the hard part<\/li>\nExample 1: a long-lived script<\/h3>\n
find -exec<\/code> or ls | xargs rm<\/code>, this is just an example for arbitrary logic).<\/p>\n$ ls -lh file*\r\n-rw-r--r-- 1 user group 0B Jul 13 00:50 file1.txt\r\n-rw-r--r-- 1 user group 0B Jul 13 00:50 file2.txt\r\n-rw-r--r-- 1 user group 531K Jul 13 00:07 file3.txt\r\n<\/pre>\n
$ find . -maxdepth 1 -type f -iname \"file*.txt\" -size +30k -print0\r\n.\/file3.txt%\r\n<\/pre>\n
function process_file {\r\n file=\"$1\"\r\n echo \"rm $file\" # 1\r\n}\r\n<\/pre>\necho<\/code> command to protect the real execution<\/p>\nfunction process_file {\r\n file=\"$1\"\r\n rm $file\r\n}\r\n<\/pre>\n$ cat s2.sh\r\n#!\/usr\/bin\/env bash\r\n\r\nfunction find_files {\r\n while IFS= read -r -d '' file; do\r\n files+=( \"$file\" )\r\n done < <(find . -maxdepth 1 -type f -iname \"file*.txt\" -size +30k -print0)\r\n}\r\n\r\nfunction process_file {\r\n file=\"$1\"\r\n rm $file\r\n}\r\n\r\nfunction main {\r\n declare -a files\r\n find_files\r\n for file in \"${files[@]}\"; do\r\n process_file \"$file\"\r\n done\r\n}\r\n\r\nmain\r\n<\/pre>\nExample 2: a one-off script<\/h3>\n
\n
process_file<\/code> to receive a single element (i.e., the function passed to map<\/code> \/ iterate).<\/li>\n$ ls -lh file*\r\n-rw-r--r-- 1 user group 0B Jul 13 00:50 file1.txt\r\n-rw-r--r-- 1 user group 0B Jul 13 00:50 file2.txt\r\n-rw-r--r-- 1 user group 531K Jul 13 00:07 file3.txt\r\n-rw-r--r-- 1 user group 531K Jul 13 00:07 file_SUPER_IMPORTANT_DO_NOT_DELETE.txt\r\n<\/pre>\n
$ find . -maxdepth 1 -type f -iname \"file*.txt\" -size +30k > candidates.txt\r\n$ cat candidates.txt\r\n.\/file3.txt\r\n.\/file_SUPER_IMPORTANT_DO_NOT_DELETE.txt\r\n<\/pre>\n
git rebase --interactive<\/code> offers: a CLI command to rebase based on your editor.<\/p>\nfile_SUPER_IMPORTANT_DO_NOT_DELETE.txt<\/code> should not be deleted. So I remove that, manually.<\/p>\n$ cat candidates.txt\r\n.\/file3.txt\r\n<\/pre>\n
%s\/^\/rm \/<\/code> will insert at the beginning of the line the command rm<\/code> that we need. The command %s\/$\/;\/<\/code> will append a semicolon at the end of the line. It\u2019s not needed for this example, but as a reminder. This replacement can also be done with sed<\/code>\/awk<\/code>.<\/p>\n$ cat candidates.txt\r\nrm .\/file3.txt;\r\n<\/pre>\n
bash candidates.txt\r\n<\/pre>\n
Limitations<\/h2>\n
When is bash enough<\/h3>\n
When it is too much for bash<\/h3>\n
Complex\/multi-stage scripts<\/h4>\n
Parameter autocompletion<\/h4>\n
<tab><\/code> for auto-completion of the goals. Bash does not offer that out of the box (but can be performed using programmable completion<\/a>). Make, on the other hand, offers goal autocompletion out of the box:<\/p>\n.PHONY: build\r\nbuild:\r\n .\/gradlew build\r\n<\/pre>\n
make b<TAB><\/code> and it will suggest make build<\/code><\/p>\nOther resources<\/h2>\n
\n