Scripting with Bash and Python

Compact Course @ Max-Planck

Tobias Neckel
February 16 - 26, 2015

Tobias Neckel: Scripting with Bash and Python

Compact Course @ Max-Planck, February 16 - 26, 2015 1
General Information

Organization
• About 1.5 hours lecture
• Slides and live demonstrations
• About 1.5 hours tutorials
• We suggest tasks to practise
• You can work on whatever you want
• I’ll try to answer any questions
• Breaks :-)

Tobias Neckel: Scripting with Bash and Python

Compact Course @ Max-Planck, February 16 - 26, 2015 2
General Information

Organization
• About 1.5 hours lecture
• Slides and live demonstrations
• About 1.5 hours tutorials
• We suggest tasks to practise
• You can work on whatever you want
• I’ll try to answer any questions
• Breaks :-)
Outline
• 4 days Bash
• 5 days Python
• One day off in second week

Tobias Neckel: Scripting with Bash and Python

Compact Course @ Max-Planck, February 16 - 26, 2015 2
General Information

About this course

• Scripting in Bash and Python (and beyond)
• Your background?
• Prior knowledge?
• Feedback!!

Tobias Neckel: Scripting with Bash and Python

Compact Course @ Max-Planck, February 16 - 26, 2015 3
General Information

About this course

• Scripting in Bash and Python (and beyond)
• Your background?
• Prior knowledge?
• Feedback!!
More information
• Slides and tutorials on website:
http://www5.in.tum.de/wiki/index.php/Scripting_with_
Bash_and_Python_-_2015

Tobias Neckel: Scripting with Bash and Python

Compact Course @ Max-Planck, February 16 - 26, 2015 3
Contents
Bash (ca. 4 days):
• Overview, most important basics:
• Files and directories: ls, dir, cd, mkdir, find, touch, . . .
• Variables, manipulation of variables, arrays, special variables
• Control structures, functions, parameters
• Shell configuration
• Aliases and variables
• Configuration examples for bash (bashrc)
• Pipes and wildcards

Tobias Neckel: Scripting with Bash and Python

Compact Course @ Max-Planck, February 16 - 26, 2015 4
Contents (cont.)
Bash (ca. 4 days):
• Advanced topics
• Regular expressions
• String-manipulation, regular expressions: grep, cut, sed,
head, . . .
• Operating on files
• Grouping, subshells
• awk
• Working remote
• ssh, scp, rsync, unison, nx . . .
• Process management
• Visualization with gnuplot
• Parsing, preprocessing data
• Interaction with shell tools

Tobias Neckel: Scripting with Bash and Python

Compact Course @ Max-Planck, February 16 - 26, 2015 5
Contents (cont.)
Scripting with Python . . . and beyond (ca. 5 days)
• Overview
•Interpreted vs. compiled programming languages
•Python command shell vs. executable Python programs,
IPython
• Where to use it and where not
• Data types and control structures
• Objects, type declarations
• Numeric types, sequences (list, tuple, dict, strings), . . .
• Control structures: if, for, break, while, . . .
• Arithmetic operations
• Functions and parameters
• String handling: strip, trim, split, ...
• File I/O
• Objects, classes, modules, and packages
Tobias Neckel: Scripting with Bash and Python
Compact Course @ Max-Planck, February 16 - 26, 2015 6
Contents (cont.)
Scripting with Python . . . and beyond (ca. 5 days)
• Object oriented programming
A short introduction to OOP
•
OOP in Python
•
• Common and useful libraries
• os, sys
• Regular expressions: re
• Command-line options: optparse
• math, random
• GUIs with tkinter
• ...
• Scientific Computing in Python
• NumPy (vectors, matrices, . . . )
• Matplotlib
• Calling gnuplot

Tobias Neckel: Scripting with Bash and Python

Compact Course @ Max-Planck, February 16 - 26, 2015 7
Requirements
Bash
• Bash as shell (yes, tcsh works, too, but slightly different)
• Editor of your choice (vim, emacs, . . . )
• Collection of standard tools
• gnuplot
• If you are using Windows, we recommend to install cygwin
full/extended

Tobias Neckel: Scripting with Bash and Python

Compact Course @ Max-Planck, February 16 - 26, 2015 8
Requirements
Bash
• Bash as shell (yes, tcsh works, too, but slightly different)
• Editor of your choice (vim, emacs, . . . )
• Collection of standard tools
• gnuplot
• If you are using Windows, we recommend to install cygwin
full/extended
Python
• Python 2.7.x, as not all packages have been ported to Python
3.X
• Packages:
• Scipy (which should include IPython as interactive Python
shell; if not, get IPython)
• PyLab/Matplotlib
• Gnuplot
Tobias Neckel: Scripting with Bash and Python
Compact Course @ Max-Planck, February 16 - 26, 2015 8
 Part I

Introduction

Tobias Neckel: Scripting with Bash and Python

Compact Course @ Max-Planck, February 16 - 26, 2015 9
Some Types of Programming Langages

• Declarative programming
• Functional (Lisp, Scheme, make, ...)
• Logical (Prolog)
• Other (Sql, ...)
• Imperative programming
• Procedural (C, Fortran, Python, Pascal, ...)
• Modular (Bash, ...)
• Object-oriented (C++, Java, Python, ...)

Tobias Neckel: Scripting with Bash and Python

Compact Course @ Max-Planck, February 16 - 26, 2015 10
Scripting vs. Compiled Languages

Scripting languages
• Bash, Python, Perl, postscript, matlab/octave, ...
• Interactive mode
• Few optimisations
• Easy to use
• No binary files (hardly useful for commercial software)
Compiled languages
• C, C++, Fortran, ...
• Efficient for computational expensive tasks
• Source code is compiled to binary code
Other
• Java

Tobias Neckel: Scripting with Bash and Python

Compact Course @ Max-Planck, February 16 - 26, 2015 11
Shell vs. Python

Shell (Bash)
• Seperate program for each simple task
• Gluing together programs with a script
• Not really a full programming language
• Powerful tools available
• Suitable for small tools (1-100 lines of code)
Python
• Full programming language
• Large number of libraries available
• Intuitive naming conventions
• Suitable for almost any task

Tobias Neckel: Scripting with Bash and Python

Compact Course @ Max-Planck, February 16 - 26, 2015 12
Use of Shell Scripts
Where to use them:
• System administration
• Automating everyday terminal tasks
• Searching in and manipulating ASCII-Files
• ...
Where not to use them:
• Lots of mathematical operations
• Computational expensive tasks
• Large programs which need structuring and modularisation
• Arbitrary file access
• Data structures
• Platform-independent programs
• ...

Tobias Neckel: Scripting with Bash and Python

Compact Course @ Max-Planck, February 16 - 26, 2015 13
General Programming Rules

• Comments

Tobias Neckel: Scripting with Bash and Python

Compact Course @ Max-Planck, February 16 - 26, 2015 14
General Programming Rules

• Comments
• Comments

Tobias Neckel: Scripting with Bash and Python

Compact Course @ Max-Planck, February 16 - 26, 2015 14
General Programming Rules

• Comments
• Comments
• Comments

Tobias Neckel: Scripting with Bash and Python

Compact Course @ Max-Planck, February 16 - 26, 2015 14
General Programming Rules

• Comments
• Comments
• Comments
• Problem ⇒ algorithm ⇒ program
• Modular programming
• Generic where possible, specific where necessary
• ...

Tobias Neckel: Scripting with Bash and Python

Compact Course @ Max-Planck, February 16 - 26, 2015 14
 Part II

Bash Basics

Tobias Neckel: Scripting with Bash and Python

Compact Course @ Max-Planck, February 16 - 26, 2015 15
Some Basic Commands
The first interactive example:

cd
mkdir bash_course
cd bash_course
touch hello . sh
chmod u + x hello . sh
< editor > hello . sh

File manipulation
• Files: touch, ls, rm, mv, cp
• Directories: cd, mkdir, rmdir, pwd
• Access with chmod: read, write and execute
• Editors: vi, emacs, gedit, nedit, ...
• Documentation: man, info, apropos, - - help

Tobias Neckel: Scripting with Bash and Python

Compact Course @ Max-Planck, February 16 - 26, 2015 16
Hello World!

The first non-interactive example (hello.sh):

# !/ bin / bash

echo " Hello World ! "

• Convention: shell script suffix .sh

• #! sha-bang (#: sharp, !: bang)
• Sha-bang is followed by interpreter (try #!/bin/rm)
• echo is a shell builtin
• ”Hello World!” is (as almost everything) a string
• hello.sh has to be executable (chmod)

Tobias Neckel: Scripting with Bash and Python

Compact Course @ Max-Planck, February 16 - 26, 2015 17
Variables
# !/ bin / bash
STR = " Hello World ! "
echo $ STR

• NO WHITESPACE in assignments
• STR: name of the variable used for assignment
• $STR: reference to the value
• $STR is a short form of ${STR}

Tobias Neckel: Scripting with Bash and Python

Compact Course @ Max-Planck, February 16 - 26, 2015 18
Variables
# !/ bin / bash
STR = " Hello World ! "
echo $ STR

• NO WHITESPACE in assignments
• STR: name of the variable used for assignment
• $STR: reference to the value
• $STR is a short form of ${STR}
Quoting
# !/ bin / bash
STR = " Hello World ! "
echo $ STR
echo " $ STR "
echo ' $ STR '

Tobias Neckel: Scripting with Bash and Python

Compact Course @ Max-Planck, February 16 - 26, 2015 18
Operations

a =3+5 # does not work

a = ` expr 3 + 5 ` # does work
a = ` expr 3+5 ` # does not work
let " a = 3 + 5 " # does work
let " a =3+5 " # does work
a = $ ((3+5)) # does work
(( a ++)) # does work ; result ?
a = $ [3+5] # does work

• No direct mathematical operations (everything is a string)

• ‘command‘ is used to call a program
• expr: evaluate expressions
• Only integer operations possible
• Operators for expr: comparison (<, ...) and arithmetics (+, -, \*, /)
• Operators for let: comp., arith., +=, ..., bitwise and logical

Tobias Neckel: Scripting with Bash and Python

Compact Course @ Max-Planck, February 16 - 26, 2015 19
Arrays

# !/ bin / bash
arr [0]=1
arr [1]= $ (( $ { arr [0]}*2))
arr [2]= $ [ $ { arr [1]}*2]
arr [5]=32

echo $ { arr [0]}

echo $ { arr [1]}
echo $ { arr [2]}
echo $ { arr [3]}
echo $ { arr [4]}
echo $ { arr [5]}
echo $ { # arr [*]}
echo $ { arr [*]}

Tobias Neckel: Scripting with Bash and Python

Compact Course @ Max-Planck, February 16 - 26, 2015 20
Special Variables
within a script
• $? Exit status variable
• $$ process ID
• $0, $1, $2, ... command line parameters
• $* all command line parameters (single string)
• $@ all command line parameters (one string per parameter)
• $# number of command line parameters
• ${#hvariablei} string length of the variable value
• ...
• shift n shift all command line parameters to the left by n (first n
parameters are lost!)

Tobias Neckel: Scripting with Bash and Python

Compact Course @ Max-Planck, February 16 - 26, 2015 21
Special Variables
within a script
• $? Exit status variable
• $$ process ID
• $0, $1, $2, ... command line parameters
• $* all command line parameters (single string)
• $@ all command line parameters (one string per parameter)
• $# number of command line parameters
• ${#hvariablei} string length of the variable value
• ...
• shift n shift all command line parameters to the left by n (first n
parameters are lost!)
Environment Variables

env

Tobias Neckel: Scripting with Bash and Python

Compact Course @ Max-Planck, February 16 - 26, 2015 21
Control Structures - if

if [ 1 = 1 ]
then
echo " Hello World ! "
fi

• Usually (most languages), something is done if test is true

• Usually, 0 is false, non-zero values are true
• In bash, test gives a return value
• Return value 0 means no error

test 1 = 2 # equals [ 1 = 2 ]
test 1 = 1

Tobias Neckel: Scripting with Bash and Python

Compact Course @ Max-Planck, February 16 - 26, 2015 22
Control Structures - test
• Usage: test EXPRESSION or [ EXPRESSION ]
• Exits with the status determined by the given expression
• And: EXPRESSION1 -a EXPRESSION2
• Or: EXPRESSION1 -o EXPRESSION2
• Negation: ! EXPRESSION
• String comparison: =, != ( test 1 = 1 is string comparison!)
• Integer comparison: -eq, -ge, -gt, -le, -lt, -ne
• File comparison: -ef, -nt, -ot
• File test with single operand: -e, -d, ...
• More details: man test

test 1 - eq 1
[ 1 - eq 1 ]
[[ 1 - eq 2 || 1 - eq 1 ]]

Tobias Neckel: Scripting with Bash and Python

Compact Course @ Max-Planck, February 16 - 26, 2015 23
First Useful Example

# !/ bin / bash
# encryption program
clear
which $ EDITOR
if [ $ ? != " 0 " ]
then
echo " Which editor would you like to use ? "
read EDITOR
fi
$ EDITOR plaintext # gedit does not work ! detached from console !
gpg -a --no - options -o cryption . gpg -c plaintext
shred -u plaintext
clear
cat cryption . gpg
shred -u cryption . gpg
exit

Tobias Neckel: Scripting with Bash and Python

Compact Course @ Max-Planck, February 16 - 26, 2015 24
Control Structures - loops

# !/ bin / bash
for i in < list >
do
< commands >
done

• List can be any possible list of elements

• seq -s <s> <x> produces a list of numbers from 1 to x with
separator s (jot - 1 x on MAC)
• for i in {1..x} does the same (no variable expansion!)
• break/break hni: stop the loop (n levels of nested loops)
• continue/continue hni: continue with the next iteration
• Other loops: while [ condition ]; do command; done
• Other loops: until [ condition ]; do command; done

Tobias Neckel: Scripting with Bash and Python

Compact Course @ Max-Planck, February 16 - 26, 2015 25
Functions

# !/ bin / bash
function function_name {
command
}

function_name () {
command
}

• Both syntactic variants do the same

• The round brackets are NOT used for parameters
• Functions have to be defined before they are used
• Functions may not be empty (use :)
• Parameter passing to functions equal to programs

Tobias Neckel: Scripting with Bash and Python

Compact Course @ Max-Planck, February 16 - 26, 2015 26
Functions with Parameters

function min {
if [ $ 1 - lt $ 2 ]
then
return $ 1
else
return $ 2
fi
}

a = ` min 4 6 `
echo $ a # does not work
min 4 6
a=$? # works
min 500 600
a=$? # does not work ! why ?

Tobias Neckel: Scripting with Bash and Python

Compact Course @ Max-Planck, February 16 - 26, 2015 27
Functions with Parameters (2)

function min {
if [ $ 1 - lt $ 2 ]
then
echo $ 1
else

echo $ 2
fi
}

a = $ ( min 4 6) # equals a = ` min 4 6 `

echo $ a

Tobias Neckel: Scripting with Bash and Python

Compact Course @ Max-Planck, February 16 - 26, 2015 28
Command Substitution

Allows output of a command to replace command itself

$ ( command )

` command `

• Both perfom expansion by executing command

• Replacing command with standard output of command, trailing
newlines deleted
• Not return code!
• May be nested: escape inner backquotes with backslashes

Tobias Neckel: Scripting with Bash and Python

Compact Course @ Max-Planck, February 16 - 26, 2015 29
Control Structures - if (2)
# !/ bin / bash

rand = $ [ $ RANDOM %2]

if [ $ # - eq 0 ]
then
echo " Usage : $ 0 <0 or 1 > "
else
if [[ $ 1 - ne 0 && $ 1 - ne 1 ]]
then
echo " parameter has to be 0 or 1 "
elif [ $ 1 - eq $ rand ]
then
echo " won "
else
echo " lost "
fi
fi

Tobias Neckel: Scripting with Bash and Python

Compact Course @ Max-Planck, February 16 - 26, 2015 30
Control Structures - case and select
# !/ bin / bash
echo " Hit a key , then hit return . "
read Key
case " $ Key " in
[[: lower :]] ) echo " Lowercase letter " ;;
[[: upper :]] ) echo " Uppercase letter " ;;
[0 -9] ) echo " Digit " ;;
* ) echo " something else " ;;
esac

Tobias Neckel: Scripting with Bash and Python

Compact Course @ Max-Planck, February 16 - 26, 2015 31
Control Structures - case and select
# !/ bin / bash
echo " Hit a key , then hit return . "
read Key
case " $ Key " in
[[: lower :]] ) echo " Lowercase letter " ;;
[[: upper :]] ) echo " Uppercase letter " ;;
[0 -9] ) echo " Digit " ;;
* ) echo " something else " ;;
esac

# !/ bin / bash
PS3 = ' Choose your favorite language : '
select language in " bash " " python " " brainfuck " " C ++ "
do
echo " Your favorite language is $ language . "
break
done
Tobias Neckel: Scripting with Bash and Python
Compact Course @ Max-Planck, February 16 - 26, 2015 31
Partial List of Commands/Variables/. . . so far

man mkdir bla $?

$STR continue rmdir clear
#! shell builtin $RANDOM break
let emacs ls $#
$3 touch echo ”$STR” ${#a}
$a -le $b chgrp if a=3
mv a=$((3+5)) expr test
$$ ((a++)) cp pwd
script suffix cd echo for
chmod $* $PS3
esac $@ u+x
${STR:-”x”} echo ’$STR’ info
apropos rm vi
read - - help a=$[3+5]

Tobias Neckel: Scripting with Bash and Python

Compact Course @ Max-Planck, February 16 - 26, 2015 32