Category: Development

We recently had our JAMF Nation User Conference here in Minneapolis.  I spent a lot of time with a lot of brilliant sysadmins from around the world.  After three nights of mingling it became pretty clear that despite how much I enjoy talking about technology solutions (and I will continue to talk as people hand me Martinis), I don’t feed a lot of that back into Mac admin community, and it was made clear that I should give it a try.

So, this will be my first installment of three posts sharing some of what I’ve learned.  This first will focus on Bash scripting.  The second will serve as an introduction to Python for the Bash scripter (let me be up front about this: Python is so hot…) and some of the benefits that it offers to us as administrators.  The last post will be entirely about the JSS API, interacting the the data in both Bash and Python as well as some examples of what you can accomplish with it.

So, on to Bash…

For the last two years as a Mac administrator I’ve learned quite a bit on the subject of shell scripting and have been shown a wealth of handy tricks that have helped refine my skill (there are a lot of wizards at JAMF).  In this post I want to show some of my favorite scripting solutions and techniques that have become staples to my writing style.  Now, if you’re reading this I’m going to assume you’re already pretty familiar with using a Mac via the command line and basic scripting (if the word shebang doesn’t conjure up the image of a a number sign with an exclamation mark, you might not be ready).

Error Checking and Handling

As admins we write a lot of code that interacts with the Mac in ways that can certainly ruin a person’s day (if not their life, or so they would claim).  When executing commands there is a variable always readily available to view:

USS-Enterprise:~ brysontyrrell$ echo $?
0

The ‘$?’ represents the result of the last command that was run.  If there were no errors you  receive a zero (0) in return.  For anything else there will be a value greater than zero which represents your error code.  Visit the manpage of any number of commands on your Mac and you will usually find a section devoted to what an error code represents (if not, we have Google).  Error codes are critical for figuring out what went wrong.

In many cases we might need to kill a script if a command failed.  After all, if data is being manipulated or moved around there’s not a whole lot of sense in executing the remainder of the script when some key actions did not perform correctly.  We can do this with a simple IF statement that triggers an ‘exit’:

cp /path/to/source /path/to/destination
if [ $? -ne 0 ]; then
    exit
fi

Now, this will exit the script if our copy operation failed, but it isn’t that great.  The script will exit, but we won’t have any tangible information about how or why.  Let’s add a few things into this to make it more helpful to us:

cp /path/to/source /path/to/destination
if [ $? -ne 0 ]; then
    echo "There was a problem with the copy operation. Error code $?"
    exit 1
fi

Now we’re getting somewhere.  With this the script will not only output onto the Terminal that there was an error, but it will return the error code and also exit our script with a value greater than zero which will be reported as a failure!  We can have error codes that mean different things. ‘1’ is just the default.  Any numerical value can be used to represent an error (or types of error if we’re lumping them together) and its meaning can be recorded either within the script or in other documentation:

# Exit 1 for general error
# Exit 10 for copy operation error
cp /path/to/source /path/to/destination
if [ $? -ne 0 ]; then
    echo "There was a problem with the copy operation. Error code $?"
    exit 10
fi

Using ‘echo’ for the output is great if we’re running the script manually, but what we’re writing will be run remotely and we will not be watching it execute live.  We’re going to need something that will allow us to go back at a later time to review what transpired:

cp /path/to/source /path/to/destination
if [ $? -eq 0 ]; then
    log "The copy operation was successful."
else
    log "There was a problem with the copy operation. Error code $?" 10
fi

The log() call you see here is actually a function that I use for almost everything I write.  We’re going to cover what exactly it does a little later, but the basic of the above script is that it will output a message upon both the success of the command as well as the failure and exit the script after the error.  Never underestimate how important it is that your scripts are telling you what they are doing.  Your life will be better for it.

Operators for Shortcuts

Our above examples all allow you to execute multiple actions in response to the success or failure of a command based upon the result.  Sometimes we might only need to trigger one command in response to an action.  We can use operators  to achieve this effect without writing out an entire IF statement:

# A copy operation using an IF statement to execute a file removal
cp /path/to/source /path/to/destination
if [ $? -eq 0 ]; then
    rm /path/to/source
fi

# The above operation using the '&&' operator
cp /path/to/source /path/to/destination && rm /path/to/source

In the second example the source file we are copying is deleted so long as the ‘cp’ command left of the ‘AND’ operator returned zero.  If there had been an error then the code on the right side won’t execute.  Both examples achieve the same result but using the operator acts as a short cut and allows you to cut down on the amount of code you need to write.  If you need to achieve the same effect but when the result is not zero we can turn to the ‘OR’ operator:

# A copy operation using an IF statement to exit upon failure
cp /path/to/source /path/to/destination
if [ $? -ne 0 ]; then
    exit 10
fi

# The same copy operation but using '||' to trigger the 'exit'
cp /path/to/source /path/to/destination || exit 10

Its a TRAP

This one is a personal favorite.  Usage of the Bash builtin ‘trap’ is actually pretty new to me, and it is one of the hands down coolest (if you’re like me and think scripting is, you know, cool) things I’ve seen.  A ‘trap’ give you the ability to determine actions that are performed when your script terminates or when commands throw errors!  Let me demonstrate with a very basic example:

# Here we define the function that will contain our commands for an 'exit'
onExit() {
rm -rf /private/tmp/tempfiles
}

# Here we set the 'trap' to execute upon an EXIT signal
trap onExit EXIT

for file in /private/tmp/tempfiles/*; do
    cp "${file}" /path/to/destination/
done

# This 'exit' command will send an EXIT signal which will trigger the 'trap'
exit 0

As you can see in the above example, ‘trap’ is very easy to use.  The syntax for ‘trap’ is:

trap 'command(s)' signal(s)

We created an onExit() function containing the actions we wanted to perform.  This became the command in the ‘trap’ line. Once triggered, the temporary directory that we were copying files from is automatically purged once the script is complete. This makes cleanup much simpler and easier on us.  It also allows far more control over the state of the system upon an error requiring we kill the script in process.  I had mentioned in my introduction that we could have traps for both terminations and errors, did I not?  Let’s expand upon that first example and make it a bit more robust:

onExit() {
rm -rf /private/tmp/tempfiles
}

# This function contains commands we want to execute every time there is an ERROR
onError() {
errorCode=$?
cmdLine=$1
cmdName=$2
echo "ERROR: ${cmdName} on line ${cmdLine} gave error code: ${errorCode}"
exit 1
}

trap onExit EXIT

# Here we introduce a second 'trap' for ERRORs
trap 'onError $LINENO $BASH_COMMAND' ERR

for file in /private/tmp/tempfiles/*; do
    cp "${file}" /path/to/destination/
done

exit 0

This one certainly has a lot more going on.  The way I approach these ‘traps’ is pretty simple: my EXIT performs a cleanup of any working files while my ERROR handles outputting the information I will need to determine what went wrong.  In this example I have an ‘exit’ command included inside the onError() function so the cleanup onExit() function is still called in the first event of an error.  That’s not a practice I’m recommending, but I am showing that it is an option.  There are plenty of cases out there where you would want the script to continue on even if an error occurs in the middle of a copy operation (user account migration, anyone?).  Those are the times when you will want to be explicit about where in your script certain errors trigger an ‘exit.’

Let’s break down that onError() function:

onError() {
# Our first action is to capture the error code of the command (remember, this changes after EVERY command executed)
errorCode=$?
# This variable is from $LINENO which tells us the number line the command resides on in the script
cmdLine=$1
# The last variable is from $BASH_COMMAND which is the name of the command itself that gave an error
cmdName=$2
# Our tidy statement here puts it all together in a human-readable form we can use to troubleshoot
echo "ERROR: ${cmdName} on line ${cmdLine} gave error code: ${errorCode}"
exit 1
}

# In this 'trap' we call not just our function, but we also pass two parameters along to it
# The $LINENO and $BASH_COMMAND variables are called 'Internal Variables' to the Bash shell
trap 'onError $LINENO $BASH_COMMAND' ERR

We’re going to make this onError() function even more powerful a little later by visiting the log() function I had mentioned.  Before that, let’s go back to the onExit() function.  This ‘trap’ ideally is where we want to perform all of our cleanup actions, and the basic example I gave it wiping out a temporary directory of working files.  While our scratch space is removed in this process it does not address any actions we may have made in other areas of the system.  So, do we want to write all of that into the onExit() function even if they may not be relevant to when the script terminated?

I’m a big fan of the idea: “If I don’t HAVE to do this, then I don’t want to.”  The meaning of this is I don’t want to execute commands on a system (especially when I’m masquerading around as root) if they’re unnecessary.  We can write our onExit() function to behave following that ideology.  I don’t quite remember where I first saw this on the internet, but it was damned impressive:

# This is an array data type
cleanup=()

onExit() {
# Once again we're capturing our error code right away in a unique variable
exitCode=$?
rm -rf /private/tmp/tempfiles
# If we exit with a value greater than zero and the 'cleanup' array has values we will now execute themif [ $exitCode -ne 0 ] && [ "${#cleanup[@]}" -gt 0 ]; thenfor i in"${cleanup[@]}"; do# The 'eval' builtin takes a string as an argument (executing it)eval"$i"donefi
echo"EXIT: Script error code: $exitCode"
}

onError() {
errorCode=$?
cmdLine=$1
cmdName=$2
echo"ERROR: ${cmdName} on line ${cmdLine} gave error code: ${errorCode}"exit 1
}

trap onExit EXIT
trap'onError $LINENO $BASH_COMMAND' ERR

for file in /private/tmp/tempfiles/*; docp"${file}" /path/to/destination/
    # After each successful copy operation we add a 'rm' command for that file into our 'cleanup' array
    cleanup+=('rm /path/to/destination/"${file##*/}"')
doneexit 0

We have now transformed our onExit() function into one giant UNDO command.  The IF statement within it will always remove the temporary working directory (which we always want, no matter what the exit status is) but will now run additional commands out of our handy ‘cleanup’ array.  Effectively, unless the script successfully completes the entire copy operation it will, on the first error, remove every file that did make it into the destination.  This leaves the system pretty much in the same state as it was before our script ran.  We can take this concept further in much larger scripts by adding new commands into a ‘cleanup’ array as we complete sections of our code.

Logging is Awesome

I’m finally getting around to explaining that log() function from earlier.  Logs are fantastic for troubleshooting as they generally contain a lot of data that helps point us towards the source of the issue.  You can approach logging of your own scripts in two ways: append and existing log or use your own customized one.  In my case, I append all of my script log output into the Mac’s system.log using the ‘logger’ command.  This command allows you to do some pretty cool things (like message priority), but my use is fairly simple.

log () {
if [ -z "$2" ]; then
    logger -t "it-logs: My Script" "${1}"
else
    logger -t "it-logs: My Script" "${1}"
    logger -t "it-logs: My Script" "Exiting with error code $2"
    exit $2
fi
}

log "This is a log entry"

***OUTPUT IN SYSTEM.LOG AS SHOWN IN CONSOLE.APP***
Nov 10 12:00:00 USS-Enterprise.local it-logs: My Script[1701]: This is a log entry

You’re probably piecing together how this function works.  the ‘-t’ flag in the command creates a tag for the log entry.  In my case I have a universal prefix for the tag I use in all of my scripts (here I’m using ‘it-logs:’ but its similar) and then I follow it with the name of the script/package for easy reference (you read right: everything I write about in this post I use for preinstall and postinstall scripts in my packages as well).  The tagging allows me to grab the system.log from a machine and filter all entries containing ‘it-logs’ to see everything of mine that has executed, or I can narrow it down to a specific script and/or package by writing the full tag.  Its really nice.

Right after the tag inside the brackets is the process ID, and then we have our message.  If you scroll back up to the example where I used the log() function you’ll see that in code that triggered on a failure I included a ’10’ as a second parameter.  That is the error code to use with an ‘exit.’  If present, log() will write the message first and then write a second entry stating that the script is existing with an error code and  ‘exit’ with that code (and trigger our onExit() trap function).

If you want to maintain your own log, instead of writing into the system.log, you can easily do so with a similar function:

# You must ensure that the file you wish to write to exists
touch /path/to/log.log

log () {
if [ -z "$2" ]; then
    # Here 'echo' commands will output the text message that is '>>' appended to the log
    echo $(date +"%Y %m %d %H:%M")" it-logs: My Script: ${1}" >> /path/to/log.log
else
    echo $(date +"%Y %m %d %H:%M")" it-logs: My Script: ${1}" >> /path/to/log.log
    echo $(date +"%Y %m %d %H:%M")" it-logs: My Script: Exiting with error code $2" >> /path/to/log.log
    exit $2
fi
}

***OUTPUT IN LOG.LOG AS SHOWN IN CONSOLE.APP***
2013 11 10 20:38 it-logs: My Script: This is a log entry

The end result is just about the same, and with a .log extension it will automatically open in the Console.app and still use the filtering.

Now I’m going to take the copy operation from above and write in the log() function so you can see how the whole package fits together:

log () {
if [ -z "$2" ]; then
    logger -t "it-logs: My Script" "${1}"
else
    logger -t "it-logs: My Script" "${1}"
    logger -t "it-logs: My Script" "Exiting with error code $2"
    exit $2
fi
}

cleanup=()

onExit() {
exitCode=$?
log "CLEANUP: rm -rf /private/tmp/tempfiles"
rm -rf /private/tmp/tempfiles
if [ $exitCode -ne 0 ] && [ "${#cleanup[@]}" -gt 0 ]; then
    for i in "${cleanup[@]}"; do
        log "ADD-CLEANUP: $i"
        eval "$i"
    done
fi
}

onError() {
errorCode=$?
cmdLine=$1
cmdName=$2
log "ERROR: ${cmdName} on line ${cmdLine} gave error code: ${errorCode}" 1
}

trap onExit EXIT
trap 'onError $LINENO $BASH_COMMAND' ERR

for file in /private/tmp/tempfiles/*; do
    cp "${file}" /path/to/destination/
    log "COPY: ${file} complete"
    cleanup+=('rm /path/to/destination/"${file##*/}"')
done

exit 0

Interacting With the User Stuff

There are a ton of examples out there for grabbing the name of the currently logged in user and finding their existing home directory.  Both items are very important when we’re executing our scripts as the root user.  To get the name of the logged in user I’ve found these three methods:

# Though the simplest I have found that this method does not work in a package preinstall/postinstall script
USS-Enterprise:~ brysontyrrell$ echo $USER
brysontyrrell

# This one is also very simple but it relies upon a command that may not be present in future OS X builds
USS-Enterprise:~ brysontyrrell$ logname
brysontyrrell

# The following is used pretty widely and very solid (the use of 'ls' and 'awk' nearly future-proofs this method)
USS-Enterprise:~ brysontyrrell$ ls -l /dev/console | awk '{print $3}'
brysontyrrell

One step beyond this is to then find the user’s home directory so we can move and/or manipulate data in there.  One piece of advice I’ve been given is to never assume I know what the environment is.  Users are supposed to be in the ‘/Users/’ directory, but that doesn’t mean they are.  If you’ve never played around much with the directory services command line utility (‘dscl’), I’m happy to introduce you:

USS-Enterprise:~ brysontyrrell$ dscl . read /Users/brysontyrrell | awk '/NFSHomeDirectory:/ {print $2}'
/Users/brysontyrrell

‘dscl’ is incredibly powerful and gives use easy access to a lot of data concerning our end-users’ accounts.  In fact, you can take that above command and change out the regular expression ‘awk’ is using to pull out all sorts of data individually:

USS-Enterprise:~ brysontyrrell$ dscl . read /Users/brysontyrrell | awk '/GeneratedUID:/ {print $2}'
123A456B-7DE8-9101-1FA1-2131415B16C1

USS-Enterprise:~ brysontyrrell$ dscl . read /Users/brysontyrrell | awk '/UniqueID:/ {print $2}'
501

USS-Enterprise:~ brysontyrrell$ dscl . read /Users/brysontyrrell | awk '/PrimaryGroupID:/ {print $2}'
20

USS-Enterprise:~ brysontyrrell$ dscl . read /Users/brysontyrrell | awk '/UserShell:/ {print $2}'
/bin/bash

Alternative Substitution

Kind of an odd header, but you’ll get it in a moment.  One of the bread ‘n butter techniques of scripting is to take the output of one command and capture it into a variable.  This process is known as “command substitution” where instead of displaying the entered command we are shown the result.  The traditional way of doing this is to enclose the command you are capturing in `backticks`.  Instead of using backticks, use a $(dollar and parenthesis) so your editor of choice still highlights the command syntax correctly.

Check out this example which will pull out the type of GPU of the Mac:

gpuType=`system_profiler SPDisplaysDataType | /awk -F': ' '/Chipset Model/ {print $2}' | tail -1`
gpuType=$(system_profiler SPDisplaysDataType | awk -F': ' '/Chipset Model/ {print $2}' | tail -1)

Functionally, both of these statements are identical, but now as we write our scripts we have an easier time going back and identifying what is going on at a glance.  Let’s take two of the examples from above for obtaining information about the logged in user and write them the same way for a script:

userName=$(ls -l /dev/console | awk '{print $3}')
userHome=$(dscl . read /Users/"${userName}" | awk '/NFSHomeDirectory:/ {print $2}')

cp /private/tmp/tempfiles/somefile "${userHome}"/Library/Preferences/