Using Bash for automation

Given the recent swath of articles covering the fundamental aspects of Bash (listed at the end of this article), it’s inevitable that one of your new colleagues tosses one into the cloud. As these things go, the next logical steps are:

1. Verify something absolutely critical depends upon the "cloud" script working flawlessly.
2. Verify the original author of the script, has completely forgotten how it actually works.
3. Verify the newest admin is tasked to fundamentally alter it without any validation whatsoever.

In this article, I assist all admin-kind and help you avoid all of the above mistakes. In turn, the result leads to happier management, and hopefully our continued employment.

How to spell "Bash script"

In order to become enlightened (and for your love of $DEITY) check your code into a source code management (SCM) tool. Even as you’re learning, use a local repository as the playground. Not only does this practice let you journal your efforts over time, but it also lets you easily undo mistakes. There are many wonderful articles on getting started with git that I highly recommend.

A quick note about using and learning Bash, because this scripting language brings a unique set of -isms and stylistic author preferences: As soon as you spot something that looks new to you (syntax, style, or a language construct) look it up immediately. Spend your time understanding the new item from the man page (first choice) or the Advanced Bash Scripting Guide (both can be accessed offline). Doing this will slow you down at first, but over time this practice will help you build your knowledge of where to find answers.

Writing reusable Bash nuggets as libraries

Scripts in automation are best written while embracing the Unix philosophy: Many small tools that only do one thing. Meaning, you’ll do much better writing small, specialized scripts and libraries, than you will with one giant "kitchen sink." Though, admittedly, I have written and maintained a few behemoths (occasionally they do serve a purpose).

Automation scripts frequently must be understandable and maintainable by more than one author. With many small scripts flying around (and being tracked in version control) you will quickly find yourself needing to share a reference to values for names, versions, paths, or URLs. Writing these common elements into libraries also provides additional mental space for maintainers to appreciate inline documentation. Additionally, doing so makes leveraging unit tests nearly trivial (we’ll get into this topic at the end).

Let’s practice good code hygiene from the start by creating a local "play" repository. Within your new repository, create a place to contain our scripts and library files. I like to stick with the well understood FHS standards for simplicity. Create the directories ./bin/ and ./lib/ in the repository’s root. In a larger automation project, I’d still use these names, but they might be deeply buried (say, under a scripts or tools subdirectory).

Talking about paths brings me to a great topic for a first library. We need a way for our future components to reference structural elements and high-level values. Using your favorite editor, create the file ./lib/anchors.sh and add the content below:

# A Library of fundamental values
# Intended for use by other scripts, not to be executed directly.

# Set non-'false' by nearly every CI system in existence.
CI="${CI:-false}"  # true: _unlikely_ human-presence at the controls.
[[ $CI == "false" ]] || CI='true'  # Err on the side of automation

# Absolute realpath anchors for important directory tree roots.
LIB_PATH=$(realpath $(dirname "${BASH_SOURCE[0]}"))
REPO_PATH=$(realpath "$LIB_PATH/../")  # Specific to THIS repository
SCRIPT_PATH=$(realpath "$(dirname $0)")

The file begins with two blank lines, and the first comment explains why. The library should not be set as executable (despite the name ending in .sh, indicating its type). If the library was executed directly, it’s possible that it could cause the user’s shell to vanish (or worse). Disabling direct execution (chmod -x ./lib/anchors.sh) is the first level of beginner admin protection. The comment at the start of the file is the second level.

By convention, comments should always describe the why (not the what) of the statements that follow. A reader can simply read the statement to understand what it does, but they cannot reliably intuit what the author was thinking at the time. However, before I go deeper, I need to detail an issue that often catches people off guard with Bash.

The Bash defaults provide an empty string upon reference to an undefined variable. The CI variable (or something analogous in your automation) is intended to indicate the probable absence of a human. Unfortunately, for the robot overlords, humans will probably need to execute the script manually at least once. At this point, it is likely that they will forget to set a value for CI before pressing Enter.

So, we need to set a default for the value and ensure that it is always either true or false. The example library code above demonstrates both how to test if a string is empty, and how to force the string to contain one of a pair of values. The way I read the first group of statements in anchors.sh is:

Define 'CI' going forward as the result of:

1. Examining the previous value of CI (it may be undefined, so an empty string).
2. The ':-' part means:
   1. If the value was an empty string, represent the string 'false' instead.
   2. If the value was not an empty string, use whatever it was (including 'DaRtH VaDeR').

Test the thing inside '[[ ' and ' ]]':

1. If the new value of 'CI' equals the literal string "false", throw the exit code 0 (meaning success or truth).
2. Otherwise, throw the exit code 1 (meaning failure or non-truth)

If the test exited with 0, continue with the next line, or (the ' || ' part), assume a novice admin set CI=YES!PLEASE or a perfect-machine set CI=true.  Immediately set the value of 'CI' to the literal string 'true'; because perfect machines is more betterer, not do mistak.

For the remainder of this library, the anchor path values are nearly always useful in scripts running in automation from a repository.  When used in a larger project, you would need to adjust the relative location of the library directory with respect to the repository root.  Otherwise, I will leave understanding these statements, as a research exercise for the reader (do it now, it will help you later).

Using Bash libraries in Bash scripts

To load a library, use the source built-in command. This command is not fancy. Give it the location of a file, and then it reads and executes the contents right there and then, meaning that the library code’s run-time context will actually be the script which sourced it.

To help prevent too much of your brain from dripping out of your ears, here is a simple ./bin/example.sh script to illustrate:

#!/bin/bash

LIB_PATH="$PWD/$(dirname $0)/../lib/anchors.sh"
echo "Before loading: $LIB_PATH"
set -ax
cd /var/tmp
source $LIB_PATH
echo "After loading: $(export -p | grep ' LIB_PATH=')"

You might immediately notice that the script altered the run-time context before loading the library. It also defines LIB_PATH locally and points it at a file (confusingly, instead of a directory) with a relative path (for illustrative purposes).

Go ahead and execute this script and examine the output. Notice that all of the operations in the library anchors.sh ran inside the /var/tmp/ directory and automatically exported its definitions. The old definition for LIB_PATH was clobbered and exported by the a in the set -ax. This fact is visible in the output from the declare -x coming from the export command. Hopefully, the debugging output (the x in the set -ax) is understandable.

When debugging like this, Bash prints all of the intermediate values when parsing every line. I included this script here to show why you would never want to set -ax or change directories using the commands from the top level of a library. Remember that library instructions are evaluated at load time in the script. So changing the environment in a library causes there to be runtime side-effects in whatever script used source to load it. Side effects like this are guaranteed to drive at least one sysadmin completely insane. You never know, that admin could be me, so don’t do it.

As a practical example, consider an imaginary library that defines a function using a username/password environment variable to access a remote service. If the library did a top-level set -ax before this function, then every time it’s loaded the debugging output will include displaying these variables, splattering your secrets all over the place for everyone to see. Worse, it will be difficult (from a calling script's perspective), for a beginner colleague to disable the output without yelling at their keyboard. 

In conclusion, the key here is to remain aware that libraries "happen" within their caller’s context. This factor is also why the example anchors.sh can use $0 (the executable script path and filename), but the path to the library itself is only available via the "magic" '${BASH_SOURCE[0]}' (array element). This factor might be confusing at first, but you should try to remain disciplined. Avoid broad, far-reaching commands in libraries. When you do, all the new admin hires will insist on paying for your doughnuts.

Writing unit tests for libraries

Writing unit tests can feel like a daunting task until you realize that perfect coverage is usually a waste of your time. However, it’s a good habit to always use and update your test code when touching your library code. The test writing goal is to take care of the most common and obvious use cases and then move on. Don’t pay much attention to corner cases or less than all-the-frickin-time uses. I also suggest initially focusing your library-testing efforts at the unit testing level instead of integration testing.

Let’s look at another example: the executable script ./lib/test-anchors.sh:

#!/bin/bash

# Unit-tests for library script in the current directory
# Also verifies test script is derived from library filename

TEST_FILENAME=$(basename $0)  # prefix-replace needs this in a variable
SUBJ_FILENAME="${TEST_FILENAME#test-}"; unset TEST_FILENAME
TEST_DIR=$(dirname $0)/

ANY_FAILED=0

# Print text after executing command, set ANY_FAILED non-zero on failure
# usage: test_cmd "description" <command> [arg...]

test_cmd() {
   local text="${1:-no test text given}"
   shift
   if ! "$@"; then
      echo "fail - $text"; ANY_FAILED=1;
   else
      echo "pass - $text"
   fi
}

test_paths() {
   source $TEST_DIR/$SUBJ_FILENAME
   test_cmd "Library $SUBJ_FILENAME is not executable" \
      test ! -x "$SCRIPT_PATH/$SUBJ_FILENAME"
   test_cmd "Unit-test and library in same directory" \
      test "$LIB_PATH" == "$SCRIPT_PATH"
   for path_var in LIB_PATH REPO_PATH SCRIPT_PATH; do
      test_cmd "\$$path_var is defined and non-empty: ${!path_var}" \
         test -n "${!path_var}"
      test_cmd "\$$path_var referrs to existing directory" \
         test -d "${!path_var}"
   done
}

# CI must only/always be either 'true' or 'false'.
# Usage: test_ci <initial value> <expected value>

test_ci() {
   local prev_CI="$CI"  # original value restored at the end
   CI="$1"
   source $TEST_DIR/$SUBJ_FILENAME
   test_cmd "Library $SUBJ_FILENAME loaded from $TEST_DIR" \
      test "$?" -eq 0
   test_cmd "\$CI='$1' becomes 'true' or 'false'" \
      test "$CI" = "true" -o "$CI" = "false"
   test_cmd "\$CI value '$2' was expected" \
      test "$CI" = "$2"
   CI="$prev_CI"
}

test_paths
test_ci "" "false"
test_ci "$RANDOM" "true"
test_ci "FoObAr" "true"
test_ci "false" "false"
test_ci "true" "true"

# Always run all tests and report, exit non-zero if any failed

test_cmd "All tests passed" \
   test "$ANY_FAILED" -eq 0
[[ "$CI" == "false" ]] || exit $ANY_FAILED  # useful to automation
exit(0)

The reason I put this script in ./lib (as opposed to ./bin) is both out of convenience and because tests must never rely on using the code they are checking. Because this test needs to check paths, it’s easier to put it in the same path as the library. Otherwise, this approach is a matter of personal preference. Feel free to execute the test now because it might help you understand the code.

Wrapping up

This article by no means represents the entirety of using Bash in automation. However, I tried to instill basic knowledge and recommendations that (if followed) will undoubtedly make your life easier. Then, even when things do become difficult, a firm understanding of the importance of runtime context will be handy.

Finally, scripting in or for automation can be unforgiving of mistakes. Having even basic unit tests in place for your libraries will both build confidence and help out the next person to come along (which could be you after five years of forgetting). You can find all of the example code used in this article online here.

Interested in brushing up on your Bash fundamentals? Check out:

• How to program with Bash: Logical operators and shell expansions 
• 5 ways to improve your Bash scripts 
• Getting started with shell scripting 
• 10 basic Linux commands you need to know
• Linux environment variable tips and tricks

[ Want to try out Red Hat Enterprise Linux? Download it now for free. ]