Safer Bash Scripting Tips

I hope you are all well and safe. As "stay safe" is the most used expression during this COVID-19 period, I thought it would be nice to extend safety to other areas as well. How about we talk about some simple tips to make your Bash scripts safer and even more robust?

Imagine you have an application that keeps reading from a directory with a lot of small static files. We're talking about file reading operations. Let's call this directory "/var/appX/data". The program appX makes constant read operations to gather data from that directory. Now, imagine that part of that data has changed and you need to update just one parameter inside all the static files that have the ".prm" extension (I've just invented that! :P).

As a good "shell scriptor" you start thinking right away about a for loop and some cryptic REGEX feeding a sed command, right? Cool! The solution is simple, but there might be something under the hood you didn't see at first! What if your script crashes (for some reason), or gets killed by someone else who has the privileges for doing that (who knows) ?

Your appX will start yelling at you and write very nasty things inside its log files, as part of the information it is gathering will be correct, and part will be miserably wrong!

How to solve that? Try to be atomic!

In short, being atomic means to do the operation in one single tick! How could you be atomic in this scenario? Well, use commands that perform that way! The mv and mkdir are examples of commands that perform atomic operation, because the operating system needs to be sure that things will be done in one single tick!

Let's see that in action. First, imagine the reckless (snippet) version of your script.

#!/bin/bash


# Saves current IFS
CUR_IFS=$IFS
IFS=$(echo -en "\n\b")


DATA_DIR="/var/appX/data"


for file in $(find "$DATA_DIR" -name "*.prm")
do
    sed -i 's/OLD_PARAM/NEW_PARAM/g' "$file"
done

Do you see the mess being setup? All the tears, blood and suffering that will rise from that script being killed in action?

Now, if a little modification, appX and people are going to be happy forever after.

#!/bin/bash


# Saves current IFS
CUR_IFS=$IFS
IFS=$(echo -en "\n\b")


DATA_DIR="/var/appX/data/"
DATA_DIR_TMP="$(echo "${DATA_DIR%%/*}_tmp"
DATA_DIR_OLD="$(echo "${DATA_DIR%%/*}_old"

cp -a "$DATA_DIR" "$DATA_DIR_TMP"

for file in $(find "$DATA_DIR_TMP" -name "*.prm")
do
    sed -i 's/OLD_PARAM/NEW_PARAM/g' "$file"
done

# Now, be atomic!
mv "$DATA_DIR" "$DATA_DIR_OLD"
mv "$DATA_DIR_TMP" "$DATA_DIR"

What have we done? Well, we just created a copy from the original data to be modified and performed the alterations on that copy. After that, we just moved the current data directory to an old one and then moved our copy to be the main one. For this to be effective, you should consider that all these copy and moving operations are done within the same partition, because in that case moving data will be just a matter of updating the inodes table within the files system. In addition, if that script was killed in action, you'd have to worry only about repeating its execution and be sure it would finish its tasks.

I hope you liked this one. If the idea behind this is clear to you, you'll definitely adapt this concept to other scenarios as well.

Happy scripting.

--FIN--