Test-driven system administration

Another day brings another incident from IT security department. Older Java versions were found on some AIX servers. We have to deinstall old Java versions 6, 7, and 7.1 from all AIX servers.

“Easy going”, says a seasoned AIX expert. Just go to every server and execute “installp -u Java6” and then the similar command for Java 7 and Java 7.1. Even every junior AIX administrator, who knows how to run smitty, can do it!

But let’s take a look from another point of view.

• I must have an evidence, which servers had older versions of Java installed.
• I must have an evidence, that these old Java versions are uninstalled.
• I must be sure, that these old Java versions will not be installed in the future.

It brings me to the point, that I must test my infrastructure. In software development the concept is called test-driven development. Before a software developer writes a line of real code, they define what they want to achieve and write a test. If you run it, it definitely fails. But then they write the code and if the code is written correctly, the test succeeds. Then the feature is implemented as it is defined by the specification.

Our specification is quite easy. The “feature request” (if I’m allowed to call it so) from IT security department, says that there should be no old Java versions installed on AIX. To test it, I have to execute the following commands:

lslpp -L Java6.*
lslpp -L Java6_64.*
lslpp -L Java7.*
lslpp -L Java7_64.*
lslpp -L Java71.*
lslpp -L Java71_64.*        

If every command fails (return code is 1), then older Java versions are not installed on the system and everything is OK.

For testing I use Chef Inspec. Chef Inspec is a platform agnostic testing engine for automated infrastructure testing.

To install it on AIX, you can download AIX version from power-devops.com.

wget https://dl.power-devops.com/inspec-5.18.5-1.powerpc.bff.gz
gunzip inspec-5.18.5-1.powerpc.bff.gz
installp -acgXYd inspec-5.18.5-1.powerpc.bff all        

It is installed in /opt/inspec and requires ca. 300MB. After the installation you have the new command called inspec. It is a swiss army knife for infrastructure testing.

Let’s create our first test. Choose a directory where your tests should reside and run:

inspec init profile java_aix        

First it will ask you to accept Chef Inspec license agreement. You are allowed to use it for non-commercial or experimental purposes. Because we are just experimenting right now, feel free to accept the license. If you’d like to use Chef Inspec at scale in the future, please contact Chef to obtain the commercial license.

Inspec creates a skeleton of the test for you:

 ─────────────────────────── InSpec Code Generator ──────────────────────────

Creating new profile at /home/user/java_aix
 ? Creating file README.md
 ? Creating directory /home/user/java_aix/controls
 ? Creating file controls/example.rb
 ? Creating file inspec.yml
        

Now we have to change some of the generated files. At first inspec.yml. The file contains the description of our test:

I would suggest, that you change title, summary and platform at least. The new inspec.yml could look like this one:

Now let’s make a step deeper and write the test’s code. Inspec created a directory named ‘controls’ and placed a file ‘example.rb’ into it. Rename the file into java6.rb and take a look into it:

As you see, it is written in almost normal English. You describe something, that you would like to test, and describe the behaviour of the object (tests). The tests begin with ‘it’ and then what should this ‘it’ do in curly braces, like:

it { should be_directory }        

As we discussed earlier, we execute a command and it should fail. Let’s try to write it in code:

describe command("lslpp -L Java6.*") do
  its('exit_status') { should eq 1 }
end        

I think it is pretty easy and straightforward, isn’t it? Let’s add some more description and our java6.rb test should look like this:

Now let’s check if it works. First check for eventually syntax errors with inspec. Go into directory where 'java_aix' resides and run inspec check:

If there are no errors, we can execute the profile on a local system:

I really didn’t expect that, but it looks like I have Java6 on the system, where I wrote the test. The test helped me to learn something new about my infrastructure.

Now let’s make the next step and add more tests. Copy java6.rb into java7.rb and change Java6 to Java7. Then do the same for Java71 - copy java7.rb to java71.rb and change the names. Now we have the complete testsuite and as we saw it earlier, it works.

Now we are ready to test our complete AIX infrastructure! Inspec relies on SSH, the same way as Ansible does it. You must have SSH access to every AIX LPAR in your infrastructure from the server, where your profiles are. You don’t need to install anything on that LPARs - only SSH access and maybe root rights. In our case we don’t even need root, because the command lslpp is available to every AIX user. To specify where the test must be executed, we should use -t option to exec, something like:

inspec exec java_aix -t ssh://user:password@server        

or

inspec exec java_aix -t ssh://user@server -i /home/user/.ssh/id_rsa        

If you use ssh-agent, you may omit password and identity parts:

inspec exec java_aix -t ssh://user@server        

Now I want to get the list of my servers and get those of them, which still have older Java versions. We can do it in one for-loop, using that inspec returns 100 if one of the tests is failed:

for h in $(< hosts); do inspec exec java_aix -t ssh://root@$h >/dev/null 2>&1; [ $? -eq 100 ] && echo $h; done | tee hosts.old_java        

“hosts” is our file with all our AIX LPARs. The resulting list of hosts will be shown on the screen and written into “hosts.old_java”. If you have many servers, it will take some time to test all of them.

Meanwhile we can write small Ansible playbook to remove the old Java versions:

When we’ve got the list of LPARs with old Java, we can run the playbook:

ansible-playbook -i hosts.old_java rmoldjava.yml        

And then test again, that everything worked right:

for h in $(< hosts); do inspec exec java_aix -t ssh://root@$h >/dev/null 2>&1; [ $? -eq 100 ] && echo $h; done | tee hosts.old_java        

If everything worked as it should, we should get no output and the file hosts.old_java must be empty. During my first run, I found, that I made a mistake in my Ansible playbook and forgot to remove 64-bit versions of Java. This is exactly the main reason to test your infrastructure - to be sure, that it behaves according to your and your customers' wishes.

It might look like an overkill for such a simple task. But if you have many servers and want be sure, that everything works as you want, you need testing. I just finished a big migration to a new storage. Everything was good and went well until I started testing. I ocasionally found that 5% of servers were not migrated as they should even if all scripts and playbooks worked well and shows me OK. Infrastructure testing helps to find such unusual problems and be sure in your infrastructure.

?

Have fun with testing!

Andrey