Automated Testing in DevOps.

What is Software Testing? Is Software Testing Necessary? How should you approach automated testing in a DevOps Environment? This article will try to address these basic questions. A clear understanding of these basic questions will help make the right decisions, investments on automated Testing in your organizations.

Software Testing can simply be defined as the process of finding bugs in the software or to provide information about the quality of software in question. It can also be defined as the process of checking if we are building the product right (Verification) and if the right product is being built (Validation). So the question now arises as to why do we build a defective product in the first place. Can’t we build the product First Time Right. There are two basic reasons for the same

1.      Humans make mistakes.

2.      Software Systems Are Complex.

So by Software Testing are we not trying to find defects in a defective software using a defective Tool (as Tests are built by Humans and have bugs)? Software systems being inherently complex, a fix at one place may create another bug at another place (that is why regression testing is done). Also Software Engineering is based on principles (which are open to interpretation) and not laws (like laws of physics) and hence test results cannot be extrapolated to arrive at the system behaviour with a degree of certainty. One of the factors that make Software systems complex is the fact that they are discrete systems, so for us to be sure about the system behaviour we need to test all the states. How many test cases do we need for a 64 state system? Are we really going to perform these many tests?

Now coming to whether the right product is being built (Validation). Validation is always done against a set of Customers requirements and/or by the Customers to determine whether the right product is being built. This assumes that the Customer knows what they want. Then how do you test a new idea in the first place. Well for a new idea, there is no need to follow the traditional Testing model in the first place, you need to get your product out in the market and test the idea first. You may have the world’s best idea but are the users willing to use/pay for it. Testing here needs to be done for the Idea and not the product.

 If the above points are true, then why bother about testing in a DevOps Initiative (leave aside automated testing). DevOps is only applicable for Tested Products. By tested product here I mean the Product Idea is market tested and you are using the product concept in managing the same. On the process of testing itself, remember that testing cannot give you the assurance that the software is bug free. It will only tell you that the tests work. It is imperative that proper thought is given to the testing strategy and options like Canary deployment are explored.

Now on the point of Automated Testing in DevOps, let me first list out the a few key points

1.      Not all Testing can be automated, you will still need to do some manual testing.

2.      Test automation project is another software development project.

3.      Automation testing is best suited for regression testing.

4.      Avoid automating GUI level tests instead focus on automating the API level tests.

5.      Do think about how you will analyse the automated test results.

6.      Each test case must be associated with a Business Impact and not with the software impact.

 Automating test cases is not easy at all and in most cases the effort involved much more as compared to a single cycle of complete manual testing. Benefits of automation can only be reaped after several cycle of testing. Below points need to be kept in mind while trying out automated testing

1.      Your automated test cases should be independent of each other

2.      Look at how you can use Data Driven Automated testing, it increases your flexibility to a great extent.

3.      Always revisit your test cases at a regular interval to see if they are still valid.

4.      Your automated test cases should finish in a short time period. For e.g. your Commit tests should not take more than 10 mins to finish.

5.      Think about how you can reuse your test cases. For e.g. using your System testing test cases to perform Performance testing.

 Your Automated Test Suite can be a great asset if maintained and used the Right way. Work the smart way and not the hard way to start your Test Automation Strategy.