Ask your Testing Partner – Should we go for Complete Automation Testing?
Today, we are talking about bringing “idea to working code” in matter of hours if not minutes. The principle objective of software testing is to give confidence in the software – that is, fewer defects, less support costs, and improved credibility.
If you look at lean software development, it preaches that it is important to optimize the “WHOLE”, which means optimize everything from ideation to release in order to improve the quality and speed of software delivery. If you want to release your software quickly in a reliable, predictable, transparent and repeatable way – ensure that the risks are clearly understood when you add more features to it.
Automation is your answer when you want to test everything in short time, but one needs to automate quickly.
Automation inherently brings various benefits - it minimizes the effect of human error to a large extent. Hence, most of the consultants suggest to look out for the most error-prone stages in the SDLC and automate them early. However, Automation should be constructed such a way that it provides early feedback and improves collaboration between dev-test-ops. A good automated test suite allows you to perform refactoring / re-architecting of your application ensuring that if the tests pass, your application’s behavior hasn’t been affected.
But then question arises as to what to automate and when? The following table gives an idea of which tests you “must” automate, which ones are under “good to have” belt, and which ones you can “avoid” as they are better done manually.
- “The Must” – These should be definitely automated
Unit, Integration and System Test cases are the easy candidates to automate, as they are the best documented test cases (if at all), and they form a key part of your regression suite. It also gives you a higher sense of achievement to see that your tests cover virtually every code-path in the system.
Acceptance tests gives completeness to the working software. It verifies that the application delivers the expected business value by the user. A well-thought out acceptance test suite reduces regression effort, and prevents defects from breaking the pre-existing functionality.
Deployment tests indicate that your application is correctly installed, configured, connected (services), and is responding as expected.
- “Good to Have” - Let’s automate these
As applications are developed and accessed across multiple devices, Saasified, and started supporting multi-tenancy, Non-functional testing is becoming even more important. However, as much as it is talked about and felt important, these are often ignored or deferred. Primary reason being that these tests take longer to run, require special resources and environment, and the skills required to perform these tests are specialized too. With changing times, tools have matured and are getting adopted mainstream to avoid experiencing bad performance just before the release. Also, the cost of fixing these defects are higher towards the release than if done earlier in the cycle. However, complex tests need proper planning and research, preparation of special test environments, and allocation of project time for preparing, executing, and debugging these tests.
- “Can be avoided” – You still can continue manually
Of course, everything need not be automated. If you automate enough tests in the above 2 categories, it allows the teams to spend more time in improving the software quality through exploratory, usability, and showcases. If your application UI (look and feel) is undergoing changes then it best done manually, as it becomes extremely difficult to verify it through automation. Exploratory tests should be done manually, as it brings the best of your testers identifying scenarios which the automation tests might not target. However, if defects are discovered while performing exploratory tests, then these tests should form the part of your mainstream regression suite.
Though it is easier said than done. Automation requires absolute discipline. However, it’s worth it for the benefits you get from it. You not only are able to reach the market faster, you can release software faster that is reliable, and do it repeatedly with better predictability and control.
Head of Testing, Entrepreneur, Coach & Advisor
8 年Bhushit, Like I have mentioned in my other blogs Automation is a tricky affair. While it may be super beneficial in one situation, it would be a disaster elsewhere. So, pick things (based on your project situations) where automation is the only way to save cost and improve productivity, and know how much to automate (or not) in other situations. There is no cookie-cutter method as every project situation is different. However, if one takes care of certain best-practices, picks the right attributes, and creates automation which can be easily maintained then the ROI will certainly be higher.
Head of Testing, Entrepreneur, Coach & Advisor
8 年Bhushit, Thank you for your comment. I will try to keep my response short, and if that doesn’t satisfy you, I would be happy to discuss the same with you offline, unless other readers also feel the same. Having seen a handful of Agile projects, I believe, Automated Acceptance Tests have its benefits: 1. It gives a faster feedback loop (to both devs and test) 2. Especially when you are dealing with a large app your acceptance tests represent a powerful regression test suite - so you can know the effect on the related modules if you made changes to certain part of the application. 3. BDD tools like Cucumber, JBehave and Twist allow BAs to write requirements as executable test scripts (Given-when-then). Hence, your requirement document is never outdated. 4. Finally, it gives the freedom to testers (by reducing their workload), and allowing them to perform other high-value activities like "exploratory testing", instead of doing boring repetitive tasks. However, the caveat is following good practices and using appropriate tools to a point where the benefits clearly exceed the costs.
Head of Testing, Entrepreneur, Coach & Advisor
8 年Nabarun, yours is a trick question because, accessibility is given more legal importance than usability. For instance, usability issues limit everyone but accessibility issues limit a select group outside of their control. Hence, many countries have laws that make it possible for users to raise a civil case against companies that do not cater for people with disabilities. Also, many countries advocate that web sites that are not accessible are discriminating against potential users and customer. In fact there is a growing demand for accessibility testing. W3C web site has enough information on what to test, and how much based on your biz requirements The good news is there are many tools available for doing this test. However, a good understanding the tools and user roles is important to do justice to this type of testing.
Software Quality, CISSP
8 年Disagree. Acceptance test isn't something to be automated. With UI changing so frequently (or UI platforms changing so frequently) and the final say with customers, Acceptance Test Automation is a bad idea.
Focussed on service developments that helps increase customer's trust.
8 年You have comprised the whole software testing in a nutshell, thanks for this post. However I want to ask what are your views on accessibility testing. I have often thought about it but when it comes to writing a test plan, have seen people either prioritizing it low or not including it in test plan. What in your opinion should be a good practice for this? If we have a b2c model should we look at including accessibility in test plan or in exploratory testing?