The future of software testing and the idea of script-less automation

The future of software testing and the idea of script-less automation

The world of software testing and quality control is moving towards a paradigm shift. As the amount of data increases and the need for more efficiency presses itself, the focus is being set more on test automation. As established worldwide, the process of software testing is a never ending, always improving process. There is no such thing as a totally bug-free, perfect application. However, a well tested application is always possible. And it is this scope for improvement that keeps the spark in testing alive.

Automating the right things

Of late, many organizations have started looking up to automation as a do-it-all solution for software testing. These pressing demands from clients have translated into a rat race for achieving the maximum automation percentage possible. This is regardless of the fact that higher automation percentage does not always guarantee better testing quality.

Automation is the right way to go ahead with testing in many situations; however it is of utmost importance to select the ‘right’ situations to automate.

For an example,

If a test case is as follows :
1. Login
2. Check if GUI element is at desired position
3. Logout.

If this test case requires 100 GUI elements to be validated, it is optimal that the 100 single page GUI validations be checked in one iteration of login and logout. Another way of making the instances could be to make 100 instances with each instance doing login and logout. This method will add to the automation percentage in a huge way. You can get a 100 instances of automation compared to just one. However, it will unnecessarily increase the resource utilization and execution time.

Selecting the right candidate for automation also depends heavily on the Return on Investment. Automation is Artificial Intelligence. Which essentially means that it is a dumb system requiring each instruction to be given correctly by a human. This means that a lot of data setup and lines of code are required to execute a well built test case. This is a significant investment on the tester’s part, and thus requires a careful selection of scenarios to give sufficient returns on this investment. An ideal candidate could be one that can get 200 instances executed through a single script in the backend, giving the tester room to focus on other areas. However, if a test case has only one instance that will only be executed once in the entire testing life cycle, it is better to be left at the bottom of the automation candidate selection pool.

These factors aside, the world is moving towards more and more automation. Industries are looking to cash in the high efficiency and productivity offered by it. There is a huge magnitude of processes just waiting to be automated. With a targeted automation strategy in place, testers have enormous potential to cash in on this surge.

Client criteria for good testing is changing

Nowadays, clients are pushing towards more quality as a criteria to evaluate testing. Gone are the days when higher number of test cases where considered as the work (and ultimately pay) parameters by the client. If the defects uncovered are high, the testing is going good else the high number of test cases do not mean anything. Some testing firms have suggested improved metrics of quality to tackle this. Few methods could be to assess how much risk coverage is present in the testing plan.

This could be achieved by working along with business to understand the risk areas of the application, and propose test solutions based on this to the client. The more the risk covered, the better the testing done. Another could be the amount and quality of defects unearthed. However, this would undermine the testing efforts in the case of a well developed application that does not have a lot of defects. At the end of the day, agreeing on a fairly quantifiable metric for assessing testing effort is a complex process, and should be worked out suitably.

The Ideal testing team of the future

With the shift towards automation, it has become essential for testers to evolve with the changing times. Testers have to adapt and get acquainted with automation skills and practices to stay in the game. Some traditionalists may feel that this will rob the quality and spirit of testing from the application. A manual tester, well versed with the application, will always figure out better ways to investigate it for defects and can also do ad hoc testing as per his/her instincts. This is not achievable through modern automation scripts which do not have a ‘soul’ and can only execute pre-scripted fixed steps.

However, it is to be understood that the shift towards automation does not necessarily mean compromising on human expertise. Automation should always be the powerful tool in the hands of the tester. The tester is the brain, the mastermind of the whole testing operation, whereas automation is the tool that will help him do his job faster. Automation will be the way to tackle huge, redundant and repetitive testing so that the manual tester can focus his energies in investigating the application further. Moreover, Application and domain experts will always be the niche segment that will drive the testing effort in the right direction.

The ideal testing team of the future should constitute a small section (15-20%) of application and domain experts involved in manual and exploratory testing. This segment will constantly aim to design and engineer quality test cases that require manual testing. They could also work on providing value addition to the client by testing for hidden defects that are not evident from the client requirements.

The majority of the team would be involved in the requirement analysis, test planning, execution and defect logging. They would constitute 55-60% of the team. It is this segment that has to be automation enabled. They should have the ability to identify right candidates for automation and automate them/send them for automation. Automation would be an added skill that will help them efficiently handle their testing efforts. Although not expected to be experts in programming, they should have the ability to run the automation test cases and pull the automation test reports. It is fair to say that this is asking too much from this group in terms of their skill sets. There should be ways to make them automation enabled without having to get trained in automation completely.

The last segment(around 15-20%) should be automation experts adept in programming. This group will be responsible for the creation and maintenance of automation frameworks and tools. They will also impart automation skills to the rest of the team. This group of automation engineers will aim for maximum achievable automation of the right candidates to achieve client expectations. Artistically speaking, this would be the engine driving the automation enabled testing team.

Challenges in automation enablement of manual testers

The path for automation enablement is not without it’s fair set of challenges. Couple of the major challenges are :

- Training and education of manual testers without compromising on ongoing testing activity.
- Tackling issues of automation artifact maintenance and uniformity (E.g : If a lot of people are accessing a single object repository, it may get corrupted)

Are Script-less automation tools the solution ?

With training and automation enablement being the biggest challenge, many programmers have tried to come up with easy and intuitive tools to enable automation. These tools bring the idea of ‘script-less automation’ to the table by generating scripts out of front end user input. The tester is asked for data inputs through a user friendly form, which are then utilized for script generation at the backend. The whole process is black boxed to avoid unnecessary code overload to the manual tester. Some tools like Tosca TestSuite, Turnkey cFactory etc. are already being used by testers worldwide. These tools address the major issue of programming skill scarcity by carrying on the major coding at the backend.

But there are still a lot of challenges faced through this approach. Some of them are :

1. Volatile applications – Some applications are volatile and non-uniform in nature. This could be due to faulty application design or on account of the dynamic nature of the application. Also, many a times WYSIWYG is not true. An element appearing as a weblist may actually be a combination of a text box and an image. In some cases, data is not entered in webedits through regular methods and require shell scripting instead. All these deviations require additional coding and dedicated functions. An SAT(Script-less Automation Tool) has to tackle these issues by offering dedicated instead of generic scripting.

2. Some flexibility is required while automation – Traditional scripting offers flexibility during debugging and execution. Many a times the script needs to be executed till a particular step to observe application behavior based on which further scripting is decided. For example, some steps might require the script to wait a bit longer to avoid object identification issues. For some test cases, the script might need to be modified frequently. For example, if multiple defects are stopping a test case flow and require some temporary workarounds to be scripted, the script will need frequent modifications. All these things should be possible to address using the tool.

3. High dependency on tool backend programmers and engineers – During transition to SATs, there will be a high dependency on the tool enablement team and engineers. Initially, the tool will face a lot of teething problems while getting aligned with the target application. During this phase, it is essential for the tool engineers to be in constant touch with the test team. This challenge is further aggravated if the said team is present at different geographical locations and time zones. Personal interactions and training can go a long way in facing this challenge.

Lastly, since it is human tendency to reject change, it might be difficult to convince clients and testing teams to transition to these new strategies. The testing world is demanding more and more automation, it is crucial that testers adapt to survive. The “If it is working fine till now, why should we change it” mentality will not do a lot of good into the evolution of the testers and will give them a hard time when automation disruption strikes the world of testing.

P.S I have been inspired to pen down my thoughts as above after attending a session by Mr.Wolfgang Platz, CPO Tricentis, when I could connect with his views regarding the future of testing. Thank you for this inspiration!