Strategy for Test Automation

Gartner defines Software Test Automation as a "subset of the overall software testing tool market focused on automating the validation of applications and services against requirements. It includes tools for unit, API/service, and user interface (UI) functional testing, as well as load and performance testing in an automated way".

This is a very broad domain which needs a good vision and strategy to be put in place to reap business benefits and avoid slowing down the development.

Software engineering has undergone huge changes over last few decades. But some things are still common. What has changed is the need to do things faster, better and with minimal manual efforts. The way that software engineering has changed is analogous to the way cars have changed in the last half century. Cars still have doors, engines, and steering wheels, but now they perform better and employ AI. They have improved security and increased resilience to better withstand an accident. Same is the ask in software engineering.

BDD (Behavior Driven Development) and Test-Driven Development (TDD) strategies have been adopted for many years now. Gherkin is one of the most popular language used in BDD. Selenium has been and is still very popular open-source software testing tool.

Question is whether these approaches, languages and tools are still very effective. Or enterprises need to look at bigger picture of what their testing needs are and then decide on the right test automation solution.

To put this discussion into context, let us consider any large life sciences and health care (LSHC) customer. They would broadly have 3 major verticals (could be more) – Medical, Life Sciences and Interventional. Each of these verticals will have sub verticals and will have many software applications getting used either built in-house or bought off the shelf (COTS) and possibly customized (PCAS: Picture archiving and communication systems, EMR: Electronic Medical Records etc). They will also have hardware devices – wearables or onsite medical devices (X-ray machines, CT scanners, MRIs). In case of onsite medical devices usually the software is embedded in these devices, making it difficult even to access the codebase. On-site medical devices: Systems that use medical imaging techniques, including X-rays, computed tomography (CT) scans, and magnetic resonance imaging (MRI), are often built with custom code. And for wearable devices, simulating network conditions to ensure the medical device continues to work across cellular (5G), cabled, or a WIFI connection is essential. Data transmitted back to an EMR is crucial to maintain quality patient care. And for any LSHC customer, HIPAA (Health Insurance Portability and Accountability Act) compliance is a mandate.

Assuming above scenario, how efficient it will be to use BDD and TDD approaches and still be able to cover “API/service, and user interface (UI) functional testing, as well as load and performance testing in an automated way” using 1-2 tools ?

This is where I believe many organizations struggle. They end up deploying many different tools and frameworks in silos for individual needs. And as the ecosystem of applications, devices grow, more customers get onboarded, all this becomes unmanageable. This is when transformational projects are discussed and executed.

So what could be a great solution?

If we look at Gartner or Forrester magic quadrants, we will see some prominent tools which feature every year. Let us consider SmartBear, Tricentis and Eggplant for this discussion. They all will have some commonalities and some differences. Few common elements which all these Next Gen software automation platforms are

1. No Code / Low Code test cases authoring. Idea is not to learn programming language to write test cases, and anyone should be able to create test cases easily.
2. Machine Learning capabilities like OCR, Computer Vision for scanning through UI without need to call APIs and do a comprehensive testing and validation. They check for subtle changes in the applications like font size, colors, font type, screen resolution etc. Features like bug hunting, coverage analysis and extremely useful.
3. Performance and Load testing - Customers do not want to use separate tools and frameworks for doing this.
4. API testing – Ability to test APIs.
5. Accessibility Testing - Compliance to WCAG and ensuring inclusivity in software has become extremely critical. Americans with Disabilities Act (ADA) is a law and nonconformance to the same can lead to (there are several examples) litigations and penalties.

They also use something called model-driven testing. It is an application of model-based design for executing artifacts to perform software testing or system testing. Model is basically a digital twin of the application through which we interface to mimic a user journey. Digital Twins have become popular and very successful in the IOT (Internet of Things) domain. The same concept has been brought now into software test automation.

This helps to create a model for an application, update it as application evolves and then use this to create test cases by replicating a user journey.

These platforms can connect to web applications, mobile devices, IOT devices, desktop applications and anything else. We can test custom developed applications written in any programming languages or packaged applications like SAP, Oracle, Salesforce.

Given these new age platforms and how they cover the breadth of testing needs, would we still want to do testing using traditional tools?

Would love to hear and learn from views from others.