The potential of AI in the Software Testing landscape

In recent conversations with multiple CTOs and developer tools founders, it's become very evident that testing continues to be one of the most manual tasks in the Software Development Lifecycle. While the wave of AI significantly has significantly impacted coding (GitHub has over 1Mn users!), various other parts of the development lifecycle still remain unimpacted, and testing continues to be the most manual task!

The last big advancement in testing happened nearly 20 years ago when Selenium was launched in 2004, and since then we've not seen any significant shifts in the landscape! There are still some key challenges that remain unsolved!

The Current Challenges in Testing:

1. Manual Test Creation: Every single test script requires meticulous crafting. Translating a test plan into a functional script isn't just resource-intensive but also bound by inherent human limitations. And let’s not even delve into the intricacies of adapting scripts for cross-browser testing!
2. Maintenance Overheads: Even minor CSS tweaks can send test scripts into disarray. It's disheartening to see the immense engineering effort that goes into merely ensuring tests remain functional after minor UI changes.
3. A Sea of Tests, But which ones to Prioritize? With thousands of potential tests, deciding which ones to run, especially given the frequent failures, becomes a difficult task.
4. Reactive Approach to testing: Our existing testing strategy is like treating a symptom, not the disease. We spend countless hours deciphering why tests failed rather than predicting and preventing those failures.

How the recent AI advancements can potentially revolutionize testing:

1. Automated Test Creation: Imagine AI utilizing inputs like code, API calls, test plans, Figma designs, or even PRDs to churn out test scripts. We’re essentially automating a manual translation process. While tools like Github copilot are focussed on unit-test creation, there's still a large whitespace in automated functional / UI tests, integration tests. The workflows will have to be specific to these types of tests.
2. Self-Healing Tests: An intriguing solution to the test maintenance conundrum. We're seeing early signs of AI essentially ‘healing’ test scripts that break due to UI changes, automatically adapting to the changes. For instance, if a 'login' button is moved around and remained to 'sign-in', AI will be able to identify the changes and heal the test scripts accordingly.
3. Synthetic Data Generation: AI can generate synthetic data based on production or sample data and mask / change PII data for testing. Though not the magnum opus solution in testing, they address the vital issue of data availability.
4. Test Prioritization: There are always 1000s of tests that can be potentially run. AI can help identify the most important tests that need to be run in that build process - the tests that are most relevant basis the changes made to the codebase / UI. Creating a test plan and automating the execution could be a game changer!
5. Predictive Analytics: QA teams spend massive amount of time figuring out why test scripts fail, and then manually make changes to test scripts and re-run the scripts. Predicting which tests are likely to fail and pinpointing the reasons, and recommending changes to scripts could truly help in moving from a reactive to proactive approach.

AI has the potential to significantly accelerate shift-left in testing, as unit testing or integration testing can now be fully owned by developers. Although, functional / UI testing is likely to remain in the hands of QA teams, but will be supercharged with AI!

For all founders, developers, and tech enthusiasts out there, the future of software testing is going to see significant changes in the next few years! If you're exploring or building solutions in the AI-driven testing domain or just want to brainstorm, I’d love to hear from you. DMs are open! ??