AI in Testing: A Three-Part Series on How AI is disrupting Testing (Part 2)

Note - This article is part of a three-part series on the evolution of AI in testing. In the previous article, we explored the Co-Pilot Era of Testing, where AI assists in decentralized, task-specific testing. In this article, we’ll dive into the next phase—the Agentic Era, where AI takes on a more central role in orchestrating integrated testing workflows.

Part 2

What’s Next: The Agentic Era of AI in Testing

The next major shift in AI-driven testing will be the emergence of what I call the Agentic Era (or wave), where AI transitions from a supporting role to a central, orchestrating force in software quality engineering. This phase is characterized by AI systems that act as agents, capable of managing entire testing workflows autonomously and intelligently. No longer limited to isolated tasks, AI will execute, analyze, and optimize testing in an integrated, end-to-end manner.

The Leap from Co-Pilot to Agentic AI

The evolution from the current “co-pilot” era—where AI assists in decentralized, task-specific automations—to the Agentic Era will be enabled by several key factors:

• Increase in LLM Context Windows: Large language models (LLMs) are becoming more powerful, with significantly larger context windows. This enables AI systems to process and understand more complex scenarios, workflows, and application data all at once, providing the intelligence needed to handle integrated testing processes.
• Reduction in Token Costs: As token costs for calling models decrease, the economic feasibility of using AI to handle more complex, resource-intensive tasks increases. This makes it more practical to deploy AI at scale, allowing it to take on larger and more dynamic roles in testing environments.
• Proliferation of Agentic Frameworks: The growing availability and democratization of AI frameworks such as Crew AI, Microsoft’s AutoGen, and Langchain are major enablers of this shift. These platforms allow for the creation of complex and integrated AI agentic workflows, where AI systems can not only perform individual tasks but also manage entire processes, make decisions, and adapt based on real-time data and feedback. These frameworks bring more powerful, accessible tools to organizations, making it easier for teams to integrate AI into testing without requiring extensive AI expertise.

This combination of technological advancements is paving the way for the Agentic Era, where AI will no longer just assist in testing but will drive the process from start to finish, using its expanded capabilities to deliver smarter, faster, and more comprehensive testing solutions.

What Defines the Agentic Era

1. AI-Driven Continuous Testing: In this new era, AI will be embedded within Continuous Integration/Continuous Delivery (CI/CD) pipelines, performing real-time, automated testing with minimal human intervention. AI will continuously assess code changes, execute tests, and provide actionable insights, ensuring that defects are caught early and that quality is maintained throughout the development lifecycle.
2. Integrated and Dynamic Risk-Based Testing: With enhanced context windows and access to more data, AI will be able to assess risks dynamically. By analyzing historical data, code changes, and user feedback, AI will prioritize testing efforts in areas with the highest risk, ensuring optimal coverage while reducing redundant tests. This allows for a more targeted, efficient approach to quality assurance.
3. Self-Sustaining Test Maintenance: The need for constant script maintenance will fade as AI takes on the task of autonomously updating and optimizing test scripts. As applications evolve, AI will recognize changes in the code or UI and adjust tests accordingly, ensuring that they remain relevant without manual updates. This capability will dramatically reduce the burden of test maintenance, freeing up human testers for more strategic tasks.
4. Autonomous Issue Detection and Resolution: In the Agentic Era, AI systems won’t just flag defects—they will also provide detailed insights into the root cause of issues and suggest potential fixes. This capability will reduce the time developers spend troubleshooting and improve overall development efficiency.

The Role of Humans in the Agentic Era

As AI takes on more responsibility, human testers will evolve into agentic architects and engineers, focusing on defining and optimizing testing agents and workflows, ensuring the right data is used, and refining the outputs generated by these systems. Rather than performing repetitive tasks, human testers will work alongside AI to ensure the highest levels of quality, adapting the system as needed and guiding its evolution.

Critical skills for testers to remain relevant in this stage:

As we move into the Agentic Era, the role of testers evolves as AI takes on more autonomous roles in managing testing workflows. Agentic AI workflows mean that AI is coordinating multiple tasks, making decisions, and even optimizing its own processes. To stay relevant, testers need to understand how these agentic frameworks operate and how to design workflows that ensure the AI operates effectively.

Key Focus Areas:

• Agentic AI Frameworks: Testers should develop expertise in agentic frameworks like Langchain, Crew AI, and Microsoft AutoGen. These platforms allow testers to design AI-driven workflows that can execute testing tasks independently, requiring a solid understanding of how these frameworks work, and how to leverage their features for integrated testing.
• Process Documentation & Optimization: In the Agentic Era, testers must ensure that AI workflows are documented properly, including detailing the processes that AI should follow. As AI takes on more responsibility, testers must focus on improving and refining workflows, ensuring that AI systems can operate efficiently while adhering to best practices.
• Data Integration & Management: Agentic workflows require extensive data interaction. Testers should be familiar with managing the integration of various data sources (APIs, databases) into testing workflows, ensuring the AI has access to all relevant information for testing.

Why It’s Important:

In the Agentic Era, AI will drive the testing processes. Testers will need to focus on strategic oversight, desiging and deploying teams of AI Agents, which will be able to leverage memory and tools to get the job done. They will be expected to ensure that these process workflows are well-designed and AI operates efficiently and effectively. Mastery of agentic frameworks and process optimization will become the new normal to speed up the work, enhance testing coverage and significantly improve the quality of the output.

Next week, I'll cover the logical evolution from this stage, where we will finally see AI take the wheel, and the focus of testers will inevitably shift towards testing AI models themselves, checking for accuracy, consistency, bias, etc to ensure transparency and trust on the models who would do the application testing themselves. Stay tuned!