Insights into the Continuous Testing in the DevOps Era

Hello readers, welcome! This article is part of the Z to A Pulse: Report spotlight, where we publish our thoughts on the findings from reputed names like: World Quality Report, Gartner, McKinsey, Forrester etc... We analyse their findings and trends with the help of our engineering leaders, then share the insights to help you achieve engineering excellence. Subscribe by clicking on the button above to read upcoming editions.??

Into the topic...?

In recent years, we've witnessed a shift in how software is built, refined, and delivered to the users. With the arrival of DevOps, engineering teams now find themselves at the forefront of a new era, one marked by a relentless drive for automation and the seamless flow of code through continuous integration and delivery (CI/CD) pipelines.?

We're at a crossroads where continuous testing isn't just a best practice; it's an important link to ensuring that your software is not just timely, but also of unwavering quality.??

Join me on this exciting journey as we explore the concept of DevOps and significance of continuous testing in today’s modern software delivery.?

Hi! I’m Keerthi V, Marketing strategist at Zuci Systems.?

Meet our experts:?

1. Saraswathi Jayaraman ?
2. Pratapsinha Suryawanshi ?

Saraswathi and Pratap have spent more than 50,000 + hours on helping engineering teams achieve exceptional user experiences within the dynamic context of CI/CD.?

Here’s what they have to share about the topic.

Keerthi: Brief about the role of continuous testing in DevOps?

Pratap: Traditional software testing in waterfall development methodology has been well known as a slow, costly process that delays releases while delivering questionable business value. As software becomes the key to creating a competitive advantage across all markets, enterprises no longer enjoy the luxury of selecting either ‘speed’ or ‘quality’ when delivering software. Both are critical.???

Now, with the maturity and acceptance of agile as well as DevOps initiatives across the corporates, Continuous Testing strategy helps enterprises accelerate and prioritize testing to meet the needs of fast-paced Agile and DevOps initiatives. This new strategy helps enterprises test smarter, so testing provides rapid insight into what matters most to the business.???

Implementing Continuous testing:?

The DevOps pipeline is a series of steps that automate the software development process, from code commit to production release. Continuous testing should be integrated into every stage of the pipeline, from unit testing to acceptance testing.?

A key technical activity is building and maintaining a set of automated test suites, including:?

Unit Tests: These tests focus on validating the functionality of a single method, class, or function in isolation. Their primary purpose is to provide developers with confidence that their code performs as intended. To ensure that code is testable, and tests remain maintainable, it's advisable to adopt the practice of writing unit tests before writing the actual code, a method commonly referred to as Test-Driven Development (TDD).?

Acceptance Tests: These tests operate at a higher level, evaluating a running application or service. Typically, they interact with the system under test (often by substituting dependencies with test doubles) to verify that broader functionality operates as expected and that no regression errors have been introduced. Acceptance tests serve to validate various aspects, such as meeting the business acceptance criteria for a user story or ensuring the correctness of an API. It's a best practice to incorporate these tests as an integral part of the development process. In fact, it's recommended that no work is deemed "development complete" unless automated acceptance tests are successfully passing.?

Below picture shows the types of automated and manual tests to run.?

When you encounter an error in an acceptance test or during exploratory testing, a proactive approach is to augment your testing with a unit test. This additional unit test serves as a critical safety net, ensuring that the same error is detected more swiftly, at an earlier stage, and with reduced costs in future testing cycles.?

This approach aligns with Mike Cohn's concept of the ideal test automation pyramid, as illustrated in the diagram below.

You might be interested in:??

Test Automation Enigma: Zuci View

Running tests continuously as part of a pipeline contributes to quick feedback for developers, a short lead time from check-in to release, and a low error rate in production environments. Developers have most of their work validated in a matter of minutes, instead of days or weeks, so they can fix bugs as soon as possible.?

The following diagram shows an example of a simple linear deployment pipeline. In this example, green means no problems were found, and red means that one or more problems were discovered.?

Running tests continuously as part of a pipeline contributes to quick feedback for developers, a short lead time from check-in to release, and a low error rate in production environments. Developers have most of their work validated in a matter of minutes, instead of days or weeks, so they can fix bugs as soon as possible.?

The following diagram shows an example of a simple linear deployment pipeline. In this example, green means no problems were found, and red means that one or more problems were discovered.?

?Ways to measure automated testing:

You can measure the results of automated testing in your environment by doing the following:?

?Keerthi: Would you share your thoughts on each of the models below? These are models adopted to enable continuous testing.?

Pratap:?

Continuous testing is widely used and effective in following development models?

1. Agile: Continuous testing is an integral part of Agile, where in product is built in small steps with regular feedback and testing. Continuous Testing helps product development team in building the product faster and better, with fewer mistakes.?

1. DevOps: Continuous testing is performed automatically throughout out software development life cycle (SDLC). Continuous testing works hand in hand with continuous integration to automatically validate any new code integrated into the application?

1. CTDD:?? Continuous test-driven development (CTDD) provides benefits of Test-Driven Development and allows for the automatic execution of tests, providing testers with continuous feedback on whether their code works.?

Keerthi: Share your thoughts on the below.?

Continuous Testing Report 2020 states that?

Test environments are one of the greatest impediments to continuous testing and Agile delivery: 36% of respondents stated that they spend over 50% of their time managing them.?

The survey also reveals that the ability to spin up environments dynamically will be key, changing and combining individual components at will in line with need. To that effect, Infrastructure such as code practices (IaC), Containerization, Cloud provisioning, and Service virtualization will play a significant role.?

Saraswathi:?

The evolution of cloud infrastructure usage has been interesting. ?

In the early days, it was straightforward - a single virtual machine accessed through SSH. But then, the second wave brought containers and provisioning tools like Dockers. Then Infrastructure became more dynamic.?

Today modern cloud infrastructure has added layers of complexity. It's not just about containers anymore; it's about serverless computing and an array of managed services, all woven into application architectures. ?

That's where Infrastructure as Code (IaC) steps in. With IaC, we can define the desired state of our infrastructure, making it more agile and adaptable.?

Moreover, IaC is a catalyst for the DevOps culture. It removes the boundaries between development and operations, fostering stronger collaboration. I believe that in coming years, we'll see a surge in IaC adoption, with many DevOps teams considering it a standard practice. ?

A typical IaC will look like this:?

• Developers initiate by defining and writing the infrastructure code.?
• The code is then committed to a version control system, often using platforms like GitHub.?
• It undergoes pull requests and peer reviews, ultimately ending up in a release branch.?
• Subsequently, the IaC tool assumes control and orchestrates the deployment of the infrastructure in the cloud or on-premises environments.?

It's like coding your infrastructure, making sure that everything, from servers to networks, is set up consistently and with precision. This means you can replicate your environments in minutes and no more hours lost on manual configurations.?

Containerization -- packages applications along with their dependencies, making them portable. Teams can run the same container on development laptop, staging server, and production environment, thus leaving less chances for environment-specific bugs. ?

Much like the importance of having test data readily available, virtual services play a crucial role in effective test environment management and the shift-left approach to attain continuous testing.?Service virtualization?mimics the behaviour of external systems, even when they're not available.? This keeps testing moving, even when external dependencies are down for maintenance or other reasons. The benefit of this approach is that any team can access any service on a virtualized basis, and test against it. It’s fast, it’s accurate and it will respond as though it’s real, which means teams can test in parallel.?

Having competence in these practices help set up and manage test environments. It’s timesaving and let teams focus on what really matters: delivering quality software faster. It's about optimizing the process and embracing agility, which is what continuous testing are all about.?

DevOps Research and Assessment (DORA) ?on implementing flexible infrastructure:?

Keerthi: State of DevOps report tells that??

Continuous delivery and version control amplify each other’s ability to promote high levels of software delivery performance.

Additionally, the teams that focus on the following practises tend to have more sophisticated DevOps processes in place.?

Loosely coupled architecture - the extent teams can make large-scale changes to the design of their system without depending on other teams to make changes in their systems??

Version control - how changes to application code, system configuration, application configuration, etc. are managed?

Continuous integration - how frequently branches are integrated into the trunk?

Continuous delivery - capabilities focused on getting changes into production safely, sustainably and efficiently?

These practices along with continuous testing foster software delivery performance that is greater than the sum of its parts.?

Can you briefly share your thoughts on these practices.?

?Saraswathi:?

A lot of organizations invest significant time and effort in technology adoption but struggle to achieve desired software delivery outcomes due to architectural limitations.?

In a tightly coupled architecture, even minor adjustments can trigger widespread, interconnected failures. This calls for?constant coordination among different teams within the system, often having to navigate convoluted and bureaucratic change management procedures.?

On the other hand, when the system's architecture is designed to empower teams to independently test, deploy, and modify systems without relying on other teams, communication becomes minimal. In essence, both the architecture and the teams are loosely coupled.?

Successful continuous integration lies in having:? Automated build process that create versioned, repeatable packages for deployment, running the process daily.?

Automated test suites: Start with reliable unit and acceptance tests, expand coverage for new features, and ensure daily feedback to developers.?

CI/CD is both?continuous integration and continuous delivery combinedly put together. To make sure your software releases are dependable and safe, it's crucial for everyone involved in the process, not just the developers, to work closely together. And your team should be open to learning new ways of doing things and acquiring new skills.?

Key Takeaways:?

Strategies to deploy continuous testing in DevOps:?

• Embrace Automation: Build your strategy on test automation for faster, more consistent testing.?
• Shift-Left Testing: Start testing early in development to catch issues sooner. New automation methodologies will continue to evolve in shift-left including model-based testing, BDD, TDD, and in-sprint security tests.?
• Data Management: Ensure comprehensive test data management for effective testing.?
• Service Virtualization: Simulate unavailable components for more thorough testing.?
• Continuous Monitoring: Keep a close eye on test results and defects for real-time feedback.?
• Parallel Execution: Run multiple tests simultaneously to save time.?
• CI/CD Integration: Integrate testing into your CI/CD pipelines for seamless automation.?
• Test Environment Management: Efficiently manage test environments to prevent delays. The ability to spin up environments dynamically will be key. To that effect, infrastructure such as code practices, containerization, and virtualization will play a significant role.?
• Collaboration and Communication: Foster teamwork for faster issue resolution.?
• Continuous Improvement: Regularly refine your strategy for ongoing success.?

Question for you:?

?Where are you in your continuous testing journey??

• Beginner?
• Intermediate?
• Expert?

Thanks for reading! We're excited about delving deeper into the insights from the World Quality Report in our future editions.?

If you like our content, show us your support by - Subscribing to our quarterly newsletter, Sound bytes here. ?and by sharing this article with your team. Your support means the world to us!?

?