Enable Continuous Testing in DevOps environment

Introduction

This is a pretty standard scenario from the quality engineering perspective where all the organization are trying to move towards DevOps. Organization have conventionally been doing automated functional and regression testing but with DevOps a change in approach towards continuous testing across DevOps phases is very important. In this small write-up I am trying to present an approach how teams/testing organization can move towards enabling continuous testing in DevOps.

Change of approach and mindset

I strongly believe that this change is driven more by approach and mindset rather than technological changes required as teams have been automating since many year but with emergence of DevOps we need to look at things in a different way. In my opinion below six mindset changes are import to be successful for CT enablement:

1.      Engage QE teams early: This is an important practice where QE teams are involved from the planning phase. This is done by following combination of processes and tools such as: TDD, BDD, static code analysis, code walkthrough, pair programming, mandatory unit test coverage etc.

2.      Be ready to discard: This is another important lesson where we need to change our mindset. In conventional automation approach we look at the ROI in the long run whereas that might not be a case here. In the fast agile there are instances where you may or may not use automation for longer time but at the same time is import to use automated testing to validate scenarios. For example validating huge data files, some intermediate changes to data which may not be there in further releases. In such scenario we should be ready to adapt mindset of developing, using and discarding scripts quickly. It does not always mean that everything we have built needs to be discarded but discard quickly as a possibility needs to be taken into consideration.

3.      Follow testing pyramid: Testing pyramid needs to be followed religiously thereby cutting down no of UI scenarios to minimum. This would help to create a more robust automation suite with less maintenance and would provide a better coverage and faster testing feedback to developers.

4.      Automate early: Focus needs to be on more automation. With increasing complexity and amount of data, it makes sense to automate test cases. It also helps to test features often and quickly. Other important thing is to automate early. This is where practices such as TDD and BDD comes into pictures and test are developed along with development and in some cases even before development has started.

5.      Service Virtualization: Virtualization helps to test component more comprehensively even when other system is not available or is not developed yet. This would help speed up testing giving more confidence to developers.

6.      Test more often and quickly: Test as often as possible – with every local build, with every new changes added, and in different environments. Including multiple quality gates and having more test coverage helps to build more reliable products and reduces chances of defect slippage in a higher environment.

Testing activities in DevOps Phases

Next step would be to understand different testing activities which are required to be performed in different DevOps phases. These can be broadly categorized as below:

1.      Planning Phase: This phase would involve creation of user stories, UACs, and coming up with solutions for the user stories. Static testing techniques such as review can be used to provide feedback in planning phase.

2.      Development and Integration: This is the phase where developers would develop features and integrate the code. Validation such as unit tests and checkstyles need to be performed at this stage to make sure that code changes are not breaking anything existing or have any conflict with any other developer who is also working on the same branch. Developers also use this phase to perform component testing of their development and virtualization can play a key role to help developers to test their changes and get more confidence.

3.      Building the code: As soon as developer checks in his/her changes based on check-in or feature branch merge the build would be triggered. This build trigger is according to the branching strategy deployed. At this point changes might either be pushed to a container image or to an environment. At this point it becomes critical to ensure that changes pushed in are not breaking anything and unit test coverage is used as a build breaker to ensure that at least at the code level things are good.

4.      Deploy and deliver: Once build is done, it is deployed in the environment and functional changes are required to be tested. All the conventional test phases comes into picture in this phase where functional tests and regression tests are performed to ensure new features are working fine and old features are intact. Functional/regression tests are used in this phase to ensure build is fine and ready to be deployed. After tests are good, build is deployed to higher environment.

You can see from above phases that same phases are automated using orchestration tools such as Jenkins and other automation tools are used to perform different activities in different phase. As the process is automated, testing is all the more important to give confidence before deploying a build and automation testing will play a critical role to perform validations across different phases in a seamless manner.

Create a continuous testing maturity model

Start with creating a basic model to represent continuous testing maturity. This will help you assess and standardize your projects and come up with a roadmap. For example: have different levels such as: Beginner, Intermediate, Advances and have set of questions associated with each level. Questions may be:

a.      Do you have automated testing in place for functional testing?

b.      Have you automated your API tests?

c.      Do you engage QE early in the development cycle?

d.      Are you following agile methodology?

e.      Do you plan to adapt DevOps? If so, where are you in the journey right now?

f.       Do you automate unit tests?

g.      Do you use checkstyles in your code?

h.      Do you follow TDD/BDD etc?

Based on answers of above questions, you may evaluate your projects and come up with a maturity level. By no means above list is exhaustive or the levels I have defines are sufficient. But this can be a good starting point to come up with your own maturity model for continuous testing.

Approach towards continuous testing in DevOps

1.      Assess current status: Use maturity model defined above to evaluate state of the project and come up with a maturity for each project.

2.      Identify gaps: Based on maturity level come up with a gap analysis report to help you understand the current state.

3.      Create transformation roadmap and strategy: Based on the gaps create a transformation roadmap which will help you to define strategy how you can make changes and integrate to DevOps cycle thereby provide value.

4.      Implement transformation strategy: Time to implement the strategy which you have defined in step 3.

I have tried to list down the approach at a very high level for transformation. I would love to hear what you think about this.