Recipe for CI-CD — Chapter Three: Automated Testing

At the times when digital disruption is the need of the hour, an organization can’t differ implementation of continuous deployment, not at least for their first line revenue products. Continuous deployment —?CD, comes together with Continuous Integration —?CI, hence one of the hottest acronyms of this time —?CI-CD.

Hoping you enjoyed the first two chapters of this series and have added the prescribed ingredients of right?product culture and teams, and?development practices?into your preparation of continuous delivery of value to your customers.

It’s time of the 3rd ingredient for the recipe,?automated?testing. The emphasis here is more on the word automated?than testing. Testing has always been an integral part of software development cycle. It is said that; testing may take up to?40–50%?of your SDLC timeframe. However, as discussed in other chapters, in the implementation of CI-CD there is no place for manual testing. All aspects and level of testing must be automated and incorporated in your delivery pipeline.

Let’s go for it.

Why testing

First let’s brush-up on the point why testing is even needed.

When you board the train of CI-CD, you will enable the business with delivery of business features, bugs and improvement in customers hand at any time of the day. That would mean continuous integration and deployment of product to production during business hours. This easier said than done job demands a huge mind-shift and changes in ways of working.

Any deployment to production had to be quality checked to ensure highest quality delivery and least possible impact to end customers. In an agile delivery world, a faster feedback cycle plays a critical role. If a deployment to production cannot be quick and safe at the same time, continuous deployment can turn suicidal very soon.

The delivery mechanism must ensure that, any issues are found and reported back as soon as possible. Testing is the key to control quality and give business the required confidence of delivering right bug-free features in the hands of consumers.

However, if the testing is manual, the delay and loss to agility is tremendous, and that’s the shift in approach you need to bring in.

Why automated

In the world of continuous delivery, testing cycle or testing phases of SDLC turn into testing stages of your build and deploy pipeline.

As the first principle of DevOps, automation should be the essence of every work carried out for CI and CD. If you have ever visited an assembly line of any manufacturing unit, you would have noticed how streamlined the flow of items take place there. In new generation line, almost all work items are carried out by one or the other way of automation, to enable speed and consistency in delivery.

Similar philosophy applies to software build, where success of the build pipeline and speedy delivery depends on how many of the build and deploy stages are automated.

Complete automation of all testing aspects, starting from unit-tests, integration-tests, contract-tests, system-tests, load-testing to site reliability testing. The aim must be to achieve fully automated testing, that becomes an integral part of the deployment pipeline for CI-CD of the project.

Good enough; the question now arises, who writes and automates these tests.

Code is incomplete without tests, so, shift-left

Just to say it one more last time, you can’t afford to have any testing cycle/phase in the delivery or any manual test suites when aspired to incorporate CI-CD as your delivery principle. Every aspect of testing must be automated. Automation demands code such that all testing suites can be run by the build pipeline.

In the chapter of?Development Practices, we talked about a phenomenon of shift-left.?In new ways of working, the shifting left makes it a responsibility of development team to write and maintain testing that is deemed necessary to ensure quality deliveries of the product. The test code must be produced alongside the production code, and at the same time.

You may argue that, an average developer may lack knowledge and expertise of right tools and frameworks that may be needed to serve the need. And, I accept that. That is where your multi-speciality team, that we have talked about in last two blogs, shall come to rescue. The QA specialist shall help the team to get right tools and techniques set for the team. Up-skill everyone with frameworks and methodologies that will set pillars for your testing. It shall be her/his responsibility to setup testing strategy that works for the product, and continuously evolve it as per the changes and refinements needed.

Once that knowledge is imparted in the team, it becomes part of everyday development to adhere to these strategies and ensuring right test coverage in automated fashion.

Build a testing strategy

Anything would lack a vision if not defined and guard-railed by a strategy, so is the case with testing. You need to have a defined strategy on how your team will approach testing, which layers of your architectural components be tested, how many tests for each layer etc. There is no silver bullet to bite, and you will need to adjust, accommodate and mould it in a way that fits best for you.

The testing strategy is better defined when described in a testing pyramid (or a new testing trophy) style. Being a pyramid, it has a wider base, indicating a big number of quick and simple unit tests to give a strong base, but converges at the top to indicate to have lesser deep but brittle end-to-end tests.

Here is an image, just for reference and not as a prescription, that is followed by a fictitious team. It broadly describes the layers the team needs to consider, at the minimum, for covering with tests. It also indicates the number of tests the team shall target in order to keep the pyramid in balance. You must also build with one such pyramid and keep evolving it with any required changes as you grow.

The key here is, not to leave any layer untested.

Do enough testing, but don’t overkill

We talked a lot about testing, test coverage, quality control, testing all layers and what not! But, as the simple principle of the nature, every benefit comes at a cost.

When we get busy adding adequate test coverage for our code, we also need to be conversant of the pit-falls where we may start testing same code path or functionality at multiple layers and places.

For an example, if you have already tested all the possible functional behaviors in your unit tests, you may need not to test them again in integration or UI tests. Of course, there may be some degree of duplication but try to keep them at the minimum.

A practice called?Example Mapping?can come handy here in order to surface all the possible scenarios right before starting with the testing, and then cover them as per the best possible fit to your testing strategy and pyramid.

Writing Testable Code

To start with tests, you first need to write code that is testable. Consider an aircraft manufacturing process — before any model first takes off to show that all its complex systems work well together, individual parts are tested to guarantee they are safe and function correctly

Software is similar. Instead of writing your entire program in one huge file with many lines of code, you write your code in multiple small modules that you can test more thoroughly than if you tested the assembled whole. In this way, writing testable code is intertwined with writing clean, modular code.

To make your app more testable, start by separating the multiple components of the whole app construct into smaller units, which are called?subject under test. For example, separate the view part of your app from your business logic in the app. This way, you can keep your business logic testing independent of the UI components themselves.

However, making a code testable is the most difficult work and a lot of patch work is needed during writing tests if the tests are not added at the right time. I can personally guarantee and assure you that, there is no other better time, to add tests and make your code testable, than before writing the functional code itself. Yes, tests come into existence before the code does, and that is one of the pillars of?eXtreme Programming?practices, called TDD.

TDD

Test Drive Development, also lovingly called Test Driven Design aka?TDD?is a practice that is not new but still not as widely followed in software industry due to lot of misconception of it being costly to write. It a very simple but radical ask to perform testing (in the form of writing?tests?for the expected behavior under development), coding (by writing the minimal code that make the test written in previous step pass) and designing (in the form of?refactoring), hence the famously known?test-code-refactor?cycles.

TDD is a whole different world of writing code, which gives us benefits such as,

• Code with almost full test coverage
• Making tests first class citizen of your code, making them the first consumer of your functional code. Resulting in better designed code.
• Minimal line of code produced and no futuristic development.
• A good opportunity to make you QA specialists give their inputs and pair with developers, if needed, to come with right amount of testing

Though many teams report it as time taken exercise in development as it forces one to think a lot to write the test scenario upfront, but they in fact lack the view of complete benefits one can reap out of it in the total course of development?and delivery.

Veteran practitioners report that TDD leads to improved design qualities in the code, and more generally a higher degree of “internal” or technical quality, for instance improving the metrics of cohesion and coupling.

Try it well before discarding it for ever!

Summary

In this blog we got close to see how and what differs in testing approach and implementation for a product that you aspire to deliver in CI-CD style. The intent was never to tell you how to write test cases, which tools to use or the various taxonomy around testing, but to ignite the radical aspiration of have fully automated test suites for your product, that can take your business further up on the path of continuous delivery of value to the customers.

In the?next chapter?we will talk about the Production Support aka Operations aspect. It’s the key driver on defining products and following product model. In the case of CI-CD, it gets even more crucial as the pace at which products change it can really get difficult to support it.