Effective Testing with Effective Pipelines

For any business to be successful, business agility is a given these days. What was relevant yesterday might be obsolete today. That's how quickly things are changing in the market. To manage delivery of highly-valued products and services while ensuring top quality involves effective delivery pipelines that incorporate automated tests at the right levels and stages.

This time, let's hear it from Michael Tomaras, Head of Technology in financial services, a truly inspiring leader in technology with a vision of fully automating delivery pipelines from feature-ready to production with no manual intervention.?

There is no doubt you'll have a ton of takeaways from his deep expertise he's kindly shared in this newsletter. Let's hear and learn from him.

Tell us about your career journey

My career has been as vanilla as it gets. I went straight into a junior developer job after university and I followed the IC (Individual Contributor) ladder pretty smoothly over the next 10 years until the Senior Software Engineer level.

Back then we were more “full stack”, “T-shaped” (or more accurately “jack-of-all-trades”) developers but the things I enjoyed most were automated testing and delivery pipelines. I was almost obsessed with getting a commit from the ‘main’ branch into production with no human interaction and maximum confidence in quality, performance and security.

At some point, I realized that no IC is more productive than a well-oiled team, and also that the system that creates and delivers work extends around the team too, across the whole business. No matter how great a single team is, it will suffer in an organization with no alignment.?

I got interested in designing and creating systems of collaboration as well as systems of software. They had many similarities but also critical differences and that’s what made them interesting.?

The next steps in my career were Team Lead, Head of Software Engineering and now Head of Technology.

How have you seen the testing landscape transform over the years?

There’s two ways to look at it: from a point of view of the practices and principles used and from a tooling one.

Test Principles:?

As far as principles are concerned, very little has changed and for good reason: they were defined decades ago when software engineering was just starting up, and proved to be working ever since (you’ll notice that a lot of software principles and protocols stand the test of time, most just being recycled under different names and frameworks).?

For example, for any test you write, you must make sure you identify and isolate your System Under Test (SUT) properly, give it some expected inputs and compare those to some expected outputs.?

Different scopes require different SUT setups and input-output parameters but the process is always the same.?

Do not test your SUT’s implementation directly, do not leave in dependencies that make the SUT’s behavior non-deterministic (e.g. a Date generation code that uses clocks outside your SUT or APIs that may or may not respond successfully in every single call).?

We could build a few semester’s worth of curriculum on this topic!

Test Tooling:

Tooling has changed dramatically over the last 10 years. The process of setting up tests of various scopes and then running them used to be a lot more manual than it is today.?

A lot of tooling has come out that pretty much abstracts all the setup and running of tests, as well as incorporating it in the deployment process, saving countless developer hours.?

You can create all kinds of tests today (unit, integration, UI, API, stress, performance, security) just by pulling together the right tools and pointing them at the right codebase or system.

Add to that the commoditization of infrastructure, that gave us the ability to spin up virtually unlimited hardware to run those tests, and the scale of testing performed has exploded in an exponential manner. And that’s a good thing!

Why is there a need for incorporating testing with delivery pipelines?

First let me take a step back and rephrase that question:?

"Why do we need automated testing at all?"

And the answer is simple and central to all steps of the SDLC: feedback.?

User feedback:

Whatever we do in software development (and business development) is about getting something in front of users and seeing if they like it or not. We fix what they don’t like, then rinse and repeat. What makes or breaks a business is the rate at which it can try out ideas, the faster the better.

Business feedback:?

So there is a business wide feedback cycle that goes from “idea” to “customer using it” and back. But, that cycle also includes many smaller concentric and overlapping cycles. Some of them are scoped to software development.?

Development feedback:

And the feedback that software engineers need is whether their code works as intended and whether it broke any previously delivered features.

The next question is:?

How fast can we get that feedback to software engineers??

Local environment:

The fastest way is locally, at the moment where they create the code. With today's? tooling, for every line of code written, build and all tests are automatically run, highlighting failures straight in the IDE. The feedback cycle is literally seconds. But it’s not perfect.?

System Integration environments:

Some kinds of tests are not able to run locally because they depend on systems that are not reachable from the developers' machine or because their scope is more than the code (for example think performance tests, they need to run against production level infrastructure, running them locally will not produce reliable results).

Also, when more than one person contributes to the same codebase, how do you know that your code works with the contributions of everyone else?

This is my long winded way of answering your initial question (finally!).

Incorporating Tests in Pipelines:

Incorporating tests in pipelines gives slower feedback to the developers than the ones run locally but:

(1) it allows us to test with scopes further than our one codebase.

(2) it is an integration point of all contributions from all team members, making sure that we have collectively provided the correct test coverage.

Most organizations have it as a rule to incorporate unit and integration tests with the delivery pipelines. What other types of testing do you see as crucial to incorporate with these pipelines?

First of all I’d say that it’s less of a “rule” and more “common sense”.?

Why would you not want that feedback as soon as possible??

Why not fix the bug that you introduced as soon as you build the codebase rather than after you’ve merged it and got a notification of a test failure several minutes later?

“Crucial”. As with many things in software, the answer is a vague and generic “it depends”. As we mentioned before, automated tests are a very important form of feedback that, on a commit per commit basis, answers the question “is this snapshot of the code ready to go into production?”.?

“Ready for production” means different things to different businesses. Some are highly regulated, some are focused on performance and scale, some have serious security considerations. If you were to put a piece of code into production what would you manually check?

The UI/UX? Specific business rules?

How fast does the system respond??

Whether audit trail was successfully updated?

Whether you added authorization correctly?

Great, take that list and automate all of it.

You’ll create UI tests for asserting the user experience.?

Performance tests to make sure your systems are as responsive as you expect them to be.?

Security tests to make sure you did not introduce any vulnerabilities.?

Regulatory tests to use as audit logs in case of an external assessment.?

Smoke test to assert the configuration transformations across deployment environments.?

Run as much as you can locally and the rest into the pipelines ordered in a way to maximize your feedback rate and by extension the rate at which you can comfortably release functionality to users while not exposing security / regulatory / performance risks.

What are some key challenges with creating and managing these pipelines? How can they be tackled?

Things that I’ve had the pleasure of wrestling with over the years include proper management of pipeline variables and secrets, pipeline dependencies and feedback optimization issues.

Managing Variables:

Pipelines use variables to run the same steps across deployment environments and as switches to conditional logic (if triggered by a PR run only the unit tests, if a manual flag is set, skip X steps).?

There are many ways to manage them and each CI solution has both ways to manage variables in the CI server but also integrate with external configuration stores.

The trick here is proper scoping. If we break it down entirely, it works like this: at first hardcode all the variables. Then find the common ones and pull them up to the job scope. Find all the common ones across jobs and pull them up to the stage scope. Do the same for stages and pull them up to the pipeline scope. Of course the more you do this the less you have to go through all the steps. Use CI built in variables as much as you can and stick to one source of variables for simplicity (built in or external).?

In all cases make sure your secrets are not committed into the repository or shown in the CI logs and are also injected properly exactly where they are needed.

Pipeline Dependencies:?

Ideally you have one pipeline per repository that deploys to one service with multiple environments.?

But often you have to add complexity to this:?

• a commit to one repository triggering multiple pipelines or
• a single pipeline triggered by multiple repositories,?
• pipelines triggering other pipelines, pipelines deploying to more than one dependent system or
• separate parts of the same system (separate pipeline for code, DB schema, infrastructure).?

If you find yourself creating these dependencies make sure you are doing it for a good reason as by definition they make the pipelines non-deterministic and the feedback slower.

Constraints of Scale and Tooling:?

Lastly, you will soon meet the constraints of scale and tooling that will force you to make decisions of feedback speed vs confidence level.?

For example, your slowest integration test suite has grown too much and is very slow. You decide to split it into “important tests” that run as part of a blocking pipeline stage on every commit and “the rest” that run in a parallel stage but non blocking. You improved your feedback time but deciding whether to go to production can no longer be done automatically.

Any advice on best practices with creating pipelines?

Lean on Tooling:

My first piece of advice is to lean on the tooling or put differently, do not reinvent the wheel. A lot of battle tested and well documented tooling makes it a breeze to spin up and run pipelines nowadays.

Source Control Setup:

Prefer YAML pipelines over UI-centered ones as you can take advantage of source controlling. This allows you to audit all pipeline changes, keep history and use the same development, collaboration and code review tools that we use for other code.

Optimize for Fast Feedback:

Optimize for feedback time. No stage should run for more than 10 minutes. If it does the developer will increase their WIP and pick up another task, context switch and lose valuable productivity momentum if the pipeline fails.?

Also, make sure the feedback (especially the failed kind) goes back to the correct person automatically and as fast as possible via alerts and notifications.

Enforce Security:

Keep security front of mind.?

Treat your secrets carefully, do not print them out in logs, commit them in repositories or expose them in other ways.?

If you use your own CI agents, make sure they are patched and only accessible from the CI server.?

Make sure the way CI talks to your cloud infrastructure is authenticated and authorized to do only what it needs to do (“least privilege” principle).

Monitoring and Maintenance:?

Get some metrics and dashboards going. Small scale observability is good to have. If you host your own agents make sure CPU/RAM/Disk space and IO are good. Pipeline status and run times are also good to monitor.

Pipelines need refactoring love too. The architecture and code evolves with the business and the same applies to your pipelines. They are not set in stone and should be reviewed and refactored in an iterative manner. Use the CI observability to identify bottlenecks and opportunities to improve.?

Can you relate? Share your voice.

Can you relate to Michael Tomaras? The true value of automated tests are realized when they are effectively used in the right stages of development and integrated appropriately with the delivery pipelines.

What sort of testing pipelines are you currently using in your workplace?

What challenges are you currently facing in reducing manual intervention with delivery pipelines?

Please share your thoughts in the comments.

Or if you would like to contribute to the newsletter with your view on testing (in any aspect), please send me a message.

Thanks for reading.

Please subscribe if you would like to be notified of new publications.