Go End-to-End From the Beginning

Most of the software teams that I see today in large companies that claim to use Agile and CI/CD methods have wrapped waterfall around DevOps pipelines.

What they do is they have a set of teams—often thirty or more for a product—that individually have “pipelines” that build and test components; but then to perform integration testing for those components, they deploy them to a shared integration test environment and perform manual tests.

That’s not Agile, and that’s not CI/CD, and that’s not DevOps. Far from it. That’s waterfall.

It seems like they have forgotten a basic concept from Agile: work iteratively instead of incrementally.

Agilists often use this diagram to illustrate incremental versus iterative work:

Notice that in the upper portion of the figure, the car is being created piece by piece: each section is complete and fully finished in its final form. The problem with this approach is that the artifact being created is not useful or testable until the whole thing is done, and so it cannot be evaluated—tested in the market. If one were to try to test it in the market, it would not be usable and would be rejected. In fact, it can only be component tested until it is complete: one can only test the wheel by itself, and then a platform with multiple wheels, and then the car without its roof, until finally all the components have been built and assembled and one can actually do some real road testing.

In contrast, an iterative approach builds an entire usable vehicle from the start. It begins as a “minimum marketable feature”, in this case a skateboard, which is actually usable. One can try it out on real sidewalks, and get value from it; but before that, one can test the whole thing in the parking lot to see if it works as a system. At that point one will find out if, say, the wheels are misaligned, or are too close together—causing the entire device to be unstable.

In other words, one can test it end-to-end, from the beginning.

Over time, features can be added, gradually evolving the device into a bicycle, then a motorcycle, and then a car—and who knows what after that! All the while, the device is being used as an entire usable system, and so feedback on its function is being received and fed back into the design concept. The result is a much more effective—and much better tested—device, from the beginning.

In most of the large organizations that I visit, they are doing what is shown in the upper part of the diagram. They have “pipelines” for the components—to use the diagram, that would be the car wheels, the chassis, the body, and the roof. Then they assemble it all and deploy those components to an integration test environment, and spend weeks trying it all out.

Of course problems are found during this integrated testing, and those problems must be fixed. Meanwhile, the business stakeholders are getting worried, because they thought that the code was only being tested—they did not expect delays for fixing problems: “I thought it was done and you were only testing it?” The business concludes that predictions about completion time are unreliable.

This brings back all of the problems with waterfall, because that is what it is—even though the teams have “CI/CD pipelines”. How ironic!

The Pain

One of the symptoms that results from this waterfall approach is that the cycle time to actually complete a feature is stuck at two months. By “complete a feature”, I mean that the feature has been integration tested. Since integration testing must wait for deployment to the shared integration test environment, there is a considerable delay between when a feature has been started and when it can actually be considered to be done.

Another symptom that results is that quality suffers: since integration testing is done all at once, it tends to be less thorough. It also tends to be manual since it doesn’t pay to automate something that is only done a few times.

A third symptom is that if one wants to make a change to the product after it is done, the lead time is very long—months. That is because one has to code the change, then deploy the changes to the shared integration test environment, and finally perform manual regression testing on everything, to make sure that the change does not break anything else. Shortcuts are usually taken on the manual regression testing, resulting in a potentially unreliable product.

A fourth symptom is that deployment tends to be manual and unreliable. Again, this results from the tendency to not automate something that is done infrequently. The result is that deployments are usually done only during off hours, over a weekend, are problem-ridden and grueling. It is not fair to staff, robbing them of their weekend every time a deployment occurs, which—today—tends to be more and more frequent.

End-to-End From the Beginning

A true DevOps and CI/CD approach is to use the iterative approach for the entire product: create something working from the very beginning, and test it all the way through, end-to-end, from the beginning. But what does that look like for an existing product?

Agile development is usually planned as a backlog of stories. When a team adds a story to its backlog, they should not start on the story unless and until they have end-to-end tooling set up, that enables them to build an empty or trivial non-functional implementation of the story. That is usually called a “hello world” piece of code. Not only that, but they should be able to run a trivial unit test, a trivial component level test, and—this is where it gets interesting—deploy the changes to an isolated test environment and execute an automated trivial end-to-end integration test.

Then, and only then, does the team start to code the story.

This is an integration test pipeline. If each component in the overall application has its own component pipeline, these all converge in an integration test pipeline. This convergence occurs because as tests are added to the integration test suite, it becomes necessary to deploy other dependent components to perform those tests. A full integration test suite requires all of the application’s components to be deployed.

The above description is an over-simplification. There are many factors that go into the design of an integration test pipeline. One consideration is how the integration tests can be “shifted left” so that some of them can be run locally on laptops. Achieving that requires designing the deployment scripts so that they are not tied to a specific build tool (such as Jenkins) or environment—that environment-specific aspects have been separated out as parameters.

Another consideration is how tests will be flagged to be omitted, to perform a quick integration check instead of a regression test. A third consideration is how dependent changes will be managed spanning multiple components.

The last problem is the problem of dependency management. I discussed that at length in this article: please read that if you are interested—dependency management is the critical enabler for true CI/CD.

Define Your Entire Process - Not Just the Team Processes

It is often more effective to define end-to-end tests at a higher level of granularity than the story level. For example, one could define “epics” or “features” that decompose into stories, and define end-to-end tests for the epics or features. That is a test design consideration that is part of the design of the Agile and CI/CD workflow of your teams.

Note that I said that the integration tests must be deployed to an isolated environment. The reason for this necessity is not obvious. It is because once teams have the ability to run integration tests on demand, they will, and their next bottleneck will be that they will want to test changes to multiple components, but if they deploy those changes to a shared test environment, they will interfere with changes that other teams are making: the environment will become “unstable”.

To avoid instability, integration tests should be deployed to a clean on-demand environment, integration tests run, and then—and only then—should the code changes be merged into master branches so that the changes can be deployed into a shared and controlled test environment.

Thus, pull requests for master branches should be approved after integration tests pass—not before.

There is also a buzzword going around today called “shift-right testing”. This is not a new idea—only the term has caught on recently. It is the practice of rolling out changes into production gradually, slowly increasing the number of users who are routed to the changes, and carefully watching a fully instrumented application to make sure that it is behaving as expected and does not need to be rolled back.

You should always do that nowadays, but before you do that you should have first fully tested things as well as you can. For a high-assurance system that, say, handles people’s bank accounts or books their airline flights, shift-right testing should not replace integration testing: it should supplement it.

One argument that is often made against automated integration testing is that if component contracts are well specified, then testing the components individually should be sufficient. That is most definitely not the case. Today’s complex systems have moved much of the design into the “outer architecture”, where services interact in real time. The temporal behavior of these real time interactions is not specified by component contracts. Further, even if APIs and message schemas are well specified, the semantics can change. Integration testing is more important today than it ever was.

Go end-to-end from the beginning, and you will have a true Agile CI/CD process. More importantly, your cycle time will decrease dramatically, your application’s reliability will increase, and your time to code and deploy changes will go from months to weeks.