Continuous Delivery is Not a Pipeline

“We have pipelines”. That is what I always hear when working with organizations that claim to use DevOps and continuous delivery (CD) methods. The claim resonates because pretty much every article one reads about DevOps mentions the “pipeline”. Graphic depictions of “DevOps” and CD almost always are a pipeline of some kind, showing a flow of software from development through various stages of testing and finally release.

Continuous Delivery is not really about the pipeline, however. In fact, in one instance, I worked with a team that had no pipeline, but nevertheless delivered continuously, and I feel that the absence of a pipeline actually improved the behavior of the team. I would claim, in fact, that the pipeline concept is a red herring, and that continuous delivery is really about (a) testing strategy and (b) branching strategy. (See my article series on testing strategy.)

The Jenkins job is really a re-invention of 1980s batch processing

If you think about it, a pipeline is an awful lot like a waterfall process - just sped up. Worse, the Jenkins job is really a re-invention of 1980s batch processing: you make some code changes, submit your job, and wait in line for it to execute, so that you can obtain your results as a report (the Jenkins console log and the JUnit test report). Is that progress?

It is not. The only real difference is that Jenkins doesn’t take punched cards, and the output reports are accessible via a browser instead of a printout.

Consider what a team would be forced to do if they did not have Jenkins: They might do as follows:

1. Deploy locally, or remotely using a script.
2. Run integration tests locally, or remotely.
3. Merge changes into master - after local integration tests pass.
4. Deploy via script to a Prod segment that receives a small percent of user traffic, and gradually scale up.

Hmmm - no mention of Jenkins anywhere. In fact, the key tools seem to be git, and the ability to run integration tests before merging code changes.

So what is the pipeline? Don’t we need it?

Yes, but not in the way that it is usually portrayed; and organizations that use it the way it is usually portrayed are using it wrong - as I have explained, they have re-created a faster version of batch programming.

The true role of the pipeline, which is manifest by the build orchestration tool (e.g., Jenkins or VSTS) and the tests that it runs, is to run tests that cannot be run locally and also to re-run all tests as a regression. It is a policeman: the tests are supposed to “stay green”. But if the team has the practice of running those same tests locally, or in isolation before they merge their code - which exposes their changes to other team members - then when merge does occur, all tests should pass. Jenkins would be “green”.

Isolation is key

The key element then is running tests before merging. You don’t need a pipeline to do that.

Notice also that in order to run tests before merging, you need a private place to deploy, so that you can run your tests. That can be your laptop, or it can be a private area (e.g., a VM) in a cloud account. You must be able to deploy the system-under-test in a place where it will not replace the components that other team members are testing. In other words, you need to deploy in isolation. Isolation is key for testing. Once isolated integration tests pass, and you merge changes into the shared development branches, then - and only then - are you ready to deploy to a shared test environment. Thus, the real integration testing should happen before code changes reach the pipeline.

Some tests cannot be run locally, but they can still be run in isolation in a cloud account or data center cluster. Tests that often cannot be run locally include behavioral tests in a true production-like environment, network failure mode tests in which network anomalies are created, soak tests that run the application for a long time, performance tests that stress the application, and so on.

The pipeline is a set of automated quality gates. However, if you are doing things right, you should have found most functional defects before code hits the pipeline. You do that by running some integration tests locally, and running quality checks such as security scans locally. This is known as “shift left” testing, and it is how advanced DevOps organizations do things. If you are debugging functional errors in your pipeline, you are doing 1980s era batch programming, and you are doing DevOps wrong.

The pipeline is important - it is an integral part of DevOps. However, it is not the central element. The central element is the practice of testing continually, using automated tests. This enables programmers to have a “red-green” feedback loop in which they find defects as soon as possible - ideally on their own workstation and before they merge their changes into the shared codebase - instead of “downstream” where defects affect everyone else’s changes and diagnosing problems is difficult.

The core to DevOps is the set of practices that make this shift-left approach possible: these include practices for branching and merging, and setting things up so that many kinds of integration testing can be performed locally on programmer’s laptops or in cloud accounts that they have direct access to, such that a programmer can initiate an integration test run that occurs in isolation from all other programmers.

DevOps is a shift-left approach. The pipeline is important, but it is not the central paradigm.