Continuous Delivery is Not a Pipeline

Continuous Delivery is Not a Pipeline

“We have pipelines”. That is what I always hear when working with organizations that claim to use DevOps and continuous delivery (CD) methods. The claim resonates because pretty much every article one reads about DevOps mentions the “pipeline”. Graphic depictions of “DevOps” and CD almost always are a pipeline of some kind, showing a flow of software from development through various stages of testing and finally release.

Continuous Delivery is not really about the pipeline, however. In fact, in one instance, I worked with a team that had no pipeline, but nevertheless delivered continuously, and I feel that the absence of a pipeline actually improved the behavior of the team. I would claim, in fact, that the pipeline concept is a red herring, and that continuous delivery is really about (a) testing strategy and (b) branching strategy. (See my article series on testing strategy.)

The Jenkins job is really a re-invention of 1980s batch processing

If you think about it, a pipeline is an awful lot like a waterfall process - just sped up. Worse, the Jenkins job is really a re-invention of 1980s batch processing: you make some code changes, submit your job, and wait in line for it to execute, so that you can obtain your results as a report (the Jenkins console log and the JUnit test report). Is that progress?

It is not. The only real difference is that Jenkins doesn’t take punched cards, and the output reports are accessible via a browser instead of a printout.

Consider what a team would be forced to do if they did not have Jenkins: They might do as follows:

  1. Deploy locally, or remotely using a script.
  2. Run integration tests locally, or remotely.
  3. Merge changes into master - after local integration tests pass.
  4. Deploy via script to a Prod segment that receives a small percent of user traffic, and gradually scale up.

Hmmm - no mention of Jenkins anywhere. In fact, the key tools seem to be git, and the ability to run integration tests before merging code changes.

So what is the pipeline? Don’t we need it?

Yes, but not in the way that it is usually portrayed; and organizations that use it the way it is usually portrayed are using it wrong - as I have explained, they have re-created a faster version of batch programming.

The true role of the pipeline, which is manifest by the build orchestration tool (e.g., Jenkins or VSTS) and the tests that it runs, is to run tests that cannot be run locally and also to re-run all tests as a regression. It is a policeman: the tests are supposed to “stay green”. But if the team has the practice of running those same tests locally, or in isolation before they merge their code - which exposes their changes to other team members - then when merge does occur, all tests should pass. Jenkins would be “green”.

Isolation is key

The key element then is running tests before merging. You don’t need a pipeline to do that.

Notice also that in order to run tests before merging, you need a private place to deploy, so that you can run your tests. That can be your laptop, or it can be a private area (e.g., a VM) in a cloud account. You must be able to deploy the system-under-test in a place where it will not replace the components that other team members are testing. In other words, you need to deploy in isolation. Isolation is key for testing. Once isolated integration tests pass, and you merge changes into the shared development branches, then - and only then - are you ready to deploy to a shared test environment. Thus, the real integration testing should happen before code changes reach the pipeline.

Some tests cannot be run locally, but they can still be run in isolation in a cloud account or data center cluster. Tests that often cannot be run locally include behavioral tests in a true production-like environment, network failure mode tests in which network anomalies are created, soak tests that run the application for a long time, performance tests that stress the application, and so on.

The pipeline is a set of automated quality gates. However, if you are doing things right, you should have found most functional defects before code hits the pipeline. You do that by running some integration tests locally, and running quality checks such as security scans locally. This is known as “shift left” testing, and it is how advanced DevOps organizations do things. If you are debugging functional errors in your pipeline, you are doing 1980s era batch programming, and you are doing DevOps wrong.

The pipeline is important - it is an integral part of DevOps. However, it is not the central element. The central element is the practice of testing continually, using automated tests. This enables programmers to have a “red-green” feedback loop in which they find defects as soon as possible - ideally on their own workstation and before they merge their changes into the shared codebase - instead of “downstream” where defects affect everyone else’s changes and diagnosing problems is difficult.

The core to DevOps is the set of practices that make this shift-left approach possible: these include practices for branching and merging, and setting things up so that many kinds of integration testing can be performed locally on programmer’s laptops or in cloud accounts that they have direct access to, such that a programmer can initiate an integration test run that occurs in isolation from all other programmers.

DevOps is a shift-left approach. The pipeline is important, but it is not the central paradigm.

Theodore von Mechow

Senior Manager at Capital One

5 年

I agree entirely. Good CI/CD implementation relies a lot more on process and mindset rather than a set of physical tools. I've seen the most success with truly small pieces of work that have extensive component (not integration!) tests combined with CDC. Having an elaborate pipeline framework and robust Jenkins infrastructure does not make CI/CD happen.

回复
Rick Vance

Transformational thought leader who successfully helps organizations solve business problems thru lean-agile mindset.

6 年

Cliff Berg is the Ricky Fowler of the IT world....??https://www.youtube.com/watch?v=Q1YqNTWOldY

回复
Frank Oelschlager

Scalable Growth for Technology | Data | SaaS Companies, Sustainable Competitive Advantage, Digital Transformation, CEO | CPO | CTO, Media/Entertainment, HealthTech, FinTech

6 年

Great article Cliff. Your point is dead on, and somewhat related to something that I've talked about for a while, which is, devops is about getting to a repeatable and reliable cycle time. It's not about having a pipeline, as you state, it's about what gets produced and how it gets produced. I've seen many a "pipeline" that produce crap repeatably, but this is not a reliable or particularly valuable pipeline as producing the wrong/broken thing over and over is essentially waste. I've also seen "devops pipelines" that don't focus on what's important- just the automation piece, that result in many an escaped defect, every time. Better to abandon the dogma and focus on best practices, the rest will follow. #shiftleft?

要查看或添加评论,请登录

Cliff Berg的更多文章

  • They Know a Collapse Is Coming

    They Know a Collapse Is Coming

    The CIO of Goldman Sachs has said that in the next year, companies at the forefront will begin to use AI agents as if…

    78 条评论
  • Agile – I Told You So

    Agile – I Told You So

    I think it is no longer controversial that the Agile movement is in decline. When I made a post about it a year ago…

    51 条评论
  • The Most Brilliant Programmer I Have Known Couldn't Work at Google

    The Most Brilliant Programmer I Have Known Couldn't Work at Google

    During the late 1980s I worked at a 30-person startup. The company was founded by the two fellows who had been the lead…

    31 条评论
  • The Most Guaranteed Way to Improve the Bottom Line

    The Most Guaranteed Way to Improve the Bottom Line

    Culture eats strategy for breakfast, but it’s leaders who generate the culture – leaders at all levels, not just at the…

    2 条评论
  • Empowerment, Not Self-Organization

    Empowerment, Not Self-Organization

    Have you heard people say something to the effect of, “Self organization is not really entirely self organization”? I…

    11 条评论
  • Join a Community that is About Learning

    Join a Community that is About Learning

    Things like leadership, product development, technical practices in DevOps, and more. Free workshops for learning.

  • Anyone Can Learn DevOps

    Anyone Can Learn DevOps

    Are you looking for ways to expand your skills, to be more effective in your organization? One way is to learn more…

  • Use a Capability-Focused Approach — Not an Agile Framework

    Use a Capability-Focused Approach — Not an Agile Framework

    Article here: https://www.agile2academy.

    3 条评论
  • Why Team Performance Is the Wrong Thing to Focus On

    Why Team Performance Is the Wrong Thing to Focus On

    Many companies today are obsessed with teams. The “old” approach of static departments and hierarchies is out.

    12 条评论
  • Leadership Is the Key Skill Today

    Leadership Is the Key Skill Today

    Join our free “Intro” to our acclaimed course, Constructive Agility? for Leaders. (Formerly Agile 2 Foundations) It…

    5 条评论

社区洞察

其他会员也浏览了