Continuous Control with Continuous Delivery
Anyone working within a software development organization that works with change control board (CCB) has been frustrated by the bureaucracy. Perhaps they think innovation grinds to a halt because of the CCB, or hours are burned with bureaucracy. They might long to work free of a CCB. Whatever varied opinions people have of CCBs, they are rarely credited as instruments of innovation.
Thus, the question, “With Continuous Integration/Continuous Delivery (CI/CD), can you get rid of the change control board?” This was the question raised by Sherry Chang (@sherrychangca) and Edward Harris (@EdwardHarrisJr) in their talk, CI/CD with Tightly Controlled Governance, at the All Day DevOps conference.
Sherry and Edward both have decades of experience in IT and work at Intel. Given the size of Intel, they have a CCB, yet they were successful in integrating CD at Intel. This is the story of how they did achieved agility and speed with compliance, governance, and decision gates.
Now some of you may be reading this and thinking, “everyone is 100% on board with CI/CD at our organization.” (You can stop reading and begin preparing your 2018 All Day DevOps conference presentation -- because you may live in rare air that others could learn from). For the rest of you living with entrenched processes and people, Sherry and Edward’s story should help you break the log jam and reorient the river.
Sherry and Edward begin by restating some inherent truths about the incompatibility of compliance within continuous delivery. In their traditional form, these are two disciplines at odds with each other:
First, compliance demands:
- Documentation requirements
- Manual review of changes and signed offs
- Feedback cycles that depend on meeting cadence
CD demands:
- Software deployable at all times
- Push-button deployment anytime
- Fast, automated feedback on production readiness of systems
As Sherry noted, on small projects, no one really cared. But when it came to mission-critical projects, you run into walls.
How do you breakthrough the wall? Speak a common language and paint a clear picture of what is good and what is bad. If you can add in change tracking, audit capabilities, and testing, it will create an opportunity to automate. It is just centralizing the data they are already gathering.
You also need to look at why we need approvals for change, and how automation can improve compliance. The below chart shows the reasons for change approvals and the benefits of automation for each one. The bottom line is that automation improves compliance, and information security review backlogs can be reduced or eliminated by automating some of the reviews.
Now, teams at Intel can automate goals and just review exceptions rather than the standard. That changes compliance from a necessary gate that delays delivery to more of an enabler to meet quality and security goals.
How can you get started?
Option A - Hybrid of Manual and Automated Releases. First, map the processes and then work out a roadmap to automate some of the tasks. Everyone needs to agree where to invest by figuring out where the greatest pain point (bottleneck) is. This process gives you invaluable visibility into the process.
Pros: Clear ownership of each step; evolve towards full automation.
Cons: One size fits all, agnostic to team maturity.
Option B - Criteria based CCB for automated releases. The CCB signs off on the CD pipeline as a standard change. When teams meet CD pipeline criteria, there is no additional CCB review. When they don’t, the change follows the existing change process. The policies are continually monitored, re-evaluated, and adjusted as needed.
Pros: Raises the bar for automation and is a model of good to other teams.
Cons: Some teams maybe left behind.
Let’s get back to our original question: Do we still need the CCD with CD?
Sherry and Edward would argue that you still need a CCB, but that its role should change. Rather than reviewing each and every change, it should set the governance policy. After all, for now, you still need humans to set the policy. But maybe not by the time the 2038 All Day DevOps conference rolls around. :)
Sherry and Edward did a great job as a team on this presentation, bringing in different perspectives. You can watch their entire talk here.
If you missed any of the other 100 practitioner-led talks from All Day DevOps, you can binge watch them here.