DevOps & Its Contradictions
CAMS Model, coined by Damon Edwards and John Willis is a good way to understand DevOps. CAMS stands for Culture, Automation, Measurement and Sharing. The 'C' can also be used to refer other characteristics like Collaboration, Continuity (Continuous Integration, Delivery, Deployment, Testing) & Consistency. But some DevOps solutions may evoke other 'C' words like - Counter intuitive, Contradictory and even Controversial. Some of these are explained as below:
Contradiction 1:
Problem : Production deployments are messy. They take longer time to execute with less chances of success.
DevOps Solution : Frequent and more production deployments.
Totally counter intuitive right ?
Justification : Reduce the batch size and do frequent deployments. Make deployments deterministic, repeatable with CI/CD & Infrastructure as Code. Culture of Experimentation and techniques like A/B testing to validate different hypotheses also require frequent deployments. Some of these experiments might fail and handling failures is planned ahead by collecting enough production telemetry (metrics) so that we can see the problem happening and if required rollback using feature flags.Techniques like Blue-Green deployments and Canary deployments are also followed. So even if deployments fail, a quicker MTTR helps. In essence, by doing deployments frequently, drama is taken out of the deployments and they are made routine and a non-event.
Contradiction 2:
Problem : Security and quality issues in applications and environment
DevOps Solution : Reduce the number of controls happening at the end of the project/sprint life-cycle.
Justification : Instead of just applying security and quality controls at the end of the project, make security and quality everyone's job and build it in every stage. Instead of CAB meetings where the variance between the person actually doing the change and the person approving the change is too much, inculcate rituals of peer-review, TDD and pair programming. Thus move quality closer to the source.
Contradiction 3:
Swarm & Solve the Problem - This might seem the opposite of practice of local problems shouldn't affect whole operations. But by following the practice of Swarm & Solve, where everyone with required expertise attend the problem, we gain the following advantages:
- No fading memories or loss of context
- No Increase of Work In Progress
- Gain organization learning. This is the best way to teach other team members
- No increase of technical debt and thus solve issues when they are cheap instead of blowing in production
- Team productivity is more important than individual productivity.
Contradiction 4:
IT and technology streams learning from manufacturing - At first this seems implausible, but once we understand the concepts of a Work Center, WIP (Work In Progress), Lean principles of how batch size is a good predictor of lead-times and quality - we see that running an IT shop is no different than a manufacturing plant. Andon Cord, TPM (Total Preventive Maintenance), TQM & Kaizen etc. are very much valid in IT world. In fact, DevOps is a culmination of adapting best principles of many other management models.
In spite of these DevOps approaches sounding contradictory at first, they work and bring amazing results in your transformation. As John Allspaw and Paul Hammond mentioned in the last slide of their seminal 2009 "10+ Deploys per day" presentation - Doing this transformation is not easy, instead you can continue and carry on shouting and blaming each other.