Automate? Always!
Some years ago I gave a talk, titled "Automate? Always!", to stimulate a discussion about automation in software engineering. Today, I am putting the argument in writing to reach a bigger audience.
Note: in this post I refer to any automation in software development lifecycle: spanning from editor macros to deployment scripts (I confess my preference for the latter as you will perceive).
Please bear with me because this will be a bit longer than other posts, although there is more to say that can fit a single article.
Common thinking: automation is cost-saving
Let's start with this quote ?our goal should be to automate all the activities that are repeated while creating and maintaining the system? (Priti Biyani, ThoughtWorks, emphasis mine). ThoughtWorks suggests that repetion triggers automation, and, I note, the implicit background is of a greenfield project and its evolution.
In a less solemn, more graphical, way xkcd #1205 conveys the idea of time-saving.
If you do not spare time, Randall suggests, it is senseless implement some automation.
You can find more examples in literature, blogs, and journals. The assumption is the same: automation is justified by time-saving, that is cost-saving.
Although this is plain common sense, I felt that reality is more nuanced and found a good list of reasons by Mattias Geniar. He enumerates six justifications:
Here we start to see two new perspectives: psychology and operation. We can easily add that automation in software helps:
The rest of this article tries investigating these additional reasons for automation so you can better judge some less direct and intangible elements that may suggest that investing in automation is the right thing to do in a specific context.
Automation creates opportunities, cost-saving is a side-effect
I articulate seven dimension that adds value to the automation effort: exploring, process improvement, the ripple effect, human satisfaction, learning, inspection & validation, and tracking.
Exploring
You face a new application, a new system: people who worked on it has left, documentation is poor. What can you do?
Add automating is a way to understand an existing system. You may see the system as black box, and try to automate configuration changes, or write some deployment scripts. Maybe you add a CI pipeline replacing ad-hoc build scripts that assume a specific developer configuration.
All these activities require some level of reverse engineering and experimenting with the system in a semi-scientific way: you experiment and validate your hypothesis how the system interact with the environment. Configuration, dependencies, relations will emerge as the result of your trial-and-error attempts to automate it.
Improvement
Any automation removes a manual process. Isn't self-evident? Not always. Removing human intervention is not simple and obvious: you may face resistance, power play, fear. Your automation must be transparent, and valuable.
Transparency has two sides: one is the visibility of automation inner workings, the other is tracking automation steps for auditing and debugging.
Value can have multiple facets:
In general, we should consider both direct positive effects and negative, preventive effects, gains as well as reduction of costs.
Documenting
All of us have met a system poorly documented. Trying to automate is a way to describe its behavior.
This approach is similar to the effect that many bloggers describe: to learn something a great technique is forcing ourselves to write about it. It is similar to implementing automated tests for an application: each test defines exactly a little bit of system's behaviour.
The value of automation is in what we may call "actionable documentation": not a passive document, that requires interpretation, but something you can use to act on the system.
Automating to document implies that the automation code is published in an easy-to-access location, where anybody interested in the target system will look for information.
领英推荐
Ripple Effect
Consider now these unintended positive effects:
Most of the items listed are psychological and reflects onto the organisation performance.
Satisfaction
Subtler psychological effects, less quantifiable though real, are:
Learning
I mentioned before the idea of automate to learn, let's talk now what one can learn.
The most obvious things to pick up are new technologies and learning scripting languages. Less prominent are infrastructure concepts, experience with real practical constraints. What a magnificent way to appreciate network latency is copying files across the pond! Master how parallelism really works trying to minimize deployment times across a bunch of machines.
Inspect & Validate
Legacy, and a thrill goes down the spine of any engineer. Legacy system, legacy code, how can we trust it, how can we touch and change it? Well, automation is a way to inspect and review an existing component. Writing a build script, a deploy script, a script to change configuration, forces the author to review existing manual configurations, question their settings, and add automated validation checks along the process.
I have often encountered configuration files that are just a chaos of unrelated properties, with no organisation nor standard names. Like coral or stalactites, it grew unnoticed in the darkness. Time to question, refactor, and clean it!
Track
One last element to address is the gain in visibility. Automation makes easy to identify usage, dependencies, and configuration. Instead of searching for a Word document which states that you need to install version X of a component, this dependency is stated in plain view in an automation script.
Automation allows collecting information and store it in a central place. An example is building an organisation-wide dependency database, crucial to manage security issues (Software Bill of Materials, SBOM).
Conclusions
“Automate ALL the things” (Niall Murphy et al., Site Reliability Engineering) is a fundamental idea of modern software engineering. It's about delegating more to machines and moving to a declarative style.
Consider what to automate first, plan for adding incrementally more automation to your daily work, prioritise considering all effects, both direct and indirect.
Don't fall into the trap of automating the wrong thing, at the wrong time: xkcd #1319
Capture the state before and after, better with at least one hard, objective metric. Communicate what you achieved, showing the benefits. Tell broadly displaying the measures: it will inspire others.
Finally, do not forget that DevOps is not about automation. Automation supports the DevOps perspective.
References
What do you think? Am I wrong?
Senior DevOps Engineer with 18+ years of experience
2 年I don't think the cost savings and productivity gains can be overstated. Two centuries ago 90% of the world lived in multi-generational poverty. Today it world poverty hovers at around 10%. Most of this prosperity is due to automation. What the XKCD comic and other illustration missed is that the cost savings of automation isn't measured in minutes and hours. It's measured in decades and centuries and on scales far larger than an individual, team or even company. They have a ripple effect that benefits the economy for current and all future generations that far outweighs the time and effort it took to implement them.