Death by a Thousand Scripts

In my last five+ years of exposure to OpenSource Software, starting with OpenStack, I have noticed an increasing trend of relying on scripts to fill the gaps left by the software system. The gaps can be varied in their scope and size, such as

• Installation
• LifeCycle Management - Upgrades, Downgrades etc
• Supporting different environments
• Troubleshooting
• Feature extensions

While this approach may help with the initial PoC steps to get things off the ground, it can cause a lot of problems later on during the productization cycle. Ultimately, this leads to a software system that is heavily propped up by a multitude of scripts. 

Once the system goes into production, there are many scenarios where the scripts make the situation more challenging. For example, 

• In case of a failure, you are forced to debug your scripts before reporting the problem to the OpenSource community
• In case of supporting a new environment, you have to rewrite all the scripts or create another set of scripts for the new environment
• In case of feature extensions, all the scripts have to be updated for every OpenSource release
• In case of troubleshooting hooks, all the scripts have to be updated for every OpenSource release
• During the upgrade cycle of the OpenSource code base, the LifeCycle Management scripts are likely to require an upgrade
• It is hard to build an end-to-end test system to validate your scripts
• As time progresses, the code base of the scripts gets more and more complicated and hard to maintain

Above all, this approach leaves you entangled in a web of customized scripts that results in a "snowflake" software system. As a result, you are largely left to solve your own problems without much support from the community or the relevant vendors.

However, this unmaintainable pile of scripts does not have to be the inevitable status quo. By carefully choosing projects (open source) or products (proprietary) it is possible to reduce the number of scripts to a manageable amount or eliminate them completely. Based on my experience, they should be able to meet the following requirements:

• Easy to install
• Support Life Cycle Management
• Support a variety of environments, as well as addition of new ones
• Easy to troubleshoot
• Support feature extensions/additions
• Production Ready which includes quality, scalability, etc

In the case of an OpenSource project, the architecture should be flexible and modular enough so that you can fill the gaps by contributing to the project. While in the case of a proprietary product, the architecture should allow the vendor to be able to fill the gaps in short order. 

At Loodse, we develop both open source and proprietary products that address this issue and help businesses avoid creating snowflake systems:

• KubeOne, our lifecycle management tool for single HA Kubernetes cluster on any cloud provider, is a great example. KubeOne makes it easy to deploy and manage Kubernetes cluster on any environment and comes with native support for major providers including AWS, Azure, DigitalOcean, GCP, OpenStack and VMware vSphere. At the same time, it gives you room to support any other new environment by contributing to the open source project. For more info - https://www.loodse.com/blog/2019-05-13-kubeone/
• Kubermatic, Loodse’s multi-cluster on multi-cloud Kubernetes platform is a great example as well. This product was developed after a lot of experimentation with scripts which were tailormade for specific environments which turned into a maintenance nightmare. With the new approach (sans scripts), it takes a few days to add support for a new cloud provider. Today, Kubermatic stands out for its reliability, flexibility and general applicability as the Kubernetes platform of choice. For more info - https://www.loodse.com/products/kubermatic/

In general, scripts are meant for PoC and prototyping phase, not for productization. At the time of production, you should be able to raise both your arms up and say "LOOK MA, NO SCRIPTS".