Continuously Changing Systems

Here's a great post from Jeff Sussna about building systems for continuous change. This is something that I've worked on for a long time in my career. There are always two schools of thought in system administration: some people prefer to keep systems stable and rarely touch them. On the other end of the spectrum, you can upgrade every time there is a minor release or patch. I'm a big believer in the latter.

If you're in the habit of making changes frequently, each change tends to be small and any fallout is easy to diagnose and deal with. If you leave systems to stagnate then eventually you're forced to do a big bang upgrade, often by external circumstances, for example if there is a security patch or you hit a bug that is only fixed in a newer release. (There's nothing worse than having a "stable" system go down due to a bug that was fixed in a later release that you hadn't upgraded to yet!) Then the scope of the change is huge, sometimes you're changing major versions of software and when something goes wrong there are a lot of variables in play.

It's a struggle sometimes to convince people that changing a working system with no apparent problems is worth the perceived risk. Really all you're doing by not changing things is banking all of that risk for a big cash out later on. It's easy to forget about all the times that older stable systems have problems, because fixing those is reactive. Making proactive changes seems risky because you're trying to stay ahead of problems, but there's nothing external driving these changes. Making upgrades a part of your workflow means that you're exercising those muscles and building those skills constantly. And management has to make it a safe environment for people to be aggressive about making changes without having negative repercussions when things go wrong - which of course they will. The idea is that they go wrong on a scope and schedule of your choosing when you're making constant changes.