The Normal Mistake

Like millions of Americans last night I received this message from HBO Max, and immediately I thought to myself "Oh I hope they are OK."

Every sys admin knew exactly what this message was. Some admin, junior, senior, or somewhere in between was doing a routine job and they picked the wrong environment. They meant to run this in some preprod and instead they selected the production environment, and instead of a handful of internal users getting the message it went out to all subscribed users. And that admin went from having a routine day to having the worst day of their year.

And here is the thing - everybody I know who has ever managed a system has made this mistake. We've all brought down the prod environment. We've all sent an email to all users. We've all deleted an unrecoverable database. We work in the only field where "undo" is always an option and found the only time that you can't control+z your way out of it. And it's because we're imperfect - because we are human.

I don't know who is responsible for sending this message. But I hope their boss isn't punishing them for it. I hope (after the mess is cleaned up) they aren't cleaning out their desk, and instead it's just a good natured drubbing on Teams or Slack by all the other admins, sharing their war stories of when they did they same thing. Because this is the most normal of mistakes.