The first time I broke production

"Hey Jenya, do you have feature toggle for your code?" Jom appeared by my desk. He was the QA master that week. Which meant he was responsible for everything that was released to production. Which meant that something that I've merged was doing something... undocumented.

"No. There's no toggle for that code. It's mostly monitoring..." I answered trying to think of all the changes that I've made.

"Then I'll rollback the code right away," Jom said and turned away going back to his desk.

I ran after him all worried. I've joined the company only a couple of months ago and didn't have any issue in production yet. I didn't know what to expect.

At the release station, the release manager did his magic. Jom told me in his kind but precise manner that the search page - our page - was broken and that they narrowed the issue down to my changes.

And that was that. I went back to my desk feeling like the biggest failure in the universe. My dev manager just glanced at me but didn't say anything. I was probably looking pretty miserable for him not to try and figure out the root cause of the bug right away.

Nut came over and went through my changes that broke everything. He found the line that I've removed cause for some reason I thought it was not an important line of code. He explained to me why this line was actually important. We've created a new PR with a fix and we've merged it in.

That's it. No one rubbed it in, no one asked me to do a detailed presentation. We had the tools to rollback or roll forward. And we didn't waste effort on blaming each other. It was pretty clear that it's better to invest in being prepared for the day when shit will happen than to try and prevent shit from every happening. There will be new joiners, there will be complexities, there will be incorrect assumptions, and there will be the dreaded Friday afternoon deploys. Are you sure you have all the preventive measures for now and the future? Or are you just ready to deal with issues no matter where they're coming from?

I got used to breaking production and messing up the master branch. Many times I had to revert code and ask for a rollback (or roll forward). Many times I had to run around the office looking for that one person who could fix the issue, because of the weird permissions and tools mismatch. But I never felt like I was blamed or punished. It was empowering, liberating.

I think we all know that the person who blames you the most is yourself. We don't need help with this. We need those other things: empathy, support, tooling.