The Power of Vulnerability
What does it mean to be vulnerable when you work in the IT department of a major corporation?
This is where most technologists stop reading. We don’t use words like vulnerable — it is equated with weakness. Hang on with me for a quick story…
It was a typical Thursday morning in the office, and I had just come back to my desk from the weekly meeting. The meeting was about a lengthy project I have been working on for months. Our team had been making good progress, but we still had a long way to go. That being said — it’s hard to stay excited when the goal is far away.
I scanned through my email quickly — no major emergencies there — so I opened a document I had been working on for the next phase of the project. Not 20 seconds after, I slipped my headphones on to really focus on work. Suddenly, my phone nearly vibrated itself off the desk from the flurry of text messages. I scanned through and saw many “system down” errors and “Hey, what’s going on with the network” messages from several of my coworkers across the country in and out of IT.
“So much for getting work done on that document,” I muttered through pursed lips. I logged into our network dashboard, and I saw most of the status gum balls had turned red — links were down everywhere. “What the,” I thought as I mentally scanned through scenarios that could have caused this widespread issue.
I opened an emergency bridge line and got all of the key people on. We started to hypothesize the issue.
“It could be our provider — is the telco having a major issue right now?” someone asked.
“Good question,” I answered the team.
“Someone give their support a call and get a ticket opened right away,” our manager barked. Jeff volunteered to do that and dropped off the call.
“Did anyone make any changes?” the database engineer asked. “I can’t connect to any of my production database instances, but I can get to a few of my development servers.”
Most people on the call said something to the effect of “No changes that I am aware of” and “The change control/ release calendar has nothing on it for this morning.”
So we went through a lot of cycles, going back and forth. Finally, we made a breakthrough. The network traffic was stopping at one of our internal firewalls and connections from all over the company were getting denied when they definitely should not be. One of the network engineers piped up, “Umm, this ruleset was changed this morning — looks like 10:17 local time.”
“Damn it, another unauthorized change biting us in the ass again,” I thought. I immediately suspected who it was — it had happened on several occasions before.
“The change was made with one of the service accounts,” the network engineer said.
I was pretty certain the change was made by John — he lives in Nashville — so I popped open my IM window and messaged him.
“We’re having a major outage this morning — made any changes to the firewall this morning?” I wrote.
He didn’t answer for what seemed like a long time, but in reality, it was just 90 seconds. “Nope,” he said. “Not me.”
“Who else could it be,” I muttered under my breath.
The network engineer that found the issue said, “I think I know who did it. It was Mike.” Mike was usually really careful about these things, I thought. But instead of chastising him right then, the network was still down, so I asked , “Can we get that change reversed right away?”
To my surprise, Mike pipes up, “Yep, I’m doing that now.”
Wait a minute, I thought. Mike was on the call the whole time? Why hadn’t he said anything?
Well, the truth is he was afraid of being vulnerable, and he allowed the fear of being the one blamed for the outage to prevent himself from showing any vulnerability to his coworkers.
At first, he tried to convince himself that his “innocent” change couldn’t have been the culprit, but as more and more evidence pointed toward his action, he had no choice but to fess up.
But that’s not how it should be. Why can’t us “techs” always default to “Hey that might have been me!” whenever there is an outage? Why can’t we be more self-effacing and not scared of making mistakes? We all make mistakes — some worse than others — and we’re not in competition with each other. We’re all on the same team, and these outages make us all collectively look bad to our customers — not just to the one who made the error.
What is it about the culture that doesn’t allow for error and makes us point fingers at one another? Couldn’t we be a much more effective tech team if we all worked together and kept each other out of hot water?
Be courageous! Vulnerability isn’t weakness — it is a strength that we can bring to our teams.
Soft skills, soft skills, soft skills! You can be the most technical genius in the world but if you cannot connect with others, you probably aren’t going to get that far.