sounding the alarm

A few years ago, I was a long way down the path towards losing my biggest customer. ?

The problems with our service ran deep, but the bigger problem, my problem, I had no idea. ?

I was blissfully ignorant.?

My team was frantically working to resolve every issue on their own. And I was letting them. I called it trust but it was closer to neglect.?

If you don’t know the difference, leave your kids home alone for a week and tell CPS it’s trust. They’ll be happy to clarify.?

And one day all the clues and whispers clicked into place when we received a surprise RFP. They were going to bid. They were looking for options. You don’t go to bid if you love your current provider. You don’t look for options when things are running smoothly.?

You go to bid and look for options when your current partner is stinking up the joint and you need to make a change.?

Lucky for us, we were not so awful as to be excluded from the RFP, but we were in a bad place.?

Nobody had sounded the alarm. And the truth was, no alarm was available to them.?

Slightly longer web 2.0 version (but with GIFs) @ sounding the alarm – Win With Flynn

Lucy & Ethel Wrap Chocolates:?

Maybe you’ve seen the video .

Lucy is frantically trying to keep up, stuffing chocolates in her mouth, then her shirt, and finally her hat.?

She needs to keep up or she’ll get fired. So, she fakes it.?

And she is rewarded with more work. Her boss calls out, “speeeed it up a little.”?

When things get moving, we are tempted to take shortcuts. Just like Lucy trying to keep up. Chocolates end up on the floor, down your shirt, or in your hat, but the work keeps coming.?

Charlie Munger would say it’s like rewarding the winner of the pie-eating contest with more pie.?

If we can’t bring?awareness to issues, eventually the train jumps the track.?

Navy SEALs use?briefings and after-action reviews. ?

Team members are asked to get involved, ask questions, and make sure everyone is clear on how to meet their objectives.?

These meetings allow them to assess what they can do better, what they can do differently, what they learned, and what they did well. It provides a forum for improvement, learning, communication, praise, and it involves everyone. It breaks down silos and hierarchy.?

In The Culture Code, Daniel Coyle takes a hard look at how Navy SEALs promote a culture of open communication.?

“During missions, [Commander] Cooper sought opportunities to spotlight the need for his men to speak up, especially with newer team members. “For example, when you’re in an urban environment, windows are bad,” he tells me. “You stand in front of one, and you can get shot by a sniper and never know where it came from. If you’re a new guy and you see me standing in front of a window in Fallujah, what are you going to say? Are you going to tell me to move my ass, or are you going to stand there quietly and let me get shot? When I ask new guys that question, they say, ‘I’ll tell you to move.’ So I tell them, ‘Well, that’s exactly how you should conduct yourself all the time around here, with every single decision.’”?

That is a powerful message for a young SEAL. You are here to contribute, and we need your contribution to be successful.?

He sets the expectation that they will share what they see, including simple reminders not to get shot in front of a window.?

If he could install a big red cord that the team could pull on when they notice an issue, I’m sure he would, but that is a tall order when you’re in the middle of a shootout in Fallujah.?

But what if you do something a bit more controlled and slightly less deadly? ?

If you trust your people, and you trust your training, use them.?

That is exactly what Toyota did when they installed Andon Cords in their factories. ?

A physical cord that ran along the production line.?

It reminds me of the Tim Ferriss quote,?“What would this look like if it were easy?”?

Installing a big red cord through the factory and telling people to pull it when things go wrong is so easy it is laughable. ?

Any Toyota worker could pull the cord and it would stop production.?

What’s important to understand about the Andon Cord is not the physical cord and the manual inspection with a supervisor. That is giving the employee a voice, but a voice is just one-half of?sounding the alarm.?

The alarm only works if someone is there to act.?

And for many companies, that’s the bigger hill to climb. Do you think Lucy would have pulled the cord with the boss she had? Not a chance.?

But you better believe Cooper’s SEALs would.?

Toyota knew it was a two-way street and focused on how that feedback was going to be received from leadership.?When supervisors arrived at the location of the issue, they would thank the employee for using the Andon Cord and bringing the issue to their attention.??

But awareness is not all you’ll need.?

Practice doesn't make perfect, practice makes permanent.?

So, practice well.?

Memos and playbooks. Half-day training sessions. Annual reviews. They don't quite cut it.?

What the team needs is practice. Practice like it's the real thing.?

When we practice this way, and we repeat it enough, good habits will be built. Practice will make it permanent.?

In school, we do fire drills. At work, we run 'fire drills' as well. We'll call in with issues, stage events, and create fictional scenarios that test the team in a variety of ways.?

Security breach: What security organizations need to know??

Systems failure: What is the business continuity plan??

Netflix went as far as putting these 'fire drills' into their code base. Sinister and awesome and I love it. They called it the Chaos Monkey.?

Chaos Monkey randomly terminates virtual machine instances and containers that run inside of your production environment. Exposing engineers to failures more frequently incentivizes them to build resilient services. It also provides practice when they don’t. ?

Everyone at Netflix was aware that the Chaos Monkey could strike at any time. They knew their systems and code would be tested. And they act accordingly.?

The thinking goes like this; the best way to avoid failure is to fail constantly, on purpose, so that you can improve constantly, on purpose.?

Fire drills work, they help, but everyone knows they’re coming. ?

I love the Chaos Monkey because of the randomness. Even the boss doesn't know. ?

What happened with the big customer??

I know this story would be better if the whole thing blew up and I had my Jerry McGuire moment, but this one does have a happy ending.?

The nice thing about being the incumbent is that you know where all the bodies are buried. If you can get the train back on the tracks, you can accelerate more quickly than the new kid.?

And that is exactly what we did.?

It paid off and we still support the work.

We put a lot of effort into improving our service. Better training, improved processes, available tools, and leveled up team culture. Once we installed these basics, we tested?with a little chaos.?

Over time, things improved. And over time we didn't need to reach for the cord as often.?

Original post: sounding the alarm – Win With Flynn