Making On-call fun

In my previous article around the Culture of DevOps, I had made a subtle reference to on-call but if you have been in a DevOps culture long-enough, you’ll know that on call is a keyword that often doesn’t excite your engineers. So how do you make it exciting and drive towards a culture where the on-call routine is fun?  

While we are not perfect in our on-call routine, we are in the continuous journey to strive for operational excellence. We are genuinely trying to make on call fun, exciting, incentivized, disciplined and rewarding.  

• Have structured tiers around your on-call routine. Tiers necessarily doesn’t mean more layers, but established practices (standard operating procedure) on what determines how you will go up the ladder of escalation.  

• Establish a graduation program for your tiered-on call system. 

The graduation concept is a simple way to incentivize Engineers in their on-call journey and let them aspire towards next tier.  

• Everyone on the team is on call, irrespective of your job title/function. For example, Engineering Leaders on my team are on the hook for ALL kinds of escalation anytime, any day. 

A good leader will take ownership and drive outcomes. 

A good leader will head-on deal with the politics during an incident and strive for outcomes.  

• Lead by example, have a strong bias for action. On call is fun, when you demonstrate you care and put yourself in the front-line when incidents happen.

In my current org, I take the pleasure of doing what’s called as the preliminary assessment when a P1 or P2 incident is triggered and collect the facts. This is not an expectation of my role, but I find this to be fun, rewarding because I get to learn more about our application Eco-system. And let my engineers work on real problems and empower me with data, dashboards with which I can complete my preliminary assessment in a nimble way.  

• Run-book is a living document that documents your standard operating procedures, links to dashboards and other relevant system documentation. If they don’t exist, start somewhere or start here. They will never be perfect, but your entire team should know that

If you see something, say something and write it down.

• Have a comprehensive on call calendar which is the single source of truth for schedule for your tiers.

Empower Engineers to collaborate with their peers directly when schedule needs adjust, and managers are kept on an inform basis.   

• At end of an on-call rotation, encourage the Engineer to do a hand-off of the torch to the next one in the line.

A simple practice could be a quick huddle with members of the team review what went well the last week, what was annoying & what to do better.   

• Celebrate individuals that took time to make existing documentation better based on their observations.  
• Figure out an incentive strategy to make on-call interesting. On-call takes a toll on the individual and one way you can motivate the individual is use the lag time he or she has the week they are on-call to work on a pet project or creative idea.  
• Recognize someone that’s gone above and beyond.

These people are your un-sung heroes, so the least you can do is give them a comp time-off where applicable and get a mental break.  

• When engineers present you problems about on-call, ask them how to fix it. Often in this collaborative chat the result is innovation leading to excellence in operations.  
• Celebrate failures. As mentioned in a diff article, failures are success in many ways because of the learning, so encourage your team to fail-fast and fail-forward. You can make it fun by having a face-palm idol that’s rewarded to the “oh sh*t” moments, but when you do this do it with common sense and understand the emotions of the employee under context before executing on it. 

If there are other practical tips, I genuinely welcome them as feedback/comments.