How to Avoid a Disastrous Deployment

I remember being part of an enterprise implementation that had taken about 9 - 12 months of planning and execution and we were finally ready to deploy and nothing could go wrong... or so we thought! This deployment involved remote teams in multiple locations and they required special access to the company's IT assets. On deployment day, we realized that a security upgrade had taken place on the company’s infrastructure previous night, locking all of the remote teams out of resources they needed to do the deployment. Needless to say, we had to postpone deployment by several weeks and appeal to key stakeholders who were eagerly waiting for the new software.  Of course by the next deployment, we had learnt our lesson and ensured that the security team was included in pre-deployment planning.

Sounds familiar? We’ve all been there and this is only one example of how minor glitches could ruin a perfectly planned and highly anticipated deployment. Here are some hard earned lessons from my experience of disasters avoided which I believe can help improve chances of a successful deployment;

1.      Create a plan – let’s call it a playbook

Planning as a key ingredient for success cannot be overemphasized. Having a plan is critical and putting that plan in a form that can be digested and followed by other members of the team, even better. I recently learnt to use properly structured playbooks for all my deployments and it has been a life saver as it is a great tool for gathering input from all key players in a deployment.

 Detailed playbooks should include the objectives of the deployment, pre-requisite tasks, approvals (if required), go/no go decision points, roles and responsibilities as well as the deployment day tasks with start and end times for each activity. Other things to consider when putting together playbooks are the contact lists, escalation lists and post deployment tasks.

 2.      Socialize the plan

Now that we have a plan, we need to ensure every stakeholder gets a chance to go through it and understand their role ahead of deployment. I once had a situation where a team member responsible for turning on critical services was absent during deployment and several attempts to reach this person were fruitless. When we finally got a hold of him, his response was that he wasn’t aware he had any role in the deployment and had turned off his phones to enjoy some alone time. Thankfully, we were able to get the deployment back on track but this set us back by several hours; therefore, making sure everyone knows their role could save you some bucks.

Socializing a plan can be tricky as it requires consensus from multiple stakeholders as well as coordination of schedules to ensure that key players and supporting team members can review the playbook and provide input on their role in the deployment. This also includes senior stakeholders and anyone that may be impacted by the change resulting from the deployment both directly and indirectly

 3.      Make sure there’s a back out plan

One of the key components of a good play book is a backup plan. Let’s say all goes well and key players are present, yet critical tests fail during post deployment checks! You troubleshoot for hours and still can’t figure out the root cause. What do you do next?

This is where having a back out plan plays out as this could be the lifeline for ensuring business isn’t completely grounded if you can’t move forward with deployment. Back out plans can entail simple tasks like unplugging some cables or undoing some line of code, and as complex as a laundry list of tasks that could take more hours to complete than the actual deployment day tasks. Along with the back out plan, we need to ensure that we have identified who is responsible for approving or deciding on a back out trigger. In some cases having pre-approved back out instructions come in handy e.g. if step X fails, then we are pre-approved to back out without needing to contact the sponsor (since most deployments happen over the weekend or after hours).

 4.      Rehearse for D-day

Sometimes referred to as a trial run and just like most performers have dress rehearsals, this can be critical to identifying huge gaps in the deployment plans. Without a trial run, the D-day roll-out essentially becomes a trial run which introduces several risks that could be avoided.

A trial run could be a test upload for deployments with a data migration component, friends and family pilot run or even a joint review of the playbooks where all key players are present to simulate a play by play of each task to be completed on the deployment day.

 5.      Remain flexible and have fun

After all said, we all know the best laid plans don’t always pan out, so my final tip would be to remain flexible as plans tend to change mid-way through most deployments and we have to allow some flexibility to ensure we get to end of job in spite of whatever hoops may appear.

 Also, try to have fun with it! We all know that deployments can be quite stressful, so one way we could take off some pressure is to motivate the team to identify errors quickly by rewarding the team member who calls out the most critical issues. Similarly, you could gamify the fixes by letting the team rank which fixes should go first. You could also reward the team member who fix issues fastest or who make the most fixes. Sometimes the reward is in the recognition of their contribution or showing that you appreciate their work.

These are a few things I have learnt in my journey as a Project Manager. I would like to know if there other tips you might like to share as well.