How can you ensure IT service availability during maintenance windows?

Here are key strategies: 1. Pre-Planning: Assess and plan meticulously. Involve stakeholders for impact understanding. #StrategicPlanning 2. Communication: Keep users informed about maintenance schedules and progress. #TransparentCommunication 3. Redundancy: Implement redundant systems for critical workloads. #SystemRedundancy 4. Testing: Conduct pre-deployment tests in a controlled setting on a regular basis. #RigorousTesting 5. Monitoring: Continuously monitor services for issues. #ProactiveMonitoring 6. Rapid Response: Be ready for unforeseen problems. #QuickResponse 7. Feedback and Improvement: Post-maintenance, gather feedback for future enhancements. #ContinuousImprovement #ITLeadership #MaintenanceStrategy #ServiceContinuity

Thinking out of the box utilizing latest tools like CI/CD pipeline for code integration, server management tools for security updates. Advance planning and tech utilization helps a lot in increasing efficiency reducing downtime.

From my perspective, this is not a challenging situation anymore - if you had asked this question even 5-8 years ago, this would have been very important and challenging question. Today there is enough maturity in this area. Products, software, tools are much matured and are highly available -using various methods and techniques. In today's world, question is more on what have you signed-up for with regards to availability - how many 9s. Based on number of 9s you are able to design the solution to meet the availability requirement (SLA). Therefore in my view what is more important is to ensure everyone understands the cost and effort required for each extra 9 !

It is crucial to have a defined objective for the maintenance window. First step is to evaluate and see if it is possible to do the system upgrade/maintenance without a system downtime. If a downtime is necessary, creating a project plan with appropriate roles/responsibilities, list of tasks and correct sequence of tasks is essential to minimize the downtime of critical applications.

A gazillion years before you start scheduling anything, you need to architect, fund, deploy and (successfully) operate your production, development, QA and DR environments. All these environments are established with a clear understanding of service, availability and reliability requirements. Lastly, customize your standard change control process (if needed) to make these maintenance changes work smoothly. Trying to make this stuff up on the fly is for amateurs.

Depending on how your application/environment is architected it may be possible to do a partial upgrade, and process Production transactions through it (e.g. a new version of the code) or build-in a software switch to be able to make a seamless transition to a new version. If you can script the update, you should also (try to) script the reversal of the change. If you cannot script the reversal or it is too complicated, or you are running out of time - or have run out of time - it may be possible to restore service to consumers, while continuing to forward-fix any issues that surface during (or shortly after) deployment. Not only should you consider your maintenance window but you must consider dependent systems maintenance windows too!

Two key items in choosing an optimal maintenance window are 1) have enough time to successfully perform the work, and 2) the window is aligned with others downstream/upstream. First, success in a change window not only has immediate consequences, but it can also impact your ability in how you perform future changes. So, it's important to be realistic about the amount of time needed to do the work, but to also address troubleshooting when things go wrong, and enough time to back out due to a unsuccessful change. Second, depending on the possibility of impact to other services that may be possible, it's imperative to communicate and ensure others downstream/upstream are in approval of your change window activities.

We need to work with the business to understand the full business calendar for the business processes supported by the application, by month, week, day and hour to determine the timeslots when interruptions are allowed and establish an error budget, as well as the times when no disruption is allowed and put a change freeze around those periods. If the allowed time slots aren't enough to do the work, the application has to be split into independent parts that can take the traffic at the slow times even when the other parts are unavailable. Then perform the changes, part by part, with lower capacity, but not completely down.

Choosing an optimal maintenance window is far too often dictated by the IT department without business input. Maintenance on a single application must take business needs into consideration before maintenance can be scheduled. Beyond peak usage times, maintenance windows should take into consideration the timing of regularly scheduled batch jobs, up and downstream data transfers, high transaction periods not related to usage, and perhaps most importantly customer experience. For example, if the system is processing online sales for a high volume online retailer, downtime during the holiday season should be avoided if at all possible for customers will take their business elsewhere due to the poor CX. Coordination with the business is key.

Selecting an optimal maintenance window is crucial for preserving service availability. Consider user activity patterns and business operations to schedule maintenance during low-impact periods, minimizing disruptions. Communicate the chosen window in advance to manage expectations and enable users to plan accordingly. This strategic approach ensures that maintenance activities align with the organization's operational rhythm, promoting a seamless IT service experience.

It is also important to provide information about any downtime in the portal where users would report issues, or the IVR message for the support team. This ensures that unnecessary tickets are not opened during maintenance.

One thing I like to do, in setting expectations, is to communicate early, and communicate often. If you're going to miss your maintenance window milestones, let others know - be open and honest - so no surprises, which engenders trust and earns a good reputation. If you go past your maintenance window, or are in danger of going past (ideally considered before the window starts) it is useful to have a catch-up strategy in place. Can you process the backlog of transactions building up and can you process two (or more) 'batches' consecutively? Don't be tempted to squeeze too much into your maintenance window - instead, be sure that what you promise, you deliver, and when you're done, be done - don't be tempted to deliver something extra!

Most organizations have established Change Management processes and activities, such as daily or weekly "stand up" calls to discuss upcoming changes. Presenting clear and complete information in such calls and properly documenting your proposed changes in your Change Management system is essential. Finally, if there are possibilities that some stakeholders or those upstream or downstream in potential impact from your work aren't going to see the usual Change Management traffic, then specifically targeting them with an e-mail or phone call would be a strong practice.

Communicating as early as possible via emails,ivr messages ,scrolling messages ,chat group messages specfying the maintennace window of unavailability and when the systemor application would be up over an are some of the common methods that I have encountered when performing CRQ.

Effectively communicate your maintenance window to users, providing clear details on when service interruptions may occur. Transparent communication helps users plan their activities accordingly and reduces the impact of maintenance on their workflow. This proactive approach fosters understanding and cooperation, contributing to a smoother IT service experience during the designated maintenance window.

While I agree with all the comments and suggestions on planned maitrnance windows. We also have few technical solutions. We can do Application and Database clustering and load balancing. Where a node can be taken for maintenance while the other nodes are available. We can enable offline mode for certain apps. We can do patch upgrades and other maintenance activities in a planned manner on cluster nodes and sync data after the upgrades. The offline apps like credit card POS machines were giving batch mode data transfers in bidirection with a Caviet, 24 hrs window in which any stolen or fraud transactions are not guaranteed. Today POS machines are online with good internet. We can also restrict few transactions which need online availability.

There will be many alerts generated during the maintenance activity, and some will be cleared when the work is complete. It is essential to investigate the ones that don't clear as those might be symptoms of subsystems that didn't recover from the change activity.

Test and Release is key..... Any maintenance window can have an unexpected impact. Hence it would always be better to have key users in place to do comprehensive test. Auto testing - through test tools is preferred, as it helps to execute fast.

Make sure you keep records of the times estimated v achieved and use the info for future schedules to maintain accuracy. Allow rollback time too.

You should have established testing criteria prior to starting the maintenance window (which would be entry criteria to having the change window approved). You should also have verified that those tests work - you don't want to be left scratching your head as to why a test is failing. Remember, it may not be your IT system that's causing the failure - there may be a dependent system that's failing. If you're operating a key business functionality system, it is wise to inform your Service Desk and provide a standard response to give users if/when they call in, even if they should/would have been aware of your maintenance window.

Always review your work during a change, including lessons learned, surprises, things that worked according to plan, and especially things that didn't. Whether it was a solo activity or work that was performed from a team function, it's a great practice to formally review change results to identify opportunity areas and how to incorporate them in the future. This might be how planning was done, what testing methodologies were incorporated that were either missing steps or lacking enough detail or the right steps. IT environments are increasingly more complex and it's not unusual to run into issues and surprises. But, how an organization can become excellent at change is to review change performance and focus on constant improvement.

Continuously evaluate and improve your maintenance window processes to enhance service availability. Conduct post-mortem analyses after each maintenance window, identifying areas for improvement and learning from any challenges faced. Implement lessons learned to refine procedures, optimize efficiency, and minimize potential disruptions. This iterative approach contributes to the ongoing enhancement of IT service availability during maintenance windows.

Feedback is critical for improvement for the next maintenance cycle. Hence record the same across each steps - what went well and what did not? This would enable to improve upon the next cycle.

Always remember: "feedback is a gift" - if somebody takes the time to provide feedback, accept it graciously. They may suggest something you're already doing or you've already done. If so, you should be looking for improvements to your communication plan and your communications next time. Remember that different channels work better for different stakeholders. You may have a senior stakeholder whom would prefer an SMS - instead of insisting that they process their inbox for receiving maintenance window notification. Senior stakeholders, however, should have some kind of presence and/or representation on the Change Approval Board anyway..

One way to look at evaluating activities taken from Scouting is: Start-what needs to be done that currently isn’t Stop-what is being done that should not (any longer) Continue-what is working well and should continue

For significant changes that are high-risk, extensive or involving many CIs, having a coordination bridge where the different teams performing the change can report progress can help ensure that coordination is maintained and any partial or full roll-back is synchronized.

Schedule maintenance during off-peak hours and communicate clearly with stakeholders. Utilise redundant systems and load balancers for uninterrupted service. Test procedures in advance and prepare rapid rollback plans for emergencies. Monitor systems in real-time and have a support team ready for unforeseen issues. Implement changes incrementally and consider canary releases. Coordinate with vendors, conduct post-maintenance reviews for continuous improvement, and ensure comprehensive documentation. Training and awareness among team members are also crucial for successful maintenance execution.

Different industries have varying definitions of a maintenance window and the expected service level during that time. Ideally: Schedule and communicate a business-minimizing maintenance window. Ensure customer awareness as part of SLAs. Identification: 24x7 apps: Implement Active-Active or Active-Passive setups for failover. For non-redundant setups with only one region, opt for a rolling restart within the same region. Noncritical apps: Align maintenance windows with usage patterns. Ensure: Proper redirection of digital apps. Clear procedures and code for service shutdown/bootup. Documented backout procedures and code for potential reversals. Success hinges on thorough redundancy and failover testing during every major release.

Sometimes you need to design for a maintenance window. For example if u have a transactional website, it would be nice if during the maintenance window, you swung to a read only website that did not take txns. Benefits of such an approach is from the customer perspective, your website is not down or in maintenance. Customers can still browse, but maybe the transactional functionality cannot be done and once maintenance is done on the main site you can swing back from the "read only" site to the main site. Another option I have seen is during maintenance, you can still take transactions/ requests but the requests are queued up for processing/ posting post maintenance window. That again gives the impression of continued availability.

Plan for the unexpected - bring in key executive oversight and on high alert from vendors, specialist, additional workforce, hardware etc that you may need to help you if things go wrong. Create worse case scenarios and plan of actions and executive awareness and alignment so decisions can be made quickly. Have a broader communication plan ready. Time is the luxury we don’t have during maintenance windows.