How do you manage IT infrastructure changes and updates without disrupting business operations?

Before making any big changes, take a good look at how things are working right now. Understand all the different programs, and how they depend on each other, and draw it all out. Break it Down: Don't tackle everything at once. Break the whole plan into smaller parts, like mini-projects. Talk to the Owners: Tell the people who own the programs (we call them stakeholders) about your plans. Ask for their thoughts and ideas. Backup Plan: Always be ready for things not going as planned. Have a backup plan that you're confident you can use if something unexpected happens during the changes. Real-Time Fixes: If something isn't working right during the changes, stakeholders can give input on the spot.

Truly understand your current environment, the applications, their dependencies, map it all out and peer review it before putting a plan together. Then break the whole programme down into manageable projects in a logical sequence. Ensure you engage and communicate your changes to your stakeholders (app owners), ask them for their input and this should inform decisions on how you implement the change. Have a robust rollback plan and be confident you can implement it in case of unexpected behaviour that might put you out of agreed outage windows. Stakeholder engagement can inform you when to roll back or what can be 'Fix forward' in real-time whilst the change is being implemented.

I’m my experience the presence of test environments allows you to test your changes prior to rolling out to a production environment. The absence of test systems increases the risk of production system downtime, availability and disruption where the change is not tested or not successful. The introduction of test environments should be added to the organisations architecture principles where the decision is to not have these environments and the risk of disruption, availability and downtime should be accepted by the stakeholders and added to the risk register. Great governance in the change management process would assist further as it will require proof of successful testing.

Understanding the blast radius of any change is key, and even more so for infrastructure supporting shared services. And communicate widely for awareness. - if possible plan for a phased/ rolling deployment monitoring and testing for any issues - have a tested rollback of plan to fix forward as necessary

I would follow the below outlined steps: 1. Prepare a ticket with detailed roadmap, rollback, risk assessment, justification, schedule of the CHG ticket window. This should also include which CI's are concerned and their inter-dependencies. 2. Send out a communication to stakeholders in advance for their approval and then present in the CAB or e-CAB. 3. Follow the detailed roadmap and ensure that all steps are followed. 4. Ensure to send communication before starting, during and after execution. 5. Take evidence during execution to review and update existing documentation. 6. Close the CHG ticket with appropriate remarks. Additionally, I would prefer to do a test on test/dev environment first and then roll out to PROD in a phased manner.

Communication is definitely important, but so is the right sort of communication. Being considerate to end users and that it is ultimately their for benefit can get them on your side. Treating them like an obstacle that is in the way of you doing your job will not. Do it right and they can be more supportive if it goes wrong.

The first step should be establishing communication, understanding what is the best moment for doing. And with that information, think if you have enough conditions to do it, in mind the total time, example, time for backing up and roll back, the last one, in a bad situation. And it must be executed after a test, trying getting good results.

Precisely, a mature change management framework that is agile enough to allow for quick iterations driven by business demands, but solid enough to bring stability and confidence.

Integrating automation and using advanced tools can significantly enhance the management of IT infrastructure modifications. Automation streamlines repetitive tasks, such as updates and backups, minimizing manual input and potential errors. Tools facilitate real-time monitoring, detailed tracking, and comprehensive reporting of changes. For example, employing configuration management solutions helps maintain consistency across infrastructure components, while change management platforms offer visibility into the status and history of updates, improving overall efficiency and accuracy.

Create a comprehensive plan: Outline a detailed plan for implementing IT infrastructure changes and updates, including a timeline, potential impact on business operations, and strategies for minimizing disruption. Test changes in a controlled environment: Before rolling out changes to the production environment, conduct thorough testing in a controlled environment to identify and address any potential issues and minimize the risk of disruptions. Implement during off hours: Schedule infrastructure changes and updates during off hours, such as nights or weekends, to minimize the impact. Rollback procedures: Have a well-defined plan in place to rollback changes and updates in case of unexpected disruptions to business operations.

In my experience, key element is the concept of high availability, which has evolved over time with tools, solutions, and frameworks that ensures infra downtime without service disruption. Virtualization, workload balance, containers, Blue/Green deployment/update strategy, infrastructure as code, are just the tip of the iceberg. Understanding the service components and creating an adequate maintenance plan will ensure a smooth operation.

Standard framework like ITIL can be very long winded, many hoops to jump through for approval. I'm some cases this may be desired, but in others batching things up with lots of changes at once can lead to more problems. If something breaks, what change caused it. That and having to wait a long time to get beneficial changes into production. It is worth considering some of the Configuration-as-Code practices of continuous delivery as used by many cloud services. When done well they can offer a lot.