A technical author's advice on avoiding a Crowdstrike: always test patches before installing them on a live system

How many technical authors have either written or contributed to the procedures for installing patches or fixes for a server? If you have, you'll know that it's crucial to never install an untested patch on a live environment.

As we have witnessed, the aftermath of a faulty patch installation leads to consequences, but who will be held responsible? Cutting corners and making rash decisions can have profound implications. The person who installed the patch may have been under pressure to do so. Can you imagine the moment when the reality of the situation sank in? Although I am speculating on the events leading up to the installation, you must follow a process before installing a patch. But first, who are the leading players?

Potential Decision Makers:

1. IT Manager: This person oversees the implementation of the patch. They would ensure the patch fits the IT strategy and approve its deployment.
2. Project Manager: Coordinates the project timeline and might expedite the process to meet deadlines.
3. Engineer: The person implementing the fix. They would follow the protocols and guidelines provided by their supervisors.
4. Junior Colleague: Less likely to make the final decision but might be involved in the initial stages of deployment or testing.
5. Change manager: implements strategies for effecting change, controlling change, and helping people adapt. Effective change management minimises - risk, ensuring a smooth transition in technology deployments.

The procedure:

Rigorous Testing:

1. Isolated Test Servers: Test the patch on isolated servers to ensure it does not disrupt the live environment.
2. Simulated Environments: Use simulated environments that mirror the live environment to catch potential issues early.

Review and Approval Process:

1. Multiple Layers of Review: Ensure that the patch goes through various layers of review, including peer reviews and approval from higher management.
2. Compliance and Best Practices: Adhere to compliance requirements and industry best practices for patch management.

Change Management:

1. Documented Procedures: Follow documented change management procedures, which include detailed steps for implementation, rollback plans, and communication strategies.
2. Stakeholder Communication: Notify all stakeholders about the upcoming changes, potential risks, and mitigation strategies.

Backup and Recovery Plans:

1. Back up: to enable system restoration if something goes wrong.
2. Rollback Plan: Have a plan to revert the changes if the patch causes issues.

Monitoring and Post-Deployment Checks:

1. Continuous Monitoring: Monitor the systems closely after the patch is applied to catch any issues early.
2. Post-Deployment Review: Conduct a post-deployment review to assess the patch's impact and learn from any issues.