How do you plan and execute the testing and validation of a change before and after implementation?

In most of the organizations, testing of the change prior to implementation is given a lot of importance but validating a change post production is often ignored. This is probably the most important aspect as it ensures realizing business benefits, ROI and overall UX. So, project managers should factor in this key aspect of change validation post production in their project planning with PIV plan and have an owner assigned for accountability prior to Go Live.

When designing a use case, consider how to validate and test it for both short and long-term durations. Validation and testing ensure long-term stability and indirectly support the change management process.

Establish the boundaries and standards for testing and validation efforts, taking into account the specifics of the change request, the underlying business case, and the anticipations of stakeholders. It's essential to delineate the goals, extent, duties, assignments, assets, instruments, approaches, and benchmarks for the testing and validation phase, and record these details in a test plan. Furthermore, you must specify the acceptance criteria, which are the quantifiable and confirmable requirements that the change needs to fulfill for approval.

The scope and acceptance criteria must be identified and documented before testing begins. The testing metrics should be SMART and traceable. Sometimes, things change, but we should always update our testing scope, plan, and approval matrix. I like the old "V" model for testing. It helps guide who should do the testing, when, and how.

Establish the Boundaries and Standards Initially, one must establish the boundaries and standards for the testing and validation tasks. These should be determined with the change request, the underlying business case, and what the stakeholders anticipate in mind. It's essential to pinpoint and document the goals, boundaries, roles, duties, resources, equipment, techniques, and regulations governing the testing and validation procedures. This documentation should take the form of a test plan. Moreover, it's important to specify the acceptance criteria – these are the quantifiable and confirmable terms that the proposed change must satisfy for approval.

Create and organize the tests needed to confirm and check the proposed change. Take into account the nature, depth, and intricacy of the change, and choose the suitable testing methods, which could include unit testing, integration testing, system testing, user acceptance testing, regression testing, performance testing, security testing, among others. Additionally, develop test cases, scenarios, scripts, data, and environments that will mimic both the anticipated and unforeseen circumstances and results stemming from the change.

In my experience i believe its very vital to also include the evaluation portion at this level of Design and preparing the testing of..... it definitely in my experience leave less room for failure as we then execute and monitor.

I would also add to the design/prep phase the testing and documentation around backout, failover and recovery design. One doesn't want to get into a change window only to discover that the prod database or a key integration is corrupted with no backout and restore plan or it will take hours/days to recover. Documenting and workshopping the detailed steps to occur in the change window always uncovers some gaps.

If it is a complex change with a few parts/phases, it is also good to consider testing in between phases and having go/no-go criteria to proceed to the next part.

Develop and Set Up the Tests Following that, the focus shifts to developing and setting up the tests to ensure the change is thoroughly checked and confirmed. Take into account the nature, scale, and intricacy of the alteration, and choose suitable test strategies, including unit, integration, system, user acceptance, regression, performance, security testing, and others. Prepare the necessary test cases, scenarios, scripts, data, and environments to mimic both anticipated and unanticipated conditions and results that may arise from the change.

Carry out and Keep Track of the Tests The subsequent action involves carrying out the tests as per the test strategy and documenting the findings and notes. Adhere to designated test processes, checklists, and timelines, and employ the tools and settings prepped earlier. Stay vigilant of the test advancement, efficiency, quality, and potential risks, and be sure to communicate any problems, flaws, or discrepancies discovered during testing. Additionally, compile test proofs, including screen captures, logs, and reports.

It is important to ensure there is separation of duties between the test team and the team executing the change to provide an independent check on the progress. There should also be a check on the elapsed time as there has to be enough time to backout if things go wrong.

Upon designing and preparing the tests, the next critical phase involves executing and monitoring these tests in alignment with the established test plan. Following the defined test procedures, checklists, and schedules is essential during this stage. Using test tools and environments prepared in earlier steps ensures consistency. Simultaneously, monitoring the test progress, performance, quality, and potential risks is imperative. Any identified issues, defects, or deviations should be promptly reported. Additionally, documentation of test evidence, including screenshots, logs, reports, etc., serves as a crucial record of the testing process and outcomes. This step ensures a systematic and well-documented testing and validation phase.

It is always helpful to identify key metrics when analyzing the reports, a before or after comparison can be done to keep track of the progress made and the possible future implications of that certain result.

Following the execution of tests, the fourth step involves evaluating and reporting results. Analysis of test data, metrics, and feedback is conducted to identify strengths, weaknesses, opportunities, and threats related to the change. This information is then compared with pre-defined acceptance criteria. The overall test status, outcome, and conclusion are determined and reported to stakeholders, including the change owner and advisory board. Recommendations for improvement, remediation, or rejection of the change are also provided. This step contributes to informed decision-making and ongoing process enhancement.

Verifying a change in IT involves monitoring for stability, gathering user feedback, comparing performance data, and updating documentation. This ensures the change meets objectives and supports ongoing improvement.

In the fifth step, the focus shifts to implementing and verifying the change in the production environment as outlined in the change implementation plan. Collaboration with the change implementer, service desk, and users is vital to ensure a correct, safe, and efficient deployment. Verification involves confirming that the change functions as anticipated, without causing disruptions or incidents, and delivers the intended benefits. Concurrently, ensuring the update of change documentation, configuration items, and service records is essential to maintain accurate and up-to-date records. This step finalizes the change implementation process with a thorough verification of its success and impact.

The best way to have a consistent method of verifying changes that affect your critical online services is to have a means of executing and verifying the primary customer journeys. Robotic Process Automation can assist greatly, eg an online banking service the automation emulates a customer to get a balance, pay a bill, locate a statement. A range of health check approaches can be developed. In one organisation they developed what they called ‘Ready to Trade’ with a range of automated and manual checks to verify services were available from a customer journey perspective after changes and before start of day. Where functionality change is involved ultimately business on-site/remote validation is required.

All issues faced during the change, even if they were resolved during the change, should be investigated using problem tickets. They may recur in future changes unless their root cause is fixed. In case incidents are reported after the change for issues that were not identified by the validation steps, the test coverage needs to be reviewed and test cases updated. In order to ensure that we are continually improving there has to be a no-blame culture so the team members are not afraid to disclose the issues they faced. It is only when all issues are progressively tackled and resolved that the overall change process gets better.

The ultimate step involves reviewing and closing the change, with a focus on evaluating its effectiveness and efficiency. Conducting a post-implementation review and gathering feedback from stakeholders, users, and customers is integral. Measuring the actual impact and value of the change and comparing them with anticipated results and objectives provides valuable insights. Identifying lessons learned, best practices, and areas for improvement is crucial, with documentation for future reference. Additionally, closing the change request and communicating the closure to stakeholders ensures a transparent and conclusive end to the change management process.

Prior to closure of the change, one must make sure the change tasks ( ctask) assigned to different teams ( and team assignee) are closed within the change window start and end date/time of the original change, otherwise change management will flag this change to the original team and assignee and request for explaination and request additional change management training too.

In my view, tests (pre and post) should never just be a check in the box. Pre tests should be done well in advance, sign off and test artefacts should be recorded in the change record when the change is raised During implementation, it’s important to have a checklist and have another individual review the implementation steps (if there’s bandwidth) to prevent any oversight. Post implementation, all the evidences will need to be recorded in the change and the peer review should happen within the change window.