Continuous UAT and DevOps

Delivering business value quickly is a main goal of most of the product and services organizations. This didn’t change since the times of Frederick Taylor or Henry Ford. However, the methods of delivery have changed significantly. With the adoption of DevOps, more and more organizations empower teams that integrate both technical and business members, to focus on self-organization and continuous process refinements in order to achieve the business goals. 

One of the foundational elements of DevOps is the continuity of delivery. Mostly we associate it with automation. We utilize concepts of Continuous Integration, Continuous Deployment, Continuous Delivery or automation of unit and integration testing. All these significantly improve the process of delivery. However, the exercise often dramatically slows down when we encounter the User Acceptance Testing (UAT) step. This is hard to automate. Although some might argue that such tests could be codified and automated, this is only partially true. Automation of UAT can get us only so far. In most cases it is not feasible or even possible to automate all possible permutations that the end user could think of while using the product. Therefore the end user review of the application, confirmation of its usability, or business rules is and still will be the manual, final proof that the application is in state of production readiness. Main reason for the delays is related to the availability of members of business teams to perform the UAT with high frequency. It is a paradox, but the delivery of functionality is often too fast for the business to be able to accommodate.

However, with DevOps we have several approaches that could help with addressing these challenges.

Canary Releases

Canary Release (or Friends and Family), is an approach to release the product to a small group of users in the market. Usually it is done in segment that is friendly, forgiving and will provide enough feedback. Product Managers usually know who are these groups. Some large companies even use entire countries, in which spoken languages are not widely used outside of that country. This helps to contain possible negative feedback, that could impact the product when it is finally launched.

Pros: ability to collect feedback from the end users.

Cons: limited to small segment of the market. It is also dependent on the skill of the Product Manager in selecting this segment, as it will ultimately define the quality of that type of test.

Dark Launches

This approach implements programming switches in the application code. It allows to release the application to the entire market in a stealth mode. In other words, when the user initiate a process, the application starts running two parallel functions - the old one that the user sees, and the invisible new one. It allows to collect information about application performance under usual load.

Pros: It allows to test entire market segment.

Cons: Since the user doesn’t see the new UI, it is hard to collect user feedback. Also, it adds administrative overhead to the team to maintain and clean the switches. Therefore Dark Launch releases are often limited to usability performance testing.

Continuous UAT

Another approach is to have dedicated business users that continuously test the business stories as soon as they pass Definition of Done. For many teams that means several tests per week.

Pros: User feedback is provided quickly and it covers all aspects from UX to business rules.

Cons: Availability of business users to perform testing several times a week. Also, the quality of the feedback might suffer, as it is done by the same users over and over again. After some time, the testing group might become desensitized.

Which approach to select?

My view on this is that a combination of all three will give the best results. The product stories are tested by the business users as part of the acceptance criteria (Continuous UAT). Once they passed the new functionality can be simultaneously released to a selected subsegment of the market (Canary Release), to gather customer feedback, and in the stealth mode, to the wide market (Dark Launch) in order to collect feedback on application performance.

However, ultimately it is up to the Product Managers and Product Owners to decide if to use one, two, or combination of three of them. This decision depends on the criticality and sensitivity of the application, risks, time and budgetary constraints.

Andre Kaminski is a highly experienced technology leader and Agile DevOps consultant. As Director - DevOps at WorkSafeBC (WSBC), he is responsible for setting up and leading Agile DevOps functional teams to deliver business value. Over the past three decades, Andre has worked in engineering, product development, project and program management, as well as has served in advisory roles across industries in both the public and private sectors.