A 5-Step Guide to Getting User Acceptance Testing (UAT) Right

I came across this tweet, a few days back and haven’t been able to get it out of my mind for a while now! As hilarious as it looks, it is, at the end of the day, a harsh reality. A bunch of us in the Product Team would have visualized our users to be using our product or a feature in a certain way but that seldom happens. So, how do we determine if people will use the product the way we would want them to? The answer — User Acceptance Testing (UAT).

User Acceptance Testing or UAT is often the final testing phase before the product is rolled out for use by end-users. Almost always done with the objective of ensuring the identification of issues, bugs, or inconsistencies that might have gotten missed during the initial testing phases, UAT is performed by actual users who are representatives of the target group/ audience.

Often, UAT gets clubbed with other forms of user testing — such as usability testing (users using the product in line with how it was originally designed) or even functional testing (all scenarios including those to induce an edge case are performed). It is important to note that UAT goes a step beyond usability and functional testing into the terrain of desirability. UAT also checks for how desirable a feature is when it is made available to the audience.

The Step-by-Step Guide to an Effective UAT

Step 1 - Get your Scope Right: Each feature that is being subjected to UAT will have a User Story associated with it. That User Story needs to be the scope of UAT for the feature. This User Story is also what defines the result against which the testing is being done.

Eg. You have built a feature in your app that allows users to book appointments with general physicians in their area. In this case, being able to successfully book appointments with general physicians will be the scope of the feature that is being tested in the UAT.

Step 2 — Get your Acceptance Criteria Right: Develop meaningful acceptance criteria. This is the single most important step to ensure that all features built under the bigger umbrella of a user story are being subjected to correct testing. Sometime back I had written a piece on how PMs can write User Stories with Gherkins — it is important for PMs to define the acceptance criteria while writing the user stories so that there is no scope for any misalignment at a later stage. Taking the above example of booking appointments with general physicians —some of the typical acceptance criteria that would define the test cases would look something like:

Can the bookings be done for a user-preferred slot? If a preferred slot is already booked, is that being shown to the user? Is the list of physicians being shown within a 5-km radius? Can the user sort the physicians by their ratings? Is sorting by consulting fee working? Can the user filter for online vs offline appointments? Can the user ultimately select the physician of their choice and book an appointment?

Typically the answers to most of the acceptance criteria will be binary — either the acceptance criteria would have been successfully met or not.

It is extremely important to ensure that your acceptance criteria carry a fair reflection of what users would want or expect and not just what you have conceived.

Step 3 — Get your Test Scenarios Right: Once you have identified the scope and the acceptance criteria, conceptualize all the test cases/ scenarios that your testers will perform during the UAT phase. Try to make these scenarios or test cases as broad-based as possible to simulate real-world interactions. Remember that your users will seldom toe the “happy path” and will use your product in a gazillion cranky ways — so be prepared for that!

Step 4 — Get your Target Audience Right: It is important to ensure that those testing your product during the UAT phase are correctly representing the users who you would want to roll the product out to. The decision is driven by two factors: a) what has been built and b) what are the insights that you are keen to gain.

Once again, taking the example of booking an appointment with a physician — say if you have built the overall MVP version of the feature, it is better to roll it out to a larger set of users to see if they use the product; on the other hand, if you have built a specific feature (say the ability to reschedule a booking), it may be useful to roll it out only to a subset of users who have shown initial affinity to the product.

It is important to ensure that your testers reflect the demographics of your actual users. However, a lot of times, the UAT phase happens in-house — which means that internal team members (read testers) who are given a script with the different test cases and acceptance criteria are the ones who do the testing.

Step 5- Execute, Evaluate, Resolve, and Roll Out: This is where the penny drops! Execute the test, ensure that testers are simulating all test cases against the acceptance criteria, and are documenting their findings. It is also rather useful for PMs to monitor the testing live to see for themselves the different paths that users are taking — this can give a lot of qualitative feedback on user journeys.

Once the testing is complete, the developers and PMs need to go through the full log of bugs to address the issues found and have them fixed and ideally retested before the feature is deployed in production. At the most fundamental level, UAT should measure the success or failure of the key user acceptance criteria. If a specific feature fails to meet the user acceptance criteria, PMs and developers need to review and decide on what needs to change.

Once tweaked, the feature can be slated to be released in the next production release. If you can afford it, try and hold on to releasing it to your entire user base — if possible, release it to a smaller chunk first, conduct more UAT rounds in the background, and slowly roll it out to your complete user base. This, however, might not be always possible (especially if your new feature seeks to solve a problem that is very here and now).

To conclude, the UAT phase is, in some ways, a chance for product teams to identify bugs, get user feedback, make tweaks to the features, and have ready the best possible product available for production push. The approach to UAT is determined by a set of factors — the most important being the scope of the release, the type of the release, the resources available, the type of users, and most importantly the end goal that the feature is trying to achieve.