Lean User Tests: Crowd Testing & What to Do with the Findings (Part 6 of 7)

TL;DR: Lean User Tests: Crowd Testing and What to Do with the Findings

This article of the lean user tests series covers what to do with the test findings and whether crowd testing might be an alternative to running user tests yourself.

What will you Do with the Findings of Your Lean User Tests?

At the end of each user test, you will end up with a list of findings. What will you do with these conclusions?

If you were testing hypotheses as part of your product discovery process, you would now be in a position to decide, which ones to bury and which ones to take to the next level, for example by building a higher fidelity prototype and retest the hypothesis in one of the upcoming user tests.

If you were testing existing applications, Lean User Testing also requires lean “fixing.” Therefore, you should solve those problems that have affected most users in a possibly simplest way. In my opinion, this could very well be an 80% solution. The main thing is that the situation will be improved. Moreover, this is important for two reasons:

1. With regards to the existing product roadmap and bug handling process, companies tend to ignore “newcomers” amongst their backlog of issues or push them on the back burner, as if for the next business value poker. This way those findings from user tests often lose momentum and run the risk getting lost in a big pool of all tasks.
2. Even if the company does work on the problems found during the user tests, you may find engineering be less cooperative to provide a quick & dirty fix and will insist on doing it right. That way, the solution to a problem might be embraced, thus crushing it.
3. However, most of the time small improvements are sufficient enough to improve the usability of a site significantly.

Conclusion: I recommend that you focus completely on the two or three most important issues and try to solve them with the least effort. This emphasis will improve the situation for a maximum of users and shows that you do not only make promises but deliver, too. Otherwise, there may be a risk that some stakeholders consider user tests as irrelevant and start either ignoring — or worse — opposing them.

If you prefer a notification by email, please sign-up for my weekly newsletter and join 9,967 peers…

Is Crowd Testing an Alternative to Do-It-Yourself Lean User Tests?

Wikipedia explains crowd testing as follows:

“Crowdsourced testing is an emerging trend in software testing which exploits the benefits, effectiveness, and efficiency of crowdsourcing and the cloud platform. It differs from traditional testing methods in that the testing is carried out by a number of different testers from different places, and not by hired consultants and professionals. The software is put to test under diverse realistic platforms which makes it more reliable, cost-effective, fast, and bug-free.[citation needed] In addition, crowdsource testing allows for remote usability testing because specific target groups can be recruited through the crowd.

This method of testing is considered when the software is more user-centric: i.e., software whose success is determined by its user feedback and which has a diverse user space. It is frequently implemented with gaming, mobile applications, when experts who may be difficult to find in one place are required for specific testing, or when the company lacks the resources or time to carry out the testing internally.”

Source: Wikipedia (License: Creative Commons Attribution-ShareAlike 3.0 Unported.)

The service providers distinguish between two main areas of application:

Usability tests and bug hunting. The latter usually receive more attention, because of the search for “hard” errors, in general, is better suited as a business model compared to the “soft” analysis of user experience, usability, and user interface design.

• The case where the product to be tested has already progressed quite far or is available online. (The test objective may be the continuous improvement of an existing application.)
• The case where a test needs to be held with a very short lead-time, for example over a weekend, to get feedback on a new feature.

When crowd testing is used for bug hunting, it is also an alternative to your quality assurance department, if that does not exist yet due to the size of the engineering department or due to the long release cycles. Some providers now specialize in outsourcing of quality assurance. Instead of exploratory tests, in these cases, it may make sense to focus your resources towards the creation of realistic test scenarios and to then run the actual test via a crowd testing provider.

The following description, however, does not focus bug hunting, but a usability analysis via crowd testing.

How Crowd Testing Works

I recommend everyone to have tried crowd testing at least once yourself before you utilize any crowd testing services.

For this purpose, I registered myself with Testcloud.de and did a usability test for the UK online shop of an outdoor equipment store.

Before the user test, which had to be done within a time frame of two days, I had to install a screen recording software first, because the offered web version was incompatible with the Java setup on my Mac.

To prepare myself, I studied the briefing for the UX test and the standard procedure for the creation of the screencast.

The actual test took around 50 minutes and was easy to follow with the test guide. There were smaller faults during the test like, for example, a provided English postal code from the test guide, which could not be validated on the website. However, in general, the test was easy to accomplish. Most of the time required — roughly three hours — was taken up by the conversion of the Camtasia screencasts into an MPEG4 file, as well as uploading the video afterward.

For my time on a Sunday afternoon, I was compensated with €20. In addition to that, I was rewarded with another €10 for the best video in eight participants. When I calculated the compensation for this group of testers, total payments to the testers were less than 10% of the amount, the outdoor equipment store had to pay for the usability test.

Conclusion: A usability test in a crowd can be an alternative to running your user tests. This does especially apply when a test is to be held on short notice — like over a weekend. Alternatively, when there are no people available in your team to organize the user test by yourself. However, it is my opinion, which any user test you can organize and run yourself is better than running a crowd test. Because you are then in charge to apply any findings flexibly and influence the actual user test in a positive way. Moreover, there is no way to do this in a crowd testing scenario: Once the briefing is filed and the test is on its way, all you can do is wait and hope for a good outcome.

Good Luck with Your First Self-Organized Lean User Test!

Practice makes you better, and this applies especially to agile methods of all kinds.

The greatest benefits of the Lean User Testing method described in this email course are that they can be quickly organized and won’t break the bank, to begin with. On top of that, they are useful from the start and can mean real progress for your product.

Alternatively, if you are a startup, it can mean reaching the next milestone with the existing funding — also known as extending your runway —, or discover product/market fit.

User tests are a great format for team building in the product and engineering team and to improve the communication with other stakeholders in the company.

User tests are in any case an excellent first attempt at the idea of product discovery and have great potential for the introduction of the agile transformation of a company. Why listen to your gut feeling, when you can ask your real customers on the way to the ideal product yourself?

I, therefore, recommend: Just do it. Moreover, if the company will not allow it, then you should simply consider hacking them. With costs of maybe $200 — $300 it is no longer a question about the budget, but if you want to do it.

Conclusion

This article of the “Lean User Testing” series covers the aftermath of lean user tests and provides insight into the crowd testing option.

See you in part 7, where I will give you all the checklists to make your life of organizing and running user tests easier.

Related Posts

Lean User Tests — Equipment and Location (Part 4 of 7)

Lean User Tests, Part 5 of 7: Best Interview Practices and the Course of the Interview

Four Lessons Learned From Making Customer Value Your Priority

Do you want to read more like this? Well:

• ?? Join 9,957 peers and sign-up for this weekly newsletter
• ?? Follow me on Twitter and subscribe to my blog Age of Product
• ?? Alternatively, join 1,325-plus peers of the new Slack team “Hands-on Agile” for free…

‘Lean User Tests, Part 6 of 7’ was first published in Age of Product.