Getting past the no user conundrum

How to do user testing before you have users.

There are hundreds if not thousands of articles about listening to your customers, A/B testing, and collecting traffic data on your site. This is great advise, unless you don’t have any customers yet. 

During this phase of a startup (be it a new organization or just a new product), the organization is vision led. Here, a leader has a vision for what the product will be, the problems it will solve, and who the ultimate customers will be. As the minimum viable product is being developed, the vision leader will make hundreds of decisions to shape the product. 

While this may be obvious, it is important to make a point because the organization will operate differently under these conditions than it will later. Often, when confronted with a challenging design decision or internal disagreement on how something should be done, the team bemoans the fact that we cannot make an informed decision because we “don’t know what the users want.” 

Fortunately, all is not lost. There are several ways you can move forward. The four I use regularly are: 

1. Leverage user research platforms
2. Test internally
3. Ask questions of potential customers or connections
4. Conduct beta testing

User Research Platform

For design decisions, you can use user research platforms that have paid audiences who will take surveys and design tests on your product or prototypes. At Astraea, we use UserTesting.com. This allows us to run many small experiments to test our design and marketing hypotheses. Another similar tool is usabilityhub.com. These tools allow you to customize your audience, so it is close to your target market sector. These tools can evaluate features, designs, workflows, or entire applications.

UserTesting.com has a large panel of testers and when you create a test, you point them at a website (or mockup drawing) and ask them to complete a task (or how they would complete a task if it is not dynamic). The system captures the screen and the user’s voice during the test. The test can have specific questions (which do you like better and why?) or general questions (what do you think of this design?).

One of the most important features of these platforms is that testers are encouraged to narrate their activity so you can hear what is going on in their head. How many times have you wanted to know what a user on your site thinks? In addition to hearing what they are thinking, the system tallies their answers to specific questions. While this can be useful when making an A or B decision, the narrative information is typically far richer. I receive the best feedback when I ask a couple of questions and but grant the tester lots of opportunities to talk through their experience. I frequently discover the answer to a different question or realize that perhaps I was asking the wrong question. This is valuable feedback so don’t just look at the numerical results.

Test Internally

Some companies offer both products and services. At Astraea, we are building a platform for analyzing imagery data, but we also have an in-house data science team. We rely heavily on this team to validate our assumptions, test our tools, and provide feedback from a user perspective.

As with any testing, set some boundaries. While it is impossible to remove all bias, suggest that your internal users forget the company vision for a moment and just treat the task as if they were completing a task in their professional life. I have a task and I have a tool - go. Lots of times developers call this "dogfooding" as in "Eat your own dogfood." In case you never heard this before – I’m not making this up:

Another great reason to test internally is that it can get the whole team on the same page. Don't forget that not everybody in the company has likely seen or used your product. Make sure your marketing team, your administrative team, your sales team, and your leadership team have all seen your work is important for keeping the company aligned and well informed.

Ask questions of potential customers or connections

Usually, when a group sets out to solve a problem, they have high connectivity with the domain and the practitioners in that domain. Taking advantage of this network for important questions is a good way to help validate your assumptions. While this is one of the most powerful approaches, you can only ask so many questions from your friends. So, save this for the big questions.

It is tempting to want to keep your idea a secret until you are ready to share it with the world. Unless you are building something that is easily replicated however, this is a mistake. Getting early feedback often far outweighs the risk of being copied. The people truly interested in what you are doing will be eager to get their hands on the product when released. These are great candidates for beta testers. Some prominent product managers believe that you should start your marketing at the same time you start development. So be bold, get out there, and move fast!

Beta Testing

Using your personal network or simply advertising for some interested beta testers on social medial can yield a handful of folks interested in checking out your product. A beta test is typically more formal that just asking questions of your contacts.

Beta testing may feel like a daunting task, but we have found that a few good beta testers give you plenty of feedback. The Nielsen Norman Group recommends only 5. The trick is finding the right person willing to not only poke around but actually interested in trying to DO SOMETHING with your system.

What questions do you ask?

When I started reaching out to potential users, I was more ad-hoc in my approach. I asked some basic questions to get a conversation going and followed it where it went. As I did this more, I realized that there were times when I was not clear in my own mind what I wanted out of the engagement and as a result, I wasn’t making the most of the opportunity. This is one area where putting on a sales hat actually helped. Our business development team put together a list of qualifying questions to assess whether or not a buyer was qualified. Interestingly enough, many of these questions were very practical from a discovery point of view.

Don’t make the mistake of turning a product discussion into a sales call – that’s not your objective. I’m just pointing out that the questions your company uses to qualify a customer may help guide you as you consider who you are building the product for and what are their needs.

Summary

The most important advice I can offer is don't be shy about getting input. I'm naturally somewhat introverted, so contacting strangers to ask them favors and questions is not always my natural instinct. However, I have found getting this feedback very rewarding. I meet interesting people, connect with them, and learn about what they are doing. I know this feedback helps me understand their perspective and use cases. The more I do this, the better I get and the more fun it becomes.

Whichever technique you use, practice asking the right questions and make sure that the answers you get align with your vision. If the feedback you are getting leads you away from the company vision - collaborate with your leadership team to ensure you are running down the right path. This is one case where collecting more data will probably be very important. Lots of companies pivot based on user feedback, but that is a big decision.

There are many other ways to get user feedback before you have users. I’ve listed my favorites. How do you collect feedback?