How to Get Accurate Feedback

One of the things I talk a lot about, is how to get accurate feedback.

The first step to getting feedback is, of course, to just go out and talk to some users, or send a survey, etc. But it's incredibly common to get false positives. That is to say, you present your idea or prototype, they give some positive responses, you take it to launch, and then nobody buys into it.

So let's talk about some common ways false positives happen due to innocuous, fussy details of experiment design:

1) You get data about their conscious opinions instead of unconscious behaviors.

This is the most common and most insidious error. If you ask someone what they would hypothetically want, they will tell you what they imagine themselves to want or imagine they should want: they are responding from what I've started to call Imagination Brain.

This reflects what they think should happen, but may or may not reflect how they would actually behave.

Instead, you want them in Reaction Brain, to give spontaneous visceral or behavioral reactions that you monitor and drill down on.

The key to getting visceral reactions is to take them out of the abstract, generalized, and hypothetical, and put them in a realistic scenario. For example, instead of asking, "check out this app idea, would you want an app that does this?", you can say, "Let's say a few of your friends have been using this app. You have some extra time to check it out, when/where are you?" and then show them the prototype, and say, "here's what you see, do what you would really do, even if it's quit and go make dinner."

It's important to let the user drive: people will do what you tell them to, and if the experimenter tells the user to engage, you will get evidence that they engage.

Note: Consumers' self-prediction is generally really bad. Consumer users' self-prediction about a) how much money they would spend, and b) whether they would continue to do something, is absolutely abysmal.

To test these things, you have to put them in a situation where they have to takeactions that are proxies or leading indicators for the desired behavior. For example, to test if they would really pay, ask very nonchalantly at the end, "Hey, we're about to pilot this soon, would you be interested in joining? No worries at all if not." and then have them fill out a slightly cumbersome signup with sharing some personal information. To test if they would continue using it, have them do a simple, manually-handled version for a week or two. Etc.

Enterprise users' self-prediction is actually mostly okay. People are generally pretty good at giving accurate feedback about the needs of their own workflow-- because managing their workflow lives in their conscious minds. So directly interviewing and asking, "What would you want / would you use this" to enterprise end users is much more reliable.

2) You've phrased the question or test in a way where positives are the path of least resistance.

I recently had a client who was planning to put a series of designs in front of a big group of users, and have them put red, yellow, or green sticky dots on the parts they liked, liked but wanted to change, or disliked.

In principle, this sounds like a fabulous way to build an efficient heat map around interest. In practice, people's brains don't actually work like this:

- Green dots are great, everyone has been trained to like or upvote things. This will mostly be accurate.

- Red dots feel more socially transgressive to place. If you're not a huge fan of a part of the design, but it's not that bad, and you're in this focus group with this beautiful prototype they clearly worked hard on, it feels like a lot to put a big angry red dot on part of it. Maybe you will just skip it. Maybe you will just put a yellow dot on it instead to soften it, like giving a bad restaurant 3 stars.

- This makes the meaning of yellow dots really ambiguous. Does yellow mean yes with specific changes? Maybe? Actually no but I'm too nice to put red? Some people will follow the directions for yellow, and some won't, but their feedback will be treated as identical in the heat map.

It doesn't help that all 3 of these take different levels of mental processing that are hard to do at the same time. "Yes" or "hell yes" is very easy to self-identify and move forward with, so that's another plus with the green dots.

But actually being able to mentally isolate, "Hey, I liked this but want to modify it" takes quite a few steps of mental processing... because it's NOT A REAL CATEGORY to the user, only to the experimenter. (See the next source of false positive errors below.)

How might we fix this?

- Literally use different colors. Green for yes and yellow for no gets rid of the "red is bad" problem.

- Allow people to put multiple dots on the same item if they want. This means that a section with a lot of greens is highly welcome, and a section with lots of yellow means something interesting is going on there. It also saves people from having to mentally sort between "I dislike this", "I want to change this", "this is sorta working for me and sorta not", which is actually all mushed up similar experiences to them.

Note that this solution requires that you give up on having a part of your heatmap that for "things users hate"-- but that's actually okay. Because the truth is that really hard no's are just as useful as hard yeses-- they are evidence that something deep and important is going on, and anything that deep and important is a design opportunity.

3) You draw categories that have meaning to your design process, but not to the user.

So, here's a fascinating thing: When I'm building something, I care a lot about which ideas I have that users like, and which ideas I have that they dislike. This matters to me, because I want to know what to build, so I can build the ones they like and avoid the ones they don't.

But.... For the user, liking and disliking something are NOT two sides of a coin; they are completely different types of internal experiences.

These are the real categories from the user point of view:

- This thing draws me in and I can't seem to forget about it, there's something about it that viscerally compels me to engage again and again (Hint: This is what we're going for in design!)

- This thing sounds nice and would be a nice to have, and I will praise it if I sit down in front of it. But at the end of the day I'm not going to bother myself to go out of my way for it. (This is the most common category of solution that gets built from "positive feedback" and then fails.)

- I am indifferent to this thing. (This is the hardest type of negative feedback to get, because we as a society don't have a great language or cultural habit to express indifference. As an experimenter, you have to work really hard to create space for in difference. This is also your only true "kill category"-- if you can prove indifference, that's enough to terminate that line of idea.)

- I dislike this thing because it's actively bothersome or unpleasant or creates genuine new friction. (This means there is a mismatch between the design and the user in terms of how it fits into their lives. Sometimes this means kill the idea, but more often it either means that you can just re-conceive the implementation for a better fit, or that you're seeing the edges of market segmentation within a previously-thought-to-be-homogenous group of useres.)

- I dislike this thing with real emotion because some part of it really gets to me and I can't let go. (This is fabulous because it hints at a real deep opportunity there! Learn as much as you can about it and then pivot like mad.)

So, given that the user's INNER experience of engagement-or-not is so nuanced, we can't actually expect them to accurately fit their response into a "go/ no go" bucket FOR us-- because the decision about what to DO about their response actually requires a layer of fairly nuanced analysis.

Another very common version of this fallacy is what I call the Premature Survey. In this, the experimenter builds a multiple choice survey and fills the answers with their own top guesses. The user then has to contort their nuance into the non-representative answers.

Surveying is an essential technique-- when you know that the range of answers you're giving accurately represents the range of experiences. But in early research, you will cut off opportunities for major impact by simply never asking the right question, if your survey options aren't validated first.

4) You made it socially hard to say no

If you're running a test with low-income users who are incentivized to participate through financial compensation, try this really simple hack: pay them at the beginning of the meeting instead of the end.

That's it. For financially-incentivized users, you get about 40% better, more animated / engaged, more nuanced, and more accurate feedback if they are not waiting to make sure you're satisfied enough to pay them. In-person, hand them cash or a gift card at the beginning. Remote, initiate the payment transfer and show it to them.

Another common way to make it socially hard to say no is just to make your prototype too attractive or polished-looking. People have a harder time pushing back on something that looks complete. As a result, I'm a huge fan of hideous, black-and-white, unevenly laid out, zero-design prototypes, because then they can focus on the content.