Avoiding a Common UX Research Pitfall

We’ve all heard the joke about the police officer out walking the beat one night, and running into a man fumbling around under a street light:

“Are you OK?” asks Officer Friendly. “What are you doing out here in the middle of the night?”

“I lost my house keys. I’m locked out.” is the man’s reply.

“Oh, let me help you look.” replies our intrepid and helpful policeman.??

After 20 minutes of searching for the man’s keys, the officer finally asks the man, “Are you sure you lost them here?”

“Oh, no” says the man “I lost them in the park a couple of blocks away.”

“Then why the heck are we looking HERE??” says the exasperated officer.

“Because this is where the light is!”

As I said, we’ve all heard the joke. The absurdity is in looking for something because it might be easier to find it under the light. Funny, right? But how many of us have this same kind of thinking sneak up in an unexpected manner when we aren’t being intentional about how we gather data?

And just lately it registered to me how much my own company had inadvertently been looking for solutions under our own street light. Sometimes the most profound ideas sneak up on you, even though you’ve been involved in something long enough to be considered an expert. The whole point of a lot of our methods in research is help us avoid the kind of cognitive biases that can lead us down the wrong path.?

A lot of established, respected organizations built successful businesses through iteration and acquisition, rather than planned growth. Additional functionality was bolted on, greatly increasing the feature set. Often, the easiest thing to do at each of those points was to just retain the old tech stack. That probably made sense in each individual case, but eventually the result was significant technical debt and a system which evolved more than was designed to work holistically in a thoughtful way. And your users meanwhile developed lots of circuitous, indirect paths based on best guesses at the time. Then that hard-won knowledge of how to use this complex system helped develop a set of users who had become experts in a complex system. And their employers became dependent on this? specialization.? This status quo can work really well to lock companies and their customers together for a long time. It is a compounding effect with many ramifications.

Eventually, however, the bill comes due, and the organization is faced with existential choices.? Often, a new agile competitor enters the market - one whose express marketing strategy is disrupting old tech. For your organization, hobbling along on the old tech stack is no longer viable.? Other times, the organization sees the writing on the wall and looks to proactively disrupt themselves before a new player comes along.? And making the new system work creates an opportunity for a new, simpler, more direct user experience.??

That sounds like an unqualified good, but the hard-won expertise your existing users built may now become obsolete.? Those switching costs matter to the users, often more than to the customers. The harder a user has to work to learn a system, and the more complex and circuitous that system is, the more entrenched those desire paths become for them. “Don’t move my cheese” is the essence of the refrain you hear, either directly from the people who represent the current (i.e. temporary) majority of your users or indirectly via the support people who have spoken to them most and feel they represent “the users.”?

Facing that past-due bill, there is pressure to move as quickly as possible while still trying to be driven by data. Recruiting users of the existing system for generative and evaluative studies can prove difficult, partly because they have a difficult job of their own to do and sometimes can’t see the benefit of contributing to your knowledge. Recruiting novice users is even more difficult when the expertise I spoke of earlier is required, because typically novices are rare and are trained by existing users.?

So UX Researchers and Designers do the best we can, and also spend time speaking with Customer Support and looking through support tickets, working with internal SME’s, and so forth. This is an excellent strategy when iterative progress is being made, but can hide another valuable source of data (novice concept models, learnability, etc.) in the dark parts of the street when major changes are made to workflows.

Companies facing a change to uproot and revamp the technology and improve on the user experience often are going to have to make a difficult choice: please the existing user group or lean more heavily on designing for future users and future customers. If we aren’t careful, we will end up finding the data that is easiest to find, right under the street light, and miss the data that would indicate a different direction is needed. So as part of our scientific mindset as User Experience Designers and Researchers, we have to be always thinking about how best to collect the full data, and interpret it in the best way, so as to better help our companies make the right decisions.

With the push to be data-driven in strategic decisions, the temptation is high to sample what is available, accidentally propagating the current state. So against that tension to move fast, UX Research argues that the cost of quickly moving in the wrong direction is likely higher than the cost of slowing down enough to strategize on who your real target users and customers are, and doing the real research needed to drive the best decisions on how to get to where you are going. The latter creates the win-win situation we crave with our customers, albeit sometimes at the expense of existing expert users if no compromise is available or a product needs to sunset. And we need to be viewing our own opinions with scientific skepticism so we don’t end up looking around under a street light, focusing on the wrong data to drive those decisions.

#uxresearch #productdesign #strategy #criticalthinking