Measuring Product Design Impact—KPIs, NPS, UX, WTF

We Product Designers have a problem: we are creative souls.

Being creative souls puts Product Designers in a precarious position. Business tends to not be creative, and yet here we are: Product Designers, working for a business. What are we doing here? How do we justify our existence? How do we find meaning in what we’re doing? Not in an existential “why am I here on this earth” kind of way, but in a “how does my work truly benefit the business” kind of way.

As creative souls, we approach our work with a certain level of personal attachment, where we have a tendency to put a part of ourselves into every project we touch. It’s this personal attachment that is both our strength and our weakness. On one hand, by connecting ourselves (our emotions, our aesthetic sensibilities, our personal opinions) to our work, we impassion it and make it special in some way. This gives us a reason to remain excited, so we can keep coming back to our designs day in and day out. On the other hand, by creating a personal attachment to our work, it makes us that much more resistant to criticism, change, and starting over, and also inhibits us from seeking out ways to better our work so it performs better. And, at the end of the day, we do want to be in it for the creativity (or at least some part of us does). So what’s all this about performance?

Performance, to be blunt, is very much the purpose of having Product Designers on staff—to help build digital products which perform better for the company. That’s why we’re here. The business depends, in part, on Product Design to imagine, create, and ship highly-performant work to (wait for it) make the business money. 

Yes, that’s right. We sold out. It (whatever “it” is) is not solely about the creativity, it’s about the money. Put on your big boy (or big girl) pants and accept it. There are numerous digital products and experiences that have been built, and are yet to be built, which have found massive success despite having no brilliant Product Designers, let alone any Product Designers, on staff. So, in a way, a product doesn’t need a Product Designer to succeed. And yet, as one of very few brilliant Product Designers on earth (and that shirt looks great on you, by the way), you do have something special about you, and you do add value to a business, but you’re really truly not getting a paycheck because you’re especially creative, so much as you are remarkably astute.

Astute about how people think and feel. Astute about the culture that surrounds us all. Astute about trends which are coming and going. You’re just plain astute, and you’re being paid to translate that into cold hard cash.

Now that you’ve settled into your newfound reality as a revenue-generating machine, you’ve yet another challenge ahead of you: proving that you are bringing the goods. In other words, proving that this remarkable astuteness you’ve got inside your gorgeous head is actually worthwhile for the business. And, fortunately for you, there are some ways to do that. But first, let’s define the broader concepts behind these ways/methods/terms before we talk about how you might leverage them for your own dastardly deeds.

Defining KPIs

In “the biz,” KPIs are everything. Also known by lay people (who we cool people laugh about) as Key Performance Indicators, KPIs are data points that help the business understand if their products are performing in the way they should. For example, a KPI at a fintech (financial technology) company might be as simple as the percentage of users who land on the website and convert to members (i.e. open a bank account). Or at the same company another KPI might be the percentage of members who begin direct depositing into their accounts. Ideally, these KPIs would be realized as high numbers (e.g. 90% of visitors open an account), and would continue growing month to month, year over year. And, if you’re lucky, you might have an awesome engineering team who built in an experimentation platform, allowing you to make little tweaks and changes here and there, which summarily allows you to measure which of them positively affects your KPIs. Whichever tweak works, you double-down on. Rinse and repeat ad nauseam.

Other KPIs might be things like gross margin, profit per employee, and other money-related figures. While these are important KPIs, they aren't purely tied to Product Design so much as they are tied to the health of the greater business, and they're impacted by a wide variety of things that aren't under your direct control.

The great thing about KPIs is: you just set up your tracking methods and watch the data pour in, and you learn what works and what doesn’t. The bad thing about KPIs is: sometimes you’re tracking the wrong things—things which don’t mean much to the business but might make a particular employee look better (I’m not mentioning any names, but you know who you are).

The hard data you get from KPIs is called quantitative data, in that it has a quantity to it...aka a number. This can get slightly fuzzy when you consider that NPS (defined below) can also be part of a KPI.

Defining NPS

Also known as Net Promoter Score, NPS is often gathered via a simple survey that asks a question like “How likely would you be to recommend My Awesome App to a friend?” These surveys are routinely delivered via multiple differing methods, like email, or in-app popup.

The recipient of said question is given a series of buttons, 1 through 10, with 10 being the very best. If people really love you, they might smash that number 10. If people really hate you, they might slam that number 1. If people feel a bit apathetic, they might lightly tap a 5 or a 6 while an audible "meh" escapes their lips. Your friendly neighborhood User Research and/or Data teams will take the sum of all those survey results and come up with a percentage. Anything less than 50% is pretty poor. Anything above 70% is mind-blowingly incredible. Why the strange discrepancy? Why is 50 awful, 60 decent, and 70 a blessing? Well, humans are very emotional and reactive creatures, and despite your best efforts you will sometimes receive survey results where a vindictive user will give you a “1 out of 10” just because something small went wrong a brief moment ago, and they’re so raving mad that they want to get back at you however they can. “Vengeance shall be mine,” they’ll say as they rate you poorly, laughing aloud.

The great thing about NPS is: it’s really simple to gather and calculate, thus giving you a regular taste of how happy (or angry) your customers are (assuming you measure on a regular basis). The bad thing about NPS is: people will sometimes blame (or reward) Product Design for NPS when it's is really the sum of a user’s entire experience, (a UX in a sense) and not just the impact of design by itself.

The results you get from NPS are sometimes referred to as quantitative data, because they’re numerical, even though NPS is a measurement of opinions. Because they measure opinions, they should really be considered qualitative data, in that they measure the quality of a user’s experience.

Leveraging the Lingo

Now that you know the lingo, how do you apply it to real life? How do you take your KPIs, your NPS, your UX, and whatever other letters are floating around your Scrabble board, and mash them into a sandwich that not only supports a creative and inspired design organization, but also drives your business? First, start with the KPIs.

In most tech-centric businesses, the Product team defines the KPIs. It’s important that you, as a Product Design representative, learn to partner up with Product Managers to ensure that your KPIs are measuring the right things—things which represent actual user needs and behaviors. If you struggle to partner with Product Managers to get proper KPIs, then partner up with the Data team and define your own KPIs. Your Head of Product Design ought to gather this data on an ongoing basis to report out.

The most applicable KPIs to measure design success can vary between products and businesses, but in general I’ve found them to be:

1. Time to Goal
2. Discovery Method
3. Success Rate
4. Failure Rate
5. Return/Repetition Rate
6. UX-Focused NPS

You don't need to adhere to my list. Defining your own KPIs, based on your company's unique needs, is also totally fine (and probably better)!

1. Time to Goal

When a user visits your product, you and that user have a goal in mind. If your business is set up properly, these two goals should be similar if not the exact same. For the sake of this example, let’s pretend you are working for Netflix and you’d like users who open your app to quickly find a video and watch it. In a perfect world, the user would open up your app and see the exact thing they wanted right in front of them (because you can read their mind). Since we live in an imperfect world, the user will probably need a bit more time to find that ideal video.

Now, some users may know what they’re looking for, while others won’t. Those users will need a bit more help from Netflix’ brilliant algorithms, which will surface videos that are likely to be right for the user. Yet, in both of these cases, design plays a major role.

The placement, size, color, and shape of iconography can affect how quickly users locate your search or browsing features. The specific language you use can confuse or clarify those same features. The hierarchy of the layout can affect how quickly a user can scroll through content options. All of these are the results of design decisions that you and your team make, and this KPI is thusly yours to own, defend, and champion.

2. Discovery Method

Using the prior example of Netflix, there are two core methods of finding content: Searching for it and Browsing for it. When a user Searches for a video, that action comes with implications: either the user knows exactly what they want to watch, or the user just can’t find what they’re looking for by Browsing alone.

With Searching, you can determine the user’s intent by watching their terminology. Did the user type “Die Hard 2” or did the user type “action movies”? If the user typed “Die Hard 2,” then the user knew exactly what they were looking for. No harm no foul. You made a competent Search experience that helped that user find the video of their dreams, and you should be applauded for it. But if the user typed “action movies,” and right on your home feed you have a section labeled “action movies,” then this suggests the user didn’t see that section at all (implying your layout may be to blame), or the user saw the section but found nothing of value (implying your algorithm may be to blame).

With Browsing, a similar determination can be made. If you’re seeing users scrolling through your Browse experience, then subsequently going to your Search feature, it suggests your User Experience may be failing them in some way. Either your Browse layout is problematic, or your algorithm just isn’t as smart as it should be, forcing users to Search for something more suiting their tastes instead.

In either case, you can gather data that visualizes your user journeys from launching your app (the beginning) to finding content and/or exiting (the end). Over time, you’ll see trends form, which can help you better understand how you are supporting or failing your users. Through experimentation, you may find ways to optimize the discovery experience and, thusly, shorten Time to Goal.

3 & 4. Success Rate and Failure Rate

This one (two?) is simple. How frequently are users successfully completing a task or a goal? Ignoring the time it takes, are users consistently clicking or tapping on options and reaching a meaningful conclusion? Or, do you perhaps see users moving in circles, typing the wrong data into forms, and generally not doing anything of value before exiting your app? Your design decisions directly affect user successes and failures, and so you should set appropriate KPIs relative to whatever you define as “success” and hold them closely.

As an example, imagine you’re still Netflix and you have a form field on your home page that asks users to sign up for a new Netflix account, but you keep seeing users typing their pre-existing account information with the intention to log in. This represents a failure, and as a Product Designer you own this failure and it is yours to find solutions for. You might also consider having a coworker punish you for this failure, to make sure you learn from it.

5. Return/Repetition Rate

In almost all tech companies, you desperately want repeat business. Repeat business not only means continued, reliable revenue, but it also means you are providing a service of value to your customers. As a Product Designer, it’s important that your User Experience is so easy to use, so free from friction, and so delightful that customers return to it again and again.

Fortunately, it’s quite easy to track return visits from the same users. Some more advanced tracking systems can also link singular users to their activity on separate devices (meaning you can see that Bob Smith is using your service on both his laptop and his phone). It’s important, though, that you combine this metric with the completion of goals. Otherwise you’ll just see that Bob Smith came back to Netflix 5 times in one day, and not that each time he came he didn’t watch a single video. If Bob failed to watch videos on each of his visits, this implies he couldn’t find a video worth watching, which could be a design problem.

6. UX-Focused NPS

The problem with most NPS methods is they focus holistically on the entirety of a business. “How likely are you to recommend Netflix to a friend” is a pretty loaded question, isn’t it? A person could love the Netflix brand, love the cost of Netflix, and even love the content selection, but every time they try to open Netflix on their iPhone it crashes. So when they’re asked if they’d recommend it, naturally they might decline. Before you know it, because of a bad app release, your NPS is 40. This isn’t the fault of Product Design, this is an engineering problem.

So how do you measure NPS in a way that digs into design and UX? You have to broaden your NPS surveys a little bit. This can be tricky, as the more questions you ask, the less likely a recipient is to answer them. But, for those that do bother to answer the questions, you can add in meaningful ones that get to the heart of something Product Design can own.

For example, you might ask a question like “How likely are you to recommend the watching experience on our iPhone app?” Assuming that the app works, of course, this helps you dig into the user’s enjoyment of watching content through your iPhone app, and not necessarily whether or not they’d recommend Netflix on the whole. This is a zone you can own! I rhymed that on purpose.

Getting the Data

It's one thing to identify your KPIs, and another thing to actually have the data. Tools like Amplitude can help designers track and visualize data using semi-intuitive filters. It's important that, once you've managed to build a functioning filter that returns accurate data, you share it with the rest of your fellow Product Designers so they may be exposed to it, learn from it, and possibly build their own version.

Tools like Amplitude, however, are not one-click installs. You don't just set it and forget it. You'll need the talents of engineers to add the right scripts and hooks into your products so you may actually gather data to begin with. This isn't the most difficult thing in the world, but it does require buy-in from the Engineering org and sometimes even Product. In order to acquire this buy-in, you'll have to clarify its purpose: you wish to quantify the impact of design. Not everyone may be on board at first, but this is a conversation worth having. Remember, your meta-existential self-worth is at risk!

Bringing In Research

After you’ve nailed the KPIs, you also need to run some research. Having a User Researcher (or more than one) will help, because a good User Researcher knows how to ask impartial, non-leading questions. But, if you can’t find (or afford) a User Researcher, you can still run some rudimentary research and gather some results on your own.

The most critical aspect of running User Research is not to run tests with employees of your company. When you place prototypes, designs, or any other idea in front of a fellow employee, they will be approaching the work from an all-too-informed, subjective position. Your regular users are better guinea pigs as they are definitely not informed, but are always objective (even if it may not seem like it).

To put it differently, your users do have selfish intentions when using your products (they want to complete a goal of their own and then get on with their lives), but they don’t have anything to gain by telling you they do or do not enjoy using your prototype. To them, the prototype you’re testing is just some fancy new idea that may never make it out into the real world, and so they approach it without a real concern for its success or failure.

With that said, let’s take a step back. User research. It’s a great idea to set a regular cadence of User Research so that once a month, or every other month, you’ve spoken directly to users about their experiences with your product and how it might improve. You can do this by building prototypes of new ideas and gathering their feedback, by recording videos of them using your live apps and asking questions along the way, or just by talking to them on the phone about what they love and hate about using your products. Just stay focused on identifying pain points as well as moments of joy so you can learn about what’s going wrong and what’s very much going right.

And please, whatever you do, do not ask users what they “want” in your app, for they do not know, even if they think they know.

Using this qualitative, straight-from-the-horse’s-mouth data can help you learn about the User Experience you are providing to your customers and, ultimately, more than a standalone NPS survey can provide. User Research also provides you with handy quotes to put into slide decks, which never get old.

Putting it All Into Practice

As mentioned previously, you’ll need to learn to partner up with other teams to find and measure the right KPIs, and run the right User Research. That’s right, you’ll have to speak to other people to make this all happen. Sorry. But the results will support your efforts as a Product Designer, whereby you’re trying to find meaning in your work. Remember how we talked about that?

By consistently monitoring these KPIs, you’ll be able to learn as you go and leverage these learnings to make new design decisions. You’ll be able to reduce friction and increase revenues. Why, you’ll be a regular money machine! You deserve a raise (tell your boss I said so).

And, as a result of monitoring these KPIs, you’ll be able to stay creative because you’ll now have clear problems to focus on. Design is really the practice of solving problems, after all, so when these KPIs go in the wrong direction it presents you an opportunity to put on your “I’m a Designer” snapback cap and get creative. Then, when you improve the KPIs, you’ll have the opportunity to present charts and graphs at your next All Hands meeting like the big executive types do (the ones who wear the big boy/big girl pants…and the ones who wear nothing but Patagonia vests).

You see, at the end of the day, we Product Designers want to be creative, but it’s really the cold, hard data that supports our work and clarifies our value to the non-creative types. By embracing that data, and setting our own goals, we are carving out a space for Product Design to not only succeed, but to be valued as a key player in the growth of the business. And as a result, we are solving our own meta-existential crisis and proving that we are, indeed, bringing the goods.

Good luck!