How can you measure user experience in Product Management?

When taking the product-led growth or product-led sales lens (PLG/PLS), product UX metrics have to be supplemented with product funnel metrics in order to understand the success of user experience decisions taken by a product team. Product funnel metrics are a set of KPIs that measure the volume, velocity and conversion rate of customers across the product funnel. These in turn help a product team understand the effectiveness of their bets and helps them continuously align the product roadmap to maximise value creation. Couple great frameworks that can be used are the popular AAARRR - or a lesser known but has worked tremendously for my teams at early stage growth is the OODA loop (observe, orient, decide, act) for product teams.

UX efforts concentrate on making the entire customer journey pleasant and productive. This begins with highlighting the value of the solution so it is clear what capabilities and benefits the product can offer to it’s potential and current customers—and that the value proposition is clearly desirable to the target market. This is where it becomes important to Product Managers

Defining UX metrics is crucial for assessing the effectiveness of a product's user experience. Here are some key steps to consider: 1. Net Promoter Score (NPS) 2. Customer Satisfaction Score (CSAT) 3. Task Success Rate 4. Time on Task 5. User Engagement Metrics: These include metrics like Daily Active Users (DAU), Monthly Active Users (MAU), and Stickiness. 6. Churn Rate 7. Retention Rate 8. Error Rate 9. User Feedback and Surveys 10. System Usability Scale (SUS) 11. Customer Effort Score (CES) 12. Feature Adoption Rate Remember, the specific metrics you choose will depend on your product, industry, and objectives. Regularly analyzing and interpreting these metrics will help you make data-driven decisions to enhance the user experience.

What are the most frequently asked questions that the customer support team receives? If these questions revolve around whether your product solves problem X, it's a good indication that the feature isn't easily discoverable. If most users can find the feature but require assistance on how to use it, it's a sign that some steps are not intuitive. Your goal is to increase the task completion rate. In both cases, collaborate with the support team to track and categorize the tickets so that you can monitor whether your efforts have reduced the number of inquiries in a specific category. The support team will appreciate your help in reducing their workload.

Your UX metrics will not be helpful if they are not around the journey that the user is going to take on your product. Ensure that you have a well-designed user journey and then try to measure the experience of the user around the touchpoints that the user has in that journey

A practical problem product managers face while analysing user behaviour data or while collecting user feedback is reducing bias-creep. Picture this: you're running a hallway test for a newly launched product module that is tailored to impact activation metrics. While observing a user playing out the module, a product manager might unknowingly nudge the user with her body language or implicit reactions to not perform a specific action. Similarly, in a customer discovery call one might ask leading questions such as "do you like feature x?" v/s say a more generic "what features have you used before". Bonus: My personal fav, The Mom Test by Rob Fitzpatrick, outlines a quick & dirty guide to open ended questions that get someone to open up.

Basically, UX professionals want to get users on the most efficient path to satisfaction and make their experience as seamless as possible. By being the user advocate and reporting findings from research and testing, UX provides important insights that helps guide the decisions of the Product Manager and hence leads to success of the product team.

While collecting user behavior and user feedback will be key, there will be some amount of bias that will exist in both collection and analysis of this data. If possible, ‘dogfooding’ may be something that could be considered as an added layer of information to validate what you are listening from your users. Many brands like, Spotify, encourage and use this technique to constantly ensure they are internally, actively, using their product and putting into context the feedback they receive from customers. This can sometimes lead to preemptively solving key UX challenges.

There are times when it is not possible to capture common UX metrics, we can rely on the technology stack to provide us insights which can then be used to determine patterns. Examples can be to ask engineering to add log lines in their code and filter out these on the log monitoring systems like logstash or splunk. And track the usage of specific features, experienced page times, retry attempts, and occurrences of validation errors and then incorporate such details into the UX design iterations.

Collecting UX data can be challenging due to issues like limited user feedback, biased responses, and data overload. To address this, focus on diverse user representation, employ multiple feedback channels, and use both quantitative and qualitative methods. Regularly iterate based on feedback, conduct usability tests, and leverage analytics for a comprehensive understanding. Conducting a System Usability Scale (SUS) test can simplify a Product Manager's life. It provides a quick and reliable measure of usability with a standardized questionnaire. The scores give a clear indication of user satisfaction and areas needing improvement.

Analyzing UX data is crucial for product success. A useful framework for this is the DIKW Pyramid, which stands for Data-Information-Knowledge-Wisdom. Data: Collect raw metrics like click-through rates or time spent on a page. Information: Process this data to produce meaningful figures, such as average time spent on a task. Knowledge: Interpret the information to identify trends or issues, like high drop-off rates at a certain step. Wisdom: Use this knowledge to make informed decisions, such as prioritizing features or implementing A/B tests. The DIKW Pyramid provides a structured approach to transform raw data into actionable insights, ensuring a more user-centric product.

The most effective way for a product manager to analyze UX data is through a "User-Centered Approach." Begin by defining clear objectives based on user needs. Collect both qualitative (e.g., user interviews, usability tests) and quantitative (e.g., analytics, heatmaps) data. Analyze the data to identify patterns, pain points, and opportunities. Prioritize findings based on impact and feasibility. Visualize insights using personas, journey maps, or storyboards. Collaborate with cross-functional teams to implement changes and continuously iterate based on feedback. This ensures product decisions are rooted in genuine user experiences, leading to enhanced usability and satisfaction.

Along with the data, Context matters. Data can tell you what is happening, but often not why. Don't just rely on numbers; contextual interviews can provide the 'why' behind the 'what,' helping you make more informed decisions.

Your research data is too valuable (and expensive to gather) to use only for a single research question. Better aim for a repository of reusable data, and then use it to answer different - future - research questions.

When starting to analyze this data, it is super important to understand and dissect the sources of the data. An age-old problem of UX interviews, especially in B2C context is that PMs and UXRs get inputs from two types of users: a. Who love your product b. Who hate your product Generally, the two do not collectively form the majority of your user base. We need to be extremely careful of our own confidence while extrapolating our insights to cater to the largest possible segment of your users.

Scrappy prototypes if you need to design. No-code launches if you need to execute. Check the feasibility with focus group discussions, same users who had highlighted issues earlier, new users who liked the earlier solution as well - see what floats - then invest!

To validate your UX hypotheses, design controlled experiments or A/B tests where one version retains the current design and another introduces the proposed changes. By comparing user interactions and feedback between the two, you can determine if the hypothesis holds true. It's essential to ensure that the sample size is large enough and that the test runs long enough to gather statistically significant data.

Before diving into costly A/B tests or user testing, consider using smoke tests or "fake door" tests to gauge user interest in a new feature. It's a low-risk way to validate demand before fully committing resources.

Validating UX hypotheses is the bridge between analysis and action. Here I prefer to leverage my Growth Hacking Mindset, always seeking the agile and efficient ways to confirm the assumptions. Whether it's through rapid prototyping, A/B testing, or user feedback sessions, this phase empowers us to refine and optimize without wasting resources. It's a critical step to ensure that the changes we make are aligned with our users' needs and actually provide value to both our audience and our business.

La validación es como asegurarte de que una nueva receta de comida sabe bien antes de servirla a tus clientes. Te ayuda a evitar perder tiempo y recursos en cambios que no funcionan y asegura que lo que haces realmente sea útil para los usuarios y beneficie a tu negocio.

OODA loop is an often used concept in business strategy. It was developed by military strategist & US Air Force Colonel named John Boyd and showcases how in dog fights, pilots use a dynamic 4-step process of Observe, Orient, Decide, and Act to quickly adapt to their enemy's actions & terrain feedback in order to win. This concept is widely used by decision makers beyond military. Business leaders swear by OODA loop to think on their feet, anticipate threats, and neutralize them before they escalate. OODA loop can very successfully applied to UX improvements. Short loops keep product decisions in tune with real-world developments which are not just connected to user behaviour or feedback - but also changes to a competitive UX landscape.

Once we've validated our hypotheses, I use agile methodologies to efficiently implement changes based on the UX data and feedback. Iteration is the key to staying ahead in today's fast-paced, ever-evolving market. By making incremental, data-driven adjustments, we ensure our product remains aligned with user expectations and competitive in the industry. It is really important to monitor the impact of these changes on our UX metrics and keep a close eye on user satisfaction.

The UX metrics help a UX/Product person understand how the user is using and engaging on what frequent basis. These are more directional to understand the business implications but should not be confused with or mixed with the financial metrics. It's important to have an intuitive and easy-to-use UX that will definitely improve the business outcomes but is not always the sole reason for a certain business outcome. UX and Product teams should work hand-in-hand to ensure that both UX and financial metrics(if P/L responsibility) are healthy and raise concerns to leadership if they see a deviation from normal.

La iteración es una parte valiosa del proceso de mejora de productos basados en UX, pero no es una solución única para todos los casos. Es fundamental considerar las limitaciones de cada proyecto y ajustar el enfoque en consecuencia; la innovación y la exploración de nuevas ideas deben ser parte de la estrategia, junto con la iteración basada en datos y la retroalimentación de los usuarios.

To iterate on your UX improvements, take the insights from your data analysis and validated hypotheses, and make targeted design changes. After implementing these changes, gather fresh feedback and re-evaluate the user experience. This cyclical process of design, test, and refine ensures that the product continuously evolves to better meet user needs and expectations.

Leveraging quantitative metrics can provide valuable insights into how users perceive a product. Qualitative methods like user interviews and usability testing can uncover in-depth feedback and pain points. By monitoring and analyzing these metrics and feedback consistently, we can make informed decisions for iterative improvements. With A/B testing and user behavior analytics the team can pinpoint specific UX issues and validate potential solutions.

To learn from your UX outcomes, review the results of your iterations against the original goals, understand the reasons behind both successful and less-effective changes, and document these insights to inform future design decisions. This reflective practice not only enhances current projects but also provides valuable lessons for upcoming endeavours.

Learning from UX outcomes is a continuous journey, one that demands a vigilant ear tuned to user feedback. As Product Managers, it's our duty to be customer-obsessed. Every project is a learning experience, and while there are best practices to guide us, adaptability is key. You begin to master techniques for better user understanding and effective team collaboration. When one project is over, for me it is just the start; it's a chance to improve. Doing a retrospective is invaluable for fine-tuning my approach and iterating theprocesses for even better results. As a Product Manager, it's all about relentless growth and improvement.

Create a centralized 'Learning Repository' where all UX insights and lessons are stored. This becomes a valuable asset for onboarding new team members and serves as a historical record for why certain decisions were made.

I have found that plain old user interviews are an incredibly effective way to determine the user's sentiments on my product. However having the right questions is very important to be able to glean data without biasing the user. There is a wonderful and a relatively short book on this calling - Talking to humans. Helps shape how to have these user conversations.

The UX of Your Analytics Tools Often overlooked, but the UX of the analytics tools you use also matters. If the tool is hard to use, it can slow down your ability to measure and improve the product's UX effectively.

Beta Testing and Asking customer Feedback for every single change in UX is the great way to calculate User experience. Select a few people from the team . And ask them to be a user of the product and then you can measure in an efficient way. We may provide various rewards to team members . who participate actively in this programme.

When considering UX outcomes, it's also important to be aware of changing user behaviours, emerging technologies, and cultural or societal shifts. Regularly updating user personas, considering accessibility and inclusivity, and being open to feedback from diverse user groups can provide a broader perspective. Additionally, fostering a culture of collaboration between UX designers, product managers, developers, and other stakeholders ensures a well-rounded approach to design decisions.

As an SEO specialist who works daily with product managers, I have a few insights to share. Measuring user experience can vary depending on the company's needs, the target audience, and the general behavior of the user. Many people have mentioned that you can measure UX by a variety of methods, but this is mostly focused on you, while the majority of organizations are primarily focused on revenue generation. For them, it does not matter if you created the world's best UX if it does not generate revenue. Therefore, I always say to create something that makes an impact on the users' lives or generates revenue.