Reflection on UX Research with Quantitative Data Sources!

User Experience research is pivotal in understanding how users interact with digital products and services. It involves gathering insights into user behavior, preferences and needs to inform design decisions and improve the overall user experience. While qualitative methods like interviews and usability testing provide valuable insights into user motivations and experiences, quantitative data sources offer a different lens to understand user behavior.

Quantitative data in UX research refers to numerical data that can be measured and analyzed objectively. This includes click-through rates, conversion rates, time on page, and other behavioral data collected through tools like analytics platforms, surveys, and A/B testing. Quantitative research methods complement qualitative approaches by providing scalable insights and enabling data-driven decision-making.

Self Assessment

Data Analysis - 3/5

Tableau - 0/5

UX Design - 3/5

I was not good at analyzing huge quantities of data, it was always intimidating to me. However, I took this up as a challenge upon myself and learned with the process. I came across a course from the Human-Computer Interaction dept of CMU, UX Research With Quantitative Data Sources (05-497) by Prof. Raelin Musuraca.

As I am making a pivot to be a Product Manager, analyzing user behavior is a crucial part. I found the combination of Product design and drawing insights from user analytics to be interesting.

Goals set

I set up goals for this course that I wished to learn. 3 things I expected to learn after this course :

1. Data Visualisation from Tableau
2. Leverage user data to extract valuable patterns and insights crucial for analyzing and refining product design.
3. Dealing with large volumes of data and not getting overwhelmed.

What does success look like to me?

Confident to draw meaningful trends from data

What does failure look like?

Get too restless with data volume and not being able to analyze

3 pitfalls that could sabotage my success

1. Not knowing the right set of tools that make life easy to visualize data
2. Overthinking and complicating data volumes
3. Overthinking and complicating data volumes

Exploring Tableau

To overcome my pitfalls and get started with my goals, I went through the course catalog of Tabluea offered by the course curriculum. I gave myself enough time to learn the basics. It was very much fun to see how easy it was to analyze and visualize huge quantities of data with just a few clicks and formulas. The ability to manipulate data and create compelling visualizations has not only expanded my skill set but also opened up new avenues for innovation and problem-solving within the realm of design. I am excited to apply my newfound expertise in Tableau to drive informed decision-making and ultimately create more impactful and user-centric products.

Project Sponsor

Analyzing Skeema's First Product Launch

Our Client: https://www.skeema.com/Skeema is a startup out of Carnegie Mellon University, HCI dept, that defines the way people manage their tabs.

Pain point of hoarding tabs:

1. Frustration?when looking for the tab you need
2. Distraction?from accidentally navigating to unrelated tabs
3. Anxiety?from seeing all those open tabs just sitting there

The evolution of understanding regarding Skeema's business has progressed through several stages.

1. Initial Understanding and Hypotheses Formation: The initial phase involved identifying user pain points and formulating hypotheses about the product's effectiveness in addressing these issues. This stage was crucial for setting the direction of the project and focusing on specific areas such as user frustration with finding tabs, distractions, and anxiety from open tabs.
2. Data-Driven Hypotheses and Testing: Moving from assumptions to evidence-based insights, the project involved collecting data to test these hypotheses. This included user surveys, engagement metrics, and A/B testing to validate whether the product effectively reduced user anxiety and improved engagement through better tab management.
3. Evaluation of Market Position and Strategy Adjustments: As the project progressed, understanding the product's position in its lifecycle was essential. Analyzing growth rates and market saturation helped in identifying the need for strategic adjustments, such as when to introduce new features or versions to stimulate growth.
4. Understanding Distribution Channels and Market Dynamics: Further analysis focused on the effectiveness of different sales and marketing channels. This involved studying how various channels contributed to user acquisition and engagement, which is vital for optimizing marketing strategies and budget allocations.
5. Ongoing Feedback and Iterative Improvement: The project emphasized the importance of continuous feedback from users to refine the product. This approach helped in adapting to user needs and expectations, ensuring that the product development was aligned with the actual requirements of the target audience.
6. Long-term Growth and Sustainability Planning: Finally, the insights from the project guided long-term planning, considering factors like the potential decline phase and strategies to maintain user engagement and market presence. This involved understanding the broader market dynamics and planning for innovation to keep the product relevant.

I had several questions in mind.

1.??What JTBD of users is Skeema trying to solve?

2.? Effectiveness of Skeema in Solving User Pain Points

3. What are the users' expectations from Skeema?

4. What does the growth rate of Skeema look like?

5. What Sales and Marketing channel does best for Skeema?

I used my Tableau data visualization skills to go through the data provided by the sponsor.

Jobs to be Done

People's Favourite

People's Favourite

The weekly user growth rate of Skeema

At this stage, I was pretty confident in exploring the data and drawing visual insights from it.

Hypothesis

A hypothesis is a belief, based on evidence, that something could be true - and can be validated with an experiment. It is an IDEA of what might work better for the customer and could be a POSSIBLE SOLUTION!

Initially, my hypotheses were very broad across many verticles including.

Hypothesis 1: Skeema Reduces User Anxiety Related to Tab Management

Hypothesis 2: Marketing Channels Significantly Affect Skeema's User Acquisition

Hypothesis 3: Higher tab management activity correlates with increased user engagement

I was able to answer a few of my original questions, - What JTBD of users is Skeema trying to solve, whether Skeema was successful in solving these pain points in its first launch, and users' expectations from Skeema, using the data provided.

An interesting finding was regarding the S curve. The product would have reached the Decline phase/Saturation phase in April 2023. However, in May 2023, the team started running ads via Facebook and Instagram which increased the lead conversion rate. This caused the growth from 4000 to 6000 weekly users.

How I refined my hypothesis -

It required a detailed examination of the provided data, along with a thoughtful consideration of the potential relationships and patterns within it. It involved identifying trends, correlations, or inconsistencies in the data that could lead to actionable insights.

The template provided - "Based on [this data/findings], I believe that if we did [hypothesis], then we can achieve [desirable outcome]" - gave a clear structure to how the hypotheses should be articulated. This helped ensure the hypotheses were specific, testable, and directly connected to the desired outcomes.

Explicitly identifying the "leaps of faith" or intuitive assumptions underlying the hypotheses was a valuable exercise. It pushed me to critically examine the rationale and potential risks associated with each hypothesis, rather than just proposing ideas at face value.

This assignment highlighted the importance of validating the hypotheses through user feedback and data analysis, rather than just proposing ideas. Past work was more focused on the ideation phase without as much consideration for testing and iteration.

Comfort Level with Hypothesizing:

Creating hypotheses based on data can be a smooth process when one has a good understanding of the dataset and the relevant field. However, I found it somewhat challenging, particularly when the data was ambiguous or incomplete. My approach was to find a balance between drawing insights from the data and applying domain knowledge. Sometimes, this required making educated guesses about user behaviors and how they might respond to new features.

Leveraging This Activity in Future Work:

I see the process of developing hypotheses based on data and user insights as extremely beneficial for future projects, especially in areas like product development, prioritizing features, and addressing complex problems. Grounding ideas in data and clearly defining assumptions and expected results makes it easier to assess, experiment with, and refine potential solutions. This methodical approach ensures that efforts and resources are directed toward the most promising and impactful initiatives.

After considerable evaluation, I started exploring one hypothesis that caught my attention:

Solution and Metrics

Periodic reminders will increase the utilization of pre-grouped tabs

Out of 2550 users who grouped tabs, only 475 people went back and opened tabs from pre-grouped categories. This suggests that people forget the groups created previously.

3 Potential Feature Designs

Metric Selection and Data Gathering

Choosing the right metrics was essential for measuring the effectiveness of the solution. Metrics were established to evaluate the effectiveness of the solution.

1. Engagement Metrics: These included the number of tabs managed per session, the time spent on tab management, and interactions with specific features.
2. User Satisfaction: User feedback and surveys helped gauge satisfaction levels, focusing on aspects like ease of use and anxiety reduction.
3. Adoption Rates: Tracking the adoption of new features among users provided insight into feature effectiveness and user engagement.

This experience underscored the importance of aligning solution development with clear, measurable objectives and iterating based on validated data. Grounding the solution in evidence not only ensured that it addressed user needs but also made it easier to refine and improve it continuously.

Key Takeaways

Throughout the project, I've gained invaluable skills that will undoubtedly enhance my professional toolkit for future endeavors. Among these, data-driven decision-making stands out as particularly pivotal, as it empowers me to ground strategic decisions in solid evidence, enhancing both the efficiency and effectiveness of outcomes. The practice of hypothesis testing has also been instrumental, fostering a rigorous analytical approach that I can apply across various contexts to validate ideas and innovations. Moreover, my enhanced ability to integrate agile development practices will allow me to adapt swiftly to changing project requirements and stakeholder feedback, ensuring that solutions remain relevant and impactful. Additionally, honing my communication skills through regular presentations and stakeholder updates has not only improved my ability to convey complex information clearly but has also bolstered my confidence in leadership roles. These competencies, combined with increased adaptability and resilience developed through navigating project challenges, form a robust foundation that will support my continued professional growth and success in future ventures.