Product management application case studies for your next application- case study?#1

Recently, I was going through my Google docs looking for a research study plan to share with a couple of innovators I have been supporting and in the process, I stumbled on case studies I did back in 2017 when I was applying for product lead roles. It is pretty common for startups to ask product candidates to work on a case study so that they get a better understanding of the candidate's abilities. This is basically similar to code tests developers do. I am not sure what UI/UX peeps do but I am sure there is something. Out of curiosity, I ended up reading both case studies and I have now decided to publish them here. I believe it might be beneficial to people looking to get into senior product roles but read on even if you are not there yet. Last disclosure — I landed both gigs but I ended up not taking either for different reasons. Instead, I ended up joining Eneza Education as their Head of Product.

I will publish this as two different posts (both were fairly long) and I will remove company identifiable information. I will not edit my answers at all but instead, present them exactly as I answered them.

Without much ado, let us get to it.

Case study 1 — applying for a Head of Product role at a growth-stage logistics startup

1. Assuming you’re a small business, take a look at our current product and suggest key improvements that would help in a product-driven — sales growth and product engagement from existing/new users

As a small business, my biggest worry, especially in a city like Nairobi, is reliable delivery time. Based on my app usage so far, this isn’t guaranteed just yet. The wait time is quite long. I made a delivery request at 1:50 pm and the app informed me that for a direct delivery the earliest I can do it was at 3:00 pm. An indirect delivery could be done at 2;26PM. I tried the schedule feature and it didn’t work as I couldn’t actually get the delivery done as per the time I wanted it scheduled. My focus would be, to some extent, to guarantee delivery time. Of course, this is something not done within the app but more of utilizing riders better. The app suggests that there are riders around me, though. This makes it even harder to understand why there is such a long wait time.

The other section I would spend some time on is ensuring that the communication between the rider and the customer is accurate. In the course of testing the app, there was a situation where the app informed the tester (I used a third party) that the package had been delivered, and yet it hadn’t. I am not sure if this is a training challenge or a technical issue. This can be a challenge for my customers.

Last but not least, I would spend more time on standardizing riders’ behaviors. At times you will find a very good rider and in other situations, one can be borderline rude. As a small business, my customer/supplier won’t know the difference between you and my business. As a result, the experience provided by the rider greatly affects how my customers perceive me. It, therefore, needs to be consistently professional.

2. What metrics do you think we should be measuring in our product and why would they be important to measure?

• Customer acquisition — What are the number of new customers we have acquired in a day/week/month? Which channels are we acquiring customers through (breakdown in numbers)? What is the split between organic and paid channels? How many unique visitors do we get on a weekly/monthly basis? What is our customer acquisition cost? Customer acquisition data will help us know the rate at which we are getting new customers and the best way to grow this
• Retention — it is one thing to acquire customers and it is another to keep them. Filling a bucket with holes is an exercise in futility. It makes sense to plug in the holes as you continue pouring water. On a weekly and monthly basis, I would track our retention rate. A churn (failure to retain) outpacing the acquisition rate will quickly kill the business. A high churn is still a huge problem for any product. By tracking this, we will know if our customers once they sign up to continue using the product
• Engagement — I would track daily, weekly, and monthly active users. I would also track the activity in the product over time. An example of an activity would be the number of deliveries done on a daily, weekly, and monthly basis. Tracking engagement allows me to know if customers actually rely on our product
• Monthly recurring revenue (MRR) — what is the new MRR (new revenue added by brand new customers), Add-on MRR ( revenue added from existing customers using more of our services), Total new MRR (add-on revenue less the churn). Tracking revenue shows the product’s growth from a financial perspective.

There are a couple of other metrics that I will specifically track so that we can use them to improve the product’s UX

• Happiness — how satisfied are our users with the product? Surveys, reviews by customers, sentiments on social media, and other sources can provide us with such information. Combining this with customer care data (number of customer support requests per week/month, average time for first response, average time to close support request) makes it easier to know how happy our customers are.
• Adoption — based on the features that we ship, how long are users taking to have it in their workflow? This data allows me to know whether to continue pushing a feature or quit
• Task success — how long does it take for someone to complete a specific task such as ordering for a rider? What % of a task initiated is completed? This data allows me to identify specific problems users face when carrying out a task and as a result, provides a data-influenced approach to improving the product

3. Product management sometimes means saying NO. When do you say NO and how do you say NO to the team’s product requests?

Product management involves a lot of NOs. I would say no if the suggested change doesn’t fit the product strategy. For any product, there is a limited number of permutations of the product. Blindly saying yes to every request will lead to a product that is bloated. Before you know, it is a product that can allow customers to chat while they are getting their deliveries made! Sometimes, a team member might argue that adding the feature won’t take long or might say that many people have requested the same. There is nothing like a small feature because of the number of hidden complexities. It is important to listen to customer feedback but if it is only used as a guideline you quickly turn into a consultancy company building different features for each client. We want to ship features that all our customers will use.

How do I say no? As a team, it is important that we all understand what our product strategy is. Everyone in the team should know how the roadmap is at least for the next two years. A shared product strategy allows me to say NO with a WHY. It allows me to say NO with confidence. It allows me to explain what we will focus on instead.

4. What would be the process of deciding which new features to build? What would go into your considerations in deciding on what to build?

I would use an ACID test with the following questions:

• Does it fit in our vision? — the ultimate guide to whatever is built in the product strategy.
• Will everyone benefit from the feature? — I am avoiding building a one-size-fits-none product and so features we ship have to benefit all our users. It doesn’t mean they will all use it, though
• How does it affect the existing workflow? Will it improve, complement, or innovate it?
• Does it grow the business? — will it lead to increased revenue? Does it reduce churn? Does it increase stickiness?
• Will it generate new meaningful engagement? — will people use it to achieve something meaningful? How does it affect the other features we have shipped?
• If it succeeds, can we support and afford it? — what does it mean for our customer support team? The dev team? Is it just a quick win or can we support it in the long run?
• Can we design it such that the reward is greater than the effort? — this can be in the long run and not necessarily over the next one year or so
• Can we do it well? — it is better not to build it if we are going to ship something mediocre
• Can we scope it well? — do we have enough information to get a clear understanding of the feature?

For us to build the feature, it would require a YES for most, if not all, of the questions above. Exceptions will only be made in rare situations.

5. How would you get the new features you build used? What would be your process and how would you measure success?

There are people who use the product and so it is important to ensure they are not inconvenienced when a new feature is rolled out. To ensure this doesn’t happen, the rollout will be steps and each step has a key outcome that will decide when to go to the next step.

• Team testing — the immediate team that is working will be the first set of testers. This is the dev team and the product team. The main outcome of this is the identification of bugs. It also allows us to release features as we continue building instead of waiting until it is complete. Being a small team, it will be fast as the team also knows which features just haven’t been built yet. At this step, we are using a copy of live data in a staging environment
• Company testing — the next step is to release the feature to the entire organization without elaborate explanations or tutorials. We would release it to the public this way. It is also a good opportunity to do cheap internal usability testing exercises. Shipping within the company allows us to identify confusing elements within the feature before we ship it to the public
• Private beta — as a product that has been in the market for a while, I would assume there is a small group of users who act as our beta testers. The feature would then be released to this specific group and communicated as a beta version. They would actually use the app and not just test it by simulation. To ensure we make the best of their response (responses have always been advocated over feedback), I would use a subset of the beta testers to carry out usability tests. The aim of this test is to see if users can discover the feature, use it (engagement), see how they use it within their workflow, see how they are using it, and identify any barriers.
• Public release — once the challenges identified in the private beta have been addressed, I would release the feature to all my customers. A huge part of this step is developing the right messaging. The messaging will focus on what the user can do with the new feature

At this point, I will monitor discoverability (how quickly it can be found), engagement (how many users are using it), and adoption (how many users are adopting it into their workflow). The higher each of these is, the better the feature is doing.

6. What are the top 3 products that you think are particularly well-designed?

• KCB banking app — the design and the marketing of the app was pulled off really well. To broaden the reach of the app, the bank made it accessible to both customers and noncustomers. KCB then used all its channels to get both customers and potential customers to download it. The app is also updated regularly showing that the bank is interested in building an app that people will actually use and not just ticking it off their list
• Overview Finance — I have always struggled with the concept of tracking my finances until I came across Overview. While still in private beta, this app has two sticky features. The first one is that it automatically tracks money received or spent via Mpesa or Airtel money. If, for example, I pay for fuel using Mpesa it can automatically log this as a fuel expenditure. This eliminates the need to key in everything. When I need to manually enter information, they have created a very simple process that has the right messaging. It does a small thing for me but a very important thing.
• Uber — everything about the Uber app is amazing. The user experience is top-notch. The frequency at which they keep updating the app is great. For each version, they ship you can see what they have been working on. Uber drivers are approximately 7 minutes away. It is a well put together experience

7. Which applications are very familiar with locally; what drives you nuts about it, how would you fix it; what would your next release look like? What would be the top features?

The Barclays USSD application drives me crazy! Basically, Barclays took their entire banking platform and availed it on USSD. The app allows you to do RTGS or Swift transactions! There are so many steps to complete any given action. If, for example, I want to check my balance I have to go through the following steps:

1. Dial *224#
2. Provide my PIN
3. Select Bank Services
4. Select Account Services
5. Select Bank enquiry
6. Select A/C number
7. Get balance

Each process is almost as long as this.

My next release would use usage patterns to decide what is presented on the menu. I would prioritize items such as withdrawal and balance inquiries as that is generally the way users interact with the app. Having usage data and not using it in any way to improve the app just doesn’t make any sense. Next, I would conduct a UX research exercise to understand what users are trying to accomplish using this channel. This will advise the options presented to them and how they are presented. From experience, the design of the app didn’t factor in the nature of the channel (USSD) being used.

This was a fun case study to work on and I hope it will help you tackle your next interview.