What I Learnt About Effective Use of Analytics

This post is about analytics, and how I learnt to use it in the context of our app Futurecam.

Why To Use Analytics

Before we get into how to use analytics effectively, let’s see why we’d want to use it in the first place. There are multiple purposes:

1. Understand how users use the app. Unlike talking to users, analytics is aggregated, so smooths out individual biases. You also see what users are doing, rather than what they’re saying they’ll do, which is completely different. You also observe people as they use the app, rather than looking over someone’s shoulder, which makes them self-conscious and changes their behavior, such as being more patient with your app. Finally, analytics also gives you detailed, fine-grained data. For example, how many people switched to a certain page in your app to use a feature. Of them, how many completed it, rather than canceling. And so on. Analytics gives you a lot of data that’s not available in any other way [1].
2. Create a funnel [2] to give you a top-down view. For example, if existing users aren’t getting value out of the app, using marketing to get more users amounts to pouring water into a leaky bucket. Absent this top-down view, your planning and therefore your product will fail.
3. To get quantified learnings. For example, we found that launching a better app icon doubled conversion rate from 13% to 26%. This is critical, since there’s no shortage of advice on the Internet on what you should do, but you can’t do everything. So you need data, which analytics provides.
4. Save time by not enhancing features nobody uses. Or at least not yet, if you have more pressing problems to fix.
5. Identify demographics of target users, for advertising or other purposes. For example, you may find that your users are men in their 30s in developed countries.
6. In addition to all the purposes above from a product point of view, you can also use analytics purely from an engineering point of view. In Futurecam, we have a lot of assertions in the code which, when they fail, log an assertion_failed event to Firebase. This lets us be more aggressive than we would otherwise be — we go ahead and make assumptions, put assertions, and fix problems that occur. If we didn’t have this analytics, we wouldn’t know how our code is behaving in the wild, so we’d have to be more conservative.

Now that we’ve seen why to use analytics, let’s see:

How To Use Analytics Effectively

1. When you use analytics in an app, start with instrumenting your funnel. Which is the sequence of steps users go through to use your app. For Futurecam, the funnel is app store impressions → product page views → app units → app open → session [3] → user tries to take a photo → user takes a photo successfully → in-app purchase.
2. The importance of analytics increases as you go down the funnel. It’s more important to know how many users are able to use your app successfully (whether to take a photo or place an e-commerce order) than to know how many downloaded your app this week.
3. Metrics earlier in the funnel are called vanity metrics. Their main purpose is to boast how well you’re doing, but it amounts to fooling yourself. And your team, if you share vanity metrics with them. So don’t. Don’t put them in your spreadsheets. Don’t even track them except to identify where in the funnel people are dropping out.
4. Start adding analytics to events later in the funnel and work backward, rather than the other way around, because events later in the funnel are more valuable (they’re not vanity metrics).
5. The most important events are those that generate value, either for the user (like photo taken successfully) or for the company (like in-app purchase).
6. One metric you can calculate from this is the percentage of users who derived value from your app. For example, in Futurecam, light trails are meant to be captured at night, on a tripod, but if someone doesn’t, they’re getting a blurry, useless photo. Similarly, users derive value from an e-commerce app if they place an order, not perform other actions like create an account, search for a product, or write a review. If the percentage of users deriving value is low, it means your UX is bad, or that you’re attracting the wrong kind of users for your app, like a doctor downloading Xcode.
7. Another metric is the breadth of usage: With Futurecam, I’d like a user who took a burst photo, a light trail, a long exposure and a timelapse, rather than only one of them. Amazon would prefer if you order electronics, shoes, tea and more, rather than only electronics.
8. A third metric is retention. To calculate retention, measure only users who derived value from your app. Don’t include users who only opened it. If 100 users placed an order on Amazon this month, of which 80 opened Amazon next month, of which 60 placed an order, the retention is 60%, not 80%. And ignore new users in the second month. When calculating retention, you consider existing users leaving, but not new users coming in.
9. Be careful not to log private information, like photos, or thumbnails of them. There’s a lot of concern being raised about Facebook and Google Analytics, so don’t go over
10. With analytics, a little goes a long way.
11. Specifically, logging events without parameters gives you a majority of the value. That is, an order_placed event in an e-commerce app captures most of the information, even if doesn’t include the amount or the number of items.
12. Don’t overdo analytics, logging every piece of information in case the data might give you some insight. This approach can take a person-quarter even for a small app, and can distract you from more important work.
13. One sign of overdoing it is that you’re logging something that always has the same value, like video resolution for an app that always records Ultra HD video.
14. Don’t future-proof your analytics. Log information you need today.
15. Log information so that you get the conclusion you want from your analytics dashboard, rather than having to process it further, say via an SQL query or a spreadsheet. You won’t end up doing such manual steps regularly, no matter what you tell youself. For example, in Futurecam, we have a light_trail_taken_successfully event that’s logged only when the user takes a light trail successfully (at night, using a tripod). Earlier we had a light_trail_taken event that’s logged in any case, with an is_successful parameter. This meant that the Firebase console would not tell us how many users captured a light trail successfully. We had to run an SQL query. But the number of successful captures is the more important piece of information. It comes later in the funnel, and remember that events later in the funnel are more important. Instead, our Firebase console showed less relevant information. So we fixed that by logging a light_trail_taken_successfully event rather than a light_trail_taken event [4].
16. Don’t tie analytics to the UX flow. For example, we used to have a permissions_granted event when the necessary permissions (camera and photo) were granted as soon as the app opens. But we later changed the UX flow to ask for photo permission after capturing the photo. So we had to split the event into camera_permission_granted and photo_permission_granted. If I had to do it again, I wouldn’t couple analytics to the UX so that the former doesn’t need to change when the latter does. I don’t know if this is clear only in retrospect, though.
17. Before solving a problem, validate that you have it, using your analytics. For example, in Futurecam, we ask for photos permission as soon as the app launches. This is not considered a good practice: apps are supposed to ask for permission only when needed. So I was about to change it, but I consulted analytics first, and I found to my surprise that fewer than 1% users were denying necessary permissions [5]. So I didn’t waste time solving a problem that didn’t exist.
18. It’s easy with analytics to mis-diagnose a problem. For example, not enough people were capturing a timelapse. I thought they weren’t going to the timelapse tab in the app, so I made a code change to encourage them to do so. But things didn’t improve. Later I realised that people were going to the timelapse tab, just not capturing a timelapse. So, before solving a problem, try to confirm that it exists, and also that your diagnosis of its cause is correct.
19. Given that it’s easy to solve problems that don’t exist, or solve the wrong root case, don’t act on problems immediately. Mull over them for a week or two if possible.
20. Analytics requires long cycles: you code it up, maybe launch it in your next release after ten days [6], wait for your app to get enough usage, analyse the collected data, take a few days to think about it, code up fixes, verify that they work, take another ten days to launch them, then again wait to get enough usage data, analyse and confirm whether your fixes did work. It’s normal to have to repeat this cycle. So, analytics is something that takes time. You can’t do it on your schedule like “We’ll fix all major problems detected by analytics in April”.
21. Analyse data you’ve already collected before collecting more data, because the former can make the latter irrelevant or less important.
22. Analytics requires a certain skillset: coding, SQL, spreadsheets, analytical thinking, etc. Non-technical people who don’t have these skills can’t implement analytics, at least not single-handedly. But in addition to skillset, analytics requires a certain mindset. You’re not standing by users when they use your app, you can’t ask them questions, and you don’t know what they’re thinking. So you have to draw conclusions from the outside, like a detective. I’ve found many junior engineers to have the skillset but not the mindset to do analytics by themselves.
23. I’ve found that effective implementation of analytics requires discussion with team members, given the varied mindsets it needs, which are hard to keep in mind at once, even if you know all of them individually — coding, SQL, spreadsheets, analytical thinking, product thinking— as in my case.
24. You need at least 20 data points to draw an inference. For example, if you’re experimenting with a higher price, you need at least 20 people to pay. Merely showing a payment screen to 20 people doesn’t count; you need to show it to a thousand people to get 20 buyers. In fact, you need 20 people to pay the lower price and 20 to pay the higher price before you can confidently say whether the higher price increased revenue. Even then, you may not be able to say that the revenue increased 42.7%, only that it increased significantly. But that’s usually good enough at an early stage. Getting it approximately right is critical. The last 20% of optimisation less so. You’re not Amazon, where a 1% increase produces millions of dollars.
25. If you change something halfway through a day, ignore that day’s data in the comparison, since some events in that day correspond to before the change, and some after. Ignore the previous and next day as well, to account for timezones.
26. When you analyse data, do so for the most recent version of the app unless you have insufficient data.
27. We had a permissions_granted event that was logged when the user granted permissions. After some time, we decided to also track users refusing permission. So we added a granted property to the event. When I I looked at the analytics after a significant gap, 100 users had generated a permissions_granted event, so I assumed that 100 users granted permission. Not so, I realised — the name had become misleading. When you find such technical debt, fix it, either by renaming the event to permissions_determined, or having separate permissions_granted and permissions_denied. Yes, it makes analytics harder in the short term, but reduces technical debt and so complexity and confusion a few months later.
28. Sometimes we may change when an event is logged, such as logging permissions_granted only when permissions are actually granted, not when they’re denied. Or maybe a calculation was wrong, which you fixed in a later release. In such cases, rename the event (say from permissions_granted to permissions_given). Otherwise, people doing analysis can easily make a mistake by not knowing that the same event’s definition has changed from version to version.
29. Be careful about data types. Don’t log a number as a string. Or as a float unless you actually need the precision. For example, we used to log ISO as a float, which causes problems because floating-point values are often logged inaccurately, like 1000 becoming 999.99. This causes a query like = 1000 to fail. Everyone doing analysis should remember to change x = 1000 to x > 999.99 AND x < 1000.0. And change x > 1000 to x > 999.99.
30. Be consistent in naming. For example, if you have a photo_taken event, also have a video_taken and timelapse_taken, rather than timelapse_captured. Then you can count how many were captured in total by querying LIKE “%taken”.
31. You can measure the number of events, or the number of users who generated that event. For example, you can measure the number of photos taken, or the number of users who took a photo. Both are valid things to measure, but don’t mix them up. For example, if you want to know whether videos or timelapses are more popular, you can compare the number of captures, or the number of users capturing, but you can’t compare the number of videos with the number of users who captured a timelapse.
32. If you want to measure how many videos were captured for 5s or longer, instead of measuring it as: videos with duration >= 5s / videos, measure it as: videos with duration >= 5s / videos with duration >= 0s. The first one gives a wrong answer if some events have null for the duration. Write your queries defensively, or you may get the wrong answer and not know it.

Also read What I Learnt About Funnels.

[1] This isn’t an argument for not talking to your users. Startups that don’t talk to users fail. Rather, the argument is that analytics is complementary to talking to users.

[2] You could say that this is part of the above purpose — understand how users use the app — but it’s important enough to call out.

[3]

• An impression is when the user sees the app in any way, such as in search results in the app store or “You may also like”.
• A product page view is when the user taps our app entry to open its app store page. Every product page is an impression but not vice-versa.
• An app unit is a download, but re-downloads by the same user, including on a new device, aren’t counted.
• A session is when the user uses the app for at least 10 seconds.
• Notice the difference between trying to take a photo and taking one successfully. Some features in Futurecam require a tripod, or are meant for usage at night, or require recording for 10 minutes. If the user fails to adhere to these requirements, he gets a blurry mess that can’t be called a photo.

[4] Notice that logging a light_trail_taken event with an is_successful parameter captures more information than logging a light_trail_taken_successfully event, since the former lets you find out how many captured unsuccessfully. So it’s a tradeoff between capturing more information and being able to see important (as opposed to vanity) metrics on your dashboard without further processing. I’d prioritise seeing important metrics at a glance.

As another example, we have events like {camera/mic/photos/location/notification}_permission_{granted/denied/restricted} so we can see at a glance in the Firebase console whether enough people are denying permission is a serious problem.

[5] People are denying permissions that aren’t needed for the app to work, like location or notification, which isn’t a critical problem.

[6] On iOS, we use Apple’s phased release, which takes a week, plus a couple of days for Apple review, for a roughly ten-day release cycle.