You're drowning in real-time data. How do you choose the right statistical indicators for quick decisions?

A lot of people, including senior data scientists, make the mistake of treating some metrics as your goal. The moment a metric becomes a goal, the metric loses its function because no metrics can comprehend the overall health of the business, and people will game it. Think about losing weight. Your goal is to become healthy and maintain balanced nutrition. If your goal is losing 20lbs when your body functions like a company with many orgs, your leg may just want to chop it off to achieve the goal in the fastest way. Express your goal in English. Once you have the goal, define your strategy to achieve it. Once you have the strategy, identify the metrics to measure the success of the strategy. Once you have the key indicators, find proxies.

Is an increase in click-through rate (CTR) good? Not necessarily if it comes from clickbaits. In my career, there are countless occasions where a KPI is increased, but the business is hurt. Thoughtless incentives are typical examples. Remember, metrics are objective reflections, but it's the data scientist's job to infer the true effects of the business from the observed effects. Data scientists shall also find lead indicators that can help prevent problems; find good proxies to make latent problems visible and quantifiable; and run good analysis on the lagging indicators to calibrate whether our decision-making process makes sense and whether our surrogates are effective.

As an Economist by training, I used to love indexes that are composite of multiple important metrics with assigned weights to reflect the tradeoff among these metrics. However, in my career as an Economist in tech and as a data scientist, I discovered that indexes never work. Why? Because metrics are used to communicate, evaluate performance, make decisions, and drive changes. Whenever we have a composite index, we lose simplicity and transparency. Also, the relative weight among different metrics is never stationary. Instead, when we have three different metrics contributing to one index, simply report them all. Everyone will have a better picture of how the business performs, and have a discussion about the tradeoff.

The best time to start an analysis is when you have an intuition. The best time to finish that analysis is 1 second after that. Every second that goes by, the value of the analysis decreases. There are two reasons. First, business changes, data distribution (including customers' preferences, environment, etc.) changes, and people's priorities change. So when a question is asked, it ideally shall be answered right away. The second and much more nuanced reason is that good data scientists develop good intuitions. These intuition, product sense, or business acumen, are vastly valuable because they encompass way more information than the visible data that goes into models. But you hunch will be gone if you stare at the problem for too long.

A newbie data analyst would look at a 30-day rolling average of business metrics, but an experienced one will only look at a 28-day rolling average. The reason is that the day-of-the-week effect is strong, and the 30-day rolling average will introduce unnecessary noises (why did our revenue decrease today? Because we headed into a weekend!) Knowing the meaningful pattern from data requires knowing the domain knowledge. We are not just moving 1s and 0s on a table, we are moving data that has a business meaning and represents human activities. We can't do a good definition of "transaction" without knowing how transactions take place, the discount and rebate, the delivery, and the refund policy.