Measuring the value from data science

In my recent post “The Unicorn in the Attic” I made a claim about how much profit my team had made for my employer, JLR. For the sake of blowing my own trumpet a second time, I said we had made ￡100 million in each of the previous 3 years. And I guess I sounded pretty pleased with my team (and by implication, myself).

Well this wasn’t good enough for some readers. They wanted to know how we came up with that number and observed that it can be pretty hard to identify the treatment effect from an analytics intervention.

And that’s a fair point.

This is a note on how practically we think about the problem. There is a big statistical/econometric literature on measuring treatment effect and this note won’t add to it. If it feels a bit wooly and anecdotal as a result, I apologise. Think how long it would have been if rigorous.

The first point is that measuring the profit impact of your work is worthwhile: it gives you direct feedback on whether what you are doing matters; and it focuses the team on delivering work that will be used.

But it can also limit the vision of the team if it is the only thing that is thought to matter. There are many useful applications of analytics which may be hard to link to profit, either because their effect is too diffuse or because the payback is in the following accounting period. You need to make sure that measurable profit is only part of your performance review.

Measuring value generation.

Measuring the incremental profit from an analytical intervention is made up of three components: Identifying the decision through which your analytics drove an operation change; Comparing to the counterfactual; and apportioning the incremental value between the various business functions involved in delivering it, only one of which is analytics.

Link to action

For the analytical work to drive profit, it must be clear that the analytics was used to change something operational that resulted in performance improvement. If your analysis provides interesting insight that motivates a manager to look into a problem, then there is a weak link to profit. If it is able to identify the exact intervention that is needed, and that intervention is implemented then the link is strong. Obviously if your work isn’t actually implemented in the business it can’t really be said to have had any value at all. So make sure you can identify what that link to action is at the start of the project.

The Counterfactual

The second challenge is to compare what has actually happened to what would have happened had no intervention taken place. This is hard because the counterfactual itself, by definition, cannot be observed.

The counterfactual can be estimated in a number of ways. In academic work and medical trials they use the randomised control trial. This gives you the correct answer, but it is expensive and takes a long time. In business we can't wait around for perfect answers. And whereas in a medical trial the measurement of the impact is the central result of the work, in data science it is often just the byproduct, so long as you aren’t materially destroying value. So you are going to have to make do with an imperfect approach.

Perhaps the most efficient way to estimate it is to use the same model you used to predict the outcome of the intervention to predict the outcome in the absence of intervention. If your original prediction turns out to be similar to the observed outcome, then your counterfactual gains credibility too.

Alternatively, you can exclude a proportion of your sales from the intervention, and treat that as a control sample. Of course you need to worry about all sorts of issues like selection bias here, but in theory it will help you learn what works. Unfortunately you will also need to persuade your business colleagues why, if you are confident in your analysis, you don’t want to apply it as widely as possible. So that you can calculate your bonus better is unlikely to be regarded as an acceptable answer.

One factor that will confound either of these approaches is endogeneity. It is hard to infer causality because of the interplay of supply, demand and price. Did you sell more units because you put more units in the market, or did you put more units in the market because there was demand. In the car industry we are helped here by the time lag between purchase decision and delivery. This means that the feedback loop between supply and demand is muted and the arrow of causation is generally more clear than with fast-turning goods.

Either way, your claim to link analytical interventions to results will become immeasurably easier if you can trace individual units. Again, the car industry may have an advantage here because we sell relatively few high-value items in a year, and we can trace them by a unique identification number. So I can see the effect of individual actions on individual units. Similarly, where an operational intervention can be linked to an individual role (such as the automation of an activity to a specific headcount reduction) the profit narrative is strong.

If you want to show the tangible value generation of your work, avoid working on projects where causation is hard to show. So working out the value driven by advertising, brand building and PR is best left to the marketeers: No one is going to give the analytics team much credit in that arena.

Apportionment

Once you can show your counterfactual, the last step is value apportionment. This is where you determine how much of the value improvement should be allocated to the business (who implemented the change) and how much to the analytics team (who showed the business what to do). In the end it will come down to a horse trade: it’s pretty difficult to introduce much science. My recommendation is to agree to the % up front, before it is known how successful the work will be. In hindsight your colleagues will either decide that your work was pretty obvious, or they will claim that they added most of the value, somehow. If the impact of your work is large, you may find your stakeholders will try to get away with under attributing on the basis that a lower proportion is still a large-ish absolute number. Get it agreed up front and in writing.

Here are some thoughts on measuring the impact of interventions, organised by the line on the income statement they affect.

Volume of Sales

You can increase profit by selling more units at the same margin. Often this is where data science focusses because it is well suited to machine learning. Recommendation engines are a nice example of this; by helping a customer choose what to buy, they will buy more. Unfortunately, these are often the hardest applications to link analytics to business performance. Often uplifts are relatively small in percentage terms, and the underlying sales profiles can be periodic and volatile, so showing a statistically significant improvement can be difficult. If you want to get recognition for your work, the trick is not to focus on increasing sales volume, at least to begin with.

One way to improve profit in an auditable way through affecting volumes is to stop selling loss-making items. Options may be loss making because the cost of procuring and fitting them may be greater than the customer’s willingness to pay. Causation in this case is clear, although the magnitude may be up for debate. Unfortunately the size of the prize can be small. Let’s hope that your business isn’t doing too much negative margin business in the first place.

We identified that certain vehicle trims were unprofitable in certain markets. By removing them from sale, even without replacing them with alternatives, we were able to show a profit.

Price

Another driver of revenue that can be addressed using data is price. But again, it is not always easy to link a price intervention to a profit increase. In anything but inelastic products, a change in price will drive a change in volume. Again if you assume that current prices are already close to optimal, allowing for only small changes, it may be hard to understand the link between the two. The only way to be sure is to keep a control sample back from the price intervention (to the extent that fair treatment and regulation allows) but that can be burdensome. One time that you can be sure of the effect of price increases is if you are at the limit of production capacity and so there is unmet demand in the market. In this case, volume is held static, and any change in price can be traced directly to revenue.

Product mix

The sales-line value driver for which it is easier to evidence impact is product mix. If the margins on high-, medium- and low-spec products are different, then upselling from low margin to higher margin segment will increase profit without affecting volumes or prices. In addition, proportions can often be more stable than total sales volumes, so it is possible to calculate the counterfactual with some confidence. Analytics can be a powerful tool to direct the sales force to upsell. Since the volume is unaffected by the intervention, it can be taken as exogenous, and the profit impact is simply the difference in margin.

My team has had some success in persuading retailers to order higher spec stock vehicles. Since one can compare the original retailer orders with the orders eventually placed, the difference can be booked as profit due to analytics.

Another example of a process which can be directly linked to margin, was when we found that certain features of our ordering process resulted in a disproportionate number of cars going to countries that bought lower spec vehicles. By changing the process we were able to increase the margin. You can see that the instrument here (the ordering process) had a direct effect on margin but can reasonably be expected to be orthogonal to the balance of supply and demand in a country, so the link between intervention and outcome can be established.

Unit Margin

Perhaps the line item that is most easy to link to analytics impact is variable cost. Assuming that any reduction in unit cost doesn’t result in a decline in quality or customer preference, then variable cost improvements flow to the bottom line, and it should be possible to link savings to specific sales units.

An obvious example of this is where the cost cannot be attributed to the quality of the vehicle, for example a logistics cost. One notable success we had was to identify where variations in the way we complied with emissions legislation was inflating the taxation payable on a vehicle. By realigning the vehicle specifications more uniformly with the legislation, we were able to manage the tax more efficiently.

We had some considerable success by analysing where similar functionality was being provided at a different cost in different vehicle ranges. So if a similar steering wheel was more expensive in a Land Rover to a Jaguar (this comparison was hard using our core systems) then you could reduce materials cost through standardisation, and this unit cost reduction can be traced through to profit quite easily.

You should bear in mind that if reducing cost requires changes to design, purchasing agreements or production, it may take a while for them to feed through, so although unit cost reduction is easy to demonstrate, it may fall into next year’s profit contribution.

You might argue that a permanent reduction in unit cost should be credited to your team indefinitely, year after year. That’s going to make you unpopular, and you’ll probably just find that your profit target goes up to compensate.

Overheads

When analytics identifies fixed cost savings, it is relatively easy to show the link to the income statement, but it may be harder to prove that the cost has not simply been shifted to another part of the organisation, or simply deferred. Generally for a cost saving to be made sustainably, an activity has to stop, and that is harder to effect than ending a procurement contract. My team built a tool to identify and evaluate one-off costs in the business which led to significant savings in a short period of time, but we only claimed the small proportion of spend that was obviously unnecessary.

Investment

Cutting capital expenditure is often an easy way to save cash, but it is particularly difficult to know what the impact will be on profit because returns from investment are uncertain and tend to be spread over a number of years in the future. My rule of thumb is that if your organisation does not have a track record of good returns on investment then cutting capex is probably a good idea. If the average return is bad, and your analysis identifies the lower end of the return distribution, then that can be counted as a real saving. For example, we were able to show that certain parts are engineered and tooling is made, but these parts are almost never ordered. Stopping the engineering of those parts is pretty much a risk free exercise, and I would allow it towards a P&L target (even if strictly speaking it isn’t a P&L item).

Working capital

Conversely reducing working capital should not be counted towards a value target because it is easily reversible. That doesn’t mean it isn’t valuable; My company was able to survive the Covid lockdown because of cash raised by a large inventory reduction exercise that was put in place the year before. It is tempting to try to value this cash at the cost of borrowing (which is not much at the moment) or at the cost of capital (but a company should be wary of term mismatch - working capital is liquid, but capital expenditure repays over a number of years).

We have experienced two cases where working capital reductions drove real value. Firstly, working capital can be a source of funding when other sources are unavailable or prohibitively expensive. Secondly, when working capital is physical, such as inventory, it might have a knock on effect on prices. We were able to drive a significant reduction in customer incentives by using the Covid shut downs to reduce the number of cars on dealer forecourts. It turned out to be a big number of which we are proud, but sadly we don’t report it on the team’s P&L because it isn’t possible to establish direct causation. 

So there you have it. No single formula. Apologies.

But think about this stuff, especially if you or your team are new to a business. The credibility that you can build by having a clear auditable link to value will make it much easier to establish analytics as a powerful tool at the centre of your commercial strategy.