Converting bits into dollars: Why and how data is able to generate business value

A post on why and how data is able to generate business value.

Introduction

Data is important because it encodes information. Information is the quantity that enables understanding (knowledge) and prediction. Prediction differs from knowledge as it implies a future and probabilistic temporal component. Information can be measured by ‘Information Entropy’, with units of ‘bits’; it's a quantifiable entity. Different data, can encode different information. Let's look at a few examples to illustrate these topics.

'My name is Bob’ in French and ‘my name is Bob’ in Japanese is different data/encoding, but the same information. In a second example, we note that the entropy of French is low (ie it is fairly predictable), while the entropy of Japanese is higher. Likewise the entropy of signed financial market data returns is much higher than the corresponding measure for volume. In a third example, a time series may show cyclic information when decoded at one resolution, and trend information when decoded at another resolution. In a final example, if a system has two users each incentivized to predict future system state, then the user with the informational advantage will win - the advantage could come from receiving the same information first, or receiving more information.

It may surprise readers to know there is a well-established duality between information theory and system growth [1]. This can be applied to the growth of a business: For a business which uses information to generate revenues, the sum of the growth rate and the entropy rate is constant. Put differently, information can be used to grow revenues. [See Appendix A].

The mechanism for this is simple: A data generating process (DGP) creates data. The data contains stationary patterns that can be recognized, from which a prediction about the future can be made. By being able to predict the future, we can generate revenues/profit. Let's take two basic use cases,

• Consumer generated data. Customer experience is about the customer saying (directly or indirectly) what they want to spend money on in the future. When repeated across enough customers, the data generated contains patterns. By detecting these patterns and then predicting our customers needs, a profit can be made. For example, a customer states what they like/ don't like about a product/service, that data is used to innovate a new and improved product/service that the customer likes more. This mechanism is referred to as the 'data flywheel' [2]. The flywheel creates a self-reinforcing competitive advantage that is nearly impossible for competitors to replicate, because they don't have the customer generated private data.

• Marketplace generated data. A tradable asset generates historical price data. This data contains patterns as a result of the behaviour of the marketplace participants. Again if we can detect these stationary patterns, we can make a prediction about their future value. If we know what the future price of an asset will be, we can profit from that information.

For both these use cases, we have the concept of intrinsic and extrinsic data. Intrinsic data is the data generated by the system that is to be predicted. Extrinsic data is data generated outside the system that is to be predicted. For example, for a food shop, intrinsic data might be customers might the number of customer queries asking for ice cream, while extrinsic data might be the weather temperature. By increasing the amount of information used to make the prediction, both the intrinsic and extrinsic data can be used to help improve prediction accuracy.

Financial Services Industry Example

Take the example of the trading fund WorldQuant , who detail how they use data-driven prediction to generate revenues [3-6].

Data: “A data group scours the globe for interesting and new data sets, including everything from detailed market pricing data to shipping statistics to footfall in stores captured by apps on smartphones”. “If we could buy,?consume,?create,?web scrape,?more data than anyone else,?we could create more alpha,?find more?opportunities”. “In 2007 we had two data sets - today [2022] we have more than 1,400.”

Data Processing: A prediction algorithm takes data as the input, processes it, and outputs an ‘alpha’ (aka predictor, signal, etc). The 'Fundamental Law of Active Management' states Sharpe is proportional to √N, where N is the number of alphas [7]. Hence, as long as an alpha is not perfectly correlated with another, then there is value in adding it to the set. ‘In 2010 we were producing several thousand alphas per year. By 2016 we had one million alphas. As of 2022 we have multiple millions of alphas’. And there is a stated ambition to get to 100 million alphas. While traditional quant finance mandates the importance of an economic rationale behind an alpha, the data-driven approach is lead purely by the patterns in the data.

Combination: Once alphas have been produced, they are intelligently merged together in a time-variant manner. Examples of signal combination methodologies include Mean Variance Optimization and Bayesian Model Averaging. “No?one?alpha is important. Our edge is putting things together, it’s the implementation.” “The idea is that with so many “alphas,” even weak signals can be useful. If counting cars in parking lots next to big box retailers has only a tiny predictive power for those retailers’ stock prices, it can still be used to enhance a bigger prediction if combined with other weak signals. For example, an uptick in cars at?WalMart?parking lots—itself a relatively weak signal—could combine with similar trends captured by mobile phone apps and credit-card receipts harvested by companies that scan emails to create a more reliable prediction.” ?

The automated process of data ingestion, processing, packaging, combination, and prediction, is referred to by World Quant as their ‘alpha factory’.

Conclusions

Lessons for the wider business community are,

1. Data holds information, and information has a fundamental link to revenue generation. All business leaders should understand this statement as this is the basis of why "being data-driven" matters.
2. Collecting large amounts of disparate data, extracting information, and stitching it together enables patterns to be predicted. To succeed in doing this in the modern world, typically large data volumes are needed and these require automatation.
3. Applications for the use of information in business are boundless. Examples include, next best action with a customer, more effective marketing, and new product design. ?

One possible vision for the future is that all businesses become miners of data patterns and generators of signals, enabling optimization of their revenue generation factors.

References

1. Kelly, J. A New Interpretation of Information Rate. The Bell System Technical Journal. 35(4): 917-926. 1956.
2. Anonymous. The AWS Data Flywheel, 2023.
3. Igor Tulchinsky . The Power of Exponential Thinking. Dec 2022.
4. Igor Tulchinsky . Finding Alphas: A Quantitative Approach to Building Trading Strategies. John Wiley & Sons. 2019.
5. Igor Tulchinsky . Performance v. Turnover: A Story by 4,000 Alphas. Arxiv. Sept 2015.
6. Hope, B. With 125 Ph.D.s in 15 Countries, a Quant ‘Alpha Factory’ Hunts for Investing Edge. Wall Street Journal. 13 April 2017.
7. Grinold, R. The Fundamental Law of Active Management. J. of Portfolio Management. 15(3): 30-37. Spring 1989.
8. Shannon, C. A Mathematical Theory of Communication. The Bell System Technical Journal, 27:379–423, 623–656. July & October, 1948.

Appendix A

This links back to Claude Shannon's 1948 fundamental information-theoretic Capacity Theorem [8]. The theorem states that any communication channel can be assigned a property termed "channel capacity", which acts as a constraint for the reliability of information transfer. In [1], the author shows the growth rate of such a system is the capacity of a hypothetical noisy channel, over which the system is getting information that distinguishes its probabilities from those of the wider system.

In the context of this article, the 'system' is a business, the 'wider system' is the business's competitors, and 'probabilities' is the information (data) being used by the business to make business decisions.