Can AI Pick Stocks? Should It?

For this week’s “Transformation Tuesday” post, we invited Sanjay Batra, Chief of Staff, Corporate, External & Legal Affairs, to share his views on the impact AI is having on financial markets.

If you’ve been wondering whether AI can be applied to stock trading, you aren’t alone. Little noticed by the general public and mostly ignored by the mainstream financial media, the application of artificial intelligence (or machine learning) to the trading of financial assets has in just the past few years become a vast and rapidly expanding field. This is a development that General Counsels and compliance professionals in any public company cannot afford to ignore.

Consider:

• Dozens of academic papers are being published with titles like Deep Learning for Event-Driven Stock Prediction, Deep learning networks for stock market analysis and prediction and Stock prediction using deep learning;
• Conferences on such topics as The Rise of Machine Learning in Asset Management are springing up;
• An academic textbook published last year on Advances in Financial Machine Learning by a principal at a leading quantitative hedge fund is already regarded as a “must” for Wall Street quants;
• While many firms of all sizes up and down Wall Street are known to be experimenting with AI, a number of hedge funds have already set up shop explicitly based on the premise of AI-powered stock picking and trading strategies, and their performance as tracked by a dedicated index already appears to be rather good (see chart).

Eurekahedge AI Hedge Fund Index 2010-2019

There are several broad approaches to using AI in investing:

1. Use AI to analyze market and company fundamentals, then use that to make stock picks.
2. Use AI to analyze short-term trends in the market, i.e., guess what other traders are doing and profit by anticipating their next moves.
3. Use AI to manage portfolios of assets rather than just trying to predict single stocks.
4. Use AI to optimize trading and investment back-office operations.

Developing clever software to make stock picks of course is not a new idea. Algorithmic trading long predates the rise of modern AI, having been around since at least the 1990s. The idea is simple: replace the labor-intensive number crunching, slow decision-making, and excess emotion of human trading with an automated rule-driven process.

In recent years so-called high-frequency trading (or HFT for short), which is a variant of algorithmic trading, has come to dominate transaction volumes on the world’s stock exchanges. HFT uses software to sift real-time streams of market data for specified patterns that trigger pre-planned buy or sell actions. The sought-for patterns can be anything from crude follow-the-trend rules to sophisticated statistical models. As its name suggests, HFT depends on high-speed automation to get in and out of the market quickly, typically in much less than a second. HFT by its nature seeks very high volumes of low-margin trades. It is thus at the opposite end of the spectrum from classic long-term buy-and-hold strategies.

In a real sense, the use of AI to guide investment decisions is just an updating of algorithmic trading with new methods. AI replaces—or more often augments—conventional statistical and rule-based algorithms with the latest breakthroughs from machine learning, in particular the advanced pattern-learning algorithms known as deep learning. AI can in principle be used to identify potentially profitable investments anywhere on the time spectrum of trading, from sub-millisecond high-frequency trades driven by real-time pattern recognition to multi-year bets based on careful analysis of company fundamentals.

But what concretely does an AI do to identify what it thinks is a good investment? Although they come in infinite varieties and combinations, there are really just two fundamental things it can do: analyze words, or analyze numbers. Stocks and other tradable financial instruments come with vast quantities of both these things attached, and they are therefore ideal targets for AI.

Deep learning models (the most influential tools of modern AI) use high-dimensional vectors to represent significant trends in numerical data and the meaning of words and documents. These models are trained by being fed many examples that exhibit the desired patterns, and are then unleashed to search for similar patterns in new, previously unseen examples.

For instance, an AI model that has been fed natural language documents such as financial statements, press releases, news stories, and tweets about a set of companies along with their past stock market performances can be trained to make predictions about future stock prices. The time scale of these predictions can be short or long. A momentum-oriented HFT AI might scan newswire stories, tweets, and market data feeds in real-time to make instant trading decisions, while a fundamentals-driven AI might analyze quarterly financials, company web sites, and broader news sources to place long-term bets.

It would be wrong to suppose that AI can only crunch hard numbers or definite facts. On the contrary, deep learning methods are especially good at evaluating subjective human sentiments toward stocks and economic events as expressed in tweets or analyst reports. Twitter has even built a profitable side business in selling its real-time “firehose” of 500 million tweets per day to firms that specialize in AI-based sentiment analysis.

You might also think that investors who deploy AI will still want to leave room for human intervention in the limit. But this isn’t necessarily so. With high-frequency training there simply isn’t time for human insight to operate. The most HFT traders can do is “backtest” their AI models against historical or synthetic market data before unleashing them on the real market. With longer term strategies, of course, a mixed approach remains possible. No one knows for sure what the AI-oriented hedge funds are doing in the privacy of their own trading rooms. One imagines that most employ algorithmic risk management strategies backstopped by human judgment. But some of the AI funds openly proclaim that their goal is to run a truly lights-out operation. The chief scientist of one such fund recently remarked, “If we all die, it would keep trading.”

Lights out or not, the bottom line is that nothing guarantees AI stock pickers will make more—or even as much—money as their human counterparts. AI doesn’t really do anything that human investors can’t do. But it can do these things much faster and at much greater scale. The potential to reap great gains or wreak great havoc seems equally clear.

While AI has by all accounts taken Wall Street by storm, investors are understandably reluctant to disclose solid data about how well these new algorithms are performing. And no matter how smart the algorithms are, the reality is that competitive advantage is likely to be fleeting. When every trader on the street is armed with the same state-of-the-art deep learning algorithms, has access to the same market data, and uses the same high-speed trading platforms, winning in the long-term will still come down to human ingenuity, luck, and the constant pursuit of the next quantum of advantage.

AI trading is also beginning to raise significant compliance questions. If AI can detect trading patterns that forecast other traders’ strategies, then it may also be possible for cheaters to create “adversarial” trading patterns that momentarily trick other traders into thinking the market for a particular stock or derivative is heading in one direction when it’s actually heading in the other direction.

Adversarial AI is a well-known problem and emerging area of concern for regulators in multiple areas of AI, including healthcare as well as stock trading. One possible safeguard for financial markets is the vast new forensic database of market transactions known as the Consolidated Audit Trail that the SEC ordered market participants to develop in the wake of the 2010 flash crash caused by HFT algorithms. The CAT has been delayed by numerous technical challenges and squabbling among the stock exchanges, but it or something like it is going to be essential if financial market regulators are to have a chance against adversarial AIs that can strike in microseconds. As always, regulators will have to balance the indispensable requirement for fair and orderly markets against the inestimable value of innovation.

Microsoft has published a book about how to manage the thorny cybersecurity, privacy, and regulatory compliance issues that can arise in cloud-based Digital Transformation—including a section on artificial intelligence and machine learning. The book explains key topics in clear language and is full of actionable advice for enterprise leaders. Click here to download a copy. Kindle version available as well here.