From Leader to Laggard: Four Areas Machine Learning is Disrupting Wall Street

Well-funded financial institutions are in a perpetual tech arms race, so it's no surprise that machine learning is shaking up the industry. Investment banking, hedge funds, and similar entities are employing the latest machine learning techniques to gain an edge on the competition - and on the markets. While the reality today is that machine learning is mostly employed in the back office--for tasks such as credit scoring, risk management, and fraud detection--this is about to change dramatically.

Machine learning is migrating to where the action is: financial market trading. Once leading-edge Wall Street platforms that companies invested many millions in are soon to become obsolete due to machine learning. Understanding how this disruption will evolve and why it matters is key to navigating the opportunities ahead. 

1. Algorithm Trading 

Algorithmic trading now dominates the derivative, equity, and foreign exchange trading markets. These trading strategies can be complex, but the essentials are straightforward: program a set of rules that takes market data as input and apply basic models (10-day moving average) to generate an automated trade workflow. Over the years, these strategies have moved beyond simple time-series momentum and mean revision models to more exotic name strategies like snipes, slicers, and boxers.  Evolved over decades, algorithm trading has replaced much of the manual trade order flow with faster static rules-based strategies. Nevertheless what was once cutting edge is now an inherent disadvantage. Static rules, no matter how complex, may work well in relatively stable markets but can’t react to evolve rapidly changing market conditions. 

A machine learning algorithm's clear advantage is it learns from experience and is not static.  Employing massive datasets and pattern recognition, these algorithms produce models that learn from experience and are orders of magnitude more powerful than old-school algorithmic trading models. Decisions on how and when to trade will be made in some cases by using multi-agent systems that can act autonomously. At some point, these static algorithms will be no match for more nimble machine learning algorithms.

Why it Matters: Reskill and Upskill

Companies that make use of algorithm trading need to reskill or risk getting left behind. In a winner-take-all market, companies employing only slightly more advanced techniques like machine learning will continuously win a bigger share of the market. In addition to machine learning, businesses should expect an increased demand for data engineers, data scientists, MLOps specials, and others that can handle this sophisticated workflow.

2. High-Frequency Trading Agents

High-frequency trading (HFT) is the flashy cousin of algorithmic trading. Employing similar rules-based models, or even predictive analytics, these strategies operate at a much more rapid pace, completing hundreds of stock trades in nanoseconds versus longer time range algorithmic trading strategies. High-frequency trading also relies on massive hardware and bandwidth infrastructure investment that often requires system colocation next to major exchanges. Given its sophistication, only 2% of financial trading firms employ high-frequency trading, yet at its peak, it accounted for 10 to 43% of stock trading volume on any given day. 

The ingredients for HFT--massive computing power, high frequency streaming big data, and ultrafast connections--are all areas where deep learning and machine learning workflows excel. 

Pre-trained models can prevent machine learning and deep learning algorithms from becoming speed-limiting factors. Coupled with techniques such as deep reinforcement learning, HTF is primed for another technological leap. However, given its increased complexity, it will remain the domain of a relatively few, but highly profitable, firms. 

Why it's Important: Trouble Ahead

The Flash Crash that occurred on  May 6, 2010, at 2:45 EST caused trillions of dollars of market equity to be wiped out in an instant (36 minutes to be precise). Regulators have struggled to keep up with algorithmic trading and high-frequency trading, and will doubtless be hard-pressed to stay ahead of the next generation. HTF AI agents will require much more sophisticated risk monitoring and compliance systems that in turn will need to employ machine learning to monitor.

3. Risk Assessment Platforms

Despite the vast sums invested in technology by financial institutions, the humble Excel spreadsheet remains the number one application on Wall Street. Risk departments, charged with ensuring traders don’t make calamitous errors, are no exception. Even the better-equipped firms employ software that relies on rule sets and analytics that are apt at catching known risks but are poorly equipped to identify evolving market risk.  

The nature of a robust risk assessment platform is a kind of catch-all. Risk scales from individual trades to companies, industry, country, and global risk profiling. Risk can be quantifiable, but often a risk assessment may need to rely on alternative data. Machine learning’s adaptability and flexibility make it a natural successor to current risk assessment software. Both supervised and unsupervised machine learning techniques can be employed to layer on more sophisticated risk strategies. Anomaly detection used to identify outliers is one such technique that can be readily employed to identify the rare events that are characteristic of risk modeling.  

Why it Matters: Risk and Repeat

The recent implosion of Archegos Capital in March cost some of the world's most sophisticated banks to lose up to $10 billion, highlighting the poor systems and oversight that many financial institutions face with trading risk exposure. Similar risk failure, albeit of a smaller magnitude, continues to abound despite the lesson learned and trillions lost due to the risk failure that gave rise to the 2007 financial crisis. Risk departments are finally waking up to the inherent advantages of pattern recognition machine learning versus manual and backward-looking analytics tools. Add to this the increased complexity due to, you guessed it, machine learning trading strategies.

4. OMS Trading Platforms

Retail traders have flocked to online trading platforms like Robinhood, Fidelity, and E*Trade.  The institutional professionals use more advanced systems called OMS (order management systems) from companies like B2Broker, Charles River, Interactive Brokers, and others. These institutional trading platforms all execute the same basic workflow. Financial market data is fed in a set of static trading, risk, and compliance rules are applied; buy and sell orders are generated; the order book is updated; and trade analytics reports are generated. 

Traditionally, these platforms were closed systems. Many provide limited APIs that allow customization of various aspects such as data feeds, order flow, and algorithms, but most work only within the confines of their particular platform. Advanced hedge fund traders are employing sophisticated machine learning and deep learning techniques that utilize platforms like Tensorflow, Keras, PyTorch, and similar frameworks and libraries. Deep learning techniques such as deep reinforcement learning, NLU (natural language understanding), and transfer learning require these platforms. These models often require alternative data whose unstructured format does not readily make itself suitable for the structured time-series format many of these present trading platforms require. 

Why it’s Important: From Closed to Open

At some point, this equation will flip. Trading platforms are very good at order workflow and trade analytics. However, data profiling, data transformation, and machine learning algorithms need something much more flexible, adaptive, and open. The existing dominant market players will need to adopt a more open API approach that gives full access to every stage of the order workflow. At some point over the next 5 years, this in turn will lead to adoption by retailed brokers and bring machine learning trading to the masses. Perhaps we may even see a return of Quantopian!  

From Leader to Laggard

For the last few decades, Wall Street has been a clear leader in rolling out complex platforms such as algorithm trading and high-frequency trading (HTF), and other innovative trading strategies.  However, many of these systems rely on static rules-based systems or predictive analytics at best. Other companies that fully embraced machine learning and deep learning earlier have come to dominate sectors of their industry.  Expect a similar shakeout in financial institutions as some companies go all-in on artificial intelligence and become the next generation of technology leaders.  

Learn More

Machine Learning in Finance is a major focus area at ODSC Europe 2021. If you’d like to learn more about finance and machine learning we have a host of topics to choose from including free sessions:

• An Introduction to Machine Learning in Quantitative Finance
• Jun 10th: 3:20 PM GMT | 10:20 AM EST  REGISTER FREE
• Machine Learning for finance with TensorFlow
• Jun 10th: 11:55 AM GMT | 6:55 AM EST  REGISTER FREE
• How to Build and Test a Trading Strategy Using Machine Learning
• Jun 10th: 1:30 PM GMT | 8:30 AM EST   
• Reinforcement Learning in Finance: Playing Atari vs Playing Markets
• Jun 8th: 11:55 AM GMT | 6:55 AM EST   
• Artificial Intelligence in Finance
• Jun 8th: 10:15 AM GMT | 5:15 AM EST   

Other relevant Machine Learning Topics for financial professionals include:

• Can Your Model Survive the Crisis: Monitoring, Diagnosis, and Mitigation
• Metal-Learning with an Older Abstract of Anomaly Detection
• PyTorch 101: Building a Model Step-by-Step
• Artificial Intelligence Risk to Companies
• Adversarial Attacks and Defence in Computer Vision 101
• Finding that Needle! Modern Approaches to Fraud and Anomaly Detection
• Production Machine Learning Monitoring: Principles, Patterns, and Techniques

ODSC EUROPE 2021: Jun 8th to 10th is a live virtual event hosting over 4,200 attendees, and 150 sessions. More information available here.