Predicting Chinese Stock Market with LSTM + Transformer: An Unveiled Secret Project

After leaving my previous role, I spent two months establishing North Path (https://www.north-path.site/) and traveling across China with my family. Following that, I embarked on a secret project that has consumed the majority of my recent efforts: the development of a stock prediction system named Kairos.

While I was working on Kairos, a lot of people reached out, but I had to go off the grid for a while. I was just so focused on getting this system finished. You see, the whole reason I started building Kairos is because of my own tough experiences trying to invest in the Chinese stock market. My friends know I've been trying to make it work, but honestly, it's been brutal. The losses have been hard to swallow, for both me and my wife. Trying to figure out if a stock is going up or down has always been a fascinating puzzle, and with all the cool stuff happening with AI, I got really curious. So, that's how Kairos was born.

First things first, I needed data. I used a package called akshare to scrape pretty much all the relevant stock data I could find. Then came the fun part (not really!) - cleaning up the data and doing a ton of feature engineering. Finally, I merged everything together.

If you take a closer look at the database I built, you'll see that I didn't just focus on individual stock trading data and financials. I also included data on various Chinese indices, US stock indices, the VIX (fear index), gold prices, and more. The reason for this is that these macroeconomic and market sentiment indicators often have a significant impact on individual stock performance. By factoring them in, I hoped to improve the accuracy of the predictions.

For the training phase, I harnessed the power of my MacBook's M3 chip and leveraged its GPU for accelerated training. I opted for Mean Squared Error (MSE) as the loss function, as it's well-suited for regression tasks and effectively measures the difference between predicted and actual values. To optimize the model parameters, I utilized the AdamW optimizer, which combines the benefits of the Adam algorithm with weight decay, helping to prevent overfitting. Furthermore, I implemented an Early Stopping mechanism to automatically halt training when the validation loss plateaus, preventing the model from overtraining.

I experimented with various model combinations, including CNN-LSTM, LSTM-Transformer, and some simpler LSTM models (see image), subjecting them to rigorous training and comparison. Each training session typically took 1-2 days to complete. Ultimately, I found that the LSTM + Transformer combination yielded the most promising results. This doesn't necessarily imply that there isn't a superior combination out there; it simply signifies that the training process is exceptionally time-consuming and demands substantial computational resources.

LSTMs excel at capturing long-term dependencies in time series data, while Transformers are better equipped to learn global information and contextual relationships within the data. Combining the two allows us to leverage their respective strengths and enhance the predictive capabilities of the model. Certainly, CNN-LSTMs also offer advantages, as they can extract local features from the data and combine that with LSTM's ability to learn temporal dependencies. However, in my experiments, it slightly underperformed compared to the LSTM + Transformer combination.

Throughout the training of Kairos, I found myself in a state of deep reflection, reminiscing about past experiences. I express my sincere gratitude to Spark, my former manager at eBay, Xtof, developer manager, and Ming, the CTO, for their guidance and support. They not only provided me with invaluable learning opportunities but, more importantly, illuminated my path forward with their long-term vision. While my achievements may seem insignificant, I deeply appreciate their encouragement, which has been instrumental in my journey. I would also like to extend my heartfelt thanks to all my former colleagues at eBay for their help and camaraderie along the way.

Finally, the accuracy of Kairos has significantly surpassed my own judgment, even surprising me at times. As you can see in the screenshot, each number represents the output from a different model, and these models can rank stocks based on their predicted probability of success. However, I am not planning to make this project public for two reasons: firstly, I don't want others to replicate my trading strategies, and secondly, I don't want to be held responsible for anyone else's investment decisions.