Introduction to Trading. From Data Science Perspective.

Short Intro

Some time ago, I decided to switch my AI focus from NLP to time series. Naturally, is there a more intriguing use case for time series than stocks and forex markets? I seriously doubt. So, I started my diving adventure into that world. These are my impressions.

First Impressions

OMG. This is an entirely new world, with its own terminology that is clearly not intuitive to the rest of the world. Talking about bools and bears; scalping; asks and bids; so-called short and so-called long, and so on.

First of all, I figured out that if I want to really trade out there, I need a broker. So I found one in my neighborhood.

First thing I learned: brokers need your money. They will do/say anything to convince you to deposit cache on your real account.

And the second thing: don't take everything you hear from the brokers within a common sense measurement. When you invest money in shares, convinced by your broker, and you buy shares for 170$ each. Within the very same day, you lose A LOT OF MONEY "just" because that price dropped to 125$/share. Your broker will tell you that that is an entirely normal thing on the stock market. You ask him, "ok, what just happened?", and you would receive an answer like "nothing special, just an ordinary price correction". Ok ... the Earth is still rolling around, so keep moving and breathing. I personally think of this as a very well spent money on personal education.

First tries to do something in your interest.

Buy/Sell signals

So, you have a broker, and you have some deposit. You are trying to make some successful deals, i.e., to make some money. So start googling, without any clear idea, what you are actually looking for. As usual, Google already has an idea of what you are looking for, and you can see many ads for so-called trading signals, i.e., somebody will tell you what and when to buy or sell something. But, they are not free. You may try some of them, and you'll see something that they act as a random number generator you are paying for. Probably there is some valid reasoning behind these signals, and they are not entirely randomized as they appeared to me. Let's say, if you act according to them, in 55% of the cases you'll gain some money. This means that you'll have 55 trades with some gains and 45 with losses from 100 executed transactions. And, there is a catch: with those 55 trades with a positive balance, you'll have to cover two things: spread in the stock prices, i.e., the difference between buying and selling price (another term I learned on the more challenging way), and the price for the signals. The real question is: will you be with a positive balance after all? Didn't have time and money enough to do the experiments, but I seriously doubt it.

Technical Analysis, by others

The next thing you'll probably try is to figure out what the technical analysis is. This is trying to figure out what will happen with the prices based on the current stock price movements and some historical price values from the past.

This is at least complicated: science by itself. But, you figure out that many sites offer such analysis already made, for free. Examples: your own broker's site, giving such analysis for their customers, or investing.com. Results: inconsistent, at least. Some of them offer just buy or sell advice, and this might be ok. Others provide 5-min, 1 hour, and 1-day advice. And they change their direction at least several times per day, so you ask yourself: how is it possible to shift 1-day advice 5 times per day? Is such advice valuable? Let it be your lucky guess.

Technical analysis by yourself

Next: you learn how to do this technical analysis by yourself. What can be wrong with that? Shortly you figure out that technical analysis is nothing but finding patterns in the charts and figure out the next price movement. That's great, isn't it? What is ML and Data Science more than finding patterns in a bunch of data?

So, you start learning about spotting patterns in the charts. There is really a lof online materials on that topic: Udemy courses, Youtube, etc. You begin to spend time on them. At one moment in time, you figure out that, literally speaking, all of them are elaborating the past market prices. They say something like: you see, it is clear from this chart that this point was the breaking point after which the price movement changed. Thank you guys a lot. I can read the past and figure this by myself. But, I didn't find one example where the instructors explain how to make the predictions, or, at least, to distinguish the more probable ones from the others.

They are continually saying: you see, this pattern is the double bottom: price hits the support line twice before it starts to move upward more significantly. The problem here is the perspective: what time frame you are focused on. If you watch 7 candles before, agreed, it is an example of a double bottom. But, expand your window to 15 candles, and you possibly correctly spot the inverted head and shoulder pattern. Ok, nothing wrong so far, still you could reasonably expect that the price will start to rise. But, move more 5 candles to the left, and voila, there is clearly a channel pattern, so the price is reasonably expected to continue bouncing within the same upper and lower limit, no clue when it will go outside from the channel.

Don't even try to start analyzing Elliot waves or Fibonacci retracements.

Eliot waves analysis goes like this: three ups, then two downs. Ok, that sounds like a predictable thing. Unless you put some focus on details: this small down here is short, it can be ignored, just a regular price oscillation. And guess what: there is no explanation about what can be considered a normal price oscillation, i.e., little noise, and what is significant enough to be taken into account when you analyze Elliot waves.

Fibonacci retracements: 0.618 and 1-0.619 are so-called Fibonacci numbers; they do have some mathematical relevance; they describe natural processes connected to beauty, etc. A vast majority of the traders take care of 0.5 as well, so why not put it among the Fibonacci numbers. Now, choose one interval, and cover the highest top and the lowest bottom with a Fibonacci rectangle, and you see, all the tops and bottoms in between are precisely at the Fibonacci numbers. Here, "precisely" means one circle that covers this top, not strictly in a cent, because this is not an exact science, you know.

Haven't seen better examples of how people delude themselves by seeing what they want to see, everywhere where they are looking at.

I can safely share what I spotted as a probable rule: if there is a significant and very steep upside in the price when it hit the top, there will be a significant but shorter steep downside in the price, and vice versa. This can be taken with feels-like 70% of probability. All of this happens very quickly, in a few minutes only. Consider yourself lucky if you spot it on time, and you act accordingly. Play on that card, and with 5 rounds played on that way, with a probability of 99%, you'll make some money.

Also, support value may occur as resistance later, and vice versa. Don't ask for an explanation of why that is so. Maybe something with the psychology of the traders.

Also, support and resistance occur frequently on round values. This is definitely connected to the trader's psychology: it seems that ordinary people prefer nicely rounded numbers, without mathematical explanation, just dull to type a lot of digits. This can be useful for your strategy: make decisions on numbers slightly over or below rounded numbers. You might be faster in most cases.

At the end of the minute/hour/day/week/month, the reasoning goes as follows:

To buy? Are you nuts? Look at the charts; the price is going down.

To sell? Are you crazy? Look at the charts: everybody see this, everybody will sell, and the price will drop.

Not to do anything? What's wrong with you? Look at the charts!

Too many times, I experienced "not again, this stock was going up, I bought it, and not it's going down". This is perfectly explainable, you know.

Explained in detail: the stock is worth buying if the price is going to rise. The stock price will increase until there are people interested in buying the stock. If you are noticing that the stock price is growing, many people are already purchased. If it is already purchased by many people, there are fewer people interested in buying it. If there are fewer people interested in buying it, the price will go down. Long story short: shorting rising stock and longing sliding stock makes a lot of sense. So: if you see the stock price going up, would you buy it? And vice versa: if the stock's price is going down, will you short it?

In short: whatever you do, it's Heinseinberg's trend: both up and down soon. Only noticing it will decide what it will be. Unfortunately, too late for financial markets.

One example is that MRNA shares went up for 8$ after the EU allowed purchasing vaccines from Moderna: a move anticipated and waited a long time ago. Then, the reasoning goes as follows: not so massive movement on that colossal news. But you think that now others will expect the price to go up, and they will immediately start to buy MRNA stocks; therefore, they will be overbought in a very short term. This should result in a price drop. Correct? Nope. The opening price the next day is 6$ up. So, if you shorted MRNA stocks, you would lose a huge. The MRNA spares started to go down an hour later, but insignificantly, probably because of the overbought effect, however too late. After a vast disappointment, you have probably left your position, saying goodbye to the lost money.

All of this is so scary well aligned with the "random dart trading": https://www.marketwatch.com/story/random-darts-beat-hedge-fund-stars-again-2019-06-26. As a result: overthinking is counterproductive, and we must figure out the possibilities of analysis applied to trading, and they are not promising.

Finally, one good material on the topic:

https://www.udemy.com/course/the-advanced-technical-analysis-course/learn/lecture/12395478#questions

and another one:

https://www.udemy.com/course/trading-with-technical-analysis-mastery-2020/learn/lecture/20793342#overview

So much for the technical analysis.

Verdict: too many technical indicators/oscillators to spot something you can already see by yourself, just looking at the charts.

There are a lot of graphs out there but use only and only for evaluating the past. You can't use them to predict the future and make buy/sell/close decisions in real-time trading. The future is unknown; there is simply no data for the next couple of seconds/minutes/hours. You can say that the price will be somewhere around the current price, plus/minus 10-20%. Therefore, no matter how robust your indicator is, some information will be lost for sure if there is a bug in the implementation. But, even it is correctly implemented, it plainly can't extract information where it doesn't exist.

One thing that appeared to be helpful: TradingView. The best app for helping traders to spot the market movements. Simply love it. Not free, but worth every cent spent on it.

Next phase: analyzing the outside world inputs, i.e., reading news

There is simply too much information to be digested, don't even try it. A couple of sites provide aggreged news: benzinga.com, investing.com, finance.yahoo.com, stocktwits.com, theinformation.com, finwiz.com, e.t.c.

Have a sense of market sentiment.

The complete market trading sentiment is more than essential: Trump signed the bill for injecting billions of dollars into the US economy. As a result, everybody is happy and optimistic and willing to put some money on stocks. As a result: stocks are rising. As a result: whatever you do, you'll probably make some money. Otherwise, you'll lose some money, on average. But, of course, a day before this happens, you have to know what will happen. In this ok, that was expected, what else he could even think to do else?

In another similar case, the EU is expected to decide whether they'll approve the Moderna vaccine or not in the day that follows. But, did they have a choice in a situation with rising numbers of deaths because of the new strain of the virus that is expected to be treatable with the Moderna's vaccine as well? Nope, not really. Therefore in similar, obvious cases, watching the news might help in proper risk management.

On the other hand: you can lose money whatever you do. One vaccinated nurse found dead while sleeping, and MRNA stock down for 1.7$ in seconds. Trump administration, "yet to be gone in hours," said that the USA might ban BABA (Alibaba), and NEO (Chinese EV maker) lost 5$ in a couple of minutes. That wasn't able to be seen by charts, either analyzing previous news, market sentiment, etc.

Soon you realize that being in line with the latest news is essential for fast, so-called intraday traders. A few minutes later, and you can lose a lot of money. So, you start looking for so-called real-time quotes and news. They are not cheap, at about 50$/month. Add here analysis from experts, and you'll have to pay at about 150$/month. Now, you soon figure out that expert estimation is not that good as you might naively expect. It can give you a hint, but it's not a crystal ball, for sure. How many experts, so much analysis/opinions. Also, there is a ton of expert analysis that appeared to be worthless on today's price movements, even counterproductive in so many situations.

Nevertheless, like some things in life, the best option you can get among others paid is free: stocktvits.com. Twitter for traders, many young kids eager to earn some money and willing to share their information all over the place. There you have real-time quotes and news, after hours and premarket, and everything you need and somebody else interested already knows, for free. Enjoy while it lasts.

Verdict: digest a lot, but only read headlines for news related to pre-selected stocks by you, and don't pay too much attention. After a couple of days, you'll be able to form your own sentiment based on such information. Not worth more than an hour per day on this.

NLP in analyzing news

Tried it. News took by finance.yahoo.com via REST API. Sentiment analyzed implemented in Python with NLTK. The result appeared to be useless: no correlation between price movements and sentiment charts.

Tried to figure out why and figured out why, analyzing news in depth. There was news that the Moderna CEO just sold a massive amount of MRNA stocks. How should this be characterized, by the machine, with a positive or negative sentiment? But, this is, of course, a piece of huge, impactful information. So, we need a specially trained semantic classifier for financial news. Has anybody heard about such? The things are even more complicated: If the CEO just sold a lot of stocks, this should generate negative sentiment for the stock, i.e., if the company is doing well, why would the CEO sell it? On the other side, selling so many stocks increases the stock price by automatism. What should be the summarized sentiment of such news? I have personally no idea, and I can't expect a machine to have its own as well. Simply put, too complicated.

Fundamental analysis

Another word for looking in numbers, this time, other than quotes (open/close/high/low/volume) trying to predict the future. This time, the numbers are expected to be seen. For example, dividends and profits. They are delivered at regular time intervals, and you know exactly which day you can expect them. Therefore, you mark that day as a potential hazard because something will happen for sure, but you don't know it's sentiment. Profits and dividends for APPL are expected and normal to be high, but the question is: will they be as expected or diverge from them? Also, among other things, as a part of fundamental analysis, you can pay attention to global financial and political happenings in the area, fiscal and monetary policy, periodic trends in economic growth. So many common sense things that may affect the results of your decisions in the very long term. If you expect to see results in a couple of days: useless. Which is commonly sensed expectable. 

Analysis Summary

Whatever we do is unreliable.

For each analysis outcome, we can make a case for both BUY, SELL, or HOLD.

Trading strategies

Finally, clues of something that looks math-backed. And, one big YES: finally: traces of time-series found in pair trading: you locate correlated stocks, create a fund and treat it as a single instrument, and trade it with averaged gain/loss. Some less and some more sophisticated strategies were found out there but promising for further analysis.

Some exciting features appeared searching for trading strategies: there are so many "secret" strategies with 70-80-95% of success. You just have to pay not that small amount of them to see them. Now: if they are so successful, why somebody that apparently made so much money with them would spend his time selling them to you? Hey, the profession of these guys is to make money. If they are good at their careers, they don't need to sell something to you. If they need to sell something to you, then they are not good at their jobs. Therefore, why paying for their stories? 90% of them are claiming that they are doing this to help others to make money. If they have a little bit of common sense, they would clearly understand that if their magic is known by a lot, it will affect the effects of their magic, and it wouldn't be magic anymore. Willing to "just help to the others, for some amount of money"? I frankly don't believe that.

NIO: To "believe" or "not to believe", the question is now

Bottom line: the result is totally confusing. Whatever you decide, you can build a case for it, no matter you do technical or fundamental analysis.

Algorithmic trading

https://medium.com/datadriveninvestor/the-best-book-to-kickstart-your-algotrading-journey-9a8e7105a1c4

In short: Define strategy that involves technical oscillators and current price for entering/exit buy/sell a position, and execute it automatically.

For. ex: SMA(15) > current.price + 10%: SELL

SMA(15) < current.price - 10%: BUY.

Backtest, analyze, and run it on a real account.

Nothing intelligent currently there.

The current state of AI in trading

Inefficient, at least, on what I was able to reach and evaluate.

finbrain.tech - tested. Registered, paid, evaluated. Price for his services: at about 160$/month. Information provided by him: useless. Nothing more than common sense, if you see charts observed by him. Some information that will help you to decide whether you should short or long something: nope. Otherwise, very eye-catching UI, a lot of high-tech smell on it.

toggle.io - same as finbrain.tech

tekton.io - friendly looking site for backtesting algotrading and building strategy. Otherwise, just one of many other sites with the same functionality, finwiz.com for ex., only this is all about algotrading. Still wondering how the guys didn't figure out to implement an automatic search for the optimal strategy with the simplest form of grid-search for hyper-parameters. 

Lessons learned

Don't be fast on the keyboard/mouse. The more moves you make, the more money you lose. Make one or two moves per stock per day. Don't revert in the first 30 minutes of the opening hours unless there is a steep downtrend, I mean very steep.

After every uphill, there is a downhill, and vice versa. But: here, you don't know how big the pivot will be, so wait until the chart stabilizes. Otherwise, you lose a lot of money.

Write down your entry and exit trading plan and stick to it religiously.

I wondered why so many Udemu courses pay attention to psychology. Now I understand: the wish to click one more time and earn some money is enormous.

Don't forget

Just don't forget to measure the time you are spending on trading: analysis, research, monitoring the stock, disruptions all through the day, only to see the price ... Then calculate how much money you actually made through the month. Then calculate how much you are earning per hour with your regular tedious job.

There are easier ways to earn money, you know. Data Science, for example. And you'll appreciate your tedious job more.

The Goal

The idea is to find an optimal ratio between time spent on trading and the money return.

What Comes Next?

Article on how to use AI in trading, the cutting edges. cu l8r.