Speedup Machine Learning 1000 Times

If you are an IT professional who had paid any attention to technology trends, you should know that machine learning is red hot these days. Google, Facebook, Amazon are urging every one of their computer programmers to learn machine learning. The whole industry forecast that eventually deep learning can be used to train and solve any computing problem you can think about, better than algorithms designed by the human programmers. So no computer company wants to lag behind in the competition.

One problem with machine learning is that it has an extremely high cost of computation time. From a recent presentation by Baidu, it was said that to train a model costs about 10 exa flops of computation. That's 10^19 floating point operations. The model trained contains roughly 100 million parameters. So that's roughly 10^11 flops to fine tune each floating point parameter of a deep learning model. Modern computing technology likes GPU makes such an astronomical amount of computation feasible for big corporations who can afford gigantic data centers. But does machine learning has to be this slow?

When I was at Google, the company open sourced its deep learning software kit called TensorFlow. Jeff Dean explained why Google was willing to open source TensorFlow. He stated that not everything is released. Google kept a small portion of secret sauces that maybe making the deep learning computation 50% or 100% faster than the open sourced version, thus giving Google a competitive advantages. I am not sure who has the best secret sauce. But it shows that making machine learning 50% or 100% faster is a big deal, a very big deal, for companies like Google.

What then, will be your response, if I tell you that I can make machine learning not 50% faster, but one thousand times faster, than the state of the art today? I am not taking about GPU. Everybody already knows that. GPUs make machine learning way much faster than a purely CPU-based solution. Every player who is doing serious job on machine learning, are already leveraging the power of GPUs, or are trying to. But I am not talking about GPU, I am talking about a radically different algorithm approach that I am developing in my spare time. If this is successful, machine learning will be thousands of times faster. Deep learning that now runs at Google's giant data centers can be put in a smart device in your pocket! This is the kind of progress needed to truly make machine learning workable in consumer devices.

Is it possible that we can see a thousand times improvement from current state of the art in machine learning? Yes it is absolutely possible. That's because current algorithms are simply inefficient, and they remain as inefficient as they were a few decades ago. Ten Exa Flops to train a model? That's ridiculous. That's because in typical deep learning algorithms, it keeps going back and forward adjusting parameters in tiny steps, two step forward and three steps backward, and repeat it many times, in hope of averaging out the noise and keep just a little bit of useful data. Each one of the deep learning model parameter is likely a half precision float point number, having 16 bits of information. There cannot be more than 65536 different values of each. You could try all 65536 of them to find one parameter that fits perfectly. But in real deep learning algorithms, each parameter is adjusted a few hundred billion times when the training is all done. There are lots of repetitions, lots of going around back to square one, lots of summing up all the noises and finally add it up to zero, in another word lots of wasted computation. This wastefulness has not changed fundamentally from decades ago. The difference is back then, such required computation is impractical, but today, thanks to Moore's law, such amount of computation is finally available. But this inherit wastefulness is the reason why there is plenty of room to look for a solution that can deliver thousands of times of improvement. I believe I am on the right track to find such solutions.

How is it possible? As I am still developing the technology and hopefully patent my ideas, I cannot disclose everything. But we need to keep in mind that natural biological neural network is the most efficient neural network we ever know. Learn how the real neural network does, and we can do way much better than current deep learning code.

1. Natural neurons are imprecise and do not store much information. It is estimated that each neuron can be in no more than 10 different states. That means a neuron stores about 3 or 4 bits of information each. We can construct neural networks with very simple neurons with very little data.
2. Natural neurons work on statistical principles. They are not deterministic. Computer chips are deterministic, unfortunately. I hope some one can invent statistical computer chips. They consume much less energy and run much faster.
3. Not all computation are equally important. When you are summing up lots of noises and you know the final result is likely zero, you might as well skip the entire thing.
4. Train individual neurons, not an entire network. Biological neurons know nothing of the whole brain. They only know their vicinity and respond only to what happens to their vicinity. When you isolate the problem to a small unit, the complexity of the problem is reduced exponentially. It becomes a linear scale problem instead of a geometric scale problem.

My experimental results are very encouraging. Model training can be done in a few thousand iterations with amazing accuracy. I tried an extremely simple neural network with a few tens of neurons, 3 or 4 of them tied to random inputs. There is a hidden relationship that one of the output is an OR value of two of the inputs but independent of the other inputs. Starting from a 50/50 accuracy (which is just random chance) and after some training, the network began to magically predict the correct result with almost 100% accuracy. The result is just amazing. I hope to continue my research on my discovery and develop this into something useful in the near future.