Deep Learning from 30,000 feet

If you fly in April on Delta, be sure to check Delta Sky Magazine for an article on Deep Learning!

I was recently contacted by Joseph Flaherty, a writer, who asked me for a brief overview of Deep Learning. Here is his article, which quotes me a few times, published in Delta Sky Magazine April 2016 issue, on pages 52, 54, and 56.

See below a really high-level overview I wrote, appropriate for reading at 30,000 feet.

Deep Learning from 30,000 feet

Deep Learning is probably the most important technical development since the invention of the web in 1990s.

It was partly inspired by the way the brain works - some of the Deep Learning pioneers like Geoff Hinton have studied and sought to emulate brain processes. However, just like the airplane flight was inspired by birds, but flies very differently, so Deep Learning works quite differently from the brain.

Deep Learning uses multiple layers of neural networks, where each layer has many artificial "neurons" - simple units that have many inputs and one output. Neural networks learns from experience - they take inputs, combine it inside neurons with different weights, and give outputs, which can go to the next layer to be final output, for example, yes/no, dog/cat, or how to move your character in a video game.

Neural networks learn by backpropagation - the output they generate is compared to correct one, and errors are propagated back to change the weights so that next time the same inputs give output closer to correct one.

Neural network methods have been tried back in 1960s, but one-layer methods were limited in their ability, and multiple-layer methods took so long to train that they were considered impractical. Work on neural networks was very unfashionable for many years, and in 2000s was mainly sustained by 3 researchers: Geoff Hinton in Toronto, Yann Lecun at Bell Labs, and Yoshua Bengio in Montreal.

Their work was mainly supported by Canadian government. They called themselves a "Canadian Conspiracy" and introduced a term "Deep Learning" to get away from unfashionable "neural networks."

The big breakthrough in Deep Learning happened around 2012, powered by very large datasets, very powerful processors (mainly GPUs), and some very clever algorithms, DL methods started to win Image Recognition and Machine Learning competitions, and then started to approach and in many cases exceed human performance levels in many areas, for example

• image recognition (DL is used to scan every image uploaded to Facebook for face and object recognition),
• automatic caption generation, see image below
• speech recognition (Google Now, Siri, Baidu). If you spoke to your smartphone recently, you probably noticed the very high quality of speech recognition
• machine translation - just try Google Translate or amazing real-time speech translation in Skype
• games: DeepMind exceeded human-level in many Atari video games, defeated a European Go champion in 2015 and in March 2016 Google AlphaGo program defeated a world champion in Go.

Deep Learning methods automatically generated a caption for this image:

Woman in white dress standing with tennis racket; two people in green behind her

More than that, DL also highlights areas on the screen corresponding to words.


Read the rest of the post on KDnuggets:

Deep Learning from 30,000 feet -

https://www.kdnuggets.com/2016/04/delta-deep-learning-from-30000-feet.html