Deep Learning - "Hello World"

This is the third in a series of articles on my experiences in deep learning with the focus of diving straight into implementing the deep learning version of "Hello World". My goal was to apply deep learning to the accurate identification the MNIST handwritten digits (e.g. 1, 2, 3, etc.) database. I decided to focus on using available frameworks rather than coding algorithms from scratch in part because it would be time consuming but mostly because it would give me a better understanding of the ecosystem around deep learning.

A quick Google search gave me the top result of MNIST For ML Beginners which fit the bill nicely for a good starting place. Installation of TensorFlow and its dependencies turned out to be easy on Mac OS X. The only challenge was getting all the paths set-up correctly to allow loading the TensorFlow module but that was rectified using the recommended virtualenv. I first started with the bare bones machine learning script enhancing it to better understand the workflow, execution times of the workflow elements and overall execution time. This machine learning version consistently resulted in a 92% accuracy with an execution time of about 5 seconds (on a 5+ year old MacBook Pro). I thought Woohoo! at least until I read "Is that good? Well, not really. In fact, it's pretty bad." With some tweaks I could have probably increased accuracy above 95% but instead I decided to put the deep in deep learning.

TensorFlow also provides Deep MNIST for Experts tutorial which definitely requires at least a basic understanding of building of a multilayer convolutional network and more than a basic understanding of Python. However it was still approachable with a encoding similar workflow as the original Python script but adding the extra layers and reworking other elements of the code. You can compare the basic deep learning version of the Python code, mnist_deep.py, to the machine learning version to see the differences. The more important difference was that the accuracy increased to 99% (Woohoo!) but at the expense of more than 56 minute run time (again this is on an old laptop using no hardware optimization). In addition to TensorFlow, I also played with the Theano framework trying their hello world tutorial of Classifying MNIST digits using Logistic Regression and Convolutional Neural Networks (LeNet) with similar results although performance didn't seem to be as good as TensorFlow and the Theano python code was more complex (however I need to spend more time understanding the differences in the frameworks).

It turns out that a much easier path to MNIST Hello World was NVIDIA's free Getting Started with Deep Learning course that leverages NVIDIA's DIGITS platform requiring no loading of software, no Python coding, very nice web interface and access to cloud-based GPU hardware (K520). The tutorial is great because it gives you a better understanding of the ML/DL workflow plus tips/tricks to enhance the results (e.g. data augmentation techniques as one example) and a lightweight introduction to the Caffe Deep Learning Framework. The tutorial also shows the power of using a GPU and leveraging GPU resources in the cloud. Even the Kepler-based GPU, two generations removed, shows the value of using GPUs giving a dramatic training and inference performance boost making even the deep learning MNIST hello world a breeze. The hello world Experience has only increased my curiosity showing that deep learning is very accessible even for the novice. Now challenge is to better understand more complex implementations of deep learning and its real-world applications.

To be continued... Part 4 - Deep Learning - Getting Real.

See also Part 1 - Learning about Deep Learning and Part 2 - Deep Learning - Getting Started.

About the Author - Darrin P Johnson has more than 25 years of experience in the computer industry largely in system software engineering with a passion for innovation. His latest quest is to understand Deep Learning and more importantly how it can be leverage to drive innovation. Any comments, suggestions, etc. will be appreciated.