What did Deep Learning actually solved ?

Deep learning has emerged as a pivotal advancement in the field of AI, adeptly solving complex challenges that once stumped traditional machine learning algorithms.

In this post, I will illustrate step by step how deep learning has addressed problems that earlier algorithms and techniques struggled with.

Classical machine learning

Let's consider a scenario where we aim to separate blue circles from green triangles. If it appears to be a circle-based problem, you're on the right track. The question then becomes: how can we draw a straight line through this data to separate these shapes? (After all, straight lines are preferable for their simplicity.)

For those with a background in geometry, a solution might involve transforming Cartesian coordinates into polar coordinates. By doing this, the polar coordinates will have one axis representing the radius, allowing us to differentiate the two shapes based on their radii, thereby achieving separation with a straight line.

This approach exemplifies classical machine learning algorithms, where we ingeniously transform data to make it more amenable to algorithmic interpretation, a process known as manual feature engineering.

However, the real world isn't always so accommodating, often presenting us with data that doesn't conform to simple shapes like circles.

Representation Learning

Take, for instance, the task of distinguishing human images from alien images using machine learning models. Given that images are essentially grids of pixel values, we're faced with the challenge of approximating some complicated functions using a simplified model. It might not necessarily be a straight line, but the goal is to simplify the model enough to make accurate classifications. The complexity of the data often outstrips our ability to manually craft features that effectively reduce its dimensionality, revealing the limitations of our human capacity.

This necessitates an automated method for feature detection, known as representation learning. Techniques like principal component analysis (PCA) and autoencoders are examples of this.

Principal component analysis (PCA)

Focusing first on PCA, imagine we have images with dimensions M x M. Each data example in our set then consists of M^2 features, which is an overwhelming number. Our objective is to identify and retain those features (in this case, pixel intensities) most likely to influence our ability to discern whether an image depicts a human or an alien. It's essential to reduce the dimensions of our input to avoid the "Curse of Dimensionality," a phenomenon in classical machine learning where an excessive number of dimensions leads to sparsely populated training spaces, complicating pattern recognition. Interestingly, deep neural networks seem to sidestep this curse despite having an order of magnitude more parameters than training instances, a topic that remains an active research area. PCA helps us in this context by identifying a smaller number, K, of the most variegated and significant features, where K is less than M. We can then apply algorithms like k-nearest neighbors or support vector machines for classification.

Autoencoders

Another example of representation learning is the use of autoencoders, which arguably fall under the umbrella of deep learning. In this technique, data is passed through an encoder function. For example, an image is processed through strided convolution operations to reduce its size, resulting in a compacted form referred to as latents. These latents are then expanded back to the original size through reverse convolution, adjusting parameters to best recreate the original image. The ability to compress a large image into a smaller latent form implies that the latent retains essential information about the original image. This compressed form can then be analyzed to extract features common to humans and aliens alike, akin to the feature transformation seen in the previous example with coordinate transformation.

Deep Learning

Some tasks in the realm of AI, such as machine translation and speech synthesis, are so intricate that learning representations for them can be as challenging as solving the problem itself. This is the juncture where deep learning steps in as a critical solution.

At the core of deep learning lies the concept of neural networks, organized into structured layers. Each layer functions like a sieve, filtering and refining the incoming information. The initial layers capture basic patterns, such as the edges in an image or rudimentary sounds in speech. As the data cascades through these layers, the neural network begins to weave these basic elements into more complex patterns, transforming lines into shapes in the visual domain, or individual sounds into coherent words in audio processing.

This stratified approach is one of the defining features of deep learning. It remarkably obviates the need for human intervention to guide the machine on what to focus on. Instead, the machine autonomously learns from the raw data it is fed, which typically includes extensive datasets. For example, in the field of image recognition, neural networks undergo training with a vast array of images, learning to discern an array of features crucial for object identification.

Training these networks involves a method known as backpropagation. This technique functions akin to a feedback loop: the network makes predictions, evaluates its errors, and adjusts its parameters incrementally for improved performance in subsequent attempts. This iterative learning process enables the network to progressively minimize errors and recognize increasingly complex patterns.

A key strength of deep learning is its proficiency in handling high-dimensional data. Whether it’s analyzing a high-resolution image or processing a lengthy audio recording, deep learning networks are equipped to manage such complexities, a feat that traditional methods struggle with. Moreover, these networks demonstrate exceptional effectiveness in their performance.