Deep Learning, Neural Networks, and Beyond

Below will be a deep dive into some information about the broad subject of AI. This article is to help identify the meaning of the terms that we sometimes use interchangeably as well as introduce some new terms to help us understand some of the incredible tools were are seeing being released today. This week will cover AI, ML, DL, and neural networks. Articles in the future will explore supervised learning, unsupervised learning, CNNs, RNNs, and transformers. Below are the resources this article is based off of. Would highly encourage looking at them.

• AI vs. Machine Learning vs. Deep Learning vs. Neural Networks | IBM
• What is a Neural Network? | IBM
• What are Recurrent Neural Networks? | IBM
• What are Convolutional Neural Networks? | IBM
• MIT Introduction to Deep Learning | 6.S191
• MIT 6.S191: Recurrent Neural Networks, Transformers, and Attention
• MIT 6.S191: Convolutional Neural Networks
• Attention Is All You Need | Research Paper | Google

What is AI, Machine Learning, and Deep Learning

Although, these terms can sometimes be used interchangeably, each interplays into the other in a bucket format. We can think of them in a step process like so: AI is like the school subject of history, Machine Learning is a discipline such as American history, and Deep Learning is a specific topic like 1800s civil war history. Now let’s define each individual term on its own.

AI (artificial intelligence) as defined by Stanford HAI is ““the science and engineering of making intelligent machines”. It includes any machine that behaves in a way that mimics human intelligence. It can be something such as a translator app, your phone camera which can recognize your face, a bot that can play chess, or chatbots such as Gemini and ChatGPT.

Machine learning is an AI subset where the machine is able to learn without explicitly being programmed. An example of a "nondeep" ML model includes your recommendation algorithm on Spotify. The machine learns over time your preferences and can adapt if you start exploring something new. "nondeep" ML models require human intervention in order to help it identify patterns and act accordingly. The songs that you listen to are already categorized into pop, rock, R&B, etc... The recommendation algorithm uses this information to recommend other songs similar to what you've listened to. Another example is a machine that can recognize the difference between an apple and an orange

Deep learning on the other hand can leverage labeled data sets and unlabeled data sets to inform the machine on how to learn. For example, a deep learning model may be able to simply tell if a piece of music is pop, rock, R&B, etc. simply by listening to it and adapt the recommendation algorithm accordingly. There is another difference between ML and DL but to understand it more clearly, let's talk about what a neural network is.

Neural Networks

Neural networks are a subset of machine learning and the backbone of DL models. They are named neural networks because the structure of the model mimics they way our brain functions with its many neurons. Neural networks are comprised of multiple nodes. These nodes make up 3 parts of the networks: the input layer, the hidden layer, and the output layer. Below is a diagram...

The input layer is where our model takes in information. It could be the volume of the music, the instruments being used, the tempo, etc. Then it takes that information and passes it through 1 or more hidden layers. These layers compute the information and allows the computer to understand what it is listening to. It allows it to combine the data of the volume, beat, and tempo together as well as establish relationships between them. Finally, this data is passed onto the output layer which the computer uses to make its final decision on what it is listening to.

Neural networks can have one hidden layer or many hidden layers (thus implying the deep-ness of the model). To understand how neural networks process the information, it involves a little explanation of the math. Let's go through a single layer neural network that processes music as an example.

This is a single layer neural network. We have input layer nodes x1 through xm, hidden layer nodes z1 through zd, and output layer nodes y1 through y2. Let's zoom in on a single hidden layer node to see the neural network in action.

Each input will be feeding a numerical value into the hidden layer node that we have here in orange. For example, the music may be 100 decibels loud, have 5 instruments in it, and is 150bpm. Each of these numbers is multiplied by an individual weight represented by variable "w". Changing these weights is what allows our model to learn and make itself more accurate.

After applying the weights and adding everything together, we add a "bias term" which will help us adapt our function further in preparation for the next step. You can see the bias term below...

The last step is putting our result through an activation function represented by "g()". This function helps turn our output into a nonlinear result (or a number between 0 and 1). If we did not have this activation function, all our outputs would be linear, leaving the model extremely limited (watch MIT Intro to Deep Learning 23:10 to understand this further). One common activation function is the sigmoid function that can be seen below...

The result of all of these processes gives us the output in the form of yhat. The neural network will use the value in yhat to make decisions about what the model is listening to (Ex: yhat could be the probability that the music is rock or not).

So how does the model change the weights and thereby learn? This is done through the process of gradient descent as well as backpropagation. This may be explored in a future article but on a basic level, the model changes the weights bit-by-bit until it reaches a level where it is making the fewest number of mistakes.

All of this is to say that neural networks allows us to use incredibly interconnected math in order to enable machines to learn how to do advanced tasks such as recognizing cars in an image, write essays in the style of Shakespeare, and create images from text. All of this comes from identifying the relationships between words, colors, shapes, etc. that we process on a daily basis without thinking about them.

Neural networks are the advanced form of AI that are the basis for the creation of the many advancements we are seeing in the technology space today. If this article was in a video format, a much clearer explanation could be given, but there are many wonderful resources out there to help deepen one's understanding of these new tech advancements. I highly encourage anyone to dive into the resources above in order to learn how these advanced machines do the incredible things they do.