Introduction to Neural Networks

Intro

In this article, my attempt is to summarize the intuition behind the neural networks for a higher level understanding.

Why Neural Networks?

When we want to represent data, we choose various functions. E.g. Linear Function, Quadratic, Cubic and many many more.

The below data set can be separated using a linear function

However as the data distribution gets complex, the patterns become difficult to identify and also impossible to represent by a single functions. In the below distribution, the data boundary cannot be represented by a single function

However several non-linear functions can be combined to produce these results. Linear functions cannot be used because when many linear functions are combined, it still leads to a linear function.

Here is an example of several non-linear functions combined. The final desired function in red is able to represent a more complex boundary than its individual functions could.

Neural Networks use non-linear functions like ReLU and Sigmoid. So with similar concept as above, the graph in figure 3 can be imagined as below using non-linear ReLU function (Theoretical graphical representation) where the green graph represents several ReLU functions combined to deliver the refined boundary.

Computing this graph manually will require a lot of effort and as the boundaries get more complex this would demand even more time.

This is where Neural Networks come in. Neural Networks are known as Universal Function Approximators. This means that a neural network can essentially represent any function. By being able to do this they are able to represent any pattern that can be imagined. This is why neural networks can learn patterns in images, language, data and so on.

How do Neural Networks do this?

The building block of a neural network is a Perceptron. It represents a single neuron in the Network. It takes several inputs which are modified with weights and biases and then a non-linear function like ReLU or Sigmoid is applied to generate an output. This single neuron will contribute one Function in the entire scheme of pattern recognition.

With each neuron we learn partial function. When multiple of these neurons are combined we get the full function that we wanted to learn.

Some patterns are even more complex like character recognition and require more hidden layers to learn those functions. The below network now has 2 hidden layers to learn more involved patterns.

How do Neural Networks Learn?

Begin - The Neural network starts with an initial assignment of weights and biases.

Forward Propagation - The weights and biases are combined and the final output is generated.

Loss Calculation - The output is compared with the desired output and the loss is calculated. Loss is the deviation from what the actual output is.

Optimization - The Optimizer algorithm decides the next change that needs to be made so that the loss will be reduced.

Back Propagation - The updated weights are propagated back and then the forward propagation is done again

End - The above process is repeated till the loss is minimized to an acceptable tolerance.

Outro

I hope this provided a brief introduction to get started into the vast world of Neural Networks.

Please contact me for any updates or corrections. I am happy to make this better with your feedback.

Credits and References

1. Supervised Machine Learning: Regression and Classification by Andrew NG https://coursera.org/share/f1160ba8e2f72c73b50cafff891ea67e
2. MIT Introduction to Deep Learning by Alexander Amini https://youtu.be/QDX-1M5Nj7s?si=wFUhRnnkhssXZbhu
3. Why Neural Networks can learn (almost) anything by Emergent Garden https://youtu.be/0QczhVg5HaI?si=Cls6wwDdui4rTAUV