Understanding Back Propagation in human terms

Deep learning neural networks and their fundamental building block, the perceptron, serve as a mathematical model inspired by the biological structure known as neurons and the complex interconnectedness they exhibit. This post endeavors to establish parallels between the deep learning process, specifically backpropagation, and the high-level functioning of the human brain.

Perceptrons like neurons react based on the values of the input signal. The perceptron calculates the weighted sum of the input values and additionally applies an activation function to normalize the output as per the needs. Each layer of a multilayer perceptron uses the output of the previous layer. The learning process happens as the weights associated with the input values are adjusted as the neural network goes through the training process. This adjustment happens as there is a desire to reduce the loss between the prediction and the true target values provided as part of the training data.

This is where the concept of backpropagation comes in. Backpropagation is the key to neural networks learning and although the math involves differentials to calculate gradients; this learning process is similar to the human brain’s ability to arrive at the correct solution to many a problem when one sleeps over them. How many times have you gone to bed frustrated trying to solve something that at the moment appeared complex, and when you woke up in the morning you already had the solution figured out. During REM phase of sleep our brain pieces together as well as cleans up the information available to it resulting in better answers/ thoughts akin to improved predictions once the neural network has had a chance to adjust/ back propagate information based on the loss calculations during the forward pass.

Granted, backpropagation can happen at every input cycle but that may not be efficient. Imagine us going into a deep REM-like phase every time we hit a roadblock. Mini-batches or in general batching process is how we gain efficiency in adjusting weights similar to brain processing plethora of information and adjusting its assessment of the said information.

Ultimately, a lot of efficiency in neural networks can be achieved by just paying additional attention to how our brains perceive, categorize and process information throughout the day.