The Backpropagation Algorithm!
Damien Benveniste, PhD
Founder @ TheAiEdge | Follow me to learn about Machine Learning Engineering, Machine Learning System Design, MLOps, and the latest techniques and news about the field.
The backpropagation algorithm is the heart of deep learning! That is the core reason why we can have those advanced models like LLMs.
In a previous video, we saw we can use the computational graph that is built as part of deep learning models to compute any derivatives of the network outputs with respect to the network inputs. Now we are going to see how we can use this computational graph to get the network to learn from the data by using the backpropagation algorithm. Let's get into it!
Watch the video for the full content!
The goal of a neural network is to generate an estimate of the target we are trying to predict. We use a loss function to compare the target to its estimate. The optimization problem is about minimizing the loss function.
Typical loss functions are the log-loss and the mean squared error loss:
To minimize the loss function, we take the gradient of the loss function with respect to the network parameters and find the zeros of the resulting function.
But solving this equation can be very hard, so instead we use optimization techniques like the gradient descent algorithm. We update the parameters following the gradient in the direction where the loss function decreases.
领英推荐
Until we reach a local minima of the loss function.
To apply the gradient descent algorithm, we are going to use the computational graph. We first compute the graph in the forward pass.
And we can back-propagate the gradients of the loss function for each computational block and node in the graph by using the chain rule in the backward pass.
Now that we have the gradient of the loss function for all the parameters in the graph, we can apply one step of the gradient descent algorithm.
Enterprise Sales Director at Zoho | Fueling Business Success with Expert Sales Insights and Inspiring Motivation
7 个月The backpropagation algorithm truly is the heart of deep learning! It's amazing how it powers advanced models like LLMs!
Vector Compute @ Superlinked | xYouTube
9 个月How do you see the role of backpropagation evolving as we push towards more advanced AI architectures? Are there any promising alternatives on the horizon?
Generative AI LLM LangChain Fine Tuning, Director AI ML Data Science, Manager Data Science, 150% branding/125% Digital, revenue steams, CTV Segmentation CDP, CV YOLO, Snowflake, Engineering AI ML Manager
9 个月I be ready to code this algorithms in python
??Blitzscaling Innovation from low-to-high Tiers for many OEM MY programs?? ex-Senior Director & Head??~20 Years in Automotive and SDV/SDX globally?? AI, Cybersecurity, Energy, Semiconductors, XaaS, AUTOSAR, Linux, IoT
9 个月Reaching the error minima descending and adjusting through all the epochs. #awesomeAI #understandingai