The Backpropagation Algorithm!

The Backpropagation Algorithm!

The backpropagation algorithm is the heart of deep learning! That is the core reason why we can have those advanced models like LLMs.

In a previous video, we saw we can use the computational graph that is built as part of deep learning models to compute any derivatives of the network outputs with respect to the network inputs. Now we are going to see how we can use this computational graph to get the network to learn from the data by using the backpropagation algorithm. Let's get into it!


Watch the video for the full content!

The goal of a neural network is to generate an estimate of the target we are trying to predict. We use a loss function to compare the target to its estimate. The optimization problem is about minimizing the loss function.

Typical loss functions are the log-loss and the mean squared error loss:

To minimize the loss function, we take the gradient of the loss function with respect to the network parameters and find the zeros of the resulting function.

But solving this equation can be very hard, so instead we use optimization techniques like the gradient descent algorithm. We update the parameters following the gradient in the direction where the loss function decreases.

Until we reach a local minima of the loss function.

To apply the gradient descent algorithm, we are going to use the computational graph. We first compute the graph in the forward pass.

And we can back-propagate the gradients of the loss function for each computational block and node in the graph by using the chain rule in the backward pass.

Now that we have the gradient of the loss function for all the parameters in the graph, we can apply one step of the gradient descent algorithm.


Articles You May Have Missed!

Hardeep Chawla

Enterprise Sales Director at Zoho | Fueling Business Success with Expert Sales Insights and Inspiring Motivation

7 个月

The backpropagation algorithm truly is the heart of deep learning! It's amazing how it powers advanced models like LLMs!

Daniel Svonava

Vector Compute @ Superlinked | xYouTube

9 个月

How do you see the role of backpropagation evolving as we push towards more advanced AI architectures? Are there any promising alternatives on the horizon?

Matthew Yeseta

Generative AI LLM LangChain Fine Tuning, Director AI ML Data Science, Manager Data Science, 150% branding/125% Digital, revenue steams, CTV Segmentation CDP, CV YOLO, Snowflake, Engineering AI ML Manager

9 个月

I be ready to code this algorithms in python

Naresh Neelakantan

Blitzscaling Innovation from low-to-high Tiers for many OEM MY programs | ex-Senior Director & Head | ~20 Years in Automotive and SDV/SDX with multiple organizations in AI, Cybersecurity, Energy, Semiconductors, XaaS ??

9 个月

Reaching the error minima descending and adjusting through all the epochs. #awesomeAI #understandingai

要查看或添加评论,请登录

Damien Benveniste, PhD的更多文章

社区洞察

其他会员也浏览了