Backpropagation: The blame game
Today, I recorded a very important lecture in the “Neural Networks from Scratch” series.
It is called Backpropagation from scratch - on a single neuron.
When most students think of backpropagation, they have the following image in mind:
Students vaguely know that it involves gradients and partial derivatives and chain rule.?
However, more than 90% of ML students are not able to write the backpropagation code from scratch.
Here is a snippet of my whiteboard as I was making this lecture:
Once you write it on the whiteboard and understand it, you realise that backpropagation is actually simple.
It is a cool trick which relies on the chain rule of calculus.
I think of backpropagation as a way to calculate how much each weight and bias should be blamed for the loss.
If you think of backpropagation as passing the blame, it becomes very intuitive to understand.
Here is a video we recorded on the topic:?
Enjoy!
Full Stack Engineer | Node.js | React.js | AWS | TypeScript | CI/CD | REST API | MongoDB | Docker | MySQL | GraphQL | Nest.js | Next.js | Jest | Mocha | PostgreSQL | SQL | Kubernetes | TDD | Express.js | Redux.js | Git
8 个月I have a question about the backpropagation algorithm in neural networks: In any implementation, when calculating the gradient of a unit in the hidden layer, do I always need to consider the gradients of all units in the next layer? Specifically, does this involve summing these errors? In other words, does the gradient of a unit in the hidden layer always depend on the errors of all units in the next layer? For instance, if my neural network has just 2 layers: a hidden layer with 3 units and an output layer with 2 units, to calculate the delta of each unit in the hidden layer, do I need to sum all errors in the next layer (in this case, the errors of the output layer)?