The Definitive Guide to Activation Functions! (Part-1)

The purpose of this article helps you to understand the last layer of neurons (activation function and loss function) used in a neural network depending on your business goal!

This article assumes the reader has a knowledge of machine learning, deep learning concepts.

Why do we use the activation function?

First, let us discuss the architecture of Perceptron. It has linear and non-linear functions.

Each perception has two-part.

1. Summation.
2. Activation Function.

Linear :

• Linear?is?Simply Addition?and?Multiplication. If you are familiar with?Linear Regression, it is just an add and multiplication term y = mx + c.
• In further, Linear refers to the?Straight Line.
• It can able to solve linear problems like separating the line and linear regression.
• Important thing?Never use Linear equations to solve non-linear problems.

Non-Linear:

• Except for linear equations, remaining all operations are considered the?non-linear operation.
• Non Linear does not have a?straight line. Look at the equation: 3x2 + 2x + 1 = 0.
• It can able to solve more?complex?problems and it adds non-linearity to our function.
• Some examples are sigma, tanh - you will see this later.
• Important thing?Never use Non-linear equations to solve linear problems.

Each perception has two functions: linear and non-linear. The work of linear function is that it simply adds the input with weight and adds bias to that.

The architecture of ANN:

• The input layer comprises independent variables (x) and the output layer comprises the output values regarding independent variables.
• In hidden layers, each circle refers to a single perceptron. In each perceptron, two processes will occur.

1. Summation.
2. Non-Linear function.

Summation:

• Adding weight and input and summing with each other.

Non-Linear Function:

• In simple words, the input will pass to the perceptron (each node), inside the perceptron the?input and weight?will be added and the output of the sum is passed to the nonlinear function (activation function) and it will give the entire output value to next hidden layer or node.

Working of single Perceptron:

This is a Single Perceptron, all the hidden nodes are perceptron, by use of this, the Network can learn from mistakes and reduce the loss functions.

In the next part, we will discuss what are all the activation functions and the types of activation functions.

+

Name: R.Aravindan

Company: Artificial Neurons.AI

Position: Content writer