Top 10 Activation Function's Advantages & Disadvantages

Sigmoid:-

Normally used as the output of a binary probabilistic function.

Advantages:

-> Gives you a smooth gradient while converging.

-> One of the best Normalised functions.

-> Gives a clear prediction(classification) with 1 & 0.

Disadvantages:

-> Prone to Vanishing Gradient problem.

-> Not a zero-centric function(Always gives a positive values).

-> Computationally expensive function(exponential in nature).

Tanh:-

Normally used as the input of a binary probabilistic function.

Advantages:

-> Zero-centric function unlike Sigmoid.

-> It is a smooth gradient converging function.

Disadvantages:

-> Prone to Vanishing Gradient function.

-> Computationally expensive function(exponential in nature).

RELU:-(Rectified Linear Unit)

Advantages:

-> Can deal with Vanishing Gradient problem.

-> Computationally inexpensive function(linear in nature).

Disadvantages:

-> Not a zero-centric function.

-> Gives zero value as inactive in the negative axis.

Leaky RELU:-

It is the same as of RELU function except it gives some partial value(0.01 instead zero as of RELU) in the negative axis.

ELU:- (Exponential Linear Unit)

Advantages:

-> Gives smoother convergence for any negative axis value.

-> For any positive output, it behaves like a step function and gives a constant output.

SoftMax:-

Normally used as the output in multi-class classification problems to find out different probabilities for different classes(Unlike Sigmoid which is prefered for a binary-class classification).

PRELU:- (Parametric RELU)

The advantage of PRELU is it has the learning parameter function which fine-tunes the activation function based on its learning rate(unlike zero in the case of RELU and 0.01 in the case of Leaky RELU).

SWISH:-

Also known as self gated function. This activation function is one of the kinds that is being inspired by the use of the Sigmoid function inside an LSTM(Long Short Term Memory) based network.

Advantages:

-> Can deal with Vanishing Gradient problem.

-> The output is a workaround between RELU and Sigmoid function which helps in normalising the output.

Disadvantage:

-> Computationally expensive function (as of Sigmoid).

MaxOut:-

Also known as the Learnable Activation Function.

It has all the advantages of a RELU function but at the same time do not have its disadvantages.

SoftPlus:-

Advantages:

-> Convergence of gradient is smoother than RELU function.

-> It can handle the Vanishing Gradient problem.

Disadvantage:

-> Computationally expensive than RELU(as of exponential in nature).

Thanks for going through this article. This is just a brief of the frequently used 10 activation function's advantages and disadvantages. Although there is a lot to cover about each of the activation functions, I hope this was meaningful to all of you.

I will be sharing my knowledge every now and then based on my availability.

Peace :)