Impact of learning rate in prediction with five layer neural network and exponential decay
We saw the importance of value learning rate in previous post: if we choose non optimal value for the learning rate, our model will not have acceptable prediction. The problem is choosing learning rate is time consuming and a trial and error process. Rather than choosing a constant value, there are some methods to choose learning rate to prevent trial and error. Exponential decay is one of the most famous methods which automatically converge learning rate to the best value. In this method, we just need to define the initial value of learning rate and the least value for it and decay speed. [read more...]