Steepest Decent v's ADAM

The Steepest Descent method is an optimization technique used to find the minimum of a function. Imagine you're hiking downhill towards the lowest point of a valley. In this method, you always move in the direction of the steepest descent, or the negative gradient, to reach the minimum point.

Key Equations

1. Gradient (?f(x)\nabla f(x)?f(x)): The vector pointing in the direction of the greatest rate of increase of the function f(x).
2. Update Rule: x_{k+1} = x{k} ? α?f(x_k) x_k: Current position α: Step size or learning rate, a small positive number determining the size of the step.?f(x_k): Gradient at the current position.

Simple Example

If you're trying to minimize f(x)=x^2 , the gradient is ?f(x)=2x. Starting from an initial guess x_0, you move in the direction of the negative gradient, ?2x_0, to find the minimum at x=0.

This method is fundamental in machine learning and finance for minimizing cost functions or optimizing portfolios.

This understanding is key in optimization, particularly in methods like Steepest Descent, where the negative gradient points to the steepest descent direction to minimize the function.

1. Prediction Vector in Finance:

In finance, the prediction vector y\mathbf{y}y could represent forecasted asset prices or returns, while yactual\mathbf{y}_{\text{actual}}yactual represents observed values. The goal is to minimize the error between these predictions and actual values.

2. Objective Function:

The objective function f(x)f(x)f(x) could be the sum of squared errors (least squares):

where y_i is the predicted value and yactual, is the actual value.

3. Gradient Calculation:

The gradient ?f(x_k) is calculated at the current parameter vector xkx_kxk (which could represent model parameters). For least squares, the gradient is:

This vector indicates the direction of the steepest increase in error.

4. Steepest Descent Update Rule:

To minimize f(x), we move in the direction opposite to the gradient:

where η is the learning rate. This update rule iteratively adjusts xkx_kxk to reduce the prediction error.

Example:

Suppose our model predicts stock prices and we use parameters x_k to adjust our model. If the actual price of a stock is $100 and our prediction is $90, the error is $10. The gradient calculated with respect to our model parameters indicates how much each parameter contributed to this error. Using steepest descent, we adjust these parameters to minimize the difference between predicted and actual prices.

This combination allows for iterative improvement of financial predictions, reducing the error in forecasts and thereby improving decision-making based on these predictions.

The steepest descent algorithm can fail to converge or converge very slowly under several conditions:

1. Poor Choice of Learning Rate (η\etaη):
2. Non-convex Cost Function:
3. Ill-conditioned Problem:
4. Incorrect Gradient Calculation:
5. High Dimensionality:
6. Plateaus or Saddle Points:
7. Noise in the Data:

For many modern applications, Adam (Adaptive Moment Estimation) is a more suitable optimization algorithm than steepest descent. Adam combines the best features of the AdaGrad and RMSProp algorithms to handle sparse gradients on noisy data. It maintains per-parameter learning rates that are adapted based on the first and second moments of the gradients. This method typically converges faster and more reliably than steepest descent, especially on complex, non-convex problems, making it popular in deep learning frameworks.

Adam Update Rule

Gradient estimation: Compute gradients of the objective function.

First moment (mean) estimate

Second moment (uncentered variance) estimate:

Bias correction:

Parameter update:

Where:

• g_t is the gradient at time step t.
• m_t and v_t are the estimates of the first and second moments.
• β_1 and β_2 are decay rates for the moment estimates.
• η is the learning rate.
• ? is a small constant to prevent division by zero.

Adam's adaptive learning rates and momentum components make it highly effective for large-scale machine learning and deep learning tasks.