How do you design the reward function and the discount factor for the actor-critic algorithms?

2 Discount factor

The discount factor is a hyperparameter that controls how much the agent values future rewards over immediate ones. It is usually denoted by gamma, and ranges from 0 to 1. A low gamma means the agent is short-sighted and only cares about the current reward, while a high gamma means the agent is far-sighted and considers the long-term consequences of its actions. For actor-critic algorithms, the discount factor affects the critic network, as it is used to calculate the expected return or the advantage function. Choosing the right discount factor depends on the characteristics of the problem, such as the horizon, the variance, and the stability.

3 Advantage function

The advantage function is a key concept in actor-critic algorithms, as it measures how much better or worse an action is compared to the average action in a given state. It is defined as the difference between the action value and the state value, or the difference between the actual return and the expected return. The advantage function is used to update the actor network, as it provides a more accurate and less noisy gradient signal than the raw reward. There are different ways to estimate the advantage function, such as using Monte Carlo methods, temporal difference methods, or generalized advantage estimation.

4 Policy gradient

The policy gradient is the core algorithm for updating the actor network in actor-critic methods. It is based on the idea of maximizing the expected return by following the current policy. The policy gradient is computed by multiplying the advantage function by the log-probability of the action taken by the actor. The policy gradient is then used to perform gradient ascent on the actor network parameters, which improves the policy by increasing the probability of good actions and decreasing the probability of bad actions. The policy gradient can be improved by using various techniques, such as entropy regularization, baseline subtraction, or actor-critic methods.

5 Value function approximation

The value function approximation is the technique for updating the critic network in actor-critic methods. It is based on the idea of minimizing the error between the predicted value and the target value for a given state or state-action pair. The value function approximation can be done by using either a state value function or an action value function, depending on the type of actor-critic algorithm. The state value function estimates the expected return from a given state, while the action value function estimates the expected return from a given state-action pair. The value function approximation can be done by using either a bootstrapping method or a Monte Carlo method, depending on the type of critic network.

6 Actor-critic variants

Actor-critic algorithms have many variants, each with its own advantages and disadvantages. A2C, or Advantage Actor-Critic, uses a single actor network and multiple critic networks to parallelize the learning process and reduce the variance. A3C, or Asynchronous Advantage Actor-Critic, takes it a step further with multiple asynchronous agents sharing a global network. DDPG, or Deep Deterministic Policy Gradient, handles continuous action spaces and learns from off-policy data. TD3, or Twin Delayed Deep Deterministic Policy Gradient, improves DDPG by using two critic networks to reduce overestimation bias. Finally, SAC, or Soft Actor-Critic, uses an entropy regularization term to encourage exploration and a reparameterization trick to simplify the policy gradient calculation.