BxD Primer Series: A3C Reinforcement Learning Models

Hey there ??

Welcome to BxD Primer Series where we are covering topics such as Machine learning models, Neural Nets, GPT, Ensemble models, Hyper-automation in ‘one-post-one-topic’ format. Today’s post is on?A3C Reinforcement Learning Models. Let’s get started:

The What:

A3C, is an evolution from Actor-Critic (A1C) → Advantage Actor-Critic (A2C) → Asynchronous Advantage Actor-Critic (A3C). We will cover it in same sequence.

Actor-Critic Architecture:

In actor-critic architecture, there are two main components: the "actor" and the "critic".

The actor is responsible for selecting actions based on current state of the environment. It learns a policy that maps states to actions, often represented by a neural network. The policy can be stochastic, meaning that it outputs a probability distribution over possible actions, or deterministic, meaning that it outputs a single action.

The critic is responsible for estimating value of each state. It learns a value function that predicts the expected future reward of being in a given state, often represented by another neural network. The value function is used to evaluate the quality of policy by estimating the expected return from following it.

During training, actor and critic work together to improve the policy. Critic provides feedback to actor by estimating the value of each state and providing a error signal, which indicates how much better the expected return could have been if actor had taken a different action. Actor then updates the policy to increase probability of actions that lead to higher value states.

Actor-critic architecture can be used with both model-based and model-free RL approaches.

Here is how you would define and build an actor-critic model:

? Define the environment?that the agent will interact with. Specify the state space, action space, and reward function.

• State space is set of possible states that the agent can observe
• Action space is set of possible actions that the agent can take
• Reward function defines the reward that agent receives for each action in each state.

? Build the actor network: It takes state as input and outputs a probability distribution over possible actions.

• Output layer of actor network should have same number of units as the action space.
• And should use a softmax activation function to ensure that the outputs sum to 1.

? Build the critic network: It takes state as input and outputs an estimate of expected future reward.

• Output of critic network can be a single scalar value, called state-value (V function).
• Or it can be a vector of values, called action-value (Q function).

? Define loss functions: Actor and critic networks are trained using different loss functions.

• Loss function for actor depends on the specific type of policy optimization algorithm being used. Most common policy optimization algorithms used are A2C and A3C.
• Critic loss function is based on the mean squared error between predicted value of current state and actual value of current state.

? Train the model: Actor-critic model is trained using a combination of gradient descent (GD) and error function (we assume TD error).

• Actor network takes the current state as input and outputs a probability distribution over possible actions.
• An action is sampled from the probability distribution and executed in the environment.
• The environment returns the reward signal and the next state.
• Critic network takes the current state as input and outputs an estimate of state-value (V) function or action-value (Q) function, depending on type of actor-critic algorithm used.
• TD error is computed using the reward signal, the next state, and the estimated value of current state.
• Critic network parameters are updated using GD to minimize TD error.
• Actor network parameters are updated using policy optimization algorithm, which is computed using TD error as a baseline.

? Evaluate the model: Once the model is trained, it is evaluated on a test set of episodes to measure its performance. This involves running the agent through environment without any further training, and measuring the average reward over a fixed number of episodes.

Note 1: If the problem requires agent to take explicit actions, then action-value (Q-function) is more appropriate. For example, in a game where agent needs to decide which move to make, a Q-function is used to estimate the value of each possible move in a given state.

Note 2: If the problem requires agent to make decisions based on value of being in a particular state, then state-value (V-function) is more appropriate. For example, in a robotics task where agent needs to navigate through a maze, a V-function is used to estimate the value of being in each location in maze.

Note 3: Variety of policy optimization algorithms can be used for actor, based on problem.?We are covering A2C and A3C in this edition:

Advantage Actor-Critic (A2C) Models:

A2C combines the Actor-Critic architecture with advantage function.

? At each time step?t, actor network takes the current state?s_t?as input and outputs a probability distribution over possible actions?π(a_t | s_t;?θ), where?θ?represents weights of actor network.

? Critic network takes the current state?s_t?as input and outputs the expected cumulative reward for that state?V(s_t;?ω), where?ω?represents weights of critic network.

? Advantage function?A_t?is defined as the difference between actual cumulative reward?r_t?obtained after taking action?a_t?and the expected cumulative reward?V(s_t;?ω)?predicted by critic network:

Where,?γ?is discount factor, which determines the relative importance of future rewards.

? Loss function for the actor and critic networks is defined as follows:

Where,?T?is total number of time steps,?G_t?is discounted sum of rewards from time step?t?onwards, defined as:

First term of loss function is the policy gradient, which encourages the actor network to increase probability of actions that result in a higher advantage.

Second term is mean squared error between predicted value and actual value, which encourages the critic network to better estimate the expected cumulative reward.

? Gradients of loss function with respect to actor and critic network weights are given by:

? These gradients are used to update the weights of actor and critic networks using stochastic gradient descent:

Where,?α_θ?and?α_ω?are the learning rates for actor and critic networks, respectively.

Note 1: Advantage function allows the algorithm to better estimate state-value of each action and improve the performance of actor network. Read more?here.

Note 2: If multiple agents are used in parallel, A2C will have synchronous training, where all agents collect experiences from environment and then update the global actor and critic networks simultaneously. This approach requires all agents to wait for each other before updating the global networks, which can be slow for large-scale problems.

Asynchronous Advantage Actor-Critic (A3C) Models:

A3C turns A2C into an asynchronous training process while the general steps remain same. Asynchronous training allows multiple agents to collect experiences and update the global networks independently and asynchronously. This approach significantly reduces the time required to collect experiences and update the global networks, making it more efficient for large-scale RL problems.

A3C maintains a single global actor and critic network and each agent has its own copy of the network that is updated independently and asynchronously. After a certain number of time steps, the agent sends its gradients to global network, which is updated by averaging the gradients from all independent agents.

Where,?N?is the number of agents.

The Why:

Reasons to use A3C:

• Effective in solving a wide range of tasks, including Atari games, robotics, and continuous control tasks.
• Time efficient compared to other deep reinforcement learning algorithms because of its parallel training approach, which allows for more efficient use of computing resources.
• Can learn optimal policies for agents in complex environments, even with high-dimensional input.
• Can learn from raw sensory input without the need for feature engineering.
• Able to learn policies that generalize well to new environments.

The Why Not:

Reasons to not use A3C:

• Difficult to implement and tune, especially for beginners.
• Parallel training require specialized hardware and software infrastructure to implement efficiently.
• May suffer from instability during training in environments with sparse rewards.
• Require a large number of training episodes to converge to an optimal policy.

Time for you to support:

1. Reply to this article with your question
2. Forward/Share to a friend who can benefit from this
3. Chat on Substack with BxD (here)
4. Engage with BxD on LinkedIN (here)

In next coming posts, we will cover four more Reinforcement Learning models: Q-Learning, Deep Q-Network, Genetic Algorithm, Multi-Agent

Let us know your feedback!

Until then,

Have a great time! ??

#businessxdata?#bxd?#A3C #Reinforcement #Learning?#primer