Actor Critic Method So, actor-critic method is basically an intersection of value-based methods and policy-based methods. Actor-critic: these are two networks. One is called an actor, and another is called a critic. They have not identical networks in DQ, and we saw them as identical. But in this case, in DQ algorithm, they were identical, but in this case, they might not necessarily be identical networks. The actor network determines the actions, given an input of state or state-action pair. So, as you might have guessed, this actor will be based on policy-based method, and a critic method determines the value of taking that action. So, this critic network will be based on value-based method. So, actor-critic method is just a combination of policy-based method and value-based method, where the actor follows the policy-based method, and the critic, the network that computes the value of a particular action, follows a value-based method. Very straightforward, simple stuff. Now, we saw reinforced in policy-based methods. Like, so this is an example of why policy-based method is good. An actor, an agent that follows a policy-based method, will be a good actor. Yeah, because again, as I explained that here, the agent will look at the final outcome. So, suppose this is the final score, and depending on that, it will either increase or decrease the probability of these actions. So, again, to recap, the agent first collects a number of episodes, and then it is following a stochastic policy. And suppose during training, you go back to training and you say, "Okay, when I saw this state, I took left action because it had higher probability, and since I took this action, it turned out to be good for the agent in the end, and the agent got agent 1 the episode." That means this left policy is good, given this state. Given this state, so I should increase its probability even more and decrease the probability of others because I don’t want to risk taking any other action, given this state. Same, the agent will do for other outputs. It will increase the probability of the action that led to a good reward and vice versa for a bad reward. And this is based on policy-based methods, so that's why policy-based method can be a good actor. Actor-critic is where we use a fast-learning policy-based method as an actor. For example, REINFORCE is just one example of a policy-based method. Again, I want to emphasize that, and I am just using REINFORCE as an example here. So, use a fast-learning policy-based method as an actor, which estimates actions better, and use a TD method-based critic to compute the action value or state value and reduce the variance. So, we get the best of both networks, the best of both concepts. That's what the actor-critic algorithm is, or an agent based on the actor-critic method does: it uses the core network for this purpose and the other network for that purpose, and combines them. And it's better, allowing faster and more stable learning. So, these are two networks: one is an actor and another is a critic. The actor will generate a policy given a state, and it might be a deterministic policy or a stochastic policy. It can also be a continuous policy, which we'll see later. A critic, given a state, computes the state value of the state. This could also be a state-action pair; I could also input a state-action pair here if it's a stochastic policy, and then this becomes Q. So, this becomes an action-value function. So, the overall basic underlying principle is that the actor takes the state, gives out a policy, and the critic will generate the action or state value given a state or state-action pair. So, like, what is the process? So, the actor, given a state, gives out an action, and this action then goes into the environment. The environment gives out a reward and the next state. Now, we need to compute the TD target.