Advantage Actor-Critic Popular actor-critic networks, ATC, also known as asynchronous advantage actor-critic (A3C), so the asynchronous part is that there are many, many agents, and each agent has its own copy of this network and its own copy of the environment. Now, this network here, I have shown only one network, and the point of showing this is that this network has two heads: one for the actor and one for the critic, and they share their parameters and train at the same time, the actor and critic. But this one network could also be two networks, one separately for the actor and one separately for the critic. Okay, so what happens is there are many agents, and they have their own local copy of the current version of the actor and critic network. Now, using the actor network and the environment, they just collect these experiences. You might remember this from the replay buffer. So, they are collecting these experiences, and then they use these experiences to update their own local version of the critic network. Right? Then, they use that critic network and update the global network. So, that's done by one agent. It uses, first of all, its own copy of the actor and critic network, generates these updates for the critic network, and then it just updates the one single global network. This is done by all these agents, and they are all doing this asynchronously. That is, they are not correlated. They are just doing it randomly: collect, update, and then update this. Collect, update their local, and then update the global. There is only one copy of the global actor network and one copy of the global critic network, but each agent has their own copy of the local actor-critic network. Using those, they are generating this, updating their critic, and then dumping it on the global networks. And they are doing this all at the same time. That’s what is called asynchronous, and the point of being asynchronous is to avoid any correlation. So, in your critic network, instead of just computing V(pi) for st+1, in some algorithms you see [Music] some networks computing V(pi) for more steps. So, they don't rely on just one step; they will compute over two steps, or three steps, or some n steps. That was called n-step bootstrapping. So, in this version, suppose it's going to be either one step and then guessing, or until the end, or one or two steps and then guessing until the end, or so it's going to be either one of these, and it's called n-step bootstrapping because n could be one or until the end. And this is done to deal with the bias in the future. In this slide, I will show another way, which is even better, which is the combination, actually, which is a modification to n-step bootstrapping. So, we already discussed this. Another popular actor-critic based algorithm is the one-edge actor-critic. As you can see, it is very similar to A3C, but again, the only thing missing here is the asynchronous nature. So, yeah, it's not an asynchronous updating of the global copies of the actor-critic networks, but everything is the same. But what they do before updating the local—and, sorry, the global—the actor and critic networks is that they wait. Wait, everyone waits until all agents have accumulated experiences and updated their networks, and then they'll update everything together. So, it's a synchronized updating. That's why it doesn't have that asynchronous element here. Yeah, so that's the advantage over A3C. In this method, you can use a CPU because not much parallel updating is taking place, whereas in A3C, you need a GPU.