MC method and TD method of Estimating returns What about the other network, critic? Because the critic is the one that is going to rank or decide how good that action was, right? Because now you're taking this, given this output—sorry, given this state—you take this action and you look at the final reward, but there has to be something that computes this as well. So, in order to determine whether this action was good or not, should I increase this probability or not, I need to compute this. So, for computing the expected return, given any state and any action, we use the critic network. And again, you might have guessed that the critic network is dependent on... well, you guessed not, because we are computing the value, the expected return of a given state and/or given state-action pair. So, there are two ways of estimating returns, right? We learned there was one MC method, Monte Carlo method, and there was another, TD method, temporal difference method. And one of those will be used for our value-based method. One of those. Later on, you will see a combination of this—we will talk about the combination later. But, so, let's see the pros and cons of MC method. When you... between... and because of this random actions taking place, so in the beginning, they all start the same, right? But in between, there might be random actions, which are different from the other episodes and which might be different from this episode. The random states that take place—sorry, I said the animation story of the random states that take place later on in the episode might be different, and that's why they might have high variance in their expected return. And that's why MC method has a very high variance. On the other hand, it has a low bias, because it's not estimating anything, it's not guessing anything, because they are going all the way until the end, right? So, it has low bias, but high variance. On the other hand, TD method, we don't go all the way until the end of the episode. Given one state at time step T, we just take one step and estimate the total expected return for the whole trajectory from this one time step, right? We don't go all the way until the end. That's why the random actions that might have taken place don't take lesser—or we don't assume, or you don't even consider them. We just make a guess, and we assume that our next guess will also be closer to our current guess. It won't be that far from our current guess, so there's not much difference in our guesses. Again, I am saying "guesses" because we are guessing the total expected return given this next state, right? So, it's going to be a reward for going in that state plus gamma times GT+1. Now, GT+1, again, all you know, it's the sum of all the cumulative rewards after that time step until the end. So, we are guessing that the sum of all the rewards until the end, discounted here, won't be much different for this episode and won't be much different for the next episode. So, they are all close to each other. So, we get low variance, but it will have a high bias, because again, we are guessing that there won't be much variance. We are guessing this expected return. So, TD method has low variance but high bias, whereas MC method has high variance but low or zero bias, almost zero bias. So, depending on our need, we need to use either MC method or TD method to compute values for our critic. So, we use one of these in our critic to compute the action value or state value function.