Here are marked as blue, so you take the gradient of the J omega, the objective function, right? So, and it turns out if you do REINFORCE, then you would actually calculate the gradient as the expectation of the gamma^t * r_t. r_t is the return, gamma^t is the t-th power of the discount factor, and then the del, you know, the gradient with respect to a logarithm of the pi. Okay, so this is what we learned about REINFORCE, and I kind of talked about why it is a logarithm of the gradient. Uh, sorry, the gradient of the logarithm. So, that's REINFORCE. But a modification of that is, instead of using, uh, you know, the return, what we can do is now in actor-critics, since we know the value, we could use, since we have an estimation of the Q functions, we can replace the return by the Q function. Okay, so now, you know, the gradient estimation is not the gamma^t * r_t and the gradient of the logarithm pi, but instead is the gamma^t * Q_pi(s_t). Okay, so this is a replacement of the return. Why? Before, we didn't have the, you know, Q function because we were doing a policy gradient, and we set that position and did not explicitly work with the Q function. But now, in the actor-critic scenario, we can because we have a critic that is evaluating the Q for us, so we can just replace the r_t (the return) with the Q_pi(s_t). Okay, so, um, I guess, you know, this is not very difficult to understand. Then, more importantly, you know, many of you might have heard about the advantage function, but this is a very, very important method for reducing variance. You know, we know that the high variance is indeed a, you know, very serious problem in reinforcement learning. So, one further modification to REINFORCE in this actor-critic form is to have this so-called advantage function. So, what is the advantage function? The advantage function is the difference between the Q and the V. So, see, here is the thing. So, first of all, the advantage function should be a function dependent on s_t, the state, right? Because for different states, the advantage function will be different. But more importantly, the advantage function is a function of the action. So, what is the action? So, no matter what the action is, the advantage function is saying if I take that action, how much I can improve by taking that action, then following pi. You know, uh, so the difference between the value... I don't know, let me say that again. So, the advantage, I could hope to improve my value function by taking action a_t at state s_t, compared with following pi at state s_t. Does this make sense? Okay, yeah, so this, so, you know, why it's called the advantage function. It is basically saying I wanted to take action a_t instead of following pi, and I might get an improvement. Definitely, the advantage function can be negative, right? If you're taking the bad action, of course, you know, it could be worse than the value following pi. But anyway, so we are looking at an advantage function instead of a Q function. Why? You know, basically, they're saying you can replace the Q with the A, with the advantage function. Why? The reason is that this part, you know... so, you know, if you replace that with the advantage function, you would have an extra term, which is the gamma^t * V_pi(s_t) and the gradient of the logarithm of pi, right? So, the good thing about that is the expectation is actually zero. Therefore, optimizing, sorry, you know, having the update function using the right term... So, I don't know if you can see my screen, so, you know, doing the update using the gradient estimated using the red term is the same as this term here. And because this term is actually zero in expectation, so this is why, you know, we use the advantage function. Physically, it has a meaning: it's telling me, you know, how much improvement in terms of value I can expect by taking action a_t. However, mathematically, um, you know, you can prove that by using the advantage function, you can reduce variance, and then, you know, the methods will be more robust. Okay, so this is, you know, a very important, I guess, you know, my technique, uh, that people use a lot in reinforcement learning: using the advantage function and using the fact that the expectation of the value function and the gradient... and so, you know, using this fact, can you see my laser pointer? Using this fact that this expectation is zero, uh, is very important. Okay, yes, the advantage function basically is a variance reduction trick.