Dueling Q-network So, Dueling Q-Network is just another, again, yet another way of doing the extension for random empirical kind of technique, which is to combine two streams to produce the Q-function. So, one is to use the state values, the other is to use the advantage function. Actually, in the next slide, I'll show you some kind of motivation why we do this. So, this way, then the network learns the state values for which actions have no effect. Also, the dueling architecture can more quickly identify the correct action in the case of redundancy. So, you can see this example here. Actually, on the left-hand side, you'll see this so-called traditional DQN and the dueling Q-Network DQN. The upper one, let me show you here, is the traditional one, and this is the dueling one. The traditional one is using, so see, the traditional one was you only have like the Q-value, but the dueling one is combining two streams to produce the Q-network: one is for state value and the other is for advantage function. So then, the example is showing here that they have, you know, you can see this is the value, and this is the advantage. So what they're saying is the value stream learns to pay attention to the road, whereas the advantage stream learns to pay attention only when there are cars immediately in front. Based on this kind of observation, they say, "You know, can we combine both for the learning?" Therefore, you have two streams, like we're saying here, you will have two streams to produce the Q-function. Although, actually, I'm not really understanding the principle of this other than this empirical observation to see in this specific game design or, you know, game design of RL agent. The definition of the advantage function \( A \) is defined as this: \( A = Q - V \), where \( A \) satisfies the expectation that \( A = 0 \). This means the agent, at the current state \( S \), takes this action, and the agent moves from this state to this point. It cannot move to the next point without the environment's information because it doesn’t know where it should land. From this point, we can, step by step, calculate the return from this point until we reach the terminal state and then come back to get the return. If this Q value is maximum, then this action \( A \) is the optimal action, and then we calculate the optimal value function. However, right now, we have this Q, but the Q is at this point, meaning it’s just at the state \( S \). After we select this action \( A \), what we want is for the agent to stand here and describe the actions—not from here. We can find the relationship between the state value function and the actions at this point. To do so, that’s why we define the advantage function \( A \). Then, using this advantage function, we can modify the Double DQN. We keep the target network the same, but we only need to modify the Q network. We keep the same input, the same output, and the same convolutional module. We only insert two output modules: one for the state value function \( V \), and another one for the advantage function \( A \). Then, we use this \( V \) and \( A \) to combine and find the output \( Q \) for all the possible actions. That’s the idea of the advantage function and what the dueling Q network does.