Double DQN So, the Double DQN, proposed in 2015, is to remove this upward bias caused by the maximum of the Q(s', a') θ. So, the θ⁻ is causing some kind of bias. The idea is to produce two Q-networks. Before, we only had one Q-network. Now, you can have two Q-networks: one is the current Q-network, and the other is the older Q-network. So, it's like a delayed copy of the current Q-network. In the current Q-network, you're going to use that for selecting actions, right? So, because you know you want to improve the Q-value based on the current estimation of the Q. So, you want to use the current Q-network, which is parameterized by θ, to select actions to improve your policy. However, you're not going to use the current network to evaluate these actions. What you do is you're going to use the older network, which is θ⁻, to evaluate actions. So, you can see this. So, the objective function actually becomes this one. So, one thing you notice is that here you have this Q(s', a') θ. So, this is still θ, and also this one is still θ. However, you have this Q(s, a) and θ⁻. Right? So, this is Q(s', a) θ. Let me just say Q⁻ because this is argmax, so it's Q'⁻, kind of. However, you're going to use an older Q-network to evaluate the action a'⁻. Okay? So, this way, you will break the kind of bias caused by this maximum of this guy here. Does this make sense? So, by using two Q-networks, you know, you first use this whatever I used, you know, I marked was Brad. So, this Q-network, parameterized by θ, is going to be used for selecting actions, right? So, this is selecting the action. It's argmax of this Q, but in evaluating it, and then evaluating your TD error and essentially your objective, you're going to use this guy here with parameters θ⁻. Okay? So, this again seems to be a simple trick, but it solves the problem of, you know, it basically is able to remove the upward bias caused by this action selection term using the same network, right? Which is, you know, if before we were using the θ, but now you're using θ⁻. Is this fine? David is smiling. Dude, that's an up arrow. Joy asks a question, and I am seconding that question. We're a little confused by upward bias. Here's the paragraph with added punctuation: Double DQ Network Q-learning uses this updated equation to update its new Q value at state \( S_t \) and action \( a \) from its current Q value and the temporal difference error. It includes two parts: one is the target, and another is the current value. So, in Q-learning, the agent starts from the current state under policy \( \Pi \), selects action \( a \), and reaches this point. We want to find the optimal policy by updating this Q value. The agent sets the target: the first term is the reward in the real world, and the second term is the Q value at the next state \( S' \) and the policy by selecting action \( a' \). However, Q-learning has an overall estimation problem because if we always use this maximum, we have a big bias. To reduce this bias, we use double Q-learning. Double Q-learning means we design two Q networks, \( Q_1 \) and \( Q_2 \), and update them at the same time. The idea is that when we update \( Q_1 \), its target is generated by \( Q_2 \), and when we update \( Q_2 \), its target is generated by \( Q_1 \). By using this approach, we can dramatically reduce the bias. Double Q Network (DQ Network) is based on the double Q-learning method. Here, this is a standard double Q Network update equation. We have two networks: one is \( \theta \), which represents the network parameters, and the other is \( \theta' \), which is another network. Right now, we don't update both networks equally. Instead, we use one of them as the Q network, which we update using the current data as input to generate the Q value for all the possible actions. Then, we select the action that generates the maximum Q value. The \( Q' \) is the parameter for this target network. The target network will only need to be updated periodically, which means it copies the parameters from the Q network, maybe not just from the last iteration, but from some previous iteration. We use the next state \( S' \) as the input, generate this target Q, and this Q is not necessarily the maximum Q. Instead, we find the Q corresponding to the selected action as the target Q in the target \( Y \), and use this target \( Y \) as a target in supervised learning to train this network. Finally, we find the optimal policy \( \Pi \). For example, if we use episodic sampling, the data will have a correlation. To break the relationship between the samples, we use replay memory. This means we store all the experiences in this memory, and we train the network by randomly picking from this replay memory. In that way, we can break the relationship in the original episodic sampling. That's the idea of the Double DQN. Now, how can we select the Q in the target? Instead of selecting the maximum Q, we find the Q corresponding to the action we selected in the Q network.