Q-networks So, Q-networks are essentially a function approximation of the Q-function, so it's using a neural network. But you can imagine two ways of designing Q-networks. So, here on the left-hand side, my left-hand side is the Q-network that takes an input \( s \) and also an \( a \)—so it takes an input state and action. So, you know, state and action might have been mapped into vectors in a, you know, \( d \)-dimensional space, then you produce the \( Q(s, a) \), so this is the value conditioned on \( s \) and \( a \), right? So that's one way of designing Q-networks. The other way of designing Q-networks, which is actually the more popular one, especially, you know, popularly used in Atari games and so on, is this on the right-hand side. So indeed, your input is not a state-action pair, but only the state. So it's a state, you know, vector representation of the state. Um, it could be an image, right? It could be a video. It could be, you know, um, it could be just like, you know, a discrete state also, right? So, then you go through a neural network, but the output is different. It's not the Q-values itself, but it's a vector of Q-values. So, each entry is corresponding to a different action. So, you can imagine this is for a finite action space, right? So, you can see, so the first entry corresponds to the Q-value corresponding to action one, the second entry to action two, and so on, right? So, all of these Q-values are corresponding to the specific \( s \) that you input to the neural network. Okay, so this is just a description of the Q-network. Here is an example of the deep Q-learning, so-called DQN network. So, you have the last four frames of the gameplay as the input, okay? And then you go through a CNN, right? So, you have like an 8x8x4 filter stripe, four, and then you have like a 4x4x16 filter stripe, two. And then you go through a fully connected neural network. Actually, you know, a second... So, you have two layers of a fully connected network. Then, the output is corresponding to the vector of the Q-values, with each entry corresponding to one action in your action space. Okay, so this is a, um, I believe this is, you know, a popular, uh, deep Q-network for games, like Atari games. So now, uh, with this deep Q-network, you know, you would ask me, "How do I do training? How do I do training?" Right? So for a neural network, you know, you choose an architecture. Um, no, so it really is a matter of choosing a loss function, right? So, a naive loss function is actually this so-called MSBE we've seen that before. So, it's, you know, it's like a TD error, right? So think about the TD error. You have the \( r \), which is the one-step reward, plus the gamma of the max \( Q \, (s') \, a' \, \theta \). So, this is very much like Q-learning, essentially. So, it's basically mimicking the Q-learning objective, right? Although Q-learning was an iterative algorithm based on TD error, here it's like using this TD error squared as an objective function and doing backpropagation. Um, so here, um, so, you know, first, let me see... so this is the MHPE, okay? So, you can see I have \( \theta \) here. \( \theta \) now are the parameters of this deep Q-network, okay? So, now I can also write it as \( L_i(\theta) \), just so you know, this is my \( i \)-step, right? So, I can write this \( y \) as, you know, this part as my \( y_i \), right? So, this is kind of like my target, and then this is something that's going through my network, okay? So, if you look at this form, this is very much like what you're familiar with in a regression problem using a deep neural network, right? So, you have the target and you have the input, and you're finally getting the output through the network. You wanted to minimize the reconstruction error, and of course, you're taking expectation over all the data examples you're looking at. But one thing interesting about this is, like, the data examples are no longer IID; they're actually from an, uh, you know, MDP, so the trajectories are correlated with each other, okay? So, as I said here, you know, this \( y_i \) can be thought of as this guy here, okay? So now, what do you do? You just take the gradient, normal, you know, normal thing that you will do for learning, uh, for deep learning. So, you take a gradient, you get this form, and essentially, this is how, you know, deep Q-networks do it. Okay, so is there any problem related to that? Right. First of all, you know, of course, I wanted to mention that Q-learning algorithm is basically a Q-learning algorithm, but the thing is that the Q-function estimate is actually a neural network. But the problem with this is that the algorithm actually provides a biased estimate, and more importantly, this algorithm could diverge because the states are actually correlated and the targets are non-stationary. Okay, so this is really problematic. To solve this problem, I think the most important contribution of the Deep Q-Network paper was to provide two key mechanisms to solve these problems. One is so-called experience replay. So what is it? It might seem like something that's really heuristic at first sight, but it has a principled consideration behind it. Right? So if you store the (s, a, r, s') tuple in the replay buffer, okay, you do that, and then you randomly sample instances from the buffer. Now, you can see instead of using, you know, my s, a, and my next state s' and a', I'm actually storing things in a replay buffer and doing random sampling. Therefore, I can break the correlations in the data. Okay, so that's a very important trick that is used in reinforcement learning, and essentially, I can imagine this being applied to many other situations where, you know, data is correlated. Okay, so more specifically, in order to deal with the correlated states, the agent builds a dataset of experience and then makes random samples from the dataset. And in order to deal with non-stationary targets, actually, the agent fixes the parameter and then, with some frequency, updates. So, what does this mean? This just means a target network. So, the parameters of the target network are updated only periodically. Like I said in the previous slide—oh, what happened?—in order to deal with the non-stationary targets, they choose to update this network periodically. So what this means is that the parameters of the target network are only updated periodically and held fixed in between to reduce the correlations between action values \( Q \) and the target, which is \( r + \gamma \max_{a'} Q(s', a') \). Okay, so basically, these are the two key mechanisms for Deep Q Networks. It is really Q-learning using a deep neural network, but because of these difficulties or challenges, the authors have proposed two mechanisms to deal with it: experience replay. I think it's really useful for other situations as well, and the target network—I also think that this trick could be used in general deep learning methods, you know, beyond reinforcement learning.