Deep Reinforcement Learning

Deep Reinforcement Learning Deep reinforcement learning really refers to using a neural network to approximate the value function, the policy, or the model, right? So, you know, we talked a lot about value-based methods and policy-based methods. Um, we kind of, at the beginning of the reinforcement learning, introduction to reinforcement learning, we said, you know, we have this transition model, right? So, but we didn't spend a lot of time talking about how, you know, we can do dynamic programming when we know the model. But I kind of, you know, um, hinted that, you know, if you know the transition model, then everything becomes really simple. You can just solve dynamic programming and you will get the solution for reinforcement learning. So, deep reinforcement learning is really just a special way of doing function approximation, either approximation of the value function, the policy, or the model. Okay, when I say the model, I really mean the transition model of the RL. And so, you know, now, since you guys are familiar with, you know, deep learning, well, you know, instead of using simple function class hypothesis class, you're using this computation of nonlinear functions. So, it's supposed to be rich, right? It's supposed to have very high expressive power, and indeed, you know, theoretical work has shown, you know, uh, the universal approximation theorem about neural nets, right? Um, so one thing, you know, deep reinforcement learning, although very powerful, it may not give any interpretation, or the estimate might be stuck at a local optimum, you know, just like, you know, we would suffer similar problems in learning in traditional deep learning, right? So, because we have this non-convexity of the optimization landscape, so indeed, we don't know very well about how you can converge to a global optimum. But there were also a lot of work recently working on understanding the behavior of first-order methods, second-order methods on deep learning, specific deep learning optimization landscapes, such as skip connections and things like that. You have heard about DQN networks, so first, let me just very, very briefly review deep neural networks. I mean, so this is just, you have a neural network that transforms input, actor, the input vector x, into an output y. And so, you have, uh, w_0 * x transpose. Okay, so, you know, you have the input, you have a linear transformation, you have a, uh, like a shift term, and then you have a non-linear transformation, so-called activation function, right? Uh, ReLU function, hyperbolic tangent, sigmoid, and so on—group source and these kind of more advanced activation functions. So, once you have that, then you go through another linear transformation, have a shift term, and then another activation, nonlinear transformation. So, you do that composition again and again. So, the number of times you do that is the depth of the neural network. And then finally, you will get the output. Okay, so the output is also a vector. An input isn't a vector; usually, input is in a very high-dimensional space, and output usually is in a smaller dimensional space. Okay, so this is just a very brief, I guess, recap of deep neural networks. So, essentially, you know, for example, if you think about regression problems, usually you would have wanted to reach, you know, minimize reconstruction error, or kind of like, um, uh, so, uh, you know, some kind of, I guess it doesn't have to be this form. It doesn't have to be like, you know, the L2 norm of the differences between the vectors. It could be other forms, but I'm just using an example here. Well, if you think about classification, then the most popular loss function is the cross-entropy loss, as you know, we might all be familiar with. So, one important thing is, you know, in deep neural networks, uh, there is kind of weight sharing, right? So, for example, in a recurrent neural network, you have the weight shared between time steps. Um, and you also have the weight sharing in convolutional neural networks, which is the more popular, uh, kind of neural networks used in, uh, you know, kind of images, language models, or video models, and so on.