Now I introduce a partially observable Markov decision process. In a Markov decision process, a system model is known, so the agent at his current state \(s\) takes the action \(a\) under the policy \(\pi\), and then the agent moves from his current state \(s\) to the next state \(s'\) and he receives the reward \(r\). If the transition probability is known, both the current state \(s\) and the next state \(s'\) are observable, that means we can fully observe this \(s\) and \(s'\). However, if the environment is unknown or the system model is unknown, the observable states are determined by the latent variables of the environment. That means the state we observe may be different from the true state. So, to describe this, we define belief \(b\), meaning the probability of the observed state being the true state. So we call this case a partially observable Markov decision process. Solving the Markov decision process and the partially observable Markov decision process is different methods. First, let's see how we can solve it. In the method, we define Alpha vectors to represent the partially observable Markov decision process, and the Alpha vectors are associated with the actions. In that case, because we don't know when an agent takes an action and how the action affects the next state, we adjust the Alpha vector. By finding the inner product, we maximize \(\alpha \times b\), and then this Alpha will be the optimal policy to solve the problem. Usually, we use Q-learning to estimate the Q-value at the state \(s\) and action \(a\) from its current value and the temporal difference error. However, in this case, in the partially observable Markov decision process, we need the Q-value at each possible belief, which is not possible to do. So, we only need the observation model, and it's only for the lower-dimensional problem. Another general solution is by using the de Q Network, and then we use this, use this Q, a Q right now at this belief and action to approximate the true Q value. Because this belief is from the observed state, we are in use the deep learning. We introduce the network parameters seta, so we do not adjust the AR vectors. Instead, now we just adjust the model parameters, and then we estimate the Q value at the belief under the action a. To use the gradient descent method to train the network, we need to define the loss function first. This loss function, first this part is the target, and this is the current value at belief and action a. Then, through the gradient descent method, we find the gradient of the objective loss function or objective function. Then, we update the network parameters. When we get the converged model network parameters, then we can estimate the Q value and find the optimal policy. However, as we discussed before, in this way, we can find the solution, but the model is unstable. To get stable results, we need to introduce replay memory and the double Q Network. That means we use two networks: one is a Q Network and another is a target network, to get a stable result. For example, here is this, the network input includes two parts. One part is fully observable states, that means those states are observable. The second part is belief, which means the state for this part is not observable; this is just a partially observable. We use these two kinds of state as input to the network, and then we output the Q value over all the possible actions. Then we find the action that produces the maximum Q, and the policy is the optimal policy.