RL Algorithms We can subcategorize reinforcement learning algorithms into main parts: one is value-based methods, and the other is policy-based methods. Value-based methods will determine a value function that quantifies the reward, and using this value function, we determine an optimal policy. Policy-based methods will determine an optimal policy directly, and by "optimal policy," it is the policy that maximizes the total reward. But wait, what is a policy? Simply put, a policy is how an agent behaves in a given situation. Examples of value-based methods include Q-learning, Deep Q Networks (DQN), and SARSA (State-Action-Reward-State-Action). Examples of policy-based methods include REINFORCE, PPO (Proximal Policy Optimization), or TRPO (Trust Region Policy Optimization). So, value-based methods determine a value function, which in turn is used to determine a policy that will maximize total rewards. The value function is a function, so it has an input and some outputs. Depending on the inputs, there are two types of value functions: One of them is the state value function, represented by *V*, and then we have the state-action value function, represented by *Q*. Wait, what are states and actions? A state is the snapshot of an environment that the agent is in, and an action is the decision taken by the agent in that environment. The state value function will take the state as input and output a real number. The state-action value function will take the state and current action as input and then output a real number. This real number is known as a *Q-value*. The state value (*V(s)*) will quantify how good it is to be in the state *s*, while the state-action value (*Q(s, a)*) will quantify how good it is to be in the state *s* and take an action. The state-action values, or Q-values, for these Q-values we need to account for the state and the action taken in that state. So, going back to our simple world, we are at state *S1* and we apply an action of right. This means that we will end up in another state. For this simple world, like we mentioned before, it's pretty easy that we will end up in a state *S2*, and if we go down, we would have ended up in a state *S4*. The maximum reward in state *S1* when going right is the reward in state *S1* plus the maximum future reward in state *S2*. From the next state *S2*, we can perform any action, and hence we would take a maximum across some action *a'* which is taken in the next state. And that's how we end up with this equation. But as we mentioned before, for the more complex case, making the action *A1* (right) doesn't necessarily mean we will end up in *S2*, and going down doesn't mean we're going to end up in *S4* because this environment is stochastic. We can represent stochastic environments with probabilities, and we compute the expected values of these state-action value functions. The expected value is the sum of the probability of transitioning to the state and the value of that state.