The key elements used in Bellman equations are: the action performed by the agent, which is referred to as a state; the state occurred by performing the action, which is referred to as "yes"; the reward or feedback obtained for each good or bad action, which are; and there is a discount factor, which is denoted by gamma in this particular case. The Bellman equation looks something like this: Phi(s) is always equivalent to the maximum of r(s, a), that is, the reward obtained by performing an action a in state s, plus gamma (the discount factor) multiplied by v(s'), which is the value in the previous state. Now, we will try to use this particular equation and assign different values to each and every cell in a given grid. For example, if a cell is present close to this particular goal, we have to assign a maximum value in this case. So, this is the grid I am going to consider for this example: the grid consists of three rows and four columns. The robot, or the agent, is present in S9. S6 is the block here, S8 is a danger or fire—whenever the agent goes here, it will be given a reward of -1—and S4 is the goal here, and whenever the agent comes to this one, it will be given a reward of +1. Now, what we do is start from a cell next to this particular goal state, which is S3. Whenever the agent performs an action (e.g., moving right), it will go towards this particular goal, so that’s why it will be given a reward of +1. The equation for the Bellman equation looks something like this: In this case: - \( R(s, a) \) is equivalent to 1, because you will get a reward of +1 whenever the agent moves right (towards the goal). - \( \gamma \) is equivalent to 0.9, which is the discount factor (we are just considering it for understanding purposes). - \( V(s') \) is equivalent to 0 in this case because this is the first state or transition we’ve performed here. So, if you put all these particular values in: Thus, \( V(s_3) \) becomes 1 in this particular case. So, I will put that particular 1 over here. Similarly, I have written the values for other states as well. We will try to calculate step by step. In this case, now that we have done with S3, we will try to calculate the value for S2. We will use the same equation: In this case: - \( R(s, a) \) will become 0 because when you perform the right action, you don't get a reward of 1 here. While moving right, you're heading towards S3, not directly to the goal (S4). The reward for reaching S4 (the goal) is 1, but you're moving to S3, which is just part of the journey. - \( \gamma \) is 0.9, the discount factor. - \( V(s') \) is the value of the previous state, which in this case is the value of S3. From the previous calculation, we know that \( V(s_3) = 1 \). So, putting everything into the equation: \[ V(s_2) = \max \left( 0 + 0.9 \times 1 \right) = \max(0.9) \] Thus, \( V(s_2) \) will be 0.9 in this case, as we calculated. That’s why you got \( V(s_2) = 0.9 \). Similarly, we try to calculate it for S1. The Bellman equation for S1 is: In this case: - \( R(s, a) \) is 0 because there's no immediate reward for moving right from S1. - \( \gamma \) is 0.9 (the discount factor). - \( V(s') \) is the value of the previous state, which is \( V(s_2) = 0.9 \). So, applying the values to the equation: V(s_1) = max ( 0 + 0.9 *0.9 ) = max(0.81) Thus, V(s_1) = 0.81 which is the value you have written here. Similarly, we will calculate it for S5. For S5: - \( R(s, a) \) is 0 because no immediate reward is obtained for moving right from S5. - \( \gamma \) is 0.9. - \( V(s') \) is the value of the previous state, which is \( V(s_1) = 0.81 \). Now, applying the equation: V(s_5) = max ( 0 + 0.9 *0.81) = max(0.729) So, V(s_5) = 0.73 (rounded to two decimal places). That’s why V(s_5) = 0.73 in this case. here now I will go to S9 here now uh the same approach we will follow and we will get 0.66 for S9 in this case now we will try to calculate it for S10 uh S10 we will get it 0.59 so that is what I have written here similarly we will go to S11 in this case so if I go to S11 I will get the 0.53 over here the same approach I have followed and then I will go to S12 I will get 0.48 over here so I have started in this case I have gone through this particular path I have got all the values here now we have one more result here that is S7 so I cannot go left side I cannot go right side because this is the fire here if I go down I am getting the previous state value is equivalent to 0.53 but if I consider the previous state is equal on to S3 this is 1 here so that is a maximum so I have to consider this one here So for S7, you can say that the previous state is equivalent to 1, and then \( R(s, a) \) is, of course, 0 here because we cannot go from S7 to S4. So, \( 0.9 \times 1 \), we can consider here. The previous state value is 1, multiplied by 0.9 (the gamma value), and we will get 0.9 over here. Now, there is one more thing we need to consider while calculating S11. We have considered this path, but this is not the only path we have here. If you want to reach this particular S4 from S11, you can follow this path as well. So, we will try to calculate S11 by considering this one as the previous state rather than considering this one as the previous state. In the earlier case, we have considered S10 as the previous state for S11. Now, we will try to consider S7 as a previous state and then we will see whether there is a modification to this particular value or not. So, we will try to get the maximum value over. Here, so I will try to calculate S11 one more time. \( R(s, a) \) is 0, of course. Now, for \( V(s') \), I will consider this one, which is 0.9 (the previous state value). So, \( 0.9 \times 0.9 \) (gamma value) will become 0.81 in this case. This particular 0.81 is greater than 0.53, so we will consider that as the value for S11. Now, when you consider 0.81 as the value for S11, we have to calculate S12 also because, while calculating earlier, we had considered this as the previous state. So, we will see and calculate it one more time. So we, S10 as well as S12, both we have to calculate. S10, if we can calculate here again, \( R(s, a) \) is equivalent to 0, that's for sure. This one is 0.81 multiplied by gamma, that is 0.9. That will become 0.73. Previously, I have written this 0.59, now it is 0.73. So, this is the best option over here, so I will replace this with 0.73 in this case. The same thing we have to follow this direction also, unless and until we will get the least value or you can say that if you are getting a better value, replace it over here. Now we will try to calculate it for S12 also. If I do the calculation for S12, I am getting 0.73 if I consider this one as the previous state. So, if I consider this 0.73 now, that is better than this particular 0.78448. So, definitely, it will be the best value over here. The same thing we have to repeat again and again unless and until we get the final values for each and every state over here. Now, once you get these particular values, let us say that the robot is present in any state. It can reach this particular goal state by considering the optimal path over here. Let us assume that the agent is present in S1. If it is present in S1, it will try to check the value of the neighboring cells. So, if it considers the right move, it will get 0.9 as a value. If it considers the down move, it'll get 0.73. So, this is the better option over here. So, it will go to this particular side, that is, S1 to S2. When it goes to S2, it will try to find the possible paths. It has two possibilities: one is right, and one more is you can say that the left here. So, left is 0.81, right is 1. So, 1 is the better option. So, it will go through this particular S3 here: S2 to S3, and then S3 to the goal state, it can reach directly over here. When it is present in S10, you can see, when it is present in S10, it has two possibilities: it can go left, or it can go right here. Okay, so when it is present in S10, it cannot go left because the value is 0.66. As the value on the right side is more in this case, so 0.81 is better than 0.66. So, it will go right side here. So, when it goes to S11, again, there are three possibilities: left, right, and what we can say that the up. Left is 0.73, right is 0.73, up is 0.9. So, 0.9 is the better option here. So, it will go up in this case. When it is present in here, it cannot go left or right. It can go down or it can go up here. So, the upper value is 1, which is better than the lower one.