First MDP Problem Let's consider the following decision problem. >From an initial state, we have a choice of three actions, a, b, and c. The immediate reward associated with each of these actions is zero. But these three actions will take us to different states with different possible rewards available from these states. Action a takes us to state S1, from which our only action is to loop at each step back into state S1, accruing a reward of P1, which is a parameter, each time. If we take action b at the initial state,we will spend the rest of the game alternating between states S2 and S4. We will gain a reward of P2 each time we move from state S2 to S4. And we will receive a reward of P3 on the return trip. And if we take action c initially, we will move into state S3, from which our only next action is to move into state S5, attaining no reward. After that, we repeatedly loop in state S5, attaining a reward of P4 at each step. Suppose the game has an independent probability, of ending after each step? And, let's call this probability, 1 minus gamma, so that the effective discount factor for the scenario is gamma. Given the parameters P1, P2, P3 and P4, our problem is to determine for which values of gamma we prefer the initial action a, the initial action b, and for which gamma we prefer the initial action c. We will solve this problem by setting up a Markov decision process encapsulating this state diagram in Burlap. We will then run value iteration procedures on the MDP for different values of gamma in order to answer this question.