SARSA Algorithm Overview Answers 1. What is the primary advantage of using epsilon in the agent's action selection? A. It guarantees optimal actions every time. B. It aids in exploration and prevents getting stuck in local optima. C. It ensures faster convergence to the optimal policy. D. It allows the agent to calculate Q-values more accurately. Answer: It aids in exploration and prevents getting stuck in local optima. (B) Epsilon introduces a degree of randomness in action selection, promoting exploration. 2. Which statement accurately describes the difference between SARSA and Q-Learning? A. Q-Learning always returns a negative reward. B. Both SARSA and Q-Learning are on-policy methods. C. SARSA can learn from actions outside its current policy. D. SARSA is on-policy, while Q-Learning is off-policy. Answer: SARSA is on-policy, while Q-Learning is off-policy. (D) SARSA learns from actions consistent with its current policy, while Q-Learning can learn from any actions. 3. What is a potential disadvantage of using SARSA? A. It is less effective in real-time environments. B. It can converge faster than Q-learning. C. Its updates are not connected to the current policy. D. It may be slower to converge in some situations. Answer: It may be slower to converge in some situations. (D) While SARSA is beneficial in many cases, it can be slower to converge compared to Q-learning. 4. What characteristic makes SARSA simpler compared to other methods? A. It requires extensive pre-computation of Q-values. B. It uses complex neural networks for function approximation. C. It learns updates that are directly connected to the policy. D. It always opts for the highest potential Q-value. Answer: It learns updates that are directly connected to the policy. (C) SARSA's connection between updates and the current policy contributes to its simplicity. 5. Why might an agent using SARSA not achieve optimal learning in some cases? A. It doesn't consider rewards in its policy updates. B. It learns exclusively from exploratory actions. C. On-policy methods are not necessarily optimal. D. It relies entirely on past experiences. Answer: On-policy methods are not necessarily optimal. (C) Since SARSA is an on-policy method, it may not always find the optimal Q-values. 6. What does SARSA primarily learn during its operation? A. A deterministic action-selection strategy B. A model of the environment dynamics C. An action-value function for expected cumulative rewards D. A value function without any policy constraints Answer: An action-value function for expected cumulative rewards (C) SARSA focuses on learning an action-value function that estimates the expected cumulative reward for actions in given states. 7. Which component of SARSA updates the Q-function? A. The agent's prior knowledge about the environment B. Predictions made by a model of future states C. Immediate reward received from the action taken D. Choices made based on a random policy Answer: Immediate reward received from the action taken (C) The Q-function is updated based on the immediate reward and the predicted next action from the current policy. 8. What is the purpose of the learning rate (α) in the Q-function update? A. To adjust the probability of taking exploratory actions B. To control how much the new information affects the existing Q-value C. To prioritize immediate rewards over future outcomes D. To decrease the exploration rate of the agent Answer: To control how much the new information affects the existing Q-value (B) The learning rate (α) determines how much the newly received information will influence the existing Qvalue. 9. In the SARSA algorithm, what does the discount factor (γ) represent? A. The fraction of actions that are exploratory B. The weight assigned to future rewards relative to immediate rewards C. The proportion of immediate rewards in the update D. The threshold for terminating an episode Answer: The weight assigned to future rewards relative to immediate rewards (B) The discount factor (γ) is crucial for determining how future rewards are valued compared to immediate rewards. 10. What strategy is commonly used to balance exploration and exploitation in the SARSA algorithm? A. Fixed policy strategy B. Epsilon-greedy strategy C. Deterministic policy strategy D. Softmax strategy Answer: Epsilon-greedy strategy (B) The epsilon-greedy strategy allows the agent to explore actions while also making use of the current policy. 11. Which step involves selecting the next action in the SARSA algorithm? A. Iteration step B. Action selection step C. Initialization step D. Observation step Answer: Action selection step (B) The action selection step is where the agent chooses its next action based on the current policy. 12. What type of learning algorithm is SARSA classified as? A. Model-based reinforcement learning B. Direct reinforcement learning C. Off-policy reinforcement learning D. Model-free on-policy reinforcement learning Answer: Model-free on-policy reinforcement learning (D) SARSA is a model-free, on-policy reinforcement learning algorithm. 13. During which phase does the SARSA algorithm update the state and action? A. Policy evaluation phase B. Initialization phase C. Termination phase D. Episode loop phase Answer: Episode loop phase (D) The update of state and action occurs within the episode loop, as the agent progresses through its interactions.