18.05 S22 Class 3: Conditional Probability, Independence and Bayes' Theorem

Conditional Probability, Independence and Bayes’ Theorem Class 3, 18.05 1 Learning Goals 1. Know the definitions of conditional probability and independence of events. 2. Be able to compute conditional probability directly from the definition. 3. Be able to use the multiplication rule to compute the total probability of an event. 4. Be able to check if two events are independent. 5. Be able to use Bayes’ formula to ‘invert’ conditional probabilities. 6. Be able to organize the computation of conditional probabilities using trees and tables. 7. Understand the base rate fallacy thoroughly. 2 Conditional Probability Conditional probability answers the question ‘how does the probability of an event change if we have extra information’. We’ll illustrate with an example. Example 1. Toss a fair coin 3 times. (a) What is the probability of 3 heads? Solution: Sample space Ω = {𝐻𝐻𝐻, 𝐻𝐻𝑇 , 𝐻𝑇 𝐻, 𝐻𝑇 𝑇 , 𝑇 𝐻𝐻, 𝑇 𝐻𝑇 , 𝑇 𝑇 𝐻, 𝑇 𝑇 𝑇 }. All outcomes are equally probable, so 𝑃 (3 heads) = 1/8. (b) Suppose we are told that the first toss was heads. Given this information how should we compute the probability of 3 heads? Solution: We have a new (reduced) sample space: Ω′ = {𝐻𝐻𝐻, 𝐻𝐻𝑇 , 𝐻𝑇 𝐻, 𝐻𝑇 𝑇 }. All outcomes are equally probable, so 𝑃 (3 heads given that the first toss is heads) = 1/4. This is called conditional probability, since it takes into account additional conditions. To develop the notation, we rephrase (b) in terms of events. Rephrased (b) Let 𝐴 be the event ‘all three tosses are heads’ = {𝐻𝐻𝐻}. Let 𝐵 be the event ‘the first toss is heads’ = {𝐻𝐻𝐻, 𝐻𝐻𝑇 , 𝐻𝑇 𝐻, 𝐻𝑇 𝑇 }. The conditional probability of 𝐴 knowing that 𝐵 occurred is written 𝑃 (𝐴|𝐵) This is read as ‘the conditional probability of 𝐴 given 𝐵’ 1 18.05 Class 3, Conditional Probability, Independence and Bayes’ Theorem, Spring 2022 2 or ‘the probability of 𝐴 conditioned on 𝐵’ or simply ‘the probability of 𝐴 given 𝐵’. We can visualize conditional probability as follows. Think of 𝑃 (𝐴) as the proportion of the area of the whole sample space taken up by 𝐴. For 𝑃 (𝐴|𝐵) we restrict our attention to 𝐵. That is, 𝑃 (𝐴|𝐵) is the proportion of area of 𝐵 taken up by 𝐴, i.e. 𝑃 (𝐴 ∩ 𝐵)/𝑃 (𝐵). 𝐵 𝐴 𝐴=𝐴∩𝐵 𝐵 𝐻𝐻𝐻 𝐻𝐻𝑇 𝑇 𝐻𝐻 𝑇 𝐻𝑇 𝐻𝑇 𝐻 𝐻𝑇 𝑇 𝑇𝑇𝐻 𝑇𝑇𝑇 𝐴∩𝐵 Conditional probability: Abstract visualization and coin example Note, 𝐴 ⊂ 𝐵 in the right-hand figure, so there are only two colors shown. The formal definition of conditional probability catches the gist of the above example and visualization. Formal definition of conditional probability Let 𝐴 and 𝐵 be events. We define the conditional probability of 𝐴 given 𝐵 as 𝑃 (𝐴|𝐵) = 𝑃 (𝐴 ∩ 𝐵) , provided 𝑃 (𝐵) ≠ 0. 𝑃 (𝐵) (1) Let’s redo the coin tossing example using the definition in Equation (1). Recall 𝐴 = ‘3 heads’ and 𝐵 = ‘first toss is heads’. We have 𝑃 (𝐴) = 1/8 and 𝑃 (𝐵) = 1/2. Since 𝐴 ∩ 𝐵 = 𝐴, we also have 𝑃 (𝐴 ∩ 𝐵) = 1/8. Now according to (1), 𝑃 (𝐴|𝐵) = 𝑃 (𝐴 ∩ 𝐵) 1/8 = = 1/4, 𝑃 (𝐵) 1/2 which agrees with our answer in Example 1 b. We start with a simple example where we can find all the probabilities directly by counting. Example 2. Draw two cards from a deck. Define the events: 𝑆1 = ‘first card is a spade’ and 𝑆2 = ‘second card is a spade’. What is the 𝑃 (𝑆2 |𝑆1 )? Solution: We’ll use formula (1) to compute the conditional probability. We have to compute 𝑃 (𝑆1 ), 𝑃 (𝑆2 ) and 𝑃 (𝑆1 ∩ 𝑆2 ): We know that 𝑃 (𝑆1 ) = 1/4 because there are 52 equally probable ways to draw the first card and 13 of them are spades. The same logic says that there are 52 equally probable ways the second card can be drawn, so 𝑃 (𝑆2 ) = 1/4. Aside: The probability 𝑃 (𝑆2 ) = 1/4 may seem surprising since the value of first card certainly affects the probabilities for the second card. However, if we look at all possible two card sequences we will see that every card in the deck has equal probability of being 18.05 Class 3, Conditional Probability, Independence and Bayes’ Theorem, Spring 2022 3 the second card. Since 13 of the 52 cards are spades we get 𝑃 (𝑆2 ) = 13/52 = 1/4. Another way to say this is: if we are not given value of the first card then we have to consider all possibilities for the second card. Continuing, we compute 𝑃 (𝑆1 ∩ 𝑆2 ) by counting: Number of ways to draw a spade followed by a second spade: 13 ⋅ 12. Number of ways to draw any card followed by any other card: 52 ⋅ 51. Thus, 𝑃 (𝑆1 ∩ 𝑆2 ) = 13 ⋅ 12 = 3/51. 52 ⋅ 51 Now, using (1) we get 𝑃 (𝑆2 |𝑆1 ) = 3/51 𝑃 (𝑆2 ∩ 𝑆1 ) = = 12/51. 𝑃 (𝑆1 ) 1/4 This case is simple enough that we can check our answer by computing the conditional probability directly: if the first card is a spade then of the 51 cards remaining, 12 are spades. So, the probability the second card is also a spade is 𝑃 (𝑆2 |𝑆1 ) = 12/51. Warning: In more complicated problems it will be be much harder to compute conditional probability by counting. Usually we have to use Equation (1). Think: For 𝑆1 and 𝑆2 in the previous example, what is 𝑃 (𝑆2 |𝑆1𝑐 )? 3 Multiplication Rule The following formula is called the multiplication rule. 𝑃 (𝐴 ∩ 𝐵) = 𝑃 (𝐴|𝐵) ⋅ 𝑃 (𝐵). (2) This is simply a rewriting of the definition in Equation (1) of conditional probability. We will see that our use of the multiplication rule is very similar to our use of the rule of product in counting. In fact, the multiplication rule is just a souped up version of the rule of product. We start by verifying the multiplication rule for the previous example. Example 3. Draw two cards from a deck. Define the events: 𝑆1 = ‘first card is a spade’ and 𝑆2 = ‘second card is a spade’. Verify the multiplication rule. Solution: From the previous example, we know 𝑃 (𝑆2 |𝑆1 ) = 12/51, 𝑃 (𝑆1 ∩ 𝑆2 ) = 3/51, 𝑃 (𝑆1 ) = 1/4. From this it is easy to check that 𝑃 (𝑆2 |𝑆1 ) ⋅ 𝑃 (𝑆1 ) = 4 12 1 3 ⋅ = = 𝑃 (𝑆1 ∩ 𝑆2 ). 51 4 51 Law of Total Probability The law of total probability will allow us to use the multiplication rule to find probabilities in more interesting examples. It involves a lot of notation, but the idea is fairly simple. We 18.05 Class 3, Conditional Probability, Independence and Bayes’ Theorem, Spring 2022 4 state the law when the sample space is divided into 3 pieces. It is a simple matter to extend the rule when there are more than 3 pieces. Law of Total Probability Suppose the sample space Ω is divided into 3 disjoint events 𝐵1 , 𝐵2 , 𝐵3 (see the figure below). Then for any event 𝐴: 𝑃 (𝐴) = 𝑃 (𝐴 ∩ 𝐵1 ) + 𝑃 (𝐴 ∩ 𝐵2 ) + 𝑃 (𝐴 ∩ 𝐵3 ) 𝑃 (𝐴) = 𝑃 (𝐴|𝐵1 ) 𝑃 (𝐵1 ) + 𝑃 (𝐴|𝐵2 ) 𝑃 (𝐵2 ) + 𝑃 (𝐴|𝐵3 ) 𝑃 (𝐵3 ) (3) The top equation says ‘if 𝐴 is divided into 3 pieces then 𝑃 (𝐴) is the sum of the probabilities of the pieces’. The bottom equation (3) is called the law of total probability. It is just a rewriting of the top equation using the multiplication rule. Ω 𝐵1 A ∩ B1 A ∩ B2 A ∩ B3 𝐵3 𝐵2 The sample space Ω and the event 𝐴 are each divided into 3 disjoint pieces. The law holds if we divide Ω into any number of events, so long as they are disjoint and cover all of Ω. Such a division is often called a partition of Ω. Our first example will be one where we already know the answer and can verify the law. Example 4. An urn contains 5 red balls and 2 green balls. Two balls are drawn one after the other. What is the probability that the second ball is red? Solution: The sample space is Ω = {rr, rg, gr, gg}. Let 𝑅1 be the event ‘the first ball is red’, 𝐺1 = ‘first ball is green’, 𝑅2 = ‘second ball is red’, 𝐺2 = ‘second ball is green’. We are asked to find 𝑃 (𝑅2 ). Let’s compute this same value using the law of total probability (3). First, we’ll find the conditional probabilities. This is a simple counting exercise. 𝑃 (𝑅2 |𝑅1 ) = 4/6, 𝑃 (𝑅2 |𝐺1 ) = 5/6. Since 𝑅1 and 𝐺1 partition Ω the law of total probability says 𝑃 (𝑅2 ) = 𝑃 (𝑅2 |𝑅1 )𝑃 (𝑅1 ) + 𝑃 (𝑅2 |𝐺1 )𝑃 (𝐺1 ) = 4 5 5 2 ⋅ + ⋅ 6 7 6 7 = 30 5 = . 42 7 (4) 18.05 Class 3, Conditional Probability, Independence and Bayes’ Theorem, Spring 2022 5 Of course, this example is simple enough that we could have computed 𝑃 (𝑅2 ) directly the same way we found 𝑃 (𝑆2 ) directly in the card example. But, we will see that in more complicated examples the law of total probability is truly necessary. Probability urns The example above used probability urns. Their use goes back to the beginning of the subject and we would be remiss not to introduce them. This toy model is very useful. We quote from Wikipedia: https://en.wikipedia.org/wiki/Urn_problem In probability and statistics, an urn problem is an idealized mental exercise in which some objects of real interest (such as atoms, people, cars, etc.) are represented as colored balls in an urn or other container. One pretends to draw (remove) one or more balls from the urn; the goal is to determine the probability of drawing one color or another, or some other properties. A key parameter is whether each ball is returned to the urn after each draw. It doesn’t take much to make an example where (3) is really the best way to compute the probability. Here is a game with slightly more complicated rules. Example 5. An urn contains 5 red balls and 2 green balls. A ball is drawn. If it’s green a red ball is added to the urn and if it’s red a green ball is added to the urn. (The original ball is not returned to the urn.) Then a second ball is drawn. What is the probability the second ball is red? Solution: The law of total probability says that 𝑃 (𝑅2 ) can be computed using the expression in Equation (4). Only the values for the probabilities will change. We have 𝑃 (𝑅2 |𝑅1 ) = 4/7, 𝑃 (𝑅2 |𝐺1 ) = 6/7. Therefore, 𝑃 (𝑅2 ) = 𝑃 (𝑅2 |𝑅1 )𝑃 (𝑅1 ) + 𝑃 (𝑅2 |𝐺1 )𝑃 (𝐺1 ) = 5 4 5 6 2 32 ⋅ + ⋅ = . 7 7 7 7 49 Using Trees to Organize the Computation Trees are a great way to organize computations with conditional probability and the law of total probability. The figures and examples will make clear what we mean by a tree. As with the rule of product, the key is to organize the underlying process into a sequence of actions. We start by redoing Example 5. The sequence of actions are: first draw ball 1 (and add the appropriate ball to the urn) and then draw ball 2. 5/7 2/7 𝑅1 4/7 𝑅2 𝐺1 3/7 𝐺2 6/7 𝑅2 1/7 𝐺2 18.05 Class 3, Conditional Probability, Independence and Bayes’ Theorem, Spring 2022 6 You interpret this tree as follows. Each dot is called a node. The tree is organized by levels. The top node (root node) is at level 0. The next layer down is level 1 and so on. Each level shows the outcomes at one stage of the game. Level 1 shows the possible outcomes of the first draw. Level 2 shows the possible outcomes of the second draw starting from each node in level 1. Probabilities are written along the branches. The probability of 𝑅1 (red on the first draw) is 5/7. It is written along the branch from the root node to the one labeled 𝑅1 . At the next level we put in conditional probabilities. The probability along the branch from 𝑅1 to 𝑅2 is 𝑃 (𝑅2 |𝑅1 ) = 4/7. It represents the probability of going to node 𝑅2 given that you are already at 𝑅1 . The muliplication rule says that the probability of getting to any node is just the product of the probabilities along the path to get there. For example, the node labeled 𝑅2 at the far left really represents the event 𝑅1 ∩ 𝑅2 because it comes from the 𝑅1 node. The multiplication rule now says 5 4 𝑃 (𝑅1 ∩ 𝑅2 ) = 𝑃 (𝑅1 ) ⋅ 𝑃 (𝑅2 |𝑅1 ) = ⋅ , 7 7 which is exactly multiplying along the path to the node. The law of total probability is just the statement that 𝑃 (𝑅2 ) is the sum of the probabilities of all paths leading to 𝑅2 (the two circled nodes in the figure). In this case, 𝑃 (𝑅2 ) = 5 4 2 6 32 ⋅ + ⋅ = , 7 7 7 7 49 exactly as in the previous example. 5.1 Shorthand vs. precise trees The tree given above involves some shorthand. For example, the node marked 𝑅2 at the far left really represents the event 𝑅1 ∩ 𝑅2 , since it ends the path from the root through 𝑅1 to 𝑅2 . Here is the same tree with everything labeled precisely. As you can see this tree is more cumbersome to make and use. We usually use the shorthand version of trees. You should make sure you know how to interpret them precisely. 𝑃 (𝑅1 ) = 5/7 𝑃 (𝐺1 ) = 2/7 𝑅1 𝐺1 𝑃 (𝑅2 |𝑅1 ) = 4/7 𝑅1 ∩ 𝑅2 6 𝑃 (𝐺2 |𝑅1 ) = 3/7 𝑅1 ∩ 𝐺2 𝑃 (𝑅2 |𝐺1 ) = 6/7 𝐺1 ∩ 𝑅2 𝑃 (𝐺2 |𝐺1 ) = 1/7 𝐺1 ∩ 𝐺2 Independence Two events are independent if knowledge that one occurred does not change the probability that the other occurred. Informally, events are independent if they do not influence one another. 18.05 Class 3, Conditional Probability, Independence and Bayes’ Theorem, Spring 2022 7 Example 6. Toss a coin twice. We expect the outcomes of the two tosses to be independent of one another. In real experiments this always has to be checked. If my coin lands in honey and I don’t bother to clean it, then the second toss might be affected by the outcome of the first toss. More seriously, the independence of experiments can by undermined by the failure to clean or recalibrate equipment between experiments or to isolate supposedly independent observers from each other or a common influence. We’ve all experienced hearing the same ‘fact’ from different people. Hearing it from different sources tends to lend it credence until we learn that they all heard it from a common source. That is, our sources were not independent. Translating the verbal description of independence into symbols gives 𝐴 is independent of 𝐵 if 𝑃 (𝐴|𝐵) = 𝑃 (𝐴). (5) That is, knowing that 𝐵 occurred does not change the probability that 𝐴 occurred. In terms of events as subsets, knowing that the realized outcome is in 𝐵 does not change the probability that it is in 𝐴. If 𝐴 and 𝐵 are independent in the above sense, then the multiplication rule gives 𝑃 (𝐴 ∩ 𝐵) = 𝑃 (𝐴|𝐵) ⋅ 𝑃 (𝐵) = 𝑃 (𝐴) ⋅ 𝑃 (𝐵). This justifies the following technical definition of independence. Formal definition of independence: Two events 𝐴 and 𝐵 are independent if 𝑃 (𝐴 ∩ 𝐵) = 𝑃 (𝐴) ⋅ 𝑃 (𝐵) (6) This is a nice symmetric definition which makes clear that 𝐴 is independent of 𝐵 if and only if 𝐵 is independent of 𝐴. Unlike the equation with conditional probabilities, this definition makes sense even when 𝑃 (𝐵) = 0. In terms of conditional probabilities, we have: 1. If 𝑃 (𝐵) ≠ 0 then 𝐴 and 𝐵 are independent if and only if 𝑃 (𝐴|𝐵) = 𝑃 (𝐴). 2. If 𝑃 (𝐴) ≠ 0 then 𝐴 and 𝐵 are independent if and only if 𝑃 (𝐵|𝐴) = 𝑃 (𝐵). Independent events commonly arise as different trials in an experiment, as in the following example. Example 7. Toss a fair coin twice. Let 𝐻1 = ‘heads on first toss’ and let 𝐻2 = ‘heads on second toss’. Are 𝐻1 and 𝐻2 independent? Solution: Since 𝐻1 ∩ 𝐻2 is the event ‘both tosses are heads’ we have 𝑃 (𝐻1 ∩ 𝐻2 ) = 1/4 = 𝑃 (𝐻1 )𝑃 (𝐻2 ). Therefore the events are independent. We can ask about the independence of any two events, as in the following two examples. Example 8. Toss a fair coin 3 times. Let 𝐻1 = ‘heads on first toss’ and 𝐴 = ‘two heads total’. Are 𝐻1 and 𝐴 independent? Solution: We know that 𝑃 (𝐴) = 3/8. Since this is not 0 we can check if the formula in Equation 5 holds. Now, 𝐻1 = {HHH, HHT, HTH, HTT} contains exactly two outcomes (𝐻𝐻𝑇 , 𝐻𝑇 𝐻) from 𝐴, so we have 𝑃 (𝐴|𝐻1 ) = 2/4. Since 𝑃 (𝐴|𝐻1 ) ≠ 𝑃 (𝐴) these events are not independent. 18.05 Class 3, Conditional Probability, Independence and Bayes’ Theorem, Spring 2022 8 Example 9. Draw one card from a standard deck of playing cards. Let’s examine the independence of 3 events ‘the card is an ace’, ‘the card is a heart’ and ‘the card is red’. Define the events as 𝐴 = ‘ace’, 𝐻 = ‘hearts’, 𝑅 = ‘red’. (a) We know that 𝑃 (𝐴) = 4/52 (4 out of 52 cards are aces), 𝑃 (𝐴|𝐻) = 1/13 (1 out of 13 hearts are aces). Since 𝑃 (𝐴) = 𝑃 (𝐴|𝐻) we have that 𝐴 is independent of 𝐻. (b) 𝑃 (𝐴|𝑅) = 2/26 = 1/13 = 𝑃 (𝐴). So 𝐴 is independent of 𝑅. That is, whether the card is an ace is independent of whether it is red. (c) Finally, what about 𝐻 and 𝑅? Since 𝑃 (𝐻) = 1/4 and 𝑃 (𝐻|𝑅) = 1/2, 𝐻 and 𝑅 are not independent. We could also see this the other way around: 𝑃 (𝑅) = 1/2 and 𝑃 (𝑅|𝐻) = 1, so 𝐻 and 𝑅 are not independent. That is, the suit of a card is not independent of the color of the card’s suit. 6.1 Paradoxes of Independence An event 𝐴 with probability 0 is independent of itself, since in this case both sides of equation (6) are 0. This appears paradoxical because knowledge that 𝐴 occurred certainly gives information about whether 𝐴 occurred. We resolve the paradox by noting that since 𝑃 (𝐴) = 0 the statement ‘𝐴 occurred’ is vacuous. Think: For what other value(s) of 𝑃 (𝐴) is 𝐴 independent of itself? 7 Bayes’ Theorem Bayes’ theorem is a pillar of both probability and statistics and it is central to the rest of this course. For two events 𝐴 and 𝐵 Bayes’ theorem (also called Bayes’ rule and Bayes’ formula) says 𝑃 (𝐴|𝐵) ⋅ 𝑃 (𝐵) 𝑃 (𝐵|𝐴) = . (7) 𝑃 (𝐴) Comments: 1. Bayes’ rule tells us how to ‘invert’ conditional probabilities, i.e. to find 𝑃 (𝐵|𝐴) from 𝑃 (𝐴|𝐵). 2. In practice, 𝑃 (𝐴) is often computed using the law of total probability. Proof of Bayes’ rule The key point is that 𝐴 ∩ 𝐵 is symmetric in 𝐴 and 𝐵. So the multiplication rule says 𝑃 (𝐵|𝐴) ⋅ 𝑃 (𝐴) = 𝑃 (𝐴 ∩ 𝐵) = 𝑃 (𝐴|𝐵) ⋅ 𝑃 (𝐵). Now divide through by 𝑃 (𝐴) to get Bayes’ rule. A common mistake is to confuse the meanings of 𝑃 (𝐴|𝐵) and 𝑃 (𝐵|𝐴). They can be very different. This is illustrated in the next example. Example 10. Toss a coin 5 times. Let 𝐻1 = ‘first toss is heads’ and let 𝐻𝐴 = ‘all 5 tosses are heads’. Then 𝑃 (𝐻1 |𝐻𝐴 ) = 1 but 𝑃 (𝐻𝐴 |𝐻1 ) = 1/16. For practice, let’s use Bayes’ theorem to compute 𝑃 (𝐻1 |𝐻𝐴 ) using 𝑃 (𝐻𝐴 |𝐻1 ).The terms 18.05 Class 3, Conditional Probability, Independence and Bayes’ Theorem, Spring 2022 9 are 𝑃 (𝐻𝐴 |𝐻1 ) = 1/16, 𝑃 (𝐻1 ) = 1/2, 𝑃 (𝐻𝐴 ) = 1/32. So, 𝑃 (𝐻1 |𝐻𝐴 ) = 𝑃 (𝐻𝐴 |𝐻1 )𝑃 (𝐻1 ) (1/16) ⋅ (1/2) = = 1, 𝑃 (𝐻𝐴 ) 1/32 which agrees with our previous calculation. 7.1 The Base Rate Fallacy The base rate fallacy is one of many examples showing that it’s easy to confuse the meaning of 𝑃 (𝐵|𝐴) and 𝑃 (𝐴|𝐵) when a situation is described in words. This is one of the key examples from probability and it will inform much of our practice and interpretation of statistics. You should strive to understand it thoroughly. Example 11. The Base Rate Fallacy Consider a routine screening test for a disease. Suppose the frequency of the disease in the population (base rate) is 0.5%. The test is fairly accurate with a 5% false positive rate and a 10% false negative rate. You take the test and it comes back positive. What is the probability that you have the disease? Solution: We will do the computation three times: using trees, tables and symbols. We’ll use the following notation for the relevant events: 𝐷+ = ‘you have the disease’ 𝐷− = ‘you do not have the disease 𝑇 + = ‘you tested positive’ 𝑇 − = ‘you tested negative’. We are given 𝑃 (𝐷+) = 0.005 and therefore 𝑃 (𝐷−) = 0.995. The false positive and false negative rates are (by definition) conditional probabilities. 𝑃 (false positive) = 𝑃 (𝑇 + |𝐷− ) = 0.05 and 𝑃 (false negative) = 𝑃 (𝑇 − |𝐷+ ) = 0.1. The complementary probabilities are known as the true negative and true positive rates: 𝑃 (𝑇 − |𝐷− ) = 1 − 𝑃 (𝑇 + |𝐷− ) = 0.95 and 𝑃 (𝑇 + |𝐷+ ) = 1 − 𝑃 (𝑇 − |𝐷+ ) = 0.9. Trees: All of these probabilities can be displayed quite nicely in a tree. 0.995 𝐷− 0.05 𝑇+ 0.005 𝐷+ 0.95 𝑇− 0.9 𝑇+ 0.1 𝑇− The question asks for the probability that you have the disease given that you tested positive, i.e. what is the value of 𝑃 (𝐷+ |𝑇 + ). We aren’t given this value, but we do know 𝑃 (𝑇 + |𝐷+ ), so we can use Bayes’ theorem. 𝑃 (𝐷+ |𝑇 + ) = 𝑃 (𝑇 + |𝐷+ ) ⋅ 𝑃 (𝐷+ ) . 𝑃 (𝑇 + ) 18.05 Class 3, Conditional Probability, Independence and Bayes’ Theorem, Spring 2022 10 The two probabilities in the numerator are given. We compute the denominator 𝑃 (𝑇 + ) using the law of total probability. Using the tree, we just have to sum the probabilities for each of the nodes marked 𝑇 + 𝑃 (𝑇 + ) = 0.995 × 0.05 + 0.005 × 0.9 = 0.05425 Thus, 𝑃 (𝐷+ |𝑇 + ) = 0.9 × 0.005 = 0.082949 ≈ 8.3%. 0.05425 Remarks: This is called the base rate fallacy because the base rate of the disease in the population is so low that the vast majority of the people taking the test are healthy, and even with an accurate test most of the positives will be healthy people. Ask your doctor for his/her guess at the odds. To summarize the base rate fallacy with specific numbers 95% of all tests are accurate does not imply 95% of positive tests are accurate We will refer back to this example frequently. It and similar examples are at the heart of many statistical misunderstandings. Other ways to work Example 11 Tables: Another trick that is useful for computing probabilities is to make a table. Let’s redo the previous example using a table built with 10000 total people divided according to the probabilites in this example. We construct the table as follows. Pick a number, say 10000 people, and place it as the grand total in the lower right. Using 𝑃 (𝐷+ ) = 0.005 we compute that 50 out of the 10000 people are sick (𝐷+ ). Likewise 9950 people are healthy (𝐷− ). At this point the table looks like: 𝐷+ 𝐷− total 𝑇+ 𝑇− total 50 9950 10000 Using 𝑃 (𝑇 + |𝐷+ ) = 0.9 we can compute that the number of sick people who tested positive as 90% of 50 or 45. The other entries are similar. At this point the table looks like the table below on the left. Finally we sum the 𝑇 + and 𝑇 − rows to get the completed table on the right. 𝐷+ 𝐷− total 𝑇 45 498 − 𝑇 5 9452 total 50 9950 10000 + 𝐷+ 𝐷− total 𝑇 45 498 543 − 𝑇 5 9452 9457 total 50 9950 10000 + Using the complete table we can compute 𝑃 (𝐷+ |𝑇 + ) = |𝐷+ ∩ 𝑇 + | 45 = = 8.3%. + |𝑇 | 543 Symbols: For completeness, we show how the solution looks when written out directly in 18.05 Class 3, Conditional Probability, Independence and Bayes’ Theorem, Spring 2022 11 symbols. 𝑃 (𝑇 + |𝐷+ ) ⋅ 𝑃 (𝐷+ ) 𝑃 (𝑇 + ) 𝑃 (𝑇 + |𝐷+ ) ⋅ 𝑃 (𝐷+ ) = + + 𝑃 (𝑇 |𝐷 ) ⋅ 𝑃 (𝐷+ ) + 𝑃 (𝑇 + |𝐷− ) ⋅ 𝑃 (𝐷− ) 0.9 × 0.005 = 0.9 × 0.005 + 0.05 × 0.995 = 8.3% 𝑃 (𝐷+ |𝑇 + ) = Visualization: The figure below illustrates the base rate fallacy. The large blue rectangle represents all the healthy people. The much smaller orange rectangle represents the sick people. The shaded rectangle represents the people who test positive. The shaded area covers most of the orange area and only a small part of the blue area. Even so, the most of the shaded area is over the blue. That is, most of the positive tests are of healthy people. 𝐷− 7.2 𝐷+ Bayes’ rule in 18.05 As we said at the start of this section, Bayes’ rule is a pillar of probability and statistics. We have seen that Bayes’ rule allows us to ‘invert’ conditional probabilities. When we study statistics we will see that the art of statistical inference involves deciding how to proceed when one (or more) of the terms on the right side of Bayes’ rule is unknown.