18.05 S22 Class 14: Continuous Data with Continuous Priors

Continuous Data with Continuous Priors Class 14, 18.05 This reading is not assigned. It goes into a little more detail on Bayesian updating where both hypotheses and data are continuous. 1 Learning Goals 1. Be able to construct a Bayesian update table for continuous hypotheses and continuous data. 2. Be able to recognize the pdf of a normal distribution and determine its mean and variance. 2 Introduction We are now ready to do Bayesian updating when both the hypotheses and the data take continuous values. The pattern is the same as what we’ve done before, so let’s first review the previous two cases. 3 Previous cases 1. Discrete hypotheses, discrete data Notation • Hypotheses ℋ • Data 𝑥 • Prior 𝑃 (ℋ) • Likelihood 𝑝(𝑥 | ℋ) • Posterior 𝑃 (ℋ | 𝑥). Example 1. Suppose we have data 𝑥 and three possible explanations (hypotheses) for the data that we’ll call 𝐴, 𝐵, 𝐶. Suppose also that the data can take two possible values, -1 and 1. In order to use the data to help estimate the probabilities of the different hypotheses we need a prior pmf and a likelihood table. Assume the prior and likelihoods are given in the following table. (For this example we are only concerned with the formal process of Bayesian updating. So we just made up the prior and likelihoods.) 1 18.05 Class 14, Continuous Data with Continuous Priors, Spring 2022 hypothesis prior ℋ 𝑃 (ℋ) A 0.1 B 0.3 C 0.6 Prior probabilities hypothesis ℋ A B C 2 likelihood 𝑝(𝑥 | ℋ) 𝑥 = −1 𝑥=1 0.2 0.8 0.5 0.5 0.7 0.3 Likelihoods Naturally, each entry in the likelihood table is a likelihood 𝑝(𝑥 | ℋ). For instance the 0.2 row 𝐴 and column 𝑥 = −1 is the likelihood 𝑝(𝑥 = −1 | 𝐴). Question: Suppose we run one trial and obtain the data 𝑥1 = 1. Use this to find the posterior probabilities for the hypotheses. Solution: The data picks out one column from the likelihood table which we then use in our Bayesian update table. hypothesis prior likelihood Bayes numerator ℋ 𝑃 (ℋ) 𝑝(𝑥 = 1 | ℋ) 𝑝(𝑥 | ℋ)𝑃 (ℋ) 𝐴 𝐵 𝐶 total 0.1 0.3 0.6 1 0.8 0.5 0.3 no sum 0.08 0.15 0.18 𝑝(𝑥) = 0.41 posterior 𝑃 (ℋ | 𝑥) = 𝑝(𝑥 | ℋ)𝑃 (ℋ) 𝑝(𝑥) 0.195 0.366 0.439 1 To summarize: the prior probabilities of hypotheses and the likelihoods of data given hypothesis were given; the Bayes numerator is the product of the prior and likelihood; the total probability 𝑝(𝑥) is the sum of the probabilities in the Bayes numerator column; and we divide by 𝑝(𝑥) to normalize the Bayes numerator. Note: As usual, the term ‘no sum’ in the likelihood column is not literally true. What it means is that the sum is not meaningful to us. In particular, we don’t expect the likelihood column to sum to 1. 2. Continuous hypotheses, discrete data Now suppose that we have data 𝑥 that can take a discrete set of values and a continuous parameter 𝜃 that determines the distribution the data is drawn from. Notation • Hypotheses 𝜃 • Data 𝑥 • Prior 𝑓(𝜃) 𝑑𝜃 • Likelihood 𝑝(𝑥 | 𝜃) • Posterior 𝑓(𝜃 | 𝑥) 𝑑𝜃. 18.05 Class 14, Continuous Data with Continuous Priors, Spring 2022 3 Note: Here we multiplied by 𝑑𝜃 to express the prior and posterior as probabilities. As densities, we have the prior pdf 𝑓(𝜃) and the posterior pdf 𝑓(𝜃 | 𝑥). Example 2. Assume that 𝑥 ∼ Binomial(5, 𝜃). So 𝜃 is in the range [0, 1] and the data 𝑥 can take six possible values, 0, 1, …, 5. Since there is a continuous range of values we use a pdf to describe the prior on 𝜃. Let’s suppose the prior is 𝑓(𝜃) = 2𝜃. We can still make a likelihood table, though it only has one row representing an arbitrary hypothesis 𝜃. hypothesis 𝜃 likelihood 𝑝(𝑥 | 𝜃) 𝑥=0 𝑥=1 𝑥=2 𝑥=3 𝑥=4 𝑥=5 (50)(1 − 𝜃)5 (51)𝜃(1 − 𝜃)4 (52)𝜃2 (1 − 𝜃)3 (53)𝜃3 (1 − 𝜃)2 (54)𝜃4 (1 − 𝜃) (55)𝜃5 Likelihoods Question: Suppose we run one trial and obtain the data 𝑥 = 2. Use this to find the posterior pdf for the parameter (hypotheses) 𝜃. Solution: As before, the data picks out one column from the likelihood table which we can use in our Bayesian update table. Since we want to work with probabilities we write 𝑓(𝜃)𝑑 𝜃 and 𝑓(𝜃 | 𝑥) 𝑑𝜃 for the pdfs. hypothesis prior likelihood (for 𝑥 = 2) Bayes numerator 𝜃 𝑓(𝜃) 𝑑𝜃 𝑝(𝑥 | 𝜃) 𝑝(𝑥 | 𝜃)𝑓(𝜃) 𝑑𝜃 𝜃 2𝜃 𝑑𝜃 (52)𝜃2 (1 − 𝜃)3 2(52)𝜃3 (1 − 𝜃)3 𝑑𝜃 posterior 𝑓(𝜃 | 𝑥) 𝑑𝜃 = 𝑓(𝜃 | 𝑥) 𝑑𝜃 = 𝑝(𝑥 | 𝜃)𝑓(𝜃) 𝑑𝜃 𝑝(𝑥) 7! 3 𝜃 (1 − 𝜃)3 𝑑𝜃 3! 3! 1 total 1 no sum 𝑝(𝑥) = ∫0 2(52)𝜃3 (1 − 𝜃)3 𝑑𝜃 = 2(52) 3!7!3! 1 To summarize: the prior probabilities of hypotheses and the likelihoods of data given hypothesis were given; the Bayes numerator is the product of the prior and likelihood; the total probability 𝑝(𝑥) is the integral of the probabilities in the Bayes numerator column; and we divide by 𝑝(𝑥) to normalize the Bayes numerator. 4 Continuous hypotheses and continuous data When both data and hypotheses are continuous, the only change to the previous example is that the likelihood function uses a pdf 𝜙(𝑥 | 𝜃) instead of a pmf 𝑝(𝑥 | 𝜃). The general shape of the Bayesian update table is the same. Notation 18.05 Class 14, Continuous Data with Continuous Priors, Spring 2022 4 • Hypotheses 𝜃. For continuous hypotheses, this really means that we hypothesize that the parameter is in a small interval of size 𝑑𝜃 around 𝜃. • Data 𝑥. For continuous data, this really means that the data is in a small interval of size 𝑑𝑥 around 𝑥. • Prior 𝑓(𝜃)𝑑𝜃. This is our initial belief about the probability that the parameter is in a small interval of size 𝑑𝜃 around 𝜃. • Likelihood 𝜙(𝑥 | 𝜃) 𝑑𝑥. This is the (calculated) probability that the data is in a small interval of size 𝑑𝑥 around 𝑥, ASSUMING the hypothesis 𝜃. • Posterior 𝑓(𝜃 | 𝑥) 𝑑𝜃. This is the (calculated) probability that the parameter is in a small interval of size 𝑑𝜃 around 𝜃, GIVEN the data 𝑥. Simplifying the notation. In the previous cases we included 𝑑𝜃 so that we were working with probabilities instead of densities. When both data and hypotheses are continuous we will need both 𝑑𝜃 and 𝑑𝑥. This makes things conceptually simpler, but notationally cumbersome. To simplify the notation we will sometimes allow ourselves to drop 𝑑𝑥 in our tables. This is fine because the data 𝑥 is fixed in each calculation. We keep the 𝑑𝜃 because the hypothesis 𝜃 is allowed to vary. For comparison, we first show the general table in simplified notation followed immediately afterward by the table showing both infinitesimals. Bayes hypoth. prior likelihood numerator 𝜃 𝑓(𝜃) 𝑑𝜃 𝜙(𝑥 | 𝜃) 𝜙(𝑥 | 𝜃)𝑓(𝜃) 𝑑𝜃 total 1 no sum 𝜙(𝑥) = ∫ 𝜙(𝑥 | 𝜃)𝑓(𝜃) 𝑑𝜃 (integrate over 𝜃) posterior 𝑓(𝜃 | 𝑥) 𝑑𝜃 = 𝜙(𝑥 | 𝜃)𝑓(𝜃) 𝑑𝜃 𝜙(𝑥) 1 = prior prob. density for data 𝑥 Bayesian update table without 𝑑𝑥 hypoth. prior likelihood Bayes numerator posterior 𝑓(𝜃 | 𝑥) 𝑑𝜃 = 𝜃 𝑓(𝜃) 𝑑𝜃 𝜙(𝑥 | 𝜃) 𝑑𝑥 𝜙(𝑥 | 𝜃)𝑓(𝜃) 𝑑𝜃 𝑑𝑥 = total 1 no sum 𝜙(𝑥) 𝑑𝑥 = (∫ 𝜙(𝑥 | 𝜃)𝑓(𝜃) 𝑑𝜃) 𝑑𝑥 𝜙(𝑥 | 𝜃)𝑓(𝜃) 𝑑𝜃 𝑑𝑥 𝜙(𝑥) 𝑑𝑥 𝜙(𝑥 | 𝜃)𝑓(𝜃) 𝑑𝜃 𝜙(𝑥) 1 Bayesian update table with 𝑑𝜃 and 𝑑𝑥 To summarize: the prior probabilities of hypotheses and the likelihoods of data given hypothesis were given; the Bayes numerator is the product of the prior and likelihood; the 18.05 Class 14, Continuous Data with Continuous Priors, Spring 2022 5 total probability 𝜙(𝑥) 𝑑𝑥 is the integral of the probabilities in the Bayes numerator column; we divide by 𝜙(𝑥) 𝑑𝑥 to normalize the Bayes numerator. 5 A digression on notational messes We have chosen to use the notation 𝜙(𝑥), 𝜙(𝑥 | 𝜃) for the pdfs of data and 𝑓(𝜃), 𝑓(𝜃 | 𝑥) for the pdfs of hypotheses. This is nice because 𝜙 is a Greek 𝑓, but the different symbols help us distinguish the two types of pdfs. Many, perhaps most, writers use the same letter 𝑓 for both. This forces the reader to look at the arguments to the function to understand what is meant. That is, 𝑓(𝑥|𝜃) is the probability of data given an hypothesis, i.e. the likelihood and 𝑓(𝜃|𝑥) is the probability of an hypothesis given the data, i.e. the posterior pdf. As mathematicians this makes us pull our hair out. But, to be fair, there is a philosophical underpinning to this notation. We can think of 𝑓 as a universal probability density which gives the probability of absolutely any combination of things. Thus 𝑓(𝑥, 𝑦) is the joint probability density for the quantities denoted by 𝑥 and 𝑦. If we just write 𝑓(𝑥) the implication is that this means the marginal density for 𝑥, i.e. the density for 𝑥 when 𝑦 is allowed to take any value. Similarly we can write 𝑓(𝑥, 𝑦|𝑧) for the conditional density of 𝑥 and 𝑦 given 𝑧. 6 Normal hypothesis, normal data A standard example of continuous hypotheses and continuous data assumes that both the data and prior follow normal distributions. The following example assumes that the variance of the data is known. Example 3. Suppose we have data 𝑥 = 5 which was drawn from a normal distribution with unknown mean 𝜃 and standard deviation 1. 𝑥 ∼ N(𝜃, 1) Suppose further, that our prior distribution for the unknown parameter 𝜃 is 𝜃 ∼ N(2, 1). Let 𝑥 represent an arbitrary data value. (a) Make a Bayesian table with prior, likelihood, and Bayes numerator. (b) Show that the posterior distribution for 𝜃 is normal as well. (c) Find the mean and variance of the posterior distribution. Solution: As we did with the tables above, a good compromise on the notation is to include 𝑑𝜃 but not 𝑑𝑥. The reason for this is that the total probability is computed by integrating over 𝜃 and the 𝑑𝜃 reminds of us that. Our prior pdf is The likelihood function is 1 2 𝑓(𝜃) = √ e−(𝜃−2) /2 . 2𝜋 1 2 𝜙(𝑥 = 5 | 𝜃) = √ e−(5−𝜃) /2 . 2𝜋 18.05 Class 14, Continuous Data with Continuous Priors, Spring 2022 6 We know we are going to multiply the prior and the likelihood, so we carry out that algebra first. In the very last step we give the complicated constant factor the name 𝑐1 . 1 1 2 2 prior ⋅ likelihood = √ e−(𝜃−2) /2 ⋅ √ e−(5−𝜃) /2 2𝜋 2𝜋 1 −(2𝜃2 −14𝜃+29)/2 = e 2𝜋 1 −(𝜃2 −7𝜃+29/2) = e (complete the square) 2𝜋 1 −((𝜃−7/2)2 +9/4) = e 2𝜋 e−9/4 −(𝜃−7/2)2 ) = e 2𝜋 2 = 𝑐1 e−(𝜃−7/2) In the last step we named the complicated constant factor 𝑐1 . hypothesis prior likelihood Bayes numerator 𝜃 𝑓(𝜃) 𝑑𝜃 𝜙(𝑥 = 5 | 𝜃) 𝜙(𝑥 = 5 | 𝜃)𝑓(𝜃) 𝑑𝜃 2 √1 e−(5−𝜃) /2 2𝜋 𝑐1 e−(𝜃−7/2) 𝜃 total 2 √1 e−(𝜃−2) /2 2𝜋 1 𝑑𝜃 no sum 2 posterior 𝑓(𝜃 | 𝑥 = 5) 𝑑𝜃 𝜙(𝑥 = 5 | 𝜃)𝑓(𝜃) 𝑑𝜃 𝜙(𝑥 = 5) 𝑐2 e−(𝜃−7/2) 𝜙(𝑥 = 5) = ∫ 𝜙(𝑥 = 5 | 𝜃)𝑓(𝜃) 𝑑𝜃 We can see by the form of the posterior pdf that it is a normal distribution. Because the 2 2 exponential for a normal distribution is e−(𝜃−𝜇) /2𝜎 we have mean 𝜇 = 7/2 and 2𝜎2 = 1, so variance 𝜎2 = 1/2. We don’t need to bother computing the total probability; it is just used for normalization 1 and we already know the normalization constant √ for a normal distribution. To 𝜎 2𝜋 summarize, The posterior pdf follows a N(7/2, 1/2) distribution. Here is the graph of the prior and the posterior pdfs for this example. Note how the data ‘pulls’ the prior (the wider bell on the left) towards the data. The posterior is the narrower bell on the right. After collecting data, we have a new opinion about the mean, and we are more sure of this new opinion. 1 2 7 0.0 0.1 0.2 0.3 0.4 0.5 18.05 Class 14, Continuous Data with Continuous Priors, Spring 2022 0 1 2 3 4 5 6 7 prior = orange; posterior = blue; data = red line Now we’ll repeat the previous example for general 𝑥. When reading this if you mentally substitute 5 for 𝑥 you will understand the algebra. Example 4. Suppose our data 𝑥 is drawn from a normal distribution with unknown mean 𝜃 and standard deviation 1. 𝑥 ∼ N(𝜃, 1) Suppose further, that our prior distribution for the unknown parameter 𝜃 is 𝜃 ∼ N(2, 1). Solution: As before, we show the algebra used to simplify the Bayes numerator: The prior pdf and likelihood function are 1 2 𝑓(𝜃) = √ e−(𝜃−2) /2 2𝜋 1 2 𝑓(𝑥 | 𝜃) = √ e−(𝑥−𝜃) /2 . 2𝜋 The Bayes numerator is the product of the prior and the likelihood: 1 1 2 2 prior ⋅ likelihood = √ e−(𝜃−2) /2 ⋅ √ e−(𝑥−𝜃) /2 2𝜋 2𝜋 1 −(2𝜃2 −(4+2𝑥)𝜃+4+𝑥2 )/2 e = 2𝜋 1 −(𝜃2 −(2+𝑥)𝜃+(4+𝑥2 )/2) e (complete the square) = 2𝜋 1 −((𝜃−(1+𝑥/2))2 −(1+𝑥/2)2 +(4+𝑥2 )/2) = e 2𝜋 2 = 𝑐1 e−(𝜃−(1+𝑥/2)) Just as in the previous example, in the last step we replaced all the constants, including the exponentials that just involve 𝑥, by by the simple constant 𝑐1 . Now the Bayesian update table becomes 18.05 Class 14, Continuous Data with Continuous Priors, Spring 2022 hypothesis prior likelihood Bayes numerator 𝜃 𝑓(𝜃) 𝑑𝜃 𝜙(𝑥 | 𝜃) 𝜙(𝑥 | 𝜃)𝑓(𝜃) 𝑑𝜃 2 √1 e−(𝑥−𝜃) /2 2𝜋 𝑐1 e−(𝜃−(1+𝑥/2)) 𝜃 2 √1 e−(𝜃−2) /2 2𝜋 𝑑𝜃 𝜃 𝜃 ∼ N(2, 1) 𝑥 ∼ N(𝜃, 1) total 1 no sum 8 2 posterior 𝑓(𝜃 | 𝑥) 𝑑𝜃 𝜙(𝑥 | 𝜃)𝑓(𝜃) 𝑑𝜃 𝜙(𝑥) 𝑐2 e−(𝜃−(1+𝑥/2)) 2 𝜃 ∼ N(1 + 𝑥/2, 1/2) 𝜙(𝑥) = ∫ 𝜙(𝑥 | 𝜃)𝑓(𝜃) 𝑑𝜃 1 As in the previous example we can see by the form of the posterior that it must be a normal distribution with mean 1 + 𝑥/2 and variance 1/2. That is, The posterior pdf follows a N(1 + 𝑥/2, 1/2) distribution. You should compare this with the case 𝑥 = 5 in the previous example. 7 Predictive probabilities Since the data 𝑥 is continuous it has prior and posterior predictive pdfs. The prior predictive pdf is the total probability density computed at the bottom of the Bayes numerator column: 𝜙(𝑥) = ∫ 𝑓(𝑥|𝜃)𝑓(𝜃) 𝑑𝜃, where the integral is computed over the entire range of 𝜃. The posterior predictive pdf has the same form as the prior predictive pdf, except it uses the posterior probabilities for 𝜃: 𝜙(𝑥2 |𝑥1 ) = ∫ 𝜙(𝑥2 |𝜃, 𝑥1 )𝑓(𝜃|𝑥1 ) 𝑑𝜃, We usually assume that 𝑥1 and 𝑥2 are conditionally independent. That is, 𝜙(𝑥2 |𝜃, 𝑥1 ) = 𝜙(𝑥2 |𝜃). In this case the formula for the posterior predictive pdf is a little simpler: 𝜙(𝑥2 |𝑥1 ) = ∫ 𝜙(𝑥2 |𝜃)𝑓(𝜃|𝑥1 ) 𝑑𝜃.