18.05 S22 Class 17: The Frequentist School of Statistics

The Frequentist School of Statistics Class 17, 18.05 1 Learning Goals 1. Be able to explain the difference between the frequentist and Bayesian approaches to statistics. 2. Know our working definition of a statistic and be able to distinguish a statistic from a non-statistic. 2 Introduction After much foreshadowing, the time has finally come to switch from Bayesian statistics to frequentist statistics. For much of the twentieth century, frequentist statistics has been the dominant school. If you’ve ever encountered confidence intervals, 𝑝-values, 𝑡-tests, or 𝜒2 tests, you’ve seen frequentist statistics. With the rise of high-speed computing and big data, Bayesian methods are becoming more common. After we’ve studied frequentist methods we will compare the strengths and weaknesses of the two approaches. 2.1 The fork in the road Both schools of statistics start with probability. In particular both know and love Bayes’ theorem: 𝑃 (𝐷|𝐻) 𝑃 (𝐻) 𝑃 (𝐻|𝐷) = . 𝑃 (𝐷) When the prior is known exactly, all statisticians will use Bayes’ formula. For Bayesian inference we take 𝐻 to be a hypothesis and 𝐷 some data. Over the last few weeks we have seen that, given a prior and a likelihood model, Bayes’ theorem is a complete recipe for updating our beliefs in the face of new data. This works perfectly when the prior was known perfectly. We saw this in our dice examples. We also saw examples of a disease with a known frequency in the general population and a screening test of known accuracy. In practice we saw that there is usually no universally accepted prior – different people will have different a priori beliefs – but we would still like to make useful inferences from data. Bayesians and frequentists take fundamentally different approaches to this challenge, as summarized in the figure below. 1 18.05 Class 17, The Frequentist School of Statistics, Spring 2022 𝑃 (𝐻|𝐷) = Probability (mathematics) 2 Everyone uses Bayes’ 𝑃 (𝐷|𝐻)𝑃 (𝐻) formula when the prior 𝑃 (𝐷) 𝑃 (𝐻) is known. Bayesian path Frequentist path Statistics (art) 𝑃Posterior (𝐻|𝐷) = 𝑃 (𝐷|𝐻)𝑃prior (𝐻) 𝑃 (𝐷) Bayesians require a prior, so they develop one from the best information they have. Likelihood 𝐿(𝐻; 𝐷) = 𝑃 (𝐷|𝐻) Without a known prior, frequentists draw inferences from just the likelihood function. The reasons for this split are both practical (ease of implementation and computation) and philosophical (subjectivity versus objectivity and the nature of probability). 2.2 What is probability? The main philosophical difference concerns the meaning of probability. The term frequentist refers to the idea that probabilities represent long term frequencies of repeatable random experiments. For example, ‘a coin has probability 1/2 of heads’ means that the relative frequency of heads (number of heads out of number of flips) goes to 1/2 as the number of flips goes to infinity. This means the frequentist finds it nonsensical to specify a probability distribution for a parameter with a fixed value. While Bayesians are happy to use probability to describe their incomplete knowledge of a fixed parameter, frequentists reject the use of probability to quantify degree of belief in hypotheses. Example 1. Suppose I have a bent coin with unknown probability 𝜃 of heads. The value of 𝜃 may be unknown, but it is a fixed value. Thus, to the frequentist there can be no prior pdf 𝑓(𝜃). By comparison the Bayesian may agree that 𝜃 has a fixed value, but interprets 𝑓(𝜃) as representing uncertainty about that value. Both the Bayesian and the frequentist are perfectly happy with 𝑝(heads | 𝜃) = 𝜃, since the longterm frequency of heads given 𝜃 is 𝜃. In short, Bayesians put probability distributions on everything (hypotheses and data), while frequentists put probability distributions on (random, repeatable, experimental) data given a hypothesis. For the frequentist when dealing with data from an unknown distribution only the likelihood has meaning. The prior and posterior do not. 3 Working definition of a statistic Our view of statistics is that it is the art of drawing conclusions (making inferences) from data. With that in mind we can make a simple working definition of a statistic. There is a more formal definition, but we don’t need to introduce it at this point. 18.05 Class 17, The Frequentist School of Statistics, Spring 2022 3 Statistic. A statistic is anything that can be computed from data and known values. Sometimes to be more precise we’ll say a statistic is a rule for computing something from data and the value of the statistic is what is computed. This can include computing likelihoods where we hypothesize values of the model parameters. But it does not include anything that requires we know the true value of a model parameter with unknown value. Examples. 1. The mean of data is a statistic. It is a rule that says given data 𝑥1 , … , 𝑥𝑛 𝑛 compute 𝑥1 +…+𝑥 . 𝑛 2. The maximum of data is a statistic. It is a rule that says to pick the maximum value of the data 𝑥1 , … , 𝑥𝑛 . 3. Suppose 𝑥 ∼ N(𝜇, 32 ) where 𝜇 is unknown. Then the likelihood 𝜙(𝑥|𝜇 = 7) = (𝑥−7)2 1 √ e− 18 3 2𝜋 is a statistic. However, the distance of 𝑥 from the true mean 𝜇 is not a statistic since we cannot compute it without knowing 𝜇 𝑥−5 4. If our data 𝑥1 , … , 𝑥𝑛 is drawn from N(𝜇, 32 ), where 𝜇 is unknown, then 𝑧 = √ is a 3/ 𝑛 statistic, since it is computed from the data and known values. More generally, if 𝜇0 is a 𝑥 − 𝜇0 𝑥−𝜇 √ is a statistic. However, since 𝜇 is not known, 𝑧 = √ is known value then 𝑧 = 3/ 𝑛 3/ 𝑛 not a statistic. Note. We will usually stick with our Bayesian practice of using the symbol 𝜙 for continuous likelihoods. This will help remind us that Frequentists don’t have prior and posterior probabilities for hypotheses. Point statistic. A point statistic is a single value computed from data. For example, the mean and the maximum are both point statistics. The maximum likelihood estimate is also a point statistic since it is computed directly from the data based on a likelihood model. Interval statistic. An interval statistic is an interval computed from data. For example, the range from the minimum to maximum of 𝑥1 , … , 𝑥𝑛 is an interval statistic, e.g. the data 0.5, 1.0, 0.2, 3.0, 5.0 has range [0.2, 5.0]. Set statistic. A set statistic is a set computed from data. Example. Suppose we have five dice: 4, 6, 8, 12 and 20-sided. We pick one at random and roll it once. The value of the roll is the data. The set of dice for which this roll is possible is a set statistic. For example, if the roll is a 10 then the value of this set statistic is {12, 20}. If the roll is a 7 then this set statistic has value {8, 12, 20}. It’s important to remember that a statistic is itself a random variable since it is computed from random data. For example, if data is drawn from N(𝜇, 𝜎2 ) then the mean of 𝑛 data points follows the distribution N(𝜇, 𝜎2 /𝑛)). Sampling distribution. The probability distribution of a statistic is called its sampling distribution. Point estimate. We can use statistics to make a point estimate of a parameter 𝜃. For example, if our data is drawn from a normal distribution with unknown mean 𝜃. Then the data mean 𝑥̄ is a point estimate of 𝜃.