Introductory Statistics, Chapter 4

Sample surveys and experiments „ Most of what we’ve done so far is data Math 140 Introductory Statistics Sample surveys and experiments „ „ „ „ „ Pollster: I asked a hundred likely voters who they planned to vote for, and fifty-two of them said they’d vote for you. Politician: Does that mean I’ll win the election? Pollster: Sorry, I can’t tell you. My stat course hasn’t gotten to inference yet. Politician: What’s inference? Pollster: Drawing conclusions based on your data. I can tell you about the hundred people I actually talked to, but I don’t yet know how to use that information to tell you about all the likely voters. exploration—ways to uncover, display, and describe patterns in data. Unfortunately, these patterns can’t take you beyond the data in hand. With exploration, what you see is all you get. Often, that’s not enough. Sample surveys and experiments „ Methods of inference can take you beyond the data you actually have, but only if your numbers come from the right kind of process. „ If you want to use 100 likely voters to tell you about all likely voters, how you choose those 100 voters is crucial. „ The quality of your inference depends on the quality of your data; in other words, bad data lead to bad conclusions. 1 4.1 Why Take samples, and How Not To. Census Versus Sample Discussion. D1 „ Population: Set of people or things we want to „ In which of these situations do you think a census is study. „ Unit: Individual element of the population. „ Population Size: Number of units. „ Sample: set of units that you do get to study. used to collect data, and in which do you think sampling is used? Explain your reasoning. „ „ „ „ „ Census: collecting data on the entire population. „ a. An automobile manufacturer inspects its new models. b. A cookie producer checks the number of chocolate chips per cookie. c. The U.S. president is determined by an election. d. Weekly movie attendance figures are released each Sunday. e. A Los Angeles study does in-depth interviews with teachers in order to find connections between nutrition and health. Discussion Bias: A problem with survey data. „ You want to estimate the average number of „ A sampling method is biased if it tends to give TV sets per household in your community. samples in which some characteristic of the population is overrepresented or underrepresented. „ An Unbiased Sample Method requires that all units in the population have a chance of being in the sample. „ A sampling frame is the list of units you use to create the sample. “bad frame, bad sample”. a. What is the population? What are the units? „ b. Explain the advantages of sampling over conducting a census. „ c. What problems do you see in carrying out this sample survey? „ 2 Bias: (dialogue page 222) Bias: Sample vs. method used for choosing the sample (dialogue p. 222) Sampling Bias Sample Bias Discussion (D5) „ Size bias: Larger units are more likely to be included. „ Voluntary Response Bias: Those who care about the issue respond. „ Convenience Sample Bias: Units are chosen because of convenience. „ Judgment Sample Bias: Units are chosen according to the judgment of someone (expert) You want to know the percentage of voters who favor state funding for bilingual education. Your population of interest is the set of people likely to vote in the next election. You use as your frame the phone book listing of residential telephone numbers. „ How well do you think the frame represents the population? „ Are there important groups of individuals who belong to the population but not to the frame? To the frame but not to the population? „ If you think bias is likely, identify what kind of bias and how it might arise. 3 Response Bias Response Bias „ Non-Response Bias: You get no data or not enough data. e.g. 80% of people contacted refuse to answer a Survey „ Questionnaire Bias: Arises from the way the questions are asked. „ Bias from incorrect responses: Might be the result of intentional lying (often, the people being interviewed want to be agreeable and tend to respond in the way they think the interviewer wants them to respond), but it is more likely to come from inaccurate measuring devices, including inaccurate memories of people being interviewed in self-reported data. Response Bias Response Bias „ Reader’s Digest commissioned a poll to determine how the wording of questions affected people’s opinions. The same 1031 people were asked to respond to these two statements: 1. 2. I would be disappointed if Congress cut its funding for public television. Cuts in funding for public television are justifi ed as part of an overall effort to reduce federal spending. „ Note that agreeing with the first statement is pretty much the same as disagreeing with the second. Agreed Disagreed Didn’t know Statement 1 54% 40% 6% Statement 2 52% 37% 10% 4 4.2 Randomizing: Playing It Safe by Taking Chances Randomize: Choose a sample by chance. This is the only method guaranteed to be unbiased. „ Simple random sample (SRS) „ Stratified random samples „ Cluster samples Simple random sample (SRS) „ In a SRS all possible samples of a given fixed size are equally likely. That is all units have the same chance of being in the sample, all possible triples of units have the same chance, and so on. „ Steps in choosing a SRS „ „ Two (or more) stage samples „ „ Systematic samples with a random start. „ 1. Start with a list of all units in the population. (a frame) 2. Number the units in the list. 3. Use a random number table or generator to choose units from the numbered list, one at a time, until you have as many as you need. Stratified random samples Why stratify „ 1. Divide the units of the sample into non- „ Convenience. It is easier to sample in smaller overlapping subgroups (strata) „ 2. Choose a SRS from each subgroup (stratum) more compact groups. „ Coverage. Each stratum is assured to be covered. (this may not happen with a SRS) „ Precision. The results may be more precise if the measurement we are interested varies a lot from stratum to stratum. Choose the relative sample sizes proportional to the stratum sizes. 5 Cluster samples Two (or more) stage samples „ 1. Create a numbered list of all the clusters in „ 1. Create a numbered list of clusters. the population. „ 2. Choose a SRS of clusters „ 3. Obtain data on each unit in each chosen cluster. „ 2. Choose a SRS of clusters. Systematic samples with a random start „ 3. From each selected cluster, create a list of individuals and choose a SRS from each (selected) cluster. Summary of sampling methods „ 1. By a method, such as counting off, divide your population into groups of the size you want for your sample. „ 2. Use a chance method to choose one of the groups for your sample. 6 Activity 4.2 Part 1. (page 225) „ Quickly choose 5 rectangles. „ Calculate the areas of each of your 5 rectangles „ Calculate the mean (average) of these areas. Results 16 14 12 10 8 Std „ Rectangles: 29, 46, 59, 71, 83 6 Keep your sample data for future reference. „ Areas: 4 10, 3, 8, 4, 2 2 „ Mean: (10+3+8+4+2)/5 0 1. 5 3. 0 0 4. 0 5 6. 0 00 7. 50 9. 10 00 . 12 50 . 13 00 . 15 50 . 16 00 .5 0 = 5.4 Activity 4.2 Part 2. „ Choose 5 random numbers between 1 and 100. Look for the rectangles associated to these numbers. Use randInt(1,100) „ Calculate the areas of each of these 5 rectangles „ Calculate the mean (average) of these areas. Results 1 (Computer Simulated n = 200) 30 50 45 25 40 35 20 15 10 30 25 20 SRS 0 5 0 1. 5 3. 0 0 4. 0 50 6. 0 7. 0 5 9. 0 10 00 12.50 13.00 15.50 16.00 .5 0 15 10 1. 5 3. 0 0 4. 0 5 6. 0 00 7. 50 9. 10 00 . 12 50 . 13 00 . 15 50 . 16 00 .5 0 5 Keep your sample data for future reference. Std 7 Results 2 (Computer Simulated n = 1000) 30 300 25 250 20 200 15 10 100 5 50 0 0 1. 5 3. 0 0 4. 0 50 6. 0 7. 0 5 9 0 10.00 12.50 13.00 15.50 16.00 18.50 .0 0 150 1. 5 3. 0 0 4. 0 5 6. 0 0 7. 0 5 9. 0 10 00 .5 12 0 . 13 00 . 15 50 .0 16 0 .5 0 Std 4.3 Experiments and Inference about Cause „ Cause and Effect SRS Experiments and Inference about Cause Experiments „ Lurking Variable: „ Goal: To establish cause and effect by A variable in the background that could explain a pattern between the variables investigated. „ How to establish cause and effect? Answer: Conduct an experiment. comparing two or more conditions (called treatments) using an outcome variable (called the response). „ To be a real experiment, the subjects must be randomly assigned to their treatments. To make this distinction sometimes we call these Randomized Experiments. 8 Example: Kelly’s Hamsters Example: Kelly’s Hamsters „ Assumptions „ Subjects: Eight golden hamsters. Golden Hamsters hibernate. Hamsters rely on the amount of daylight to trigger hibernation. „ An animal’s capacity to transmit nerve impulses depends in part on an enzyme called Na+K+ ATPase. „ Question: If you reduce the amount of light a hamster gets, from 16 hours to 8 hours per day, what happens to the concentration of Na+K+ ATP-ase. „ „ „ Treatments: Raised in long days (16 hours) or short days (8 hours) of daylight. „ Random Assignment of Treatments: Kelly randomly assigns four of the hamsters to short days, and four to long days. „ Replication: Each treatment was given to four hamsters. „ Response Variable: Enzyme concentration. Kelly’s Hamsters (Results) Kelly’s defense of her design „ Results Kelly: Enzyme concentrations in milligrams per 100 milliliters. Short Days 12.500 11.625 18.275 13.225 Long Days 6.625 10.375 9.900 8.800 I claim that the observed difference in enzyme concentrations between the two groups of hamsters is due to the difference in daylight. Skeptic: Wait a minute. As you can see, the concentration varies from one hamster to another. Some just naturally have higher concentrations. If you happened to assign all the high-enzyme hamsters to the group that got short days, you’d get results like the ones you got. Kelly: I agree, and I was concerned about that possibility. In fact, that’s precisely why I assigned day lengths to hamsters by using random numbers. The random assignment makes it extremely unlikely that all the high-enzyme hamsters would get assigned to the same group. If you have the time, I can show you how to compute the probability. Skeptic: (Hastily) That’s OK for now. I’ll take your word for it. But maybe you can catch me in Chapter 6. 9 Discussion D21 (page 245) Confounding in Observational Studies „ D21. Kelly has shown that hamsters raised in „ Confounded: mixed-up, confused, at a dead less daylight have higher hormone concentration than hamsters raised with more daylight. In order for Kelly to show that less daylight causes an increase in the hormone concentration, she must convince us that there is no other explanation. Has she done that? end. „ Two possible influences on an observed outcome are said to be confounded if they are mixed together in a way that makes it impossible to separate their effects. Confounding in Observational Studies Confounding in Observational Studies „ Studies that claim to show that review courses increase SAT Imagine yourself in this situation: „ You know that many infants are dying of what seem to be respiratory obstructions. „ You begin to do autopsies on infants who die with respiratory symptoms. „ The infants all have thymus glands that look too big in comparison to body size. Aha! That must be it: „ The respiratory problems are caused by an enlarged thymus. „ It became quite common in the early 1900s for surgeons to treat respiratory problems in children by removing the thymus. Even though a third of the children who were operated on died. scores often ignore the important concept of confounding. „ In one study, students at a large high school were offered an SAT preparation course, and SAT scores of students who completed the course were higher than scores of students who chose not to take the course. „ The positive effect of the review course was confounded with the fact that the course was taken only by volunteers, who would tend to be more motivated to do well on the SAT. „ Consequently, you can’t tell if the higher scores of those who took the course were due to the course itself or to the higher motivation of the volunteers. 10 Confounding in Observational Studies Experiments vs. Observational Studies „ The best solution to guard against confounding: To randomize. „ Observational Study: No treatment gets assigned to „ The doctors couldn’t know whether children with a large thymus tend to have more respiratory problems, because they have no evidence about children with a smaller thymus. Age and size of thymus were confounded. the subjects by the experimenter. „ (Randomized) Experiment: Comparing results of treatments assigned to subjects at random. „ Clinical Trial: Randomized experiment comparing medical treatments. „ For observational studies the conditions are called factors, (not treatments) Factors and Levels Why randomization makes inference possible? „ The term factor is also used for experiments when there are „ By assigning treatments to units at random, many characteristics that want to be compared. „ The different values that a factor may take are called levels. „ Example. If Kelly added the type of diet to her experiment. there are only two possible causes for a difference in the responses to the treatments: chance or the treatments. Factor 1 Type of Diet Factor 2 Length of Day Light Heavy Short Light-Short Heavy-Short Long Light-Long Heavy-Long „ If the probability is small that chance alone will give such a difference in the responses, then we can infer that the cause of the difference was the treatment. 11 Control or Comparison Group „ Anecdotal evidence is not proof. Why? „ Placebo Effect: When people believe they are getting „ „ „ „ special treatment they tend to improve. Control Group: A group of people given a placebo. Comparison Group: A group of people given the standard treatment (when comparing against a new treatment). Blind Experiment: People do not know which treatment they are given. Double Blind Experiment: patients and doctors do not know which treatment they are assigned. 12