Critical Thinking Sampling

The Big Picture of Statistics Chapter 1 Section1-3: Critical Thinking Sampling Section 1-4: Design of Experiments 2 1 Population versus sample  Population: The entire group of individuals in which we are interested but can’t usually assess directly
Sample: The part of the population we actually examine and for which we do have data How well the sample represents the population depends on the sample design. Example: All humans, all working-age people in California, all crickets parameter is a number describing a characteristic of the population. A Sampling statistic is a number describing a characteristic of a sample.
A The goal is to select a representative sample from the population for costeffective data collection so that inferences can be drawn about the population it comes from. 3 Sampling 4 Bad sampling methods--Bias
But how can we choose a sample that we can trust to represent the population?
A sampling design describes exactly how to choose a sample from the population. 5
The sample design is biased if it systematically favors certain outcomes.
Example: consider a research project on attitudes toward sex. Collecting the data by publishing a questionnaire in a magazine and asking people to fill it out and send it in would produce a biased sample. People interested enough to spend their time and energy filling out and sending in the questionnaire are likely to have different attitudes toward sex than those not taking the time to fill out the questionnaire. 6 1 Bad sampling methods: Bad sampling methods: Convenience sampling: Just ask whoever is around.
“Man on the street” survey (cheap,
Voluntary response sampling: Individuals choose to be involved. These samples are very susceptible to being biased because different people are motivated to respond or not. They are often called “public opinion polls” and are not considered valid or scientific.
Bias: Sample design systematically favors a particular outcome. convenient, often quite opinionated or emotional) 
Ask about gun control or legalizing marijuana “on the street” in Berkeley, CA and in some small town in Idaho and you would probably get totally different answers. Bias: Opinions limited to individuals present 8 7 On-line surveys: Example of voluntary response sample: Bias: People have to care enough about an issue to bother replying. This sample is probably a combination of people who hate “wasting the taxpayers’ money” and “animal lovers.” Ann Landers summarizing responses of readers: Seventy percent of (10,000) parents wrote in to say that having kids was not worth it—if they had to do it over again, they wouldn’t. Bias: Most letters to newspapers are written by disgruntled people. A random sample showed that 91% of parents WOULD have kids again. 10 9 Good sampling methods: Simple Random Sample (SRS) Probability or random sampling: Individuals are
randomly selected. No one group should be overrepresented.
Sampling randomly gets rid of bias. In SRS each member of the population has an equal chance to be selected. We can use a Random Number Table, or calculator to generate the random numbers to select our SRS. Random samples rely on the absolute objectivity of random numbers. There are books and tables of random digits available for random sampling. 11 12 2 Stratified Random Samples Stratified Random Sample: A stratified random sample is essentially a series of SRS performed on subgroups of a given population. The subgroups are chosen to contain all the individuals with a certain characteristic. For example: 1. 2. Classify the population into groups of similar individuals, called strata. Choose a separate SRS in each stratum and combine these SRSs to form the full sample. For example:  Divide the population of VC students into males and females. • Stratified random sample of 100 male and 150 female VC students  Divide the population of California by major ethnic group. •  Divide the counties in America as either urban or rural based on a criterion of population density. Stratified random sample of a total of 100 Californians, representing proportionately the major ethnic groups 13 Stratified Random Sampling 14 Other methods strata
Cluster sample: first select clusters at random, and then use all the individuals in the clusters as your sample. The SRS taken within each group in a stratified random sample need not be of the same size. 15 Other methods 16 Other methods Multistage samples use multiple stages of stratification. They are
Systematic sampling: “count off” your population by a number k with a random start often used by the government to obtain information about the U.S. population. Data are obtained by taking an SRS for each substrata. Example: Sampling both urban and rural areas, people in different ethnic and income groups within the urban and rural areas, and then individuals of different ethnicities within those strata. k=5 17 18 3 Caution about sampling surveys Summary of good sampling methods:
Nonresponse: People who feel they have something to hide or who don’t like their privacy being invaded probably won’t answer. Yet they are part of the population.
Response bias: Fancy term for lying when you think you should not tell the truth. Like if your family doctor asks: “How much do you drink?” Or a survey of female students asking: “How many men do you date per week?” People also simply forget and often give erroneous answers to questions about the past.
SRS
Stratified Random Sampling
Cluster Sampling
Multistage Sampling
Systematic Sampling with Random Start
Wording effects: Questions worded like “Do you agree that it is awful that…” are prompting you to give a particular response. 19  20 Learning about populations from samples Undercoverage Undercoverage occurs when parts of the population are left out in the process of choosing the sample. The techniques of inferential statistics allow us to draw inferences or conclusions about a population from a sample. Because the U.S. Census goes “house to house,” homeless people are not represented. Illegal immigrants also avoid being counted. Geographical districts with a lot of undercoverage tend to be poor ones. Representatives from richer areas typically strongly oppose statistical adjustment of the census.  Your estimate of the population is only as good as your sampling design  Work hard to eliminate biases.  Your sample is only an estimate—and if you randomly sampled again, you would probably get a somewhat different result. Historically, clinical trials have avoided including women in their studies because of their periods and the chance of pregnancy. This means that medical treatments were not appropriately tested for women. This problem is slowly being recognized and addressed.  The bigger the sample the better. 21 22 Sampling Sampling In 1993, presidential candidate Ross Perot appeared on television to voice his opinions on government reform. To gauge public opinion, Perot urged viewers to fill out the survey appearing in that week’s issue of TV Guide. Of the approximately 1.4 million respondents, 98% agreed with Ross Perot’s platform on health care reform. What type of sampling method was used? a) Convenience sample b) Voluntary response sample c) Simple random sample d) Stratified sample e) Multistage sample Which of the following is NOT a type of probability sample? 23 a) b) c) d) e) Voluntary response sample Simple random sample Stratified sample Multistage sample All are probability samples. 24 4 Sampling Sampling Suppose you want to estimate the proportion of students at a large university that approve of the new health care bill. You take an SRS of 200 of the 25,000 undergraduate students and an SRS of the 100 of the 5,000 graduate students. This overall sample is a Which of the following sampling methods describes a multistage sample of 200 undergraduate students at a large university? a) b) c) d) e) a) b) Voluntary response sample Simple random sample Stratified sample Multistage sample None of the above c) Obtain a list of the undergraduate students at the large university; assign consecutive numbers to the students on the list; use a random number table to select 200 university students. Obtain a list of all freshmen, sophomores, juniors, and seniors; use a random number table to randomly select 50 students from each class. Randomly select 10 colleges; within each college, randomly select 5 majors; from each of the selected majors randomly select 4 undergraduate students. 26 25 Bias Bias A Gallup poll sponsored by the disposable diaper industry asked “It is estimated that disposable diapers account for less than 2% of the trash in today’s landfills. In contrast, beverage containers, third-class mail, and yard waste are estimated to account for about 21% of the trash in landfills. Given this, in your opinion, would it be fair to ban disposable diapers?” Which type of bias does this poll suffer from? a) Under-coverage bias b) Non-response bias c) Response bias d) Question wording bias e) Interviewer bias True or false: Bias can occur in both probability and non-probability samples. a) b) True False 28 27 Bias Probability samples Using a local telephone book to select a simple random sample could introduce what type of bias? An important characteristic of probability samples is that a) Results of one sample are usually the same as the results for the entire population. b) Results of one sample are usually the same as the results for another sample of the same size. c) Principles of chance apply and the size of error due to sampling is measurable. d) Smaller samples give more accurate results than larger samples. a) b) c) d) e) Under-coverage bias Non-response bias Response bias Question wording bias Interviewer bias 29 30 5 Observation

Observes individuals and measures variables of interest but does not attempt to influence the responses. It is difficult to establish convincing evidence of a causal relationship because of lack of control over outside variables (lurking variables). Terminology Experiment

Deliberately imposes some treatment on individuals in order to observe their responses. Allows researchers to take control of lurking variables by randomized assignment to treatments, which helps provide more convincing evidence of causation
The individuals in an experiment are the experimental units. If they are human, we call them subjects.
The explanatory variables in an experiment are often called factors.
A treatment is any specific experimental condition applied to the subjects. 31
32 How to experiment badly If the experiment involves giving two different doses of a drug, we say that we are testing two levels of the factor.
A response to a treatment is statistically significant if it Subjects Treatment Measure response In a controlled environment of a laboratory (especially if human subjects are not being used), a simple design like this one, where all subjects receive the same treatment, can work well. is larger than you would expect by chance (due to random variation among the subjects). In a study of sickle cell anemia, 150 patients were given the drug hydroxyurea, and 150 were given a placebo (dummy pill). The researchers  counted the episodes of pain in each subject. Identify: Field experiments and experiments with human subjects are exposed to more variable conditions and deal with more variable subjects. •The subjects (patients, all 300) A simple design often yields worthless results because of confounding with lurking variables.  •The factors/treatments (hydroxyurea and placebo) •And the response variable (episodes of pain) 33 3 Principles of experimental design 34 Principles of comparative experiments 1. Control the effects of lurking variables on the response, most simply by comparing two or more treatments. Experiments are comparative in nature: We compare the response to a treatment versus to  another treatment 2. Randomize—use impersonal chance to assign subjects to treatments.  no treatment (a control)  a placebo  or any combination of the above 3. Replicate—use enough subjects in each group to reduce chance variation in the results. 35 36 6 The Placebo Effect A control is a situation in which no treatment is administered. It serves as a reference mark for an actual treatment (e.g., a group of subjects does not receive any drug or pill of any kind). The “placebo effect” is an improvement in health due not to any treatment but only to the patient’s belief that he or she will improve. A placebo is a fake treatment, such as a sugar pill. It is used to test the hypothesis that the response to the treatment is due to the actual treatment and not to how the subject is being taken care of.  The placebo effect is not understood, but it is believed to have therapeutic results on up to a whopping 35% of patients.  It can sometimes ease the symptoms of a variety of ills, from asthma to pain to high blood pressure and even to heart attacks.  An opposite, or “negative placebo effect,” has been observed when patients believe their health will get worse. 37 38 Designing “controlled” experiments Fisher’s solution: Sir Ronald Fisher—The “father of statistics” He was sent to Rothamsted Agricultural Station in the UK to evaluate the success of various fertilizer treatments. Randomized comparative experiments Fisher found the data from experiments that had been going on for decades to be basically worthless because of poor experimental design.
Fertilizer had been applied to a field one year and not in another in order to compare the yield of grain produced in the two years. BUT
It may have rained more, or been sunnier, in different years.
The seeds used may have differed between years as well.  Or fertilizer was applied to one field and not to a nearby field in the same year. BUT
The fields might have different soil, water, drainage, and history of previous use.  Too many factors affecting the results were “uncontrolled.” 
In the same field and same year, apply fertilizer to randomly spaced plots within the field. Analyze plants from similarly treated plots together. This minimizes the effect of variation of drainage and soil composition within the field on yield as well as controlling for weather. F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F 39 F 40 Getting rid of sampling biases Completely randomized designs The best way to exclude biases in an experiment is to randomize the design. Both the individuals and treatments are assigned randomly. A double-blind experiment is one in which neither the subjects nor the experimenter know which individuals got which treatment until the experiment is completed. In a completely randomized experimental design, individuals are randomly assigned to groups, then the groups are randomly assigned to treatments. Another way to make sure your conclusions are robust is to replicate your experiment—do it over. Replication ensures that particular results are not due to uncontrolled factors or errors of manipulation. 41 42 7 Matched pairs designs The “best” experiment
The most reliable way to determine whether the explanatory variable is actually causing changes in the response variable is to carry out a randomized controlled double-blind experiment. Depending on the variables of interest, such a design may not be entirely feasible but the closer researchers get to achieving this ideal design, the more convincing are their claims of causation (or lack thereof). Matched pairs: Choose pairs of subjects that are closely matched— e.g., same sex, height, weight, age, and race. Within each pair, randomly assign who will receive which treatment. It is also possible to just use a single person and give the two treatments to this person over time in random order. In this case, the “matched pair” is just the same person at different points in time. The most closely matched pair studies use identical twins. 43 44 Block designs Matched pair designs example In a block design, subjects are divided into groups, or blocks, prior to the experiment to test hypotheses about differences between the groups.
The blocking, or stratification, here is by gender. 45 Suppose researchers want to compare the relative merits of toothpastes with and without tartar control ingredients. In order to make the comparison between individuals who are as similar as possible with respect to background and diet, they could obtain a sample of identical twins. One of each pair would randomly be assigned to brush with the tartar control toothpaste, while the other would brush with regular toothpaste of the same brand. These would be provided in unmarked tubes so that the subjects would be blind. To make the experiment double-blind, dentists who evaluate the results would not know until afterwards who used which toothpaste. 46 Matched pair design example 47 8