The Validity of Intuitions

Agenda for today Go over the homework problems that were due today Intuitive inference of data & information What makes a good project Example -- study from last week Some final questions Critical thinking Critical thinking • Observing and recording data is the central aspect of social science • Sometimes, the data at hand is not ideal! • The question we will start addressing today is what can we learn from different types of data. Questions • Please provide an answer to each of the following questions. • • For each, indicate: • what you believe in • and why. Q1: Cholesterol & heart diseases • A large study in Framingham found that people with high cholesterol levels are much more likely to have heart attacks • It was also found that diets and medication can reduce cholesterol • Physicians recommendations for diets and medications reduced heart attacks only by a small number and did not reduce mortality! • What is going on? Q2: Uri Geller • In one TV show Uri Geller told the viewers that he will activate broken clocks in their homes. • After Geller concentrated, many people called in to report that their clocks started working • Does this means that Uri Geller had such powers? How would you test it? Q3: Reward and punishment • It has been well documented that rewarding good behavior is more effective than punishing bad behavior. • It was also observed that in flight school complementing excellent performance is usually followed by bad performance, while criticizing bad performance is usually followed by better performance. • What is going on? Q4: The “hot hand” • Many people believe that athletes (such as basketball players) can be in a “hot” state (for example have a “hot hand”), such that they are more likely to score after they have just scored • The hot hand can have many reasons • Such as …. • But, is it real? Q5: Discrimination? • UC Berkeley in the fall of 1973 • 8,442 men and 4,321 women applied • 44% of the men and 35% of the women were admitted • The quality of the applicants was the same • Is this discrimination? Q6: Stock predictions • One stock broker provides monthly predictions that correctly predicted 7 of the 9 downturns in the economy over the past 3 years • Is he an expert? • Do you want to invest with this person? Q7: Medical diagnosis • Imagine that a blood test identified a person with a rare disease (base rate of 0.1% in the population). • This test identifies as a positive result 98% of the people who have the disease but also 4% of the healthy population. • Given this performance, how likely is it that the person who was identify as have the disease, indeed has it? Q8: “The bible code” • It turn out that if you start from the last letter of the first word in the bible and count every 50th letter, you get the word “TURA” -- which means bible in Hebrew • There are many such examples of hidden meanings in the bible and this has been suggested as evidence that the bible was not written by men. • What do you think? Evidence…. • We encounter evidence like those presented here every day. • We make inferences based on such data and sometimes even make recommendations. • We should try to think critically about data, whether it is research, our research, or just the news. Correlation & causation Every day we see in the news claims such the ones below. These could be reliable statistical relationships, but are they causal? 1. 2. 3. 4. 5. Better weather makes people happier Boys who mature later suffer from depression Runners outlive other athletes Kids who were breast feed have a higher IQ There is a negative correlation between # of hours watching TV and grades in school Examples: For each statement provide the causal story: 1. Women who exercise regularly have less natural miscarriages. 2. Famous conductors in the US live about 7 years longer than the average 3. Students who come to all the classes do better Correlation & causation • Famous conductors (C) in the US live about 7 years longer (L) than the average. C L C C E L E L C L Interpreting correlations • Sometime we have very strong causal models (good weather causes happiness, etc). • Sometime strong models can be wrong • Running prevents heart attacks (making people run can cause heart attacks). • Watching TV decreases learning So??? • The only way to draw valid conclusions is to have a control group! • In the Uri Geller case we should have tested how many clocks were observed to start working 1) on any other hour, 2) if someone else concentrated, etc. • In many cases, the only way to have a good control group is to conduct an experiment! Self-selection! • Runners select themselves • Famous conductors • Berkeley Men Women Apply Accepted Apply Accepted A 1,000 60% 100 60% B 1,000 30% 1,000 30% Both 2,000 900 1,100 360 45% 33% 5 year cholesterol experiment • P(death|no treatment) = 21% • P(death|treatment) = 21% • What about regularity? • P(death|regularity) = 15% • P(death|non regularity) = 24.6% • But now we have self selection again • For regulars P(D|T)= P(D|no T) ~ 15% • For non regulars P(D|T)= P(D|no T) ~ 26% Experiments • Are important but not a perfect method • They are susceptible to: • Placebo • The Hawthorne effect • Self fulfilling prophecies • Desire to cooperate Placebo I • In 1794 Gerbi discovered that by squeezing a certain worm between the thumb and the finger of the right hand, and placing the fingers on a painful tooth, the pain went away for a year. • Hundreds of patients were treated with this medication, and 68% reported no pain for a year! Placebo II • In a study on the effects of birth control, three groups were used • Pills with warning of side effects • Placebo without warning of side effects • Placebo with warning of side effects • Group #2 reported 6% side effects • Groups #1 & #3 reported 20% side effects Placebo III • Old French medical book suggest to use new medicines as fast, when its healing power is the strongest • Probably when the physician believes in it the most Placebo V • The Hawthorne effect • In 1924 a study was done on effectiveness at work • The recommendation was to improve lighting condition • This helped for a while but not long term • Other changes had the same effect …. Placebo -- summary • Placebo are real and strong effects • Threatens the conclusions of many experiments • Important to have good controls • Placebo is real: • Opioids • Sensitivity • Attention Self fulfilling prophecies • Teachers who are told that some kids are very smart, find that these kids get better grades • Researchers who believed that they have genetically inferior rats found them to have a lower performances • Smart Hans … Desire to cooperate • In many cases subjects want to help the experimenter. • If you tell subjects what is your hypothesis they will help you find it.. • This is why we prefer blind experiment • Even better are double blind experiments • Computer controlled experiments are another approach. Some answers Q1: Cholesterol & heart diseases • A large study in Framingham found that people with high cholesterol levels are much more likely to have heart attacks • It was also found that diets and medication can reduce cholesterol • Physicians recommendations for diets and medications reduced heart attacks only by a small number and did not reduce mortality! • What is going on? Not clear. Maybe the link is not causal Q2: Uri Geller • In one TV show Uri Geller told the viewers that he will activate broken clocks in their homes. • After he concentrated, many people called in to report that their clocks started working • Does this means that Uri Geller had such powers? Text No, he was a fraud. Q3: Reward and punishment • It has been well documented that reward good behavior is more effective than punishing bad behavior. • It was also observed that in flight school complementing excellent performance is usually followed by bad performance, while criticizing bad performance is usually followed by better performance. • What is going on? Regression to the mean Q4: The “hot hand” • Many people believe that athletes (such as basketball players) can be in a state called “hot hand.” • They are more likely to score after they have just scored • The hot hand can have many reasons • But, is it real? It is not real Q5: Discrimination? • UC Berkeley in the fall of 1973 • 8,442 men and 4,321 women applied • 44% of the men and 35% of the women were admitted • The quality of the applicants was the same • Is this discrimination? Not necessary -- “the Samson paradox” Q6: Stock predictions • One stock broker correctly predicted 7 of the 9 downturns in the economy over the past 3 years • Is he an expert? • Do you want to invest with this person? What about the other predictions? What if this person said “down”34 times? Q7: Medical diagnosis • Imagine that a blood test identified a person with a rare disease (base rate of 0.1% in the population). • This test identifies as a positive result 98% of the people who have the disease but also 4% of the healthy population. • Given this performance, how likely is it that the person who was identify as have the disease indeed has it? 0.0239% Q8: “The bible code” • It turn out that if you start from the last letter of the first word in the bible and count every 50th letter, you get the word “TURA” -- which means bible in Hebrew • There are many such examples of hidden meanings and this has been suggested as evidence that the bible was not written by men. • What do you think? “War and peace by Dostoyevsky shows the same effects Evidence…. • We encounter evidence like those presented here every day. • We make inferences based on such data and sometimes even make recommendations. • We should try to think critically about data, whether it is research, our research, or just the news. Summary • People are bad measurement devices • We observe and infer “rules” where there are non! Correlations / causation Self-selection Placebo / the Hawthorne effect Self fulfilling prophecies • Statistics and research methods are here to protect us against ourselves! Reading some more: • A Prayer Before Dying (story in Wired magazine) • In July 1995, back when AIDS was still a death sentence, psychiatrist Elisabeth Targ and her co-researchers enrolled 20 patients with advanced AIDS in a randomized, double-blind pilot study at the UC San Francisco Medical Center. All patients received standard care, but psychic healers prayed for the 10 in the treatment group. The healers lived an average of 1,500 miles away from the patients. None of the patients knew which group they had been randomly assigned to, and thus whether they were being prayed for. During the six-month study, four of the patients died - a typical mortality rate. When the data was unblinded, the researchers learned that the four who had died were in the control group. All 10 who were prayed for were still alive. Agenda for today Go over the homework problems that were due today Intuitive inference of data & information What makes a good project Example -- study from last week Some final questions 3 Conditions no lying lying lying & honor code + + Examples # of problem Solved 8 6.8 7 4 SS reported having solved all 20! 6 5 4 4.2 3.3 3 2 1 14 SS 22 SS 23 SS no lying lying lying & honor code 0 65 SS participated in total, 6 surveys were discarded What are good questions? Applied or theory? Applied meaning you are trying to estimate particular parameters Examples: Politeness, opening doors, etc. Theory meaning you are trying to understand a directional effect Examples: Politeness, opening doors, etc. What are good questions? Applied or theory? Applied is more specific, theory is more general Engineering vs. Science “There is nothing more practical than a good theory” (Kurt Levin) At what stage of product development is each more appropriate? When is represented sampling more important? When is exact implementation more important? What are good questions? Relevant is a good criteria But relevant to who? “Big” is a good criteria More general (theoretical) questions are likely to be “larger” & more relevant Ethnics in research There have been a lot of cases of abuse Medical cases where they gave people different diseases. Shaking basic beliefs Philip Zimbardo’s jail experiment Stanley Milgram’s Electrical shocking experiment The Human subject review board COUHES is the local branch that is in charge of ethical behavior toward subjects Being wrong It is fine (good?) to be wrong The important think is knowledge If you are wrong, try to figure out why. Hypothesis or method? It is good to be wrong try to venture out, take risks Exercise in generating ideas In small groups propose questions you are interested in finding the answers for. Practice “armchair” psychology Think of your own weird behavior (and on that of the people around you). Observe, introspect Also name 3 “bad” (useless) thins to test Agenda for today Go over the homework problems that were due today Intuitive inference of data & information What makes a good project Example -- study from last week Some final questions