Class 2 Notes - The Replication Crisis Importance

The Replication Crisis Introduction 2011, Daryl Bem publishes a research paper Paper uses standard psych research methods Finds that participants are affected by things that have not yet happened Defies basic physics Psychologists began asking questions: Why was Bem able to use standard psychological research methods to show something impossible? Why did a reputable journal publish it? What does this mean for the past and future of psychology? The Sin of Bias Scientists are prone to confirmation bias, as any other person Find evidence to support research hypotheses p < .05 means reject the null hypothesis p > .05 does not mean we accept the null E.g., could be not enough participants An hypothesis exists because we think it's true and interesting Journals are also biased towards publishing "exciting" results Null results are not "exciting" and are much harder to publish Example: A psychologist runs a study 5 times and gets these p-values: p = .013, p = .048, p = .071, p = .036, p = .093. Which one gets published? How to Address the Problem? Option 1 - Preregistration Tell the world exactly what you plan to do before you do it Registered Reports even better Option 2 - Publish null results Example: Bruce is comparing two therapies for anxiety He runs an independent samples t-test: t(31) = -1.99, p = .055, two-tailed test Appears strange - plot them Outliers - check their Z-scores Re-analyze: t(36) = -2.44, p = .020, two-tailed test The Sin of Hidden Flexibility Range of ways of analyzing every data set Want to find significant results Rarely report all analyses they tried How to Address the Problem? Option 1 - Preregistration Tell the world exactly what you plan to do before you do it The Sin of Unreliability Studies produce unreliable results 1 reason is poor replication Example: Psychologist publishes a study that finds that British 15-yr olds (n=30) are faster at word puzzles than 30-yr olds (n=30). What happens next? No direct replication Mohan runs a conceptual replication in India Western vs. Non-Western ideas/Different puzzles Two possible outcomes: Same result (younger adults sig. faster) - This is further evidence that teens are faster at word puzzles Different Result (No sig. age diff.) - The effect shown in the previous study must have been specific to the language/puzzle used Too few participants - making claims about the human race based on behavior of 30 participants Lack of reporting of methods - method sections that do not allow for full replication Reward researchers for running direct replication Power analysis - make sure study has enough power before you run it The Sin of Data Hoarding Not standard in Psych - to make data publicly available Final paper is often the only public record of what happened How to address the problem? Easy: make data available Internet makes this very simple The Sin of Corruptibility Science is competitive Exciting results: Money Promotion Prestige Financial stability Extreme scientific fraud is rare Trimming data to produce good findings is common How to address: De-incentivize excluding results Preregistration/registered reports Open data The Sin of Interment Many articles are behind expensive paywalls Journal publishers: Do not pay researchers for their work The Sin of Bean Counting Scientists' publication record directly linked to career success "Good" journals: High impact factor Exciting, novel results "Less good" journals: Lower impact factor Less exciting results Encourages bias, flexibility, corruption How do we address this problem? Start valuing quality over quantity Quality = scientific rigor, not exciting results The Sin of Interment Do not pay scientists to peer review others' work Charge universities massive fees for access to articles Prevent the public from accessing science (often paid for by public funding) Makes access difficult for people at small universities or from poorer countries How do we address this problem? Open access publishing