Like two dark clouds in the history physics, there is a dark cloud over the field of psychology, and it's huge.
Repeat 100 studies published in top journals of psychology and only 36% of the results were repeated. Last August, a group of scientists named "Open Scientific Cooperation" (COS) published the results in the journal Science.
So what does the repeatable crisis of psychological experimentation mean? Psychology is not science? A big psychological crisis?
Or is there a crisis in the research methods of psychology?Like the experimental method.
This is a complex question. First, the studies in question are not randomly selected, so it is hard to establish the prevalence of replicable findings. Second, what level of replicability would you expect? We are considering effects estimated in highly variable contexts. E.g., if you took a 95% CI for the original effect you'd expect the recapture rate to be around 80% (around 80% of new effect sizes would be in the old CI). So if you expect 100% replication then you have misunderstood the original strength of evidence those studies showed. Third, replicability varies between replication studies (e.g., some show levels around 50-60% for certain areas of research).
So it seems likely that replication is lower than we'd expect and that we also ver-interpret the original studies in terms of the strength of evidence they provide for an effect.
Last, this isn't a crisis of psychology per se - it seems to be arise in any discipline that deals with uncertain phenomena and that uses a particular set of methods (particularly valuing surprising, novel findings, using statistical significance as a threshold for publication etc.).
For me, this crisis (there have been others, but this one has reached more of a tipping point and as Thom Baguley says, is pretty general ... note he does not limit it to those using stats! That's important!) shows two things:
1. The pressures to publish and the publishing mechanisms are not set up solely to make sure every paper is accurate. Yes, good peer review helps, and there are improvements (and in science some refinements underway way be helpful), but the individual reward structure particularly for news coverage, click bait, citations, etc., mean Gelman will continue to have content for those papers in his blog (https://andrewgelman.com/ is a good read and he has lots of views on this). It is worth stressing that no system should ensure all papers are reproducible/replicable/right/(or whatever) as this would mean nothing is published. We can argue that there is too much published now, and that much peer review is not a good first hurdle, but it seems like publishers are going to keep creating new journals if they (and this often academic societies) think that they can make more money.
2. People should not treat a single paper or even a few papers as the end of a scientific story. I think this is true more in areas like many in psychology where we can create lots of data sets that inform a theory from lots of different perspectives (this is a bit tougher for say the "discovery" of the Higgs Boson, but that is why they have 5 sigma for "discoveries" (it is also there for what physicists call the "look elsewhere effect" and what Gelman and Loken call the garden of forking paths based on the Borges story), but you can published research with less. This means the wider research community can critique it, replicate it in varied ways (or exactly), and let the research literature grow without hiding findings from all but a few viewers. The problem is when people read a paper without being skeptical. It is tough to show skepticism when writing headlines for the World Weekly News (which their website says is the world's [I think this one] only reliable newspaper, so is a good example to teach students the difference between reliability and validity).
Thom Baguley
Like the uncertainty of quantum mechanics?
Psychological phenomena are not uncertain, maybe the "repetition" of your experiment has always been wrong? that is, you should not repeat some "effect", "psychological phenomenon", but should repeat a "psychological process", because, "psychological phenomenon" is the manifestation of "Psychological process".
In this case I'm referring to the uncertainty inherent in measuring psychological phenomena. This includes measurement error, but also heterogeneity in sampling from individuals that are inherently variable and other sources of heterogeneity in stimuli, contexts and so on. Effect is just the term we used for a quantified finding whether the outcome of a process or not. It is worth noting that there are many psychology studies that do replicate and people are pretty good at predicting which ones do replicate: Article Participant Nonnaiveté and the reproducibility of cognitive psychology
http://www.pnas.org/content/112/50/15343
Thom Baguley
Thank you for sharing the article. I have an article,too.
- Badges
- Science topic
- Similar topics
- Psychology
- Mental Processes
- Learning