Humans are remarkably good at self-deception, and growing concern about reproducibility is driving many researchers to seek ways to fight their own instincts. Scientists are susceptible to the same cognitive fallacies as everyone else – so how can they mitigate the risk?
The journal Nature sheds light on four common fallacies that are found in scientific research, and several techniques to “de-bias” results. Some of these go by different names, but they are basically sharing the same set of ideas:
- Hypothesis Myopia: Collecting evidence to support a hypothesis, not looking for evidence against it, and ignoring other explanations.
- Texas Sharpshooter: Seizing on random patterns in the data and mistaking them for interesting findings.
- Asymmetric Attention: Rigorously checking unexpected results, but giving expected results a free pass.
- Just-So Storytelling: Finding stories after the fact to rationalize whatever the results turn out to be.
Scientists can avoid these fallacies by applying the following “debiasing” techniques:
- Devil’s Advocacy: Explicitly consider alternative hypotheses – then test them out head-to-head.
- Pre-Commitment: Publicly declare a data collection and analysis plan before starting the study.
- Team of Rivals: Invite your academic adversaries to collaborate with you on a study.
- Blind Data Analysis: Analyse data that look real but are not exactly what you collected – and then lift the blind.