pubmed.ncbi.nlm.nih.gov

Testing for Questionable Research Practices in a Meta-Analysis: An Example from Experimental Parapsychology - PubMed

  • ️Fri Jan 01 2016

Meta-Analysis

Testing for Questionable Research Practices in a Meta-Analysis: An Example from Experimental Parapsychology

Dick J Bierman et al. PLoS One. 2016.

Abstract

We describe a method of quantifying the effect of Questionable Research Practices (QRPs) on the results of meta-analyses. As an example we simulated a meta-analysis of a controversial telepathy protocol to assess the extent to which these experimental results could be explained by QRPs. Our simulations used the same numbers of studies and trials as the original meta-analysis and the frequencies with which various QRPs were applied in the simulated experiments were based on surveys of experimental psychologists. Results of both the meta-analysis and simulations were characterized by 4 metrics, two describing the trial and mean experiment hit rates (HR) of around 31%, where 25% is expected by chance, one the correlation between sample-size and hit-rate, and one the complete P-value distribution of the database. A genetic algorithm optimized the parameters describing the QRPs, and the fitness of the simulated meta-analysis was defined as the sum of the squares of Z-scores for the 4 metrics. Assuming no anomalous effect a good fit to the empirical meta-analysis was found only by using QRPs with unrealistic parameter-values. Restricting the parameter space to ranges observed in studies of QRP occurrence, under the untested assumption that parapsychologists use comparable QRPs, the fit to the published Ganzfeld meta-analysis with no anomalous effect was poor. We allowed for a real anomalous effect, be it unidentified QRPs or a paranormal effect, where the HR ranged from 25% (chance) to 31%. With an anomalous HR of 27% the fitness became F = 1.8 (p = 0.47 where F = 0 is a perfect fit). We conclude that the very significant probability cited by the Ganzfeld meta-analysis is likely inflated by QRPs, though results are still significant (p = 0.003) with QRPs. Our study demonstrates that quantitative simulations of QRPs can assess their impact. Since meta-analyses in general might be polluted by QRPs, this method has wide applicability outside the domain of experimental parapsychology.

PubMed Disclaimer

Conflict of interest statement

Competing Interests: The authors have declared that no competing interests exist.

Figures

Fig 1
Fig 1. Distribution of Sample Sizes (= number of samples) in the database.
Fig 2
Fig 2. Funnel plot of hit rates in the database.
Fig 3
Fig 3. Distribution of binomial P-values.

Upper pane: the experimental GF database. Lower pane: Simulated Null-distribution.

Fig 4
Fig 4. The fitting value as a function of generation of the Genetic Algorithm.

The QRP-parameters are kept within a reasonable interval. Circular points show the mean fitness for each generation, with error bars of 1 SE, diamond points show the best fitness per generation.

Fig 5
Fig 5. Fitness values when using reasonable QRP parameters and allowing for a small excess hit rate.

The right Y-axis indicates the probability that simulations and experimental data are the same.

Similar articles

Cited by

References

    1. John LK, Loewenstein G, Prelec D. Measuring the prevalence of questionable research practices with incentives for truth telling. Psychol Sci 2012. May 1;23(5):524–532. 10.1177/0956797611430953 - DOI - PubMed
    1. Ioannidis JP. Why most published research findings are false. PLoS medicine 2005;2(8):e124 - PMC - PubMed
    1. Simmons JP, Nelson LD, Simonsohn U. False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol Sci 2011. November;22(11):1359–1366. 10.1177/0956797611417632 - DOI - PubMed
    1. Open Science Collaboration. Estimating the reproducibility of Psychological Science. Science 2015;349(6251). - PubMed
    1. Price H. The Cold Fusion Horizon. Is cold fusion truly impossible or is it just that no respectable scientist can risk their reputastion working on it. Aeon essays 2015 21 dec 2015.

Publication types

MeSH terms

Grants and funding

The authors have no support or funding to report.