pubmed.ncbi.nlm.nih.gov

Evaluation of the potential excess of statistically significant findings in published genetic association studies: application to Alzheimer's disease - PubMed

  • ️Tue Jan 01 2008

Comparative Study

. 2008 Oct 15;168(8):855-65.

doi: 10.1093/aje/kwn206. Epub 2008 Sep 8.

Affiliations

Comparative Study

Evaluation of the potential excess of statistically significant findings in published genetic association studies: application to Alzheimer's disease

Fotini K Kavvoura et al. Am J Epidemiol. 2008.

Abstract

The authors evaluated whether there is an excess of statistically significant results in studies of genetic associations with Alzheimer's disease reflecting either between-study heterogeneity or bias. Among published articles on genetic associations entered into the comprehensive AlzGene database (www.alzgene.org) through January 31, 2007, 1,348 studies included in 175 meta-analyses with 3 or more studies each were analyzed. The number of observed studies (O) with statistically significant results (P = 0.05 threshold) was compared with the expected number (E) under different assumptions for the magnitude of the effect size. In the main analysis, the plausible effect size of each association was the summary effect presented in the respective meta-analysis. Overall, 19 meta-analyses (all with eventually nonsignificant summary effects) had a documented excess of O over E: Typically single studies had significant effects pointing in opposite directions and early summary effects were dissipated over time. Across the whole domain, O was 235 (17.4%), while E was 164.8 (12.2%) (P < 10(-6)). The excess showed a predilection for meta-analyses with nonsignificant summary effects and between-study heterogeneity. The excess was seen for all levels of statistical significance and also for studies with borderline P values (P = 0.05-0.10). The excess of significant findings may represent significance-chasing biases in a setting of massive testing.

PubMed Disclaimer

Figures

Figure 1.
Figure 1.

Representative forest plots of meta-analyses for typical patterns of included studies. A) Statistically significant summary effect and all significant studies point in the same direction as the overall estimate (cholesterol 25-hydroxylase (CH25H) rs13500); B) non-statistically significant summary estimate with one or several significant studies pointing in the same direction (amyloid beta (A4) precursor protein-binding, family B, member 1 (APBB1) rs1799755); C) non-statistically significant summary estimate and significant studies pointing in both directions (interleukin-10 (IL10) rs1800871 (−819)). In each meta-analysis (parts A–C), studies are ordered by year of publication. CI, confidence interval. (The articles used as sources for these meta-analyses are listed in a supplement posted on the Journal’s website (

http://aje.oxfordjournals.org/

).)

Figure 2.
Figure 2.

Cumulative effect estimates (odds ratios) per year after publication of the first study for meta-analyses that showed a statistically significant (P < 0.10) excess of observed significant studies compared with the expected ones. The first year of publication is shown at time point 0. Each circle corresponds to the point estimate of the cumulative meta-analysis in a given year. The point estimate is updated as new studies are published.

Figure 3.
Figure 3.

Scatterplot of the heterogeneity I2 metric and O:E ratio (number of observed studies (O) with statistically significant results (P = 0.05 threshold) vs. the number expected (E)) for the 175 meta-analyses included in the analysis. The filled circles stand for meta-analyses with a nonsignificant excess of observed versus expected significant meta-analyses, and the open diamonds stand for meta-analyses with a significant excess of observed versus expected significant meta-analyses.

Figure 4.
Figure 4.

Ratio of observed (O) to expected (E) numbers of studies with results in a specific range of P values. A) All meta-analyses; B) meta-analyses with significant results; C) meta-analyses with nonsignificant results. Dotted lines correspond to 1, where O and E are equal.

Figure 5.
Figure 5.

Ratio of observed (O) to expected (E) numbers of studies with results in a specific range of P values. A) Meta-analyses with I2 ≤ 50% and nonsignificant results; B) meta-analyses with I2 > 50% and nonsignificant results. Dotted lines correspond to 1, where O and E are equal.

Similar articles

Cited by

References

    1. Ioannidis JP, Gwinn M, Little J, et al. A road map for efficient and reliable human genome epidemiology. Nat Genet. 2006;38(1):3–5. - PubMed
    1. Hirschhorn JN, Lohmueller K, Byrne E, et al. A comprehensive review of genetic association studies. Genet Med. 2002;4(2):45–61. - PubMed
    1. Ioannidis JP. Non-replication and inconsistency in the genome-wide association setting. Hum Hered. 2007;64(4):203–213. - PubMed
    1. Patterson M, Cardon L. Replication publication. PLoS Biol. 2005;3(9):e327. - PMC - PubMed
    1. Chanock SJ, Manolio T, Boehnke M, et al. Replicating genotype-phenotype associations. Nature. 2007;447(7145):655–660. - PubMed

Publication types

MeSH terms