pubmed.ncbi.nlm.nih.gov

On summary ROC curve for dichotomous diagnostic studies: an application to meta-analysis of COVID-19 - PubMed

  • ️Sat Jan 01 2022

On summary ROC curve for dichotomous diagnostic studies: an application to meta-analysis of COVID-19

ShengLi Tzeng et al. J Appl Stat. 2022.

Abstract

In a systematic review of a diagnostic performance, summarizing performance metrics is crucial. There are various summary models in the literature, and hence model selection becomes inevitable. However, most existing large-sample-based model selection approaches may not fit in a meta-analysis of diagnostic studies, typically having a rather small sample size. Researchers need to effectively determine the final model for further inference, which motivates this article to investigate existing methods and to suggest a more robust method for this need. We considered models covering several widely-used methods for bivariate summary of sensitivity and specificity. Simulation studies were conducted based on different number of studies and different population sensitivity and specificity. Then final models were selected using several existing criteria, and we compared the summary receiver operating characteristic (sROC) curves to the theoretical ROC curve given the generating model. Even though parametric likelihood-based criteria are often applied in practice for their asymptotic property, they fail to consistently choose appropriate models under the limited number of studies. When the number of studies is as small as 10 or 5, our suggestion is best in different scenarios. An example for summary ROC curves for chemiluminescence immunoassay (CLIA) used in COVID-19 diagnosis is also illustrated.

Keywords: Summary ROC curve; meta analysis; random effects; sensitivity; specificity; systematic review.

© 2022 Informa UK Limited, trading as Taylor & Francis Group.

PubMed Disclaimer

Conflict of interest statement

No potential conflict of interest was reported by the author(s).

Figures

Figure 1.
Figure 1.

MIAE of different candidate models or selected models for Scenario 1 under first (left panel) and second (right panel) families. (a) and (b): N = 5; (c) and (d): N = 10; (e) and (f): N = 50. Different colors stand for different α: 0 in cyan, 0.6 in black, 1 in red, 1.4 in green, and 2 in gray. The numbers in the parentheses on the horizontal axis represent (

αp

,

αq

, family index).

Figure 2.
Figure 2.

MIAE of different candidate models or selected models for Scenario 2 under first (left panel) and second (right panel) families. (a) and (b): N = 5; (c) and (d): N = 10; (e) and (f): N = 50. Different colors stand for different α: 0 in cyan, 0.6 in black, 1 in red, 1.4 in green, and 2 in gray. The numbers in the parentheses on the horizontal axis represent (

αp

,

αq

, family index).

Figure 3.
Figure 3.

Estimated sROC curves with the confidence region (red line) and prediction region (dashed line) for CLIA(IgG) using (a) the model selected by AIC, (b) MSL method, (c) HSROC model, and (d) the model with

αp=αq=1.4

, and index

=1

, where gray circles (with the area proportional to the sample size) are the sensitivity and 1-specificity observed from individual studies and the black triangle indicates the summary sensitivity and 1-specificity.

Figure 4.
Figure 4.

Estimated sROC curves with the confidence region (red line) and prediction region (dashed line) for CLIA(IgM) using (a) the model selected by AIC, (b) MSL method, (c) HSROC model, and (d) the model with

αp=αq=1.4

, and index

=1

, where gray circles (with the area proportional to the sample size) are the sensitivity and 1-specificity observed from individual studies and the black triangle indicates the summary sensitivity and 1-specificity.

Similar articles

Cited by

References

    1. Akaike H., Information theory and an extension of the maximum likelihood principle, Proceedings of 2nd International Symposium on Information Theory, Akadémiai Kiadó, 1973, pp. 267–281.
    1. Bachmann L.M., Puhan M.A., Ter Riet G., and Bossuyt P.M., Sample sizes of studies on diagnostic accuracy: Literature survey, BMJ 332 (2006), pp. 1127–1129. - PMC - PubMed
    1. Baggerly K.A., Empirical likelihood as a goodness-of-fit measure, Biometrika 85 (1998), pp. 535–547.
    1. Bastos M.L., Tavaziva G., Abidi S.K., Campbell J.R., Haraoui L.P., Johnston J.C., Lan Z., Law S., MacLean E., Trajman A., Menzies D., Benedetti A., and Ahmad Khan F., Diagnostic accuracy of serological tests for covid-19: Systematic review and meta-analysis, BMJ 370 (2020), pp. m2516. - PMC - PubMed
    1. Burnham K.P. and Anderson D.R., Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach, Springer Science & Business Media, New York, 2003.

Grants and funding

Financial support for this study was provided in part by a grant from National Science Council, Taiwan [grant number NSC102-2118-M038-002], and Ministry of Science and Technology, Taiwan [grant numbers MOST104-2314-B039-037, MOST107-2118-M-110-004-MY3 and MOST108-2628-M-008-005-MY3]. The funding agreement ensured the authors' independence in designing the study, interpreting the results, writing, and publishing the report.

LinkOut - more resources