The normality assumption on between-study random effects was questionable in a considerable number of Cochrane meta-analyses

Ziyu Liu; Fahad M Al Amer; Mengli Xiao; Chang Xu; Luis Furuya-Kanamori; Hwanhee Hong; Lianne Siegel; Lifeng Lin

doi:10.1186/s12916-023-02823-9

The normality assumption on between-study random effects was questionable in a considerable number of Cochrane meta-analyses

BMC Med. 2023 Mar 29;21(1):112. doi: 10.1186/s12916-023-02823-9.

Authors

Ziyu Liu¹, Fahad M Al Amer², Mengli Xiao³, Chang Xu^{4

5}, Luis Furuya-Kanamori⁶, Hwanhee Hong⁷, Lianne Siegel⁸, Lifeng Lin⁹

Affiliations

¹ Department of Statistics, Florida State University, Tallahassee, FL, USA.
² Department of Mathematics, College of Science and Arts, Najran University, Najran, Saudi Arabia.
³ Department of Biostatistics and Informatics, University of Colorado Anschutz Medical Campus, Aurora, CO, USA.
⁴ Ministry of Education Key Laboratory for Population Health Across-Life Cycle & Anhui Provincial Key Laboratory of Population Health and Aristogenics, Anhui Medical University, Anhui, China.
⁵ School of Public Health, Anhui Medical University, Anhui, China.
⁶ UQ Centre for Clinical Research, Faculty of Medicine, University of Queensland, Herston, Australia.
⁷ Department of Biostatistics and Bioinformatics, School of Medicine, Duke University, Durham, NC, USA.
⁸ Division of Biostatistics, University of Minnesota School of Public Health, Minneapolis, MN, USA.
⁹ Department of Epidemiology and Biostatistics, Mel and Enid Zuckerman College of Public Health, University of Arizona, Tucson, AZ, USA. lifenglin@arizona.edu.

Abstract

Background: Studies included in a meta-analysis are often heterogeneous. The traditional random-effects models assume their true effects to follow a normal distribution, while it is unclear if this critical assumption is practical. Violations of this between-study normality assumption could lead to problematic meta-analytical conclusions. We aimed to empirically examine if this assumption is valid in published meta-analyses.

Methods: In this cross-sectional study, we collected meta-analyses available in the Cochrane Library with at least 10 studies and with between-study variance estimates > 0. For each extracted meta-analysis, we performed the Shapiro-Wilk (SW) test to quantitatively assess the between-study normality assumption. For binary outcomes, we assessed between-study normality for odds ratios (ORs), relative risks (RRs), and risk differences (RDs). Subgroup analyses based on sample sizes and event rates were used to rule out the potential confounders. In addition, we obtained the quantile-quantile (Q-Q) plot of study-specific standardized residuals for visually assessing between-study normality.

Results: Based on 4234 eligible meta-analyses with binary outcomes and 3433 with non-binary outcomes, the proportion of meta-analyses that had statistically significant non-normality varied from 15.1 to 26.2%. RDs and non-binary outcomes led to more frequent non-normality issues than ORs and RRs. For binary outcomes, the between-study non-normality was more frequently found in meta-analyses with larger sample sizes and event rates away from 0 and 100%. The agreements of assessing the normality between two independent researchers based on Q-Q plots were fair or moderate.

Conclusions: The between-study normality assumption is commonly violated in Cochrane meta-analyses. This assumption should be routinely assessed when performing a meta-analysis. When it may not hold, alternative meta-analysis methods that do not make this assumption should be considered.

Keywords: Cochrane Library; Effect measure; Heterogeneity; Meta-analysis; Normality assumption; Q–Q plot.

The normality assumption on between-study random effects was questionable in a considerable number of Cochrane meta-analyses

Authors

Affiliations

Abstract

Publication types

MeSH terms

Grants and funding