Perspective: NutriGrade: A Scoring System to Assess and Judge the Meta-Evidence of Randomized Controlled Trials and Cohort Studies in Nutrition Research

Lukas Schwingshackl; Sven Knüppel; Carolina Schwedhelm; Georg Hoffmann; Benjamin Missbach; Marta Stelmach-Mardas; Stefan Dietrich; Fabian Eichelmann; Evangelos Kontopantelis; Khalid Iqbal; Krasimira Aleksandrova; Stefan Lorkowski; Michael F Leitzmann; Anja Kroke; Heiner Boeing

doi:10.3945/an.116.013052

Perspective: NutriGrade: A Scoring System to Assess and Judge the Meta-Evidence of Randomized Controlled Trials and Cohort Studies in Nutrition Research

Adv Nutr. 2016 Nov 15;7(6):994-1004. doi: 10.3945/an.116.013052. Print 2016 Nov.

Authors

Lukas Schwingshackl¹, Sven Knüppel², Carolina Schwedhelm², Georg Hoffmann³, Benjamin Missbach³, Marta Stelmach-Mardas^{2

4}, Stefan Dietrich², Fabian Eichelmann^{5

2}, Evangelos Kontopantelis⁶, Khalid Iqbal², Krasimira Aleksandrova^{5

2}, Stefan Lorkowski^{7

8}, Michael F Leitzmann⁹, Anja Kroke¹⁰, Heiner Boeing²

Affiliations

¹ Department of Epidemiology, German Institute of Human Nutrition Potsdam Rehbruecke, Nuthetal, Germany; lukas.schwingshackl@dife.de.
² Department of Epidemiology, German Institute of Human Nutrition Potsdam Rehbruecke, Nuthetal, Germany.
³ Department of Nutritional Sciences, University of Vienna, Vienna, Austria.
⁴ Department of Pediatric Gastroenterology and Metabolic Diseases, Poznan University of Medical Sciences, Poznan, Poland.
⁵ Nutrition, Immunity, and Metabolism Start-Up Lab.
⁶ Centre for Primary Care, Institute of Population Health, University of Manchester, Manchester, United Kingdom.
⁷ Institute of Nutrition, Friedrich Schiller University Jena, Jena, Germany.
⁸ Competence Cluster of Nutrition and Cardiovascular Health, Halle-Jena-Leipzig, Germany.
⁹ Department of Epidemiology and Preventive Medicine, University of Regensburg, Regensburg, Germany; and.
¹⁰ Department of Nutritional, Food, and Consumer Sciences, University of Applied Sciences, Fulda, Germany.

Abstract

The objective of this study was to develop a scoring system (NutriGrade) to evaluate the quality of evidence of randomized controlled trial (RCT) and cohort study meta-analyses in nutrition research, building upon previous tools and expert recommendations. NutriGrade aims to assess the meta-evidence of an association or effect between different nutrition factors and outcomes, taking into account nutrition research-specific requirements not considered by other tools. In a pretest study, 6 randomly selected meta-analyses investigating diet-disease relations were evaluated with NutriGrade by 5 independent raters. After revision, NutriGrade was applied by the same raters to 30 randomly selected meta-analyses in the same thematic area. The reliability of ratings of NutriGrade items was calculated with the use of a multirater κ, and reliability of the total (summed scores) was calculated with the use of intraclass correlation coefficients (ICCs). The following categories for meta-evidence evaluation were established: high (8-10), moderate (6-7.99), low (4-5.99), and very low (0-3.99). The NutriGrade scoring system (maximum of 10 points) comprises the following items: 1) risk of bias, study quality, and study limitations, 2) precision, 3) heterogeneity, 4) directness, 5) publication bias, 6) funding bias, 7) study design, 8) effect size, and 9) dose-response. The NutriGrade score varied between 2.9 (very low meta-evidence) and 8.8 (high meta-evidence) for meta-analyses of RCTs, and it ranged between 3.1 and 8.8 for meta-analyses of cohort studies. The κ value of the ratings for each scoring item varied from 0.32 (95% CI: 0.22, 0.42) for risk of bias for cohort studies and 0.95 (95% CI: 0.91, 0.99) for study design, with a mean κ of 0.66 (95% CI: 0.53, 0.79). The ICC of the total score was 0.81 (95% CI: 0.69, 0.90). The NutriGrade scoring system showed good agreement and reliability. The initial findings regarding the performance of this newly established scoring system need further evaluation in independent analyses.

Keywords: meta-analysis; meta-evidence; nutrition; reliability; scoring.

MeSH terms

Biomedical Research / standards*
Cohort Studies
Evaluation Studies as Topic*
Humans
Meta-Analysis as Topic
Nutritional Sciences / standards*
Randomized Controlled Trials as Topic / standards*
Reproducibility of Results
Research Design