Concordance of randomised controlled trials for artificial intelligence interventions with the CONSORT-AI reporting guidelines

Alexander P L Martindale; Benjamin Ng; Victoria Ngai; Aditya U Kale; Lavinia Ferrante di Ruffano; Robert M Golub; Gary S Collins; David Moher; Melissa D McCradden; Lauren Oakden-Rayner; Samantha Cruz Rivera; Melanie Calvert; Christopher J Kelly; Cecilia S Lee; Christopher Yau; An-Wen Chan; Pearse A Keane; Andrew L Beam; Alastair K Denniston; Xiaoxuan Liu

doi:10.1038/s41467-024-45355-3

Concordance of randomised controlled trials for artificial intelligence interventions with the CONSORT-AI reporting guidelines

Nat Commun. 2024 Feb 22;15(1):1619. doi: 10.1038/s41467-024-45355-3.

Authors

Alexander P L Martindale¹, Benjamin Ng^{2

3}, Victoria Ngai⁴, Aditya U Kale^{5

6

7}, Lavinia Ferrante di Ruffano⁸, Robert M Golub⁹, Gary S Collins¹⁰, David Moher¹¹, Melissa D McCradden^{12

13

14}, Lauren Oakden-Rayner¹⁵, Samantha Cruz Rivera^{16

17}, Melanie Calvert^{7

16

17

18

19}, Christopher J Kelly²⁰, Cecilia S Lee²¹, Christopher Yau^{22

23}, An-Wen Chan²⁴, Pearse A Keane²⁵, Andrew L Beam^{26

27}, Alastair K Denniston^{5

6

7

16

25}, Xiaoxuan Liu^{28

29

30}

Affiliations

¹ Brighton and Sussex Medical School, Brighton, UK.
² Birmingham and Midland Eye Centre, Sandwell and West Birmingham NHS Trust, Birmingham, UK.
³ Christ Church, University of Oxford, Oxford, UK.
⁴ University College London Medical School, London, UK.
⁵ Institute of Inflammation and Ageing, University of Birmingham, Birmingham, UK.
⁶ University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK.
⁷ National Institute for Health and Care Research (NIHR) Birmingham Biomedical Research Centre, University of Birmingham, Birmingham, UK.
⁸ York Health Economics Consortium, University of York, York, UK.
⁹ Northwestern University Feinberg School of Medicine, Chicago, IL, USA.
¹⁰ Centre for Statistics in Medicine//UK EQUATOR Centre, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK.
¹¹ Centre for Journalology, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottowa, ON, Canada.
¹² Department of Bioethics, The Hospital for Sick Children, Toronto, ON, Canada.
¹³ Genetics & Genome Biology Research Program, Peter Gilgan Centre for Research & Learning, Toronto, ON, Canada.
¹⁴ Division of Clinical and Public Health, Dalla Lana School of Public Health, Toronto, ON, Canada.
¹⁵ Australian Institute for Machine Learning, University of Adelaide, Adelaide, SA, Australia.
¹⁶ Birmingham Health Partners Centre for Regulatory Science and Innovation, University of Birmingham, Birmingham, UK.
¹⁷ Centre for Patient Reported Outcomes Research (CPROR), Institute of Applied Health Research, College of Medical and Dental Sciences, University of Birmingham, Birmingham, UK.
¹⁸ NIHR Applied Research Collaboration (ARC) West Midlands, University of Birmingham, Birmingham, UK.
¹⁹ NIHR Blood and Transplant Research Unit (BTRU) in Precision Transplant and Cellular Therapeutics, University of Birmingham, Birmingham, UK.
²⁰ Google Health, London, UK.
²¹ University of Washington, Seattle, WA, USA.
²² Nuffield Department of Women's and Reproductive Health, University of Oxford, Oxford, UK.
²³ Health Data Research UK, London, UK.
²⁴ Department of Medicine, Women's College Hospital. University of Toronto, Toronto, ON, Canada.
²⁵ NIHR Biomedical Research Centre at Moorfields, Moorfields Eye Hospital NHS Foundation Trust and UCL Institute of Ophthalmology, London, UK.
²⁶ Department of Epidemiology, Harvard. T.H. Chan School of Public Health, Boston, MA, USA.
²⁷ Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA.
²⁸ Institute of Inflammation and Ageing, University of Birmingham, Birmingham, UK. xliuphone@gmail.com.
²⁹ University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK. xliuphone@gmail.com.
³⁰ Birmingham Health Partners Centre for Regulatory Science and Innovation, University of Birmingham, Birmingham, UK. xliuphone@gmail.com.

Abstract

The Consolidated Standards of Reporting Trials extension for Artificial Intelligence interventions (CONSORT-AI) was published in September 2020. Since its publication, several randomised controlled trials (RCTs) of AI interventions have been published but their completeness and transparency of reporting is unknown. This systematic review assesses the completeness of reporting of AI RCTs following publication of CONSORT-AI and provides a comprehensive summary of RCTs published in recent years. 65 RCTs were identified, mostly conducted in China (37%) and USA (18%). Median concordance with CONSORT-AI reporting was 90% (IQR 77-94%), although only 10 RCTs explicitly reported its use. Several items were consistently under-reported, including algorithm version, accessibility of the AI intervention or code, and references to a study protocol. Only 3 of 52 included journals explicitly endorsed or mandated CONSORT-AI. Despite a generally high concordance amongst recent AI RCTs, some AI-specific considerations remain systematically poorly reported. Further encouragement of CONSORT-AI adoption by journals and funders may enable more complete adoption of the full CONSORT-AI guidelines.

Publication types

Systematic Review

MeSH terms

Artificial Intelligence*
China
Randomized Controlled Trials as Topic
Reference Standards

Grants and funding

MC_PC_19005/MRC_/Medical Research Council/United Kingdom