Validation of the Learner Engagement Instrument for Continuing Professional Development

David A Cook; Christopher R Stephenson

doi:10.1097/ACM.0000000000005749

Validation of the Learner Engagement Instrument for Continuing Professional Development

Acad Med. 2024 Apr 29. doi: 10.1097/ACM.0000000000005749. Online ahead of print.

Authors

David A Cook¹, Christopher R Stephenson²

Affiliations

¹ D.A. Cook is Professor of Medicine and Medical Education; Director of Education Science, Office of Applied Scholarship and Education Science, Mayo Clinic College of Medicine and Science; and Consultant, Division of General Internal Medicine, Mayo Clinic, Rochester, MN; ORCID ID: https://orcid.org/0000-0003-2383-4633.
² C.R. Stephenson is Assistant Professor of Medicine and Consultant, Division of General Internal Medicine, Mayo Clinic College of Medicine and Science, Rochester, MN; ORCID ID: https://orcid.org/0000-0001-8537-392X.

PMID: 38683885
DOI: 10.1097/ACM.0000000000005749

Abstract

Purpose: Learner engagement is the energy learners exert to remain focused and motivated to learn. The Learner Engagement Instrument (LEI) was developed to measure learner engagement in a short continuing professional development (CPD) activity. The authors validated LEI scores using validity evidence of internal structure and relationships with other variables.

Method: Participants attended 1 of 4 CPD courses (1 in-person, 2 online livestreamed, and 1 either in-person or online) in 2018, 2020, 2021, and 2022. Confirmatory factor analysis was used to examine model fit for several alternative structural models, separately for each course. The authors also conducted a generalizability study to estimate score reliability. Associations were evaluated between LEI scores and Continuing Medical Education Teaching Effectiveness (CMETE) scores and participant demographics. All statistical methods accounted for repeated measures by participants.

Results: 415 unique participants attended 203 different CPD presentations and completed the LEI 11,567 times. The originally hypothesized 4-domain model of learner engagement (domains: emotional, behavioral, cognitive in-class, cognitive out-of-class) demonstrated best model fit in all 4 courses, with Comparative Fit Index ≥0.99, Standardized Root Mean Square Residual ≤0.031, and Root Mean Square Error of Approximation ≤0.047. The reliability for overall scores and domain scores were all acceptable (50-rater G-coefficient ≥ 0.74) except for the cognitive-in-class domain (50-rater G-coefficient 0.55 to 0.66). All findings were similar for both in-person and online delivery modalities. Correlation of LEI scores with teaching effectiveness was confirmed (rho 0.58), and a small correlation was found with participant age (rho 0.19); other associations were small and not statistically significant. Using these findings, we generated a shortened 4-item instrument, the LEI Short Form.

Conclusions: This study confirms a 4-domain model of learner engagement, and provides validity evidence that supports using LEI scores to measure learner engagement in both in-person and livestreamed CPD activities.