The Reliability of Rating via Audio-Recording Using the Mindfulness-Based Interventions: Teaching Assessment Criteria

Erin Floyd; Shelley R Adler; Rebecca S Crane; Judson Brewer; Patricia Moran; Robert Richler; Wendy Hartogensis; Willem Kuyken; Frederick M Hecht

doi:10.1177/27536130221149966

The Reliability of Rating via Audio-Recording Using the Mindfulness-Based Interventions: Teaching Assessment Criteria

Glob Adv Integr Med Health. 2023 May 17:12:27536130221149966. doi: 10.1177/27536130221149966. eCollection 2023 Jan-Dec.

Authors

Erin Floyd^{1

2}, Shelley R Adler¹, Rebecca S Crane³, Judson Brewer⁴, Patricia Moran¹, Robert Richler¹, Wendy Hartogensis¹, Willem Kuyken⁵, Frederick M Hecht¹

Affiliations

¹ Osher Center for Integrative Medicine, University of California, San Francisco, CA, USA.
² Department of Medicine, University of Wisconsin Hospital and Clinics, Madison, WI, USA.
³ Centre for Mindfulness Research and Practice, Bangor University, Bangor, UK.
⁴ Mindfulness Center at Brown, Brown University, Providence, RI, USA.
⁵ Oxford Mindfulness Centre, Oxford University, Oxford, UK.

Abstract

Background: The Mindfulness-Based Interventions: Teaching Assessment Criteria (MBI:TAC) is an important tool for assessing teacher skill and aspects of the fidelity of mindfulness-based interventions, but prior research on and implementation of the MBI:TAC has used video recordings, which can be difficult to obtain, share for assessments, and which increase privacy concerns for participants. Audio-only recordings might be a useful alternative, but their reliability is unknown.

Objective: To assess evaluator perception of the rating process and inter-rater reliability of MBI:TAC ratings using audio-only recordings.

Methods: We prepared audio-only files from video recordings of 21 previously rated Mindfulness-Based Stress Reduction teachers. Each audio recording was rated by 3 trained MBI:TAC assessors drawn from a pool of 12 who had previously participated in rating the video recordings. Teachers were rated by evaluators who had not viewed the video recording and did not know the teacher. We then conducted semi-structured interviews with evaluators.

Results: On the 6 MBI:TAC domains, the intraclass correlation coefficients (ICCs) for audio recordings ranged from .53 to .69 using an average across 3 evaluators. Using a single rating resulted in lower ICCs (.27-.38). Bland-Altman plots showed audio ratings had little consistent bias compared to video recordings and agreed more closely for teachers with higher ratings. Qualitative analysis identified 3 themes: video recordings were particularly helpful when rating less skillful teachers, video recordings tended to provide a more complete picture for rating, and audio rating had some positive features.

Conclusions: Inter-rater reliability of the MBI:TAC using audio-only recordings was adequate for many research and clinical purposes, and reliability is improved when using an average across several evaluators. Ratings using audio-only recordings may be more challenging when rating less experienced teachers.

Keywords: MBI:TAC; MBSR; intervention fidelity; intervention integrity; mindfulness.

Abstract

Grants and funding