Four-Year Analysis of a Novel Milestone-Based Assessment of Faculty by General Surgical Residents

Abstract

Objective: In response to our faculty's concerns about the quality and reliability of feedback from general surgery residents, we developed a novel faculty assessment tool. This study was designed as an interim analysis of the tool's effectiveness and discriminatory ability.

Methods: Our department's educational leadership developed milestones in 7 domains that were scored from 1 to 4, with each level representing an educational approach that ranged from ineffective (1) to ideal (4). Each postgraduate year (PGY) class meets annually to develop a consensus regarding each faculty member's effectiveness in each of the 7 domains: (1) operative supervision, (2) operative teaching, (3) clinic and/or hospital supervision, (4) clinic and/or hospital teaching, (5) conference participation, (6) availability, and (7) overall contribution to the training program. We reviewed the results from the initial 4 years of this project. We also analyzed the annual national faculty survey administered by the Accreditation Council for Graduate Medical Education (ACGME) to evaluate faculty satisfaction regarding feedback during the same study period. Data were assessed using the Levene test for homogeneity, analysis of variance, and Wilcoxon-Mann-Whitney tests.

Results: Forty-two faculty members were annually evaluated by 29 to 32 residents. Each resident PGY class assigned faculty milestone scores that varied across the 7 domains, demonstrating that faculty scores reflected variable opinions about each specific domain, while avoiding labeling an effective faculty member with all high scores and a less effective member with all poor scores.(p < 0.0001). Milestone scores for a given faculty member differed across PGY classes, indicating that junior residents might evaluate a specific faculty member differently than senior residents (p < 0.0001). Eleven faculty members received low scores of 1 or 2 on the overall contribution to training domain and 8/11 (73%) improved to 3 or 4, the following year. Twenty core faculty members were included on the annual ACGME survey. The results from the study period on the ACGME anonymous faculty survey reflected enhanced satisfaction with resident feedback during the study period, improving from 68% to 88% compliance with ACGME standards and our mean program score improved from 4.1 to 4.4 compared to the national mean of 4.3 (p = 0.02).

Conclusions: This milestone-based faculty assessment tool improves the quality of the feedback from surgical residents when evaluating faculty. When residents assign a negative statement to describe faculty educational effectiveness in a specific domain, performance improves. A milestone-based faculty assessment strategy should be explored on a national level.

Keywords: Evaluations; Interpersonal and Communication Skills; Patient Care; Practice-Based Learning and Improvement; Professionalism; faculty; feedback; milestones.