Improving reliability of a residency interview process

Am J Pharm Educ. 2013 Oct 14;77(8):168. doi: 10.5688/ajpe778168.

Abstract

Objective: To improve the reliability and discrimination of a pharmacy resident interview evaluation form, and thereby improve the reliability of the interview process.

Methods: In phase 1 of the study, authors used a Many-Facet Rasch Measurement model to optimize an existing evaluation form for reliability and discrimination. In phase 2, interviewer pairs used the modified evaluation form within 4 separate interview stations. In phase 3, 8 interviewers individually-evaluated each candidate in one-on-one interviews.

Results: In phase 1, the evaluation form had a reliability of 0.98 with person separation of 6.56; reproducibly, the form separated applicants into 6 distinct groups. Using that form in phase 2 and 3, our largest variation source was candidates, while content specificity was the next largest variation source. The phase 2 g-coefficient was 0.787, while confirmatory phase 3 was 0.922. Process reliability improved with more stations despite fewer interviewers per station-impact of content specificity was greatly reduced with more interview stations.

Conclusion: A more reliable, discriminating evaluation form was developed to evaluate candidates during resident interviews, and a process was designed that reduced the impact from content specificity.

Keywords: interview; psychometrics; reliability; residency.

MeSH terms

  • Humans
  • Interviews as Topic*
  • Pharmacy Residencies*
  • Psychometrics
  • Reproducibility of Results