Biases related to variations among patients and settings may be reduced by having faculty evaluate the clinical skills of students as they interact with simulated patients. The Student Clinical Performance Scale is a rating scale developed to standardize assessment of videotaped simulated patient encounters with family nurse practitioner students. In a pilot study of faculty interrater reliability, correlation coefficients for independent scoring by faculty pairs using the Student Clinical Performance Scale varied widely among six case studies, and several scenario- and faculty-related factors served as systematic sources of faculty variability. The findings demonstrate the importance of a rigorous approach to developing and testing instruments that guide measurement of clinical competence. The use of simulated patients to evaluate nurse practitioner students provides opportunities to assess a wide range of attributes and holds promise for assessing advanced clinical skills in controlled educational environments.