Measuring teaching effectiveness in a pre-clinical multi-instructor course: a case study in the development and application of a brief instructor rating scale.

Abstract

BACKGROUND: Despite widespread use, misunderstandings persist about student evaluations of teaching. These evaluations have not been well examined in the common medical school setting of the multi-instructor, preclinical lecture course. PURPOSE: The study evaluated the psychometrics of a brief student evaluation of a teaching instrument developed for a multi-instructor 2nd-year course and described its application. METHODS: An 11-item instrument was developed and administered to 276 students to evaluate 27 lecturers per year in 3 years of an introductory clinical psychiatry course. A fully crossed research design allowed for a thorough analysis of variability in ratings. RESULTS: Generalizability analysis showed good reliability and relatively large Student x Lecturer interactions. Profile analysis generated distinct lecturer teaching profiles. CONCLUSIONS: Judicious use of a psychometrically sound student evaluation of a teaching instrument can be used to assist faculty and course development. Administering the evaluation instrument to an entire class produces no better reliability than administration to randomly selected subgroups of students.