Introducing progress testing in McMaster University's problem-based medical curriculum: psychometric properties and effect on learning.

Abstract

BACKGROUND: The progress test (or Quarterly Profile Examination), invented concurrently by the University of Missouri-Kansas City School of Medicine and the University of Limburg, is used to provide useful summative and formative judgments about students' knowledge without distorting learning. All students in all classes sit the same examination at regular intervals through the year, and their individual progress is noted. This paper reports on four years' experience with a progress test, the Personal Progress Index (PPI), at the McMaster University Faculty of Health Sciences. METHOD: The PPI, a 180-item multiple-choice test with items drawn from all disciplines of medicine, is administered to medical students in all three classes three times per year. Individual feedback is provided, and accumulated student performance is determined for summative purposes. This paper examines extensive evidence of reliability, validity, and effect on student learning, using samples from the entering classes of 1992-1995 (a total of 400 students). RESULTS: Reliabilities across test administrations (within classes) ranged from .46 to .63. The PPI demonstrated strong construct validity, with highly significant statistical tests of differences between classes and changes within classes on successive administrations. The predictive validity of the PPI, i.e., whether it could predict performance on the licensing examination of the Medical Council of Canada, increased monotonically from a correlation of .12 for the first test administration one month into medical school to a high of about .60 for the cumulative score across all administrations three months prior to the examination. CONCLUSION: The PPI seems to be performing as intended, with students showing growth in performance across the three years of the MD program. Additional benefits are that many more students now self-refer for remediation (based on low PPI scores) and that the consistent relative performances of individual students across test administrations allow for the identification of students who have severe and persistent problems.