PubMed · 9286843
The source did not provide an abstract. Follow the original record for more information.
Explore connections, maps & timelines
I Oransky. 1997-09-03. Evaluating the evaluations.. https://doi.org/10.1001/jama.278.9.788
Cite the original work for its findings. Save a collection to share your selection of sources.
KEEP EXPLORING
PURPOSE: Despite their widespread use, medical school admission interviews often are unstructured and lack reliability. This report describes the development of a structured admission interview designed to eliminate bias and provide valid information for selecting medical students, with preliminary information about the interview's reliability and validity. METHOD: After screening applications, 490 applicants to a public medical school residency program were interviewed by two faculty members using a structured interview format. Interview scores were compiled and correlated with undergraduate grade-point averages (GPAs); Medical College Admission Test (MCAT) scores; Iowa Evaluation Form (IEF) scores, an in-house evaluation of applicants' noncognitive abilities; and eventual admissions status. RESULTS: Interrater agreement was good; the percentages of rater pairs whose scores differed by one point or less ranged from 87% to 98%. Scores on the structured interview revealed low to moderate correlations with other admission criteria: 10 (p < 0.05) for cumulative GPA, 0.18 (p < 0.01) for MCAT Biological Science, 0.08 (p > 0.05) MCAT Physical Science, and 0.10 (p < 0.05) MCAT Verbal Reasoning. None of the correlations between the overall interview scores and the IEF scores reached statistical significance (p = 0.05). Higher overall scores on the structured interview did predict a greater likelihood of being accepted into the medical school and the interview score accounted for 20% of the incremental variance in admission status beyond GPA, MCAT, and IEF scores. CONCLUSIONS: The moderate-to-low correlations with other admission criteria suggest that the interview provided information about candidate credentials not obtained from other sources and accounted for a substantial proportion of the variance in admission status. This finding supports the considerable time and resources required to develop a structured interview for medical student admissions. Final judgment on the validity and utility of this interview should be made after follow-up performance data have been obtained and analyzed.
Evaluation and feedback are fundamental components of graduate medical education. Paper-based evaluation systems are inefficient and costly and the evaluation data they provide are difficult to retrieve and analyze. In view of these problems, in 1996-1997, the authors developed and implemented a World Wide Web-based electronic evaluation system for the internal medicine residency program at the University of Minnesota. Residents were evaluated using the American Board of Internal Medicine Resident Evaluation Form. Custom evaluations were created for the assessment of sites, rotations, and faculty. The evaluations were completed by accessing an evaluation Web site from any location using standard computers and Web browsers. The evaluations were submitted electronically and automatically entered into a database. The system tracked compliance and automatically sent out reminders. Other features of the system included extensive reporting capabilities, automatic notification of substandard performance, and the ability to send confidential information to the program director. The total compliance rate ranged between 81% and 92% during the first 12 months of operation, with no significant difference in compliance observed between faculty and residents. The system was easy to use and could quickly and confidentially identify performance problems of residents and faculty from large numbers of evaluations.
Explore the source record for details and available documents.