Biomedical subjects

Harold I Reiter

Publications and source records attributed to Harold I Reiter.

13 recordsLinked to original sources

PubMed2006

Impact of field of study, college and year on calculation of cumulative grade point average.

A consistent finding from many reviews is that undergraduate Grade Point Average (uGPA) is a key predictor of academic success in medical school. Curiously, while uGPA has established predictive validity, little is known about its reliability. For a variety of reasons, medical schools use different weighting schemas to combine years of study. Additional concerns relate to the equivalence of grades obtained from different fields of study and institutions, with little hard data to guide conclusions. At the Michael G. DeGroote School of Medicine Class of 2007 at McMaster University, every undergraduate grade of 2,138 applicants, along with field of study and post-secondary educational institution, was analyzed. Individual grades were aggregated into an overall uGPA using published algorithms from several medical school, and correlated with a non-weighted sum. Correlations of the different schemas with equal weights ranged from 0.973 to 0.990. The extent of the difference between fields of study was small, accounting for only 1.5% of the variance. However, differences among 16 Ontario universities were larger, and accounted for 9.3% of the variance. The results of this study suggest that all weighting schemas are virtually equivalent, making any formulation reasonable. Differences by field of study are small, but do not show any bias against non-science students. Differences by institution are larger, amounting to a range in average score from 78.7 to 84.6; however it is not clear whether this reflects candidate ability or institutional policy, so attempts to correct for institution may be difficult.

Algorithms↗

PubMed2006

Can self-declared personal values be used to identify those with family medicine career aspirations?

PURPOSE: Self-declaration of personal values has been suggested as a means of identifying students with greater predilection for future primary care careers. While statistically significant differences have been demonstrated, absolute differences between those interested in primary care and those interested in specialist careers tend to be small. This study's objective was to determine whether limited differences can nevertheless be used to identify those with particular interest in primary care. METHODS: Schwartz' values were submitted to a paired comparison design in which respondents were asked to identify the value they held most dearly for possible pairings of values. 88 medical school applicants, 57 first year medical students, 78 final year medical students, and 34 admissions interviewers participated. Applicant and medical student subgroups were analyzed as a whole and as a function of self-identified career interest (i.e., primarily interested in Family medicine or other specialty careers). RESULTS: The values statements were remarkably consistent (r > 0.90) between groups, regardless of which subgroups were analyzed. DISCUSSION: Despite apparent differences in the literature between those interested in primary care and those interested in other specialist careers, the differences are small and do not correlate with career aspirations in a way that could inform admissions decisions.

Attitude↗

PubMed2006

A cost efficiency comparison between the multiple mini-interview and traditional admissions interviews.

A major expense for most professional training programs, both financially and in terms of human resources, is the interview process used to make admissions decisions. Still, most programs view this as a necessary cost given that the personal interview provides an opportunity to recruit potential candidates, showing them what the program has to offer, and to try and gather more information about the candidates to ensure that those selected live up to the espoused values of the institution. We now have five years worth of experience with a Multiple Mini-Interview (MMI) process that, unlike traditional panel interviews, uses the OSCE model to have candidates interact with a larger number of interviewers. We have found that the MMI is more reliable and has better predictive power than our traditional panel interviews. Still, the extent to which any measurement is valuable depends also on the feasibility of use. In this paper we report on an exploration of the cost effectiveness of the MMI as compared to standard panel-based interviews by considering the generation of interview material, human resource (i.e., interviewer and support staff) use, infrastructure requirements, and other miscellaneous expenses. Our conclusion is that the MMI requires greater preparatory efforts and a larger number of rooms to carry out the interviews relative to panel-based interviews, but that these cost disadvantages are offset by the MMI requiring fewer person-hours of effort. The absolute costs will vary dependent on institution, but the framework presented in this paper will hopefully provide greater guidance regarding logistical requirements and anticipated budget.

Humans↗

PubMed2006

Medical school admissions: enhancing the reliability and validity of an autobiographical screening tool.

BACKGROUND: Most medical school applicants are screened out preinterview. Some cognitive scores available preinterview and some noncognitive scores available at interview demonstrate reasonable reliability and predictive validity. A reliable preinterview noncognitive measure would relax dependence upon screening based entirely on cognitive tendencies. METHOD: In 2005, applicants interviewing at McMaster University's Michael G. DeGroote School of Medicine completed an offsite, noninvigilated, Autobiographical Submission (ABS) preinterview and another onsite, invigilated, ABS at interview. Traditional and new ABS scoring methods were compared, with raters either evaluating all ABS questions for each candidate in turn (vertical scoring-traditional method) or evaluating all candidates for each question in turn (horizontal scoring-new method). RESULTS: The new scoring method revealed lower internal consistency and higher interrater reliability relative to the traditional method. More importantly, the new scoring method correlated better with the Multiple Mini-Interview (MMI) relative to the traditional method. CONCLUSIONS: The new ABS scoring method revealed greater interrater reliability and predictive capacity, thus increasing its potential as a screen for noncognitive characteristics.

Autobiographies as Topic↗

PubMed2006

The effect of defined violations of test security on admissions outcomes using multiple mini-interviews.

INTRODUCTION: Heterogeneous results exist regarding the impact of security violations on student performances in objective structured clinical examinations (OSCEs). Three separate studies investigate whether anticipated security violations result in undesirable enhancement of MMI performance ratings. METHODS: Study 1: low-stakes: MMI station stems provided to a random half of 57 medical school applicants 2 weeks in advance of participation in a research study. Study 2: high-stakes: 384 medical school applicants sat a 12-station MMI to determine admission. Each half received 1 of 2 pilot MMI station stems 2 weeks in advance. Study 3: high-stakes: 38 interviewees with dual applications to occupational therapy and physiotherapy experienced the same 7-station MMI twice on the same date. RESULTS: No statistically significant differences in MMI performances were detected. CONCLUSIONS: Predictable violations of MMI security do not unduly influence applicant performance ratings.

Interviews as Topic↗

PubMed2005

Reflecting the relative values of community, faculty, and students in the admissions tools of medical school.

BACKGROUND: In defining the characteristics of medical students that society and the medical profession find desirable, little effort has been spent assessing the relative value of the dozens of characteristics that have been identified. Furthermore, many institutions go to great lengths to ensure equal representation across stakeholder groups in an effort to maximize the heterogeneity of the pool of students accepted to study medicine; however, the extent to which different stakeholders value different characteristics has yet to be determined. PURPOSE: This study was an attempt to assess the relative value of the characteristics of medical students that society and the medical profession find desirable. METHODS: Using documents created internationally to identify the core competencies of medical personnel, a series of 7 characteristics were generated for inclusion in a study that adopted the paired comparison technique. Of 347 surveyed, 292 respondents indicated the rank ordering they would assign to each characteristic by circling the more important characteristic in all possible pairings. RESULTS: Overwhelmingly, "ethical" was deemed to be the most important characteristic on which selection tools should be based. Surprisingly, the pattern of responses was highly consistent regardless of stakeholder group and degree of affiliation with the undergraduate medical program. CONCLUSIONS: The generalizable features of this study not only include the empirical findings but also demonstrate useful survey protocol that can be adapted by any admission committee to guide the generation of an institution-specific admissions blueprint. A novel protocol that provides the necessary flexibility is discussed.

Faculty↗

PubMed2004

Where judgement fails: pitfalls in the selection process for medical personnel.

Despite the critical importance of maintaining a valid and transparent selection process that serves the values held by all stakeholders involved in medical education (i.e., students, faculty, society), there continue to be problems with the current state of available admissions protocols. Some problems derive from inertia induced by inaccurate intuitions pertaining to the nature of admissions protocols and the underlying qualities being measured. Others arise from the lack of reliable and valid admissions protocols to capture the non-cognitive qualities of candidates. Still other issues arise from the problem of rumor-based perceptions regarding the qualities for which selection protocols select. Three articles in this issue of Advances in Health Sciences Education present evidence pertaining to the selection of medical personnel. This commentary represents an attempt to bind together these unique perspectives on the admissions process while also casting light on other ways in which human judgment can fail in this domain in the hope that it might help steer decision-makers away from these pitfalls.

Decision Making↗

PubMed2004

Do clinical clerks provide candidates with adequate formative assessment during Objective Structured Clinical Examinations?

CONTEXT: Various research studies have examined the question of whether expert or non-expert raters, faculty or students, evaluators or standardized patients, give more reliable and valid summative assessments of performance on Objective Structured Clinical Examinations (OSCEs). Less studied has been the question of whether or not non-faculty raters can provide formative feedback that allows students to take advantage of the educational opportunity that OSCEs provide. This question is becoming increasingly important, however, as the strain on faculty resources increases. METHODS: A questionnaire was developed to assess the quality of feedback that medical examiners provide during OSCEs. It was pilot tested for reliability using video recordings of OSCE performances. The questionnaires were then used to evaluate the feedback given during an actual OSCE in which clinical clerks, residents, and faculty were used as examiners on two randomly selected test stations. RESULTS: The inter-rater reliability of the 19-item feedback questionnaire was 0.69 during the pilot test. The internal consistency was found to be 0.90 during pilot testing and 0.95 in the real OSCE. Using this form, the feedback ratings assigned to clinical clerks were significantly greater than those assigned to faculty evaluators. Furthermore, performance on the same OSCE stations eight months later was not impaired by having been evaluated by student examiners. DISCUSSION: While evidence of mark inflation within the clinical clerk examiners should be addressed with examiner training, the current results suggest that clerks are capable of giving adequate formative feedback to more junior colleagues.

Adult↗

PubMed2004

How can I know what I don't know? Poor self assessment in a well-defined domain.

As the rapidity with which medical knowledge is generated and disseminated becomes amplified, an increasing emphasis has been placed on the need for physicians to develop the skills necessary for life-long learning. One such skill is the ability to evaluate one's own deficiencies. A ubiquitous finding in the study of self-assessment, however, is that self-ratings are poorly correlated with other performance measures. Still, many educators view the ability to recognize and communicate one's deficiencies as an important component of adult learning. As a result, two studies have been performed in an attempt to improve upon this status quo. First, we tried to re-define the limits within which self-assessments should be used, using Rosenblit and Keil's argument that calibration between perceived and actual performance will be better within taxonomies that are regularly tested (e.g., factual knowledge) compared to those that are not (e.g., conceptual knowledge). Second, we tried to norm reference individuals based on both the performance of their colleagues and their own historical performance on McMaster's Personal Progress Inventory (a multiple choice question test of medical knowledge). While it appears that students are able to (a) make macro-level self-assessments (i.e., to recognize that third year students typically outperform first year students), and (b) judge their performance relatively accurately after the fact, students were unable to predict the percentage of questions they would answer correctly with a testing procedure in which they have had a substantial amount of feedback. Previous test score was a much better predictor of current test performance than were individuals' expectations.

Adult↗

PubMed2004

An admissions OSCE: the multiple mini-interview.

CONTEXT: Although health sciences programmes continue to value non-cognitive variables such as interpersonal skills and professionalism, it is not clear that current admissions tools like the personal interview are capable of assessing ability in these domains. Hypothesising that many of the problems with the personal interview might be explained, at least in part, by it being yet another measurement tool that is plagued by context specificity, we have attempted to develop a multiple sample approach to the personal interview. METHODS: A group of 117 applicants to the undergraduate MD programme at McMaster University participated in a multiple mini-interview (MMI), consisting of 10 short objective structured clinical examination (OSCE)-style stations, in which they were presented with scenarios that required them to discuss a health-related issue (e.g. the use of placebos) with an interviewer, interact with a standardised confederate while an examiner observed the interpersonal skills displayed, or answer traditional interview questions. RESULTS: The reliability of the MMI was observed to be 0.65. Furthermore, the hypothesis that context specificity might reduce the validity of traditional interviews was supported by the finding that the variance component attributable to candidate-station interaction was greater than that attributable to candidate. Both applicants and examiners were positive about the experience and the potential for this protocol. DISCUSSION: The principles used in developing this new admissions instrument, the flexibility inherent in the multiple mini-interview, and its feasibility and cost-effectiveness are discussed.

Adult↗

PubMed2004

The relationship between interviewers' characteristics and ratings assigned during a multiple mini-interview.

PURPOSE: To assess the consistency of ratings assigned by health sciences faculty members relative to community members during an innovative admissions protocol called the Multiple Mini-Interview (MMI). METHOD: A nine-station MMI was created and 54 candidates to an undergraduate MD program participated in the exercise in Spring 2003. Three stations were staffed with a pair of faculty members, three with a pair of community members, and three with one member of each group. Raters completed a four-item evaluation form. All participants completed post-MMI questionnaires. Generalizability Theory was used to examine the consistency of the ratings provided within each of these three subgroups. RESULTS: The overall test reliability was found to be .78 and a Decision Study suggested that admissions committees should distribute their resources by increasing the number of interviews to which candidates are exposed rather than increasing the number of interviewers within each interview. Divergence of ratings was greater within the pairing of community member to faculty member and least for pairings of community members. Participants responded positively to the MMI. CONCLUSION: The MMI provides a reliable protocol for assessing the personal qualities of candidates by accounting for context specificity with a multiple sampling approach. Increasing the heterogeneity of interviewers may increase the heterogeneity of the accepted group of candidates. Further work will determine the extent to which different groups of raters provide equally valid (albeit different) judgments.

Adult↗

PubMed2004

The ability of the multiple mini-interview to predict preclerkship performance in medical school.

PROBLEM STATEMENT AND BACKGROUND: One of the greatest challenges continuing to face medical educators is the development of an admissions protocol that provides valid information pertaining to the noncognitive qualities candidates possess. An innovative protocol, the Multiple Mini-Interview, has recently been shown to be feasible, acceptable, and reliable. This article presents a first assessment of the technique's validity. METHOD: Forty five candidates to the Undergraduate MD program at McMaster University participated in an MMI in Spring 2002 and enrolled in the program the following autumn. Performance on this tool and on the traditional protocol was compared to performance on preclerkship evaluation exercises. RESULTS: The MMI was the best predictor of objective structured clinical examination performance and grade point average was the most consistent predictor of performance on multiple-choice question examinations of medical knowledge. CONCLUSIONS: While further validity testing is required, the MMI appears better able to predict preclerkship performance relative to traditional tools designed to assess the noncognitive qualities of applicants.

Clinical Clerkship↗

PubMed2002

Self and peer assessment in tutorials: application of a relative-ranking model.

PURPOSE: While self assessment continues to be touted as being of paramount importance for continuing professional competence, problem-based learning curricula, and adult learning theory, techniques for ensuring valid judgments have proven elusive. This study tested the applicability of an innovative relative-ranking procedure to problem-based learning tutorials. METHOD: A total of 36 students in the McMaster University Faculty of Health Sciences' MD program were provided relative-ranking forms listing seven domains of competence along with their definitions. The student, two of the student's peers, and the student's tutor were asked to complete the ranking exercise after their second, fourth, and sixth tutorials. RESULTS: Combining each level of the time and rater variables generated 66 correlation coefficients, none of which was significantly different from zero. Re-performing the analysis on only the extreme domains did not improve this result. CONCLUSION: The relative-ranking instrument developed did not prove to be a reliable measure of tutorial performance. Ratings were inconsistent from one week to the next as well as across raters within a week.

Attitude of Health Personnel↗