PubMed Health⌕ Search

Biomedical subjects

Kevin W Eva

Publications and source records attributed to Kevin W Eva.

At least 19 recordsLinked to original sources
PubMed2006

Using a sampling strategy to address psychometric challenges in tutorial-based assessments.

INTRODUCTION: Tutorial-based assessment, despite providing a good match with the philosophy adopted by educational programmes that emphasize small group learning, remains one of the greatest challenges for educators working in this context. The current study was performed in an attempt to assess the psychometric characteristics of tutorial-based evaluation upon adopting a multiple sampling approach that requires minimal recording of observations. METHOD: After reviewing the literature, a simple 3-item evaluation form was created. The items were "Professional Behaviour," "Contribution to Group Process," and "Contribution to Group Content." Explicit definition of these items was provided on an evaluation form. Twenty five tutors in five different programmes were asked to use the form to evaluate their students (N=169) after every tutorial over the course of an academic unit. Each item was rated using a 10-point scale. RESULTS: Cronbach's alpha revealed an appropriate internal consistency in all five programmes. Test-retest reliability of any single rating was low, but the reliability of the average rating was at least 0.75 in all cases. The construct validity of the tool was supported by the observation of increasing ratings over the course of the academic unit and by the finding that more senior students received higher ratings than more junior students. CONCLUSION: Consistent with the context specificity phenomenon, the adoption of a "minimal observations often" approach to tutorial-based assessment appears to maintain better psychometric characteristics than do attempts to assess tutorial performance using more comprehensive measurement tools.

Curriculum↗
PubMed2006

Can self-declared personal values be used to identify those with family medicine career aspirations?

PURPOSE: Self-declaration of personal values has been suggested as a means of identifying students with greater predilection for future primary care careers. While statistically significant differences have been demonstrated, absolute differences between those interested in primary care and those interested in specialist careers tend to be small. This study's objective was to determine whether limited differences can nevertheless be used to identify those with particular interest in primary care. METHODS: Schwartz' values were submitted to a paired comparison design in which respondents were asked to identify the value they held most dearly for possible pairings of values. 88 medical school applicants, 57 first year medical students, 78 final year medical students, and 34 admissions interviewers participated. Applicant and medical student subgroups were analyzed as a whole and as a function of self-identified career interest (i.e., primarily interested in Family medicine or other specialty careers). RESULTS: The values statements were remarkably consistent (r > 0.90) between groups, regardless of which subgroups were analyzed. DISCUSSION: Despite apparent differences in the literature between those interested in primary care and those interested in other specialist careers, the differences are small and do not correlate with career aspirations in a way that could inform admissions decisions.

Attitude↗
PubMed2006

A cost efficiency comparison between the multiple mini-interview and traditional admissions interviews.

A major expense for most professional training programs, both financially and in terms of human resources, is the interview process used to make admissions decisions. Still, most programs view this as a necessary cost given that the personal interview provides an opportunity to recruit potential candidates, showing them what the program has to offer, and to try and gather more information about the candidates to ensure that those selected live up to the espoused values of the institution. We now have five years worth of experience with a Multiple Mini-Interview (MMI) process that, unlike traditional panel interviews, uses the OSCE model to have candidates interact with a larger number of interviewers. We have found that the MMI is more reliable and has better predictive power than our traditional panel interviews. Still, the extent to which any measurement is valuable depends also on the feasibility of use. In this paper we report on an exploration of the cost effectiveness of the MMI as compared to standard panel-based interviews by considering the generation of interview material, human resource (i.e., interviewer and support staff) use, infrastructure requirements, and other miscellaneous expenses. Our conclusion is that the MMI requires greater preparatory efforts and a larger number of rooms to carry out the interviews relative to panel-based interviews, but that these cost disadvantages are offset by the MMI requiring fewer person-hours of effort. The absolute costs will vary dependent on institution, but the framework presented in this paper will hopefully provide greater guidance regarding logistical requirements and anticipated budget.

Humans↗
PubMed2006

The difficulty with experience: does practice increase susceptibility to premature closure?

INTRODUCTION: A recent review of the physician performance literature concluded that the risk of prematurely closing one's diagnostic search increases with years of experience. To minimize confounding variables and gain insight into cognitive issues relevant to continuing education, the current study was performed to test this conclusion. METHODS: Physician participants were shown a series of case histories and asked to judge the probability of a pair of diagnoses. The order in which features were presented was manipulated across participants and the probabilities compared to determine the impact of information order. Two groups of participants were recruited, 1 older than and 1 younger than 60 years. RESULTS: The probability assigned to a diagnosis tended to be greater when features consistent with that diagnosis preceded those consistent with an alternative than when the same features followed those consistent with the alternative. Older participants revealed a greater primacy effect than less experienced participants across 4 experimental conditions. DISCUSSION: Physicians with greater experience appear to weigh their first impressions more heavily than those with less experience. Educators should design instructional activities that account for experience-specific cognitive tendencies.

Aged↗
PubMed2006

Giving learners the best of both worlds: do clinical teachers need to guard against teaching pattern recognition to novices?

PURPOSE: There has been much debate in the medical education literature regarding the extent to which feature-driven and nonanalytic (similarity-based) reasoning strategies define expertise, but the relative value of teaching these strategies, together or in isolation, remains uncertain. The purpose of this study was to compare the diagnostic accuracy achieved upon receiving instruction to use each strategy in isolation to that of a combined approach. METHOD: In 2003-04, 48 undergraduate psychology students from McMaster University in Ontario, Canada, were taught to diagnose ten cardiac disorders (including normal) via electrocardiogram (ECG) presentation. Twelve students were instructed to carefully identify all features present before assigning a diagnosis (feature first). Twelve were given the same instruction with notice that some test ECGs had been seen during training (implicit combined). Twelve were simply instructed to trust familiarity and diagnose based on this impression (similarity-based). Finally, 12 students were given feature first and similarity-based instructions in combination (explicit combined). RESULTS: No difference in diagnostic accuracy was observed between the groups given the feature first (42%) and first impression (41%) instructions (p > .4), but the groups instructed to use both strategies (explicitly or implicitly) performed significantly better (56% and 53%, respectively; p < .01). CONCLUSIONS: The results support an additive model of clinical reasoning in which instructions to be feature oriented and to trust similarity improve performance in novice diagnosticians.

Adult↗
PubMed2006

Medical school admissions: enhancing the reliability and validity of an autobiographical screening tool.

BACKGROUND: Most medical school applicants are screened out preinterview. Some cognitive scores available preinterview and some noncognitive scores available at interview demonstrate reasonable reliability and predictive validity. A reliable preinterview noncognitive measure would relax dependence upon screening based entirely on cognitive tendencies. METHOD: In 2005, applicants interviewing at McMaster University's Michael G. DeGroote School of Medicine completed an offsite, noninvigilated, Autobiographical Submission (ABS) preinterview and another onsite, invigilated, ABS at interview. Traditional and new ABS scoring methods were compared, with raters either evaluating all ABS questions for each candidate in turn (vertical scoring-traditional method) or evaluating all candidates for each question in turn (horizontal scoring-new method). RESULTS: The new scoring method revealed lower internal consistency and higher interrater reliability relative to the traditional method. More importantly, the new scoring method correlated better with the Multiple Mini-Interview (MMI) relative to the traditional method. CONCLUSIONS: The new ABS scoring method revealed greater interrater reliability and predictive capacity, thus increasing its potential as a screen for noncognitive characteristics.

Autobiographies as Topic↗
PubMed2006

The effect of defined violations of test security on admissions outcomes using multiple mini-interviews.

INTRODUCTION: Heterogeneous results exist regarding the impact of security violations on student performances in objective structured clinical examinations (OSCEs). Three separate studies investigate whether anticipated security violations result in undesirable enhancement of MMI performance ratings. METHODS: Study 1: low-stakes: MMI station stems provided to a random half of 57 medical school applicants 2 weeks in advance of participation in a research study. Study 2: high-stakes: 384 medical school applicants sat a 12-station MMI to determine admission. Each half received 1 of 2 pilot MMI station stems 2 weeks in advance. Study 3: high-stakes: 38 interviewees with dual applications to occupational therapy and physiotherapy experienced the same 7-station MMI twice on the same date. RESULTS: No statistically significant differences in MMI performances were detected. CONCLUSIONS: Predictable violations of MMI security do not unduly influence applicant performance ratings.

Interviews as Topic↗
PubMed2006

Comparison of aboriginal and nonaboriginal applicants for admissions on the Multiple Mini-Interview using aboriginal and nonaboriginal interviewers.

BACKGROUND: Achievement on grade point average and Medical College Admissions Test contribute as unintentional barriers to advancement of underrepresented minorities. So long as noncognitive measures mimic random number generators, they merely perpetuate such discrepancies. As reliable noncognitive measures are developed, it is crucial to ensure immunity from bias, enabling them to better dilute unintended discrimination of cognitive measures. PURPOSE: The Multiple Mini-Interview (MMI) is a recently developed, reliable (overall reliability = .70), noncognitive measure used for assessment of medical school applicants. Our purpose in this study was to evaluate whether any suggestion of bias existed in application of the MMI in its assessment of aboriginal medical school applicants. METHODS: In this study of the MMI (overall reliability = .70), each of 5 self-declared aboriginal applicants and 7 general-pool applicants experienced the same 11 vetted interview stations with the same 6 aboriginal raters and 5 nonaboriginal raters. RESULTS: The Interviewer Type x Interviewee Type interaction was nonsignificant, p > .7. CONCLUSION: Based on the results of this study, it is recommended that MMI stations be vetted by aboriginally sensitive personnel, but neither aboriginal-specific rater training nor aboriginal rater assignment is required to ensure a level playing field for the assessment of applicants' personal qualities.

Canada↗
PubMed2005

What every teacher needs to know about clinical reasoning.

CONTEXT: One of the core tasks assigned to clinical teachers is to enable students to sort through a cluster of features presented by a patient and accurately assign a diagnostic label, with the development of an appropriate treatment strategy being the end goal. Over the last 30 years there has been considerable debate within the health sciences education literature regarding the model that best describes how expert clinicians generate diagnostic decisions. PURPOSE: The purpose of this essay is to provide a review of the research literature on clinical reasoning for frontline clinical teachers. The strengths and weaknesses of different approaches to clinical reasoning will be examined using one of the core divides between various models (that of analytic (i.e. conscious/controlled) versus non-analytic (i.e. unconscious/automatic) reasoning strategies) as an orienting framework. DISCUSSION: Recent work suggests that clinical teachers should stress the importance of both forms of reasoning, thereby enabling students to marshal reasoning processes in a flexible and context-specific manner. Specific implications are drawn from this overview for clinical teachers.

Clinical Competence↗
PubMed2005

Reflecting the relative values of community, faculty, and students in the admissions tools of medical school.

BACKGROUND: In defining the characteristics of medical students that society and the medical profession find desirable, little effort has been spent assessing the relative value of the dozens of characteristics that have been identified. Furthermore, many institutions go to great lengths to ensure equal representation across stakeholder groups in an effort to maximize the heterogeneity of the pool of students accepted to study medicine; however, the extent to which different stakeholders value different characteristics has yet to be determined. PURPOSE: This study was an attempt to assess the relative value of the characteristics of medical students that society and the medical profession find desirable. METHODS: Using documents created internationally to identify the core competencies of medical personnel, a series of 7 characteristics were generated for inclusion in a study that adopted the paired comparison technique. Of 347 surveyed, 292 respondents indicated the rank ordering they would assign to each characteristic by circling the more important characteristic in all possible pairings. RESULTS: Overwhelmingly, "ethical" was deemed to be the most important characteristic on which selection tools should be based. Surprisingly, the pattern of responses was highly consistent regardless of stakeholder group and degree of affiliation with the undergraduate medical program. CONCLUSIONS: The generalizable features of this study not only include the empirical findings but also demonstrate useful survey protocol that can be adapted by any admission committee to guide the generation of an institution-specific admissions blueprint. A novel protocol that provides the necessary flexibility is discussed.

Faculty↗
PubMed2004

Where judgement fails: pitfalls in the selection process for medical personnel.

Despite the critical importance of maintaining a valid and transparent selection process that serves the values held by all stakeholders involved in medical education (i.e., students, faculty, society), there continue to be problems with the current state of available admissions protocols. Some problems derive from inertia induced by inaccurate intuitions pertaining to the nature of admissions protocols and the underlying qualities being measured. Others arise from the lack of reliable and valid admissions protocols to capture the non-cognitive qualities of candidates. Still other issues arise from the problem of rumor-based perceptions regarding the qualities for which selection protocols select. Three articles in this issue of Advances in Health Sciences Education present evidence pertaining to the selection of medical personnel. This commentary represents an attempt to bind together these unique perspectives on the admissions process while also casting light on other ways in which human judgment can fail in this domain in the hope that it might help steer decision-makers away from these pitfalls.

Decision Making↗