PubMed Health⌕ Search

Biomedical subjects

John Norcini

Publications and source records attributed to John Norcini.

11 recordsLinked to original sources
PubMed2006

mini-PAT (Peer Assessment Tool): a valid component of a national assessment programme in the UK?

PURPOSE: To design, implement and evaluate a multisource feedback instrument to assess Foundation trainees across the UK. METHODS: mini-PAT (Peer Assessment Tool) was modified from SPRAT (Sheffield Peer Review Assessment Tool), an established multisource feedback (360 degrees ) instrument to assess more senior doctors, as part of a blueprinting exercise of instruments suitable for assessment in Foundation programmes (first 2 years postgraduation). mini-PAT's content validity was assured by a mapping exercise against the Foundation Curriculum. Trainees' clinical performance was then assessed using 16 questions rated against a six-point scale on two occasions in the pilot period. Responses were analysed to determine internal structure, potential sources of bias and measurement characteristics. RESULTS: Six hundred and ninety-three mini-PAT assessments were undertaken for 553 trainees across 12 Deaneries in England, Wales and Northern Ireland. Two hundred and nineteen trainees were F1s or PRHOs and 334 were F2s. Trainees identified 5544 assessors of whom 67% responded. The mean score for F2 trainees was 4.61 (SD = 0.43) and for F1s was 4.44 (SD = 0.56). An independent t test showed that the mean scores of these 2 groups were significantly different (t = -4.59, df 390, p < 0.001). 43 F1s (19.6%) and 19 F2s (5.6%) were assessed as being below expectations for F2 completion. The factor analysis produced 2 main factors, one concerned clinical performance, the other humanistic qualities. Seventy-four percent of F2 trainees could have been assessed by as few as 8 assessors (95% CI +/-0.6) as they either scored an overall mean of 4.4 or above or 3.6 and below. Fifty-three percent of F1 trainees could have been assessed by as few as 8 assessors (95% CI +/-0.5) as they scored an overall mean of 4.5 or above or 3.5 and below. The hierarchical regression when controlling for the grade of trainee showed that bias related to the length of the working relationship, occupation of the assessor and the working environment explained 7% of the variation in mean scores when controlling for the year of the Foundation Programme (R squared change = 0.06, F change = 8.5, significant F change <0.001). CONCLUSIONS: As part of an assessment programme, mini-PAT appears to provide a valid way of collating colleague opinions to help reliably assess Foundation trainees.

Clinical Competence↗
PubMed2006

The medical education of United States citizens who train abroad.

BACKGROUND: Graduates of international medical schools (IMGs) make up approximately one quarter of the physician workforce in the United States. Among IMGs are a number of US citizens (USIMGs) who take graduate training positions and ultimately practice in the United States. Compared with graduates of US medical schools (USMGs), relatively little is known about the undergraduate educational experiences of these US citizens. The objective of this study was to identify the schools that produce the most USIMGs and to describe the educational experiences and examination performance of graduates of these schools. METHODS: The 10 largest schools were identified based on the number of USIMGs who were certified by the Educational Commission for Foreign Medical Graduates between 2001 and 2004. Information about the institutions was collected from the International Medical Education Directory; educational experience information was gathered from 100 randomly selected transcripts and from a survey that was completed by 418 graduates of the schools. These data were compared with information about USMGs from the curriculum management and information tool and the graduation questionnaire of the Association of American Medical Colleges. Performance on steps 1 and 2CK of the United States Medical Licensing Examination were also analyzed. RESULTS: Some differences existed between USIMGs and USMGs in the required clinical clerkships, the sites where educational experiences occurred, and the special topics that were covered. USMGs and non-US citizen IMGs had better examination scores than USIMGs. CONCLUSION: There are many similarities; however, there are some noteworthy differences between the educational experiences of USIMGs and USMGs. Further work is needed to better understand the educational experiences of USIMGs, particularly in the clinical clerkships.

Clinical Clerkship↗
PubMed2006

Setting school-level outcome standards.

BACKGROUND: To establish international standards for medical schools, an appropriate panel of experts must decide on performance standards. A pilot test of such standards was set in the context of a multidimensional (multiple-choice question examination, objective structured clinical examination, faculty observation) examination at 8 leading schools in China. METHODS: A group of 16 medical education leaders from a broad array of countries met over a 3-day period. These individuals considered competency domains, examination items, and the percentage of students who could fall below a cut-off score if the school was still to be considered as meeting competencies. This 2-step process started with a discussion of the borderline school and the relative difficulty of a borderline school in achieving acceptable standards in a given competency domain. Committee members then estimated the percentage of students falling below the standard that is tolerable at a borderline school and were allowed to revise their ratings after viewing pilot data. RESULTS: Tolerable failure rates ranged from 10% to 26% across competency domains and examination types. As with other standard-setting exercises, standard deviations from initial to final estimates of the tolerable failure rates fell, but the cut-off scores did not change significantly. Final, but not initial cut-off scores were correlated with student failure rates (r = 0.59, P = 0.03). DISCUSSION: This paper describes a method to set school-level outcome standards at an international level based on prior established standard-setting methods. Further refinement of this process and validation using other examinations in other countries will be needed to achieve accurate international standards.

China↗
PubMed2005

Use of SPRAT for peer review of paediatricians in training.

OBJECTIVE: To determine whether a multisource feedback questionnaire, SPRAT (Sheffield peer review assessment tool), is a feasible and reliable assessment method to inform the record of in-training assessment for paediatric senior house officers and specialist registrars. DESIGN: Trainees' clinical performance was evaluated using SPRAT sent to clinical colleagues of their choosing. Responses were analysed to determine variables that affected ratings and their measurement characteristics. SETTING: Three tertiary hospitals and five secondary hospitals across a UK deanery. PARTICIPANTS: 112 paediatric senior house officers and middle grades. MAIN OUTCOME MEASURES: 95% confidence intervals for mean ratings; linear and hierarchical regression to explore potential biasing factors; time needed for the process per doctor. RESULTS: 20 middle grades and 92 senior house officers were assessed using SPRAT to inform their record of in-training assessment; 921/1120 (82%) of their proposed raters completed a SPRAT form. As a group, specialist registrars (mean 5.22, SD 0.34) scored significantly higher (t = - 4.765) than did senior house officers (mean 4.81, SD 0.35) (P < 0.001). The grade of the doctor accounted for 7.6% of the variation in the mean ratings. The hierarchical regression showed that only 3.4% of the variation in the means could be additionally attributed to three main factors (occupation of rater, length of working relationship, and environment in which the relationship took place) when the doctor's grade was controlled for (significant F change < 0.001). 93 (83%) of the doctors in this study would have needed only four raters to achieve a reliable score if the intent was to determine if they were satisfactory. The mean time taken to complete the questionnaire by a rater was six minutes. Just over an hour of administrative time is needed for each doctor. CONCLUSIONS: SPRAT seems to be a valid way of assessing large numbers of doctors to support quality assurance procedures for training programmes. The feedback from SPRAT can also be used to inform personal development planning and focus quality improvements.

Child↗
PubMed2005

The FAIMER Institute: creating international networks of medical educators.

One of the many products of Miriam's career was an international network of medical educators. She knew we would learn from each other and gain access to the perspectives, resources, and experiences that such a community brings. More importantly, Miriam understood the need for shared values, support, encouragement, and a sense of global citizenship that can come only from an international network of colleagues and friends. The FAIMER Institute, described in this paper, is a formalization and extension of Miriam's work and we hope it will be as successful as she was. The FAIMER Institute is a two-year fellowship program designed for medical school faculty from developing countries who have the potential to improve medical education. The first year consists of two residential sessions in the US before and after an intersession of distance learning and implementation of an educational project at the participant's home institution. The second year, completed from the Fellow's home country, involves co-mentoring a new Fellow and active engagement in the Institute's Internet discussion group. The program is designed to teach educational methods and leadership skills, as well as to develop strong professional bonds with other medical educators around the world. Preliminary data concerning the efficacy of the program have been encouraging. Fellows' perceptions of their knowledge, skills and attitudes show significant improvement. These self-assessments are supported by the outcomes, which indicate considerable scholarship as well as academic and administrative advancement. There have also been changes in the nature of the professional networks of these medical educators, which enhance their ability to undertake more complex projects in an innovative fashion. Finally, plans for the future focus on conducting regional Institutes in South Asia, sub-Saharan Africa and South America with the goal of fostering the creation of networks of medical educators. The current model will be modified to meet local needs, FAIMER will coordinate its other programs to support development in the regions, and partners will be sought to support and expand this effort.

Curriculum↗
PubMed2003

Construct validity of the miniclinical evaluation exercise (miniCEX).

PURPOSE: To investigate the construct validity of the miniclinical evaluation exercise (miniCEX). METHOD: Forty faculty participants from 16 internal medicine residency programs enrolled in a randomized, controlled trial of faculty development. Using a standard nine-point miniCEX rating form, participants watched and rated performances of standardized residents on nine scripted clinical videotapes depicting three levels of performance (unsatisfactory, marginal/satisfactory, and high satisfactory/superior). The nine-point rating scale was 1-3 = unsatisfactory, 4-6 = marginal/satisfactory, and 7-9 = superior. The performances were rated for three clinical skills, history taking, physical examination, and counseling. RESULTS: For each of the three clinical skills, the faculty participants were able to successfully discriminate among the three levels of performance using the miniCEX scale. Differences among ratings of the three performance levels were statistically significant; however, the range in ratings among the participants for each videotape was wide. CONCLUSION: The authors believe this to be the first study to document the construct validity of the miniCEX. Although the miniCEX appears to have reliability and construct validity, further research is needed to improve individual faculty observation skills and reduce interrater variability.

Adult↗
PubMed2002

Association between licensure examination scores and practice in primary care.

CONTEXT: Standards for licensure are designed to provide assurance to the public of a physician's competence to practice. However, there has been little assessment of the relationship between examination scores and subsequent practice performance. OBJECTIVE: To determine if there is a sustained relationship between certification examination scores and practice performance and if licensing examinations taken at the end of medical school are predictive of future practice in primary care. DESIGN, SETTING, AND PARTICIPANTS: A total of 912 family physicians, who passed the Québec family medicine certification examination (QLEX) between 1990 and 1993 and entered practice. Linked databases were used to assess physicians' practice performance for 3.4 million patients in the universal health care system in Québec, Canada. Patients were seen during the follow-up period for the first 4 years (1993 cohort of physicians) to 7 years (1990 cohort of physicians) of practice from July 1 of the certification examination to December 31, 1996. MAIN OUTCOME MEASURES: Mammography screening rate, continuity of care index, disease-specific and symptom-relief prescribing rate, contraindicated prescribing rate, and consultation rate. RESULTS: Physicians achieving higher scores on both examinations had higher rates (rate increase per SD increase in score per 1000 persons per year) of mammography screening (beta for QLEX, 16.8 [95% confidence interval [CI], 8.7-24.9]; beta for Medical Council of Canada Qualifying Examination [MCCQE], 17.4 [95% CI, 10.6-24.1]) and consultation (beta for QLEX, 4.9 [95% CI, 2.1-7.8]; beta for MCCQE, 2.9 [95% CI, 0.4-5.4]). Higher subscores in diagnosis were predictive of higher rates in the difference between disease-specific and symptom-relief prescribing (beta for QLEX, 3.9 [95% CI, 0.9-7.0]; beta for MCCQE, 3.8 [95% CI, 0.3-7.3]). Higher scores of drug knowledge were predictive of a lower rate (relative risk per SD increase in score) of contraindicated prescribing for MCCQE (relative risk, 0.88; 95% CI, 0.77-1.00). Relationships between examination scores and practice performance were sustained through the first 4 to 7 years in practice. CONCLUSION: Scores achieved on certification examinations and licensure examinations taken at the end of medical school show a sustained relationship, over 4 to 7 years, with indices of preventive care and acute and chronic disease management in primary care practice.

Certification↗