PubMed Health⌕ Search

Biomedical subjects

John R Boulet

Publications and source records attributed to John R Boulet.

9 recordsLinked to original sources
PubMed2003

Quality assurance methods for performance-based assessments.

Performance assessments are subject to many potential error sources. For performance-based assessments, including standardized patient (SP) examinations, these error sources, if left unchecked, can compromise the validity and reliability of scores. Quality assurance (QA) measures, both quantitative and qualitative, can be used to ensure that candidate scores are accurate and reasonably free from measurement error. The purpose of this paper is to outline several QA strategies that can be used to identify potential content- and score-related problems with SP assessments. These approaches include case analyses and various comparisons of primary and observer scores. Specific examples from the ECFMG Clinical Skills Assessment (CSA) are used to educate the reader concerning appropriate statistical methods and legitimate data interpretations. The results presented in this investigation highlight the need for well-defined training regimes, regular feedback to those involved in rating/scoring performances, and detailed statistical analyses of all scores.

Clinical Competence↗
PubMed2003

Evaluating the spoken English proficiency of international medical graduates: detecting threats to the validity of standardised patient ratings.

PURPOSE: To investigate potential threats to the validity of the spoken English proficiency ratings provided by standardised patients (SPs) in high-stakes clinical skills examinations. METHOD: Spoken English ratings from 43 327 patient encounters were studied. These involved over 5000 candidates, 40% of whom were female and 33% of whom self-reported English to be their native language. Over 100 SPs were involved in the study, 51% of whom were female and 90% of whom were native English speakers. Possible performance differences in English ratings were studied as a function of candidate and SP gender, and as a function of candidate and SP native language (English versus all other languages). RESULTS: No significant candidate by SP gender effect was detected. There were no meaningful differences in mean English ratings as a function of SP or candidate gender. Likewise, English ratings did not vary as a function of either candidate or SP native language. While candidate mean English ratings were not associated with the native language of the SP, native English-speaking candidates did achieve significantly higher ratings. DISCUSSION: The lack of significant interaction between candidate and SP gender, and candidate and SP native language, suggests that the SPs provided unbiased English ratings. These results, combined with the expected higher English ratings given to candidates with English-speaking backgrounds, provides additional evidence to support the validity and fairness of spoken English proficiency ratings provided by standardised patients.

Adult↗
PubMed2003

Setting defensible performance standards on OSCEs and standardized patient examinations.

Recently, standardized patient assessments and objective structured clinical examinations have been used for high-stakes certification and licensure decisions. In these testing situations, it is important that the assessments are standardized, the scores are accurate and reliable, and the resulting decisions regarding competence ar equitable and defensible. For the decisions to be valid, justifiable standards, or cut-scores, must beset. Unfortunately, unlike the body of research specifically dedicated to multiple-choice examinations, relatively little research has been conducted on standard-setting methods appropriate for use with performance-based assessments. The purpose of this article is to provide the reader with some guidance on how to set defensible standards on performance assessments, especially those that utilize standardized patients in simulated medical encounters. Various methods are discussed and contrasted, highlighting the relevant strengths and weaknesses. In addition, based on the prevailing literature and research, ideas for future studies and potential augmentations to current performance-based standard setting protocols are advanced.

Certification↗
PubMed2003

Using National Medical Care Survey data to validate examination content on a performance-based clinical skills assessment for osteopathic physicians.

Patient characteristics, chief complaints, and diagnoses can be used to specify the examination content for performance-based assessments of clinical skills. The purpose of this investigation was to explore osteopathic and allopathic medical practice patterns and to provide summary statistics that can be used to delimit potential assessment content areas for a clinical skills assessment targeted at osteopathic physicians. Analyses of the National Ambulatory Medical Care Survey indicated that the types of patients seen by osteopathic and allopathic physicians in office-based settings are somewhat different. Furthermore, the reasons that patients seek care, and accompanying diagnostic outcomes, can vary by physician type. These differences suggest that from a content perspective, a performance-based clinical skills evaluation targeted at osteopathic physicians should be characteristically different from one designed for allopathic physicians.

Clinical Competence↗
PubMed2002

The changing pool of international medical graduates seeking certification training in US graduate medical education programs.

International medical graduates (IMGs) consistently represent approximately one fourth of both the physician workforce and the graduate medical education (GME) population of the United States. To enter into accredited US GME programs, IMGs must be certified by the Educational Commission for Foreign Medical Graduates (ECFMG). Changes in the number and characteristics of those seeking certification directly affect the GME population and the future physician workforce in the United States. In July 1998, in response to concerns that IMGs might be lacking in basic clinical skills (eg, history taking, physical examinations, communicating with patients in spoken English), the ECFMG initiated a requirement that IMGs pass a clinical skills assessment (CSA) to achieve ECFMG certification. In this study we examined the pool of IMGs seeking certification, using databases reporting on all individuals beginning the certification process from 1995 through 2001. For this period, we found that the number of IMG candidates taking the Step 1 examination decreased by 45.5% (36,983 vs 16,828), and the number of IMGs registered to take Step 2 decreased by 38.1% (31 751 vs 12 122). The number of ECFMG certificates issued annually decreased, from a range of 9000 to 12,000 (1995-1998) to fewer than 6000 (1999-2001). Although the number of IMGs annually seeking and receiving certification has decreased, the quality of the applicants appears to have improved and the number of IMGs certified annually continues to adequately fill GME positions not taken by US medical graduates.

Certification↗
PubMed2002

Assessing the comparability of standardized patient and physician evaluations of clinical skills.

Accumulating evidence to defend decisions based on scores from evaluations is an ongoing process. The purpose of this investigation was to gather additional data to support the validity of inferences made from scores on the Educational Commission for Foreign Medical Graduates' Clinical Skills Assessment (CSA). This was accomplished by contrasting CSA candidate scores, and pass/fail decisions, with those obtained from the American Board of Internal Medicine's Mini-CEX (Clinical Evaluation Exercise). Data gathering performance based on the number of unweighted history taking and physical examination checklist items adequately predicted the global ratings provided by physician observers. CSA ratings of doctor-patient communication skills correlated with mini-CEX ratings of like constructs, indicating that physician observers, using mini-CEX rating scales, are able to make realistic assessments of interpersonal skills. These results provide evidence of the convergent validity of CSA scores.

Certification↗
PubMed2002

Attrition rates of residents in postgraduate training programs.

BACKGROUND: One goal of program directors is to attract and retain high-quality residents. It is therefore important to study attrition of residents to determine specific cohorts that may be less likely to complete residency training. PURPOSE: The purpose of this study was to investigate attrition rates in targeted programs. METHOD: Surveys were sent to 540 program directors whose programs contained at least 25% international medical graduates. The program directors reported both voluntary attrition and termination rates. RESULTS: Overall attrition rates for international medical graduates and U.S. medical graduates were similar, although U.S. medical graduates were terminated at a slightly lower rate than international medical graduates (2.5% vs. 3.6%) and were somewhat more likely to transfer to other programs (9.9% and 8.9%). CONCLUSIONS: Attrition from these programs remains a small but persistent issue. In addition, the differences in termination and transfer rates for the 2 years of the study were greater than the differences in rates between U.S. and international medical graduates.

Data Collection↗
PubMed2002

State of the art in standardized patient programs: a survey of osteopathic medical schools.

In 2001, the authors sent a survey to all 19 schools of osteopathic medicine requesting information about how standardized patients are used for clinical teaching and skills assessment. The purpose of this survey was to identify key people involved in standardized patient programs (SPPs), to gather information on the resources allocated by osteopathic medical schools, to determine the "state of the art" of standardized patient teaching and assessment in osteopathic medical schools, and to assess which SPPs would be available to assist in the development of the new component of the National Board of Osteopathic Medical Examiners licensing examination, the Comprehensive Osteopathic Medical Licensing Examination Performance Evaluation. The survey results revealed that 12 osteopathic medical schools have SPPs that teach and assess clinical skills, including history taking; physical examination (complete and focused); genital, rectal, and breast examinations; physician-patient communication, osteopathic manipulative medicine or osteopathic manipulative treatment. Details of the 12 schools' SPPs are highlighted.

Clinical Competence↗