Large-scale high-stakes testing with an OSCE: report from the Medical Council of Canada.
Explore the source record for details and available documents.
Canada↗
Biomedical subjects
R K Reznick
Publications and source records attributed to R K Reznick.
The effect of early performance on examiners' marking patterns during an oral examination.
Explore the source record for details and available documents.
Achievement↗
Who should rate candidates in an objective structured clinical examination?
PURPOSE: To determine who is the better rater of history taking in an objective structured clinical examination (OSCE): a physician or a standardized patient (SP). METHOD: During the 1991 pilot administration of an OSCE for the Medical Council of Canada's qualifying examination, five history-taking stations were videotaped. Candidates at these stations were scored by three raters: a physician (MD), an SP observer (SPO), and an SP rating from recall (SPR). To determine the validity of each rater's scores, these scores were compared with a "gold standard", which was the average of videotape ratings by three physicians, each scoring independently. Analysis included both correlations with the standard and a repeated-measures analysis of variance (ANOVA) comparing raters' mean scores on each station with mean scores of the gold standard. RESULTS: Ninety-one videotapes were scored by the "gold-standard" physicians. Correlations with the standard showed no clear preference for MD, SPO, or SPR raters. ANOVAs revealed significant differences from the standard on three stations for the SPR, two stations for the SPO, and one stations for the MD. CONCLUSIONS: An MD rater is less likely to differ from a standard established by a consensus of MD ratings than are SP raters rating from recall. If an MD cannot be used, an SP observer is preferable to an SP rating from recall.
Analysis of Variance↗
The efficacy of a computer-assisted preoperative tutorial for clinical clerks.
OBJECTIVE: To assess the effectiveness of computer-assisted preoperative tutorials on human anatomy in improving the operating-room learning experience for clinical clerks. DESIGN: Crossover trials with immediate assessment and approximately 1-week delay between trials. SETTING: General surgery operating room of a university teaching hospital. SUBJECTS: Eight 4th-year clinical clerks on a single 8-week surgical rotation. MAIN OUTCOME MEASURES: The senior surgeon's assessment of the clerk's knowledge and understanding of the operation, according to a six-item questionnaire, and the clerk's own assessment of the experience in the operating room, according to an eight-item questionnaire. RESULTS: Surgeons rated the clerks' performances in the operating room as better when they had received the preoperative tutorial (mean [and standard deviation], 3.7 [0.4]) than when they had not received the preoperative tutorial (3.0 [0.3]). The difference (0.7 [0.6]) was statistically significant (t7 = 3.3, p < 0.01). Similarly, clerks rated their own experience more positively when they had received the tutorial (4.0 [0.2]) than when they had not (3.1 [0.3]), with the difference (0.9 [0.5]) being statistically significant (t7 = 4.9, p < 0.001). CONCLUSIONS: Short, preoperative, computer-assisted tutorials on human anatomy can have a positive impact on the clerk's level of knowledge and confidence in the operating room. Further research is warranted into the extent to which students spontaneously make use of these tutorials.
Anatomy↗
Description and evaluation of the Surgeons as Educators course.
BACKGROUND: Since 1993, the American College of Surgeons has sponsored an annual 6-day course entitled the Surgeons as Educators. The course was designed to provide academic surgeons with the knowledge and skills necessary to enhance the surgical education curriculum, teaching strategies, educational program administration, and performance evaluation. This article describes the development, implementation, and effect of the course on the classes graduating in 1993 and 1994. STUDY DESIGN: The effect of the course was studied by using a longitudinal survey approach. A survey was mailed to participants 3 to 6 months after they completed the course. Graduates were asked to describe any education-related actions taken attributable to attending the Surgeons as Educators course. The quality of course content and presentations were evaluated by using end-of-course evaluation forms and daily feedback forms and by an external reviewer. RESULTS: Within 6 months of returning from the course, more than one half of the graduates initiated actions related to curriculum development, teaching strategies, or educational administration. One third or more of the graduates modified their performance and program evaluation systems. Using a five-point scale, ratings of the course content ranged from 3.78 to 4.64 for "value of topic" and from 3.77 to 4.76 for "quality of presentation." Items evaluated by the graduates on the end-of-course evaluation forms ranged from 7.8 to 8.7 on a nine-point scale. CONCLUSIONS: The Surgeons as Educators course offered an opportunity for participants to interact among themselves and with course faculty about educational issues and to practice teaching skills. The course was highly rated for educational quality and value. The retreat environment and the length of the program helped attendees become immersed during this "protected time" to analyze strengths and weaknesses of their programs and devise achievable plans to improve their abilities as educators and the effectiveness of their programs.
Clinical Competence↗
Reliability and construct validity of a structured technical skills assessment form.
Current methods of evaluating technical competence of surgical residents are subjective and potentially unreliable. This study assesses the reliability and construct validity of a new format for the assessment of technical ability, the two part Structured Technical Skills Assessment Form (STSAF). Part I, which is completed while an operation is proceeding consists of approximately 120 essential components of the procedure. Part II, completed at the end of the operation, is a 10-point global rating form. Forty-one operations were evaluated using the STSAF, with multiple observers present at 26. Inter-rater reliability of both Parts I and II were high (.78 and .73, respectively). Statistically significant differences were noted between senior-resident and junior-resident performances, suggesting construct validity. The incorporation of structured guidelines to the assessment of technical skill leads to high inter-rater reliability and construct validity, which ultimately may result in improved and reproducible evaluations of surgical trainees.
Cholecystectomy↗
Meta-analysis of highly selective vagotomy versus truncal vagotomy and pyloroplasty in the surgical treatment of uncomplicated duodenal ulcer.
OBJECTIVE: To compare critically by meta-analysis the postoperative outcomes after truncal vagotomy and pyloroplasty (TVP) and after highly selective vagotomy (HSV). DATA SOURCES: A search was conducted on MEDLINE with the keywords "peptic ulcer" and "vagotomy". Bibliographics of the articles retrieved and review articles pertaining to the subject were scanned further. STUDY SELECTION: Included were only published prospective, randomized controlled trials comparing TVP and HSV in the English-language literature up to April 1991. Twelve studies were found. DATA EXTRACTION: Two authors, working independently, abstracted raw data on mortality, recurrence, dumping, diarrhea and postoperative Visick grading. DATA SYNTHESIS: A ratio greater than 1.0 indicates a higher proportion occurring with TVP and less than 1.0 a higher proportion with HSV. [Table: See Text] CONCLUSION: The results suggest that the likelihood of adverse long-term sequelae is higher with TVP whereas the likelihood of recurrence is higher with HSV.
Confidence Intervals↗
Teaching and testing technical skills.
Teaching technical skills is one of the most important tasks of a surgeon. This article discusses current issues in teaching and testing technical skills. For the most part, the level of technical skills cannot be predicted before a surgical resident starts a program. Different methods of teaching technical skills are reviewed (in and out of the operating room). For optimal effectiveness in teaching residents, we must apply principles of adult learning to the surgical domain. A methodologic framework for skill acquisition, adapted from the educational psychology literature, is discussed. Five methods of assessing technical skills are presented. Structuring the assessment process has resulted in higher levels of reliability and improved validity.
Clinical Competence↗
Guidelines for estimating the real cost of an objective structured clinical examination.
A major impediment to the use of the objective structured clinical examination (OSCE) is that it is a labor-intensive and costly form of assessment. The cost of an OSCE is highly dependent on the particular model used, the extent to which hidden costs are reported, and the purpose of the examination. The authors detail hypothetical costs of running a four-hour OSCE for 120 medical students at one medical school. Costs are reported for four phases of this process: development, production, administration, and post-examination reporting and analysis. Costs are reported at two ends of the spectrum: the high end, where it is assumed that little is paid for by the institution and that faculty receive honoraria for work put into the examination; and the low end, where it is assumed that the sponsoring institution defrays basic costs and that faculty do not receive honoraria for their participation. The total costs reported for a first-time examination were $104,400 and $59,460 (Canadian dollars) at the high and low ends, respectively. These translate to per-student costs of $870 and $496. The cost of running an OSCE is high. However, the OSCE is uniquely capable of assessing many fundamental clinical skills that are presently not being assessed in a rigorous way in most medical schools.
Canada↗
An objective structured clinical examination for the licentiate of the Medical Council of Canada: from research to reality.
Explore the source record for details and available documents.
Analysis of Variance↗
A technical-skills course for 1st-year residents in general surgery: a descriptive study.
The proper teaching of operative skills to surgical residents is increasingly constrained by operative time, complex procedures and medicolegal concerns. A technical-skills program was developed and introduced over a 3-year period to 28 1st-year residents in general surgery. To introduce the residents to the principles of surgical techniques in a simulated environment outside the operating room, the program consisted of a combination of two didactic sessions and six "wet labs" taking 3 to 4 hours per week for 8 weeks between January and March each year. The didactic sessions included instruction on suture material and the use of stapling devices; the "wet lab" used a "hands-on" approach. Educational objectives in the "wet lab" included instruction on preparation of the patient and draping, aseptic technique, principles of bowel anastomosis, incisions, instrument use and handling, principles of hemostasis, intraoperative surgical emergencies, surgical assisting and overall conduct in the operating room. The residents' surgical technique and skills improved over the course period. The overall value, teaching and understanding of surgical principles were rated highly. Problems cited during resident feedback were the use of live animals and insufficient time to practise. The efficacy of a surgical-skills program has been demonstrated, but its effectiveness requires further evaluation.
Canada↗
The structured oral examination as a method for assessing surgical residents.
The purpose of this study was to determine the overall reliability, inter-rater reliability, and criterion validity of the structured oral examination (SOE) for assessing surgical residents. An SOE consisting of four predetermined clinically oriented scenarios was administered to 23 second postgraduate year surgical residents. Each scenario had five to six questions, each with a specific marking scheme. Candidates were assessed by two examiners and scores were derived independently. Overall reliability (Cronbach's alpha) was 0.75. Inter-rater reliability was significant for each pair of examiners and each question (r = 0.78 to 0.91: p less than 0.0001). Criterion validity was measured by correlating SOE scores with multiple-choice examination (MCQ) and objective structured clinical examination (OSCE) scores. Correlations between the SOE and MCQ and OSCE were significant and fell into the moderate range (0.48 to 0.51). The results of this study show that the SOE is useful in the assessment of clinical knowledge and problem-solving abilities of the surgical resident. Overall and inter-rater reliabilities achieved exceed those of traditional oral examination formats.
Adult↗
B.F. Skinner in the operating room.
Explore the source record for details and available documents.
History, 20th Century↗
Reliability and validity of the objective structured clinical examination in assessing surgical residents.
The purpose of this research was to assess reliability and construct validity of the objective structured clinical examination (OSCE) for evaluating the clinical skills of surgical residents. Reliability refers to precision of the examination and construct validity to the degree to which the examination can discriminate between different levels of training. Twenty-seven second postgraduate year surgical residents took a 38-station OSCE representing seven surgical specialties and that tested history-taking, physical examination, problem-solving, technical skills, and attitudes. A couplet methodology was used wherein a patient encounter was followed by written questions aimed at testing problem-solving and patient management capabilities. Thirty-six standardized patients were trained and 36 surgeons served as examiners marking from structured checklists. Overall reliability, Cronbach's alpha, was 0.89. Construct validity was examined by comparing the scores of the residents with those of a group of graduates of foreign medical schools applying for a "pre-internship" program. For 17 of 19 stations that both groups took, the residents performed significantly better (p less than 0.01). Individual station validity was significant for 32 of 38 stations (r = 0.36 to 0.82, p less than 0.05). The examinations took 3.83 hours at a cost of $5,293 (Canadian dollars). The OSCE has been shown to be a reliable method of assessing clinical skills of surgical residents, construct validity has been established, and inter-item validity confirmed. Reliabilities achieved exceed those traditionally required for both acceptance and promotion decisions.
Analysis of Variance↗
The surgeon simulator.
A new technique of small group surgical teaching has been developed wherein the surgeon takes on the role of the patient. This technique, which incorporates extensive and immediate formative evaluation, has all the advantages of simulation techniques while avoiding the major problems of training, scheduling and cost. This method has been used to teach a wide variety of surgical disease processes with the major emphasis being teaching patient management strategies. Sixty-one medical students have been taught using this method and they have found it superior to conventional seminar teaching, particularly in the domains of problem solving, patient management strategies and thought provocation.
Education, Medical, Undergraduate↗
Colitis cystica profunda. Review of the literature.
Colitis cystica profunda is a rare benign disorder of the colon and rectum. The disease is important clinically in that it often mimics malignant processes. The clinical features of 144 cases of colitis cystica profunda reported in the literature are reviewed. Controversies surrounding the cause and pathogenesis of the disease are discussed.
Colitis↗
Survival analysis: a practical approach.
Survival analysis is a statistical method used to calculate the probability of an event such as death or relapse of disease occurring in a patient over time. Survival analysis is important in the interpretation of clinical research, and is frequently encountered in the colorectal literature. In this article, the terminology used in survival analysis is explained, specific examples are presented, and common methods of calculation demonstrated.
Humans↗
Reliability of different grading systems used in evaluating surgical students.
Inter-rater agreement in assigning grades using five different grading systems was determined. The performance of 16 students in a surgery clerkship was rated by 21 faculty raters using a pass-fail grading system, a pass-fail-honors system, a letter grade system, a number grade scale from 1 to 10, and a number grade scale from 1 to 100. Inter-rater agreement coefficients were used to assess relative and absolute reliabilities, respectively. Both the letter grade and 1 to 10 number grade systems provided good discrimination, had high to moderate reliability, and required only five raters to achieve a mean rating with the commonly recommended reliability of 0.80. Using the letter grade system, however, a majority of raters agreed on a specific grade assignment for 14 of 16 students, in contrast to the 1 to 10 scale, for which this was true for only 4 of 16 students. The results of this reliability study favor the use of a letter grading system.
Clinical Clerkship↗