Biomedical subjects

H MacRae

Publications and source records attributed to H MacRae.

10 recordsLinked to original sources

PubMed2001

Exporting a technical skills evaluation technology to other sites.

BACKGROUND: The Objective Structured Assessment of Technical Skill (OSATS) is a multistation performance-based examination that assesses the technical skills of surgery residents. This study explores the implementation issues involved in remote administration of the OSATS focusing on feasibility and the psychometric properties of the examination. METHODS: An eight-station OSATS was administered to surgical residents in Los Angeles and Chicago. The University of Toronto and the local institutions shared responsibility for organization and administration of the examination. RESULTS: There was good reliability for both the checklist (alpha = 0.68 for LA, 0.73 for Chicago) and global rating forms (alpha = 0.82 for both sites). Both iterations also showed evidence of construct validity, with a significant effect of training year for the checklist and global rating forms at both sites (analysis of variance: F = 8.66 to 19.93, P <0.01). Despite some challenges, the model of central organization and peripheral delivery was effective for the administration of the examinations. CONCLUSIONS: Two iterations of the OSATS at remote sites demonstrated psychometric properties that are highly consistent with previously reported data suggesting that the examination is portable. Both faculty and residents indicated satisfaction with the examination experience. A model of central administration with peripheral delivery was feasible and effective.

California↗

PubMed2001

Structured assessment of minor surgical skills (SAMSS) for family medicine residents.

PURPOSE: To develop a valid and reliable examination to assess the technical proficiency of family medicine residents' performance of minor surgical office procedures. METHOD: A multi-station OSCE-style examination using bench-model simulations of minor surgical procedures was developed. Participants were a randomly selected group of 33 family medicine residents (PGY-1 = 16, PGY-2 = 17) and 14 senior surgical residents who functioned as a validation group. Examiners were qualified surgeons and family physicians who used both checklists and global rating scales to score the participants' performances. RESULTS: When family medicine residents were evaluated by family physicians, interstation reliabilities were .29 for checklists and .42 for global ratings. When family medicine residents were evaluated by surgeons, the reliabilities were .53 for checklists and .75 for global ratings. Interrater reliability, measured as a correlation for total examination scores, was .97. Mean scores on the examination were 60%, 64%, and 87% for PGY-1 family medicine, PGY-2 family medicine, and surgery residents, respectively. The difference in scores between family medicine and surgery residents was significant (p < .001), providing evidence of construct validity. CONCLUSION: A new examination developed for assessing family medicine residents' skills with minor surgical office procedures is reliable and has evidence for construct validity. The examination has low reliability when family physicians serve as examiners, but moderate reliability when surgeons are the evaluators.

Ambulatory Surgical Procedures↗

PubMed2000

Coping behavior and social support contribute independently to quality of life after surgery for inflammatory bowel disease.

PURPOSE: The purpose of this study was to examine the association between coping behavior at the time of surgery and inflammatory bowel disease-related quality of life after surgery. We also investigated the relationship between perceived social support and both coping style and postsurgical quality of life. Finally, the value of the Medical Outcomes Study Social Support Scale for preoperative screening was assessed. METHODS: Eighty-six subjects who had surgery during a 12-month period completed the Inflammatory Bowel Disease Questionnaire, the Ways of Coping, a measure of inflammatory bowel disease symptom severity, and the Medical Outcomes Study Social Support Scale. Analysis of variance was used to test an association between Ways of Coping score and membership in a high quality of life (Inflammatory Bowel Disease Questionnaire > mean) or low quality of life (Inflammatory Bowel Disease Questionnaire < mean) cohort. Comparison of group means between the high quality of life and low quality of life cohorts identified Ways of Coping behavior scales that differed between the high quality of life and low quality of life cohorts. Stepwise linear regression analysis was then used to determine the independent contribution of 1) current inflammatory bowel disease symptoms, 2) current perceived social support, and 3) identified coping behaviors (self-control, self-blame, and escape, summed as a single index named "maladaptive coping") to postsurgical quality of life. The sensitivity, specificity, and negative predictive value of the Medical Outcomes Study Social Support Scale were assessed. RESULTS: The lower quality of life group distinguished itself by more frequent use of maladaptive coping. Regression analysis revealed that current inflammatory bowel disease-related symptoms, current perceived social support, and maladaptive coping behaviors at the time of surgery each made a highly significant independent contribution to postsurgical quality of life. The sensitivity of the Medical Outcomes Study Social Support Scale in identifying patients with poor postsurgical quality of life was 81 percent, and the specificity was 77 percent. The negative predictive value was 93 percent. CONCLUSIONS: Three coping behaviors which seem to be maladaptive (self-control, self-blame, and escape) are associated with lower quality of life after surgery for inflammatory bowel disease. These coping behaviors make a contribution to postsurgical quality of life independent of the negative effect on quality of life of inflammatory bowel disease symptoms. Perceived social support is a third factor that makes an independent contribution to postsurgical quality of life. The Medical Outcomes Study Social Support Scale has properties associated with an effective screening tool and merits further investigation as an instrument to screen presurgically for individuals at higher risk of poor subjective outcome of inflammatory bowel disease surgery.

Adaptation, Psychological↗

PubMed2000

A comprehensive examination for senior surgical residents.

BACKGROUND: Two complimentary examinations designed to comprehensively assess competence for surgical practice have been developed. The Objective Structured Assessment of Technical Skill (OSATS) evaluates a resident's operative skill, and the Patient Assessment and Management Examination (PAME) evaluates clinical management skills. METHODS: Twenty-four postgraduate year (PGY)-4 and PGY-5 general surgery residents from four training programs were examined. Each examination had eight stations, with a total of 6 hours of testing time. RESULTS: Interstation reliability for the OSATS was 0.64, for the PAME was 0.71, and for the total test was 0. 74. Examination scores discriminated between PGY-4 and PGY-5 residents for the OSATS (t = 4.39, P <.01), the PAME (t = 1.86, P <. 05), and the total examination (t = 3.90, P <.01). Year of training accounted for 40%, of the variance of scores. CONCLUSIONS: This comprehensive examination is a reliable and valid method of assessing critical skills in senior surgical residents and may be useful for the formal assessment of readiness for practice.

Clinical Competence↗

PubMed2000

Using operative outcome to assess technical skill.

BACKGROUND: This study examined whether an operative product and time to completion could serve as measures of technical skill. METHODS: Nine final-year (PGY5) and 11 penultimate-year (PGY4) general surgery residents participated in a 6-station bench model examination. Time to completion was recorded. Twelve faculty surgeons (2 per station) evaluated the quality of the final product using a 5-point scale. RESULTS: The mean interrater reliability was 0. 59 for product quality. Interstation reliability was 0.59 for analysis of the final product and 0.72 for time to completion. There was 63% and 78% agreement between attendings' ratings and product quality and time scores respectively. PGY5s' mean product quality score was 4.14 +/- 0.26, compared with 3.82 +/- 0.33 for PGY4s (P < 0.05). PGY5s' mean time was 110 +/- 19 minutes compared with PGY4s' 132 +/- 15 (P < 0.05). CONCLUSIONS: Analysis of the operative end product and time to completion offer efficient alternatives to on-line examiner scoring for bench model examinations of technical competence.

Benchmarking↗

PubMed2000

Use of community resources before inflammatory bowel disease surgery is associated with postsurgical quality of life.

BACKGROUND: Research in chronic illness shows that community resources can have a lasting influence on the course of the illness; however, little research has been done to evaluate the community agencies that specifically address the needs of inflammatory bowel disease (IBD) patients. OBJECTIVES: To survey awareness of community agency resources among patients who have surgery for IBD, and to analyze the association between using these resources and qualitative postsurgical outcomes. SUBJECTS AND METHODS: Ninety-two subjects who had surgery over a 12-month period completed, in full, the Inflammatory Bowel Disease Questionnaire (IBDQ), and a self-report instrument used to probe awareness and use of local community resources. Community resources were divided into two groups: those involving primarily social and educational participation ('social/ educational') and those involving some individualized attention, usually from a professional or trained lay facilitator ('professional/individual'). The contribution of presurgical participation in each type of resource to postsurgical quality of life was tested using ANOVA, with IBDQ score as the dependent variable. The ANOVA was repeated with postsurgical disease activity as a covariable. IBDQ subscale scores were compared between groups that were found to differ in the ANOVA. RESULTS: Almost all subjects were aware of at least one available resource. Participation in resources before surgery was variable, but 50% of the sample participated in at least one social/educational resource, and 46.9% participated in at least one professional/individual support. For the 92 subjects who completed both the IBDQ and the survey of resources, ANOVA revealed a main effect of professional/individual resource use on postsurgical quality of life but no main effect of social/educational resources and no interaction. DISCUSSION: The association between presurgical participation in professional or individualized community resources and better subjective outcome of IBD surgery may be explained by a positive contribution of participation to coping with surgery for IBD. The data do not support the alternative explanation that subjects with less severe disease (and thus better outcome) have greater ability to participate, although further research is required.

Adolescent↗

PubMed1998

Comparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE-format examination.

PURPOSE: To compare the psychometric properties of checklists, global rating scales preceded by a checklist, and global rating scales alone in assessing surgery residents' performances on an OSCE-like technical skills examination. METHOD: In 1996, 53 general surgery residents with one to six years of postgraduate training participated in a performance-based examination of technical skills consisting of eight 15-minute stations (bench-model simulations of operative procedures in general surgery). Two qualified surgeons marked at each station, one using a task-specific checklist (C) and a subsequent global rating scale (Gc), the other using a global rating scale only (G). RESULTS: Interstation reliabilities measured by Cronbach's alpha were .79 for C, .89 for Gc, and .85 for G. A series of multiple regressions predicting level of training from test scores revealed an R2 of .584 for C alone, which increased to .711 when Gc was entered after (p < .001), and increased to .704 when G was entered after C (p < .001). However, R2 for Gc alone was .711, and for G alone was .704, neither of which changed when C was entered into the prediction (p > .10). The R2 for Gc and G predicting level of training (.725) was not significantly greater than that of either Gc or G alone. A very similar pattern of results was seen when C, Gc, and G were used to predict independent evaluations of the operative outcomes. CONCLUSIONS: Global rating scales scored by experts showed higher inter-station reliability, better construct validity, and better concurrent validity than did checklists. Further, the presence of the checklists did not improve the reliability or validity of the global rating scale over that of the global rating scale alone. These results suggest that global rating scales administered by experts are a more appropriate summative measure when assessing candidates on performance-based examinations.

Educational Measurement↗

PubMed1997

Testing technical skill via an innovative "bench station" examination.

BACKGROUND: A new approach to testing operative technical skills, the Objective Structured Assessment of Technical Skill (OSATS), formally assesses discrete segments of surgical tasks using bench model simulations. This study examines the interstation reliability and construct validity of a large-scale administration of the OSATS. METHODS: A 2-hour, eight-station OSATS was administered to 48 general surgery residents. Residents were assessed at each station by one of 48 surgeons who evaluated the resident using two methods of scoring: task-specific checklists and global rating scales. RESULTS: Interstation reliability was 0.78 for the checklist score, and 0.85 for the global score. Analysis of variance revealed a significant effect of training for both the checklist score, F(3,44) = 20.08, P <0.001, and the global score, F(3,44) = 24.63, P <0.001. CONCLUSIONS: The OSATS demonstrates high reliability and construct validity, suggesting that we can effectively measure residents' technical ability outside the operating room using bench model simulations.

Clinical Competence↗

PubMed1997

Objective structured assessment of technical skill (OSATS) for surgical residents.

BACKGROUND: The technical skill of surgical trainees is not well assessed. This study aimed (1) to compare the reliability of three scoring systems, (2) to compare live and bench formats and (3) to assess construct validity of a test of operative skill. METHODS: Parallel examinations of operative skill, one using live animals and one using simulations, were developed. Performance was graded using operation-specific checklists, detailed global rating forms and pass/fail judgements. Twenty surgical residents each took both formats. RESULTS: Disattenuated correlations between live and bench scores were high (0.69-0.72). Mean interrater reliability across stations ranged from 0.64 to 0.72. Internal consistency was moderate to high (alpha: 0.61-0.74) for the live format using the checklist and for live and bench formats using global ratings. Global ratings discriminated between resident levels for both formats (bench: F(2,17) = 4.45, P < 0.05; live: F(2,17) = 3.55, P < 0.05), checklists did not. CONCLUSION: This preliminary study suggests that the Objective Structured Assessment of Technical Skill can reliably and validly assess surgical skills. Global ratings are a better method of assessment than task-specific checklists. Bench model simulation gives equivalent results to use of live animals for this test format.

Clinical Competence↗

PubMed1996

Teaching effectiveness of surgeons.

BACKGROUND: Evaluation of clinical teaching is essential for feedback to teachers, and for reliable and valid data for the purpose of promotion and tenure. METHODS: The teaching effectiveness of 43 surgeons was evaluated continuously over a 9-year period. Stability of Teaching Effectiveness Scores (TES) over the 9-year period, correlation with age, and the changes after academic promotion were analyzed. RESULTS: A total of 3,750 evaluations were completed. The average of 10 evaluations per surgeon per year gave an intraclass correlation of 0.65. The mean TES did not show any significant change over the 9 years. The majority of the "good" and "average" surgeons maintained their TES ratings, and most of the "poor" group improved their TES. The age of the surgeon was not a factor. Most surgeons had a decrease in their TES after academic promotion. CONCLUSIONS: A reliable system of teaching evaluation was achieved. Teaching scores were stable over time for good and average teachers, whereas the poor teachers tended to improve their scores.

Adult↗