Biomedical subjects

C Van der Vleuten

Publications and source records attributed to C Van der Vleuten.

6 recordsLinked to original sources

PubMed2006

Regulatory end-point assessment of the consultation competence of family practice trainees in Kuwait.

BACKGROUND: No single approach to the regulatory assessment of global consultation competence has been shown to possess the required levels of validity, reliability and feasibility. OBJECTIVE: To evaluate the approach adopted in Kuwait to the regulatory end-point assessment of the global consultation competence of family practice trainees with particular reference to validity, reliability and feasibility. METHODS: Family practice trainees in Kuwait were individually and directly observed for 3 hours in consultation with a minimum of 10 patients by a pair of examiners. Performance was judged against the explicit criteria of consultation competence as contained in the Leicester Assessment Package (LAP). RESULTS: The marks independently allocated by the pairs of examiners to 126 trainees between 1994 and 2001 were within five percentage points on 91% of occasions. A reliability coefficient of 0.82 was achieved when two examiners independently marked candidates consulting with 10 real patients; this rose to 0.95 at the critical 50% pass-fail margin. The main sources of variance contributing to the reliability of marks allocated were candidate performance (42%) and the interaction of candidate performance across cases, i.e., case specificity (30%). The clinical challenges presented by the patients were judged by both examiners to be sufficient to enable performance to be assessed across the seven LAP consultation categories as follows: behaviour and relationship with patients (100% of consultations), interviewing/history taking (100%), record keeping (99%), patient management (99%), problem solving (98%), physical examination (95%), and anticipatory care (86%). Each assessment involved a pair of examiners and lasted approximately 3.5 hours. CONCLUSION: The Kuwait clinical examination achieves high content validity and authenticity as it uses direct observation of performance, validated and explicit criteria against which performance is judged, and real patient challenges. It can discriminate between different levels of consultation performance and satisfies the recognized reliability threshold for regulatory examinations (0.82 vs 0.80). Accordingly, we recommend the use of such an approach in the regulatory end-point assessment of the global consultation competence of trainees in family practice. Such an approach is more valid, and is likely to be more feasible, than simulated surgeries or the short-case OSCE format.

Family Practice↗

PubMed2005

The role of a computerised case-based testing procedure in practice performance assessment.

INTRODUCTION: For postgraduate training of doctors there is a need for valid and reliable instruments to assess their daily performance. Various instruments have been suggested, some of which use incognito simulated patients (SPs). These methods are resource intensive. Computerised Case-based testing (CCT) is logistically simpler and may still predict performance well. The research question was to evaluate the predictive validity of CCT for performance. METHODS: Seventeen rheumatologists were each visited by eight incognito SPs presenting various rheumatological complaints, and scoring the performance of the rheumatologists using a predefined checklist. From this checklist a panel of experts identified essential items. In addition the rheumatologists sat a CCT test containing 55 cases with a total of 121 items. RESULTS: Negative correlations were found between the SP scores and the CCT scores. This was unexpected. Therefore, background variables on experience were used to compare both methods. The correlation between these and CCT were high and positive and with the SP scores high and negative. This pattern did not differ when using the essential items of the checklist. Reliabilities of the SP scores were markedly high. DISCUSSION: Although CCT was not predictive of SP scores, it was related to working experience. There are good reasons to assume that although SP-scores were more authentic, they were less valid than CCT scores, mainly because they focussed more on thoroughness than on efficiency in data gathering. The results underpin the assumption that for valid performance assessment the most important issue is what information about the candidate is collected and now how authentic the method is.

Clinical Competence↗

PubMed2001

Assessment of clinical competence.

Tests of clinical competence, which allow decisions to be made about medical qualification and fitness to practise, must be designed with respect to key issues including blueprinting, validity, reliability, and standard setting, as well as clarity about their formative or summative function. Multiple choice questions, essays, and oral examinations could be used to test factual recall and applied knowledge, but more sophisticated methods are needed to assess clincial performance, including directly observed long and short cases, objective structured clinical examinations, and the use of standardised patients. The goal of assessment in medical education remains the development of reliable measurements of student performance which, as well as having predictive value for subsequent clinical competence, also have a formative, educational role.

Clinical Competence↗

PubMed2001

Measurement of clinical reflective capacity early in training as a predictor of clinical reasoning performance at the end of residency: an experimental study on the script concordance test.

BACKGROUND: The script concordance (SC) test was conceived to measure knowledge organization, the presence of links between items of knowledge which allow for interpretation of data in clinical decision making situations. Earlier studies have shown that the SC test has good psychometric qualities and overcomes some of the limitations of simulation clinical testing. This study explores the predictive validity of the test. OBJECTIVES: To verify whether scores obtained by students at the end of clerkship predict their clinical reasoning performance at the end of residency. DESIGN: Comparison of scores obtained on a SC test taken at the end of clerkship with those obtained 2 years later at the end of residency on two clinical reasoning assessments of known validity, called the short-answer management problems (SAMPs) and the simulated office orals (SOOs), and an objective structured clinical examination (OSCE) aimed at assessing hands-on skills and clinical reasoning. Data were treated by Pearson correlation analysis. SUBJECTS AND SETTING: A cohort of 24 students from a medical school in Quebec was followed up to the end of their residency in family medicine, completed in several schools across Quebec. RESULTS: The observed Pearson correlation coefficients of the SC test were statistically significant (0.451, P=0.013; 0.447; P=0.015) when compared with the SAMPs and the SOOs, respectively. They were not statistically significant (0.340, P=0.052) when compared with the OSCE. CONCLUSION: The authors assumed that the richness of knowledge organization, as indicated by SC test scores, would predict part of the performance on the measures of clinical reasoning (SAMP and SOO), but would predict less well performance on the OSCE which measures both clinical skills and clinical reasoning. Data found in the study are coherent with this hypothesis. This is evidence in favour of the construct validity of the SC test. It also indicates that, in the context of certification assessment, if a candidate has shown good organization of clinical knowledge at an early point in training, it can be expected that he/she will show good organization at subsequent measurements of this kind of knowledge. This appears to be true even if the later measures bear on a wider clinical domain.

Clinical Clerkship↗

PubMed2001

Standardized or real patients to test clinical competence? The long case revisited.

BACKGROUND: In undergraduate clinical examinations, the use of real patients as long cases is being replaced by objective structured clinical examinations (OSCEs) which use simulated scenarios, although we lack published psychometric data on long cases to support the move from real to simulated patients. AIM: To assess candidate performance across two history-taking long cases to estimate the number of cases required for a reliable assessment. Results are compared with psychometric data from an OSCE. SETTING: A final-year qualifying undergraduate clinical examination. METHOD: Two observed history-taking long cases were included, alongside an OSCE. Candidates interviewed two unstandardized real patients. The history-taking part (14 minutes) was observed, uninterrupted, by examiner(s) who assessed data gathering, interviewing, and diagnostic and management skills. The presentation (7 minutes) was unstructured; the examiner(s) intervened as appropriate. Marks were expressed as a percentage of the total possible score and analysed using generalizability theory to estimate intercase reliability. RESULTS: Two examiner pairs independently rated both long cases for 79 (36.7%) of the 214 candidates. Projections based on generalizability theory showed that 10 20-minute cases would give reliabilities of 0.84 for single-marked and 0.88 for double-marked candidates, compared with a projected reliability of 0.73 for the same 214 candidates taking the OSCE. CONCLUSION: If history-taking long cases are observed, three-and-a-half hours of testing time using 10 unstandardized patients would produce a reliable test. Long cases therefore are, in terms of reliability, no worse and no better than OSCEs in assessing clinical competence.

Clinical Competence↗

PubMed2001

Composite undergraduate clinical examinations: how should the components be combined to maximize reliability?

BACKGROUND: Clinical examinations increasingly consist of composite tests to assess all aspects of the curriculum recommended by the General Medical Council. SETTING: A final undergraduate medical school examination for 214 students. AIM: To estimate the overall reliability of a composite examination, the correlations between the tests, and the effect of differences in test length, number of items and weighting of the results on the reliability. METHOD: The examination consisted of four written and two clinical tests: multiple-choice questions (MCQ) test, extended matching questions (EMQ), short-answer questions (SAQ), essays, an objective structured clinical examination (OSCE) and history-taking long cases. Multivariate generalizability theory was used to estimate the composite reliability of the examination and the effects of item weighting and test length. RESULTS: The composite reliability of the examination was 0.77, if all tests contributed equally. Correlations between examination components varied, suggesting that different theoretically interpretable parameters of competence were being tested. Weighting tests according to items per test or total test time gave improved reliabilities of 0.93 and 0.81, respectively. Double weighting of the clinical component marginally affected the reliability (0.76). CONCLUSION: This composite final examination achieved an overall reliability sufficient for high-stakes decisions on student clinical competence. However, examination structure must be carefully planned and results combined with caution. Weighting according to number of items or test length significantly affected reliability. The components testing different aspects of knowledge and clinical skills must be carefully balanced to ensure both content validity and parity between items and test length.

Clinical Competence↗