PubMed Health⌕ Search

Biomedical subjects

Cees van der Vleuten

Publications and source records attributed to Cees van der Vleuten.

15 recordsLinked to original sources

Learning in practice: experiences and perceptions of high-scoring physicians.

PURPOSE: To increase understanding of informal learning in practice (e.g., consulting with colleagues, reading journals) through exploring the experiences and perceptions of physicians perceived to be performing well. Objectives were to find out how physicians learned in practice and maintained their competence, and how they learned about the communication skills domain specifically. METHOD: Of 142 family physicians participating in a formal multisource feedback (360-degree) formative assessment, 25 receiving high scores were invited to participate in interviews conducted in 2003 at Dalhousie University Faculty of Medicine. Twelve responded. Interviews were 1.5 hours each, recorded, transcribed, and analyzed by the research team using accepted qualitative procedures. RESULTS: While formal learning appeared important to most, informal learning, especially through patients and colleagues, appeared to be fundamental. The physicians appeared to learn intentionally from practice and work experiences, and reflection appeared integral to learning and monitoring the impact of learning. Two findings were surprising: participants' conceptions of competence and perceptions that communication skills were innate rather than learned. CONCLUSIONS: These physicians' ways of intentional learning from practice concur with current models of informal learning. However, informal learning is largely unrecognized by formal institutions. Additionally, the physicians did not in general share notions of professional competence held by educators and others in authority. These findings suggest the need to make implicit content and learning processes more explicit. Additional research areas include exploring whether physicians across the range of performance levels demonstrate similar processes of reflective learning.

Adult↗

Chronic patients in undergraduate education: didactic value as perceived by students.

OBJECTIVES: Medical education should prepare students for future clinical practice. However, this preparation is inadequate for the most prevalent problem in health care: chronic disease. This applies to the continuous aspect of chronic disease. Within the context of a newly developed programme, we investigated what makes a chronic patient interesting in the eyes of medical students, what they learned from a specific programme in which each student had contact with a chronic patient 4 times in 8 months, and what they learned from their patients. METHODS: A total of 240 Year 3 students were enrolled in the programme, 89 of whom filled in questionnaires at both the start and end of the programme. Topics included the characteristics of the ideal and the actual patient, the Ideal Physician Questionnaire, and several questions on the expected and actual amount of knowledge gained from the patient. RESULTS: Students preferred patients who demonstrated clear symptoms and had frequent contacts with health care professionals during the programme to 'well adapted' patients. The perceived knowledge obtained from the patient was less than they had expected at the start of the programme. A didactic gain perceived as low was mainly due to low expectations of gaining knowledge at the start of the programme, a doctor-centred attitude and a high level of discrepancy between the student's ideal patient and the actual patient. CONCLUSIONS: Programmes that aim to present chronic patients to medical students focus on patient selection so that patients who differ only very slightly from healthy persons are eliminated. In addition, realistic information on the types of patients with whom students can expect to have contact may help students appreciate the knowledge to be gained from these patients.

Adolescent↗

Assessment of clinical reasoning in the context of uncertainty: the effect of variability within the reference panel.

The Script Concordance Test (SCT) assesses reasoning in the context of uncertainty. Because there is no single correct answer, scoring is based on the comparison of answers provided by examinees with those provided by members of a reference panel made up of experienced practitioners. The study aimed to assess the discriminatory power of the SCT based on the variability of the reference panel's answers. Items from a bank covering different family medicine domains were classified into 3 groups according to the degree of variability of answers provided by a pool of experienced doctors. A variability index (mean squared error) was used to select items in the low, moderate and high variability categories. A 102-item test (Cronbach's alpha 0.70), made up of 3 subtests of each category, was administered to 3 contrasting groups in family medicine: 157 clerkship students, 30 residents and 30 practising doctors. anova and effect size (ES) were used to quantify and test the discrimination power of the 3 subtests. The high variability subtest showed high effect size for discrimination between extreme groups (ES = 1.5; F = 16.3, P < 0.001), whereas the moderate variability subtest showed less effect size (ES = 0.56; F = 57, P = 0.041). The low variability subtest did not discriminate significantly (ES = 0.31; F = 2.9, P = 0.06). Variability of answers within the reference panel is a key component of the discriminatory power of the SCT. In accordance with theory, the presence of variability ensures discrimination between levels of clinical experience. These results imply important considerations for the construction of efficient SCTs.

Analysis of Variance↗

The cognitive validity of the Script Concordance Test: a processing time study.

BACKGROUND: According to the theory on which the Script Concordance Test (SCT) is based, scripts contain expectations on features that are associated with each illness and about the range of values that are typical, atypical, or incompatible. PURPOSE: To document the construct validity of the SCT, we investigated the theory prediction that once a script is activated, new incoming information (e.g., additional clinical features) is processed faster if it is typical for that script than if it is atypical. If it is incompatible, processing time falls in between. METHODS: We presented 2 groups of participants (30 fourth-year medical students and 30 full-time geriatricians) with 64 clinical vignettes (divided over 5 types of prevalent clinical presentations in geriatrics), each accompanied by a diagnostic hypothesis aimed to instantiate an appropriate script. Next, we presented a new finding, which could be typical, atypical, or incompatible given the hypothesis. Participants had to decide as quickly and accurately as possible whether the new finding increased, decreased, of did not affect the likelihood of the diagnostic hypothesis. We administered the test on a computer. The dependent variable was processing time. We analyzed data with a repeated measure 2 x 3 analysis of variance. RESULTS: Typical information was processed faster than atypical and incompatible information (M = 10.6 sec vs. 19.2 sec and 16.4 sec, respectively; p lt; .001 for both). Incompatible information was processed faster than atypical information (16.4 sec vs. 19.2; p < .001). There was no significant difference between the groups of geriatricians and students. CONCLUSION: It is possible to predict what kind of information will be processed faster depending of the typicality and compatibility of clinical data for given hypotheses. Results support SCT construct validity.

Clinical Clerkship↗

Effective improvement of doctor-patient communication: a randomised controlled trial.

BACKGROUND: Doctor-patient communication is an essential component of general practice. Improvement of GPs' communication patterns is an important target of training programmes. Available studies have so far failed to provide conclusive evidence of the effectiveness of educational interventions to improve doctor-patient communication. AIM: To examine the effectiveness of a learner-centred approach that focuses on actual needs, to improve GPs' communication with patients. DESIGN OF STUDY: Randomised controlled trial. SETTING: One hundred volunteer GPs in the Netherlands. METHOD: The intervention identified individual GPs' deficiencies in communication skills by observing authentic consultations in their own surgery. This performance assessment was followed by structured activities in small group meetings, aimed at remedying the identified shortcomings. Outcomes were measured using videotaped consultations in the GPs' own surgery before and after the intervention. Communication skills were rated using the MAAS-Global, a validated checklist. RESULTS: The scores in the intervention group demonstrated a significant improvement compared with those of the control group (95% confidence interval = 0.04 to 0.75). The effect size was moderate to large (d-value = 0.66). The level of participation significantly contributed to the effectiveness. Largest improvement was found on patient-centred communication skills. CONCLUSION: The approach of structured individual improvement activities based on performance assessment is more effective in improving communication skills than current educational activities.

Communication↗

The Sharp/van der Heijde method out-performed the Larsen/Scott method on the individual patient level in assessing radiographs in early rheumatoid arthritis.

OBJECTIVE: To test the reliability of two radiologic scoring methods in rheumatoid arthritis (RA)--the Sharp/van der Heijde (SvH) and the Larsen/Scott (LS)--with generalizability analyses. STUDY DESIGN AND SETTING: Films of 51 patients representing the spectrum of early RA were read by two raters for each method. The discriminative ability and responsiveness were expressed as: intraclass correlation coefficients (ICCs), two types of smallest detectable difference (SDD), and two types of smallest detectable change (SDC); reflecting measurement error when discriminating between or detecting changes within (1) individuals or (2) groups. They were calculated for (average) scores of one to three raters. RESULTS: The discriminative capacity (0.85-0.97) and responsiveness (0.91-0.97) were good when expressed by ICC. On the group level the SDDs and SDCs ranged between 0.6-3.3% of the max. obtainable score. On the individual level, the scores showed better reliability measured with the SvH (SDDs 2.0-3.4%) than with the LS (SDDs 5.3-9.2%). The SvH also assessed changes in scores in individuals with less measurement error (SDCs 1.3-2.2%) than the LS (SDCs 2.3-3.9%). CONCLUSION: For early RA patients, the SvH seems preferable if analyses on individual level are included.

Analysis of Variance↗

Measurement of perception and interpretation skills during radiology training: utility of the script concordance approach.

Imaging specialties require both perceptual and interpretation skills. Except in very simple cases, data perception and interpretation vary among clinicians. This variability makes for difficulty in measuring these skills with traditional assessment tools. The script concordance approach is conceived to allow standardized assessment in contexts of uncertainty. In this exploratory study, the authors tested the usefulness of the approach for assessment of perceptual and interpretation skills in radiology. A perception test (PT) and an interpretation test (IT) were designed according to the approach. Both tests used plain chest X-rays. Three groups were tested: clerkship students (20), junior residents (R1-R3; 20), senior residents (R4-R5; 20). Eleven certified radiologists, all currently appointed to chest reading, provided the answers by aggregate scoring method. Statistics included descriptive, ANOVA, regression analysis, Pearson and Spearman correlation coefficients. Cronbach alpha values were 0.79 and 0.81 for the PT and IT respectively. Score progression was statistically significant in both tests. Perception scores progressed more rapidly than interpretation scores during training. Effect size was large in discriminating low versus higher level of expertise, 2.2 (PT) and 1.6 (IT). The Pearson correlation coefficient between both tests was 0.58. Cronbach alpha coefficient values indicate reasonable reliability for both tests. The linear progression of scores, each at its own pace, and the positive and moderate magnitude of the Pearson correlation coefficient are arguments suggesting measurement of two different skills. More studies are necessary to document the approach usefulness for assessment in radiology training.

Clinical Competence↗

The long case.

BACKGROUND: The long case has been gradually replaced by the objective structured clinical examination (OSCE) as a summative assessment of clinical skills. Its demise occurred against a paucity of psychometric research. This article reviews the current status of the long case, appraising its strengths and weaknesses as an assessment tool. ISSUES: There is a conflict between validity and reliability. The long case assesses an integrated clinical reaction between doctor and real patients and has high face validity. Intercase reliability is the prime problem. As most examinations traditionally used a single case only, problems of content specificity and standardisation were not addressed. DISCUSSION: Recent research suggests that testing across more cases does improve reliability. Better structuring of tests and direct observation increases validity. Substituting standardised cases for real patients may be of little benefit compared to increasing the sample of cases. CONCLUSIONS: Observed long cases can be useful for assessment depending on the sample size of cases and examiners. More research is needed into the exact nature of intercase and interexaminer variance and consequential validity. Feasibility remains a key problem. More exploration of combined assessments using real patients with OSCEs is suggested.

Clinical Competence↗

Standardized assessment of reasoning in contexts of uncertainty: the script concordance approach.

Current written tools of assessment are mostly measuring the capacity to solve well-defined problems by the application of rules and principles, while the essence of expertise in the professions lies in the capacity to solve ill-defined problems, that is, reasoning in contexts of uncertainty. The purpose of this study is to describe an approach that allows assessing ill-defined problems and to present and discuss research findings related to this approach. The tool has been used up to now mainly in medicine, however it can be applied in all health professions. The approach is based on three principles: (a) examinees are faced with a challenging authentic situation in which several options are relevant; (b) the response format is a Likert-type scale that reflects the way information is processed in problem-solving situations, according to the script theory; and (c) scoring is based on the aggregate scoring method to take into account the variability of reasoning processes among experts. Research findings suggest that the approach permits one to reliably discriminate examinees across their level of experience, and so in very different domains. It makes it possible to measure skills or domains that were up to now difficult to measure.

Clinical Competence↗

Comparison of a rational and an empirical standard setting procedure for an OSCE. Objective structured clinical examinations.

PURPOSE: Earlier studies of absolute standard setting procedures for objective structured clinical examinations (OSCEs) show inconsistent results. This study compared a rational and an empirical standard setting procedure. Reliability and credibility were examined first. The impact of a reality check was then established. METHODS: The OSCE included 16 stations and was taken by trainees in their final year of postgraduate training in general practice and experienced general practitioners. A modified Angoff (independent judgements, no group discussion) with and without a reality check was used as a rational procedure. A method related to the borderline group procedure, the borderline regression (BR) method, was used as an empirical procedure. Reliability was assessed using generalisability theory. Credibility was assessed by comparing pass rates and by relating the passing scores to test difficulty. RESULTS: The passing scores were 73.4% for the Angoff procedure without reality check (Angoff I), 66.0% for the Angoff procedure with reality check (Angoff II) and 57.6% for the BR method. The reliabilities (expressed as root mean square errors) were 2.1% for Angoffs I and II, and 0.6% for the BR method. The pass rates of the trainees and GPs were 19% and 9% for Angoff I, 66% and 46% for Angoff II, and 95% and 80% for the BR method, respectively. The correlation between test difficulty and passing score was 0.69 for Angoff I, 0.88 for Angoff II and 0.86 for the BR method. CONCLUSION: The BR method provides a more credible and reliable standard for an OSCE than a modified Angoff procedure. A reality check improves the credibility of the Angoff procedure but does not improve its reliability.

Clinical Competence↗

Reproducibility of clinical performance assessment in practice using incognito standardized patients.

BACKGROUND: The reproducibility of authentic assessment methods has been investigated for objective structured clinical examinations (OSCEs) and video assessment in general practice, but not for assessment with incognito standardized patients. PURPOSE: To investigate the reproducibility of assessment with incognito standardized patients. METHODS: A total of 27 Dutch rheumatologists in 16 hospitals were each visited by 8 incognito standardized patients presenting with different rheumatological disorders. After each visit, the standardized patient completed a case-specific checklist containing items on medical history, physical examination and management. Over a 20-month period, 254 incognito visits took place, of which 201 were first visits. The standardized patient was detected by the rheumatologist in 2 cases only. These encounters were not included in the analysis. Generalizability theory was used to investigate the reproducibility of the assessment. RESULTS: One fifth of the variance can be attributed to variation between rheumatologists. The largest variance is due to the variation in difficulty among cases. A reproducible assessment requires 3 hours of testing time (6 cases) if it is obtained through a norm-referenced interpretation of scores and 7 hours of testing time (14 cases) if it is obtained through an absolute interpretation of scores. CONCLUSION: The reproducibility of performance assessment in clinical practice by incognito standardized patients is similar to that of other authentic measurements for the assessment of clinical competence and performance.

Analysis of Variance↗

Stability of clinical reasoning assessment results with the Script Concordance test across two different linguistic, cultural and learning environments.

The Script Concordance (SC) test is designed to measure the organization of knowledge that allows interpretation of data in clinical reasoning. An originality of the test is that answer keys use an aggregate scoring method based on answers given by a panel of experts. Previous studies have shown that the SC test has good construct validity. This study, done in urology, explores (1) the stability of the construct validity of the test across two different linguistic and learning environments and (2) the effect of the use of experts who belong to different environments. An 80-item SC test was administered to participants from a French and a Canadian university. Two levels of experience were tested: 25 residents in urology (11 from the French university and 14 from the Canadian university) and 23 students (15 from the French faculty, eight from the Canadian faculty). Reliability analysis was studied with Cronbach's alpha coefficient. Scores between groups were compared by analysis of variance. Reliability coefficient of the 80 items test was 0.794 for the French participants and 0.795 for the Canadian participants. Scores increased with clinical experience in urology in the two sites. Candidates obtained higher scores when correction was done using the answer key provided by the experts from the same country. These data support the stability of the construct validity of the tool across different learning environments.

Adult↗

Comparison of an aggregate scoring method with a consensus scoring method in a measure of clinical reasoning capacity.

BACKGROUND: Diversity of clinical reasoning paths of thought among experts is well known. Nevertheless, in written clinical reasoning assessment, the common practice is to ask experts to reach a consensus on each item and to assess students on a unique "good answer." PURPOSES: To explore the effects of taking the variability of experts answers into account in a method of clinical reasoning assessment based on authentic tasks: the Script Concordance Test. METHODS: Two different methods were used to build answer keys. The first incorporated variability among a group of experts (criterion experts) through an aggregate scoring method. The second was made with the consensus obtained from the group of criterion experts for each answer. Scores obtained with the two methods by students and another group of experts (tested experts) were compared. The domain of assessment was gynecology-obstetric clinical knowledge. The sample consisted of 150 clerkship students and seven other experts (tested experts). RESULTS: In a context of authentic tasks, experts' answers on items varied substantially. Amazingly, 59% of answers given individually by criterion group experts differed from the answer they provided when they were asked in a group to provide the "good answer" required from students. The aggregate scoring method showed several advantages and was more sensitive to detecting expertise. CONCLUSIONS: The findings suggest that, in assessment of complex performance in ill-defined situations, the usual practice of asking experts to reach a consensus on each item reduces and hinders the detection of expertise. If these results are confirmed by other researches, this practice should be reconsidered.

Cluster Analysis↗

Doctor-patient interaction: standardized patients' reflections from inside the rheumatological office.

OBJECTIVE: To assess appreciation and quality of doctor-patient interaction by exploring standardized patients' (SP) opinions on aspects of the interaction between doctors and standardized patients. METHODS: A focus group interview was organized with SP who had completed 254 incognito visits to 26 Dutch rheumatologists in a study assessing rheumatologists' daily practice performance; 13 of 16 SP attended the interview. Patients discussed aspects of interaction with the physicians. The interview was audiotaped and transcribed literally. Recurring themes were identified. RESULTS: Participants were on the whole very satisfied with the rheumatological care received. Factors contributing to satisfaction included "being approached as a person," "being treated respectfully," and "being given enough room to mention all complaints." On the other hand, SP were struck by the variation in performance among the rheumatologists. CONCLUSION: Physicians may not be aware of the influence of their behavior on patients. Most critical comments from patients regarding communication and behavior were on small things, which should not be too difficult to change in daily practice.

Aged↗