Biomedical subjects

Bernard Charlin

Publications and source records attributed to Bernard Charlin.

11 recordsLinked to original sources

PubMed2006

On line clinical reasoning assessment with Script Concordance test in urology: results of a French pilot study.

BACKGROUND: The Script Concordance test (SC) test is an assessment tool that measures the capacity to solve ill-defined problems, that is, reasoning in a context of uncertainty. This study assesses the feasibility, reliability and validity of the SC test made available on the Web to French urologists. METHODS: A 97 items SC test was developed based on major educational objectives of French urology training programmes. A secure Web site was created with two sequential modules: a) The first one for the reference panel to elaborate the scoring system; b) The second for candidates with different levels of experience in urology: Board certified urologists, chief-residents, residents, medical students. All participants were recruited on a voluntary basis. Statistical analysis included descriptive statistics of the participants' scores and factorial analysis of variance (ANOVA) to study differences between groups' means. Reliability was evaluated with Cronbach's alpha coefficient. RESULTS: The on line SC test has been operational since June 2004. Twenty-six faculty members constituted the reference panel. During the following 10 months, 207 participants took the test online (124 urologists, 29 chief-residents, 38 residents, 16 students). No technical problem was encountered. Forty-five percent of the participants completed the test partially only. Differences between the means scores for the 4 groups were statistically significant (P = 0.0123). The Bonferroni post-hoc correction indicated that significant differences were present between students and chief-residents, between students and urologists. There were no differences between chief-residents and urologists. Reliability coefficient was 0.734 for the total group of participants. CONCLUSION: Feasibility of Web-based SC test was proved successful by the large number of participants who participated in a few months. This Web site has permitted to quickly confirm reliability of the SC test and develop strategy to improve construct validity of the test when applied in the field of urology. Nevertheless, optimisation of the SC test content, with a smaller number of items will be necessary. Virtual medical education initiative such as this SC test delivered on the Internet warrants consideration in the current context of national pre-residency certification examination in France.

Academic Medical Centers↗

PubMed2006

Assessment of clinical reasoning in the context of uncertainty: the effect of variability within the reference panel.

The Script Concordance Test (SCT) assesses reasoning in the context of uncertainty. Because there is no single correct answer, scoring is based on the comparison of answers provided by examinees with those provided by members of a reference panel made up of experienced practitioners. The study aimed to assess the discriminatory power of the SCT based on the variability of the reference panel's answers. Items from a bank covering different family medicine domains were classified into 3 groups according to the degree of variability of answers provided by a pool of experienced doctors. A variability index (mean squared error) was used to select items in the low, moderate and high variability categories. A 102-item test (Cronbach's alpha 0.70), made up of 3 subtests of each category, was administered to 3 contrasting groups in family medicine: 157 clerkship students, 30 residents and 30 practising doctors. anova and effect size (ES) were used to quantify and test the discrimination power of the 3 subtests. The high variability subtest showed high effect size for discrimination between extreme groups (ES = 1.5; F = 16.3, P < 0.001), whereas the moderate variability subtest showed less effect size (ES = 0.56; F = 57, P = 0.041). The low variability subtest did not discriminate significantly (ES = 0.31; F = 2.9, P = 0.06). Variability of answers within the reference panel is a key component of the discriminatory power of the SCT. In accordance with theory, the presence of variability ensures discrimination between levels of clinical experience. These results imply important considerations for the construction of efficient SCTs.

Analysis of Variance↗

PubMed2006

The cognitive validity of the Script Concordance Test: a processing time study.

BACKGROUND: According to the theory on which the Script Concordance Test (SCT) is based, scripts contain expectations on features that are associated with each illness and about the range of values that are typical, atypical, or incompatible. PURPOSE: To document the construct validity of the SCT, we investigated the theory prediction that once a script is activated, new incoming information (e.g., additional clinical features) is processed faster if it is typical for that script than if it is atypical. If it is incompatible, processing time falls in between. METHODS: We presented 2 groups of participants (30 fourth-year medical students and 30 full-time geriatricians) with 64 clinical vignettes (divided over 5 types of prevalent clinical presentations in geriatrics), each accompanied by a diagnostic hypothesis aimed to instantiate an appropriate script. Next, we presented a new finding, which could be typical, atypical, or incompatible given the hypothesis. Participants had to decide as quickly and accurately as possible whether the new finding increased, decreased, of did not affect the likelihood of the diagnostic hypothesis. We administered the test on a computer. The dependent variable was processing time. We analyzed data with a repeated measure 2 x 3 analysis of variance. RESULTS: Typical information was processed faster than atypical and incompatible information (M = 10.6 sec vs. 19.2 sec and 16.4 sec, respectively; p lt; .001 for both). Incompatible information was processed faster than atypical information (16.4 sec vs. 19.2; p < .001). There was no significant difference between the groups of geriatricians and students. CONCLUSION: It is possible to predict what kind of information will be processed faster depending of the typicality and compatibility of clinical data for given hypotheses. Results support SCT construct validity.

Clinical Clerkship↗

PubMed2005

Online clinical reasoning assessment with the Script Concordance test: a feasibility study.

BACKGROUND: The script concordance (SC) test is an assessment tool that measures capacity to solve ill-defined problems, that is, reasoning in context of uncertainty. This tool has been used up to now mainly in medicine. The purpose of this pilot study is to assess the feasibility of the test delivered on the Web to French urologists. METHODS: The principle of SC test construction and the development of the Web site are described. A secure Web site was created with two sequential modules: (a) The first one for the reference panel (n = 26) with two sub-tasks: to validate the content of the test and to elaborate the scoring system; (b) The second for candidates with different levels of experience in Urology: Board certified urologists, residents, medical students (5 or 6th year). Minimum expected number of participants is 150 for urologists, 100 for residents and 50 for medical students. Each candidate is provided with an individual access code to this Web site. He/she may complete the Script Concordance test several times during his/her curriculum. RESULTS: The Web site has been operational since April 2004. The reference panel validated the test in June of the same year during the annual seminar of the French Society of Urology. The Web site is available for the candidates since September 2004. In six months, 80% of the target figure for the urologists, 68% of the target figure for the residents and 20% of the target figure for the student passed the test online. During these six months, no technical problem was encountered. CONCLUSION: The feasibility of the web-based SC test is successful as two-thirds of the expected number of participants was included within six months. Psychometric properties (validity, reliability) of the test will be evaluated on a large scale (N = 300). If positive, educational impact of this assessment tool will be useful to help urologists during their curriculum for the acquisition of clinical reasoning skills, which is crucial for professional competence.

Clinical Competence↗

PubMed2004

Measurement of perception and interpretation skills during radiology training: utility of the script concordance approach.

Imaging specialties require both perceptual and interpretation skills. Except in very simple cases, data perception and interpretation vary among clinicians. This variability makes for difficulty in measuring these skills with traditional assessment tools. The script concordance approach is conceived to allow standardized assessment in contexts of uncertainty. In this exploratory study, the authors tested the usefulness of the approach for assessment of perceptual and interpretation skills in radiology. A perception test (PT) and an interpretation test (IT) were designed according to the approach. Both tests used plain chest X-rays. Three groups were tested: clerkship students (20), junior residents (R1-R3; 20), senior residents (R4-R5; 20). Eleven certified radiologists, all currently appointed to chest reading, provided the answers by aggregate scoring method. Statistics included descriptive, ANOVA, regression analysis, Pearson and Spearman correlation coefficients. Cronbach alpha values were 0.79 and 0.81 for the PT and IT respectively. Score progression was statistically significant in both tests. Perception scores progressed more rapidly than interpretation scores during training. Effect size was large in discriminating low versus higher level of expertise, 2.2 (PT) and 1.6 (IT). The Pearson correlation coefficient between both tests was 0.58. Cronbach alpha coefficient values indicate reasonable reliability for both tests. The linear progression of scores, each at its own pace, and the positive and moderate magnitude of the Pearson correlation coefficient are arguments suggesting measurement of two different skills. More studies are necessary to document the approach usefulness for assessment in radiology training.

Clinical Competence↗

PubMed2004

Standardized assessment of reasoning in contexts of uncertainty: the script concordance approach.

Current written tools of assessment are mostly measuring the capacity to solve well-defined problems by the application of rules and principles, while the essence of expertise in the professions lies in the capacity to solve ill-defined problems, that is, reasoning in contexts of uncertainty. The purpose of this study is to describe an approach that allows assessing ill-defined problems and to present and discuss research findings related to this approach. The tool has been used up to now mainly in medicine, however it can be applied in all health professions. The approach is based on three principles: (a) examinees are faced with a challenging authentic situation in which several options are relevant; (b) the response format is a Likert-type scale that reflects the way information is processed in problem-solving situations, according to the script theory; and (c) scoring is based on the aggregate scoring method to take into account the variability of reasoning processes among experts. Research findings suggest that the approach permits one to reliably discriminate examinees across their level of experience, and so in very different domains. It makes it possible to measure skills or domains that were up to now difficult to measure.

Clinical Competence↗

PubMed2002

Assessment of clinical reasoning competence in urology with the script concordance test: an exploratory study across two sites from different countries.

OBJECTIVES: The script concordance (SC) test is designed to measure the organisation of knowledge that allows interpretation of data in clinical reasoning. This study explores the use of this new written examination tool in urology. MATERIALS AND METHODS: An 80 items SC test was administered to participants from a French and a Canadian university. Three levels of experience were tested: urologists (n = 22), residents in urology (n = 25) and students (n = 23). Scores between groups were compared by analysis of variance. Reliability analysis was studied with Cronbach alpha coefficient. RESULTS: Mean global scores were 51.45 +/- 5.29 for students, 58.19 +/- 3.81 for residents and 62.27 +/- 5.46 for urologists. The difference between the three groups was statistically significant (P < 0.00001). Interaction between levels of expertise and sites was apparently not significant (P = 0.326). Cronbach alpha was 0.79 for the test. CONCLUSIONS: This study shows that the SC test is able to discriminate among participants according to their levels of clinical experience in urology. The results are similar in two different learning environments. The SC test appears as a simple and direct approach to testing organisation and use of knowledge in urology.

Clinical Competence↗

PubMed2002

Stability of clinical reasoning assessment results with the Script Concordance test across two different linguistic, cultural and learning environments.

The Script Concordance (SC) test is designed to measure the organization of knowledge that allows interpretation of data in clinical reasoning. An originality of the test is that answer keys use an aggregate scoring method based on answers given by a panel of experts. Previous studies have shown that the SC test has good construct validity. This study, done in urology, explores (1) the stability of the construct validity of the test across two different linguistic and learning environments and (2) the effect of the use of experts who belong to different environments. An 80-item SC test was administered to participants from a French and a Canadian university. Two levels of experience were tested: 25 residents in urology (11 from the French university and 14 from the Canadian university) and 23 students (15 from the French faculty, eight from the Canadian faculty). Reliability analysis was studied with Cronbach's alpha coefficient. Scores between groups were compared by analysis of variance. Reliability coefficient of the 80 items test was 0.794 for the French participants and 0.795 for the Canadian participants. Scores increased with clinical experience in urology in the two sites. Candidates obtained higher scores when correction was done using the answer key provided by the experts from the same country. These data support the stability of the construct validity of the tool across different learning environments.

Adult↗

PubMed2002

Communication between consultants and referring physicians: a qualitative study to define learning and assessment objectives in a specialty residency program.

BACKGROUND: Outpatient consultation constitutes a major part of medical practice. However, little is known about the skills which should be taught to residents in order for them to improve their consultant-referring physician relationships. PURPOSES: To specify the consultant skills which are required to ensure an effective communication between specialists and referring physicians. METHODS: A qualitative study based on (a) a literature search and (b) focus group interviews. RESULTS: Skills thus identified and described are classified in two groups: observable skills and principles/attitudes. CONCLUSIONS: The consensual specification of these abilities permits a greater efficacy in the teaching of consultant skills.

Ambulatory Care↗

PubMed2002

Comparison of an aggregate scoring method with a consensus scoring method in a measure of clinical reasoning capacity.

BACKGROUND: Diversity of clinical reasoning paths of thought among experts is well known. Nevertheless, in written clinical reasoning assessment, the common practice is to ask experts to reach a consensus on each item and to assess students on a unique "good answer." PURPOSES: To explore the effects of taking the variability of experts answers into account in a method of clinical reasoning assessment based on authentic tasks: the Script Concordance Test. METHODS: Two different methods were used to build answer keys. The first incorporated variability among a group of experts (criterion experts) through an aggregate scoring method. The second was made with the consensus obtained from the group of criterion experts for each answer. Scores obtained with the two methods by students and another group of experts (tested experts) were compared. The domain of assessment was gynecology-obstetric clinical knowledge. The sample consisted of 150 clerkship students and seven other experts (tested experts). RESULTS: In a context of authentic tasks, experts' answers on items varied substantially. Amazingly, 59% of answers given individually by criterion group experts differed from the answer they provided when they were asked in a group to provide the "good answer" required from students. The aggregate scoring method showed several advantages and was more sensitive to detecting expertise. CONCLUSIONS: The findings suggest that, in assessment of complex performance in ill-defined situations, the usual practice of asking experts to reach a consensus on each item reduces and hinders the detection of expertise. If these results are confirmed by other researches, this practice should be reconsidered.

Cluster Analysis↗

PubMed2002

[Systematic planning of technical training in urologic surgery: methodologic trial].

OBJECTIVES: Trainee urologists must be helped and guided in order to acquire surgical techniques and professional skills. This methodological trial was designed to define the training objectives based on the needs formulated in terms of basic technical procedures. MATERIALS AND METHODS: Systematic planning based on the training cycle was used in this study. Training needs were identified by a questionnaire sent to 4 groups composed of a limited number (about twelve) of urologists, but representative of the various modes of urological practice in several regions of France. Data were acquired according to the Delphi method. RESULTS: A list of 10 operations to be mastered at the end of urology training was defined consensually based on the replies (79.2%) to the questionnaire. This list was used as the basis for the design of general and specific training objectives for each operation. These objectives were validated by the urologists consulted. Three examples of these objectives are presented. CONCLUSION: A clear determination of the surgical technique training needs and objectives could lead to standardization of the practical training of urology residents. This would subsequently need to be completed by determination of the methods of evaluation of achievement of these objectives for tutors and trainee urologists.

Education, Medical↗