PubMed Health⌕ Search

Biomedical subjects

Lambert W T Schuwirth

Publications and source records attributed to Lambert W T Schuwirth.

12 recordsLinked to original sources
PubMed2006

Broadening perspectives on clinical performance assessment: rethinking the nature of in-training assessment.

CONTEXT: In-training assessment (ITA), defined as multiple assessments of performance in the setting of day-to-day practice, is an invaluable tool in assessment programmes which aim to assess professional competence in a comprehensive and valid way. Research on clinical performance ratings, however, consistently shows weaknesses concerning accuracy, reliability and validity. Attempts to improve the psychometric characteristics of ITA focusing on standardisation and objectivity of measurement thus far result in limited improvement of ITA-practices. PURPOSE: The aim of the paper is to demonstrate that the psychometric framework may limit more meaningful educational approaches to performance assessment, because it does not take into account key issues in the mechanics of the assessment process. Based on insights from other disciplines, we propose an approach to ITA that takes a constructivist, social-psychological perspective and integrates elements of theories of cognition, motivation and decision making. A central assumption in the proposed framework is that performance assessment is a judgment and decision making process, in which rating outcomes are influenced by interactions between individuals and the social context in which assessment occurs. DISCUSSION: The issues raised in the article and the proposed assessment framework bring forward a number of implications for current performance assessment practice. It is argued that focusing on the context of performance assessment may be more effective in improving ITA practices than focusing strictly on raters and rating instruments. Furthermore, the constructivist approach towards assessment has important implications for assessment procedures as well as the evaluation of assessment quality. Finally, it is argued that further research into performance assessment should contribute towards a better understanding of the factors that influence rating outcomes, such as rater motivation, assessment procedures and other contextual variables.

Clinical Competence↗
PubMed2006

A plea for new psychometric models in educational assessment.

OBJECTIVE: To describe the weaknesses of the current psychometric approach to assessment as a scientific model. DISCUSSION: The current psychometric model has played a major role in improving the quality of assessment of medical competence. It is becoming increasingly difficult, however, to apply this model to modern assessment methods. The central assumption in the current model is that medical competence can be subdivided into separate measurable stable and generic traits. This assumption has several far-reaching implications. Perhaps the most important is that it requires a numerical and reductionist approach, and that aspects such as fairness, defensibility and credibility are by necessity mainly translated into reliability and construct validity. These approaches are more and more difficult to align with modern assessment approaches such as mini-CEX, 360-degree feedback and portfolios. This paper describes some of the weaknesses of the psychometric model and aims to open a discussion on a conceptually different statistical approach to quality of assessment. FUTURE DIRECTIONS: We hope that the discussion opened by this paper will lead to the development of a conceptually different statistical approach to quality of assessment. A probabilistic or Bayesian approach would be worth exploring.

Bayes Theorem↗
PubMed2005

The use of observational diaries in in-training evaluation: student perceptions.

INTRODUCTION: In health science education clinical clerkships serve the twofold purpose of guiding student learning and assessment of performance. Evidently, both formative and summative assessment procedures are needed in clerkship assessment. In-training evaluation (ITE) has the potential to serve both assessment functions. Implementation of effective ITE, however, has been shown to be problematic, partly because integration of assessment functions may have negative consequences for teaching and learning. This study investigates student perceptions of the impact of an integrated assessment approach, seeking to refine criteria for effective ITE. METHOD: In the curriculum of Maastricht Midwifery School (MMS), clerkship assessment is based on ITE serving both assessment functions. The ITE model is based on principles of extensive work sampling, and frequent documentation of performance. A focus group technique was used to explore student perceptions on the impact of the ITE approach on student learning and supervisor teaching behaviour, and on the usefulness of information for decision making. RESULTS: Results indicate that the assessment approach is effective in guidance of student learning. Furthermore, students consider the frequent performance documentation essential in clerkship grading. Acceptance and effectivity of ITE requires a learning environment which is safe and respectful. Transparency of assessment processes is the key to success. Suggestions for improvement focus on variation in evaluation formats, improvement of feedback (narrative, complete) and student involvement in assessment. CONCLUSION: ITE can fulfill both its formative and summative purposes when some crucial conditions are taken into account. Careful training of both supervisors and students in the use of ITE for student learning and performance measurement is essential.

Curriculum↗
PubMed2005

Assessing professional competence: from methods to programmes.

INTRODUCTION: We use a utility model to illustrate that, firstly, selecting an assessment method involves context-dependent compromises, and secondly, that assessment is not a measurement problem but an instructional design problem, comprising educational, implementation and resource aspects. In the model, assessment characteristics are differently weighted depending on the purpose and context of the assessment. EMPIRICAL AND THEORETICAL DEVELOPMENTS: Of the characteristics in the model, we focus on reliability, validity and educational impact and argue that they are not inherent qualities of any instrument. Reliability depends not on structuring or standardisation but on sampling. Key issues concerning validity are authenticity and integration of competencies. Assessment in medical education addresses complex competencies and thus requires quantitative and qualitative information from different sources as well as professional judgement. Adequate sampling across judges, instruments and contexts can ensure both validity and reliability. Despite recognition that assessment drives learning, this relationship has been little researched, possibly because of its strong context dependence. ASSESSMENT AS INSTRUCTIONAL DESIGN: When assessment should stimulate learning and requires adequate sampling, in authentic contexts, of the performance of complex competencies that cannot be broken down into simple parts, we need to make a shift from individual methods to an integral programme, intertwined with the education programme. Therefore, we need an instructional design perspective. IMPLICATIONS FOR DEVELOPMENT AND RESEARCH: Programmatic instructional design hinges on a careful description and motivation of choices, whose effectiveness should be measured against the intended outcomes. We should not evaluate individual methods, but provide evidence of the utility of the assessment programme as a whole.

Clinical Competence↗
PubMed2004

Education research at the Faculty of Medicine, University of Maastricht: fostering the interrelationship between professional and education practice.

An academic department of education serving the entire university and a strategic choice by the Faculty of Medicine to support educational innovation through education research are the historical cornerstones of the education research program of the University of Maastricht. Over the years, the department's initial exclusive research focus on the evaluation of problem-based learning has widened to include theory-based applied research covering the broad domain of education. The program focuses on themes: the learning of students and teachers, characteristics of powerful learning environments, and assessment and evaluation of learning and teaching. Although modest in terms of resources, the program is firmly anchored within the Faculty's organizational structure. Educational relevance and professional alignment are the most prominent determinants of the success of the program. These features sustain the institutional mission of educational excellence as well as the high ranking of the Faculty of Medicine's medical training program among the training programs of the Netherlands' medical schools. A break in this self-perpetuating mechanism--due either to internal politics or to staffing problems--forms the main risk factor for the continuation of the department.

Faculty, Medical↗
PubMed2004

Changing education, changing assessment, changing research?

BACKGROUND: In medical education, assessment of medical competence and performance, important changes have taken place in the last 5 decades. These changes have affected the basic concepts in all 3 domains. DEVELOPMENTS IN EDUCATION AND ASSESSMENT: In education constructivism has provided a completely new view on how students learn best. In assessment the change from trait-orientated to competency- or role-orientated thinking has given rise to a whole range of new approaches. Certain methods of education, such as problem-based learning (PBL), and assessment, however, are often seen as almost synonymous with the underlying concepts, and one tends to forget that it is the concept that is important and that a particular method is but 1 way of using a concept. When doing this, one runs the risk of confusing means and ends, which may hamper or slow down new developments. LESSONS FOR RESEARCH: A similar problem seems to occur often in research of medical education. Here too, methods--or, rather, methodologies--are confused with research questions. This may lead to an overemphasis on research that fits well known methodologies (e.g. the randomised controlled trial) and neglect of what are sometimes even more important research questions because they do not fit well known methodologies. CONCLUSION: In this paper we advocate a return to the underlying concepts and a careful reflection of their use in various situations.

Clinical Competence↗
PubMed2004

Different written assessment methods: what can be said about their strengths and weaknesses?

INTRODUCTION: Written assessment techniques can be subdivided according to their stimulus format--what the question asks--and their response format--how the answer is recorded. The former is more important in determining the type of competence being asked for than the latter. It is nevertheless important to consider both when selecting the most appropriate types. Some major elements to consider when making such a selection are cueing effect, reliability, validity, educational impact and resource-intensiveness. RESPONSE FORMATS: Open-ended questions should be used solely to test aspects that cannot be tested with multiple-choice questions. In all other cases the loss of reliability and the higher resource-intensiveness represent a significant downside. In such cases, multiple-choice questions are not less valid than open-ended questions. STIMULUS FORMAT: When making this distinction, it is important to consider whether the question is embedded within a relevant case or context and cannot be answered without the case, or not. This appears to be more or less essential according to what is being tested by the question. Context-rich questions test other cognitive skills than do context-free questions. If knowledge alone is the purpose of the test, context-free questions may be useful, but if it is the application of knowledge or knowledge as a part of problem solving that is being tested, then context is indispensable. CONCLUSION: Every format has its (dis)advantages and a combination of formats based on rational selection is more useful than trying to find or develop a panacea. The response format is less important in this respect than the stimulus.

Clinical Competence↗
PubMed2003

Controlled trial of effect of computer-based nutrition course on knowledge and practice of general practitioner trainees.

BACKGROUND: Nutrition education is not an integral part of either undergraduate or postgraduate medical education. Computer-based instruction on nutrition might be an attractive and appropriate tool to fill this gap. OBJECTIVE: The study objective was to assess the degree to which computer-based instruction on nutrition improves factual knowledge and practice behavior of general practitioner (GP) trainees. DESIGN: We carried out a controlled experimental study, using a 79-item knowledge test and 3 incognito standardized patients' visits in a pre- and posttest design with 49 first-year GP trainees. The experimental group (n = 25) received an average of 6 h of a newly developed computer-based instruction on nutrition. The control subjects (n = 24) took the standard vocational training program. RESULTS: The percentage of correct answers on the knowledge test increased from 30% at pretest to 42% at posttest in the experimental group, and from 36% to 37% in the control group. Analysis of covariance, with the pretest scores as covariate, showed a significant experimental versus control group difference at posttest: 9.2% (P = 0.002). The mean percentage of correctly performed items during the 3 standardized patients' visits (assessed by checklists) showed an increase in the experimental group from 20% at pretest to 36% at posttest, whereas the control group changed from 20% to 22%. Analysis of covariance, with the pretest scores as covariate, revealed a significant group difference at posttest: 13.7% (P < 0.001). CONCLUSION: The computer-based instruction proved its effectiveness, both by increasing factual knowledge and by substantially enhancing GP trainees' practice behavior on the subject of nutrition.

Adult↗
PubMed2002

Optimising the reproducibility of a performance-based assessment test in midwifery education.

Despite problems concerning generalisability of test results - largely due to limited sampling of the task domain - and questions about test efficiency, the popularity of OSCEs in medical education has motivated the use of similar assessment methods in other health care domains. Purpose of this study was to investigate reproducibility of scores on an OSCE-based test in the relatively narrow domain of midwifery, as compared to the broad medical domain. The influence of global rating scales and adoption of a mastery-oriented test perspective on reproducibility of test scores was investigated in order to explore possibilities to increase test efficiency.A 3-hr, 6-station performance based test was administered to third and fourth year students at Kerkrade Midwifery School (the Netherlands). Students' performance was recorded using a station specific checklist and a global rating scale. For the 3-hr OSCE the generalisability coefficient based on checklist scores is 0.48. Based on global ratings alone, the generalisability coefficient is 0.61. Adjusted dependability indices are 0.85 based on checklist scores and 0.95 for global ratings respectively. Results suggest that even for small domains the problem of case-specificity remains a major impediment to high stakes performance testing. Findings furthermore suggest that use of global rating scales and professional expert judgements in professional competence evaluation is to be preferred to task specific checklists for reasons of reproducibility and efficiency in test development and administration. Adoption of a mastery oriented test perspective may reduce testing time requirements even further.

Certification↗
PubMed2002

Procedures for establishing defensible programmes for assessing practice performance.

The assessment of the performance of doctors in practice is becoming more widely accepted. While there are many potential purposes for such assessments, sometimes the consequences of the assessments will be 'high stakes'. In these circumstances, any of the many elements of the assessment programme may potentially be challenged. These assessment programmes therefore need to be robust, fair and defensible, taken from the perspectives of consumer, assessee and assessor. In order to inform the design of defensible programmes for assessing practice performance, a group of education researchers at the 10th Cambridge Conference adopted a project management approach to designing practice performance assessment programmes. This paper describes issues to consider in the articulation of the purposes and outcomes of the assessment, planning the programme, the administrative processes involved, including communication and preparation of assessees. Examples of key questions to be answered are provided, but further work is needed to test validity.

Clinical Competence↗