Biomedical subjects
Publications and source records attributed to Clarence D Kreiter.
The academic performance consequences of relying solely on non-cognitive factors for selecting applicants above a GPA and MCAT threshold have not been fully considered in the literature. This commentary considers the impact of using a "threshold approach" on academic performance as assessed with the USMLE Step 1.
BACKGROUND: Grading standards vary widely across undergraduate institutions. If, during the medical school admissions process, GPA is considered without reference to the institution attended, it will disadvantage applicants from undergraduate institutions employing rigorous grading standards. METHOD: A regression-based GPA institutional equating method using historical MCAT and GPA information is described. Classes selected from eight applicant pools demonstrate the impact of the GPA adjustment. The validity of the adjustment is examined by comparing adjusted and unadjusted GPAs' correlation with USMLE and medical college grades. RESULTS: The adjusted GPA demonstrated significantly improved congruence with MCAT estimates of applicant preparedness. The adjustment changed selection decisions for 21% of those admitted. The adjusted GPA enhanced prediction of USMLE and medical school grades only for students from institutions which required large adjustments. CONCLUSION: Unlike other indices, the adjustment described uses the same metric as GPA and is based only on an institution's history of preparing medical school applicants. The institutional adjustment is consequential in selection, significantly enhances congruence with a standardized measure of academic preparedness and may enhance the validity of the GPA.
BACKGROUND: Researchers generally recommend a structured format for the medical school preadmission interview (MSPI). However, the relative benefits of various elements of structure remain unexamined. PURPOSE: In this study, we compared the performance of a highly structured interview format with a semistructured format. Specifically, we examined how the reliability of interview ratings is likely to change when using the same versus different questions for each applicant being interviewed. METHOD: Variance components from a generalizability (G) study of a structured interview are used in decision studies to compare the relative efficiency of using the same versus different questions for each applicant. RESULTS: Using different questions for each interviewee is practically as reliable as using the same questions for all applicants (G = .55 vs. .57, respectively). CONCLUSIONS: Because there are a number of drawbacks to using the same questions for all applicants (i.e., security and validity) and little advantage in terms of increased reliability, the semistructured question format should be considered when conducting the MSPI. A suggested method of implementing a semistructured interview is by presenting each applicant a set of questions randomly drawn from a pool of interview questions.
PURPOSE: The Script Concordance Test (SCT) is designed to measure cognitive ability related to successful clinical decision making. An SCT's usefulness for medical education depends on establishing its construct validity. The SCT's present construct relates examinee's scores to experts' response patterns, which does not require a single-best-answer format. Because medical education assessments do require a single best answer, the authors compared the psychometric properties of two aggregate scoring methods with three single-best-answer scoring methods for an SCT. METHOD: A nephrology SCT was developed and administered to 85 examinees. Examinees' scores derived from a key developed using eight experts and a traditional aggregate scoring method on a five-point Likert-based scale were compared with four alternate scoring methods (one method eliminated the multipoint Likert-type scale and three eliminated the Likert-type scale and employed single-best-answer scoring). RESULTS: Two of the four alternate scoring methods performed as well as the traditional Likert-type aggregate scoring method. Scores from all five methods were highly intercorrelated. In addition, each method produced scores similarly correlated with level of experience, and none exhibited an intermediate effect. CONCLUSIONS: Single-best-answer scoring with three answer choices produced results similar to aggregate scoring on a Likert-type scale. Because SCT items appear to assess an examinee's understanding of the interrelatedness of medical knowledge, single-best-answer scoring on an SCT may be valid as an educational assessment. More research is needed to assess differential validity compared with multiple-choice question exams and the predictive validity related to clinical performance.
PURPOSE: Surveys of medical students are widely used to evaluate course content and faculty teaching within the medical school. Gathering information that accurately reflects student perceptions requires that students buy into the evaluation process and be willing to provide thoughtful responses to the teaching evaluation. To maintain student commitment, it is important that medical students are not overburdened with poorly planned evaluations. Sampling might decrease the number of evaluations required of students and might also reduce the proportion of non-responses and other forms of inattentive response biases. METHODS: A sampling technique employed within a large medical lecture is described and evaluated. A generalisability study of the teacher evaluations is conducted. RESULTS: A high response rate and high levels of reliability were obtained by sampling a small proportion of the total class. The largest source of error was related to rater and utilising sufficient numbers of student-raters is critical to achieving reliable results. CONCLUSION: Sampling can reduce evaluation demands placed on students, and preserve reliability and increase the validity of mean evaluation scores. With computer presentation, efficient sampling techniques become practical and should be part of software packages used to present teacher evaluations.
BACKGROUND: The U.S. Supreme Court ruled in 2003 in the Gratz et al. v. Bollinger et al. and the Grutter v. Bollinger et al. cases that affirmative action selection policies aimed at student diversity in higher and professional education are acceptable on constitutional grounds. The court also ruled that "holistic," "individualized" selection procedures, not "mechanical" methods, must be used to achieve student diversity goals. SUMMARY: This brief essay reviews and critiques professional and academic literature that addresses the two court cases and their likely sequelae. The weight of scientific evidence dating from at least 1954 shows that actuarial procedures are superior to holistic, individualized procedures to fulfill a priori student selection policy goals fairly and consistently. CONCLUSIONS: The higher and professional education community, including medicine, should support the Supreme Court's student diversity policy but should insist the court change its reliance on holistic, individualized decision making.
BACKGROUND AND OBJECTIVES: Identification of reliable methods to evaluate the newly mandated American Board of Medical Specialties (ABMS)/Accreditation Council for Graduate Medical Education (ACGME) competencies of the board-certified physician is in its early stages. In this study, we evaluated a comprehensive faculty peer evaluation system designed to assess the six competencies as well as faculty performance in their primary departmental roles and teaching. METHODS: Using a one-page form containing 19 items, all faculty members evaluated all other faculty within a single department. Annual individual faculty reviews included discussion of these aggregated evaluations. RESULTS: The reliabilities for the ACGME competency subscales ranged from .61 to .79. While overall scores were relatively high, there was variability across faculty. Factor analysis demonstrated that evaluation items load onto three scales. The first relates to clinical practice and teaching, the second to departmental citizenship, and the third to research. An item related to systems-based practice loaded on none of the factors. Research faculty outscored other faculty on the items reflecting research skills. Faculty who had primary administrative responsibility scored higher than other faculty on measures related to role within the department. No differences in subgroup scores for clinical skills were observed. CONCLUSIONS: Using a method in which all faculty evaluate each other can result in objective, reliable measures of faculty performance.
BACKGROUND: Information prescriptions (IPs) are prescriptions of specific, evidence-based information to manage health problems. OBJECTIVE: To determine whether a pediatrician-provided IP would change parents' attitudes/behaviors about using Internet health information resources. DESIGN, SETTING, AND PARTICIPANTS: We conducted a randomized controlled trial of parents visiting an academic general pediatric practice. Both groups received a preintervention paper survey and a telephone survey 2 to 3 weeks after their clinic visit. The intervention group was offered computer training and received the IP and training summary handout. RESULTS: Final groups included 100 control and 97 intervention subjects. Of all parents, 68.0% had used the Internet in the past 6 months for health information and 52.8% used it for children's health information. The intervention group used the Internet more for general health information (P =.05) and child health information (P<.001) than the control group. At follow-up, 51 (66.2%) of 77 Internet information resources used by the intervention group were prescribed by the pediatricians. Parents who used the IP (31 [32%] of 97 intervention parents) used the Internet for general health information (P<.001) and child health information (P =.001) more than nonusers. Compared with nonusers, IP users were more likely to state they would use the IP again in the future (P =.02) and had already recommended the IP to family or friends more than nonusers (P =.001). CONCLUSIONS: Parents of children in pediatric practices commonly use the Internet for general and children's health information. In this study, IPs were associated with specific parental attitude and behavior changes resulting in increased Internet utilization for general and child health information and for specific high-quality information resources. Pediatricians can implement IPs in their office.
PURPOSE: To examine the validity of using scores from a clinical evaluation form as an assessment of clinical competence. METHOD: Investigators collected a longitudinal clinical skills assessment database that included scores reflecting performance on standardized patient interactions, case-based learning performance, scores on multiple-choice clinical examinations, and preceptors' ratings of students during their clerkships. Pearson correlation coefficients and coefficients corrected for attenuation were calculated between the mean preceptor rating CEF score and the other measures collected during years one and two. RESULTS: Estimates from an earlier across-clerkship generalizability study of the CEF demonstrated that a mean rating computed across 28 forms and four clerkships (the average measurement frequencies observed in this study), yielded an estimated G-coefficient of 0.62 (Kreiter and Ferguson, 2001). Reliabilities for the other measures ranged from 0.23 to 0.56. Nine of the twelve clinical skill measures correlated with the CEF. For those significantly correlated measures where reliabilities could be calculated, correlations corrected for attenuation ranged from 0.46 to 0.58. CONCLUSION: This study indicates that the skills measured by the CEF are related to other clinical performance measures and, conversely, that pre-clinical measures of skills that are believed to be important during clinical years are in fact predictors of preceptors' ratings of clinical performance later in medical school. In addition, the magnitude of the disattenuated coefficients suggests that ratings on the CEF are dependent on important aspects such as clinical knowledge. This study suggests that when averaged over a large number of observations, mean CEF scores demonstrate validity coefficients large enough to support their use as part of an evaluation of students' clinical performance.
PURPOSE: Determining the valid and fair use of the interview for medical school admissions is contingent upon a demonstration of the reproducibility of interview scores. This study seeks to establish the generalizability of interview scores, first assessing the existing research evidence, and then analyzing data from a non-experimental independent replications research design. METHODS: Multivariate and univariate generalizability analyses are conducted using data from a structured interview obtained from a population of medical school applicants over two years. RESULTS: The existing literature does not provide sufficient evidence regarding interview reliability. In this study, interview scores derived from a standardized interview were found to display low to moderate levels of reliability. Interview scores do not appear to possess the level of precision found with other measures commonly used to facilitate admissions decisions. DISCUSSION/CONCLUSION: Given the results obtained, the fairness of using the interview as a highly influential component of the admission process is called into question. Methods for using interview data in a psychometrically defensible fashion are discussed. Specifically, attention to decision reliability provides guidance on how interview scores can best be integrated into the admissions process.
BACKGROUND: Performance-based assessments are often not reliable enough to be used as the sole method for determining a grade. PURPOSE: Using more than 1 assessment format has several advantages and can improve the quality of the information used to calculate course grades. To achieve a valid and reliable total score representing a combination of multiple assessment formats requires appropriate weighting procedures. METHODS: The classical test theory rationale and methodology for appropriately weighting in-course assessments is presented. Two reliablility-related equations are used to make decisions about weighting for a computerized performance assessment and a pattern recognition multiple-choice exam. RESULTS: Using the equations presented, the outcomes of various weighting scenarios are graphically presented. This technique produced weights that allow the instructor to obtain acceptable reliability while retaining a substantial emphasis on performance assessment. CONCLUSION: The weights assigned to component tests used to derive a total score have important reliability and validity implications. Course instructors need to consider both empirical reliability and logical validity evidence in determining component weights. When used in conjunction with classical methods, objectively scored item formats can augment performance assessments and enhance overall validity and reliability.
OBJECTIVE: Usage of computer resources at the point of care has a positive effect on physician decision making. Pediatricians' information-seeking behaviors are not well characterized. The goal of this study was to characterize quantitatively the information-seeking behaviors of general pediatricians and specifically compare their use of computers, including digital libraries, before and after an educational intervention. METHODS: General pediatric residents and faculty at a US Midwest children's hospital participated. A control (year 1) versus intervention group (year 2) research design was implemented. Eligible pediatrician pools overlapped, such that some participated first in the control group and later as part of the intervention. The intervention group received a 10-minute individual training session and handout on how to use a pediatric digital library to answer professional questions. A general medical digital library was also available. Pediatricians in both the control and the intervention groups were surveyed using the critical incident technique during 2 6-month time periods. Both groups were telephoned for 1- to 2-minute interviews and were asked, "What pediatric question(s) did you have that you needed additional information to answer?" The main outcome measures were the differences between the proportion of pediatricians who use computers and digital libraries and a comparison of the number of times that pediatricians use these resources before and after intervention. RESULTS: A total of 58 pediatricians were eligible, and 52 participated (89.6%). Participant demographics between control (N = 41; 89.1%) and intervention (N = 31; 70.4%) were not statistically different. Twenty pediatricians were in both groups. Pediatricians were slightly less likely to pursue answers after the intervention (94.7% vs 89.2%); the primary reason cited for both groups was a lack of time. The pediatricians were as successful in finding answers in each group (95.7% vs 92.7%), but the intervention group took significantly less time (8.3 minutes vs 19.6 minutes). After the intervention, pediatricians used computers and digital libraries more to answer their questions and spent less time using them. CONCLUSION: This study showed higher rates of physician questions pursued and answered and higher rates of computer use at baseline and after intervention compared with previous studies. Pediatricians who seek answers at the point of care therefore should begin to shift their information-seeking behaviors toward computer resources, as they are as effective but more time-efficient.
BACKGROUND: The rationale for treatment of patients with port-wine stain (PWS) birthmarks has focused on the belief that eventual soft-tissue hypertrophy and negative psychological effects might be diminished or avoided. Pulsed-dye laser (PDL) therapy has become widely accepted as the treatment of choice for PWS, and a variety of studies have been conducted that evaluate its short-term efficacy. Long-term data regarding outcome and patient satisfaction are lacking. OBJECTIVE: Our objective was to evaluate the long-term efficacy of pulsed-dye laser therapy from the patient's perspective. METHODS: A survey was mailed to all patients or parents of minor patients whose last known laser treatment was performed between January 1, 1989, and January 1, 1996. Patients were asked to quantify changes in their PWS as well as their psychological well-being. A formal qualitative analysis was performed on their written comments as well. RESULTS: A total of 164 surveys were mailed to the most recent address available by the patient record. Fifty-two surveys were returned as undeliverable, and 55 evaluable surveys were returned, yielding a 49% overall return rate for patients who could be located. The mean age of respondents was 28.8 years, 96% of whom were white and 62% female. The mean number of years since last treatment was 7.04. The vast majority of patients noted little or no change in texture, height, or dimension of their PWS, whereas 62% noted color improvement. A majority or patients (60%) worried less about their appearance after treatment, whereas a similar number (61%) believed their ability to make friends or meet others was unaffected by treatment. Only 19% thought others looked at or treated them differently because of their PWS. Overall, 48% of patients indicated satisfaction with treatment, 24% dissatisfaction, and 28% neutral. On a 10-point scale indicating their likelihood of recommending treatment to someone similarly affected, the mean score was 7.42. Men were significantly and consistently less satisfied with treatment than women, despite rating the degree of color improvement similarly to women. Qualitative analysis of patient comments helped validate quantitative data and revealed gender differences in satisfaction, as well as correlations with adverse events from treatment, desire for additional treatment, and a trend toward more positive comments with the passage of time. CONCLUSION: The pulsed-dye laser improves the color of PWS over long periods of time in a majority of patients. Patients tended to worry less about their appearance after treatment, although most believed treatment did not substantially affect their relationship with others or others' view of them. Most patients were satisfied or neutral with regard to satisfaction with therapy and would recommend treatment to others. A minority of patients was dissatisfied with treatment, and men were more likely to be dissatisfied. Additional long-term and prospective studies will be helpful in assessing the physical and psychosocial impact of PDL for PWS.
PURPOSE: The underrepresentation of certain minorities within medical education and the medical profession continues to be a problem. A review of the relevant research literature suggests current strategies are inadequate to address this important problem. Psychometric issues important in differentiating the unique concerns of medical education must be defined. SUMMARY: A new model that may attain diversity goals and meet standards related to validity and legality is presented. Admissions data describing applicants for one year at a large midwestern medical college are analyzed. The impact of selection techniques on both majority and underrepresented minority applicants is presented. CONCLUSION: The results of the analyses support further research designed to meet the quantitative objectives implied by diversity goals and suggests initiatives aimed at enhancing minority representation within medical education.
OBJECTIVE: We developed an instructional program to teach aspiration and injection techniques of the knee and shoulder to medical students and residents. METHODS: Residents and fourth-year medical students participating in a rheumatology elective were assigned by deterministic allocation into 3 groups: the Traditional group received no specific instruction in arthrocentesis but simply rotated through rheumatology, learning injection techniques only if they saw patients who required them; the Lecture-only group received only the didactic lecture and did not have the opportunity to practice on the models; the Program group participated in the newly developed program of instruction that combined a didactic lecture and a hands-on workshop using the anatomic models to practice arthrocentesis techniques. RESULTS: The scores on the written examination for those in the Program group (mean score 37.46 out of 40 possible) and the Lecture-only group (mean 37.75) were significantly higher than those of the Traditional group (mean 33.15) (P <.05). The scores on the practical examination for those in the Program group (mean score 24.08 out of 26 possible) were significantly higher than those of the Lecture-only (mean 20.50) and Traditional (mean 17.33) (P <.05) CONCLUSION: The addition of this type of instruction to supplement a traditional internal medicine rotation can enhance a learner's ability to perform joint/soft-tissue injection and aspiration.
Weighting models do not adequately represent admission goals. A constrained optimization (CO) decision-analysis model is an alternative that allows a more useful and accurate translation of goals in the admission process. CO allows easy translation of class composition objectives into the model. The author used simulated data to demonstrate the CO technique within a medical-school-admission context. Variables related to class diversity and individual achievement were used to select a class from a simulated final applicant pool. Constraints in the model were evaluated for their effects on the class. The model performed well and yielded significant advantages over weighting equations.
PURPOSE: Many preceptors fill out students' clerkship-performance forms with all items marked with the maximum value. Because items are usually designed to measure somewhat specific behaviors, the authors questioned the validity of straight-line five (SL5) responses, and conducted an empirical validity study to investigate the measurement properties of these SL5 forms. METHOD: In 1999-00 the authors undertook a correlation study and a generalizability study to assess the measurement characteristics of the SL5 forms. For the correlation study, they calculated mean scores for each of 168 students across forms, excluding the SL5 forms, then correlated this mean with the proportion of straight-line forms the student received. They conducted a generalizability analysis with and without the straight-line forms to determine the impact on score reliability. RESULTS: The proportion of SL5 forms was significantly correlated (r =.48, p <.0001) with the mean student score. Inclusion of the SL5 forms did not negatively affect the reliability of the mean score. CONCLUSION: The SL5 forms appear to be valid ratings of students' performances. An informal comparison of the comments on SL5 forms and those not displaying this format supported this conclusion.
Explore the source record for details and available documents.