SEARCH · PubMed Health

Results for “Reliability”

Explore indexed PubMed citations for clinical trials, systematic reviews and public health research. Read source abstracts and follow each citation to its original PubMed record.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 recordsLinked to original sources

PubMed2003

A breakdown of reliability coefficients by test type and reliability method, and the clinical implications of low reliability.

The author presented descriptive statistics for 937 reliability coefficients for various reliability methods (e.g., alpha) and test types (e.g., intelligence). He compared the average reliability coefficients with the reliability standards that are suggested by experts and found that most average reliabilities were less than ideal. Correlations showed that over the past several decades there has been neither a rise nor a decline in the value of internal consistency, retest, or interjudge reliability coefficients. Of the internal consistency approaches, there has been an increase in the use of coefficient alpha, whereas use of the split-half method has decreased over time. Decision analysis and true-score confidence intervals showed how low reliability can result in clinical decision errors.

Decision Support Techniques↗

PubMed2004

Exploring variability and sources of measurement error in alcohol expectancy questionnaire reliability coefficients: a meta-analytic reliability generalization study.

OBJECTIVE: Reliability Generalization, a relatively new meta-analytic score reliability technique, was used to examine reliability coefficients for Alcohol Expectancy Questionnaire (AEQ) subscales in published research. Specifically, the present study identified the typical reliability coefficients of scores on AEQ subscales across published studies and examined sources of measurement error across AEQ subscales and studies. METHOD: Based on literature reviews of PsyclNFO and PubMed, a total of 71 studies were analyzed and coded on 10 different dimensions believed to affect score reliability. RESULTS: Only 37% of the studies surveyed reported reliability coefficients for the data in hand. Analysis of these studies revealed that the average score reliability across studies varied considerably in subscales and samples, with the Arousal/Interpersonal Power and Sexual Enhancement subscales evidencing the greatest variability. Gender homogeneity and racial homogeneity were found to be the two most important predictors of the magnitude of score reliability coefficients across subscales and studies. CONCLUSIONS: Results of the study indicate that the AEQ tends to generate reliable scores, with some noted exceptions. Because the Arousal/Interpersonal Power and Sexual Enhancement subscales tend to generate unacceptably low reliability coefficients and because the AEQ holds promise in furthering prevention and treatment outcome research, it is suggested that future research with the AEQ explore instrument factor structure and item consistency. Because reliability estimates can influence substantive statistical analyses, it is recommended that authors using the AEQ report reliability information in their published work.

Alcohol Drinking↗

PubMed2006

Making noncatastrophic health care processes reliable: Learning to walk before running in creating high-reliability organizations.

Health care clinicians successfully apply proven medical evidence in common acute, chronic, or preventive care processes less than 80 percent of the time. This low level of reliability at the basic process level means that health care's efforts to improve reliability start from a different baseline from most other industries, and therefore may require a different approach. This paper describes The Institute for Healthcare Improvement's (IHI) current approach to improving health care reliability, including a useful nomenclature for levels of reliability, and a focus on improving reliability of basic health care processes before moving on to more sophisticated high reliability organization concepts. Early IHI work with a community of health care reliability innovators has identified four themes in health care settings that help to explain at least a portion of the gap in process reliability between health care and other industries. These include extreme dependence on hard work and personal vigilance, a focus on mediocre benchmark outcomes rather than process, great tolerance of provider autonomy, and failure to create systems that are specifically designed to reach articulated reliability goals. This paper describes our recommendations for the initial steps health care organizations' might take, based on these four themes, as they begin to move toward higher reliability.

Benchmarking↗

PubMed2000

Increasing the reliability of Wechsler Intelligence Scale for Children--Third Edition difference scores with reliable component analysis.

Wechsler Intelligence Scale for Children--Third Edition index score differences are generally interpreted cautiously, if at all, primarily because of their poor reliability. On the basis of prior analyses with the Wechsler Adult Intelligence Scale--Third Edition (J. C. Caruso & N. Cliff, 1999), it was hypothesized that differences between scores defined by reliable component analysis would have higher reliability than those defined by traditional equal weighting. Differences between the reliable component scores showed substantially higher reliability than equally weighted score differences. The differences between reliable component scores were also substantially more reliable than those derived from the weighted scores suggested by K. C. H. Parker and L. Atkinson (1994). Using the weights provided in this article will allow researchers and practitioners to compute the RCA scores and have the assurance of high reliability with its attractive consequences.

Adolescent↗

PubMed2006

High-Level Mobility Assessment Tool (HiMAT): interrater reliability, retest reliability, and internal consistency.

BACKGROUND AND PURPOSE: The High-Level Mobility Assessment Tool (HiMAT) assesses high-level mobility in people who have sustained a traumatic brain injury (TBI). The purpose of this study was to investigate the interrater reliability, retest reliability, and internal consistency of data obtained with the HiMAT. SUBJECTS: Three physical therapists and 103 people with TBI were recruited from a rehabilitation hospital. METHODS: Three physical therapists concurrently assessed a subset of 17 subjects with TBI to investigate interrater reliability. One physical therapist assessed a different subset of 20 subjects with TBI on 2 occasions, 2 days apart, to investigate retest reliability. Data from the entire sample of 103 subjects were used to investigate the internal consistency of this new scale. RESULTS: Both the interrater reliability (intraclass correlation coefficient [ICC]=.99) and the retest reliability (ICC=.99) of the HiMAT data were very high. For retest reliability, a small systematic change was detected (t=3.82, df=19), indicating a marginal improvement of 1 point at retest. Internal consistency also was very high (Cronbach alpha=.97). DISCUSSION AND CONCLUSION: The HiMAT is a new tool specifically designed to measure high-level mobility, which currently is not a component of existing scales used in TBI. This study demonstrated that the HiMAT is a reliable tool for measuring high-level mobility.

Adult↗

PubMed2004

Increasing the reliability of ipsative interpretations in neuropsychology: a comparison of reliable components analysis and other factor analytic methods.

Ipsative approaches to neuropsychological assessment typically involve interpreting difference scores between individual test scores. The utility of these methods is limited by the reliability of neuropsychological difference scores and the number of comparisons between scores. The present study evaluated the utility of difference scores using factor analytic methods, including reliable components analysis (RCA), equally weighted composites and individual neuropsychological measures. Data from 1,364 individuals referred for neuropsychological assessment were factor analyzed and the resulting solutions were used to compute composite scores. Reliabilities and confidence intervals were derived for each method. Results indicated that RCA outperformed other factor analytic methods, but produced a slightly different factor structure. Difference scores derived using orthogonal solutions were slightly more reliable than oblique methods, and both were more reliable than those from equally weighted composites and individual measures. Confidence intervals for difference scores were considerably smaller for factor methods relative to those for individual test comparisons, due to the greater reliability of factor based difference scores and the smaller number of comparisons required. These findings suggest that difference scores derived from orthogonal factor solutions, particularly RCA solutions, may improve reliability for clinical assessment purposes.

Adolescent↗

PubMed2000

Reliability of computerized perimetric threshold tests as assessed by reliability indices and threshold reproducibility in patients with suspect and manifest glaucoma.

BACKGROUND: High reproducibility of test measurements is often considered an indication of high reliability of test results. The aim of the current study was to analyse the role of the traditional perimetric reliability indices, False Negative and False Positive responses, and Fixation Losses, as indicators of test reliability in comparison with threshold reproducibility in patients with suspect or manifest glaucoma. METHODS: Perimetry was performed in one eye in each of 76 patients. Each eye was tested twice within approximately one week using the Humphrey II 30-2 SITA STANDARD program. Frequencies of False Positive and False Negative answers, and rates of Fixation Losses were related to threshold reproducibility and to general field status as expressed by Mean Deviation from age-normal threshold values using stepwise multiple linear regression analysis. RESULTS: Substantial field loss was associated both with low threshold reproducibility (p<0.0001) and with increased frequency of False Negative answers (p=0.047). The traditional reliability indices contributed marginally compared to amount of field loss when predicting threshold reproducibility; the coefficient of determination decreased non-significantly from 0.37 to 0.33 when excluding the three reliability indices as explanatory variables from the regression model. Frequencies of False Positive answers and Fixation Losses showed no association to field status or to threshold reproducibility. CONCLUSION: Reliability of visual field test results in patients with glaucoma expressed as threshold reproducibility, can be predicted by amount of field loss alone, and traditional patient reliability indices contribute surprisingly little in this regard.

Aged↗

PubMed2003

Zebris versus Myrin: a comparative study between a three-dimensional ultrasound movement analysis and an inclinometer/compass method: intradevice reliability, concurrent validity, intertester comparison, intratester reliability, and intraindividual variability.

STUDY DESIGN: Experimental study. OBJECTIVES: To compare two devices for measuring cervical range of motion, a three-dimensional ultrasound motion device (Zebris) and a gravity-reference goniometer (Myrin). SUMMARY OF BACKGROUND DATA: Assessment of cervical range of motion is used to evaluate the effect of different treatments, determine impairment, and ascertain the relationship between neck disorders and cervical spine mobility. METHODS: Sixty "neck-healthy" volunteers (25 men, 35 women; mean age 38 years, range 22-58 years) performed active maximal movements in flexion-extension, rotation, and lateral flexion. Maximal cervical range of motion was recorded simultaneously with the Zebris and Myrin devices. Intradevice reliability, concurrent validity, intertester comparison, intratester reliability, and intraindividual variability were computed. RESULTS: Our study showed good agreement of full-cycle cervical range of motion measurement between devices, testers, and the test and retest (intraclass correlation [ICC] was >0.90 for intradevice reliability, >0.93 for concurrent validity, and >0.92 for intratester reliability). Method error, assessed with the within-subject coefficient of variation for 95% of the measurements, was 5.4% to 11.1% for intradevice reliability, 4.4% to 7.6% for concurrent validity, 3.6% to 7.6% for intratester reliability, and 5.3% to 9.9% for individual variability. Individual variability did not increase with an increased cervical range of motion. CONCLUSION: Both devices are reliable and showed good agreement. We conclude that the two techniques can be used interchangeably. Our study supports the continued use of the Myrin-a gravity-reference goniometer in routine clinical orthopedic work. The more sophisticated three-dimensional method adds information and allows evaluation of combined motion in two and three dimensions and is suitable for research.

Adult↗

PubMedDate not supplied

Maintaining reliability in a long-term psychiatric study: an ongoing inter-rater reliability monitoring program using the longitudinal interval follow-up evaluation.

The Longitudinal Interval Follow-up Evaluation (LIFE), has been shown to be a valid and reliable instrument for characterizing the week-by-week course of anxiety disorders examined retrospectively over the period of 1 year. Due to the chronic nature of these disorders, there is a need for reliable, valid instruments for measuring course over periods of several years if we are to learn more about the natural history of these disorders. This paper describes a rater-monitoring program designed to ensure long-term inter-rater reliability and prevent "rater drift". In this program, clinical interviewers score taped interviews and are required to maintain a median intra-class correlation coefficient (ICC) of at least 0.80 with the other raters. Raters also assess tapes from previous years, to ensure that they are using the same diagnostic criteria as earlier generations of interviewers. A reliability study was conducted to compare psychiatric status ratings (PSRs) collected using biweekly telephone interviews with the semi-annual interviews. The ICCs for panic, agoraphobia, social phobia, and generalized anxiety disorder were very good to excellent. Another reliability study examined the PSRs of subjects who had been previously lost to follow-up. ICCs for panic, agoraphobia, generalized anxiety disorder and depression were good to excellent. These results show that the LIFE, when used in conjunction with an intensive training and rater monitoring system, is a reliable instrument for use in longitudinal studies of the course of anxiety disorders.

Adult↗

PubMed2001

Reliability generalization: exploring reliability variations on MMPI/MMPI-2 validity scale scores.

The study utilized the measurement meta-analytic method called "reliability generalization" to examine (a) the typical reliability, (b) the variability of score reliability, and (c) the factors predicting variability of score reliability on the three MMPI validity scales. Across studies the reliability coefficients on the three validity scales ranged from .16 to .93, with the F scale having the most variability, as might be expected. Regression analyses suggested that the age of participant best predicted the variation of reliability coefficients.

Female↗

PubMed2003

Test-retest reliability, practice effects and reliable change indices for the recognition memory test.

OBJECTIVES: The Recognition Memory Test (RMT) is widely used; however, important characteristics such as reliability and stability over time were largely unknown. In this study, we document for the first time test-retest reliabilities, practice effects, and Reliable Change (RC) indices for this test. DESIGN: A sample of 206 normal adults (aged 40-70) were administered twice either the same version or two different versions of the RMT, with a 1-month interval between assessments. The normal sample was split into two groups; a young (aged 40-54) and an old (55-70) group. RESULTS: Test-retest reliabilities were modest when using either the same versions or different versions. Practice effects were abolished when different versions of the RMT were used. In contrast, practice effects were clearly present on the same version of the non-verbal subtest for both control groups. However, practice effects were present on the same versions of the verbal subtest only in the old group. RC indices were rather large when using the same or different versions. CONCLUSION: Although modest, the test-retest reliability of the RMT is no worse than those reliabilities reported for other commonly used recall memory tests. Thus, the inherent clinical advantages of using a recognition paradigm make its use desirable. Usage of different versions of the RMT enables us to avoid practice effects. However, the RC indices indicate that large changes in scores are needed to detect a significant improvement or decline in an individual's performance.

Adult↗

PubMed1995

How to evaluate intraexaminer reliability using an interexaminer reliability study design.

Examiner reliability is often investigated to make generalizations about a profession's performance with various diagnostic tests. Although the evaluation of interexaminer reliability is straightforward, assessment of intraexaminer reliability can be problematic. Estimations of intrarater reliability can be inflated due to correlated error and the difficulty in blinding the examiners. A statistical method is presented that permits the investigator to compute intraexaminer as well as interexaminer reliability and precision from the findings of an interexaminer reliability investigation. In this approach, intraclass correlation coefficients are constructed from variance components estimated from a simple repeated-measures design: each of two or more raters evaluates each subject one time. The method can be applied to nominal and ordinal as well as interval data. Not only can spurious results be avoided, but time and funds can be saved by reducing the required number of subject ratings.

Analysis of Variance↗

PubMed2004

Toward reliable operative assessment: the reliability and feasibility of videotaped assessment of laparoscopic technical skills.

BACKGROUND: Decision making on the competency of surgical trainees to perform laparoscopic procedures has been hampered by the lack of reliable methods to evaluate operative performance. The goal of this study was to develop a feasible and reliable method of evaluation. METHODS: Twenty-nine senior surgical residents were videotaped performing a low anterior resection and a Nissen fundoplication in a pig. Ten blinded laparoscopists rated the videos independently on two scales. Rating time was minimized by allowing raters to fast-forward through the tapes at their discretion. Interrater reliability and the time required to rate a procedure were assessed. RESULTS: Rating time per procedure was a median of 15 min (range, 6-40). The mean interrater reliability for the two scales was 0.74. CONCLUSIONS: The use of videotapes of operations enabled multiple raters to assess a performance reliably and shortened assessment times by 80%. This assessment technique shows potential as a means of evaluating the performance of advanced laparoscopic procedures by surgical trainees.

Clinical Competence↗

PubMed1998

Long-term mechanical circulatory support system reliability recommendation: American Society for Artificial Internal Organs and The Society of Thoracic Surgeons: long-term mechanical circulatory support system reliability recommendation.

Jointly developed by members of the American Society for Artificial Internal Organs and the Society of Thoracic Surgeons along with staff from the Food and Drug Administration, the National Heart, Lung and Blood Institute and other experts, this recommendation describes the reliability considerations and goals for Investigational Device Exemption and Premarket Approval submissions for long-term, mechanical circulatory support systems. The recommendation includes a definition of system failure, a discussion of an appropriate reliability model, a suggested in vitro reliability test plan, reliability considerations for animal implantation tests, in vitro and animal in vivo performance goals, the qualification of design changes during the Investigational Device Exemption clinical trial, the development of a Failure Modes Effects and Criticality Analysis, and the reliability information for surgeons and patient candidates. The document will be periodically reviewed to assess its timeliness and appropriateness within five years.

Animals↗

PubMedDate not supplied

Long-term mechanical circulatory support system reliability recommendation: American Society for Artificial Internal Organs and Society of Thoracic Surgeons: long-term mechanical circulatory support system reliability recommendation.

Clinical Trials as Topic↗

PubMed2006

Identifying reliable change in tactile temporal thresholds in multiple sclerosis: test-retest reliability.

BACKGROUND: Tactile temporal thresholds are typically significantly higher (ie, prolonged) in multiple sclerosis (MS) patients when compared to controls and increase significantly during relapses, probably reflecting integrity of conduction across a portion of the corpus callosum. As part of an ongoing validation study of tactile temporal thresholds, the test-retest reliability of these thresholds was examined in patients with MS. METHODS: Tactile temporal thresholds were measured in 61 MS patients during two separate test sessions within three weeks. Test-retest reliability and the standard error of measurement were calculated. The threshold of change in tactile temporal thresholds in MS patients that would correspond to real change beyond measurement error with 95% certainty was also calculated. RESULTS: The test-retest reliability of this measure of tactile temporal thresholds was 0.93. The threshold indicating change beyond chance or measurement error with 95% certainty was 19 ms. CONCLUSIONS: This measure of tactile temporal thresholds has excellent test-retest reliability and a change of greater than 19 ms is highly likely to represent real change. This measure is promising as a precise, reliable outcome measure in MS.

Adult↗

PubMed2004

Expanding reliability generalization: confidence intervals and Charter's combined reliability coefficient.

Methods of reliability generalization characterize the reliability of a scale's scores across studies. Charter in 2003 presented useful formulae for computing combined estimates of reliability. The present article illustrates use of Charter's formulae and application of confidence intervals for reliability coefficients, including the combined estimate. Researchers may find these methods for reliability generalization useful.

Confidence Intervals↗

PubMed2003

[Reliability of measurement and the methods of estimating reliability].

OBJECTIVE: To explore the methods of estimating measurement reliability and their limitations. METHODS: According to the given examples, reliabilities of measurement were calculated and limitations of the methods of estimating reliability were analyzed. RESULTS: The Kappa value of interobserver reliability was 0.793 between two pathologists. Between the two populations with different prevalence rates, the values of Kappa were 0.800 and 0.137 respectively, and with big difference. Cronbach's alpha coefficient of compositive index for A type behavior was 0.55. CONCLUSION: The Kappa index and alpha coefficient were both inherently population-specific. Before generalizing to different populations, the reliability needs to be measured.

Algorithms↗