PubMed Health⌕ Search
Home/Archive/PubMed

PubMed · 16431865

Assessing the reliability of standardized performance indicators.

Scott C Williams·Ann Watt·Stephen P Schmaltz·Richard G Koss·Jerod M Loeb
View original source ↗Export metadata ↓

Abstract

OBJECTIVE: . To investigate the reliability of self-reported standardized performance indicators introduced by the Joint Commission on Accreditation of Healthcare Organizations in July 2002 and implemented in approximately 3400 accredited US hospitals. The study sought to identify the most common data quality problems and determine causes and possible strategies for resolution. DESIGN: Data were independently reabstracted from a random sample of 30 hospitals. Reabstracted data were compared with data originally abstracted, and discrepancies were adjudicated with hospital staff. Structured interviews were used to probe possible reasons for abstraction discrepancies. RESULTS: The mean data element agreement rate for the 61 data elements evaluated was 91.9%, and the mean kappa statistic for binary data elements was 0.68. The rate of agreement for individual data elements ranged from 100 to 62.4%. The mean difference between calculated indicator rates was 4.88% (absolute value) and the range of differences was 0.0-13.3%. Symmetry of disagreement among original abstractors and reabstractors identified eight indicators whose differences in calculated rates were statistically unlikely to have occurred through random chance (P < 0.05). CONCLUSION: Although improvement in the accuracy and completeness of the self-reported data is possible and desirable, the baseline level of data reliability appears to be acceptable for indicators used to assess and improve hospital performance on selected clinical topics.

Explore related subjects

HospitalsInterviews as TopicJoint Commission on Accreditation of Healthcare OrganizationsQuality Indicators, Health CareSelf DisclosureUnited StatesJournal ArticleResearch Support, U.S. Gov't, P.H.S.Validation Study

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Scott C Williams, Ann Watt, Stephen P Schmaltz, Richard G Koss, Jerod M Loeb. 2006-01-23. Assessing the reliability of standardized performance indicators.. https://doi.org/10.1093/intqhc%2Fmzi098

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related citations

PubMed2006

Methods for identifying surgical wound infection after discharge from hospital: a systematic review.

BACKGROUND: Wound infections are a common complication of surgery that add significantly to the morbidity of patients and costs of treatment. The global trend towards reducing length of hospital stay post-surgery and the increase in day case surgery means that surgical site infections (SSI) will increasingly occur after hospital discharge. Surveillance of SSIs is important because rates of SSI are viewed as a measure of hospital performance, however accurate detection of SSIs post-hospital discharge is not straightforward. METHODS: We conducted a systematic review of methods of post discharge surveillance for surgical wound infection and undertook a national audit of methods of post-discharge surveillance for surgical site infection currently used within United Kingdom NHS Trusts. RESULTS: Seven reports of six comparative studies which examined the validity of post-discharge surveillance methods were located; these involved different comparisons and some had methodological limitations, making it difficult to identify an optimal method. Several studies evaluated automated screening of electronic records and found this to be a useful strategy for the identification of SSIs that occurred post discharge. The audit identified a wide range of relevant post-discharge surveillance programmes in England, Scotland and Wales and Northern Ireland; however, these programmes used varying approaches for which there is little supporting evidence of validity and/or reliability. CONCLUSION: In order to establish robust methods of surveillance for those surgical site infections that occur post discharge, there is a need to develop a method of case ascertainment that is valid and reliable post discharge. Existing research has not identified a valid and reliable method. A standardised definition of wound infection (e.g. that of the Centres for Disease Control) should be used as a basis for developing a feasible, valid and reliable approach to defining post discharge SSI. At a local level, the method used to ascertain post discharge SSI will depend upon the purpose of the surveillance, the nature of available routine data and the resources available.

Hospitals↗