SEARCH · PubMed Health

Results for “interreviewer agreement”

Explore indexed PubMed citations for clinical trials, systematic reviews and public health research. Read source abstracts and follow each citation to its original PubMed record.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

8 recordsLinked to original sources

PubMed2006

Distinguishing benign from malignant pulmonary nodules with helical chest CT in children with malignant solid tumors.

PURPOSE: To retrospectively assess whether computed tomographic (CT) findings can indicate the benign or malignant nature of pulmonary nodules in pediatric patients with malignant solid primary tumors. MATERIALS AND METHODS: With institutional review board approval, waived parental and patient consent, and HIPAA compliance, the authors determined the incidence of malignancy among 81 pulmonary nodules that were sampled at biopsy within 3 weeks after chest CT (January 1999 to September 2003) in 41 young patients with malignant solid tumors. Three radiologists independently and retrospectively reviewed these scans and the available previously obtained scans, classifying nodules as benign, malignant, or indeterminate on the basis of their number, unilateral versus bilateral distribution, size, margins (indistinct vs distinct), calcification, growth, and associated adenopathy. These classifications were compared with nodule histologic type, and interreviewer agreement was assessed. RESULTS: The median patient age was 14.8 years (mean, 13.7 years; range, 5-21 years). Twenty-four of the 41 patients (58%) had at least one biopsy-proved malignant nodule. Four (10%) patients had both benign and malignant nodules; 17 (42%) had only benign nodules. Reviewer 1 classified 65% (39 of 60) of nodules correctly; reviewer 2, 57% (37 of 65); and reviewer 3, 67% (43 of 64). Interreviewer agreement was slight to moderate (kappa </= 0.43, P </= .03). In contrast to findings in adults, sharply defined nodules in younger individuals were more likely to be malignant (P = .03) and nodule size was not associated with malignancy (P >/= .32). CONCLUSION: The frequency of benign nodules and the inconsistency of predictions based on CT features suggest the need for better predictors of pulmonary nodules being malignant or benign, so as to reduce unnecessary thoracotomy in pediatric patients with solid malignancy. .

Adolescent↗

PubMed1995

The extent of inter- and intrareviewer agreement on the classification and assessment of designs of single-practice research.

The aim of this study was to investigate how much inter- and intrareviewer agreement there was with regard to the classification of research designs and the assessment of the methodological quality of research reports. This was done by means of formal criteria and criteria relating to content. Three successive reliability tests were performed on two random samples of reports of single-practice research in general practice. Inter- and intrareviewer agreement on the classification tended to be satisfactory. The results of the tests to assess the methodological quality were, however, more deviant, the interreviewer agreement ranging from predominantly mediocre/poor to mostly good and intrareviewer agreement ranging from mediocre to good. These results emphasize the need for further discussion and follow-up research.

Evaluation Studies as Topic↗

PubMed2003

Surgical quality assurance in the Ischemic Optic Neuropathy Decompression Trial (IONDT).

The purpose of this article is to report the methods and results of the surgical quality assurance program associated with the Ischemic Optic Neuropathy Decompression Trial (IONDT). A surgical quality assurance committee developed and implemented a quality assurance program for a randomized clinical trial requiring surgical intervention. A surgical technique questionnaire was administered at two times during the study course, and maintenance of surgeon certification required submission and approval of a masked videotape of an optic nerve sheath decompression surgery by each study surgeon. Surgical quality was assessed through completion of surgical report forms and standardized, masked review of operative notes. Rates of compliance and intra- and interreviewer agreement were assessed for each aspect of the program. Twenty-five of 32 surgeons (81%) successfully completed and maintained certification. Item agreement varied from 21-92% among reviewers of satisfactory videotapes and 22-89% on unsatisfactory videotapes. Intrarater agreement for videotape acceptability was 11 of 13 (85%), and for specific surgical steps, 147 of 182 (81%). Operative notes were submitted for 123 of 125 (98%) patients receiving surgery. Interrater agreement on individual items ranged from 73-100%. Classification of individual items was identical on first and second review for 1285 of 1344 (95.6%) items. Overall agreement for individual reviewers was 93.8-97.8%. We conclude that use of a small peer review committee, which developed and oversaw a quality assurance program, allowed for consistent certification and monitoring of surgical performance. This in turn increased the credibility of the IONDT results, which demonstrated no difference in outcome between operated and unoperated groups of patients.

Certification↗

PubMedDate not supplied

Emergency medical services assessment and treatment of children with special health care needs before and after specialized paramedic training.

INTRODUCTION: This study evaluates whether a continuing education program for paramedics, focusing on Children with Special Health Care Needs, improved paramedics' assessment and management. METHODS: Emergency Medical Services responses for children, 21 years of age or younger, with a congenital or acquired condition or a chronic physical or mental illness, were identified. The responses before and after the specialized education program were reviewed by a multidisciplinary team to evaluate assessment and management of the children. Interreviewer agreement between the nurses on the team and between the physicians on the team was assessed. We also evaluated whether there was an improvement in assessment and care by paramedics completing our education program. RESULTS: Significant improvement was seen in appropriate assessment and overall care by paramedics who completed our specialized education program. Reviewers also noted an appropriate rating for the initial assessment category more often for responses involving paramedics who had the training. Agreement on whether assessment and treatment was appropriate for all five reviewers varied considerably, ranging from 32% to 93%. Overall there was a high percentage of agreement (>70%) between the nurses and between the physicians on most items. However, kappa statistics did not generally reflect good agreement except for most of the focused assessment items and some treatment and procedure items. CONCLUSION: Most of the documentation on the EMS records indicated appropriate assessment and treatment during all responses for Children with Special Health Care Needs. Nevertheless, the results indicate that paramedics may improve their assessment and management of these children after specialized continuing education.

Arizona↗

PubMed1992

The reliability of peer assessments of quality of care.

OBJECTIVE: To critically examine the literature regarding the interreviewer reliability of the standard practice of peer assessment of quality of care. DATA SOURCES: Computerized searches of the English-language literature from 1966 through 1990 using MEDLINE, HEALTHLINE, and SCISEARCH databases were performed to identify studies reporting data on interreviewer agreement of implicit evaluations of patient care episodes. STUDY SELECTION: Seventeen studies were identified. Five studies were excluded from this review because of deficiencies in the methods or lack of data on chance-corrected indexes of agreement. DATA EXTRACTION SYNTHESIS: The degree of agreement beyond chance was compared with accepted standards in the 12 remaining studies. Most of these studies found agreement corrected for chance to be in the range regarded as poor, indicating that physician agreement regarding quality of care is only slightly better than the level expected by chance. CONCLUSIONS: Given the magnitude of the resources devoted to quality assurance and the centrality of peer assessment to these efforts, there is a need for a global reexamination of the peer review process. A number of proposals appear to have potential for improving the peer review process including more objective assessment procedures, multiple reviewers, higher standards for reviewers, elimination of systematic reviewer bias, use of outcome judgements, and adoption of practice guidelines.

Bias↗

PubMed2004

Interrater reliability in grading abstracts for the orthopaedic trauma association.

Only a small proportion of submitted abstracts to the annual meeting of the Orthopaedic Trauma Association can be accepted for podium presentation. Annual program committee members must ensure that the selection of abstracts is free from bias and transparent to investigators. The objectives of this study are to examine the consistency of reviewers in grading abstracts submitted for podium presentations at the 2001 and 2002 Annual Meetings of the Orthopaedic Trauma Association and to evaluate whether the grades of the actual podium presentations at the meeting are consistent with the grades based on abstracts only. Reviewers independently graded all abstracts submitted to the Orthopaedic Trauma Association for presentation in a blinded manner. Abstracts submitted by members of the review panel were independently adjudicated by six reviewers who were not members of the committee. Before final decision-making, all reviewers met to discuss the abstracts submitted for oral presentation. Among the 440 papers reviewed in 2001 and 438 papers reviewed in 2002, the interreviewer reliability for abstract review was 0.23 and 0.27, respectively. Despite disagreements in the quality of the abstracts, reviewers achieved consensus by discussions to determine the final program. Agreement among unblinded reviewers of the 67 and 73 podium presentations during the 2001 and 2002 meetings, respectively, did not improve interreviewer agreement. Of the papers of the 2002 meeting that ultimately ranked in the top 20 after the full presentation of the papers, 15 papers originally had been ranked less than 20 in the initial grading. Only one of the top three papers of the meeting originally was ranked in the top three before the meeting.

Abstracting and Indexing↗

PubMed2026

Artificial Intelligence Cannot Replace Peer Reviewers but May Help Editors Triage: A Comparative Analysis of a Large Language Model and Human Reviewer Recommendations at the American Journal of Sports Medicine.

BACKGROUND: The peer review system faces increasing strain from rising manuscript volumes, reviewer fatigue, and well-documented interreviewer disagreement. Large language models (LLMs) have shown potential to support the peer review process, but their ability to replicate editorial decisions at high-impact medical journals and their utility as manuscript screening tools remain unknown. PURPOSE: To compare the agreement between an LLM and the final editorial decision on manuscripts submitted to the American Journal of Sports Medicine and to evaluate the potential of LLMs as a manuscript screening tool. STUDY DESIGN: Cross-sectional agreement study. METHODS: Fifty-four manuscripts randomly selected from submissions to the American Journal of Sports Medicine (September 2024-October 2024) were reviewed by a locally deployed LLM (Ministral 3 14B; Mistral AI) using a standardized prompt. The artificial intelligence (AI) produced a categorical recommendation (reject, cascade, revision, or accept) and a numerical score (0-100) for each manuscript. Agreement with the final editorial decision was assessed by Cohen kappa (4-category model) for pooled human reviewers (n = 139 reviews) and the AI (n = 54). Screening performance was evaluated by positive predictive value (PPV), sensitivity, and specificity. RESULTS: Pooled human reviewers demonstrated fair agreement with the final decision (κ = 0.181 [P < .001]; 42.4% agreement), while the AI demonstrated slight, nonsignificant agreement (κ = 0.126 [P = .099]; 37.0% agreement). The AI recommended revision for 61.1% of manuscripts, of which 72.7% were ultimately rejected or cascaded, demonstrating systematic "revision bias." When the AI recommended rejection, 54.5% of those manuscripts were ultimately rejected and 27.3% were cascaded; when the AI recommended cascade, 50% were rejected and 50% were cascaded. However, when the AI recommended rejection or cascade (n = 21), 90.5% received a final decision of rejection or cascade (PPV, 90.5%; specificity, 81.8%). Manuscripts with an AI score <70 were rejected or cascaded 88.0% of the time (PPV, 88.0%). CONCLUSION: AI cannot replicate the nuanced judgment of human peer reviewers at a high-impact sports medicine journal. When AI recommended rejection or cascade, 90.5% of manuscripts received that final decision (descriptive PPV, 90.5%; 95% CI, 71.1%-97.3%), suggesting potential utility as an exploratory first-pass screening tool warranting further validation in larger cohorts. However, AI could not reliably distinguish manuscripts destined for outright rejection from those that would be cascaded to a sister journal-an important limitation for editorial triage applications.

Sports Medicine↗

PubMed2003

Reproducibility and prognostic variability of grade and lamina propria invasion in stages Ta, T1 urothelial carcinoma of the bladder.

PURPOSE: We assessed the reproducibility and prognostic variability of grade and lamina propria invasion in stages Ta, T1 urothelial carcinoma of the bladder. MATERIALS AND METHODS: A total of 130 consecutive stages Ta, T1 urothelial carcinomas routinely diagnosed by 15 pathologists (original diagnosis) were reviewed by 3 independent experienced pathologists using 1999 WHO criteria (diagnoses 1 to 3 and reviewer consensus diagnosis). Interreviewer disagreement cases were blindly reviewed again. Each remaining disagreement case was discussed in a multihead microscope session to attempt to solve remaining disagreements. In cases of continuing disagreement the majority diagnosis on stage and grade was considered the consensus diagnosis. Stage progression at followup was the dependent variable. Stage progression-free Kaplan-Meier survival curves and hazard ratios of each stage and grade diagnosis were calculated and prognostic variability was determined. RESULTS: There was complete interobserver agreement on stage and grade among reviewers in 80% and 59% of cases, while it was 87.7% and 75.4%, respectively, after the second review. More than 1 grade difference occurred in 1.5% of cases (0% after the second review). The consensus and original diagnoses agreed on stage and grade in 68.5% and 62.3% of cases, respectively. Assignment of individual cases to 1 category of the 1999 WHO classification per reviewer varied considerably. The incidence of cases classified as stage T1 grade 3 by the reviewers was between 12.3% and 18.9% (average 14.1%). Consensus diagnosis grade had the strongest prognostic value (HR 68.8, range 8.9 to 528.0). Of the 63 original diagnoses of stage T1 tumors the consensus diagnosis down staged 35 (55.6%) to Ta and up staged 8 (12.7%) to T2-3. Progression was more common in the 20 consensus diagnosis stage T1 cases (5 or 25%) than in the 55 original diagnosis stage T1 cases (11 or 20%). Original diagnosis stage T1 tumors that were down staged by the consensus diagnosis showed less progression than consensus diagnosis confirmed stage T1 tumors (17% versus 25%). The prognostically worst subgroup (T1 grade 3) also showed considerable prognostic variation among reviewers (28% to 76% at 5 years of followup), in that the consensus diagnosis again had the highest prognostic significance (HR 3.5, range 1.2 to 10.2). At the end of the study all pathologists expressed that they were regularly uncertain about stage and grade assessment in an individual case in a considerable percent of all cases. CONCLUSIONS: Observer prognostic variability in staging and grading is considerable with potentially strong implications for patients. Interobserver variation did not decrease using the new 1999 WHO grading classification.

Carcinoma, Transitional Cell↗