PubMed Health⌕ Search

Biomedical subjects

Sergey V Beiden

Publications and source records attributed to Sergey V Beiden.

6 recordsLinked to original sources
PubMed2004

Multireader, multicase receiver operating characteristic analysis: an empirical comparison of five methods.

RATIONALE AND OBJECTIVES: Several statistical methods have been developed for analyzing multireader, multicase (MRMC) receiver operating characteristic (ROC) studies. The objective of this article is to increase awareness of these methods and determine if their results are concordant for published datasets. MATERIALS AND METHODS: Data from three previously published studies were reanalyzed using five MRMC methods. For each method the 95% confidence intervals (CIs) for the mean of the readers' ROC areas for each diagnostic test, the P value for the comparison of the diagnostic tests' mean accuracies, and the 95% CIs for the mean difference in ROC areas of the diagnostic tests were reported. RESULTS: Important differences in P values and CIs were seen when using parametric versus nonparametric estimates of accuracy, and there were the expected differences for random-reader versus fixed-reader models. Controlling for these differences, the Dorfman-Berbaum-Metz (DBM), Obuchowski-Rockette, Beiden-Wagner-Campbell, and Song's multivariate Wilcoxon-Mann-Whitney (WMW) methods gave almost identical results for the fixed-reader model. For the random-reader model, the DBM, Obuchowski-Rockette, and Beiden-Wagner-Campbell methods yielded approximately the same inferences, but the CIs for the Beiden-Wagner-Campbell method tend to be broader. Ishwaran's hierarchical ROC sometimes yielded significance not found with other methods. Song's modification of DBM's jack-knifing algorithm sometimes led to different conclusions than the original DBM algorithm. CONCLUSION: In choosing and applying MRMC methods, it is important to recognize: (1) the distinction between random-reader and fixed-reader models, the uncertainties accounted for by each, and thus the level of generalizeability expected from each; (2) assumptions made by the various MRMC methods; and (3) limitations of a five- or six-reader study when the reader variability is great.

Analysis of Variance↗
PubMed2002

Independent versus sequential reading in ROC studies of computer-assist modalities: analysis of components of variance.

RATIONALE AND OBJECTIVES: The authors analyzed two methods for arranging the temporal sequencing of unaided versus computer-assisted reading in multiple-reader, multiple-case receiver operating characteristic studies of detection of solitary pulmonary nodules on chest radiographs. In the "independent" mode the readings are separated by about I month; in the "sequential" mode, the computer-assisted reading immediately follows the unassisted reading. MATERIALS AND METHODS: The authors used the method of Beiden, Wagner, and Campbell (BWC) to decompose the components of variance of receiver operating characteristic accuracy measures into those that are correlated and those that are uncorrelated across reading conditions. Only the latter contribute to uncertainty in estimates of the difference in accuracy measures across reading conditions (unaided vs aided). This method was used to analyze data from two independent studies of the detection of solitary pulmonary nodules on chest radiographs. RESULTS: In the sequential reading mode the components that were correlated across reading conditions increased compared to the independent reading mode, as might be expected. What was not anticipated was the fact that the total reader variance was approximately the same for the two reading modes. The results were remarkably similar across the two independent studies analyzed. CONCLUSION: The sequential reading mode may thus be the more sensitive probe of the difference between unassisted and computer-assisted reading, if the mean effect is unperturbed (as here). It is also the least demanding on the logistics and investment of reader time.

Analysis of Variance↗
PubMed2002

Assessment of medical imaging and computer-assist systems: lessons from recent experience.

In the last 2 decades major advances have been made in the field of assessment methods for medical imaging and computer-assist systems through the use of the paradigm of the receiver operating characteristic (ROC) curve. In the most recent decade this methodology was extended to embrace the complication of reader variability through advances in the multiple-reader, multiple-case (MRMC) ROC measurement and analysis paradigm. Although this approach has been widely adopted by the imaging research community, some investigators appear averse to it, possibly from concern that it could place a greater burden on the scarce resources of patient cases and readers compared to the requirements of alternative methods. The present communication argues, however, that the MRMC ROC approach to assessment in the context of reader variability may be the most resource-efficient approach available. Moreover, alternative approaches may also be statistically uninterpretable with regard to estimated summary measures of performance and their uncertainties. The authors propose that the MRMC ROC approach be considered even more widely by the larger community with responsibilities for the introduction and dissemination of medical imaging technologies to society. General principles of study design are reviewed, and important contemporary clinical trials are used as examples.

Diagnosis, Computer-Assisted↗
PubMedDate not supplied

Reader variability in mammography and its implications for expected utility over the population of readers and cases.

The multiple-reader, multiple-case (MRMC) approach to receiver operating characteristic (ROC) analysis is becoming the dominant assessment paradigm in medical imaging. Its most common version involves having many readers read every patient case in the study, a critical feature since differences among competing imaging modalities are often dominated by differences in reader performance. The present authors have carried out MRMC ROC analysis on a uniquely large data set for mammography. The analysis quantifies the great range of observed reader skill in that data set. It also demonstrates that the sample sizes are sufficiently large that the conclusions generalize to the populations sampled here with little uncertainty from the finite sample size. A schematic approach to bracketing the utility matrix is then used to study trends in the resulting expected utility functions that correspond to the range of observed ROC curves. This is done for both the screening and the diagnostic context. The results raise 2 hypotheses for further investigation. First, it is possible that the present ambiguity surrounding the effectiveness of mammography is due in part to the observed range of reader skills and corresponding expected utility functions. Second, it is possible that computer-assisted modalities for mammography may lead to improvements in the expected utility function not only for screening but also in the diagnostic context, especially for the lower performing readers.

Data Interpretation, Statistical↗