SEARCH · PubMed Health

Results for “roc”

Explore indexed PubMed citations for clinical trials, systematic reviews and public health research. Read source abstracts and follow each citation to its original PubMed record.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3Linked to original sources

PubMed2003

Novel ROC-type method for testing the efficiency of multivariate statistical methods in fMRI.

The receiver operating characteristic (ROC) method is a useful and popular tool for testing the efficiency of various diagnostic tests applicable to functional MRI (fMRI) data. Typically, the diagnostic tests are applied on simulated and pseudo-human fMRI data, and the area under the ROC curve is used as a measure of the efficiency of the diagnostic test. The effectiveness of such a method depends on how well the simulated data approximate the real data. For multivariate statistical methods, however, this technique is usually inadequate, as the spatial dependence among voxels is ignored for simulated data. In this work a modified ROC method using real fMRI data with a broader scope is proposed. This method can be applied to most fMRI postprocessing techniques, including multivariate analyses such as canonical correlation analysis (CCA). Also, the relationship of the modified ROC method with the conventional ROC method is discussed in detail.

Algorithms↗

PubMed1989

Nonparametric comparison of entire ROC curves for computerized ECG left ventricular hypertrophy algorithms using data from the Framingham Heart Study.

A computer program may be capable of several different statements for left ventricular hypertrophy (eg, possible LVH, probable LVH, consistent with LVH), but such statements resulting from discretized levels of sensitivity/specificity would represent only isolated points on a receiver-operating characteristic (ROC) curve, which is a plot of all levels of sensitivity versus specificity. Even if two algorithms use the same discrete scales, their performances may not readily be compared. The authors present a comparison methodology for ROC curves using ROC area as a nonparametric measure of the ability of the algorithm to separate the two populations; the ROC area ranges from 0.5 (no ability) to 1.0 (perfect separation) and is unbiased if the normal versus abnormal populations have no common values for the measurement. The methodology compares the performance of ECG algorithms on the same population of cases by testing for significant differences of ROC areas and incorporating correlation of the algorithms in a nonparametric way. To illustrate this methodology, they use ECG and echocardiographic data from the Framingham Heart Study.

Algorithms↗

PubMed2005

ROCs in recognition with and without identification.

Receiver operating characteristics (ROCs) for the recognition of unidentified word fragments were analysed and compared with ROCs for the recognition of identified word fragments. In all conditions reported here, the z-transformed ROCs (z-ROCs) for recognition without identification were linear, with slopes near 1.0. In contrast, the z-ROCs for recognition with identification each had both a linear and a quadratic component, and a slope of substantially less than 1.0. Because the recognition of unidentified word fragments likely results from a familiarity process, the present study examined how well the effect can be explained by global matching variants of signal detection theory. Global matching models predict that memory should be better for strong items from a mixed list than for strong items from a pure list, and should be worse for weak items from a mixed list than for weak items from a pure list; however, only the latter pattern was shown for the recognition of unidentified items.

Cues↗

PubMed2006

ROC curves predicted by a model of visual search.

In imaging tasks where the observer is uncertain whether lesions are present, and where they could be present, the image is searched for lesions. In the free-response paradigm, which closely reflects this task, the observer provides data in the form of a variable number of mark-rating pairs per image. In a companion paper a statistical model of visual search has been proposed that has parameters characterizing the perceived lesion signal-to-noise ratio, the ability of the observer to avoid marking non-lesion locations, and the ability of the observer to find lesions. The aim of this work is to relate the search model parameters to receiver operating characteristic (ROC) curves that would result if the observer reported the rating of the most suspicious finding on an image as the overall rating. Also presented are the probability density functions (pdfs) of the underlying latent decision variables corresponding to the highest rating for normal and abnormal images. The search-model-predicted ROC curves are 'proper' in the sense of never crossing the chance diagonal and the slope is monotonically changing. They also have the interesting property of not allowing the observer to move the operating point continuously from the origin to (1, 1). For certain choices of parameters the operating points are predicted to be clustered near the initial steep region of the curve, as has been observed by other investigators. The pdfs are non-Gaussians, markedly so for the abnormal images and for certain choices of parameter values, and provide an explanation for the well-known observation that experimental ROC data generally imply a wider pdf for abnormal images than for normal images. Some features of search-model-predicted ROC curves and pdfs resemble those predicted by the contaminated binormal model, but there are significant differences. The search model appears to provide physical explanations for several aspects of experimental ROC curves.

Algorithms↗

PubMed2005

Regularized ROC method for disease classification and biomarker selection with microarray data.

MOTIVATION: An important application of microarrays is to discover genomic biomarkers, among tens of thousands of genes assayed, for disease classification. Thus there is a need for developing statistical methods that can efficiently use such high-throughput genomic data, select biomarkers with discriminant power and construct classification rules. The ROC (receiver operator characteristic) technique has been widely used in disease classification with low-dimensional biomarkers because (1) it does not assume a parametric form of the class probability as required for example in the logistic regression method; (2) it accommodates case-control designs and (3) it allows treating false positives and false negatives differently. However, due to computational difficulties, the ROC-based classification has not been used with microarray data. Moreover, the standard ROC technique does not incorporate built-in biomarker selection. RESULTS: We propose a novel method for biomarker selection and classification using the ROC technique for microarray data. The proposed method uses a sigmoid approximation to the area under the ROC curve as the objective function for classification and the threshold gradient descent regularization method for estimation and biomarker selection. Tuning parameter selection based on the V-fold cross validation and predictive performance evaluation are also investigated. The proposed approach is demonstrated with a simulation study, the Colon data and the Estrogen data. The proposed approach yields parsimonious models with excellent classification performance.

Algorithms↗

PubMed2004

Ideal observers and optimal ROC hypersurfaces in N-class classification.

The likelihood ratio, or ideal observer, decision rule is known to be optimal for two-class classification tasks in the sense that it maximizes expected utility (or, equivalently, minimizes the Bayes risk). Furthermore, using this decision rule yields a receiver operating characteristic (ROC) curve which is never above the ROC curve produced using any other decision rule, provided the observer's misclassification rate with respect to one of the two classes is chosen as the dependent variable for the curve (i.e., an "inversion" of the more common formulation in which the observer's true-positive fraction is plotted against its false-positive fraction). It is also known that for a decision task requiring classification of observations into N classes, optimal performance in the expected utility sense is obtained using a set of N-1 likelihood ratios as decision variables. In the N-class extension of ROC analysis, the ideal observer performance is describable in terms of an (N2-N-1)-parameter hypersurface in an (N2-N)-dimensional probability space. We show that the result for two classes holds in this case as well, namely that the ROC hypersurface obtained using the ideal observer decision rule is never above the ROC hypersurface obtained using any other decision rule (where in our formulation performance is given exclusively with respect to between-class error rates rather than within-class sensitivities).

Bayes Theorem↗

PubMedDate not supplied

The area under an ROC curve with limited information.

The area under the receiver operating characteristic (ROC) curve of a diagnostic test can be used as a summary measure for its discriminative ability. If only a single point of an ROC curve is available, then the entire form of the ROC curve is unknown and the area under it cannot be calculated. Assuming that the unknown ROC curve is either monotone or concave, lower and upper bounds are derived for the area. From these bounds, the minmax approximations are obtained. Compared to only assuming monotonicity, assuming that the unknown ROC curve is concave renders a higher minmax approximation for the area under it, with tighter bounds.

Area Under Curve↗

PubMedDate not supplied

Multiple classification and receiver operating characteristic (ROC) analysis.

The receiver operating characteristic (ROC) curve was applied to observer performances in a multiple-alternative decision task. It was shown that the probability of correct classification, a performance criterion often maximized in multiple-classification procedures, corresponds to the area under an appropriately constructed ROC curve. Degrees of confidence in the observer's judgment of 0, 1, ..., 10 were used for both classification and ROC rating. To demonstrate the validity of the method, 1,190 photofluorograms were examined by experienced staff radiologists to identify four cardiovascular conditions distinguishable on the basis of images of structural elements of the contours of the heart and great vessels. The classification matrices for three radiologists who achieved high, medium, and low performance ratings in this experiment are reported. The ROC curves are symmetric, with their points located around the off-diagonal. Differences between the overall probability of correct classification and the ROC curve index calculated from the same evaluator's data were very small, 0.004 to 0.011.

Cardiovascular Diseases↗

PubMedDate not supplied

An exponential model used for optimal threshold selection on ROC curves.

A two-parameter exponential equation for modeling a receiver operating characteristic (ROC) curve is presented, where the area under the curve is a simple function of one of the parameters. The model makes no distributional assumptions about the underlying normal and abnormal patient populations or about the shape of the resulting ROC curves. In a computer simulation of 75 ROC curves, the model provides a fit equivalent to the maximum likelihood estimate method commonly used for ROC curve fitting. Similar results are obtained using the model to fit ROC curve data from the literature. The model's equation calculates the true-positive ratio as a function of the false-positive ratio, and has a first derivative that is useful for finding the optimal decision threshold for a diagnostic testing procedure. In particular, the model is useful in a computer program for finding jointly optimal thresholds for multiple sequential tests.

Computer Simulation↗

PubMedDate not supplied

Testing for fetal pulmonary maturity: ROC analysis involving covariates, verification bias, and combination testing.

The lecithin/sphingomyelin ratio (L/S) and the measured value of saturated phosphatidylcholine (SPC), amniotic fluid determinations obtained to assess fetal pulmonary maturity, were evaluated with receiver operating characteristic (ROC) curve analysis. The effects of covariates on the ROC curves were analyzed with a regression methodology that took into account all the available data when constructing an ROC curve for each subgroup. To correct for verification bias the authors used a logistic regression analysis to model the probability of verification, thereby permitting correction for verification bias of a fully stratified data set in spite of small cell frequencies. They examined combination testing with prediction rules using prospective logistic modeling, including as variables test results and clinical features. The L/S was found to be significantly better than SPC for assessing fetal pulmonary maturity. For older gestational age the L/S and SPC performed better than for younger gestational age. Contamination of the specimen degraded the ROC curves. Correcting for verification bias did not influence the ROC curves significantly but changed the cutoff value of the test variable for any particular operating point. Prediction rules to evaluate combination testing showed that obtaining the SPC level in addition to the L/S ratio added no significant information compared with the L/S only. Including gestational age in the prediction rule of either test improved the prediction.

Bias↗

PubMedDate not supplied

ROC curves and the areas under them for dichotomized tests: empirical findings for logistically and normally distributed diagnostic test results.

Many measures, including sensitivity and specificity, predictive values, and likelihood ratios, are available for the assessment of diagnostic tests. A drawback of the use of these measures is that continuous test results are often dichotomized, with consequent loss of information. Receiver operating characteristic (ROC) curves do not depend on discrimination thresholds, and therefore the area under the ROC curve (AUC) is one of the preferred measures. Although quantitative test results are often presented dichotomized, it would be convenient still to be able to estimate the ROC curve and the AUC. The authors present equations for such estimates when only one pair of a true- and a false-positive rate is given, for inherently logistically and normally distributed data. Illustrative empirical data are provided for both distributions. In contradiction to earlier reports, the authors also show that differential disease verification may skew the ROC curve. The ROC curve is thus not invariant to selection bias.

Bayes Theorem↗

PubMedDate not supplied

Goodness-of-fit issues in ROC curve estimation.

Zhou recently considered goodness-of-fit (GOF) testing for receiver operating characteristic (ROC) curves estimated by applying the binormal and other bidistributional models to rating method data. He interpreted significant GOF tests as evidence that different decision thresholds were applied to diseased and nondiseased subjects and concluded that, in such circumstances, an ROC curve does not exist. In this article the author demonstrates that the GOF test accommodates many alternative hypotheses and that a significant test result need not be equated with an interaction between disease status and decision criteria or with non-existence of an ROC curve. He develops a new family of ROC curves based on a fully parameterized bidistributional model. The family includes the binormal ROC curve, but generalizes its structure by signifying all identifiable deviations in parameters of the latent distributions that define the model. The family provides a unified framework of alternative models to the binormal assumption and of alternative hypotheses to the GOF test for that assumption.

Binomial Distribution↗

PubMed2006

Regularized binormal ROC method in disease classification using microarray data.

BACKGROUND: An important application of microarrays is to discover genomic biomarkers, among tens of thousands of genes assayed, for disease diagnosis and prognosis. Thus it is of interest to develop efficient statistical methods that can simultaneously identify important biomarkers from such high-throughput genomic data and construct appropriate classification rules. It is also of interest to develop methods for evaluation of classification performance and ranking of identified biomarkers. RESULTS: The ROC (receiver operating characteristic) technique has been widely used in disease classification with low dimensional biomarkers. Compared with the empirical ROC approach, the binormal ROC is computationally more affordable and robust in small sample size cases. We propose using the binormal AUC (area under the ROC curve) as the objective function for two-sample classification, and the scaled threshold gradient directed regularization method for regularized estimation and biomarker selection. Tuning parameter selection is based on V-fold cross validation. We develop Monte Carlo based methods for evaluating the stability of individual biomarkers and overall prediction performance. Extensive simulation studies show that the proposed approach can generate parsimonious models with excellent classification and prediction performance, under most simulated scenarios including model mis-specification. Application of the method to two cancer studies shows that the identified genes are reasonably stable with satisfactory prediction performance and biologically sound implications. The overall classification performance is satisfactory, with small classification errors and large AUCs. CONCLUSION: In comparison to existing methods, the proposed approach is computationally more affordable without losing the optimality possessed by the standard ROC method.

Biomarkers, Tumor↗

PubMed2004

ROC curves in clinical chemistry: uses, misuses, and possible solutions.

BACKGROUND: ROC curves have become the standard for describing and comparing the accuracy of diagnostic tests. Not surprisingly, ROC curves are used often by clinical chemists. Our aims were to observe how the accuracy of clinical laboratory diagnostic tests is assessed, compared, and reported in the literature; to identify common problems with the use of ROC curves; and to offer some possible solutions. METHODS: We reviewed every original work using ROC curves and published in Clinical Chemistry in 2001 or 2002. For each article we recorded phase of the research, prospective or retrospective design, sample size, presence/absence of confidence intervals (CIs), nature of the statistical analysis, and major analysis problems. RESULTS: Of 58 articles, 31% were phase I (exploratory), 50% were phase II (challenge), and 19% were phase III (advanced) studies. The studies increased in sample size from phase I to III and showed a progression in the use of prospective designs. Most phase I studies were powered to assess diagnostic tests with ROC areas >/=0.70. Thirty-eight percent of studies failed to include CIs for diagnostic test accuracy or the CIs were constructed inappropriately. Thirty-three percent of studies provided insufficient analysis for comparing diagnostic tests. Other problems included dichotomization of the gold standard scale and inappropriate analysis of the equivalence of two diagnostic tests. CONCLUSION: We identify available software and make some suggestions for sample size determination, testing for equivalence in diagnostic accuracy, and alternatives to a dichotomous classification of a continuous-scale gold standard. More methodologic research is needed in areas specific to clinical chemistry.

Chemistry, Clinical↗

PubMed2004

The sampling distributions of Gaussian ROC statistics.

For a discrimination experiment, a plot of the hit rate against the false-alarm rate--the ROC curve--summarizes performance across a range of confidence levels. In many content areas, ROCs are well described by a normal-distribution model and the z-transformed hit and false-alarm rates are approximately linearly related. We examined the sampling distributions of three parameters of this model when applied to a ratings procedure: the area under the ROC (Az), the normalized difference between the means of the underlying signal and noise distributions (da), and the slope of the ROC on z-coordinates (s). Statistical bias (the degree to which the mean of the sampling distribution differed from the true value) was trivial for Az, small but noticeable for da, and substantial for s. Variability of the sampling distributions decreased with the number of trials and was also affected by the number of response categories available to the participant and by the overall sensitivity level. Figures in the article and tables available on line can be used to construct confidence intervals around ROC statistics and to test statistical hypotheses.

Humans↗

PubMed2000

ROC and LROC analyses of the effects of lesion contrast, size, and signal-to-noise ratio on detectability in PET images.

UNLABELLED: Image quality in PET is typically assessed using measures such as contrast recovery, noise variation, and signal-to-noise ratio (SNR). However, these criteria do not directly reflect performance in the clinical use of the images. Lesion detection is a critical task in the clinical interpretation of many PET studies. A receiver operating characteristic (ROC) study is an accepted method for quantitatively evaluating detection performance with respect to factors that influence image quality. ROC and localization ROC (LROC) analyses were conducted to investigate the effects of lesion contrast, SNR, and size on detectability of hot lesions in PET images. METHODS: A thorax phantom was imaged with spheres of 3 sizes simulating lesions (0.45, 1.0, and 1.9 mL). The relative activity in the lesions and the total number of counts acquired were each varied by factors of 2 to ascertain the effects of contrast and SNR, respectively. Measured attenuation correction and a standard reconstruction protocol were used. Three nuclear medicine physicians and 6 medical physicists participated as readers, rating each image and indicating the suspected lesion location. The area under the calculated ROC and LROC curves (Az and Az,LROC) were used as measures of detection performance. RESULTS: Detection performance was shown to increase from virtually random (Az approximately 0.5, Az,LROC approximately 0.2) to superior (Az > 0.9, Az,LROC > 0.9) as lesion contrast was increased by 50% and as lesion SNR was doubled. Detection performance was not seen to vary when comparison was made using image-based measures alone. CONCLUSION: This study quantitatively shows that moderate increases in the image-based measures of lesion contrast and SNR give a relatively large increase in the task-based measure of lesion detection as measured by ROC and LROC analyses. Thus, techniques that give modest increases in lesion contrast or SNR are expected to improve detection. Results will be useful in evaluating improvement in detection for various reconstruction, acquisition, and data analysis methods that enhance contrast or noise performance.

Heart↗

PubMed1997

Analysis of correlated ROC areas in diagnostic testing.

This paper focuses on methods of analysis of areas under receiver operating characteristic (ROC) curves. Analysis of ROC areas should incorporate the correlation structure of repeated measurements taken on the same set of cases and the paucity of measurements per treatment resulting from an effective summarization of cases into a few area measures of diagnostic accuracy. The repeated nature of ROC data has been taken into consideration in the analysis methods previously suggested by Swets and Pickett (1982, Evaluation of Diagnostic Systems: Methods from Signal Detection Theory), Hanley and McNeil (1983, Radiology 148, 839-843), and DeLong, DeLong, and Clarke-Pearson (1988, Biometrics 44, 837-845). DeLong et al.'s procedure is extended to a Wald test for general situations of diagnostic testing. The method of analyzing jackknife pseudovalues by treating them as data is extremely useful when the number of area measures to be tested is quite small. The Wald test based on covariances of multivariate multisample U-statistics is compared with two approaches of analyzing pseudovalues, the univariate mixed-model analysis of variance (ANOVA) for repeated measurements and the three-way factorial ANOVA. Monte Carlo simulations demonstrate that the three tests give good approximation to the nominal size at the 5% levels for large sample sizes, but the paired t-test using ROC areas as data lacks the power of the other three tests and Hanley and McNeil's method is inappropriate for testing diagnostic accuracies. The Wald statistic performs better than the ANOVAs of pseudovalues. Jackknifing schemes of multiple deletion where different structures of normal and diseased distributions are accounted for appear to perform slightly better than simple multiple-deletion schemes but no appreciable power difference is apparent, and deletion of too many cases at a time may sacrifice power. These methods have important applications in diagnostic testing in ROC studies of radiology and of medicine in general.

Analysis of Variance↗

PubMed1991

ROC analysis of observer-response subjective rating data--application to periodontal radiograph assessment.

Many physical anthropological studies require that an observer or device discriminate between states that can be easily confused. Receiver operating characteristic (ROC) analysis currently offers the best method for determining the accuracy of such choices, particularly for small sample sizes. Although ROC analysis is widely accepted in psychophysical and biomedical testing, its use in anthropological studies has not been reported. ROC analysis is used here to determine the usefulness of enhanced dental radiographs to assess vertical alveolar bone defects for quantitative studies of human variation with regard to periodontal disease. The presence or absence of vertical-bony defects (truth) for 75 human skulls was established by the consensus of two trained observers. Dental bitewing-radiographs were taken of the alveolar processes, the radiographs digitized, and the brightness and contrast of the digital images enhanced. The two observers who established truth then rated 1) plain bitewing radiographs, 2) unenhanced digital images of bitewings, and 3) enhanced digital images of bitewings for vertical bony defects. The rating scale varied from 1 (vertical defect definitely or almost definitely present) to 5 (definitely or almost definitely absent). ROC analysis was used to compared the diagnostic value of the 3 imaging modalities. All modalities had nearly identical diagnostic performance, measured as Az values (areas beneath ROC curves) that were less than 0.80, which indicates only moderate usefulness. It is concluded that enhancement does not increase success in vertical-bony-defect diagnosis from digital dental radiographs processed in this manner. Moreover, it is suggested that conventional bitewing radiographs may be unsuitable for accurate quantification of such defects.

Alveolar Bone Loss↗