SEARCH · PubMed Health

Results for “roc”

Explore indexed PubMed citations for clinical trials, systematic reviews and public health research. Read source abstracts and follow each citation to its original PubMed record.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4Linked to original sources

PubMed1978

Basic principles of ROC analysis.

The limitations of diagnostic "accuracy" as a measure of decision performance require introduction of the concepts of the "sensitivity" and "specificity" of a diagnostic test. These measures and the related indices, "true positive fraction" and "false positive fraction," are more meaningful than "accuracy," yet do not provide a unique description of diagnostic performance because they depend on the arbitrary selection of a decision threshold. The receiver operating characteristic (ROC) curve is shown to be a simple yet complete empirical description of this decision threshold effect, indicating all possible combinations of the relative frequencies of the various kinds of correct and incorrect decisions. Practical experimental techniques for measuring ROC curves are described, and the issues of case selection and curve-fitting are discussed briefly. Possible generalizations of conventional ROC analysis to account for decision performance in complex diagnostic tasks are indicated. ROC analysis is shown to be related in a direct and natural way to cost/benefit analysis of diagnostic decision making. The concepts of "average diagnostic cost" and "average net benefit" are developed and used to identify the optimal compromise among various kinds of diagnostic error. Finally, the way in which ROC analysis can be employed to optimize diagnostic strategies is suggested.

Cost-Benefit Analysis↗

PubMed1998

COMPROC and CHECKNORM: computer programs for comparing accuracies of diagnostic tests using ROC curves in the presence of verification bias.

To assess relative accuracies of two diagnostic tests, we often compare the areas under the receiver operating characteristic (ROC) curves of these two tests in a paired design. Standard methods for analyzing data from a paired design require that every patient tested has the known disease status. In practice, however, some of the patients with test results may not have verified disease status. Any analysis using only verified cases may result in verification bias. COMPROC is an easy to use program for comparing the effectiveness of two diagnostic tests based on the area under the ROC curve in the presence of verification bias. COMPROC compensates for verification bias by implementing the maximum likelihood (ML) estimation of the areas and covariance matrix of two ROC curves under the missing at random (MAR) assumption as described by Zhou (Biometrics 54 (1998) 349-366). This method assumes normality of the difference of the two ROC curve area estimators. We also describe a program CHECKNORM that does a bootstrap analysis to test this normality assumption (B. Efron, R.J. Tibshirani, An Introduction to the Bootstrap, Chapman and Hall, London, 1993). COMPROC allows for the inclusion of observed covariates that may influence the decision to verify the disease status of a patient. The program computes the estimates of the area under the ROC curve for the two diagnostic tests along with the variance of each area, the covariance between the two areas, a two-sided p-value, and a confidence interval for the difference of the areas. The programs COMPROC and CHECKNORM require the scripting language Perl and the statistical software SAS and can be run on both UNIX machines as well as PCs. The use of COMPROC and CHECKNORM is illustrated in a clinical study designed to compare relative accuracies of MRI and CT in detecting pancreatic cancer.

Area Under Curve↗

PubMed1992

Evaluation of diagnostic tests using relative operating characteristic (ROC) curves and the differential positive rate. An example using the total serum bile acid concentration and the alanine aminotransferase activity in the diagnosis of canine hepatobiliary diseases.

The value of a diagnostic test depends on most cases on its ability to discriminate between patients with and without a certain disease. One way of evaluating a diagnostic test is to use the relative operating characteristic curve (ROC curve) and the differential positive rate (DPR). The ROC curve displays the relationship between the true positive ratio and the false positive ratio for a range of cutoff values and it can be used to compare various diagnostic tests under equivalent conditions (equal true positive ratios or false positive ratios) and over the entire range of cutoff values. The DPR is the difference between the true positive ratio and the false positive ratio at various cutoff values and it can be used to obtain the cutoff value associated with the highest sensitivity and specificity. The purpose of this study was to describe the evaluation and comparison of diagnostic tests using ROC curves and DPR. Eventually, the positive and negative predictive values were used to assess the differences between the sensitivity and specificity obtained when the upper limit of the reference interval, or the optimal cutoff value indicated by the DPR, was used as cutoff value. To illustrate the methods, the 2 h post-prandial total serum bile acid concentration (PSBA) and the alanine aminotransferase activity (ALAT) in the diagnosis of primary or secondary hepatobiliary diseases in dogs were used. The ROC curves showed, as expected from previous studies, that PSBA was superior to ALAT in diagnosing dogs with hepatobiliary diseases. Using DPR, the optimal cutoff value for PSBA was suggested to be 15.48 mumol/l. Compared to the traditionally used cutoff value of 22.24 mumol/l, no decisive difference in the positive predictive values were observed. However, the cutoff value of 15.48 mumol/l appeared to produce higher negative predictive values compared to a cutoff value of 22.24 mumol/l. Seemingly, ROC curves and DPR are simple methods useful to the evaluation of diagnostic tests and due to the simplicity, there seems to be a great potential for these methods in the evaluation of diagnostic tests in veterinary medicine.

Alanine Transaminase↗

PubMed1998

Maximum likelihood estimation of receiver operating characteristic (ROC) curves from continuously-distributed data.

We show that truth-state runs in rank-ordered data constitute a natural categorization of continuously-distributed test results for maximum likelihood (ML) estimation of ROC curves. On this basis, we develop two new algorithms for fitting binormal ROC curves to continuously-distributed data: a true ML algorithm (LABROC4) and a quasi-ML algorithm (LABROC5) that requires substantially less computation with large data sets. Simulation studies indicate that both algorithms produce reliable estimates of the binormal ROC curve parameters a and b, the ROC-area index Az, and the standard errors of those estimates.

Algorithms↗

PubMed2000

Regional confidence bands for ROC curves.

The performance of a diagnostic test is characterised by its specificity and sensitivity. For a quantitative diagnostic test these criteria depend on the selected cut-off point. The receiver operating characteristic (ROC) curve of a quantitative diagnostic test is generated by plotting sensitivity against specificity as the cut-off point runs through the whole range of possible test values. In practice, the ROC curve is estimated from clinical data. One important goal is to select an optimal cut-off point. For this purpose the sample variability has to be taken into account. Recently, Campbell has introduced nonparametric asymptotic simultaneous confidence bands that are valid for the whole ROC curve. In this paper a nonparametric asymptotic approach for the construction of regional confidence bands for ROC curves is proposed. It can be applied for any specificity interval of interest. Our approach is based on the asymptotic theory of empirical and quantile processes. To investigate the small sample properties of the different approaches, a Monte Carlo study was conducted using normal and log-normal data. A method for sample size calculation is presented. Finally, the approaches are applied to a tumour marker in the diagnosis of bone marrow metastases.

Bone Marrow Neoplasms↗

PubMed1996

ROC methods for evaluation of fMRI techniques.

Receiver operating characteristic (ROC) methods provide a standardized and statistically meaningful means for comparing signal-detection accuracy. A brief overview of ROC methods is presented. Example applications include a comparison of four different postprocessing algorithms operating on simulated fMRI time-course data sets and on human null data sets to which a simulated fMR response had been added. ROC methods also were used to reanalyze one data set from a previously published work. Additional ROC methods that also may be useful for fMRI comparisons are described.

Algorithms↗

PubMed2002

Receiver operating characteristic (ROC) analysis for diagnostic examinations with uninterpretable cases.

Receiver operating characteristic (ROC) analysis plots true positive rates over false positive rates to describe the discriminatory power of a test to differentiate between two specifiable populations (that is 'normal' from 'abnormal') using a continuous dichotomous threshold. This assumes that the test separates cases into observed 'normal' or 'abnormal'. However, in practice many tests have some uninterpretable results as an inherent feature of the test itself, whether independent or dependent on the sample populations. This paper defines a method to describe the ability of tests to discriminate between specifiable populations when uninterpretable results occur non-informatively about disease status. A mixed model modified ROC curve is developed. Formulae to estimate the area under the modified ROC curve are given. Comparisons of conventional with the mixed model ROC analysis are shown in mathematical formulae as well as simulated experiments. An example of diagnosis of spinal fracture in renal transplant patients is presented.

Area Under Curve↗

PubMed2003

ROC methodology within a monitoring framework.

Receiver operating characteristic (ROC) methodology is widely used to evaluate and compare diagnostic tests. Generally, each diagnostic test is applied once to each subject in a population and the results, reported on a continuous scale, are used to construct the ROC curve. We extend the standard method to accommodate a framework in which the diagnostic test is repeated over time to monitor for occurrence of an event. Unlike the usual situation in which event status is static, the problem we address involves event status that is not constant over the monitoring period. Subjects generally are classified as non-events, or controls, until they experience events that convert them to cases. Viewing the data as incomplete discrete failure time data with time-varying covariates, potentially useful diagnostic markers can be related appropriately in time with the true condition and varying amounts of information per individual can be taken into account. The ROC curve provides an assessment of the performance of the test in combination with the schedule of testing. Within this framework, a computational simplification is introduced to calculate variances and covariances for the areas under the ROC curves. Periodic monitoring for reperfusion following thrombolytic treatment for acute myocardial infarction provides a detailed example, whereby the lengths of the testing interval combined with different diagnostic markers are compared.

Angioplasty, Balloon, Coronary↗

PubMed2004

Adjusting the generalized ROC curve for covariates.

Receiver operating characteristic (ROC) curves and in particular the area under the curve (AUC), are widely used to examine the effectiveness of diagnostic markers. Diagnostic markers and their corresponding ROC curves can be strongly influenced by covariate variables. When several diagnostic markers are available, they can be combined by a best linear combination such that the area under the ROC curve of the combination is maximized among all possible linear combinations. In this paper we discuss covariate effects on this linear combination assuming that the multiple markers, possibly transformed, follow a multivariate normal distribution. The ROC curve of this linear combination when markers are adjusted for covariates is estimated and approximate confidence intervals for the corresponding AUC are derived. An example of two biomarkers of coronary heart disease for which covariate information on age and gender is available is used to illustrate this methodology.

Adult↗

PubMed2004

Ordered multiple-class ROC analysis with continuous measurements.

Receiver operating characteristic (ROC) curves have been useful in two-group classification problems. In three- and multiple-class diagnostic problems, an ROC surface or hyper-surface can be constructed. The volume under these surfaces can be used for inference using bootstrap techniques or U-statistics theory. In this article, ROC surfaces and hyper-surfaces are defined and their behaviour and utility in multi-group classification problems is investigated. The formulation of the problem is equivalent to what has previously been proposed in the general multi-category classification problem but the definition of ROC surfaces here is less complex and addresses directly the narrower problem of ordered categories in the three-class and, by extension, the multi-class problem applied to continuous and ordinal data. Non-parametric manipulation of both continuous and discrete test data and comparison between two diagnostic tests applied to the same subjects are considered. A three-group classification example in the context of HIV neurological disease is presented and the results are discussed.

AIDS Dementia Complex↗

PubMed2005

The partial area under the summary ROC curve.

The area under the curve (AUC) is commonly used as a summary measure of the receiver operating characteristic (ROC) curve. It indicates the overall performance of a diagnostic test in terms of its accuracy at various diagnostic thresholds used to discriminate cases and non-cases of disease. The AUC measure is also used in meta-analyses, where each component study provides an estimate of the test sensitivity and specificity. These estimates are then combined to calculate a summary ROC (SROC) curve which describes the relationship between-test sensitivity and specificity across studies. The partial AUC has been proposed as an alternative measure to the full AUC. When using the partial AUC, one considers only those regions of the ROC space where data have been observed, or which correspond to clinically relevant values of test sensitivity or specificity. In this paper, we extend the idea of using the partial AUC to SROC curves in meta-analysis. Theoretical and numerical results describe the variation in the partial AUC and its standard error as a function of the degree of inter-study heterogeneity and of the extent of truncation applied to the ROC space. A scaled partial area measure is also proposed to restore the property that the summary measure should range from 0 to 1. The results suggest several disadvantages of the partial AUC measures. In contrast to earlier findings with the full AUC, the partial AUC is rather sensitive to heterogeneity. Comparisons between tests are more difficult, especially if an empirical truncation process is used. Finally, the partial area lacks a useful symmetry property enjoyed by the full AUC. Although the partial AUC may sometimes have clinical appeal, on balance the use of the full AUC is preferred.

Area Under Curve↗

PubMed2005

A permutation test sensitive to differences in areas for comparing ROC curves from a paired design.

The area under the receiver operating characteristic (ROC) curve (AUC) is a widely accepted summary index of the overall performance of diagnostic procedures and the difference between AUCs is often used when comparing two diagnostic systems. We developed an exact non-parametric statistical procedure for comparing two ROC curves in paired design settings. The test which is based on all permutations of the subject specific rank ratings is formally a test for equality of ROC curves that is sensitive to the alternatives of AUC difference. The operating characteristics of the proposed test were evaluated using extensive simulations over a wide range of parameters. The proposed procedure can be easily implemented in experimental ROC data sets. For small samples and for underlying parameters that are common in experimental studies in diagnostic imaging the test possesses good operating characteristics and is more powerful than the conventional non-parametric procedure for AUC comparisons. We also derived an asymptotic version of the test which uses an exact estimate of the variance in the permutation space and provides a good approximation even when the sample sizes are small. This asymptotic procedure is a simple and precise approximation to the exact test and is useful for large sample sizes where the exact test may be computationally burdensome.

Biometry↗

PubMed2006

An ROC-type measure of diagnostic accuracy when the gold standard is continuous-scale.

ROC curves and summary measures of accuracy derived from them, such as the area under the ROC curve, have become the standard for describing and comparing the accuracy of diagnostic tests. Methods for estimating ROC curves rely on the existence of a gold standard which dichotomizes patients into disease present or absent. There are, however, many examples of diagnostic tests whose gold standards are not binary-scale, but rather continuous-scale. Unnatural dichotomization of these gold standards leads to bias and inconsistency in estimates of diagnostic accuracy. In this paper, we propose a non-parametric estimator of diagnostic test accuracy which does not require dichotomization of the gold standard. This estimator has an interpretation analogous to the area under the ROC curve. We propose a confidence interval for test accuracy and a statistical test for comparing accuracies of tests from paired designs. We compare the performance (i.e. CI coverage, type I error rate, power) of the proposed methods with several alternatives. An example is presented where the accuracies of two quick blood tests for measuring serum iron concentrations are estimated and compared.

Adolescent↗

PubMed2008

On the exact interval estimation for the difference in paired areas under the ROC curves.

An important measure for comparison of accuracy between two diagnostic procedures is the difference in paired areas under the receiver operating characteristic (ROC) curves. Non-parametric and maximum likelihood methods have been proposed for interval estimation for the difference in paired areas under ROC curves. However, these two methods are asymptotic procedures and their performance in finite sample sizes has not been thoroughly investigated. We propose to use the concept of generalized pivotal quantities (GPQs) to construct an exact confidence interval for the difference in paired areas under ROC curves. A simulation study is conducted to empirically investigate the probability coverage and expected length of the three methods for various combinations of sample sizes, values of the area under the ROC curve and correlations. Simulation results demonstrate that the exact confidence interval based on the concept of GPQs provides not only sufficient probability coverage but also reasonable expected length. Numerical examples using published data sets illustrate the proposed method.

Clinical Trials as Topic↗

PubMed1993

Combining independent studies of a diagnostic test into a summary ROC curve: data-analytic approaches and some additional considerations.

We consider how to combine several independent studies of the same diagnostic test, where each study reports an estimated false positive rate (FPR) and an estimated true positive rate (TPR). We propose constructing a summary receiver operating characteristic (ROC) curve by the following steps. (i) Convert each FPR to its logistic transform U and each TPR to its logistic transform V after increasing each observed frequency by adding 1/2. (ii) For each study calculate D = V - U, which is the log odds ratio of TPR and FPR, and S = V + U, an implied function of test threshold; then plot each study's point (Si, Di). (iii) Fit a robust-resistant regression line to these points (or an equally weighted least-squares regression line), with V - U as the dependent variable. (iv) Back-transform the line to ROC space. To avoid model-dependent extrapolation from irrelevant regions of ROC space we propose defining a priori a value of FPR so large that the test simply would not be used at that FPR, and a value of TPR so low that the test would not be used at that TPR. Then (a) only data points lying in the thus defined north-west rectangle of the unit square are used in the data analysis, and (b) the estimated summary ROC is depicted only within that subregion of the unit square. We illustrate the methods using simulated and real data sets, and we point to ways of comparing different tests and of taking into account the effects of covariates.

Carcinoma, Non-Small-Cell Lung↗

PubMed2006

ROC and loss function analysis in sequential testing.

Sequential testing is applied to reduce costs in SP-based tests (OSCEs). Initially, all candidates take a screening test consisting of a part of the OSCE. Candidates who fail the screen sit the complete test, whereas those who pass the screen are qualified as a pass of the complete test. The procedure may result in a reduction of testing resources, but at the cost of false positives (candidates who pass the screen but would fail the complete test). In a diagnostic test an optimum cutpoint is obtained by minimizing the weighted sum of false negatives and false positives using Receiver Operator Characteristic (ROC) analysis. However, in the sequential procedure there are no false negatives, because the result of the complete test is considered the final outcome. Several authors ignored this difference and applied classic ROC analysis to a sequential test. Earlier, we doubted the validity of the procedure and proposed to set the cutpoint by minimizing the loss, defined as the weighted sum of the screen negatives and the false positives. Recently, Regehr and Colliver (2003) argued that under certain theoretically derived conditions the use of the loss formula in sequential testing is functionally identical to using classic ROC analysis. The current study shows (a) the conclusion of Regehr and Colliver is based on assumptions that can be challenged, and (b) ROC analysis indeed can be used in sequential testing, but only if the procedure is modified according to the results of a corresponding loss function analysis.

Diagnostic Tests, Routine↗

PubMed1990

Evaluation of neural network performance by receiver operating characteristic (ROC) analysis: examples from the biotechnology domain.

A need exists for an unbiased measure of the accuracy of feed-forward neural networks used for classification. Receiver operating characteristic (ROC) analysis is suited for this measure, and has been used to assess the performance of several different network weights. The area under an ROC and its standard error were used to compare different network weight sets, and to follow the performance of a network during the course of training. The ROC is not sensitive to the prior probabilities of examples in the testing set nor to the system's decision bias. The area under an ROC curve is a readily understood measure, and should be used to evaluate neural networks and to report results of learning experiments. Examples are provided from experiments with data from the biotechnology domain.

Artificial Intelligence↗

PubMed2006

Reader variance in ROC studies--generalizability to reader population at high and low performance levels.

RATIONALE AND OBJECTIVES: To investigate the variability between discriminative performances of readers as a function of average performance levels during receiver operating characteristic (ROC) studies. MATERIALS AND METHODS: Four subsets of cases from previously ascertained ROC rating data by 12 observers when detecting interstitial disease and pneumothorax on posteroanterior chest films were selected for each abnormality and reanalyzed to assess changes in "reader" variance component. The subsets were selected based on a prestudy subjective assessment of the subtleness of depicted abnormality (positive cases) and the difficulty in determining its absence (negative cases). Reader variance component was estimated using a bootstrap approach for each subset and the results were used to assess a general relationship between variability and average performance level. RESULTS: The reader variance component decreased substantially (from 0.007704 to 0.000426), as expected, when the areas under the ROC curves (AUC) for detecting pneumothoraces increased from 84% to 97%. On the other hand, reader variance component increased substantially (from 0.000890 to 0.005181) when AUC for detecting interstitial disease increased from 59% to 87%. The large magnitude of and changes in the reader variance component resulted in a consistent nonmonotone relationship as a function of AUC when other related variance components were included in addition to the reader component. CONCLUSION: Among several factors affecting generalizability of ROC results to the population of readers, the reader variance component depended nonmonotonically on the average diagnostic performance and is lowest at both very high and very low levels of performance.

Clinical Competence↗