COMPROC and CHECKNORM: computer programs for comparing accuracies of diagnostic tests using ROC curves in the presence of verification bias.

Abstract

To assess relative accuracies of two diagnostic tests, we often compare the areas under the receiver operating characteristic (ROC) curves of these two tests in a paired design. Standard methods for analyzing data from a paired design require that every patient tested has the known disease status. In practice, however, some of the patients with test results may not have verified disease status. Any analysis using only verified cases may result in verification bias. COMPROC is an easy to use program for comparing the effectiveness of two diagnostic tests based on the area under the ROC curve in the presence of verification bias. COMPROC compensates for verification bias by implementing the maximum likelihood (ML) estimation of the areas and covariance matrix of two ROC curves under the missing at random (MAR) assumption as described by Zhou (Biometrics 54 (1998) 349-366). This method assumes normality of the difference of the two ROC curve area estimators. We also describe a program CHECKNORM that does a bootstrap analysis to test this normality assumption (B. Efron, R.J. Tibshirani, An Introduction to the Bootstrap, Chapman and Hall, London, 1993). COMPROC allows for the inclusion of observed covariates that may influence the decision to verify the disease status of a patient. The program computes the estimates of the area under the ROC curve for the two diagnostic tests along with the variance of each area, the covariance between the two areas, a two-sided p-value, and a confidence interval for the difference of the areas. The programs COMPROC and CHECKNORM require the scripting language Perl and the statistical software SAS and can be run on both UNIX machines as well as PCs. The use of COMPROC and CHECKNORM is illustrated in a clinical study designed to compare relative accuracies of MRI and CT in detecting pancreatic cancer.