Multiple classification and receiver operating characteristic (ROC) analysis.

Abstract

The receiver operating characteristic (ROC) curve was applied to observer performances in a multiple-alternative decision task. It was shown that the probability of correct classification, a performance criterion often maximized in multiple-classification procedures, corresponds to the area under an appropriately constructed ROC curve. Degrees of confidence in the observer's judgment of 0, 1, ..., 10 were used for both classification and ROC rating. To demonstrate the validity of the method, 1,190 photofluorograms were examined by experienced staff radiologists to identify four cardiovascular conditions distinguishable on the basis of images of structural elements of the contours of the heart and great vessels. The classification matrices for three radiologists who achieved high, medium, and low performance ratings in this experiment are reported. The ROC curves are symmetric, with their points located around the off-diagonal. Differences between the overall probability of correct classification and the ROC curve index calculated from the same evaluator's data were very small, 0.004 to 0.011.