PubMed Health⌕ Search
Home/Archive/PubMed

PubMed · 12600955

Comparison of eight computer programs for receiver-operating characteristic analysis.

Carsten Stephan·Sebastian Wesseling·Tania Schink·Klaus Jung
View original source ↗Export metadata ↓

Abstract

BACKGROUND: ROC analysis is widely accepted to assess and compare diagnostic validity of laboratory tests. Within the last few years, many new ROC programs have become available but have not been systematically evaluated. The aim of this study was to assess different ROC programs regarding their ease of use, mathematical correctness, final output, and their compatibility with other graphics programs. METHODS: Eight available programs running under Windows (AccuROC, Analyse-It, CMDT, GraphROC, MedCalc, mROC, ROCKIT, and SPSS) were evaluated. ROC analyses of prostate-specific antigen and related values were performed from a dataset of 928 men with prostate cancer and benign prostatic hyperplasia and corresponding subsets. Criteria such as data input, data output, and correctness and completeness of results were used to evaluate the practicability of the programs. RESULTS: Although the programs produced equivalent results (areas under the curves and their characteristics), we observed deficiencies concerning input of data, processing of the output data, and completeness of the results. Analyse-It, AccuROC, and MedCalc exhibited good performance, but each program had different shortcomings. Only GraphROC could compare curves at a certain sensitivity or specificity cutoff. CONCLUSIONS: Adequate ROC analysis and ROC plotting cannot be performed with a single program. Analyse-It, AccuROC, and MedCalc can be recommended with certain limitations. Further improvements of the programs are necessary.

Explore related subjects

Area Under CurveHumansMaleNeural Networks, ComputerProstate-Specific AntigenProstatic HyperplasiaProstatic NeoplasmsROC CurveSoftwareComparative StudyJournal Article

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Carsten Stephan, Sebastian Wesseling, Tania Schink, Klaus Jung. 2003. Comparison of eight computer programs for receiver-operating characteristic analysis.. https://doi.org/10.1373/49.3.433

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related citations

PubMed2007

Smooth semiparametric receiver operating characteristic curves for continuous diagnostic tests.

We propose a semiparametric kernel distribution function estimator, based on which a new smooth semiparametric estimator of the receiver operating characteristic (ROC) curve is constructed. We derive the asymptotic bias and variance of the newly proposed distribution function estimator and show that it is more efficient than the traditional non-parametric kernel distribution estimator. We also derive the asymptotic bias and variance of our new ROC curve estimator and show that it is more efficient than the smooth non-parametric ROC curve estimator proposed by Zou et al. (Stat. Med. 1997; 16:2143-2156) and Lloyd (J. Am. Stat. Assoc. 1998; 93:1356-1364). For our proposed estimators, we derive data-based methods for bandwidth selection. In addition, we present some results on the analysis of two real data sets. Finally, a simulation study is presented to show that our estimators are better than the non-parametric counterparts in terms of bias, standard error, and mean-square error.

Area Under Curve↗
PubMed2007

Redox potential measurement as a rapid method for microbiological testing and its validation for coliform determination.

The redox potential is one of the most complex indicators of the physiological state of microbial cultures and its measurement could be a useful tool for the qualitative and quantitative determination of the microbial contamination. During the bacterial growth, the redox potential of the medium decreases. The shape of the redox potential curve is characteristic on the type of microorganism, and the rate of the change (dE/dt) is proportional to the living cell concentration. Defining the time required to reach a significant change in redox potential as Time to Detection (TTD), similarly to the impedimetric measurements, a strict linear correlation could be established between the TTD and the logarithm of the initial concentration of microorganisms. On the base of this calibration curve, the determination of living cell concentration could be simplified. For the experiments, a computer-controlled multi-channel measuring system and software was developed by the authors. The redox potential measurement method was tested and validated for the determination of coliform bacteria. The results have proved the high efficiency and reliability of the new method.

Area Under Curve↗
PubMed2007

Determination of N-methyl-4-isoleucine-cyclosporin (NIM811) in human whole blood by high performance liquid chromatography-tandem mass spectrometry.

A liquid chromatographic method with tandem mass spectrometric detection (LC-MS/MS) for the determination of N-methyl-4-isoleucine-cyclosporin (NIM811) was developed and validated over the concentration range 1-2500 ng/mL in human whole blood using a 0.05 mL sample volume. NIM811 and the internal standard, d(12)-cyclosporin A (d(12)-CsA), were extracted from blood using MTBE via liquid-liquid extraction. After evaporation of the organic solvent and reconstitution, a 10 microL aliquot of the resulting extract was injected onto the LC-MS/MS system. Chromatographic separation of NIM811 and internal standard was performed using a Waters Symmetry RP-8 (50 x 4.6 mm, 3 microm particle size) column. The mobile phase consists of 10 mm ammonium acetate in water (A) and acetonitrile (B), with 45% B from 0 to 0.2 min, 45 to 85% B from 0.2 to 0.8 min and 85% B from 0.8 to 2.2 min. The total run time was 3.5 min with a flow rate of 0.8 mL/min. The method was validated for sensitivity, linearity, reproducibility, stability, dilution integrity and recovery. The precision and accuracy of quality control samples at low (2.00 ng/mL), medium (20.0 and 400 ng/mL) and high (2000 ng/mL) concentrations were in the range 1.1-4.3% relative standard deviation (RSD) and -2.5-10.0% (bias), respectively, from three validation runs. The method has been used to measure the exposure of NIM811 in human subjects.

Area Under Curve↗