PubMed2006
OBJECTIVE: Many proposals for the comparison of diagnostic devices refer to the computation of ROC curves or sensitivity / specificity-based parameters, thereby strictly assuming the presence of a reliably parameterized clinical reference method. When none of the devices under consideration can be regarded as a reference, Cohen's kappa coefficient for assessing the methods' relative agreement becomes increasingly popular. If, however, not only the agreement between two diagnostic devices, but also the devices' reliability must be taken into account (for example, if multiple parallel readings are obtained from one or both of the devices), no corresponding coefficients can be obtained from standard software. Bearing the recent modifications in the German Medicinal Devices Law (Medizinproduktegesetz) in mind, such methods will soon become necessary and strongly demanded for the sake of immediate re-evaluation of previously certified medicinal devices. METHODS: Generalizations of Cohen's kappa (kappa) for complex multi reader designs can be found by estimating weighted averages of the observed and expected agreement among subsets of parallel readings. A flexible, although instructive, strategy for designing kappa coefficients in the context of method comparison trials is proposed, which measures the two methods' overall agreement while correcting for each method's underlying inter / intra observer reliability. Cluster algorithms will be outlined, which allow to identify (in)compatible clusters of readings. Their application will be illustrated by means of the intraindividual comparison of two different strategies in radiographical imaging, where none of the underlying imaging methods can be regarded as a reference. RESULTS: The algorithms are illustrated by the comparison of two radiological imaging devices R and F, where none of these imaging methods could be considered as a valid reference, i.e. replicate readings by three independent radiologists were taken from each device, respectively. The setting allowed for intraindividual comparison of the imaging methods, since each of the three involved radiologists took one reading from both devices on each of 120 individuals. The algorithm identifies a subset of compatible reading patterns with an overall agreement of kappa = 0.83 (95% confidence interval 0.78 - 0.88) despite the fact, that the underlying readings arose from two different imaging devices. An obvious interpretation suggests, that the gradient in experience between the readers was more relevant to their reading patterns' outcome than any difference between the imaging devices. CONCLUSIONS: The generalized kappa coefficients can be modified according to the study design at hand to instructively identify (in)compatible clusters of multiple parallel reading patterns; the relative agreement of imaging methods can be estimated as well as each imaging method's internal reliability as assessed by parallel readings from the respective methods.