Observer variation--how many observers for generalizability?
Explore the source record for details and available documents.
Adult↗
SEARCH · PubMed Health
Results for “Observer Variation”
Explore indexed PubMed citations for clinical trials, systematic reviews and public health research. Read source abstracts and follow each citation to its original PubMed record.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
[Observer variation in clinical practice. A review].
Observer variation denotes the discrepancy between two consecutive observations of the same indicant. Principles and results of observer variation studies are outlined. Observer variation seems to be present wherever it is searched for, in history taking as well as in physical examination. It has been found unanimously to be smaller for intra- than for inter- observer variation while it is still controversial whether clinical experience lessens observer variation. Even fundamental clinical observations and particularly evaluations of borderline cases between normal and abnormal exhibit observer variation. The possibilities of reducing observer variation, and the importance of observer variation to clinicians and patients, and also the influence on research and society are referred to briefly. The future role of observer variation studies is discussed. Observer variation often is evaluated by the Kappa statistics whose statistical advantages and clinical disadvantages are outlined.
Observer Variation↗
Kappa statistics in the assessment of observer variation: the significance of multiple observers classifying ankle fractures.
Studies using kappa statistics have been conducted with a varied but limited number of observers. The aim of this study was to evaluate the significance of multiple observers on kappa as a measure of observer variation. One hundred orthopedic specialists were asked to assess a random sample of ten sets of standard radiographs of 94 consecutive patients with ankle fractures. The observers were randomly allocated into four groups, which again were divided into subgroups with an increasing number of observers. Random subgroups of three observers revealed kappa values from 0.20 to 0.64 in the Lauge-Hansen and 0.27 to 0.90 in the Weber classification system. With an increasing number of observers in the subgroups, kappa stabilizes around a mean value, indicating that the sampling variation and standard error decrease. The standard error found in this study makes kappa questionable as a measure for agreement among a small number of observers. Thus, kappa values obtained for a given diagnostic tool at one department are not directly comparable with results from other departments. We conclude that kappa cannot stand alone as a simple measure of observer variation.
Ankle Injuries↗
Melanocytic nevi in children. II. Observer variation in counting nevi.
A survey of benign melanocytic nevi was conducted among children in Perth, Western Australia, in 1985. Children were examined by one of a team of five nurses who counted all of their nevi. Linear regression was performed on the logarithmically transformed mole counts to assess the degree of interobserver variation. Among 2,354 white children aged 5-14 years, 3.9% of the variation in numbers of nevi of all sizes and 7.7% of the variation in nevi 2 mm or more in diameter was attributed to variation among observers. The observer variation was less for the back than for any other body site. Differences between observers were greater in children with heavy freckling than in children with little or no freckling. However, the interaction between degree of freckling and observer accounted for more than 1% of the variation in counts only at sites where freckling was common (the face and neck and the upper limbs). The reproducibility of counts made by pairs of observers on 236 of the children, estimated by intraclass correlation coefficients, was 0.79 for all nevi and 0.65 for nevi 2 mm or more in diameter. The mean count of nevi on the back obtained by examining photographs of 100 children aged 8-9 years (11.9 nevi) was similar to that recorded by the examiners (12.3; p = 0.31), and the two measures were highly correlated (r = 0.89).
Adolescent↗
Inter-observer and intra-observer variation in bone histomorphometry.
Inter-observer variation has been examined for a number of histomorphometric indices in 20 normal human iliac crest biopsies. Quantitation was performed using an eye-piece graticule and eye-piece micrometer. The same sections were examined by two observers and the methodology was identical. Intra-observer variation was also assessed. Significant inter-observer differences were found for the measurement of total trabecular bone volume, osteoid volume and surface, double plus single and double tetracycline labeled surfaces, and the mean osteoid seam width. The percentage variance due to inter-observer variation was highest for osteoid surface and volume, total resorption surface, and mean osteoid seam width. Intra-observer variation in both observers was small. We conclude that a large inter-observer variation may occur in the measurement of a number of histomorphometric indices, even when section preparation and methodology are identical. Caution should be used in basing the diagnosis of metabolic bone disease on strictly defined control data from other observers, particularly when this has been obtained from centers where the effects of inter-observer variation may be magnified by differences in methodology.
Bone Resorption↗
Is pancreatogram interpretation reliable?--a study of observer variation and error.
Observer variation in the interpretation of endoscopic pancreatograms has been examined by asking four experienced observers to assess 40 sets of well-documented radiographs (from 20 patients with cancer and 20 with pancreatitis), both without ("blind") and with clinical details, each on three occasions. Individual consistency for "blind" diagnoses ranged from 61% to 78%, increasing significantly with clinical information. Overall diagnostic accuracy with clinical information varied from 52% to 83% for cancer, and from 87% to 95% for pancreatitis. However, unanimous and correct opinions were given by the four observers for only 53% of all cases, even when clinical details were provided. Clinical information changed the radiographic diagnosis in 43% of assessments, 83% of these changes leading to improved accuracy. ERCP gives direct information about the major pancreatic and biliary duct systems and often provides an accurate diagnosis. Caution must be exercised in relying upon radiological appearances alone.
Diagnosis, Differential↗
Validity in diagnosing osteoporosis. Observer variation in interpreting spinal radiographs.
Observer variation in interpreting radiograph of the spine in 70-year-old Danish women has been examined. The women joined a general health investigation as part of an epidemiologic screening for osteoporosis. Two experienced radiologists examined independently 280 sets of lateral radiographs of the spine to disclose fractures. The radiographs were examined four times, each examined at two different occasions by each radiologist. Overall intra-observer agreement was 88% and 87% when asked for presence or absence of fractures, irrespective of fracture localization. When accidental occurrence of agreement was taken into account (kappa analysis) intra-observer variation was reduced to 64% and 62%. The overall inter-observer agreement was 75%, but reduced to 47% when accidental occurrence of agreement was noted. Kappa analysis in testing observer variation indicates that a considerable part of agreement can be ascribed to chance alone. This seriously questions the hitherto used definition of osteoporosis.
Aged↗
Observer variation in assessing renal scarring.
To examine the extent of variation between experienced observers in recognizing renal scarring, 60 intravenous urograms from schoolgirls with bacteriuria were presented separately to three radiologists with a major interest in the radiology of the urinary tract. Observers were asked to state whether scarring was "absent", "doubtful" or "present" at each of three sites in each kidney: the poles and the midportion. The proportion of overall agreement, i.e. the proportion of coincident opinions, was 86% to 89%. Results were also analysed by using the kappa statistic, which corrects for agreement due to chance. Kappa values ranged from 0.67 to 0.73. These values indicate higher levels of inter-observer agreement than those recorded for many other radiological examinations and are much greater than those noted by others for inter-observer variation in the recognition of renal scarring in urograms in adults. The likely explanations for these high levels of agreement are discussed. Observers disagreed about the extent of scarring in one-third of cases. Studies of the prevalence of scarring should take account of observer variation and should be based on judgments by more than one experienced observer.
Adolescent↗
Diagnostic quality and observer variation in radiographic diagnoses of approximal caries.
The aim of the study was to compare observer variations, observer strategy, and diagnostic quality with regard to radiographic caries diagnoses made at different depth levels. Qualitative assessment of approximal carious lesions on the basis of radiographs were made for sound (n = 28) and carious (n = 123) lesions by seven dentists. The inter- and intra-observer variations were lowest when lesions were diagnosed as being in the outermost parts of the teeth. The frequency of false positive scores was lower when dentin was examined as compared with enamel. The quality of radiographic diagnoses showed small variations (p greater than 0.05) when different levels of pulpal depths were interpreted.
Bicuspid↗
Observer variation in the diagnosis of thyroid disorders. Criteria for and impact on diagnostic decision-making.
The thesis is concerned with the reliability of thyroid diagnostic modalities. When no "true" diagnosis was available the reliability could not be evaluated by assessing the accuracy; in these cases the reproducibility was evaluated. This was done by evaluating the observer variation: comparison of observers' evaluation in pairs was assessed. The observer variation was expressed by the kappa coefficient, which adjusts to the amount of agreement that can be expected to occur by chance alone. Clinical estimation only of thyroid functional status was subject to considerable observer variation. Likewise clinical estimation the thyroid gland (size and morphology) was subject to considerable observer variation, and the observer variation was greatest concerning the palpation of a solitary thyroid nodule where even experienced doctors had difficulties in reproducing their own evaluations. In contrast high reproducibility was found concerning the interpretation of routine thyroid blood tests. Thyroid scintigram is not suitable for thyroid size estimation. Assessment of thyroid morphology (diffuse, multinodular, solitary hot or cold lesion) based on scintigram alone seldom resulted in acceptable reproducibility: four doctors diagnosed a solitary cold lesion in 100 of 480 scintigrams, however, they only agreed in 30 of the cases. A solitary cold lesion may in a few per cent of the cases imply malignancy and will therefore lead to further investigations. The reproducibility was not improved when the doctors had additional access to patient history and clinical examination as well as the scintigram. Ultrasound is increasingly often regarded as the "true" diagnosis in the morphological description of the thyroid gland. In a study of two specialists in thyroid ultrasound these descriptions were, however, found not to be reproducible. Thus ultrasound should be applied with caution as the golden standard with regard to thyroid type. It has been demonstrated that thyroid imaging is subject to considerable observer variation. This has great implications when thyroid imaging is used as the golden standard, for example when patients are selected for scientific studies. Furthermore, this may influence the comparison of different studies because of inconsistency of the material included. Therefore the observer variation should appear from description of material and methods in scientific studies. Palpation of thyroid size is neither accurate nor reproducible if an exact volume is aimed at. A rough division into small, medium, and large goitres is reproducible and rather accurate and may therefore be applied. In a prospective randomised study it was demonstrated that this division could replace ultrasound estimation of thyroid size in the calculation of 131I dose in the treatment of hyperthyroidism. Evaluation of thyroid functional status cannot be based on patient history and clinical examination alone, it should include thyroid blood tests. Exact estimation of thyroid size should be performed by means of ultrasound. When morphology of the thyroid is assessed the observer variation should be kept in mind. The existence of observer variation implies uncertainty concerning patient material and inclusion criteria in scientific studies. It may complicate comparison of scientific results. In thyroid diagnostic decision-making doctors should keep in mind that even though there is agreement concerning thyroid blood tests there is not agreement concerning thyroid imaging and the clinical evaluation of patients suspected of thyroid disorders. The patients may experience these disagreements even if the same doctor follows them. Doctors are obliged to use the existing modalities and should be aware of not only problems with technology but also variations originating from the observers, the doctors themselves--the human factor.
Humans↗
Observer variation in histological classification of cutaneous malignant melanoma.
To evaluate the variations within and between observers in the interpretation of important histological prognostic factors, a series of 96 melanoma patients was randomly selected from a database of 1691 patients with cutaneous malignant melanoma. The stained sections were examined on two occasions by four experienced pathologists. Analysis by observed agreement and kappa statistics showed maximal tumour thickness to be the best reproducible variable, with ulceration the second best. Regression was the least reproducible, with level of invasion and type of melanoma in the mid range. Intra-observer variation was uniformly less than inter-observer variation for each variable. For tumour thickness a variance component analysis was done to quantify the variability further. The clinician should not base his choice of treatment entirely on the microscopic classification but take into consideration the clinical course and appearance of the tumour.
Analysis of Variance↗
Observer variation in the sonographic measurement of optic nerve sheath diameter in normal adults.
AIM: To quantify the observer variation in the sonographic measurement of optic nerve sheath diameter (ONSD) in normal adults. MATERIALS AND METHODS: Sixty-seven normal adult volunteers underwent ultrasound examination of each eye by three independent observers using a 7 MHz sector probe. Three measurements were made of each eye by each observer and a mean value calculated for each eye. Median and 5th-95th centile values for both intra- and inter-observer variation were calculated for the three observers. RESULTS: The median intra-observer variation was +/-0.1 mm, with 5th-95th centile values of +/-0-0.4 mm. The median inter-observer variation was +/-0.2-0.3 mm, with 5th-95th centile values of +/-0-0.7 mm. Careful review of examination technique by the three observers after the first 17 examinations was shown to reduce both intra- and inter-observer variation. CONCLUSION: The sonographic measurement of ONSD is a readily learned, reproducible technique with low intra- and inter-observer variation. The average inter-observer variation (+/-0.2 mm) is comparable to the inherent variability of the ultrasound machine. The importance of standardisation of examination technique is stressed.
Adult↗
Assessment of dysplasia in colorectal adenomas: an observer variation and morphometric study.
Observer variation in the grading of dysplasia in 100 colorectal adenomas has been analysed by kappa statistics. Intraobserver agreement was only 70% and 67% for the two principal observers, and, as would be expected, interobserver agreement was even lower at 59% and 66%. Although the kappa values were significantly different from chance at the 0.1% level, there were substantial disagreements. When the study was extended to four observers, agreement between observer pairings was considerably worse (as low as 34%), and in four pairings the kappa values did not differ significantly from those expected by chance alone even at the 5% level. In an endeavour to improve agreement we adopted a percentage estimation grading method; but this failed to achieve any improvement when comparing overall grades. The percentage estimates of the two observers, however, showed a highly significant correlation. To identify the cytological features given most weight by the principal observers in assessing dysplasia we undertook morphometry on 30 adenomas using an image analysis computer. The nuclear to cytoplasmic ratio, variation in nuclear area, and variation in nuclear height above the basement membrane showed significant differences between mild, moderate, and severely dysplastic epithelia. While evaluation of these parameters therefore appears to be most important in the subjective interpretation of dysplasia, this study has shown that such evaluation is poorly standardised between observers and poorly reproduced within observers. Our findings of poor agreement in the grading of dysplasia in colorectal adenomas has serious implications for the assessment of dysplasia in inflammatory bowel disease, where the added problem of reactive cellular atypia brings greater complexity to these subjective judgments.
Adenoma↗
Observer variation in skeletal radiology.
The factors that affect observer variation in bone radiology are analysed from data in the literature and on the basis of studies carried out at McMaster University on the hands and sacroiliac joints. A plea is made for presenting results in terms of Kappa statistics so that agreement due purely to chance is eliminated. In the conclusions the main variables that affect concordance are listed so that strategies can be developed to reduce observer variation. This is important in serial studies to ensure that the observer variations are smaller than the effect one wishes to measure.
Age Determination by Skeleton↗
Inter-observer variation in masked and unmasked images for quality evaluation of clinical radiographs.
PURPOSE: To investigate the influence of masking on the inter-observer variation in image quality evaluation of clinical radiographs of chest and lumbar spine. BACKGROUND: Inter-observer variation is a big problem in image quality evaluation since this variation is often much bigger than the variation in image quality between, for example, two radiographic systems. In this study, we have evaluated the effect of masking on the inter-observer variation. The idea of the masking was to force every observer to view exactly the same part of the image and to avoid the effect of the overall 'first impression' of the image. A discussion with a group of European expert radiologists before the study indicated that masking might be a good way to reduce the inter-observer variation. METHODS: Five chest and five lumbar spine radiographs were collected together with detailed information regarding exposure conditions. The radiographs were digitised with a high-performance scanner and five different manipulations were performed, simulating five different exposure conditions. The contrast, noise and spatial resolution were manipulated by this method. The images were printed onto the film and the individual masks were produced for each film, showing only the parts of the images that were necessary for the image quality evaluation. The quality of the images was evaluated on ordinary viewing boxes by a large group of experienced radiologists. The images were examined with and without the masks with a set of image criteria (if fulfilled, 1 point; and not fulfilled, 0 point), and the mean score was calculated for each simulated exposure condition. RESULTS: The results of this study indicate that-contrary to what was supposed-the inter-observer variation increased when the images were masked. In some cases, especially for chest, this increase was statistically significant. CONCLUSIONS: Based on the results of this study, image masking in studies of fulfilment of image criteria cannot be recommended.
Computers↗
Inter-observer variation in the results of the clinical andrological examination including estimation of testicular size.
Inter-observer variation in andrological examination by 10 clinical investigators from five Nordic and Baltic countries was investigated. In addition, information on intra-observer variation was obtained for six of the 10 investigators. Testicular size was measured using Prader's orchidometer and one of the investigators also performed an ultrasound estimate of testicular size. A highly significant difference (p < 0.001) between observers was found with an inter-observer error of 16% in estimating testicular size in 23 young men. The difference in the estimate tended to increase with increasing testicular size. There was no significant intra-observer difference in two measurements performed on consecutive days. Only differences in median testis size, which were greater than 31% between measurements by two investigators, were found to be significant at the 5% level. The ultrasound estimate of testicular size was significantly lower than the orchidometer estimate, with a mean difference of 3.6 mL for the left testis and 4.3 mL for the right testis. Tanner staging of genitalia and diagnosis of a varicocele was subject to great inter-observer variation, and for the diagnosis of varicocele only one-third of the investigators was able to reproduce their results on a second examination. In conclusion, it was found that the clinical andrological examination of young men is subject to great inter-observer variation. This should be kept in mind when results from different studies are compared as well as in daily clinical practice.
Adult↗
Observer variation in the assessment of the histopathologic diagnosis of acute appendicitis.
BACKGROUND: The objectives of this study were to determine observer variation in the assessment of the histopathologic diagnosis of acute appendicitis. METHODS: Two consultant pathologists independently evaluated 415 appendectomy specimens (set I). After a consensus conference at which the observers unified the criteria of the diagnosis of acute appendicitis, another 396 appendectomy specimens (set II) were evaluated. To calculate the intra-observer variation, one observer evaluated set II once more. Finally, using the consensus diagnosis as the final diagnosis, the diagnostic accuracy of the surgeon, the primary pathology report, and the two observer evaluations was calculated. RESULTS: In set I no difference was found in the level of agreement on the diagnosis of acute appendicitis versus other diagnoses, and a kappa value of 0.85 was obtained. In set II a kappa value of 0.88 was obtained, which was not significantly different from the kappa value in set I. However, the consensus conference did increase the level of agreement on the diagnosis of acute appendicitis. The intra-observer variation was calculated, and a kappa value of 0.88 was obtained. The cases in which the observers disagreed on a positive diagnosis were cases of mild appendicitis. The question remains how many specimens would have shown these changes if all tissue had been developed for microscopy. The observers' diagnoses were the most reliable, and there was a significant decrease in the diagnostic accuracy compared with both the primary pathology report and the surgeon's opinion, the last one being the least accurate. CONCLUSIONS: These observations stress how important it is for the pathologist to discuss the diagnostic criteria of the diagnosis of acute appendicitis and for the surgeon to go back to the patient's record and look up the histopathologic diagnosis.
Acute Disease↗
Observer variation in quantification of immunocytochemistry by image analysis.
This paper reports the findings of a study designed to examine observer variation as a source of inaccuracy inherent in the use of computer-assisted image analysis to measure areas of stained tissue. The rat pituitary immunostained for prolactin and galanin was used as an example to estimate patterns of immunoreactivity exhibited by different cell types. Six observers, with differing experience, selected grey level threshold values on 40 fields of images of stained tissue making three repeats of each field. The 40 fields consisted of 20 serial pairs of colocalized fields, one immunostained for prolactin, the other for galanin. The 20 pairs consisted of four pairs from each of five animals. Analysis of observer variation in the selection of threshold values showed large differences in the within- and between-observer variation. Analysis of the components of variance in the estimation of the ratios of stained tissues showed that the major source of variation was the within-observer component. An additional experiment using two observers, where half of the images were compared to the original microscope images before setting threshold levels, showed that the opportunity to make a comparison did not reduce observer variation. It is suggested that any study which uses semi-automatic methods to segment regions of a digital image can benefit from an analysis of this kind so that the sources of variation can be determined to enable maximum discriminating power in future studies.
Animals↗