Inter-observer variation in masked and unmasked images for quality evaluation of clinical radiographs

Purpose: To investigate the influence of masking on the inter-observer variation in image quality evaluation of clinical radiographs of chest and lumbar spine. Background: Inter-observer variation is a big problem in image quality evaluation since this variation is often much bigger than the variation in image quality between, for example, two radiographic systems. In this study, we have evaluated the effect of masking on the inter-observer variation. The idea of the masking was to force every observer to view exactly the same part of the image and to avoid the effect of the overall 'first impression' of the image. A discussion with a group of European expert radiologists before the study indicated that masking might be a good way to reduce the inter-observer variation. Methods: Five chest and five lumbar spine radiographs were collected together with detailed information regarding exposure conditions. The radiographs were digitised with a high-performance scanner and five different manipulations were performed, simulating five different exposure conditions. The contrast, noise and spatial resolution were manipulated by this method. The images were printed onto the film and the individual masks were produced for each film, showing only the parts of the images that were necessary for the image quality evaluation. The quality of the images was evaluated on ordinary viewing boxes by a large group of experienced radiologists. The images were examined with and without the masks with a set of image criteria (if fulfilled, 1 point; and not fulfilled, 0 point), and the mean score was calculated for each simulated exposure condition. Results: The results of this study indicate that - contrary to what was supposed - the inter-observer variation increased when the images were masked. In some cases, especially for chest, this increase was statistically significant. Conclusions: Based on the results of this study, image masking in studies of fulfilment of image criteria cannot be recommended. (authors)