Observer variation in target volume delineation of lung cancer related to radiation oncologist-computer interaction: A 'Big Brother' evaluation

Background and purpose: To evaluate the process of target volume delineation in lung cancer for optimization of imaging, delineation protocol and delineation software. Patients and methods: Eleven radiation oncologists (observers) from five different institutions delineated the Gross Tumor Volume (GTV) including positive lymph nodes of 22 lung cancer patients (stages I-IIIB) on CT only. All radiation oncologist-computer interactions were recorded with a tool called 'Big Brother'. For each radiation oncologist and patient the following issues were analyzed: delineation time, number of delineated points and corrections, zoom levels, level and window (L/W) settings, CT slice changes, use of side windows (coronal and sagittal) and software button use. Results: The mean delineation time per GTV was 16 min (SD 10 min). The mean delineation time for lymph node positive patients was on average 3 min larger (P=0.02) than for lymph node negative patients. Many corrections (55%) were due to L/W change (e.g. delineating in mediastinum L/W and then correcting in lung L/W). For the lymph node region, a relatively large number of corrections was found (3.7 corr/cm2), indicating that it was difficult to delineate lymph nodes. For the tumor-atelectasis region, a relative small number of corrections was found (1.0 corr/cm2), indicating that including or excluding atelectasis into the GTV was a clinical decision. Inappropriate use of L/W settings was frequently found (e.g. 46% of all delineated points in the tumor-lung region were delineated in mediastinum L/W settings). Despite a large observer variation in cranial and caudal direction of 0.72 cm (1 SD), the coronal and sagittal side windows were not used in 45 and 60% of the cases, respectively. For the more difficult cases, observer variation was smaller when the coronal and sagittal side windows were used. Conclusions: With the 'Big Brother' tool a method was developed to trace the delineation process. The differences between observers concerning the delineation style were large. This study led to recommendations on how to improve delineation accuracy by adapting the delineation protocol (guidelines for L/W use) and delineation software (double window with lung and mediastinum L/W settings at the same time, enforced use of coronal and sagittal views) and including FDG-PET information (lymph nodes and atelectasis)