Modeling inter and intra observer variability of GTV contouring using deep learning

Seventeen patients with soft-tissue and bone sarcomas underwent pre-operative non-contrast CT at Massachusetts General Hospital (MGH), in accordance with the Institutional Review Board (IRB). The dataset consisted of 11 patients with soft-tissue sarcoma (including 5 extremity tumors) and 6 patients with bone sarcomas (chordomas).

The dataset was augmented by using the public soft-tissue sarcoma dataset from The Cancer Imaging Archive (TCIA) [19], which contains 51 patients imaged by CT with soft-tissue sarcoma confirmed by histology. The total number of patients used in this work was 68.

In order to evaluate the performance of the proposed neural network, a representative subset of the patients was excluded from the training procedure and reserved for evaluation. This evaluation dataset was selected to cover different types of tumors and was composed of two bone sarcomas (chordomas), and six soft-tissue sarcomas: one pelvic and five in extremities.

For each patient, 4 readers (radiologists and radiation oncologists) performed GTV delineation using the MIMVista software (MIM Software Inc., Cleveland, OH). GTV delineation was performed slice by slice and repeated three times by each reader, in a randomized order to avoid recall bias. Due to missing data for some patients, the total number of GTV contouring trials for each patient varied between 6 and 12. Example contours for a representative patient are shown in Figure 1(a), demonstrating the typical variability in GTV contours.

In order to model the variability, the contours were combined into a discretized confidence map. Each contour was first rasterized to generate a binary mask; masks were then summed and the result was quantized into 6 values (from 0 for pixels that were never included in a GTV contour to 5 for pixels included in every available GTV contour). An example of confidence map overlaid on the corresponding CT image is shown in Figure 1(b). Using discretized maps allows flexibility in the number of contours available for training.

In addition to the three trials, readers jointly determined a consensus GTV contour for each patient in a separate session for accuracy evaluation of both human contours and contours predicted by the network.

We developed a convolutional neural network to predict GTV confidence maps in order to model inter- and intra-reader variability. The training task is viewed as a multi-class segmentation problem, i.e. learning the discrete confidence level for each pixel, with the important characteristic that classes (i.e. confidence levels) are ordered.

The network was based on the U-Net [17] which has been extensively used for image segmentation tasks in the recent years [7, 20, 21]. The U-Net architecture is based on the combination of contracting/expanding paths with skip connections capturing spatial information. The contracting path is composed of successive convolutional layers interleaved with nonlinear activation (ReLU), batch normalization and pooling layers responsible for enforcing a sparse representation at the bottleneck level and increasing the receptive field by downsampling images at each level. The expanding path is constructed as the dual of the contracting path, replacing pooling with upsampling and adding skip connections. The final layer of the network is a convolutional layer with $C$ channels (where $C$ is the number of output classes). It predicts a probability for each class and each pixel.

In this work, the U-Net architecture was improved using several techniques (Figure 2). First, due to the limited amount of training data available and to limit the memory requirements, a so-called 2.5D U-Net was used where images are processed slice by slice while stacking neighboring slices in the channel dimension of the input images (5 slices were used in this work). This sliding-window approach enables the use of 3D contextual information without dramatically increasing the size of the network (and therefore the amount of labeled data required for training). Additionally, the network included attention gates in the skip connections as described in [22]. The goal of attention-based approaches is to focus the training on image features directly related to the output class. By using self-trained attention maps, the proposed network becomes less sensitive to class imbalance.

The loss function used to train the network was a modified cross-entropy loss, which can be expressed as:

where $C$ is the total number of confidence levels (6 in this work), $w_{c}$ is the class weight for class $c$ defined by the reciprocal of the class frequency, $\tilde{y}$ is the network output, $y$ is the ground-truth confidence map, $y_{j}^{(c)}$ is the known confidence level for pixel $j$ and class $c$, and $\tilde{y}_{j}^{c}$ is the network-generated probability for pixel $j$ and class $c$. To account for the ordering between confidence levels, network labels are transformed as described in [18]. Instead of encoding the target vector for a pixel $\left\{y_{j}^{(c)}\right\}_{c=1,\ldots,C}$ as $(0,\ldots,0,1,0,\ldots,0)$ with a probability 1 for the true class index $k$ (known as “one-hot” encoding), it is expressed as: $(1,\ldots,1,1,0,\ldots,0)$ where the first $k$ entries are 1. This change implies changing the final activation function from soft-max (traditionally used for classification) to sigmoid.

Network training was performed using the Adam optimizer [23]. Several techniques were used to avoid over-fitting [24]. Firstly, an $\ell_{2}$ penalty on the network weights was added to the loss. Secondly, dropout layers were inserted in the network. Dropout layers randomly disable a subset of network weights during training which typically improves the robustness of the network. Finally, data augmentation was used during training, including image mirroring, zoom and translation operations. Data augmentation adds variety to the training dataset and typically reduces over-fitting. To select hyper-parameters (learning rate, dropout rate, $\ell_{2}$ weight regularization), a 4-fold cross validation on the training dataset was performed [25]. Complete network parameters are given in Supplementary material S1.

Several metrics are commonly used to compare binary masks (e.g. Dice, accuracy, etc.). In this work, the output of the proposed neural network is a confidence map instead of a binary mask. Therefore, we use generalizations of common metrics to continuous masks as described in [26]. To calculate the metrics, the generalized true positive ($\mathrm{TP}$), false positive ($\mathrm{FP}$), true negative ($\mathrm{TN}$) and false negative ($\mathrm{FN}$) counts between a reference mask $\overline{m}$ and a given mask $m$, both continuous are defined as follows:

where $N$ is the total number of voxels in a region around the tumor. The region is based on the bounding box of the tumor, extended on all dimensions by 20% of the bounding box dimension. These definitions match the traditional binary definitions when the masks are binary. The volumetric metrics used for evaluation are:

Besides the continuous overlap metrics, the 95th percentile Hausdorff distance was also calculated [27]. Since this metric requires two binary masks (defined by their contours $\mathcal{X}$ and $\mathcal{Y}$), it can be calculated after thresholding the confidence maps and can be expressed as:

where $P_{95\%}(.)$ is an operator extracting the 95-th percentile.

Each metric can be computed between the predicted confidence map and the confidence map obtained by combining GTV contours from all readers and between predicted or manual contours and the consensus GTV contour established by all the readers.

The trained network was used to predict confidence maps on the evaluation dataset. Predicted confidence maps for eight patients are compared to the ground-truth human confidence maps in axial and sagittal views in Figure 3. The figure demonstrates a good agreement between predicted (bottom row) and ground truth maps (top row). The proposed deep neural network successfully predicts confidence maps for GTV that are comparable to confidence maps obtained from multiple readers.

The performance of the proposed network on the evaluation dataset was quantified using continuous Dice and accuracy metrics between predicted and true confidence maps. The proposed method achieves an average continuous Dice coefficient of 86.8% ($\pm 5.4\%$) and 95% ($\pm 1.5\%$) accuracy on average, demonstrating a good agreement between predicted and true confidence maps.

Besides the continuous metrics, a comparison between binary masks obtained by thresholding confidence maps was performed (see Table 1). This comparison helps characterize the performance of the proposed neural network for each confidence level. Results suggest a good agreement for all thresholds, with a slight degradation for the highest threshold, i.e. the highest confidence level. The threshold confidence level selected as starting point to draw clinical target volume (CTV) contours was three, for which a Dice score of 86.7% ($\pm 5.4\%$) was measured. For comparison, the inter-reader variability, measured by calculating the Dice score between pairs of contours from different readers, was also calculated. The Dice score between pairs of human was around 90.5% ($\pm 4.3\%$), suggesting that the level of agreement between predicted and true confidence maps is similar to the level of agreement expected between human readers.

Confidence maps for the evaluation dataset obtained with the proposed deep neural network and from the multiple human readers were compared to the consensus GTV, established by all the readers in a joint contouring session. When comparing a (discretized) confidence map with the consensus GTV, a threshold on the confidence level is first selected to convert the confidence map to a binary GTV mask.

The resulting metrics for varying thresholds are compared in Figure 4. The figure shows the Dice, sensitivity, specificity and 95th percentile Hausdorff distance between thresholded confidence maps (from human readers and the proposed neural network) and the consensus GTV. The range of metric values largely overlap which suggests that the proposed network generated reasonable confidence maps. While human GTVs tended to better match the consensus, Dice scores between thresholded predicted GTV and the consensus GTV were around 84.6% ($\pm 6.6\%$). This difference can be explained by the fact that consensus GTVs were drawn by the same human readers who performed the multiple GTV delineations, therefore introducing an unavoidable bias. The 95th percentile Hausdorff distance to the consensus GTV for predicted GTVs was typically within 10 mm of that of human GTVs. This confirms that the proposed network calculated GTV masks that are comparable to masks from different readers.

The human consensus GTV and predicted GTV obtained by thresholding the predicted confidence map for each patient in the evaluation were used by a human reader to determine the CTV. The CTV was drawn according to current standard clinical practice which includes a 3 cm longitudinal expansion into tissues of least resistance and 1 to 1.5 cm margin radially across tissues of greater resistance. The resulting CTV contours are shown, along with the corresponding GTV contours in Fig. 5. The agreement (Dice score) between CTVs derived from human consensus and predicted GTVs was around 89.5% ($\pm 1.8\%$), which is even higher than the agreement between the corresponding GTVs (84.6% $\pm 6.6\%$).