Multi-Attribute Attention Network for Interpretable Diagnosis of Thyroid Nodules in Ultrasound Images

There has been a lot of research on developing CAD systems for thyroid cancer diagnosis. Most of the traditional approaches combine feature extraction methods with classical machine learning classifiers. For example, Keramidas et al. [12] extracted fuzzy local binary patterns as noise-resistant textural features and used the support vector machine (SVM) as the classifier. Similarily, Ardakani et al. [13] also used SVM to classify 49 morphological or textural features extracted from nodule images. Tsantis et al. [14] calculated the wavelet local maxima and the shape features to characterize the segmented nodules. Singh et al. [15] used the gray level co-occurrence matrix features to construct a k-nearest neighbor (KNN) model for thyroid nodule classification. Acharya et al. [16] utilized gray-scale features calculated based on stationary wavelet transform. They also compared the performance of several commonly-used classifiers. Raghavendra et al. [17] used the fusion of Spatial Gray Level Dependence Features (SGLDF) and fractal textures to decipher the intrinsic nodular structure. Although of the efforts, as thyroid nodules vary in location, shape, size, and internal characteristics, these hand-crafted features may have difficulty in accommodating all scenarios and differentiating the malignant nodule from the benign ones.

Recently, deep learning (DL) based methods have been adopted to design similar CAD systems. Some of them require human-intervention or pre-processing before analyzing. For example, in Ma et al. [18], they required radiologists to select the regions of interest (ROIs) in original US images. They then used a cascaded CNN model to classify these ROIs. Similarly, Guan et al. cropped the nodule region based on manual annotation and used the cropped images to train an Inception-V3 model [10]. In Chi et al. [11] , all US images are pre-processed to calibrate their scanning scale and remove possible artifacts. The authors then fine-tuned a GoogLeNet model as a feature extractor to train a Random Forest classifier. These methods could add clinical burden or may have difficulty in adapting to new datasets with different image appearances.

Others, therefore, turn to fully automated approaches by adding an additional localization stage to remove the background regions. For example, Liu et al. [8] proposed a two-stage CAD system, which used a multi-scale region-based detection network for nodule detection and a multi-branch classification network to capture and enhance group characteristics for the final malignancy classification . Buda et al. [9] first detected the nodules using the Faster-RCNN model and cropped the US images to the nodule area. The cropped images are then fed into the second CNN network to classify whether the nodules are benign or malignant. Despite exhibiting good performance, these two-staged methods take risk of depleting potential context information and error propagation from the detection stage to the classification stage. In other words, incorrect detection results will propagate the errors to the classification stage and result in incorrect or unreliable predictions. Therefore, an end-to-end approach is desired, to can handle the origninal US images directly.

Note that most of the aforementioned methods only offer the malignancy prediction result and fail to explain how the decision was made [19]. It can cause confusion and misunderstanding when applied in a real clinical scenario, which may block a wider adoption of DL algorithms. There has been an increasing need for interpretable deep learning diagnosis systems to overcome this dilemma [20].

Beside the main task, many methods try to combine different information from different tasks to improve the performance and generalizable ability of the model [21]. The joint training can help the model learn the specific features of the tasks and share the common features representation to reduce the overfitting. For example, Wang et al. [22] modified the structure of U-Net by adding a classification branch for classification and segmentation of bone surfaces. Murugesan et al. [23] proposed a deep model to predict the shape and boundary information for optic cup and disc segmentation. Sbhankar Roy et al. [24] developed a model to simultaneously predict the COVID-19 disease severity score and provides location of pathological artefacts . These methods are not only help improving the tasks together but also provide the auxiliary predictions to assist the diagnosis.

Considering the TI-RADS attributes are widely used to translate the diagnosis to human-understandable characteristics. We incorporate the learning and prediction of TI-RADS attributes into our model design, which adds the model interpretability naturally.

The visual attention-based method allows a deep model to focus on important regions of an image adaptively. To provide visual attention maps for deep neural networks, many interesting approaches have been proposed in [25],[26], which are mainly two types: gradient-based visual explanation and activation-based visual explanation. The former uses gradients of a target concept to produce a heatmap of regional importance, while the latter uses the response of a feed-forward propagation. Local interpretable model-agnostic explanations (LIME) is a technique to explore the behavior of the models from variously deformed input to provide visualization [27]. Visual attention-based CNNs have also been applied in medical image processing. For example, Ouyang et al. proposed a dual-sampling attention network to classify the COVID-19 and CAP infection [28]. They proposed an online module to utilize the segmented pneumonia infection regions to refine the attention for the network. Wang et al. used an attention module to focus the learning on small targets to segment tumors in high-resolution T2-weighted MRI [29]. Yan et al. proposed an attention-based method for melanoma recognition, in which the attention maps are learned together with other network parameters [30].

Note that these attention methods usually only produce one attention map for each image, as they mostly deal with images containing one single class. On the contrary, our task here is classifying multiple attributes and malignancy for each image. This task is different as an ideal model needs to put attention on various image regions to predict each task correctly. To solve this, we propose a novel attribute-specific multi-attention scheme that can adapt to different attributes freely. Experiment results show that this design can help boost the model performance and robustness.

Figure 2 offers a schematic of the overall framework. An original image is first processed by the backbone $f(I)$, which maps the inputs $I$ to high-level features. In specific, ResNet50 is adopted. The extracted features are then fed into the attention branch to generate suitable attention heatmaps for each task. These heatmaps are then multiplied with the obtained feature maps and fed into the attribute branch.

The attribute branch consists of several paralleled streams, corresponding to each desired nodular attribute, i.e. the shape, calcification, ratio, boundary, margin, and echo uniformity. Each stream has three residual blocks (3x3 convolutions, the first block has a stride of 2) to learn attribute specific features, a global average pooling (GAP) layer to summarize those features and a fully connected (FC) layer to yield the final prediction. To imitate the clinician diagnosis process, the model concatenates the six attribute-specific FC layers together to predict the nodule malignancy (shown as ‘cancer’ branch in Fig. 2). By this way, the proposed model explicitly imitating the clinical work-flow in the ‘diagnosis-making’ process. Note that no additional information is used except for the features of the nodular attributes, reinforcing the model interpretability.

The ground truth of the model is a 1D vector, denoted as $L=\left[l_{1},l_{2},\ldots,l_{c}\right]$ in which $l_{c}\in\{0,1\}$, $\mathrm{C}=7$, $l_{c}$ presents the probability of being malignancy or belonging to a certain class of the corresponding attribute. A sigmoid layer is added to normalize the predicted output $p_{attr}(c\mid I)$ by:

, where $I$ is the input image, $\tilde{p}_{attr}(c\mid I)$ presents the probability score of I belonging to the $c^{th}$ class, $c\in\{1,2,\ldots,C\}.$ We optimize the parameter of the attribute branch using the binary cross-entropy loss (BCE) loss:

, where $l_{c}$ is the ground truth label of the $c^{th}$ class.

The human can direct their attention to the region of an image that is the most relevant to a specific task. For example, given the same US image, doctors usually look at the edge of the nodule to recognize whether its boundary is clear or blurry, while paying more attention to the internal region of the nodule to detect the existence of calcification. Inspired by this, we propose a flexible attribute-adaptive attention scheme to instruct the model to focus on different image regions in predicting various attributes.

The structure of the attention branch is illustrated in Fig. 3. The extracted features are convoluted by 3x3 kernels with a stride of 1. A 1x1 convolution is used to aggregate the feature maps into a spatial size of 7x14x14. It is then sequentially normalized by the sigmoid function to get 7 different attention maps. Six of them are connected to the six attribute streams, respectively. The other attention map is used to impose an additional spatial constraint and will be explained in detail later. The 6 attribute-specific attention maps can highlight different regions of the image that is the most relevant to the corresponding attribute. By multiplying them with the original feature map, the model can put more emphasis on relevant information while avoiding distractions from the other. Formally, the filtered feature map $f_{c}\left(I\right)$ for the attribute $c^{th}$ is calculated as:

where $M_{c}\left(I\right)$ are the the attention maps. Note that, the attribute branch is also passed to a GAP layer and a sigmoid layer to add an auxiliary supervision for the malignancy and 6 attributes. This is then used to calculate the attention loss to supervise the attention branch directly, in specific:

where the $\tilde{p}_{attn}(c\mid I)$ presents the probability score of the attributes at the end of the attention branch. The $L_{attn}$ is used to put extra supervision for the attention maps.

Thyroid nodules have varying sizes and indefinite spatial locations within images. Existing methods usually crop or segment the regions of thyroid nodules first, which may lose context information and could lead to error-propagation. Instead, we propose to use the delineation of nodules to constrain the model’s attention while retaining all available information. In specific, we force the model to attentively learn the prior spatial information of the 7th attention map (introduced in Section III.B) by matching with the mask of the original nodule (see Fig. 2).

To achieve this, the model penalizes the following loss:

, where N denotes the number of pixels in the attention map, $p_{i}$ and $g_{i}$ represent pairs of corresponding pixel values of attention map and the delineation mask, respectively. The pixel value of $p_{i}$ and $g_{i}$ is normalized in range of [0;1]. Note that this additional loss is not directly imposed on any of the attribute streams as an ideal attention map may not be limited to the nodule region. For example, classifying the ‘echo uniformity’ attribute requires knowledge of both the nodule region and all surrounding tissues. However, the spatial location and the shape of nodules are expected to be informative. Therefore, we generate an extra attention map for this loss which influences the generation of other attention maps through weight-sharing kernels. It can also serve as a localization head during the test. Later experiments validate that this spatially constrained attention map aligns well with the nodule, and it can increase the classification accuracy of nodular attributes as well.

The overall loss function of the whole framework is therefore defined as the combination of three. Formally:

, where $w_{attr}$, $w_{attn}$, $w_{spatial}$ are the weights of the corresponding loss $L_{attr}$, $L_{attn}$, $L_{spatial}$. Empirically, the $w_{attr}$ is set to 1 for the main malignancy and attributes classification task. The $w_{attn}$, $w_{spatial}$ of attention loss and spatial loss are secondary for extra supervision and localization task and are set to 0.5 to not dominate the main task and better training convergence.

The thyroid dataset was collected from over 20 different sites and hospitals to capture various kinds of scans in a real clinical setting. In total, there are 4554 ultrasound images of thyroid nodules of 4554 patients with an age range from 9 to 82 years. The collected images contain only single nodule with the corresponding attributes and spatial mask information. The ground truth classification labels were obtained based on the corresponding biopsy report, while other labels were annotated by doctors with over 20 years of experience. The annotation process followed the TI-RADS guidelines [3] and the American Thyroid Association (ATA) guidelines [12]. The images were randomly shuffled and split to training (60%), validation (20%), and testing (20%) sets.

This rich dataset contains 2D ultrasound images acquired from a wide range of ultrasound machines e.g. Mindray DC-8, Mindray RESONA7, Philips-cx50, Philips EPIQ7, GE VOLUSON S8, TOSHIBA Aplio 300, SIEMENS ACUSONS 2000, SIEMENS ACUSON NX3 Elite, and Esaote Mylab. Note that the image contrast, overall intensity distribution, and the scanning window size were different. Fig. 4 shows some typical examples. It also demonstrates that the sizes of the nodule have large variations (e.g. Fig. 4(b) and (h)). Moreover, their shape also varies (see Fig. 4(a) and (f)). The nodules can be cystic (Fig. 4(h)) or solid (Fig. 4(g)) and appear at different positions in the images. The total numbers of benign and malignant nodules and other attributes are shown in Fig. 5. It should be pointed out that this dataset was collected to approximate real-world clinical settings, while most existing medical datasets were collected using certain types of machines in one center/hospital. A robust algorithm that can perform well on such a challenging dataset is expected to perform similarly in a real-world application.

The images are first converted to gray-scale in PNG format. To preserve the original aspect ratio of the nodules, the images are padded to a square shape before resizing to the same size of 224x224. No other preprocessing was performed. In training, data augmentation was carried out by resizing, random cropping, and random horizontal flipping. The ResNet50 with ImageNet pre-trained weights was used as the baseline. The network was trained for 150 epochs with a batch size of 20. The initial learning rate was 0.01 and divided by 10 after 20 epochs. The network was optimized using stochastic gradient descent (SGD) with a weight decay of 0.0001 and a momentum of 0.9. The MAA-Net was implemented with the Pytorch framework. We used a Geforce GTX 1080 Ti GPU for the training and evaluation.

Comparison with state-of-the-art methods. To demonstrate the efficacy of the proposed framework, we implemented some state-of-the-art methods using our dataset for a fair comparison. Intuitively, we implemented some classical classification methods such as ResNet50 [31], VGG-16 [32], and DenseNet201 [33]. Furthermore, we also chose some of the most related works on DL-based thyroid nodule diagnosis [9][10][11]. Note that these methods [9],[10],[11] require an additional stage of localizing or pre-processing the ROIs using either human annotation or additional models.

To investigate whether a single simple CNN model can combine the two tasks, we designed two additional comparison models which can solve the recognition and the classification task. In specific, the classical U-Net [34] model was chosen as the backbone for its simplicity. In ‘UNet-Simul’ (Table I), a classification head is added to the middle of the U-Net to predict the nodule malignancy. In the ‘UNet-2Stage’ model, we investigated another way to combine the two tasks where the segmentation result is used to crop the nodule ROI and is then fed to another ResNet50 classification network. The experiment results are shown in Table I.

Ablation study. To evaluate the effectiveness of different components in our MAA-Net framework, we performed the following ablation experiments: 1) Feature extractor + single malignancy output (Baseline). 2) Feature extractor + Attribute branch (Baseline+Attr). 3) Feature extractor + Attribute branch+ Attention branch (Baseline+Attr+Attn). 4) Feature extractor + Attribute branch+ Attention branch + Spatial loss (the proposed MAA-Net).

Cross-validation. To further estimate if the proposed model can generalize to the whole dataset, a 5-fold cross-validation experiment was performed. The dataset was randomly partitioned equally into 5 sub-sets. Each cross-validation model was trained using the same hyper-parameters and data augmentation method described in Section IV-B.

To quantitatively evaluate the proposed MAA-Net model, we used the accuracy (Acc), F1score, specificity (Spec), precision (Prec), recall (Rec), and the area under the receiver operating characteristic curve (AUC) as performance metrics, which are defined as:

, where TP, FN, TN, and FP refer the number of true positive, false negative, true negative and false positive cases, respectively. $t_{pr}$ is the true positive rate, $f_{pr}$ is the false positive rate, and X0 and X1 are the confidence scores for a negative and positive instance, respectively.

The experiment results of the performance comparison between the proposed MAA-Net and other state-of-the-art methods are shown in Table I. The column ‘Multi-stage’ indicates whether the method has two stages, while the column ‘Pre-process’ shows whether it requires pre-processing. It shows that the proposed model achieved the highest accuracy (up to 0.836), AUC score (up to 0.906) on this challenging dataset while requiring no human intervention or multi-stage. This proves that the proposed MAA-Net can process original thyroid nodule images collected from various types of machines without pre-processing and is capable of dealing with variation of nodule location, size and appearance through learning the regions and context information.

Table I also shows that popular computer vision models such as ResNet50, VGG-16, Densenet201 ([31],[32],[33]) achieved relatively inferior performances. It might result from that these models were not specialized for this challenging original images with high variation characteristics of nodules. Among them, the DenseNet201 achieved higher accuracy, which may result from its rich connection among layers.

Note that some two-stage models achieved better accuracy and the AUC (e.g. [9] Acc=0.786 and AUC=0.842). However, it is still lower than our methods. We conjecture that it might suffer from false detection in the first stage, which leads to incorrect cropping of the image and depleting the informative regions. The classification network cannot make accurate predictions on those images that might not even contain the targeted nodules. On the contrary, our method process the original images to predict in one shot therefore avoids similar errors.

Other methods (e.g. [10]) avoid these kinds of errors by using manual pre-processing. We implemented these methods faithfully by using manually cropped images to train the corresponding models. According to Table I, [10] achieved better accuracy than those two-staged methods (e.g. [9], UNet-2Stage), but was inferior to the MAA-Net. We conjecture that the surrounding tissues around the nodules may contain useful context information, such as whether the nodule tissue is solid or cystic. Therefore, the removal of the non-nodule image region might not cause information loss.

This finding also validates our model design which uses nodule delineation to constraint the model attention, rather than eliminating the background region completely. Note that the predicted attention maps can serve as a rough localization tool. We calculate the percentage of the predicted attention maps that have larger than 50% overlap with the ground truth (shown as hit rate in Table I). It can be seen that the predicted malignancy attention correlates well with the nodule location. It achieved a high hit rate (ours=0.88) similar to the separately trained localization models (e.g. that of [9]=0.91).

It is also interesting to see that the UNet-Simul did not achieve comparable performance as expected. We reckon it might result from the competition between the two tasks (segmentation and classification) that lead to performance degradation in both. The UNet-2Stage avoids this competition by separating the two tasks. Table I shows that the latter scored higher classification performance compared to UNet-Simul, validating our previous conjectures. These experiments were also part of our inspiration to design a model that can exploit nodule location information without sacrificing classification accuracy. The approach in [11] also achieved relatively good accuracy. However, it necessitates pre-processing to calibrate the scanning scale and remove possible artifacts. It then requires time and effort to generalize to images collected from new or un-seen types of US machines, which is frequently encountered in real-world settings. Meanwhile, only the proposed method and [9] can predict the nodular attributes (marked ‘yes’ in the ‘Interp’ column), thus provide more interpretable results for the doctors. In the ablation study, we report the model accuracy in predicting these nodular attributes in detail.

Table II reports the results of the ablation study. It shows that adding the attribute branch with single attribute to the baseline (Baseline+C,S,R,B,M,E) can help to improve the malignancy classification accuracy. In addition, combining some attributes can not only improve the malignancy prediction but also simultaneously improve the prediction of attributes itself. As a result, combining all six attributes together (Baseline+Attr) help the baseline get the most improvement to accuracy of 0.795. It proves that the extra knowledge of the nodular attributes is indeed beneficial to improve the final malignancy prediction. Meanwhile, adding the attention branch (Baseline+Attr+Attn) also played a positive role in boosting model performance (see Table II). The AUC results of six attributes are improved substantially compared to the Baseline+Attr. As a result, the classification accuracy for malignancy increases from 0.795 to 0.804, and the average AUC increases from 0.790 to 0.826. We conjecture that predicting multiple attributes is complex and may require the model to concentrate on different regions or utilize different features. Therefore, the proposed multi-attention scheme enables such dynamic learning. Visualization results also prove that the attention branch highlighted various local regions to better discriminate different attributes (see Fig. 7).

Finally, the adding of the spatial attention loss resulted in the full MAA-Net, which further improved the average AUC of attribute prediction to 0.834 and Acc to 0.836. The spatial loss give a significant improvement in our method. Due to the challenge of using the original images instead of cropping the nodule, we conjecture that the spatial loss can give the rough localization of the nodule area which benefits the prediction of malignancy. In clinical routine, the localization is also a importance step to analyze the thyroid nodule. Compared with results showed in Table I, we argue that this design is more efficient and less error-prone than the two-staged methods. Furthermore, this attention maps can be generated automatically during the test and provide a rough localization of the nodule. Specifically, the precision is improved to 0.859 showing that our method can have less false positive and prevent the unnecessary biopsy as we expected.

The Fig. 6 visualizes the ROC curves of the ablation experiments over the nodule malignancy and attributes predictions. We can see that the attention mechanism and spatial loss can consistantly improve the true positive rates and AUC of all the attributes showing the effectiveness of the proposed components. As a result, the AUC of the malignancy diagnosis of MAA-Net can be largely improved (0.906) compared with the baseline (0.839).

To further investigate whether the model performs consistently on this challenging dataset, we carried out a 5-fold cross-validation experiment. Table III reports the average accuracy of the MAA-Net model trained and tested using different folds. It shows that the proposed model is effective for all partitions of the dataset. It is also interesting to see that the model scored higher accuracy in recognizing the existence of calcification while achieved lower accuracy in predicting whether the boundary is clear or not. These results coincide with clinical studies on the inter-observer consistency in labeling these attributes [6],[7].

To better understand the reasoning of the diagnosis, we provide visualization of the attention heatmaps generated for recognizing the attributes and the malignancy. Fig.7 provides four different examples for comparison in terms of malignancy and attributes. The prediction decisions and corresponding probability of the attributes provide the useful information for the doctor to understand. For example, the first row image is diagnosed as benign with high confidence because it has high probability of no calcification, round shape and clear boundary. In contrast, the third row image is malignant because it has high probability of calcification, H/W ratio $>$ 1, irregular shape and margin. These information consist with the TI-RADS guidelines [5]. Generally, the malignant attributes in row 3 and 4 have more high probability than benign cases in row 1,2 showing the better discrimination ability of model.

The two right-most columns show the original US images and the heatmaps generated to provide nodule location information. Note that the nodule regions are highlighted consistently thanks to the spatial attention loss, despite their contrasting sizes. Meanwhile, the column 1-7 are heatmaps correspond to the targeted nodular malignancy and attributes. It shows that the model concentrates on varying parts of the image to recognize different attributes. For instance, the ‘Calcification’ and the ‘Echo uniform’ heatmaps learn the region inside the nodule. The ‘Boundary’, ‘Margin’ heatmaps highlight the pixels around the nodule edge. This finding echoes with the previous conjecture that the model should focus on different image regions to better classify different attributes. Our attribute-based multi-attention scheme suits better to our multi-label task than the popular single attention scheme.