Evidence-aware multi-modal data fusion and its application to total knee replacement prediction

The overall framework of the proposed evidence-aware multi-modal data fusion is shown in Fig. 1. In general, it contains backbone classification networks, evidence estimation networks, and the DST fusion module. The backbone network commonly includes two branches [9, 22, 27], namely, image branch and non-image branch. Concretely, each branch comprises a feature extraction backbone and a classifier to output a probability based on the corresponding data. The features from two branches are concatenated to form the third branch and are attached to a third classifier for prediction.

Decision-making level fusion of multi-modal data are shown to be effective in medical condition prediction [9, 22, 27]. In our task, considering that the quality of various modalities may vary, the decisions from each branch vary too, which results in conflict of prediction. The aim of our work is to provide a reliable fusion strategy for multi-modal data to improve the prediction accuracy. To this end, we introduce an evidence network to output an evidence score for each branch. The evidence network is designed to estimate the reliability of the data and the output of each branch. Finally, Dempster’s combination rule is employed to fuse the information of image branch, non-image branch, and fusion branch according to the estimated evidence scores to obtain the final prediction results. In this way, the decision from each branch can be considered accordingly based on the reliability estimation, alleviating the clash of the decision during the decision fusion stage, leading to a reliable prediction.

To reliably use the decisions from multiple modalities, we need to estimate the reliability of each modality. However, the backbone networks alone are not able to determine whether the data source is of good use and the output is reliable or not. Therefore, an additional network branch, named as evidence network, is employed to evaluate the reliability of the output results. As illustrated in Fig. 1, we introduce three evidence networks for image branch, non-image branch, and feature fusion branch, respectively. The input of the evidence network is the extracted features of corresponding branch and the output indicates the reliability. The evidence network contains a few linear layers, layer normalisation layers, and non-linear ReLU activation functions.

We define the reliability of the current output $p_{i}$ of branch $i$ as evidence score $s_{i}\in[0,1]$. The more reliability a decision has, the larger its $s_{i}$ is. This means the current branch is making a confident and right decision. On the contrary, when the branch makes a wrong decision, the $s_{i}$ needs to be a small value, indicating the output is of high uncertainty and not reliable. Given a well-trained classifier of branch $i$, we pass all the samples $x^{j}|_{j=1}^{N}$, where $N$ is the number of samples, through the network for classification and record if it makes a mistake or not. When classifier of the $i^{th}$ branch returns a wrong decision for a sample $x^{j}$, this means the decision $p_{i}^{j}$ are not reliable, and the $s_{i}^{j}$ for this sample should be $0$. On the contrary, when the classifier of branch $i$ returns an accurate result for the sample, $s_{i}^{j}$ should be 1, denoting a trust-worthy decision. We take these binary scores as the evidence network’s learning target $s_{i}^{j*}$. The evidence networks are trained to generate the evidence scores $\hat{\mathbf{s}}$ that are close to the true evidence score $\mathbf{s^{*}}$.

Dempster-Shafer theory (DST) [3, 4], also known as evidence theory or the theory of belief functions, is a framework for reasoning with uncertainty. DST combines the evidence from various sources with a degree of belief. Studies have shown DST offers superior fusion results in many field [5, 6, 10, 11, 18]. The Dempster’s combination rule is based on DST and is applied to reasonably fuse the estimated evidence scores $\hat{\mathbf{s}}$ with the probability $\mathbf{p}$ for all the branches.

Specifically, let $\Omega=\{\omega_{1},\omega_{2},...,\omega_{K}\}$ be a list of $K$ hypotheses, and the evidence can be represented by a basic probability assignment, which is a mapping $m(\cdot):2^{\Omega}\rightarrow[0,1]$. We have $\sum_{A\subseteq\Omega}{m(A)=1}$, and $m(\cdot)$ is called a mass function.

The evidence of two mass function $m_{1}(X_{1})$ and $m_{2}(X_{2})$ can be fused by the Dempster’s combination rule:

where $M=\sum_{{X_{1}}\cap{X_{2}}\neq\emptyset}{m_{1}({X_{1}})}{m_{2}({X_{2}})}$. Eq. (1) is extendable to multiple evidence:

where $M=\sum_{{X_{1}}\cap{X_{2}}\cap...{X_{T}}\neq\emptyset}\prod_{i=1}^{T}m_{i}({X_{i}})$.

In our task, we have two hypotheses, positive and negative, so we define $\Omega=\{T,F\}$. $T$ represents the positive outcome, and $F$ represents the negative outcome. Then, we can construct the set $\{\varnothing,T,F,U\}$, where $U=\{T,F\}$ represents the uncertain prediction.

The mass function for each branch $i$ can be defined as the evidence score calibrated output. We ignore the sample index $j$ here for simplicity:

We take two pieces of evidence as an example, the combination of two branches can be formulated as:

where $M=m_{f}(T)+m_{f}(F)+m_{f}(U)$ is the normalisation factor. The combination rule can be extended to multiple branches according to equation (2).

We first train the backbone classification networks with the cross entropy loss. Once the backbone network is fully-trained, we fixed the weights and generate the evidence score labels as described in section 2.2. For each branch, we train an evidence network with a mean absolute error (MAE) loss $L_{evid}=\frac{1}{N}\sum_{j=1}^{N}|\hat{\mathbf{s}}^{j}-\mathbf{s}^{j*}|$. The true evidence scores are imbalanced in the training samples, where the majorities are 1, and the minorities are 0. Therefore, similar to [27], we duplicated the minority cases during the training to make the dataset balanced.

In this study, we apply the proposed methods to TKR prediction and follow the settings in [27] to conduct the experiments. The data was obtained from Osteoarthritis Initiative (OAI) [23] dataset. 3D Double Echo Steady-state (DESS) MRI images are used as the imaging data, and 27 non-imaging variables are used for the non-image predictions, as in [27].

The DESS MRIs were cropped in the center region to have size of $320\times 320\times 120$. The patients who underwent TKR in 5 years are labelled as positive cases; otherwise, the samples are non-TKR control group. The class imbalance problem is significant in the dataset as the majority of the patients did not get the knee replaced. To avoid the bias introduced by the class imbalance, which can affect the evaluation of the model [31], we randomly sample the non-TKR cases to have a similar number of TKR cases in our dataset. We repeated the random sampling process to create three datasets to evaluate the proposed method. In total, there are 1,717 samples in each of our dataset, and we use $80\%$ for training, $10\%$ for validation, $10\%$ for testing.

The experiments were conducted on an NVIDIA Tesla V100 GPU and we implemented the networks with PyTorch library. The evidence networks in our study contain three interleaved linear, normalisation and ReLU layers, and the size of all the linear layers is 32. We train the evident networks for 100 epochs and saved the best model on the validation dataset. We use the Adam optimiser with a learning rate of 0.0001 and a weight decay to prevent over-fitting.

There are three settings in our experiments, which are image-only, clinical-only, and multi-modal on the three datasets. The backbone networks are mainly based on [27] and [9, 22]. For image-only setting, we employed DenseNet-121 as our backbone using the code and configurations from [27]. For the clinical-only experiments, we trained three models. (1) logistic regression (LR): we first used a simple LR method to perform the classification [27]. (2) fully-connected network (FC): since LR is a simple approach and only contain one linear layer, we improved the non-imaging branch with a fully-connected network (FC). In FC, the categorical variables go through embedding layers, and the continuous variables are fed into linear layers. After layer normalisation and ReLU activation function, the projected features are summed up and fed into another linear layer. (3) Transformer (Trans): to further improve the non-imaging branch, we use the transformer architecture [9, 22]. Then, we implemented multi-modality models and our proposed method. The image model is combined with each of the three clinical models, respectively. The LR in multi-modality section of Table 1 combined imaging data with non-imaging data with logistic regression (LR) for multi-modal prediction [27]. For FC and Trans in the multi-modality part of Table 1, the corresponding imaging and non-imaging branches’ features are concatenated and fed into a third fusion branch, which contains a linear layer for the decision making. Finally, we implemented the proposed fusion method on all baselines, labelled as DST in Table 1. Specifically, the features from the backbone networks are used as the evidence network input. The input of the evidence network for imaging branch draws from the DenseNet-121 and the feature size is 1018. The input of the evidence network for LR, FC, Trans and fusion branch has feature size of 34, 32, 64, and 4, respectively. We use accuracy, specificity, sensitivity, precision, F1 score and area under the ROC curve (AOC) in the scikit-learn package to evaluate the algorithms’ performance.

The comparison of various methods over all datasets is shown in Table 1. The image-only predictions have good accuracy, which are around 82$\%$, 81$\%$ and 83$\%$ on datasets 1, 2,and 3, respectively. In comparison, the clinical-only predictions are normally inferior, which are less than 80$\%$, except that FC and Trans on dataset 3 are close to 85$\%$.

The multi-modal networks without DST combine the information from both sources and consistently obtain better results than the single-model predictions on all datasets. The improvement in accuracy of the multi-model networks are around 2-4$\%$ higher than the best single model. This means that combination of information from multiple sources improves the prediction performance.

However, the multi-modal networks without DST still conduct a simple combination treating all the data sources and branches equally, and cannot make good prediction when multiple branches output conflicting predictions and have significantly different reliability. The proposed evidence-aware approach, labelled as DST, considers the reliability of the source data and the network prediction, greatly reducing the uncertainties of the prediction. As illustrated in Table 1, the methods with DST almost consistently outperform the counterparts in terms of all metrics on all datasets. This shows the effectiveness of the proposed evidence network and fusion strategy.

We further investigated the effectiveness of the proposed methods. First, we compared the proposed fusion strategy with the average fusion method [18], which simply averages the classification probabilities of all the branches. The comparison results of the proposed method (DST) and the average fusion (Avg.) is shown in Table 2. The proposed methods have around 1$\%$ higher accuracy compared to the average fusion method on dataset 1. The improvement is also seen on datasets 2 and 3. Therefore, it is important to consider the reliability of each branch’s output for final results.

Second, we examine the evidence score estimation to confirm that the evidence network has indeed learned to perform its task. Specifically, we check the evidence scores of correctly classified samples and misclassified samples, respectively. The correctly classified samples should have larger evidence scores, and the evidence scores of the rest samples should be near $0$. We use histogram to visualise it. Fig.S1-S4 in the supplementary shows an example histogram of the fuse branch of LR backbone on dataset 1. For most of the correct results, the scores falls near $1$; and for the wrong classifications, the scores are much smaller and close to $0$. This means the evidence network can learn the reliability of the backbone networks.