NEEDED: Introducing Hierarchical Transformer to Eye Diseases Diagnosis

Given a OEMR document $O_{\ast}$, in this stage we aims to use a Transformer to embed each sentence $S_{i}$ in $O_{\ast}$ into the sentence representation $a_{i}\in\mathbb{R}^{d}$. To retain information about the text type of each sentence and thus help the Transformer model its semantics, we first add a type-token $t_{0}$ at the beginning of each sentence. Formally, given an sentence $S_{i}$ with token sequence $[t_{0}^{(i)},t_{1}^{(i)},t_{2}^{(i)},\dots,t_{|S_{i}|}^{(i)}]$, the sentence representation $a_{i}$ can be obtained by:

where $Transformer_{w}$ is a $N_{w}$-layers Transformer. For each token $t_{j}^{(i)}$ in the sentence $S_{i}$, its input embedding is obtained by summing its token embedding and positional encoding, and the token embedding is randomly initialized. $[h_{0}^{(i)},h_{1}^{(i)},h_{2}^{(i)},\dots,h_{|S_{i}|}^{(i)}]$ is a list of output embedding of tokens in the sentence $S_{i}$, where $h_{j}^{(i)}$ is a $d$ dimensional vector corresponding to $t_{j}^{(i)}$. The $pooling(\cdot)$ denotes the average-pooling on the output embedding representations to obtain the sentence embedding representation $a_{i}$.

In the above stage, we obtain the representation of each sentence in the OEMR document. Most of the sentences are descriptions of the patient’s symptoms or medical history. It is likely that there are many associations between them, which means that the meaning represented by one sentence may vary depending on its context. Thus, the embedding representation of each sentence should not only depend on its content but also consider its context.

At this stage, we aims to use the Transformer to obtain contextualized sentence representations by learning the dependencies between sentences. Given a OEMR document $O_{\ast}$, we can obtain the sentences representations set $\left\{a_{1},a_{2},...,a_{N}\right\}$ from the above word-level Transformer. The setence-level transformer takes sentences representations set as the input and outputs a matrix $C$, which can be formulated as:

where $Transformer_{s}$ is a one layer Transformer. $C=[c_{1},c_{2},\dots,c_{N}]$ is a set of contextualized sentence representations. With the multi-head self-attention mechanism, Transformer can model the dependencies between sentence representations in the input from multiple perspectives, enabling each sentence representation can collect global information of context. In a sense, the matrix $C$ can be seen as a multi-perspective representation of the input OEMR document.

The hierarchical transformer encoder has the following main advantages: First, Transformer has strengths in learning long-range dependencies [10, 11]. The hierarchical Transformer encoder can effectively and efficiently model the semantic information of OEMR by learning both word-level and sentence-level dependencies. Second, embedding documents into multiple sentence-level representations allows us to implement traceable disease prediction based on these representations in combination with our attention-based predictor proposed below. Third, by adding the type-token at the beginning of sentences, the differences between text types are taken into account, helping the encoder to model sentence semantics more accurately.

The goal of the first experiment is to compare the performance of the NEEDED and baseline models on the ADED task. In this comparison, we used Macro-F1, Micro-F1, Macro-AUC, and Micro-AUC as evaluation metrics, which are widely used in multi-label classification problems [11, 20, 23, 24]. The results are shown in Table 1. From the overall results, we observed that: First, our framework achieves the best performance in overall evaluation metrics, showing it is better at extracting disease-related information from the OEMR document. Second, our model performs better than w/o c by learning both word-level and sentence-level dependencies and preserving text type information. Third, BERT as pre-trained model does not perform as well as other baseline models on most metrics, probably due to the large gap between the text in its pre-training corpus and the OEMR documents.

To explore the effectiveness of each part of our framework, we conduct ablation experiments. We observed that our framework outperforms all its variants, illustrating that each component in our framework is indispensable. In addition to this, there are the following points: First, compared to our model, both w/o c and w/o w show a significant performance drop due to the fact that they both remove the hierarchical Transformer. Meanwhile, w/o c is better than w/o w in most metrics because it retains the attention-based predictor. Second, w/o s outperforms w/o w in all metrics and w/o c is better than w/o w in most metrics, indicating the effectiveness of attention-based predictor which can capture disease-specific information. Third, although the performance of w/o l is degraded compared to our framework, it performs better than w/o w in all metrics. This is because it removes the attention-based predictor while retaining the hierarchical Transformer, indicating the importance of learning the dependency between sentences. Fourth, w/o p is the worst and has a very significant degradation compared to the other models because it removes the preprocessing component, resulting in clutter and sparsity of information in the model input.

To explore the effect of distinguishing descriptions of different eyes, we conduct the following experiment. We first filter part of the asymptomatic descriptions in the OEMR document according to the previous section. Then, instead of distinguishing the remaining descriptions to form two documents, we use all filtered texts as model inputs to obtain the diagnosis results for both eyes. We perform this experiment on our framework and all baseline methods, then compare the results with the case where distinction were made. The results are shown in figure 2. We observe that the performance of all models degrades significantly if the descriptions are not distinguished. The reason is the information clutter caused by descriptions of different eyes mixed up. Besides, we observed that our model is the most affected, probably because it utilize the hierarchical Transformer to model the document’s information at sentence level, which exacerbate the information clutter. In addition to the above, we observed that BERT is the least affected, probably because it has a huge number of model parameters and is pre-trained on a large corpus, making its fitting ability more stable despite the information clutter.

In our framework, we filter out templated asymptomatic descriptions that occur more frequently than a certain threshold before the OEMR document is fed into the downstream model. In this experiment, we explore the impact of this way by setting different filtering thresholds and not filtering at all. Specifically, we set the filtering thresholds $B$ to $0.05$, $0.1$, $0.2$, and $0.3$, as well as no filtering, to observe the changes in the performance of our framework. The results are shown in figure 3, where $0$ represent the ablation of filter. We can find that after the threshold $B$ is bigger than $0.1$, the model’s performance tends to decrease as $B$ increases. The performance is worst in the absence of filtering, which indicates the validity of filtering the templated asymptomatic description.

We also observe that the performance is worse when $B$ is $0.05$ than when $B$ is $0.1$, indicating that the too-low filtering threshold is harmful. This may be due to some valuable asymptomatic descriptions being filtered out.

Our predictor utilizes dot-product attention to extract disease-specific information, which improves the model’s effectiveness and can provide some explainability to the diagnostic results. Specifically, assume that our model diagnoses the disease $l$ based on the given input, we can obtain the attention weight vector $\alpha_{l}$ from the predictor. The $\alpha_{l}^{i}$ denotes the attention weight of the $i$-th sentence’s representation for the disease $l$, which can be viewed as the importance of the $i$-th sentence for diagnosis of disease $l$. Thus, for each disease, we can obtain the important sentences associated with it in the input based on the attention weights. We show a case to demonstrate the effect of providing explainability in this way. The left part of Figure 5 shows a portion of an OEMR document describing the patient’s right eye. Specifically, the patient’s right eye was diagnosed with cataract(CAT), meibomian gland dysfunction(MGD), dry eye syndrome(DES), and pterygium by doctor. Our model yields the same diagnosis. The right part of Figure 5 shows a heatmap in which each column is the distribution of attention weights for one disease. We can observe that: First, our model can focus on different information and make the correct diagnosis when diagnosing various diseases, demonstrating that our model can capture disease-specific information from the document by the attention mechanism. Second, based on the distribution of attention weights, we can track the sentences essential for diagnosing the disease, which provides explainability to our model. Third, we can learn each sentence’s role in diagnosing different diseases based on the distribution of attention weights, which can informing clinical research on the condition.

The number of Transformer layers in the word-level Transformer and the number of self-attention heads in the hierarchical Transformer are two critical parameters in our model. To assess the impact of these two hyperparameters, we investigate the sensitivity of them and show the results in Figure 4. We observe that the performance of our model shows a trend of increasing and then decreasing as the number of Transformer layers varies from one to eight. In addition, the performance of our model shows the same trend as the number of self-focused heads increases. This may be because the fitting ability of the model improves with more parameters introduced, but after introducing too many parameters, the model appears to be over-fitted. Besides, we collected the loss values and accuracy of each epoch during the training process and visualized them to study the convergence of our model. We observe that the model’s accuracy increases rapidly in the first ten training epochs, then tends to converge. The loss curve shows that the model’s training process is stable.