Integration of Domain Knowledge using Medical Knowledge Graph Deep Learning
for Cancer Phenotyping

Our dataset consists of cancer pathology reports obtained from the Louisiana Tumor Registry (LTR), Kentucky Cancer Registry (KCR), Utah Cancer Registry (UCR), and New Jersey State Cancer Registry (NJSCR) of the SEER Program²²2NJSCR is no longer in the SEER Program, but is included in the current data release.. The study is executed in accordance to the institutional review board protocol DOE000152. Each pathology report in our dataset is associated with a unique tumor ID; the same tumor ID may be associated with one or more pathology reports. For each tumor ID, Certified Tumor Registrars (CTRs) manually assigned ground truth labels for key data elements – cancer site, histology, grade, etc. – based off all data available for that tumor ID according to the SEER program coding and staging manual³³3https://seer.cancer.gov/tools/codingmanuals/index.html. In this paper, site (70 classes), subsite (324 classes), laterality (7 classes), behavior (4 classes), histology (572 classes), and grade (9 classes) are defined as the target classification labels for each cancer pathology report as these are fundamental information extraction tasks for cancer phenotyping. Documents generated within 10 days between the date of pathological diagnosis and either path specimen collection date or the surgery date are identified as relevant to the specific case ID. We split the dataset based on the pathology report specimen collection date into train, validation and test sets. Specifically, reports collected prior to 2017 are used for train and validation with 80:20 ratio, while the rest of the reports are used for testing. The resulting train, validation, and test datasets consist of 598,228, 149,315, and 131,322 pathology reports respectively, yielding a total of 878,865 documents for our experiment.

For each pathology report, we extract the text content and apply standard preprocessing steps to tokenize it. First, we remove XML tags as well as identification tags. In addition, we set all alphabetical characters to lowercase. Then, we create a vocabulary list including all words that have pathology report document frequency of at least five. Since pathology reports have different lengths, we accommodated this by specifying a fixed length of $3,000$ words for all reports.

The UMLS is a repository of files and software that integrate various biomedical vocabularies and standards developed by the National Library of Medicine (NLM) [5]. The motivation behind the development of the UMLS is to help computer programs to understand the semantics of biomedicine and health [6]. UMLS provides three knowledge sources. First, the Metathesaurus that contains numerous vocabulary terms and codes. Second, the Semantic Network provides semantic types and relations. Third, it contains the Specialist Lexicon for natural language processing [5]. In this paper, we use three vocabulary sources to build our knowledge graph: i) Systematized Nomenclature of Medicine-Clinical Terms (SNOMED CT US Edition), which provides major general terminology for the electronic health record (EHR) and has hierarchies made from specific conceptual meanings and formal logic-based definitions [7]. ii) National Cancer Institute (NCI) Thesaurus, which is a product of NCI Enterprise Vocabulary Services (EVS) and includes the vocabularies about clinical cancer care, translational, and basic research. It consists of public information on cancer together with administrative activity vocabularies [8]. iii) International Classification of Diseases, Tenth Revision (ICD10) [9], which is introduced and maintained by the World Health Organization (WHO). It is made up of titles and codes for causes of death, inclusion and exclusion terms for cause-of-death titles, an alphabetical index to diseases and nature of injury, external causes of injury, table of drugs and chemicals, and description, guidelines, and coding rules [10].

To build our graph, we download the latest version of the UMLS release (2018AA) from [11] for local installation. It contains 3,665,926 concepts from 154 sources [12]. We create MySQL database load scripts by using the UMLS installation tool. Then, we develop a Python software tool that reads words from the vocabulary list (mentioned in Section II-A), connects them to MySQL database, and fetches the similar UMLS concept names by using the SQL LIKE operator. The software collects all similar UMLS concept names from three vocabularies mentioned earlier (SNOMED CT, NCI, and ICD-10). Concept names are ordered by the UMLS concept unique identifiers (CUIs). We observe that in the UMLS database sometimes one CUI represents multiple similar concept names. In that situation, we use the first concept name.

The enhanced word embeddings used in this paper are pre-trained in two phases. In phase one, the word embeddings are first trained using traditional distributional similarity approaches such as word2vec without incorporating the domain knowledge from UMLS knowledge graph. The outcome is pre-trained word vectors corresponding to the set of vocabularies $V=\{v_{1},v_{2}\dots,v_{n}\}$ from the cancer pathology reports. In the second phase, the retrofitting algorithm, presented in [13], is used to refine the pre-trained word embeddings resulting from the first phase. Specifically, a undirected graph is created from the vocabulary words $V$, the vertices of the graph, and the UMLS graph that finds the relationships between clinical concepts in $V$ and defines the edges $E$ of the graph. Then, the retrofitted word embeddings $\hat{W}$are initialized to be equal to the word embeddings $W$ resulting from phase one. The objective is to update the word vectors to be close to their counterparts in $W$ and close to the adjacent vertices in the graph. The distance that needs to be minimized to ascertain these objectives can be defined as follows:

Where $n$ is the vocabulary list size, $w_{i}$ is the feature representation of word $v_{i}$, $\hat{w}_{i}$ is the retrofitted feature representation of word $v_{i}$, $w_{j}$ is the feature representation of the adjacent words in the graph to the word $v_{i}$, where $(i,j)\in E$. $\hat{W}$ words vectors are updated by taking the first derivative of equation 1 with respect to $\hat{w}_{i}$ as follows:

This method can be used to update word embeddings produced from any model. Therefore, we can use any existing method to pre-train word embeddings and then apply our retrofitting algorithm to enrich the pre-trained word embedding vectors with domain knowledge from a relevant knowledge graph.

The deep learning (DL) model used in this paper is a previously developed MT-CNN model [2]. The model consists of three parallel 1D convolution layers that run across the document matrix, i.e. the word embeddings for a given document. These convolution layers have 300 filters each and use window sizes of 3, 4, and 5 consecutive words to capture features with variable context lengths. The outputs from the convolution layers are concatenated and fed into a max-pooling layer that filters out the most important features from the convolution layers. Finally, the output from the max-pooling layer is fed into six separate soft-max fully connected layers – one for each cancer characteristic. The Baseline MT-CNN model uses pre-trained word embeddings generated by word2vec. The embeddings are trained on our entire corpus of pathology reports. Our enhanced MT-CNN uses enriched word embeddings – these consist of the same word2vec embeddings that are enhanced with external knowledge from medical terminology ontologies using our retrofitting algorithm.

In this work, we tested our proposed method on automated information extraction from cancer pathology reports for six cancer key characteristics – site, subsite, laterality, behavior, histology, and grade. We developed an MT-CNN model to extract all six tasks simultaneously. We built a graph based on the vocabulary list from the cancer pathology reports and the UMLS knowledge graph that combines three vocabulary resources – SNOMED, NCI, and ICD10. The size of the graph is 59,604 nodes and 239,665 edges. For all the experiments, we included 95% confidence intervals for each performance metric derived using bootstrapping [14].

The performance results on all six tasks are summarized in Table I. The results show that MT-CNN model with word embeddings enriched by UMLS knowledge graph consistently outperform the the baseline MT-CNN model that uses pre-trained word embeddings using word2vec. The average performance of the baseline DL model micro- and macro-averaged F1 scores are 0.805 and 0.480, respectively. While the proposed approach achieves micro- and macro-F1 scores of 0.845 and 0.588; that is an improvement of 4.97% and 22.5% for micro- and macro-F1 scores, respectively.

Class imbalance is one of the challenges in real-world datasets, especially in clinical domains. Our cancer pathology report dataset has extreme variability in class prevalence. This is a common challenge in cancer registries because some cancer types are highly prevalent (e.g., breast, lung, prostate) while others are very rare (e.g., esophagus, gum, sinuses). To study in more detail the impact of class prevalence on the classification accuracy, we show in Figure 1 the average accuracy of the two models on the most prevalence and the least prevalence classes across all six tasks. As expected, both the models perform the best on the more prevalence classes. However, our proposed approach outperform the baseline MT-CNN on the low prevalence classes. These results highlight the advantage of incorporating domain knowledge with deep learning to boost the classification performance on the low prevalence classes by enriching the text representation with external knowledge.

Finally, we study the performance of our proposed MT-CNN and the baseline MT-CNN models on predicting cancer phenotype, i.e. correctly classifying multiple cancer characteristics for a given document. When considering all six cancer characteristics in this six, the model with retrofitted embeddings correctly phenotypes 41.66% of the pathology reports, while the baseline model correctly phenotypes only 31.47% of the pathology reports. When considering the site, laterality, behavior, and histology characteristics, the proposed model correctly phenotypes 68.89% of the pathology reports, while the baseline model correctly phenotypes only 62.88% of the pathology reports.