Contextual Semantic Embeddings for Ontology Subsumption Prediction

OWL ontologies DBLP:journals/ws/GrauHMPPS08 are based on the $\mathcal{SROIQ}$ description logic (DL) baader2017introduction . An ontology comprises a TBox and an ABox. The TBox defines atomic concepts and roles, and uses DL constructors such as conjunction (e.g., $C\sqcap D)$, disjunction (e.g., $C\sqcup D$) and existential restriction (e.g., $\exists r.C$) to compose complex concepts, where $C$ and $D$ denote concepts and $r$ denotes a role. The TBox also includes General Concept Inclusion (GCI) axioms (e.g., $C\sqsubseteq D$) and Role Inclusion axioms (e.g., $r\sqsubseteq s$), where $s$ denotes another role. The ABox is a set of assertions such as concept assertions (e.g., $C(a)$), role assertions (e.g., $r(a,b)$) and individual equality assertions (e.g., $a\equiv b$), where $a$ and $b$ denote two individuals.

In OWL, the aforementioned concept, role and individual are modeled as class, object property and instance, respectively. An atomic concept corresponds to a named class, while the class of a concept by DL constructors is sometimes called complex class. Class, object property and instance can all be referred to as entity. Each entity is uniquely represented by an Internationalized Resource Identifier (IRI). As shown in Figure 1, these IRIs may be lexically “meaningful” (e.g., vc:ProcessedLegumes of HeLiS) or consist of internal IDs that do not carry useful lexical information (e.g., obo:FOODON_00002809 of FoodOn); in either case the intended meaning may also be indicated via annotation axioms which can be represented by RDF triples using annotation properties as the predicates; e.g., the class obo:FOODON_00002809 (edamame) is annotated using rdfs:label — a built-in annotation property by RDFS to specify a name string, and using obo:IAO-0000115 (definition) — a bespoke annotation property to specify a natural language “definition”. To facilitate understanding we will from now on refer to an entity by one of its readable labels.

A GCI axiom $C\sqsubseteq D$ corresponds to a subsumption relation between the class $C$ and the class $D$. When an ontology is serialised as RDF¹¹1https://www.w3.org/TR/rdf11-concepts/ triples, each of which is a tuple composed of a subject, a predicate and an object, $C\sqsubseteq D$ is represented as ($C$, rdfs:subClassOf, $D$) where rdfs:subClassOf is a built-in property in RDFS²²2https://www.w3.org/TR/rdf-schema/. For shortness, we abbreviate the subsumption triple as $(C,D)$. Real-world ontologies often use a property existential restriction as the subsumer (superclass) of a GCI axiom to (partially) specify the semantics of classes. For example, in FoodOn soybean milk is subsumed by an existential restriction on the property obo:RO_0001000 (derives from) to the class obo:FOODON_03411452 (soybean plant); this specifies that every instance of soybean milk is related to some instance of soybean plant via the property derives from. Such property existential restrictions are widely used in many real-world ontologies such as FoodOn and SNOMED CT³³3https://www.snomed.org/ due to their relatively high expressivity in representing class semantics and polynomial time complexity in reasoning baader2005pushing . Note that an OWL ontology can support entailment reasoning as implemented by, e.g., HermiT glimm2014hermit , through which hidden subsumption axioms can be entailed, e.g., via the transitivity of the subsumption relation and the logical definition of complex classes. Entailing class subsumptions using the transitivity of the subsumption relation is sometimes also called inheritance reasoning. For example, in FoodOn, we can entail that gluten soya bread is a subclass of bean food product via the intermediate class soybean food product and the transitivity of the subsumption relationship.

A class hierarchy, i.e., the pre-order on named classes induced by the subsumption relation (rdfs:subClassOf), can usually be extracted from an OWL ontology. It can simply be the set of all the declared subsumptions between named classes, or computed with entailment reasoning, through which the entailed subsumptions between named classes (except for these by inheritance reasoning) are added. Sometimes the class hierarchy can be extended with complex classes and logic expressions such as the property existential restriction.

BERT is a contextual word embedding model based on deep bidirectional transformer encoders devlin2019bert . It is often pre-trained with a large general purpose corpus, and then can be applied following a typical fine-tuning paradigm, i.e., it is attached by a classification layer and its parameters are further adjusted towards a specific downstream task with labeled samples, as shown in Figure 2. In pre-training, each input is a sequence composed of a [CLS] special token, two sentences (denoted as A and B), each of which is tokenized e.g., into sub-words by WordPiece schuster2012japanese , and a [SEP] special token that separates A and B. The embedding of each token is initialized by its one-hot encoding, segment encoding (in A or B) and position encoding. The parameters of the stacked encoders are learned with two self-supervision tasks: Next Sentence Prediction (NSP) which predicts whether B is following A using the embedding of [CLS], and Masked Language Modeling (LM) which predicts randomly masked tokens in both A and B.

In fine-tuning, the input can be either one sentence with the [CLS] special token or two sentences as in pre-training, depending on the downstream tasks. Figure 2 presents the fine-tuning for the classification of two sentences, where the [CLS] token embedding is fed to a (binary) classification layer for a probabilistic output. This architecture can support tasks such as predicting whether one sentence (the premise) entails another sentence (the hypothesis) and predicting whether two sentences (phrases) are synonyms, and is adopted by BERTSubs. Since labeled samples are given in fine-tuning, the model parameters are updated by minimizing a task-specific loss over the given labels.

It is worth mentioning that the pre-trained BERT can also be applied without or with only a little task-specific fine-tuning, by transforming the downstream task into a task adopted in pre-training (e.g., predicting the masked token in a natural language sentence). Such a paradigm is sometimes known as prompt learning, and its key advantage is relying on no or only a small number of task-specific labeled samples liu2021pre ; gao2021making . In our ontology subsumption prediction, an ontology already has quite a large number of given subsumptions that can be used for fine-tuning and it has no high requirement on the computation time as an offline application. Thus we prefer the BERT fine-tuning paradigm instead of prompting learning.

As a contextual word embedding model, BERT and its variants (such as BioBERT which is pre-trained on biomedical corpora lee2020biobert ) have been widely investigated, with better performance achieved than traditional word embedding and sequence feature learning models such as Recurrent Neural Networks in many Natural Language Processing tasks. In contrast to non-contextual word embedding models such as Word2Vec mikolov2013efficient , which assign each token only one embedding, BERT can distinguish different occurrences of the same token taking their contexts (attentions of surrounding tokens) into consideration. Considering the sentence “the bank robber was seen on the river bank”, BERT computes different embeddings for the two occurrences of “bank” which have different meanings.

This study aims to predict two kinds of class subsumptions by one general method:

• •

  Intra-ontology subsumptions. Given an OWL ontology $\mathcal{O}$, it is expected to discover plausible class subsumptions so as to complete its class hierarchy. For a named class $C$ in $\mathcal{O}$, the method predicts a score $s$ in $[0,1]$ for each class $D$ in $\mathcal{O}$ to indicate the likelihood that $D$ subsumes $C$; i.e., the method ranks the classes in $\mathcal{O}$ according to their likelihood of being a superclass of $C$. Note $D$ can be either a named class or a property existential restriction. For example, the named class soybean food product and the existential restriction (derives from some seed (anatomical part)) are expected to be predicted with high scores and ranked in high positions, as subsumers of soybean milk (see Figure 1). In ranking, we exclude the class $C$ itself and its superclasses that can already be logically entailed, such as plant food product w.r.t. soybean milk. For simplicity, in this paper we call the subsumption between two named classes as named subsumption and the subsumption between a named class and a property existential restriction as existential subsumption.

• •

  Inter-ontology subsumptions. Given two ontologies $\mathcal{O}$ and $\mathcal{O}^{\prime}$, and a named class $C$ in $\mathcal{O}$, the method predicts a similar score $s$ as in the first problem, for each named class $D$ in $\mathcal{O}^{\prime}$ to indicate the likelihood that $D$ subsumes $C$. For example, the named class Soy Product in HeLiS is expected to be the subsumer of soybean milk in FoodOn. The class pair ($C$, $D$) is regarded as a subsumption mapping between $\mathcal{O}$ and $\mathcal{O}^{\prime}$. Such mappings are usually for knowledge integration and do not consider complex classes in $\mathcal{O}^{\prime}$.

Figure 3 shows the overall framework of BERTSubs. Following a typical BERT fine-tuning paradigm, it mainly includes three parts:

• •

  Corpus Construction. Given an ontology $\mathcal{O}$ (or two ontologies $\mathcal{O}$ and $\mathcal{O}^{\prime}$), BERTSubs extracts a corpus, i.e., a set of sentence pairs $\mathcal{S}$ by three steps. First, it extracts positive class subsumptions from the class hierarchy of $\mathcal{O}$ (or from the class hierarchies of both $\mathcal{O}$ and $\mathcal{O}^{\prime}$). The current BERTSubs implementation uses the class hierarchy extracted from declared subsumptions without entailment reasoning. Second, for each positive subsumption $(c_{1},c_{2})$, it generates a negative class subsumption by randomly replacing $c_{2}$ by a named class or existential restriction in the class hierarchy, with all the declared or entailed subsumers of $c_{1}$ excluded for preventing from generating false negative samples. Note the entailed subsumers of $c_{1}$ are very efficiently computed by inheritance reasoning over the class hierarchy. Third, it transforms each subsumption into one or multiple sentence pairs. It currently has three templates to transform a named class or an existential restriction into a sentence: Isolated Class, Path Context, and Breadth-first Class Context, all of which will be introduced in the remainder of this section. Note for predicting named subsumptions, we extract named subsumptions for training, while for predicting existential subsumption, we extract existential subsumptions using the class hierarchy extended with property existential restrictions.

• •

  Model Fine-tuning. To construct a classifier, a linear layer with dropout is first attached to a pre-trained BERT. It takes as input the embedding of [cls] token from the BERT’s last-layer outputs, and transforms the embedding into a 2-dimensional vector. The linear layer is then further attached with a softmax layer to output the score $s$ in $[0,1]$ which indicates the truth degree of the subsumption relationship. The parameters of the pre-trained BERT encoders and this new classifier are jointly optimized using the Adam algorithm by minimizing the cross-entropy loss over the samples (labeled sentence pairs). Note that the original input sentences, which are sequences of words, are parsed into sub-word (token) sequences by an inherent WordPiece tokenizer wu2016google of the pre-trained BERT, before they are input to the BERT encoders. For example, the original word “soybean” is parsed into two tokens “soy” and “##bean”.

• •

  Prediction. In prediction, the fine-tuned model predicts a score for each candidate subsumption to test (i.e., a pair of classes) which will be transformed into one or multiple sentence pairs using the same template as used in generating the fine-tuning samples. When one subsumption is transformed into multiple sentence pairs, they are predicted independently, and the scores are averaged as the score of the subsumption.

In this class-to-sentence transformation template, we use the name information of a named class or an existential restriction alone without considering its surrounding classes. One subsumption $(c_{1},c_{2})$ will lead to sentences of $\left\{(s_{1},s_{2})|s_{1}\in L(c_{1}),s_{2}\in L(c_{2})\right\}$, where $L(\cdot)$ denotes a class’s labels. For a named class, BERTSubs uses the label defined by rdfs:label by default, and can also use the labels defined by some other annotation properties for synonyms, such as obo:hasExactSynonym⁴⁴4obo: is short for the prefix of http://www.geneontology.org/formats/oboInOwl#. and obo:hasSynonym in FoodOn. Using multiple labels by different annotation properties lead to many more sentence pairs, and this setting is compared with using one English label by rdfs:label in our evaluation (see Section 4.3.2).

For an existential restriction in form of $\exists r.C$ where $C$ is a named class, BERTSubs generates its natural language descriptions as its labels using the template of “something $L(r)$ _ $L(C)$”. $L(r)$ and $L(C)$ are the labels of the object property (relation) $r$ and the class $C$, respectively, while _ is an optional preposition that is added when the property label itself does not end with a preposition, so as to generating natural language text (e.g., “of” is added after the FoodOn property obo:RO_0000086 (“has quality”)). As an example, the existential restriction (obo:RO_0001000 some obo:FOODON_03411452) in Figure 1 will get the label of “something derives from soybean plant”. BERTSubs can also be easily extended to address those existential restrictions in forms of $\exists r.(C_{1}\sqcap C_{2},...)$ and $\exists r.(C_{1}\sqcup C_{2},...)$, where the restriction class is the conjunction or disjunction of multiple named classes. Their templates are “something $L(r)$ _ $L(C_{1})$ and $L(C_{2})$ …” and “something $L(r)$ _ $L(C_{1})$ or $L(C_{2})$ …”, respectively. For more complicated existential restrictions where the restriction class is a complex class with negation or some other existential restriction (e.g., $\exists r_{1}.(C_{1}\sqcap\exists r_{2}.C_{2})$), the label generation could be quite complicated and we leave it for future extension, where some previous works on generating natural language descriptions from OWL ontology concepts (e.g., stevens2011automating ; kaljurand2007attempto ) could be referred to.

For a class subsumption $(c_{1},c_{2})$, the PC template first extracts paths starting from $c_{1}$ and $c_{2}$, respectively, from the class hierarchies of their corresponding ontologies. The paths of $c_{1}$ are generated by depth-first traversal which starts from itself to one of its child classes, continues to its descendants, and stops when it has arrived at a leaf class or the depth has reached the maximum value $d$. One path can be denoted as $p(c_{1})=[c_{1},c_{1,1},...,c_{1,i-1},c_{1,i},...,c_{1,d}]$ where $(c_{1},c_{1,1})$, $\cdots$, $(c_{1,i-1},c_{1,i})$ are class subsumptions. The paths of $c_{2}$ are also generated by depth-first traversal which starts from itself to one of its parent classes, continues to its ancestors, and stops when it arrives at the top class owl:Thing or the depth exceeds the maximum depth of $d$. When $c_{2}$ is a property existential restriction, the depth-first traversal is not applied and the path is $[c_{2}]$. When traversing from one class to the next, there could be multiple subclasses or superclasses, and we randomly select at most $w$ of them to reduce the size of paths. $w$ and $d$ are two hyper-parameters. We extract the path down for $c_{1}$ and up for $c_{2}$, because the downside context of $c_{1}$ has its more fine-grained information, the upside context of $c_{2}$ has its more general information, and they are more useful for determining whether $c_{1}$ is a subclass of $c_{2}$.

With the paths, sentences are then generated by replacing the classes by their labels. One path of $c_{1}$, briefly denoted as $p(c_{1})=[c_{1},c_{1,1}...,c_{1,d}]$, is transformed into sentences of

where the special token [SEP] separates the labels of every two classes. The path of $c_{2}$ is transformed into sentences in the same way, denoted as $S_{p}(c_{2})$. For a property existential restriction, its label is generated using the same template as in IC. Note we have considered introducing a new special token [SUB] instead of using the BERT built-in special token [SEP], for representing the semantics of rdfs:subClassOf, but it does not lead to better performance in our evaluation. One potential reason is that it may need a much larger training corpus to let the model learn the new special token’s representation. With $S_{p}(c_{1})$ and $S_{p}(c_{2})$, the final set of the sentence pairs of the subsumption $(c_{1},c_{2})$ can be represented as $\left\{(s_{1},s_{2})|s_{1}\in S_{p}(c_{1}),s_{2}\in S_{p}(c_{2})\right\}$. Take the candidate named subsumption (obo:FOODON_03302389, obo:FOODON_00002266) in Figure 1 as an example, one potential sentence pair with $d$ = $2$ is (“soybean beverage [SEP] soybean milk”, “soybean food product [SEP] bean food product [SEP] plant food product”). Since one subsumption could have quite a few sentence pairs due to different paths and different class labels , we set $w$ and $d$ to small numbers (e.g., $4$ and $2$ respectively).

Given a class subsumption ($c_{1}$, $c_{2}$), the BC template first extracts context class sequences for each class via breadth-first traversal starting from this class over the class hierarchy of its corresponding ontology. For $c_{1}$, it first extracts subclasses of $c_{1}$ for context subsumptions of depth $1$, then extracts subclasses of each subclass achieved at depth $1$ for context subsumptions of depth $2$, and so on. It stops when the depth has arrived at $d$ or all the subclasses of this depth have been leaf classes. $c_{1}$ and the context subsumptions achieved in order are concatenated to generate the sequence, denoted as $b(c_{1})=[c_{1},c_{1,1},c_{1},\cdots$,$c_{1,i},c_{1,j},\cdots$], where ($c_{1,1}$, $c_{1}$), …, ($c_{1,i}$, $c_{1,j}$) are class subsumptions. For $c_{2}$, if it is a named class, its sequence is generated in the same way except that the BC template traverses up for context subsumptions until the depth arrives at $d$ or the superclasses have only the top class owl:Thing. Its sequence is denoted as $b(c_{2})=[c_{2},c_{2},c_{2,1},\cdots$,$c_{2,i},c_{2,j},\cdots$], where ($c_{2}$, $c_{2,1}$), …, ($c_{2,i}$, $c_{2,j}$) are class subsumptions. If $c_{2}$ is an existential restriction, its sequence is simply $[c_{2}]$ without surrounding classes. For both $c_{1}$ and $c_{2}$, to limit the sequence length, the BC template randomly extracts at most $w$ subclasses at each depth. It also does multiple times of traversal, leading to multiple sentences for $c_{1}$ and $c_{2}$.

For each sequence the classes are replaced by their labels to generate sentences. The sentences of a sequence of $c_{1}$ are computed as

As in the PC template, we originally considered using a new special token [SUB] to separate two class labels of a subsumption (e.g., $s_{1,i}$ and $s_{1,j}$) but the performance is not as good as simply using the BERT built-in token [SEP] which is to separate two sentences. Consider obo:FOODON_00001015 in Figure 1 as an example of $c_{1}$, one of its sentences with depth of $1$ can be “plant food product [SEP] bean food product [SEP] plant food product [SEP] soybean beverage [SEP] plant food product”. The sentences of $c_{2}$, i.e., $S_{b}(c_{2})$, can be calculated in the same way as $c_{1}$. The final sentence pairs of $(c_{1},c_{2})$ are calculated as $\left\{(s_{1},s_{2})|s_{1}\in S_{b}(c_{1}),s_{2}\in S_{b}(c_{2})\right\}$. In evaluation, we also only consider small numbers for $w$ and $d$ to limit the sentence length. In comparison with PC, the BC template contains more complete contexts of $c_{1}$ and $c_{2}$, but also leads to higher redundancy in the sentences.

The overall results of intra-ontology named subsumption prediction are shown in Table 3 where the best results are bolded. On the one hand, we can find considering the textual information is beneficial. On both FoodOn and GO, Word2Vec itself already performs a bit better than the best of the geometric KG embedding methods; Text-aware TransE and $\text{Text-aware}^{+}\text{ TransE}$ both significantly outperform the original TransE; the Word2Vec-based OWL ontology embedding methods, especially $\text{OWL2Vec}^{*}$, all perform quite well. On the other hand, BERTSubs (IC), which only considers the labels of two isolated classes but adopts contextual word embedding with BERT, further outperforms the OWL ontology embedding methods as well as the other baselines; for example, in comparison with $\text{OWL2Vec}^{*}$, BERTSubs improves MRR from $0.462$ to $0.586$ on GO. Regarding the templates for utilizing the class contexts, the PC template is quite effective in view of the facts that BERTSubs (PC) outperforms BERTSubs (IC) on MRR and H@$1$ on both FoodOn and GO, while the BC template plays a slight negative impact w.r.t. most metrics.

The overall results on intra-ontology restriction subsumption prediction are shown in Table 4. As intra-ontology named subsumption prediction, the textual information is still quite effective, with much better performance achieved by $\text{Text-aware}^{+}\text{ TransE}$ than the original TransE. The ontology embedding methods have better performance than the geometric KG embedding methods on GO, but on FoodOn, methods of the latter, including TransR, DistMult and HAKE, in contrast, have better performance, which could be due to that the associated owl:onProperty triples and owl:someValuesFrom triples can contribute important information in distinguishing the correct and wrong restriction subsumers. Meanwhile, BERTSubs (IC) dramatically outperforms all the baselines (e.g., the MRR value is improved from $0.757$ to $0.781$ on FoodOn and from $0.700$ to $0.898$ on GO, in comparison with the best baseline), and this improvement is higher than that in intra-ontology named subsumption prediction. Both the PC template and the BC template are quite effective w.r.t. the task on both ontologies, in comparison with using isolated classes, and the PC template has slightly better performance than the BC template. For example, H@$1$ on FoodOn is improved from $0.670$ to $0.723$ by the PC template, and to $0.706$ by the BC template.

The overall results on inter-ontology named subsumption prediction are shown in Table 5. Note the geometric KG embedding baselines cannot not applied in this task since the subsumption is between two independent ontologies. The observations on BERTSubs are very close to those of intra-ontology named subsumption prediction: BERTSubs (IC) can already achieve better results than all the baselines (except for H@$1$ on HeLiS-FoodOn in comparison with $\text{OWL2Vec}^{*}$); the PC template is also effective, leading to some performance improvement especially towards the metrics of MRR and H@$1$ on both NCIT-DOID and HeLiS-FoodOn, while the BC template is less effective.

It is worth noting BERTSubs (IC) has better performance than BERTSubs (BC) in all the three above scenarios. One potential reason is that the BC template attempts to encode more complete contexts of the two classes. This leads to more complex input sequences that the model needs to learn from, with higher information redundancy and potential noise. Due to this observation, we do not consider generating more complicated input sequences by using the IC and the BC templates at the same time.

In the past decade, quite a few methods have been developed for embedding and completing knowledge graphs (KGs) that are composed of relational facts wang2017knowledge . These methods such as TransE bordes2013translating and TransR lin2015learning can be applied to predict subsumption relationships in OWL ontologies by transforming the ontologies into RDF graphs (triple sets). Besides the W3C OWL Mapping to RDF Graphs we used in evaluation, some other transformations, e.g., the projection rules used on ontology visualization soylu2018optiquevqs , could be considered, but these transformations often miss some semantics in OWL, especially the logical expressions. Meanwhile, some geometric KG embedding methods, such as HAKE zhang2020learning , Poincaré Embedding nickel2017poincare and Box Lattice Embeddings lees2020embedding , have been proposed to consider the hierarchical structure. But they do not consider other semantics beyond the graph structure in OWL ontologies, making them fail to utilize these semantics especially the literals and the logical expressions. Some literal-aware KG embedding methods, such as Text-aware TransE mousselly2018multimodal evaluated in our study, have also been proposed to jointly embed the graph and the literals that are associated by some data properties gesese2019survey , but they usually aim at relational facts instead of OWL ontologies with little attention to e.g., the hierarchical classes.

Recently, the geometric KG embedding methods have been further extended to embed logical relationships in OWL ontologies with more complex modeling in the vector space (e.g., classes are represented by areas instead of points). Typical methods include EL Embedding kulmanov2019embeddings , Box EL Embedding xiong2022box and Logic Tensor Networks ebrahimi2021towards for Description Logic $\mathcal{EL}^{++}$, and Quantum Embedding garg2019quantum for Description Logic $\mathcal{ALC}$. As those classic geometric KG embeddings, these methods currently only consider (a part of) the logical relationships in an OWL ontology. On the other hand, some recent OWL ontology embedding methods such as OWL2Vec* chen2021owl2vec , OPA2Vec smaili2019opa2vec and Onto2Vec smaili2018onto2vec , which first transform the ontology contents into entity and/or word sequences with entity and word correlation kept, and then learn word embedding models to get word and entity vectors, have achieved quite good performance on ontology relevant prediction tasks such as class subsumption prediction, class clustering and gene–disease association prediction. Their weakness mainly lies in the adopted non-contextual word embedding model which has been proven to perform worse than the more recent Transformer-based contextual word embedding models such as BERT in many natural language understanding and sequence learning tasks.

Transformer-based PLMs such as BERT and T5 have achieved great success in many taks of natural language inference such as text classification, question answering and machine reading comprehension devlin2019bert ; lee2020biobert ; raffel2020exploring . With the recent development of prompt techniques, these PLMs can be further fine-tuned and applied to different kinds of tasks, where input with different contexts and formats are transformed, embedded and exploited liu2021pre . There have been some works that apply PLMs to address KG completion and some other prediction tasks of KGs. For example, KG-BERT yao2019kg fine-tunes a BERT to predict a relational fact (triple) by encoding its text (i.e., the labels of its entities and relation); similarly, COMET bosselut2019comet uses a pre-trained Transformer Language Model to predict the fact of ConceptNet by its text; Wang et al. wang2021structure follow the text encoding paradigm of KG-BERT by augmenting the text embedding with the graph embedding so as to take the entity’s context into consideration. There are also some studies that utilize the symbolic knowledge in a KG as the reference to probe the sub-symbolic knowledge in PLMs. One typical work is LAMA petroni2019language which uses the pre-trained BERT to predict the masked tokens of natural language sentences, each of which is generated from one KG fact by a simple template. These works are quite different from BERTSubs which aims at OWL ontologies that have more complex knowledge formats than relational facts and require different templates. BERTSubs can also support both inter-ontology and intra-ontology subsumptions, as well as the existential restrictions as the subsumers. All these tasks are quite different from the related works presented above.

BERT has been applied to ontology subsumption prediction and ontology matching, but the research of this direction is still in an early stage. Liu et al. liu2020concept predict subsumers in the SNOMED ontology for new and isolated classes to insert, where templates are used to generate sentences for the class context as BERT input. In contrast, BERTSubs aims at the subsumption between two existing classes, both of which have contexts, with different templates and fine-tuning mechanisms, and more importantly considers the property existential restriction as the subsumer. Our previous work BERTMap he2022bertmap predicts the equivalence mapping (i.e., equivalent classes) between two OWL ontologies, using the labels of the two classes as the IC template introduced in this paper. BERTSubs is a follow-up work of BERTMap with new research on different ontology curation tasks, more complicated templates for utilizing the class context, and the support of existential restrictions.