Joint Multimodal Entity-Relation Extraction Based on Edge-enhanced Graph Alignment Network and Word-pair Relation Tagging

Previous studies usually used RNN (Recurrent Neural Networks) to encode text and CNN (Convolution Neural Network) to encode image as a vector. Then, designing an implicit interaction module to model the information between modalities for the MNER task (Zhang et al. 2018; Lu et al. 2018; Moon, Neves, and Carvalho 2018). For example, Zhang et al. (2018) constructed an MNER dataset and proposed a baseline model based on a bi-directional long-term memory network using an attention mechanism to align the image representation with text. However, encoding image as a vector cannot benefit extract different type entities, e.g., Curry (Per) and O’Brien Trophy (Misc).

To address limitation of image alignment methods, the previous models extracted different visual objects using Mask-RNN or Fast-RNN (He et al. 2017) and aligned visual objects with the text representation (Wu et al. 2020b; Yu et al. 2020; Zheng et al. 2021c; Zhang et al. 2021a; Xu et al. 2022). Wu et al. (2020b) proposed an interactive attention structure to align text with visual objects. In addition, Zheng et al. (2021c) designed a gated bilinear attention network (Kim, Jun, and Zhang 2018) with an adversarial strategy to better extract the fine-grained objects from the image. However, the object alignment method does not consider the relations of entity-entity and object-object, and the model will ineffectively match overlapping visual objects with textual entities. For example, the trophy of text may align with the 1_man, since the 1_man contains the region of trophy.

To address the above limitation, the most current researches align the entities with objects in the visual and textual graphs constructed by the latent relationships of objects and words (Zhang et al. 2021a, b; Zheng et al. 2021a). Zhang et al. (2021a) proposed a graph-based multimodal fusion model based on the syntactic dependency text graph and full connection visual graph to exploit the fine-grained semantic alignment different modalities. In the MRE task (Zheng et al. 2021b), Zheng et al. (2021a) designed a graph alignment module to align nodes in textual graph and visual graphs. However, the node alignment methods only consider the nodes in the cross-graph and ignore the edge information. The edge information in the cross-graph can effectively improve the matching precision of nodes and contain some clues about the classifying relation between entities.

The joint multimodal entity-relation extraction task is defined: given an input text $w=\left\{w_{1},w_{2},\cdots,w_{n}\right\}$ with a corresponding image I, the model is required to extract a set of quintuples $y=\left\{\left(e_{1},t_{1},e_{2},t_{2},r\right)_{c}\right\}_{c=1}^{C}$, where ${\left(e_{1},t_{1},e_{2},t_{2},r\right)_{c}}$ represents the c-th quintuple, consisting of two entities $e_{1}$ and $e_{2}$ with the corresponding entity types $t_{1}$ and $t_{2}$, where ${e_{1}}\neq{e_{2}}$. Furthermore, $r$ represents the relation between the entities $e_{1}$ and $e_{2}$. Figure 1 gives an example to better understand the JMERE task, which aims to extract all quintuples, e.g., (Thompson, Per, NBA, Org, Member_of), where Per and Org indicate the entity types of Thompson and NBA, and Member_of denotes their relation type.

Inspired by the grid tagging scheme in aspect-based sentiment triplet extraction (Wu et al. 2020a), we design a word-pair relation tagging to extract all elements of JMERE in one step. By word-pair relation tagging, the JMERE task is converted into extracting the relations $Y$ between each word-pair $(w_{i},w_{j})$ and avoid the error propagation caused by the pipeline framework. These relations can be explained below, and we also give an example in Figure 3 to better understand the word-pair relation tagging.

• •

  N indicates that the word-pair does not have any relation.

• •

  $T$ indicates that the word-pair belongs to an entity type, which is contained 4 defined types in the previous work (Zheng et al. 2021c)

• •

  $R$ indicates that the word-pair belongs to defined relation (Zheng et al. 2021a) and each word is an entity.

Given the textual graph $G_{T}=\left\{H_{T},Z_{T}\right\}$, and the visual graph $G_{I}=\left\{H_{I},Z_{I}\right\}$. We aim to align the nodes and edges of the cross-graphs simultaneously and to transfer the matched semantic information from objects into entities. Formally, an optimal transport method (Chen et al. 2020) is first used to match the nodes and edges in the cross-graph. Furthermore, we use image2text attention to transfer matched semantic information from the visual object to text modality and obtain the refined textual representation.

In this subsection, we aim to mine the different dependency features between $w_{i}$ and $w_{j}$ to help detect relations between them. As shown in Figures 5: (a) we consider that part of speech (Pos) can provide lexical information for word pairs. For example, the Pos of most entities belong to the NOUN and PEROPN, e.g., NBA, Curry, and Thompson; (b) encoding the syntactic distance (Sd) between word-pair can improve the ability of the model to capture long-range syntactic information; (c) the word co-occurrences matrix (Co) can provide corpus-level information between word pairs. For example, Curry and NBA appear some times in the corpus. The details about constructing each feature matrix are added in Appendix-A.

After data preprocessing, three feature matrices are obtained, ${M^{l}}\in{\mathbb{R}^{n\times n}},l\in\{Pos,Sd,Co\}$. We propose a W-GCN module to model each matrix, obtaining each channel representation. Each matrix ${M^{l}}$ firstly sends an embedding layer yielding a trainable representation ${R^{l}}\in{\mathbb{R}^{n\times n\times d_{l}}}$ and $d_{l}$ is the dimension of representation. The calculation W-GCN process of i-th word in l-th matrix is shown as,

$\displaystyle\begin{array}[]{l}S_{i}^{l}={\rm{W\mbox{-}GCN}}_{l}(R_{i}{{{}^{l}}},{O})\\ ={\rm{Softmax}}({\rm{ReLU}}(W_{l}^{{r_{1}}}R{{}_{i}^{l^{T}}}+{b_{l}}))\cdot(W_{l}^{{r_{2}}}(O))\end{array}$

(11)

where $R_{i}^{l}\in{\mathbb{R}^{n\times d_{l}}}$ is the i-th word in l-th linguistic matrix and $W_{l}^{{r_{1}}}{\in\mathbb{R}^{1\times d_{l}}}$ and $W_{l}^{{r_{2}}}{\in\mathbb{R}^{d_{T}\times d_{T}}}$ are shared weights used to perform a linear layer to learn linguistic features and representational abilities. We combine the representations and send them to the MLP (MultiLayer Perception) layer for obtaining the final word representation,

$\displaystyle{S_{i}}={\rm{MLP}}[S_{i}^{Pos};S_{i}^{Sd};S_{i}^{Co}]$

(12)

where $S_{i}\in\mathbb{R}^{d_{T}}$ is the i-th word representation. Thus, the output representation of the multi-channel layer is denoted as $S=[{S_{1}},{S_{2}}\cdots,{S_{n}}]$. Finally, we concatenate the enhanced representations of $S_{i}$ and $S_{j}$ to represent the word-pair $({w_{i}},{w_{j}})$, i.e., $r_{i,j}=[S_{i};S_{j}]$. Then, send the $r_{i,j}$ to a linear prediction layer and obtain the probability distribution,

$\displaystyle{p_{i,j}}={\rm{Softmax(}}{W_{p}}{r_{i,j}}+{b_{p}}{\rm{)}}$

(13)

where $W_{p}\in\mathbb{R}^{d_{y}\times 2d_{T}}$ and $b_{p}\in\mathbb{R}^{d_{y}}$ are trainable parameters and $d_{y}$ is the number of tags. Then, we used the cross-entropy error to measure the ground truth distribution $\mathcal{Y}$ and predicted tagging distribution,

$\displaystyle\mathcal{L}_{main}(\theta)={-}\sum\limits_{s=1}^{S}{\sum\limits_{i=1}^{n}{\sum\limits_{j=1}^{n}{\mathcal{Y}_{i,j}^{s}\log(p_{i,j}^{s}\theta)}}}$

(14)

where S and $\theta$ denote the number of training samples and all trainable parameters, respectively.

The final objective is a combination of the main task and optimizing the cross-graph as follows,

$\displaystyle\mathcal{L}={\mathcal{L}_{main}}+\lambda\cdot{\mathcal{L}_{graph}}$

(15)

where $\lambda$ are trade off hyper-parameters to control the contribution of optimizing the cross-graph.

Furthermore, the EEGA surpasses all baselines. Compared with the best results of existing baselines, EEGA still achieves absolute F1-score increases of 1.67% and 1.37% on JMERE and #MNER. The experimental results strongly prove that simultaneously aligning the nodes and edges in the cross-graph can effectively improve the precision of matching the objects with entities and capture more relationships between entities. In addition, the proposed attribute transformer enhances the ability to mine relations between nodes by incorporating edge information into the key and value in the transformer. Meanwhile, the multi-channel layer can take linguistic relations between word pairs to refine the final representation and improve prediction performance.

W/o edge-enhanced means removing the edge-enhanced graph optimal transport from EEGA. The performance of the model is highly degraded after removing the edge-enhanced, showing that simultaneously aligning nodes and edges in the cross-graph can be beneficial for matching the visual objects with textual entities more precisely and finding the entity classification clues from the relationship between objects.

To understand the effectiveness of our proposed model, Figure 6 presents three examples with the predicted results. Meanwhile, the important objects and relations are detected from images. In example (a), only the pipeline-based model UMGF+MEGA extracts incorrectly since the pipeline model easily suffers from error propagation, i.e., the extraction entity Arsene by UMGF is incomplete, and the final model UMGF+MEGA extract the incorrect quintuple. In example (b), UMGF+MEGA imprecisely extracts NBA as an entity, and *MEGA incorrectly predicts the relationship between Kobe and NBA MVP as the Present_in. In example (c), the situation is similar to example (b). Since lacking effective ways to map the semantic relationship of objects Man-near-Woman to entities (LILI-COLE), the UMGF+MEGA and *MEGA incorrectly predicts relations peer between entities.

For these three examples, the proposed EEGA makes accurate judgments. Benefiting from the edge-enhanced graph optimal transport module, the EEGA can align the nodes and edges in the cross-graph to match the entities with objects more precisely. Meanwhile, the EEGA also effectively captures the relation clue from the visual graph to the textual graph shown in examples (b) and (c). In addition, the attribute transformer and multi-channel layer can further enhance the ability to model the relationships of objects and word pairs.

In this section, we visualize the example (b) of Figure 6 at $\alpha=1$ and $\alpha=0.4$ to test whether our edge alignment strategy helps to learn fine-grained entity-object matching. As shown in Figure 7 (a), when only node alignment means $\alpha=1$, since the proposed model lacks the edge constraints, the attention weight is relatively scattered and affects the precision of matching entities with objects. Particularly, the model easily classifies the type of entity NBA as Per. Meanwhile, as shown in the NBA and Kobe in Figure 7 (b), benefiting from the edge alignment can find the mapping between the object-object relationships and entity-entity relationship; the EEGA effectively reduces the ambiguity and match objects with entities more precisely.