Text Classification through Glyph-aware Disentangled Character Embedding and Semantic Sub-character Augmentation

Embedding methods based on character images have been proposed with some excellent success Chen et al. (2015); Sun et al. (2016); Yu et al. (2017); Sun et al. (2019); Dai and Cai (2017); Shimada et al. (2016); Liu et al. (2017); Kitada et al. (2018); Ke and Hagiwara (2017); Aldón Mínguez et al. (2016). These methods are also called glyph-aware embedding as they generate embeddings that take into account the shape of the characters or sub-characters. These image-based methods mainly use convolutional neural networks (CNNs) or convolutional auto-encoders (CAEs) Masci et al. (2011) for character-embedding learning, and they perform well because of the following advantages: (1) they operate without the cumbersome word segmentation required by some Asian languages, and (2) they can apply additional image-based data augmentation.

For NLP tasks, it is challenging to apply data augmentation methods because of the need to consider the context of the text Sennrich et al. (2016); Jia and Liang (2016); Silfverberg et al. (2017); Edunov et al. (2018). Several data augmentation methods that do not require text analysis have been proposed for word embedding Iyyer et al. (2015); Zhang et al. (2016) and character embedding Shimada et al. (2016). In particular, Shimada et al. (2016) achieved significant performance improvements by applying dropout Hinton et al. (2012)-based data augmentation to a type of character embedding called wildcard training (WT). However, these methods have little interpretability of what the data augmentation means in the input text, partly due to the lack of interpretability of the embedding itself. Our proposed SSA is improved WT, and it replaces the sub-character of a character with another readable sub-character.

For learning a latent representation that can be interpreted, InfoGAN Chen et al. (2016) and $\beta$-VAE Higgins et al. (2017) are well known. Unlike InfoGAN, $\beta$-VAE is stable while training, requires less assumptions about the data, and relies on only a single hyperparameter $\beta$. Because of these advantages, several improved models based on $\beta$-VAE have been proposed (e.g., Factor-VAE Kim and Mnih (2018), HFVAE Esmaeili et al. (2019)). Therefore, in this paper, we use $\beta$-VAE as a VCE to learn interpretable character embeddings.

We obtain the GDCE using the VCE based on the $\beta$-VAE. Since the GDCE provides dimensionally independent features, we expect to solve the problem of the poorly interpretable character embedding obtained by the CAE.

$\beta$-VAE is a generative model that estimates the data distribution $p(\bm{x})$, where $\bm{x}\in\mathbb{R}^{d}$ is a $d$-dimensional input. Let $\bm{z}\in\mathbb{R}^{d^{\prime}}$ be a $d^{\prime}$-dimensional latent variable, which is derived from the GDCE in this paper; $p(\bm{z})$ is a normal distribution, which is the prior distribution of the latent variables, $q(\bm{z}|\bm{x})$ is the posterior distribution, and $p(\bm{x}|\bm{z})$ is a generative model. We optimize the following function:

where $\beta$ is a balancing coefficient for the second term. The first term represents the reconstruction error of the character image. The second term represents the regularization of the latent variables that are learned so as to follow the prior distribution by the KL divergence $D_{\mathrm{KL}}[\cdot||\cdot]$. If the coefficient $\beta$ increases, it is possible to obtain a representation of the features where each dimension is independent Higgins et al. (2017).

However, the latent variables themselves are a probability distribution and cannot be backpropagated to the encoder. Hence, the reparameterization trick Kingma and Welling (2013) of the approximation method is used. We let $\bm{\alpha}$ be a sampled random variable from $\mathcal{N}(\bm{0},\bm{I}_{d^{\prime}})$ and calculate the latent variables as follows:

where $\odot$ is an element-wise product, $\mu$ is the mean of the distribution, and $\sigma$ is the variance of the distribution. Here, $\mu(\bm{x})$ and $\sigma(\bm{x})$ are $d^{\prime}$-dimensional vectors obtained from the $\beta$-VAE.

The sequence of $c$ embedded characters $C=\{\bm{z}^{(1)},\bm{z}^{(2)},\cdots,\bm{z}^{(c)}\}$ from the GDCE, where $\bm{z}^{(t)}$ is the $t$-th character embedding, in the VCE are provided to the following character-level text classifier. The parameters of the classifier are optimized in the back-propagation using the cross-entropy error.

In this paper, we propose SSA as a data augmentation method. Taking advantage of the preferred features of the embedding created by the GDCE, we expect that the sub-character of a character will be replaced by another readable sub-character, using the SSA.

Let $\gamma$ be the perturbation range, and the formula of the SSA for the $i$-th dimension $\bm{z}^{(t)}_{i}$ of the character embedding $\bm{z}^{(t)}$ is defined as follows:

where $u\sim\mathcal{U}(a,b)$ indicates that the random variable $u$ has a uniform distribution with the minimum $a$ and the maximum $b$. Since each dimension of the GDCE follows $\mathcal{N}(\bm{0},\bm{I}_{d^{\prime}})$, the character embedding converted in Eq. 3 falls within the range of trained character-embedding values.

We evaluated our framework with the following datasets: newspaper and livedoor. These datasets were split into two parts: 80% for training and 20% for evaluation. Because these datasets contain new words and/or meanings related to current affairs, accurate word segmentation through morphological analysis has been a challenge in conventional word-level processing for Japanese. Therefore, we can avoid such difficulties by using character-level input instead of word-level input²²2It is generally known that a character-level model performs better than a word-level model in Chinese and Japanese Zhang and LeCun (2017)..

We trained the VCE based on $\beta$-VAE and character-level CNN (CLCNN) Zhang et al. (2015) as text classifier independently. The hyperparameters of these models were adjusted with a validation set split from the training set, and the predicted results of the evaluation set were reported.

Table 3 presents a comparison of the proposed GDCE and CAE-based embedding. The GDCE showed better document- and sentence-level classification performance than the conventional CAE-based character embedding without data augmentation. This may be due to the fact that the characters to be learned by the VCE are distributed in a limited embedded space centered on zeros, so the later stage of the CLCNN training became more effective. The WT, which randomly set all representations of a particular character embedding to zero, enhanced the discrimination of both models. The effect on the CAE-based model was particularly large, as reported in previous studies. We can confirm an effect of the WT as a dropout for preventing overfitting, but it did not provide an interpretation of what was changed in the character embeddings.

The proposed SSA provided us with an idea of what the embedding changes would look like, while also providing the same discriminatory capacity as the WT. This may be due to the fact that the GDCE had standardized metrics in the embedding space (i.e., the embedding had a normal distribution), so that the distances between the character embeddings were within the range of what could be assumed. Hence, the size of the perturbations applied could be designed, allowing for meaningful data augmentation. However, the CAE with SSA did not show an improved classification performance. This may be due to the fact that the CAE with SSA does not change to a meaningful character representation.

Figure 2 shows a comparison of the reconstructed character images when a $\pm 2.0\sigma$ perturbation is placed on the 2 (the GDCE) and 2 character embedding obtained by the CAE. In Figure 2, it is confirmed that the shape of the character replaced a different interpretable character or characters with a similar different subcomponent in the input space. In particular, by adding a perturbation to the fifth dimension of the embedding of “迫” or “追” (containing a sub-char. of “辶,” meaning road), it can be interpreted that it changed to “氵” (sub-char. of water) or “辶” (sub-char. of road, the same as “辶”). In addition, by adding a perturbation to the first dimension of the embedding of “綱” or “縄” (containing a sub-char. of “糸,” meaning yarn), it can be interpreted that it changed to “扌” (sub-char. of hand) or “金” (sub-char. of gold). From these results, we are convinced that such a replacement in the embedding resulted in more effective data augmentation for training the model.

As seen in Figure 2, in contrast, we were unable to identify these trends. We consider this is one of the typical benefits of our framework in that each dimension of the GDCE is independent and each of them affects each character component (e.g., sub-char. or radical of the character) with independence. In other words, we can change only some part of the character by changing certain dimensions of the embedding.

Since the SSA is a local transformation for the parts of the character shown above, even some characters that do not actually exist are generated by the combination of parts. These are not readable as correct characters, but we can make certain interpretations of them. In sum, the combination of the proposed GDCE and SSA provides us with the interpretability of the data augmentation as well as embedding the character while providing a high discriminative power.

To understand the effect of hyperparameters, we analyzed the coefficient $\beta$ and perturbation size $\gamma$ using the livedoor, as shown in Figure 3.

At present, the role of each dimension in the character reconstruction of the GDCE cannot be clearly defined because it depends on the training of the model. Also, since the VCE was independently trained from the classifier (i.e., not in an end-to-end manner), trained embedding can only consider visual features, not the semantic ones. We will be working on these in the future.