Disentangled and Robust Representation Learning
for Bragging Classification in Social Media

The bragging classification task aims to automatically classify bragging type of a given text from social media. Since pre-trained language models show their excellent performance in natural language processing (NLP) tasks, the general paradigm for NLP classification tasks is to fine-tune pre-trained language models (e.g. BERT [19], RoBERTa [20]). In this work, our base bragging classification model is composed of a pre-trained transformer encoder and an MLP with $\mathrm{softmax}$. Given a text ${\bm{T}}_{i}$, the model is to distinguish its bragging type $c$, i.e,

	$\displaystyle{\bm{h}}^{i}$	$\displaystyle=\operatorname*{Trans-Enc}({\bm{T}}_{i};\theta^{(enc)})$		(1)
	$\displaystyle{\bm{p}}^{i}$	$\displaystyle=\mathrm{softmax}(\operatorname*{MLP_{clf}}({\bm{h}}^{i};\theta^{(clf)}))$		(2)

where $i$ denotes $i$-th training example; ${\bm{h}}^{i}$ denotes representation of the text ${\bm{T}}_{i}$, and ${\bm{p}}^{i}$ refers to a probability distribution over all bragging types ${\bm{C}}$. Lastly, we train the pre-trained transformer encoder $\theta^{(enc)}$ and MLP $\theta^{(clf)}$ by maximum likelihood estimation, and its loss function is written as:

$\displaystyle{\mathcal{L}}^{(clf)}=-\dfrac{1}{|{\mathcal{T}}|}\sum_{{\mathcal{T}}}\log{\bm{p}}^{i}_{[y=c]},$

(3)

where ${\mathcal{T}}$ denotes a collection of training texts; $c$ refers to a ground truth bragging type, and $c\in{\mathcal{C}}$.

To alleviate data imbalance problem, previous works [2] propose data augmentation to enrich text embedding representation by injecting external knowledge from the NRC word-emotion lexicon, Linguistic Inquiry and Word Count(LIWC) and vectors clustered by Word2Vector. However, these methods achieve only a limited gain (i.e., -0.5% $\sim$ 1.09% on F1) compared to the base bragging classification model. The reason is that injected external knowledge is fixed (not trainable) and is full of noise information unrelated to bragging classification, which interferes with the model training. Therefore, we introduce a disentangled-based feature augmentation method instead of external knowledge injection.

Since the text is encoded as representation, it is reasonable to assume that representations contain separable content and specific features from a feature disentanglement perspective [21]. Therefore, we separate the representation ${\bm{h}}^{i}$ into two features: one is closely related to bragging types, denoted as ${\bm{h}}^{i}_{t}$, and the other contains separable content unrelated to bragging, denoted as ${\bm{h}}^{i}_{c}$. To disentangle these two features from representation ${\bm{h}}^{i}$, we employ two MLPs for content and bragging type features disentangle, respectively, i.e.,

	$\displaystyle{\bm{h}}^{i}_{c}$	$\displaystyle=\operatorname*{MLP_{cont}}({\bm{h}}^{i};\theta^{(cont)})$		(4)
	$\displaystyle{\bm{h}}^{i}_{t}$	$\displaystyle=\operatorname*{MLP_{type}}({\bm{h}}^{i};\theta^{(type)})$		(5)

where ${\bm{h}}^{i}_{c}$ and ${\bm{h}}^{i}_{t}$ denote content and bragging type disentangled features, respectively.

Based on disentangled features, we can obtain new augmented features by integrating content and bragging type features from different samples, i.e,

$\displaystyle\hat{{\bm{h}}}^{ji}={\bm{h}}^{i}_{c}+{\bm{h}}^{j}_{t}$

(6)

where $\hat{{\bm{h}}}^{ji}$ denotes an augmented feature; ${\bm{h}}^{j}_{t}$ refers to a bragging type feature from $j$-th training example. It is remarkable that bragging type of $\hat{{\bm{h}}}^{ji}$ is the same as bragging type feature ${\bm{h}}^{j}_{t}$ because it contains bragging type features ${\bm{h}}^{j}_{t}$. Augmented feature $\hat{{\bm{h}}}^{ji}$ is passed into Eq.2 to derive a probability distribution ${\bm{p}}^{ji}$, and loss function can be defined as:

$\displaystyle{\mathcal{L}}^{(clf^{+})}=-\dfrac{1}{|{\mathcal{T}}|}\sum_{{\mathcal{T}}}\log{\bm{p}}^{ji}_{[y=c]},$

(7)

Moreover, we employ Kullback-Leibler (KL) divergence loss to guide representation disentangling and representation reconstruction. Specifically, we disentangle $\hat{{\bm{h}}}^{ji}$ to reconstruct content feature $\hat{{\bm{h}}}^{i}_{c}$ and bragging type feature $\hat{{\bm{h}}}^{j}_{t}$ by Eq.4 and Eq.5. Then, we optimize the reconstruction loss by minimizing KL divergence, defined as follow:

	$\displaystyle{\mathcal{L}}^{(kl)}$	$\displaystyle=\mathbb{E}(\log{\bm{h}}^{x}_{y}-\log\hat{{\bm{h}}}^{z}_{y}),$
		$\displaystyle\text{where,}~{}x,z\in{\mathcal{T}},y\in\{c,t\}$		(8)

Although model trained on augmented features circumvents data imbalance problem, it inevitably suffers from domain discrepancy between encoder representations and augmented features, which is verified to undermine the performance [22]. We thus design an adversarial strategy to constrain the augmented features to follow the same domain with representations. Formally, a discriminator is built upon representation and augmented features:

	$\displaystyle p^{(adv)}$	$\displaystyle=\operatorname*{Sigmoid}(\operatorname*{MLP_{dis}}({\bm{h}};\theta^{(dis)})),$
		$\displaystyle\text{where,}~{}{\bm{h}}\in\{{\bm{h}}_{i},{\bm{h}}_{j},\hat{{\bm{h}}}^{ji},\hat{{\bm{h}}}^{ij}\}.$		(9)

$p^{(adv)}$ is the probability of the feature not augmented. The discriminator is trained to minimize:

$\displaystyle{\mathcal{L}}^{(dis)}=-\mathbb{I}_{(rep)}\log{\bm{p}}^{(adv)}-\mathbb{I}_{(aug)}\log(1-{\bm{p}}^{(adv)})$

(10)

where $\mathbb{I}_{(aug)}$ denotes if the feature is an augmented feature; $\mathbb{I}_{(rep)}$ denotes if the feature is encoder representation. On the contrary, augmented features are learned to fool by minimizing an adversarial loss, i.e.,

$\displaystyle{\mathcal{L}}^{(adv)}=-\mathbb{I}_{(aug)}\log{\bm{p}}^{(adv)}.$

(11)

During training, we alternately train model and discriminator. we train our model jointly by minimizing four loss functions:

$\displaystyle{\mathcal{L}}\!\!=\!\!\alpha\times{\mathcal{L}}^{(clf)}\!+\!\beta\times{\mathcal{L}}^{(clf^{+})}\!+\!\lambda\times{\mathcal{L}}^{(kl)}\!+\!\gamma\times{\mathcal{L}}^{(adv)}$

(12)

Meantime, discriminator is trained by ${\mathcal{L}}^{(dis)}$.

In experiments, we train and evaluate our proposed approach on the dataset built by [2]. This dataset is collected from tweeter by Premium Twitter Search API and further annotated by human-craft. Bragging types include “Not Bragging”, “Achievement”, “Action”, “Feeling”, “Trait”, “Possession” and “Affiliation”. We follow the official data split with 3,382/3,314 samples in training/test sets. Following [2], evaluation metrics are macro precision, recall and F1 score.

The pre-trained transformer encoder we used is BERTweet [23]. We use AdamW optimizer[24, 25] with learning rate of $3\times 10^{-6}$ for training, and learning rate for discriminator is $3\times 10^{-4}$. The maximum training epoch and batch size are set to 26,500 and 35. The maximum sequence length, weight decay and gradient clipping are set to 128, 0.01 and 1.0. The dropout of model and discriminator are set to 0.2 and 0.5, separately. In Eq.12, $\alpha$, $\beta$, $\lambda$ and $\gamma$ are set to 1.5, 1.5, 1.0 and 1.5, respectively. Experiments are conducted on a NVIDIA RTX2070 GPU, and training time is around 5 hours.

Comparison results are shown in Table 1. On the table, it is clear that 1) using pre-trained language model as base model can significantly improve performance, mainly because pre-trained language models have stronger text representation capability; 2) our approach is superior to the method proposed by [2] and achieves state-of-the-art,demonstrating the proposed disentangled and robust representation learning is effective. From these two observations above, it is insightful to assert that learning better representation from data itself is more critical than external knowledge injection.

As shown in Table 2, we conduct an ablation study on our method. First, we remove domain-aware adversarial strategy, and results show that performance drops. It demonstrates that constraining the domain of augmented features using adversarial strategy is beneficial for model to learn robust representation. Moreover, we verify the effectiveness of augmented features by removing disentangle-based feature augmentation. Results show that the performance drops a lot, which demonstrates that disentangled representation learning is able to mine text information deeper.

To comprehensively evaluate our method, we also evaluate our model on bragging binary-classification task (i.e., bragging and non-bragging). As shown in Table 3, we can observe that our model still achieves better performance as compared with other methods. Since data imbalance is greatly alleviated in binary-classification setting, the improvement of our method is not as significant as the multi-classification setting.

To delve into the effectiveness of disentangled features, we employ MLP to classify disentangled feature ${\bm{h}}^{i}_{t}$ directly, and results are shown in Table 4. From Precision results, we can observe that Only ${\bm{h}}^{i}_{t}$ outperforms BERTweet, which shows the effectiveness of feature disentanglement. Moreover, Only ${\bm{h}}^{i}_{t}$ underperforms our method on F1 Score, which indicates that content feature disentanglement and domain consistency benefit bragging classification.

We apply t-SNE visualization(Fig 2.) on features obtained by Eq.1 to visually demonstrate how our approach works. It is obvious that scatters of our method represent a more clustering trend than BERTweet, which indicates our method is able to learn a type-relevant representation.

To investigate the impact of domain-aware adversarial strategy, we show classification loss and discriminator loss in Figure 3. The discriminator loss quickly drops and then slowly goes up, indicating that the discriminator is functional at first and fooled later by domain-consistency features. Meanwhile, The classification loss decreases rapidly and remains very low in the following steps, showing that domain-aware adversarial strategy is well integrated within the whole process.