Assume that we have a text classification task to train a model to assign one label from $\mathcal{C}=\{c_{1},c_{2},...,c_{k}\}$ to a sentence or document. The training set is $\mathcal{D}$ and its vocabulary is $\mathcal{V}$. For a word $w_{i}\in\mathcal{V}$ and a category $c_{j}\in\mathcal{C}$, we can examine their relationship from two perspectives:
$\bullet$ Statistical Correlation. This measures how frequent the word $w_{i}$ co-occurs with class $c_{j}$ while not with other classes in the training set.
$\bullet$ Semantic Similarity. This measures how much semantics the word $w_{i}$ share with the label of class $c_{j}$.

With the two perspectives, all the words in a training document can be divided into four different roles (Figure 1):

• 1.

  Gold words: these are useful class-indicating words, with high statistical correlation and high semantic similarity with the corresponding category;

• 2.

  Venture words: these words usually co-occur frequently with their corresponding categories but have low semantic similarities. Thanks to their high statistical correlation with the class, these Venture words can usually bring extra information about the class. However, their low semantic similarity puts them at greater risk to be the noise or misleading words, which are harmful for model training;

• 3.

  Bonus words: these words have low statistical correlation but high semantic similarity. Bonus words usually don’t occur frequently in the training data, but are quite useful for model’s generalization ability;

• 4.

  Trivial words: these are those with low statistical correlation and low semantic similarity, which are perhaps less important in model prediction.

To recognize the words of different roles in a dataset, we should decide proper metrics to measure the above two perspectives.

For the measurement of statistical correlation with the class, we employ weighted log-likelihood ratio (WLLR) to select the class-correlated words from the text sample. This is inspired by (Yu and Jiang 2016) where WLLR is used to find out the ”pivot words” for sentiment analysis. The WLLR score is computed by:

$\displaystyle wllr(w,y)=p(w|y)log(\frac{p(w|y)}{p(w|\bar{y})})$

where $w$ is a word, $y$ is a certain class and $\bar{y}$ represents all the other classes in the classification dataset. $p(w|y)$ and $p(w|\bar{y})$ are the probabilities of observing $w$ in samples labeled with $y$ and with other labels respectively. We use the frequency of a word occurring in the certain class to estimate the probability.

To measure the semantic similarity between a word and the meaning of the class label, a straightforward way is to use word vectors pre-trained with skip-gram (Mikolov et al. 2013) or Glove (Pennington, Socher, and Manning 2014). We are not using transformer-based models like BERT for similarity measuring due to their high inference cost. Some also reveal that static word-embeddings can achieve comparable and even better performance than BERT-like models in similarity measurement tasks, especially in word-level (Reimers and Gurevych 2019). We compute the cosine similarity between a word and a class label to see their semantic distance:

$\displaystyle similarity(w,l)=\frac{v_{w}\cdot v_{l}}{\|v_{w}\|\|v_{l}\|}$

where $l$ represents the label and $v_{w}$, $v_{l}$ are word vectors for the word $w$ and the label $l$. We can also use a description of the label to obtain $v_{l}$ by averaging the word vectors of each word in the description for better label representation. In our experiments, we find that simply using the word or phrase of the label itself is enough to measure the similarity between a word and the category.

We compute the WLLR and similarity scores of each word, and set a threshold to divide the high and low scores (or set different thresholds respectively). We call the words with high (low) WLLR socres as $C_{h}$ ($C_{l}$) and words with high (low) similarity socres as $S_{h}$ ($S_{l}$). Then we can extract these words with the following rules:

	$\displaystyle W_{G}$	$\displaystyle=\{w|w\in C_{h}\cap S_{h}\}$
	$\displaystyle W_{V}$	$\displaystyle=\{w|w\in C_{h}\cap S_{l}\}$
	$\displaystyle W_{B}$	$\displaystyle=\{w|w\in C_{l}\cap S_{h}\}$
	$\displaystyle W_{T}$	$\displaystyle=\{w|w\in C_{l}\cap S_{l}\}$

where $W_{G}$, $W_{V}$, $W_{B}$ and $W_{T}$ are Gold, Venture, Bonus, and Trivial words respectively.

Given a training sample, there are two strategies to assign the roles to the words from the text: global or local. The global strategy determine the thresholds for roles division based on all the words in the vocabulary, while for local strategy the division is based on the words from the current sample. In other words, the local strategy use a relative view to distinguish high and low scores, therefore the a word’s role may change when it occurs in another sample.

In our experiments, we use the median number as the threshold for the local strategy, and choose the upper (lower) quartile as the high (low) score threshold for the global strategy. We find these choices of thresholds to work well in our first try and did not tune these thresholds. Tuning these hyper-parameters might result in further gains for our proposed method.

Here we provide an actual example to show the differences of these roles more vividly. We randomly sample 100 documents from the 20 Newsgroup¹¹1https://archive.ics.uci.edu/ml/datasets/Twenty+Newsgroups text classification dataset, which consists of 20 categories. We do the word roles recognition using the global strategy and some examples are show in Table 1.

As shown in Table 1, Gold words are highly related to their corresponding category, like ”transistor”, ”circuit” and ”batteries” for the electronics class. Venture words are not semantically related to their classes, but these words co-occur frequently with the corresponding categories in the given dataset. Intuitively, we expect the models to learn class-indicating features from these Gold words, while less from the Venture words, since some of these Venture words might be misleading. For instance, words like ”sufficient” and ”signature” appear lots of times in electronics documents of the dataset, but we shouldn’t rely on these words to tell if a document is about electronics or not. However, according to our preliminary experiments, these Venture words are usually learned by the classifiers as evidence for certain categories, and are likely to result in wrong predictions. Bonus words are semantically similar to the categories, but are less commonly found in these categories. Trivial words seem to be uninformative and are less important for prediction. The examples illustrate that our proposed four kinds of word roles are reasonable and can provide insights for text classification and data augmentation. Note that we only use a subset with 100 documents to identify the roles in these examples, more precise roles recognition can be achieved when more data are provided.

We also conduct experiments with increasing training size ($n=\{50,100,500,1000\}$) on the NG and BBC datasets as shown in Table 3. STA consistently outperforms the competitive EDA-w2v baseline when more labeled data is available.

To verify the effectiveness of STA on stronger text classification models, we also evaluate different augmentation methods on BERT-base on NG and IMDB datasets and compare their averaged results in Figure 3. We can see that STA outperforms other methods both with smaller and larger classification models. Interestingly, even with smaller classification model (DitilBERT), STA helps the model to achieve higher or comparable results than other methods with larger model.

STA is composed of four operations: selective replacement, selective insertion, selective deletion and positive selection. We now evaluate each of these operations, and also compare them with non-selective operations where the words are randomly selected for text-editing. We conduct experiments on NG, IMDB and Yahoo datasets, and use the global strategy for STA. The results are shown in Table 4.

From these experiments, we have two important findings:
1. The positive selection of STA is particularly powerful as an augmentation operation for low-resource text classification, which is significantly better than all the other operations. The generated new text by positive selection is mainly composed of the Gold words from the text with high statistical and semantic relationships with the label. We speculate that adding these samples into the training set can help the model to learn the most class-indicating features about the class, even if only small number of examples are available.
2. Selective insertion is more effective than random insertion with very few examples (n = 50) but are less useful when more examples are provided (n = 100). In the contrary, selective deletion and selective replacement are more effective with larger training set. This is probably due to the fact that the word roles recognition is more accurate when providing more labeled data.

These findings can further guide us to design a more powerful selective augmentation method, which we leave for future work.

Recall that STA is based on word roles recognition, which relies on two different relationship measurements between a word and a class: statistical correlation and semantic similarity. It is natural to ask: do we really need both statistical correlation and semantic similarity with the class for defining word roles? To verify this, we conduct ablation experiments of by designing two variants of STA:
1. STA w/o Cor.: During role words recognition, we only use semantic similarity to separate the words in the text into two types: high-similarity and low-similarity words. By doing so, Gold and Bonus words, or Venture and Trivial words can no longer be distinguished from each other.
2. STA w/o Sim.: During role words recognition, we only use statistical correlation to separate the words in the text into two types: high-correlation and low-correlation words. This way, the previous Gold and Venture words are combined together, Trivial and Bonus words are combined together.

We then follow the same word selection rules with the original STA, and experiment on NG and IMDB datasets with 100 training examples. For NG, STA uses global strategy while for IMDB STA uses local strategy. The results are reported in Table 5. The results illustrate that removing the statistical or semantic factor both lead to performance degradation compared to original STA for NG and IMDB tasks. STA w/o Sim. is worse than STA w/o Cor. which shows that semantic similarity plays a more important part than statistical correlation for proper word selection in STA. Though inferior to the original STA, STA w/o Sim. and STA w/o Cor. both surpass the performance of other non-selective baselines, which illustrates that word selection is vital for text augmentation.