Learning from Bootstrapping and Stepwise Reinforcement Reward:
A Semi-Supervised Framework for Text Style Transfer

The overview of our proposed semi-supervised framework is shown in Figure 1. Given the non-parallel datasets $\mathcal{D}_{S}$ and $\mathcal{D}_{T}$, we use lexical or semantic features for pseudo parallel pair matching. The training process consists of two stages: (1) the generator model $G$ is trained on the pseudo parallel samples, where cross-entropy loss over the target sentence tokens is used to optimize generated output probabilities, i.e. the bootstrapping step; (2) we incorporate reconstruction and style rewards to enhance attribute rewriting and content preservation, where reinforcement learning is used to optimize the generation, i.e. the reward-based learning. Moreover, the second stage can use pseudo parallel pairs as well as the non-parallel samples.

To build the pseudo parallel data for bootstrapping, we investigate lexical similarity and semantic similarity for sentence matching.
Lexical Similarity In text style transfer, rewriting is often accomplished by changing a few words or phrases that are indicative of a particular attribute in the source sentence, namely attribute markers, while leaving the rest largely unaltered Li et al. (2018). For example, “Moving past the shape, they were dry and truly tasteless.”, a sentence with a negative sentiment style, can be transferred to a positive style by changing or replacing sentiment-specific words “dry” and “tasteless”, while keeping other words intact. This intuition has inspired the template-based and editing-based rewriting approaches Li et al. (2018). Here we employ it for the lexical feature extraction. First, from unaligned corpora of two styled subsets (e.g., positive, negative), we identify attribute markers by sorting phrases that occur with far higher frequency within one attribute than the other (e.g., “worst” and “very disappointed” are negative markers). Second, for each sentence in the two subsets, we remove those markers, and regard the remaining words as its content-preserved spans. Then we match the content-preserved spans of style $S$ to those of style $T$ with the smallest Levenshtein editing distance (see examples shown in Table 1).

With the pseudo parallel data, we can conduct supervised learning with token-level maximum likelihood estimation (MLE). In our framework, we use a sequence-to-sequence neural network. Since the large-scale pre-trained language models boost the performance of various downstream tasks, we use BART (Lewis et al., 2020) as the language backbone, which is a denoising autoencoder with strong language generation capability. Given a source sentence $\bm{x}$ and a reference sentence $\bm{y}$, the cross-entropy loss is calculated between the decoder’s output and the reference sentence:

Moreover, to avoid the generation becoming over-fitting to the pseudo parallel data, we add the label smoothing on the cross-entropy loss Müller et al. (2019), with the smoothing weight $\lambda$ = 0.15.

Upon the supervised learning from the pseudo parallel data, the model can be further improved by unsupervised learning from the massive unlabeled data. For the unsupervised stage, we adopt reinforcement learning, and use two rewards to enhance style rewriting and content preservation.

For extensive experiments, in this paper, we select three representative text style transfer corpora: Yelp (business reviews), Amazon (product reviews), and Grammarly’s Yahoo Answers Formality Corpus (GYAFC) Li et al. (2018); Rao and Tetreault (2018). The training, validation, and test split are the same as previous work Luo et al. (2019); Chawla and Yang (2020), and their task types and statistics are shown in Table 2. In the non-annotated corpora Yelp and Amazon, human-written references are only available for the test set. Therefore, to build the pseudo parallel data described in the previous section, we filter out the sentence pairs with lexical or semantic similarity lower than a threshold, and remove sentences that are shorter than 5 words. The pseudo parallel set is used for the bootstrapping training (Section 3.3), and the rest samples are used for the unsupervised stage (Section 3.4).

The framework is implemented with Pytorch and Hugging Face Transformers²²2https://github.com/huggingface/transformers. The ‘BART-base’ model is selected as the generator $G$. For style classification, ‘RoBERTa-base’ is used. We fine-tune models with AdamW (Kingma and Ba, 2015) with batch size 32; initial learning rates are all set at $2e^{-5}$. Style reward penalty $\gamma$ is $0.2$. Values for $\lambda$ are set to $1.0$ for style reward and $0.8$ for cyclic reward. Beam search size is set at $6$. Test results are reported with best validation scores (see Appendix Table 9 for environment and hyper-parameter setting details, and Algorithm 1 for the training process).

As previous work Luo et al. (2019); He et al. (2020); Sancheti et al. (2020), we adopt the following evaluation metrics: (1) Style Accuracy is calculated via binary classification to measure the style strength of re-writing. While the TextCNN (Kim, 2014) is used in previous studies, we also adopt a Transformer ‘RoBERTa-base’ classifier, where the reported scores are similar in our settings; (2) BLEU score is calculated on the prediction and human references to measure the content preservation; (3) We also compute the geometric mean (G2) and harmonic mean (H2) of style accuracy and BLEU score; (4) Since recent metrics with semantic similarity show better correlation with human judgments than traditional lexical measures. We also calculate BertScore between generation and references Zhang et al. (2020a).

A number of representative unsupervised baseline models are selected for extensive comparison on the Yelp corpus: (1) models that adopt content-style disentanglement such as Cross Aligned Shen et al. (2017) and Style Embedding Fu et al. (2018); (2) models that adopt back-translation such as Unsupervised MT Zhang et al. (2018), and Dual RL Luo et al. (2019), and recent state-of-the-art models Deep Latent He et al. (2019) and Direct Rewards w/ GPT-2 Liu et al. (2021). For our semi-supervised framework, we first (1) apply vanilla supervised learning to assess the effectiveness of the pseudo parallel data construction; (2) bootstrap the model with 30K pseudo parallel pairs, then further train it via reward-based learning; (3) apply semi-supervised learning by bootstrapping the model with 30K pseudo parallel pairs, and using 70K non-parallel samples for the reward-based training. As shown in Table 3, vanilla supervised training on the 30K pseudo parallel data lead to favorable scores of style accuracy, though they do not perform well in terms of BLEU scores, as the pseudo pairs emphasize style converting rather than content preservation. Further training with rewards improves both the style accuracy and BLEU score, and models with both lexical and semantic pseudo data produce comparable results with only 30k samples. Performance is further improved by using additional non-parallel data (70k samples), where our models outperform state-of-the-art baselines significantly.

For the Amazon sentiment transfer corpus, we adopt the same training strategies described in Section 4.3. Aside from unsupervised models, we also select the semi-supervised model Semi-LM-MMI w/ BART Chawla and Yang (2020), which adopted a language model-based discriminator for maximizing token-level conditional probabilities for training. Due to label noise in online-crawled data, the style accuracy for all models becomes lower than those trained on Yelp, and the classifier precision is only 86% (see Table 4). We also observed that the lexical similarity of pseudo parallel pairs is smaller than Yelp samples, and results in lower BLEU scores, especially when we apply supervised training on the 30K pseudo parallel data. On the other hand, content preservation largely benefits from the reward-based learning. Unsurprisingly, after bootstrapping, training with rewards significantly improves the generation quality, and our framework achieves state-of-the-art performance. Moreover, bootstrapping with lexical-based and semantic-based pseudo data resulted in a similar final performance with reward learning.

In recent work, it is shown that style transfer models trained on parallel data can benefit from additional reward-based learning Lai et al. (2021). Here we conduct additional experiments to assess our semi-supervised framework on the GYAFC formality transfer corpus with well-annotated data. We evaluate the proposed model on the informal-to-formal task as previous work Chawla and Yang (2020), and compare them with strong baselines. As shown in Table 5, while the baselines show impressive BLEU scores on the formality transfer task, our framework outperforms them significantly in terms of style accuracy, approaching upper-bound human performance. Moreover, compared with the contemporary supervised work Lai et al. (2021), which also introduced additional RL-based optimization, our model still achieves higher G2 and H2 scores. The examples shown in Appendix Table 13 demonstrate that our approach generates sentences with accurate formality paraphrasing.

Additionally, we conducted a human evaluation on Yelp, Amazon and GYAFC datasets. Following previous work Chawla and Yang (2020); Liu et al. (2021), we evaluated the generated sentences from three aspects: style transfer strength (Style), text fluency (Fluency), and content preservation (Content), separately. The three aspects are rated with range [1, 5], then their average value is calculated and reported as Mean (see Table 14 in Appendix). For each corpus, we randomly selected 80 test samples and compared the outputs of representative and previous state-of-the-art models. Each candidate was rated by three linguistic experts, and we report the average scores. Our model achieves better overall performance when considering all three evaluation metrics on each dataset. Moreover, we observe that leveraging the pre-trained language models such as BART and GPT-2 is beneficial for the text fluency.

We conduct an ablation experiment to assess the effectiveness of stepwise reward re-weighting. As shown in Table 6, the performance degrades significantly without the stepwise reward re-weighting, especially the BLEU score. In particular, we observed that when removing stepwise optimization, the generator was prone to mode collapse. In one manifestation of mode collapse, the model appended a limited set of phrases to the source sentences, resulting in generation with disfluency and low diversity. It demonstrates that token-level reward optimization provides finer-granularity for policy gradient of sequence-to-sequence training. This approach can also be potentially extended to other text generation tasks.

We utilize attentive scores from the top-2 multi-head layers for stepwise reward re-weighting. To study the effect of layer selection, we compared the results using attention scores extracted from different Transformer layers in the style classifier described in Section 3.4. As shown in Table 7, the performance shows an overall increasing trend from the 7-th to the 12-th layer, and we obtained better results with the top layers. In scores of lower layers, we found that the model tended to assure content preservation rather than style accuracy. This is consistent with the observations from recent linguistic probing and model interpretation studies Hewitt and Manning (2019); Xu et al. (2020): the information modeled in the Transformer-based networks, especially the pre-trained language backbones, is represented in a hierarchical manner, and the higher layers provide more effective information on scoring the span importance for text classification (see visualization in Appendix Figure 4).

We investigate the effect of different pseudo parallel sample sizes. As shown in Table 8, the result shows that the evaluation result by automatic metrics becomes acceptable when training reaches 10K samples. Results comparable to state-of-the-art are achieved with merely 30K data (10% of the Yelp training set). We speculate that the relatively weak performance with 10K samples is because the BART model uses a denoising autoencoding paradigm Lewis et al. (2020), which is trained to reconstruct the input sentence, and style strength of sentence rewriting is strongly affected in this low resource scenario.

Additionally, we conduct an ablation study on the bootstrapping step, and the result shows that with the same training sample size, the generation performance (considering both style accuracy and content preservation) obtained significant improvement by adding the bootstrapping learning stage (see Appendix Table 15).