Discriminative Pre-training for Low Resource Title Compression in Conversational Grocery

Our problem can be classified under the broad category of text summarization. This area has evolved quite a bit with the advent of RNNs. (Nenkova and McKeown, 2012; Dalal and Malik, 2013; Allahyari et al., 2017; Gambhir and Gupta, 2017) present comprehensive surveys of neural as well as classical text summarization techniques. While document level summarization (Cheng and Lapata, 2016; Bing et al., 2015; Nallapati et al., 2017) deals with the problem of generating document level summaries of the content, our work is best described as sentence compression (Wang et al., 2013; Viet et al., 2017; Chopra et al., 2016; Miao and Blunsom, 2016; Zhao et al., 2018; Févry and Phang, 2018; Filippova et al., 2015) which involves summarizing long sentences to shorter ones while preserving the core intent.

There are 2 distinct flavors of summarization independent of the source granularity; Abstractive and Extractive. Abstractive summarization (Chopra et al., 2016; Nallapati et al., 2016; Shi et al., 2018; Bing et al., 2015) can produce summaries consisting of words not present in the input, while Extractive summarization (Filippova et al., 2015; Févry and Phang, 2018; Zhao et al., 2018; Viet et al., 2017) aims to generate summaries using words or sentences extracted from the original input. We will focus on neural approaches for extractive flavor of sentence compression for the remainder of this section.

A popular approach to extractive sentence compression is to model the problem as a sequence-to-sequence learning problem using an encoder-decoder based architecture (Filippova et al., 2015; Févry and Phang, 2018; Miao and Blunsom, 2016). The encoder learns a distributed representation of the input sentence which is then fed to the decoder which is trained to produce a binary 0/1 label for each input word denoting whether to delete or keep that word in the output summary. Beam search can be used (Filippova et al., 2015; Wang et al., 2013; Wiseman and Rush, 2016) to sample from the decoder output probability distribution to generate the most likely compression.

(Viet et al., 2017) and (Zhao et al., 2018) present 2 approaches different from the encoder-decoder based approaches. (Zhao et al., 2018) uses Reinforcement Learning with KEEP and DELETE policies. Words are kept or deleted based on the policy while a pre trained language model provides feedback on the generated summary. This results in the system learning the optimal policy over time. Like all the other deletion based compression models, (Viet et al., 2017) still tags words in the input sequence with a 0/1 label, but treats it as a sequence labeling problem, instead of a sequence generation problem. Hence it does away with the decoder and employs stacked BiLSTM layers with a final CRF layer at the output that does the 0/1 classification. Our current work closely follows this approach with minor modifications.

Most of the techniques discussed here, with the exception of (Viet et al., 2017), require a considerable amount of training data. While (Miao and Blunsom, 2016; Févry and Phang, 2018; Zhao et al., 2018) are trained on the Gigaword corpus, (Filippova et al., 2015) is trained on 2 Million news headline and summary pairs. Though these 2 Million examples were synthetically generated following the method in (Filippova and Altun, 2013), it is based on the syntactic structure of the English language and its parse trees. (Févry and Phang, 2018) also proposes an unsupervised approach involving generation of training data by adding noise, but the results lag behind supervised approaches. Since the distribution of the words in the product titles, and in general the vocabulary of our problem domain, is different from general English, data sparsity and unavailability of e-commerce specific embeddings pose a challenge for us. Interestingly, (Viet et al., 2017) reported good performance with only 10000 pairs of original and compressed sentences, which is the reason we chose to implement a modified version of this approach to solve our problem of product title compression.

Product title summarization in the context of voice and mobile shopping has been studied in (Wang et al., 2018; Sun et al., 2018; Zhang et al., 2018; Xiao and Munro, 2019; Gong et al., 2019). Amongst them, (Gong et al., 2019) defined title summarization as a sequence classification problem, where a binary decision is made at each word, given a long product title. A feature vector has been applied, comprising term frequency (tf) and inverse document frequency (idf) for each word. However, in this approach, large training data (500,000 samples) has been used and the results didn’t prove to be significantly better than a simple BiLSTM approach. Another interesting approach has been proposed, where the problem has been defined as multi-task learning objective (Wang et al., 2018) to compress the product title using user search log data. The multi-task learning objective involves two networks, one network to select the most informative words from the product title as a compressed title and the other network to generate the user search query from the product title. These two networks have been constrained to share the same product title encoder and additionally, the attention distributions from these two networks were constrained to agree with each other. However, in this approach, the size of the training data is large (185,386 samples) and additionally requires user search data. Another recent approach has been proposed in which the product title summarization has been framed as a Binary Named Entity Recognition problem (Xiao and Munro, 2019). The model architecture involves a simple bi-directional LSTM encoder/decoder network (2 layer LSTMs) with an attention mechanism. ANOVA and post-hoc tests showed that with this approach, there was no statistical difference between model outputs and human-labeled short titles.

Intuitively, higher the quality of the replaced token, harder it is for the network to guess that it is replaced, and hence better is the latent representations learnt during the pre-training process. Unlike (Clark et al., 2020) which samples the replacement tokens from a small masked language model trained jointly, but not adversarially, with the network, we obtain our replacement tokens from a skip gram model that maximizes the log likelihood in a window centered on the token to be replaced. This allows us to improve our compute efficiency while obtaining reasonably good candidate replacement tokens. If $w_{i}$ is the token at the $i_{th}$ position that is to be replaced, then we choose $w_{r}$, the replacement token as

where $V$ is the vocabulary, $V^{{}^{\prime}}=V-\{w_{i+k};-n\leq k\leq+n\}$, $l_{w}=2n+1$ is the window size, and $P_{s}(w_{i}|w_{j};l_{w})$ is the conditional distribution for occurrence of $w_{i}$ in a window of length $l_{w}$ centered on $w_{j}$.

We minimally normalize our datasets by converting everything to lower case and squeezing consecutive white space characters. Given the nature of the Walmart catalog data, 2 additional normalization steps were necessary; converting all “&” symbols to “and” and padding commas with whitespaces so that they are treated as separate tokens. We follow a simple tokenization scheme where we split each product title by whitespace. We keep the maximum sequence length at 35, truncating longer product titles and padding shorter titles with the PAD token. The maximum token length we use is 15, with similar padding and truncating scheme. We extract our word and character vocabularies from the product titles dataset instead of the title summary dataset since the former is a super set of the later. Our corpus has a word vocabulary size of 67634 including the “UNK” and “PAD” tokens, while the character vocabulary size stands at 69. We set aside 20% of the human generated title summary dataset as our test set, 8% as validation set and the remaining 72% as our training set.

We perform several ablations, as well as some additions, on the model proposed in section 4 to underscore the effect of each of the components. We name our model CB3SA, choosing to retain the first character for each layer (CharCNN, BiLSTM(3 stacks) and self attention). Additions and ablations are denoted with a + and - operator respectively after the model name. For example CB3SA+PT refers to the pre-trained version of our model, while CB3SA-CharCNN refers to the version of our model with the character level word embedding layer removed. Table 3 lists the various combinations we tried and the results on the test set.

Clearly, the pre-training process (model CB3SA+PT) contributed a significant boost in performance over the non pretrained model (CB3SA) while removing the charCNN layer (CB3SA-CharCNN) causes the most significant drop in performance. Apart from these, there are a few interesting observations. Firstly, removing a BiLSTM layer (model CB3SA-BLSTM1) has one of the least negative impacts together with reduced training and inference times. Secondly, removing the vanilla self attention layer and adding a multi-headed self attention layer with 8 attention heads (model CB3SA-SA+MHSA8) proves to be as detrimental as removing the self attention layer altogether (model CB3SA-SA). Finally, using a narrow width attention with window length of 7 instead of our global self attention (model CB3SA-SA+NWSA7) causes the least drop in performance.

We fine tune our pre-trained model (CB3SA+PT) on the entire training dataset and compare it’s performance against several strong baselines; XLNet, BERT, RoBERTa, DistillBERT, and ELECTRA - all fine tuned on the same training set. Table 4 lists the results. Additionally, the F1 column shows the percentage by which our F1 lags the respective models and the Params column shows the factor by which our parameter set is smaller.

Our model is able to use 55X lesser parameters than BERT and yet obtain a comparable performance, lagging BERT Base by 2.78% of F1 score on the test set. Among the SOTA models, ELECTRA-Small seems to provide the best parameter efficiency by uisng 7.8X lesser parameters than BERT-Base while lagging by only 1.3% on F1 score on the test set. Interestingly, RoBERTa, the biggest model that we tried with 125M parameters, under performed by a huge margin, lagging our F1 score by 11.96%.

We experiment in a low resource setting by allowing all the models to be fine tuned only on a small fraction of the training data and then evaluating their performance on the same test dataset as was used in Section 6.3. Figure 2 compares the performance of our pre trained model with that of the SOTA models which shows our model’s performance deteriorates much lesser compared to others. When allowed to be fine tuned only on 5% of the training data (around 1600 parallel examples) our model outperforms the best performing SOTA model, BERT-Base, by 24.3%. In fact, our model continues outperforming all SOTA models for a large part of the low data regime, till 20%, after which others catch up.