Product Title Generation for Conversational Systems using BERT

Smartphones and voice-activated smart speakers, such as Amazon Alexa, Google Home, and Apple Siri, have led to increased adoption of voice-enabled shopping experiences. In such voice-enabled shopping experiences, reducing user friction and saving time is key, especially for low consideration purchases and repeat grocery purchases. Display-based experiences typically utilize a short product title when presenting a product, but these short titles do not naturally fit in a typical conversational flow. For example, for a display-based experience showing “Jergens Natural Glow Daily Moisturizer, Medium to Tan, 7.5 oz” as a short title of the product may be okay but it is not suitable for voice-based applications as it is not a naturally spoken title. At the same time, display-based experiences have the added benefit of being able to show a lot of additional meta-data which is not possible in conversational systems. The product title for a conversational system needs to encapsulate the important information in a succinct, grammatically correct natural language sentence, such that it naturally fits with the conversational flow during the dialogue rounds between the user and the conversational system. E-commerce companies can have millions to billions of products in their ever-changing product catalogs, so it is typically prohibitively expensive to manually annotate naturally spoken titles for all products. In such industry settings, it is ideal to have a model that can generate naturally spoken titles for an evolving catalog. The primary goal of this work is to examine the methods and challenges to convert short titles of products into a naturally spoken language that is grammatically correct.

This problem is similar to the text summarization task which is well studied in natural language processing (NLP). However, using it for e-commerce application is not a trivial task. Text summarization approaches can be classified into two broad sub-categories, extractive text summarization and abstractive text summarization. Extractive text summarization-based approaches usually try to extract a few sentences (or keywords) from lengthy documents [Dorr et al., 2003, Neto et al., 2002, Nallapati et al., 2016b]. To generate natural language titles from web titles, model should have capability to generate conjunctions, articles etc. at the appropriate position.

Abstractive text summarization attempts to understand the content of the document and produce summaries which may contain novel words or phrases. Recurrent Neural Networks(RNN)  [Hochreiter and Schmidhuber, 1997, Cho et al., 2014] based sequence to sequence (seq2seq) models [Sutskever et al., 2014] or recently developed attention models  [Vaswani et al., 2017] have been shown to perform well on abstractive summarization tasks  [Nallapati et al., 2016a]. However these models tend to generate repetitive words and this can cause negative customer experience in voice e-commerce applications. For traditional text summarization tasks we can introduce novel words which are not part of the input sequence but in an e-commerce setting paraphrasing of factual details like brand, quantity, etc. may imply a different product. In addition to this, new products are added to e-commerce product catalogs continuously, which can introduce out-of-vocabulary words. The summarization model should be able to generalize to these words.

In this paper, we investigate application of text summarization techniques to voice based e-commerce application. Our major contributions are

1. 1.

   We adapt different state-of-the-art NLP models to a real-world e-commerce dataset with limited labels.

2. 2.

   We perform extensive evaluation of these models on established evaluation metrics, as well metrics relevant to our application. We also perform human judgement evaluation.

The following sections provides a summary of related work followed by a description of methods applied to convert web-based short titles of products (sequence of words in English) into more naturally spoken summary titles (sequence of words in English) for voice-based applications. In our problem setting, we are more interested in building an abstractive text summarization based model that can generate novel words in the decoded summary. Section 4 provides the salient features of the dataset and implementation details of the methods described earlier. This is followed by a discussion of results observed and conclusion.

Text summarization is a long-studied problem in natural language processing. With the advent of deep learning based approaches, seq2seq models have proven highly successful in abstractive text summarization. Some of these models and relevant developments in the field are mentioned below.

Ptr-Net [See et al., 2017] develops on the seq2seq model with attention for the summarization task. It uses the concept of the pointer network introduced in Vinyals et al. [Vinyals et al., 2015] to decide which words from the main text should directly be copied to the summary. This helps to preserve the important factual information from the input text and also assists in handling out-of-vocabulary words. Ptr-Net model also adds coverage loss, which examines the difference between the attentions of previous words generated and the current attention, in an attempt to fix the issue of word repetition, a persistent issue in seq2seq models. Gehrmann et al. [Gehrmann et al., 2018] try to improve the fluency of the generated text through various constraints applied during model training. Soft constraints on the size of text are used to constrain the length of generated descriptions, while constraints on the output probability distribution of words ameliorates word repetition.

Development in language models have subsequently led to increased use of pre-trained models such as BERT [Devlin et al., 2018] and GPT-2 [Radford et al., 2019], which are trained on huge text corpuses and are used to generate the word embeddings for input texts. While Khandelwal et al. [Khandelwal et al., 2019] examines the feasibility of pre-trained language models in a low-data setting and moves away from a seq2seq framework, the work of Liu and Lapata [Liu and Lapata, 2019] use BERT in a seq2seq model to summarize data. Details of this model are discussed further in Section3. In our study, this is the primary model adapted for our use-case.

Text summarizarion finds natural application in e-commerce where products may have a long description but only the salient features of a particular product is what the end user is in interested in. Increased interaction with mobile devices and voice based interfaces such as Amazon Alexa and Google Home present new challenges as the product title now need to be succinct. There has been development in rule based methods as in Camargo de Souza et al. [Camargo de Souza et al., 2018]. Deep learning based methods find natural application and attempt to use multi-model information (images and text) to generate product title. Chen et al. [Chen et al., 2019] generates personalized product titles utilizing user personas and an external knowledge base. Mathur et al. [Mathur et al., 2018] attempts at generating titles in different languages for the same product. Sun et al. [Sun et al., 2018] develops further on the work of Ptr-Net using a separate encoder network for important attributes like quantity, brand of products and then then uses 2 pointers to decide where to copy data from. The work of Zhang et al. [Zhang et al., 2019] attempts a novel method of generating short descriptions for e-commerce use case and uses multimodal information. A word sequence symbolizing description and the image of a product in the catalog are chosen as descriptors for the product and are used to generate short titles for the product. An adversarial network based approach is then used to decide if the generated title is machine or human generated to improve the quality of generated titles and make it more human like.

In this section, we formulate the problem of automatic natural language title generation and discuss various approaches that can be used to solve this problem.

Problem Definition: The goal of this task is to build a system that can automatically generate natural language product titles which are easily interpretable in a voice-enabled shopping experience. Given the short web-title $w$ represented as a sequence of words $w=\{w_{1},w_{2},....w_{n}\}$, the goal of this system is to generate the corresponding natural language title $y=\{y_{1},y_{2},....y_{m}\}$.

In the following sub-sections, we discuss various models which we apply for the automatic natural language title generation task.

Table 3 provides a summary of model performance on different evaluation metrics. We observe that transformer and BERT have ROUGE-1 F1 scores of $0.92715$ and $0.92298$ respectively, outperform seq2seq-based approaches in terms of both ROUGE and avg. # duplicates metrics. The Ptr-Net model outperforms basic seq2seq+attention approach, however, paradoxically Ptr-Net with coverage underperforms compared to Ptr-Net model. The coverage loss is present to specifically address the issue of repetition of words and instead of fixing it, lead to an increase in the avg.# of duplicates to $0.3041$ from $0.239839$. While transformer model does have a better ROUGE score than BERT model, BERT has lower repeated words in output ($0.17262$ compared to $0.18795$) which has a greater impact on the title qualitatively. Since, BERT is pre-trained on a large corpus, it should be able to generalize better especially in low data scenarios. Human evaluation of generated titles reinforce this, showing that BERT performs better than transformer based model on output quality.

Table 4 and Table 5 lists examples of BERT Abstractive Model prediction on the test dataset where the model performs well and poorly, respectively. The corresponding ground truth and web titles have also been provided for comparison. The model seems to have repeated words in certain cases, for example Row 2 of Table 5.Given that most of the data has been trained on products with ounce and pound as the units of measurement, it can be seen that liter is incorrectly converted to fluid ounce by the model in Row 3 of Table 5 and pound is added to a single bulb of garlic in Row 4 of Table 5. However, it can be observed that in some cases, the model clearly does better than ground truth evaluation and even fixes the incorrect quantities in ground truth (rows 3-6 in Table 4). The model is able to add important attributes like frosted in the case of row 4 of Table 4. From row 2 of Table 4 we can see that the model is able to maintain the brand name as it is like great value and provide the correct measurements units for the different products like 6 count or pack of 6. Thus the model is able to fulfill requirements of preserving quantity and brand between web and voice titles.

We observe that overall BERT based model performs better both quantitatively and qualitatively in maintaining factual details in the output title and also reducing repeated words in the output.

In this paper, we studied the problem of generating succinct, grammatically correct voice titles for products in a large e-commerce catalog with limited labels. We evaluate 4 different baselines and demonstrate that BERT summarization can generate good titles through ROUGE metrics and human evaluation, even when there is extremely limited data. Generating personalized titles for different user segments based on rich user metadata and incorporating web data with additional product attributes that may be product dependent are some directions to extend this work.