OA-Mine: Open-World Attribute Mining for E-Commerce Products with Weak Supervision

Product attributes are essential for online shoppers, as they provide valuable information to help search, recommendation, and product comparison. The massive, constantly changing products bring the open-world challenge. Products we already know may gain new attributes over time – “HDR compatibility” barely comes to mind when people buy a TV ten years ago, but is relevant for many shoppers nowadays. Not to mention, new types of products with distinctively new sets of attributes may emerge in the future. Automatic extraction of product attributes has to meet the open-world challenge. The model must be able to discover both attribute types (e.g., brand, referred to as “attributes” hereafter) and attribute values (e.g., Samsung, referred to as “values”). Moreover, they must operate with limited supervision, as human annotation can never keep up with the forever-expanding product catalog.

Existing works on attribute mining mostly carry a closed-world assumption on attributes. Pioneer studies such as OpenTag (Zheng et al., 2018) bear “open” in the name, but their open-world assumption is on values only. They assume the set of attributes of interest is given as input to the model. The model is trained to find more values for the given attributes, but does not expand to unseen attributes without additional training data. Moreover, they usually rely on extensive human labeling (Zheng et al., 2018) or noisy distantly supervised data (Xu et al., 2019b; Yan et al., 2021; Wang et al., 2020).

In this paper, we study the open-world attribute mining problem with weak supervision. We let the user provide a few example values for a few known attributes. Neither the attributes nor the values given as examples are required to be comprehensive. In addition, we also assume the type of the products are known a priori, as each product type holds a distinct set of applicable attributes (e.g., “resolution” applies to TVs but not shoes), and the attribute mining process should be done with respect to each product type. With limited supervision and abundant product raw text, we aim to discover new attributes and values for products of different types.

Product titles, as the cleanest and most prominent part of product text, possess a number of unique properties. As shown in Figure 1, product titles are concise, non-repeating, not written in a way as a general domain natural language. More specifically, we highlight three observations from the data. First, to maximize exposure of products to customers, sellers usually pack the highlights of their product in the title (observation 1: title first). Second, a product title rarely contains irrelevant information, and is a collection of attribute values (observation 2: bag-of-values). Third, with limited space in the title, the values seldom repeat (observation 3: value exclusiveness). For example, if one word describes the resolution of a TV product, the rest of the title will mention other attributes such as screen size. These observations pave the way for a mining-based approach that exploits the concise product titles, is tailored to the language of products, and discovers new attributes and values in an open-world setting.

We propose OA-Mine, a principled framework for Open-world product Attribute MIN(E)ing with weak supervision. Our framework has two main steps, candidate value generation and attribute value grouping. The first step of our framework is designed based on the title first and the bag-of-values data observations. It probes an in-domain fine-tuned language model that captures compositional relations between words, and segments product titles into phrases. These phrases are candidates that will either be marked as an attribute value or be excluded as noise in our subsequent step.

Once we obtain the attribute value candidates, our framework finds the major attributes for each product type, and puts the candidates into different attribute groups. We rely on a proper value embedding to propagate supervisions from seed values to other values when they together form a clique, and also to discover new values when new cliques emerge. We find that pre-trained language model embedding (e.g., BERT embedding) cannot fully capture attribute-specific information, even after MLM (Devlin et al., 2019) pre-training using in-domain corpus, as seen in Figure 2. We propose an attribute-aware fine-tuning method, which takes full advantage of value exclusiveness of product titles as well as the weak supervision, and optimizes a multitask objective function. After the fine-tuning, embedding becomes more expressive in distinguishing values from different attributes. With well-tuned embedding, we then apply a self-ensemble-based inference method. A density-based clustering component, which can discover open-world new attributes and scrap noise from the candidates, is joint force with a classification component, which improves the model recall by finding more values of the discovered attributes. Our OA-Mine framework improves itself with iterative training, where the discovered attribute types and values help gradually shape the value space for each attribute, as the model becomes more knowledgeable.

We run extensive experiments on over 750K products from 100 product types. In addition to distantly annotated development data, we recruit crowd workers and domain experts to label a gold standard test set for end-to-end evaluation. Our experiments show the superiority of our framework over various state-of-the-art baseline methods. Besides, we also show the model’s ability to discover unseen attributes by holding out attributes from training, and demonstrate the model’s strong generalization by evaluating on product types with zero supervision.

Our main contributions can be summarized as follows:

• •

  We propose a principled framework for open-world product attribute mining. The framework discovers both new attributes types and new attribute values.

• •

  We propose a novel attribute-aware representation fine-tuning framework for pre-trained language models. The framework combines human given weak supervision with abundant unlabeled product text, and adapts the language model to distinguish values from different product attributes.

• •

  We release a human annotated evaluation dataset for end-to-end evaluation of the open-world product attribute mining task.

In this section, we formally define the open-world product attribute mining task, and specify our input and output. In addition, we offer some additional insights through data analysis.

The problem is defined for a specific type of product $t$, because each product type has its unique sets of applicable attributes $\mathcal{A}_{t}$. Product types are identified by their surface names. Putting all types of products together $\mathcal{P}=\bigcup_{t\in T}\mathcal{P}_{t}$, the open-world attribute mining aims to discover all the attributes $\mathcal{A}_{t}$ for $t\in T$.

In this paper, we use product titles from $\mathcal{P}$, and we assume the number of products $\mathinner{\!\left\lvert\mathcal{P}\right\rvert}$ is large.

We would like to know what percentage of attribute values are already presented in the structured information of those online product profiles. In total, the human annotators found 51 distinct attributes from all the products across 10 product types. Of those, only 30 attributes are found in existing structured product profiles. Figure 3 shows the percentage of values identified by humans compared to the structured attribute information in product profiles. Rows represent product types, and columns represent attributes. Comparing different rows, we can see different types of products have different applicable attributes. Judged by the color of the cells, lots of values are apparently missing. The data analysis clearly shows the need for open-world attribute mining, as we see missing attributes and values in the existing product profiles.

OA-Mine is a two-step framework for weakly supervised open-world product attribute mining (Figure 4), with candidate value generation followed by attribute value grouping. The first step aims to harvest phrases that are likely attribute values from raw text product titles. Data observations from Section 1 show that product titles are collections of attribute values. Our model segments the titles into phrases, essentially breaking down bags-of-attribute-values into individual value candidates. Established phrase mining tools cannot meet our needs, as pre-trained NLP pipelines (Honnibal et al., 2020; Akbik et al., 2019) are not adapted to the product text, while the unsupervised methods (Gu et al., 2021) typically has assumptions that do not work well on the product titles (shown in Section 6.2). We designed a language model probing-based technique, which captures the compositional relations between words for effective phrase segmentation.

Although product titles are primarily bags-of-attribute-values, there are exceptions where things that are not attribute values can be mentioned (e.g., “packaging may vary”). In our first step, we leave them as candidate values and let our second step remove them from the candidate pool. We design our first step to be recall-focused and the second step to be noise-aware, as information lost from the first step cannot be recovered later, while noise can be effectively handled.

Having obtained the attribute value candidates, we find novel attributes and put the candidates into attribute groups in our next step. Our model navigates the embedding feature space of candidate values and discovers dense areas that are likely attribute clusters. Fundamentally, the embedding of candidate values has to be attribute-aware, a requirement we have shown that pre-trained language model embedding cannot satisfy. To achieve this, we draw on the value exclusiveness observation from data and combine it effectively with the limited supervision we have from seed values. We optimize a multitask objective, with a binary meta-classification loss to generalize across product types, a contrastive loss to enforce embedding similarity resembles attribute-level similarity, and a classification objective to impose attribute distinctiveness.

Once we have well-trained embedding, a self-ensemble of different parts of the model first discovers new attributes and removes noise by DBSCAN (Ester et al., 1996) clustering, and then puts more values into attribute groups with classification. Finally, the model improves itself by leveraging the prediction for iterative training.

The first step of our framework aims to generate attribute value candidates from product titles. As product titles are mostly bags-of-attribute-values, we formalize the task as a product title segmentation task. For example, for the product in Figure 1, we would like to segment its title into “TCL — 50-inch — 5-Series — 4K UHD —…” Our goal of this step is to include as many value candidates as possible, i.e., it is recall-focused. We allow phrases that are not attribute values to be kept, but do not want actual attribute values to be lost.

We leverage a pre-trained language model for candidate generation. As observed in prior work (Kim et al., 2020; Gu et al., 2021; Wu et al., 2020), pre-trained language models can capture phrases through the attention scores computed by the Transformer model.

We first fine-tune a BERT (Devlin et al., 2019) language model using the Masked Language Model (MLM) (Devlin et al., 2019) objective on the product text. This step is necessary because the product text differs from the general domain natural language text on which BERT is pre-trained. Then we probe the BERT model to generate a score that resembles how likely two words are from the same phrase.

$W/\{\cdot\}$ denotes the original text with words in the set under slash replaced with [MASK]. $\mathrm{BERT}(\cdot)$ computes the sequence embedding by applying the BERT model, and $(\cdot)_{i}$ takes the i-th embedding, i.e., the token embedding of $w_{i}$. $d(\cdot,\cdot)$ is a distance function. We use cosine distance in our model.

One drawback of relying entirely on the language model is that it does not guarantee the completeness of the phrases. For example, “QLED TV” may be combined into one phrase in one product, but separated into two words in another product. This is because the BERT representation is context-dependent, and cannot leverage global frequency signals from the whole corpus. We, therefore, apply frequent pattern mining to combine words that frequently co-occur in the corpus, which further improves the completeness of the attribute value candidates.

Once we have generated candidate values from product titles, we are ready to group them into attribute clusters. Given the open-world challenge in our problem setting, standard token classification approaches cannot suffice. Fully unsupervised clustering methods also fall short, as they rely on the pre-trained representation of phrases, which is not attribute-aware (Figure 2).

We identify two main requirements for discovering new attributes and values: (1) The candidate value representation must be attribute-aware, placing values from the same attribute close to each other, and those from different attributes far apart; (2) The model must generalize to unseen attributes and to various product types. Bearing these two requirements in mind, we first design a multitask fine-tuning approach that focuses on attribute awareness and generalization. Then we apply a self-ensemble inference to discover new attributes and values.

The major different between our work and the existing works is that we approach the problem with open-world assumptions on both attributes and values, while prior works assume the attributes of interest are given as input. For supervision, we adopt weak supervision, which is a small amount of high quality data, instead of distant supervision, where a large amount of noisy data is used.

Our problem setting is more challenging than open-world text classification, as we have to extract attribute values from raw text. In terms of methodology, our proposed model can take advantage of unique characteristics of the product text, and can learn better representation. As seen in our experiments, the adaptive clustering-based methods do not work well out-of-the-box, as the product text is noisy and leads to poor convergence of adaptive clustering.

In this paper, we proposed a principled framework for open-world product attribute mining. The framework includes a novel candidate generation method, an attribute-aware representation fine-tuning model, and achieves open-world inference through self-ensemble. Our framework offers strong performance and generalize to unseen attributes and product types. In the future, we will extend our model to handle additional text fields besides product titles, such as using product introductions. Improving the classifier to make open-world inference is also a promising direction.