Heterogeneous Network Embedding for Deep Semantic Relevance Match in E-commerce Search

In a real-world search application, queries and items and their relationships naturally form a heterogeneous bipartite network based on the multi-type user behaviors like view, click, and purchase. However, as mentioned earlier, user behaviors are typically biased and noisy. So if one directly conducted embedding learning on the original user behavior network, it would be difficult for the model to estimate explicit relevance relationships. To solve this problem, we introduce external knowledge to refine the user behavior network and then construct a knowledge-enhanced heterogeneous network. Here the external knowledge is provided by the BERT model that is pre-trained on a large text corpus and then fine-tuned on some in-house data. The whole network construction includes the following phrases.

Fine-tuning BERT. Transformer-based models such as BERT and ERNIE (Sun et al. 2019) have been preferably used as NLP benchmark in recent years. Here we use the BERT-Base model²²2https://github.com/google-research/bert composed of stacked Transformer units and fine-tune it on some in-house data. The positive and negative examples in the data are human-labeled and cover various categories of items. The fine-tuned BERT is equipped with a high relevance discrimination ability and thus can act as an expert on filtering noisy data.

Behavior network formation. The user behavior network in the left network of Fig. 2 is built on some user search log (over several months) which records user clicks and purchase behaviors as well as their frequencies. In this network, an edge represents an existing click behavior or purchase behavior between a query-node and an item-node. The edge of purchase behavior is sparse but important. The edge of click behavior is denser but highly noisy, making it difficult to learn a good model and predict reasonable results. Therefore, we introduce BERT to refine this network.

Knowledge-enhanced network refinement. Given the scarcity of user purchase feedback and noisiness of click feedback, we retain all the original purchase behavior edges while use the external knowledge from BERT to refine the click behaviors. Specifically, we set two different thresholds $\alpha$ and $\beta$, i.e. $\alpha$ is used for judging whether a clicked item is truly relevant to its query, and $\beta$ is used for judging whether an unclicked item is truly irrelevant to its query. This strategy can help remove noises in user behaviors and at the same time extract the missing but crucial relevance signals not captured by user behaviors. To preserve the high-quality neighbor set, for each central node, we rank its neighboring nodes with the priority of “purchase$\rightarrow$high click$\rightarrow$low click” and sample top-2 of them as the final neighbor set.

In HG4SM, the basic units are query nodes, item nodes and the refined user-behavior-oriented edges between them. Based on this schema, we employ two metapaths, “$Q$-$I$-$Q$” and “$I$-$Q$-$I$”, where “$Q$” and “$I$” stand for “$Query$” and “$Item$”. These two metapaths correspond closely to the second-order relevance definition defined earlier. Compared to some more complex metapaths (such as “$Q$-$I$-$Q$-$I$-$Q$” and “$I$-$Q$-$I$-$Q$-$I$”), the adopted metapaths in our model is both effective (with less information density loss) and computationally efficient. We further choose two instances for each metapath whenever they are available and pad with zero embeddings otherwise . Take “$Q$-$I$-$Q$” as an example, its instances can include “$Q$-$I_{\mathrm{top1}}$-$Q_{\mathrm{{top1}}}$” and “$Q$-$I_{\mathrm{{top1}}}$-$Q_{\mathrm{{top2}}}$”, as shown in Fig.2.

The whole framework of the new HG4SM model includes both the first-order relevance modeling and second-order relevance modeling. We first introduce the first-order relevance modeling which captures macro and micro semantic match signals by incorporating both the representation-focused and interaction-focused designs.

Word embedding in e-commerce scene. In e-commerce applications, it is infeasible to represent queries and items as individual embeddings since their entity space is effectively unbounded. Instead, we adopt word embedding in HG4SM, which dramatically reduces the representation space and deals well with the cold-start situation. Numerous product type names or attribute names (like “iphone11” and “256GB”) exist, but the basic word segmentation based on N-gram or WordPiece word segmentation splits these special names and thus cannot reveal their complete semantics. To adapt to this feature, we first treat single Chinese characters, contiguous numerals or English letters as single words (Fig. 3). We then retain only the high-frequency words in this list. Compared to potentially million queries and billion items, our vocabularies are only in the tens of thousands, which saves memory consumption and lookup time of id and embedding by a large margin. We represent each word with a $d$-dimensional vector and train these word vectors or embeddings using Word2Vec (Mikolov et al. 2013). We use $\bm{E}_{Q}^{i}$ (and $\bm{E}_{I}^{i}$) to denote the $i$-th word’s embedding of query $Q$ (and the item’s title $I$).

Macro matching element. Most of the first-order relevance methods are either representation-focused (capturing macro matching signals) or interaction-focused (capturing micro matching signals). We employ their mixture in the HG4SM’s first-order relevance modeling. For the representation-focused part, take query $Q$ with $l_{Q}$ words as an example, its sequence embedding $\bm{E}_{seq}^{Q}$ is obtained by calculating the element-wise mean value of $\left\{\bm{E}_{Q}^{i}\right\}$:

$$\bm{E}_{seq}^{Q}=\frac{1}{l_{Q}}\sum_{i=1}^{l_{Q}}\bm{E}_{Q}^{i}$$

(3)

The above $\bm{E}_{seq}^{Q}$ depicts the whole semantic information of query $Q$, so that it can be viewed as a simplified version of representation-focused embedding.

Micro matching element. To capture micro matching signals, we need to model the word-level interaction information (Pang et al. 2016a) between queries and titles. Suppose the sequence lengths of query $Q$ and title $I$ are $l_{Q}$ and $l_{I}$, respectively, we build an interaction matrix $\bm{M}_{int}$:

$$\bm{M}_{int}=[\bm{E}_{Q}^{1},...,\bm{E}_{Q}^{l_{Q}}]^{\mathrm{T}}\times[\bm{E}_{I}^{1},...,\bm{E}_{I}^{l_{I}}]$$

(4)

The interaction embedding is derived by reshaping $\bm{M}_{int}$ into a one-dimension vector:

$$\bm{E}_{int}=reshape(\bm{M}_{int})$$

(5)

Position encoding. Besides, considering the sequential structure of texts, we further add a position embedding to each word embedding before calculating the correlation matrix. The position embedding is set as a trainable embedding vector and has the same dimension as the word embedding.

Most semantic match methods focus on the first-order relevance, but ignore the second-order relevance (which integrates the neighbor information on metapaths into the central node and is essentially important in many real context-aware scenes). A complete semantic relevance estimation model should integrate them together. Here we consider how to generate second-order relevance embeddings so that it can incorporate context information in the network to enrich and improve the central nodes’ semantics. In general, the second-order relevance model consists of a node-level encoder and a metapath instance-level aggregator.

Node-level encoder. Metapath instance bridges the communication gap between heterogeneous nodes and thereby can infer the node’s global semantic embedding (rather than the local semantic embedding). For each metapath instance, to derive the global node semantics, we integrate the neighboring node embedding into the central node embedding with mean encoder. Take the instance “$Q$-$I_{\mathrm{{top1}}}$-$Q_{\mathrm{{top1}}}$” as an example, its corresponding embedding is:

$$\bm{E}_{Q-I_{\mathrm{{top1}}}-Q_{\mathrm{{top1}}}}=\mathrm{MEAN}\left\{\bm{E}_{seq}^{Q},\bm{E}_{seq}^{I_{\mathrm{top1}}},\bm{E}_{seq}^{Q_{\mathrm{top1}}}\right\}$$

(6)

Instance-level aggregator with graph attention. Different meatapath instances convey different information, so they should have different effects on the final metapath’s embedding. However, the mapping relationship between the metapath instance’s embedding and the metapath’s embedding is unknown. To learn their relationship automatically, we introduce a “graph attention” mechanism to the formation progress of the metapath’s embedding. The attention mechanism enables the model to learn a weight distribution and assign different weights to different components, which has been successfully applied in computer vision and natural language processing (Vaswani et al. 2017). Here we introduce graph attention to represent the mapping relationship between metapath and its instances. The final metapath’s embedding is then obtained by accumulating the embeddings of each metapath’s instances with attention scores. Take metapath “$Q$-$I$-$Q$” as an example, its corresponding embedding is defined as:

	$\displaystyle\bm{E}_{Q-I-Q}=$
	$\displaystyle\sigma(\mathrm{Att}_{1}\cdot\bm{E}_{Q-I_{\mathrm{{top1}}}-Q_{\mathrm{{top1}}}}+\mathrm{Att}_{2}\cdot\bm{E}_{Q-I_{\mathrm{{top1}}}-Q_{\mathrm{{top2}}}})$		(7)

where $\sigma(.)$ is the activation function. Though $\mathrm{Att}_{i}$ can be set as a fixed value, we adopt a more flexible way, i.e., we use a single-layer neural network to learn it automatically. Specifically, we feed the concatenation of embeddings of the central node and the metapath instances into a one-layer neural network and output an attention distribution in the softmax layer:

$$\bm{E}_{\mathrm{concat}}=\mathrm{CONCAT}(\bm{E}_{seq}^{Q},\bm{E}_{Q-I_{\mathrm{{top1}}}-Q_{\mathrm{{top1}}}})$$

(8)

$$\mathrm{Att}_{i}=\mathrm{softmax}(\sigma(W_{att}*\bm{E}_{\mathrm{concat}}+b))$$

(9)

where $W_{att}$ is a 1*2d trainable vector, shared across all metapaths.

Embedding fusion. Based on the first-order and second-order relevance modeling, three types of embeddings can be generated, including the representation- focused, interaction-focused and metapath-guided embeddings. To combine them, we concatenate these three embeddings together, feed it to the deep neural networks, and output a relevance score. To alleviate the model’s high time-cost, we use efficient three-layer DNNs without considering more complex neural network structures such as CNN and LSTM.

The HG4SM model is compared to several existing state-of-the-art semantic models. To facilitate their implementation, we use the open-source codebase packages which are as follows.
^†: Nine methods are implemented using MatchZoo (Fan et al. 2017), which is a toolkit for deep text matching, based on TensorFlow and Keras. We compare these typical methods (Guo et al. 2016; Pang et al. 2016b; Hu et al. 2014; Wan et al. 2016b; Mitra, Diaz, and Craswell 2017; Xiong et al. 2017) in MatchZoo with HG4SM. We use the default hyper-parameter setting for all the methods in MatchZoo³³3https://github.com/NTMC-Community/MatchZoo.
^§: Two other baselines DSSM (Huang et al. 2013) and ESIM (Chen et al. 2017) are also included in the comparison. DSSM is a classical representation-focused method using pairwise examples to train the model. ESIM is a successful interaction-focused method which incorporates the syntactic parsing information into chain LSTM for natural language inference.

To comprehensively measure the performance of our HG4SM and these baseline methods, we use six metrics, including: 1) Area Under the receiver operating characteristic Curve, 2) Accuracy, 3) Precision, 4) Recall, 5) F1-score and 6) False Negative Rate (since it is more serious than False Positive Rate for e-commerce ranking). They are respectively denoted by AUC, Acc, Pre, Recall, F1 and FNR for short. For the first five metrics, the higher the metric value, the better the model’s performance. For FNR, the lower its value, the better the model’s performance. Note that AUC often serves as the most important metric while the others also provide auxiliary supports for our analysis.

We compare HG4SM with eleven state-of-the-art deep semantic matching methods using the in-house e-commerce search log data. The results are shown in Table 1. Because some baseline methods (e.g., DRMM and ESIM) have relatively high time complexities, we sample ten million training data from the all-categories dataset. For fairness, all methods including our HG4SM are trained on these data.

As shown in Table 1, HG4SM nearly always outperforms the other methods compared, across all six metrics. More specifically,

• •

  Compared to the second best method, HG4SM obtains 1.8% (and 5.4%) gains under AUC in the mobile-phone- dataset (and all-categories dataset). Furthermore, HG4SM achieves the best (smallest) FNR on both these datasets. This implies that HG4SM has a high discrimination power on negative examples, so that it can return a list of satisfactory items which are relevant to the user’s shopping needs.

• •

  The collected training data have imbalanced classes (i.e. the positive examples are far more than the negative examples), making model learning a challenge. Most of the methods compared, such as DSSM, are vulnerable to the class imbalance and often fail to correctly estimate many testing examples in this case. Fortunately, our HG4SM learns explicit node semantics benefiting from the neighboring node’s effect, so that it has a robust performance even though the training data are highly imbalanced.

To further examine the importance of each component in the HG4SM model, we remove one or two components from HG4SM at a time and examine how the components affect the overall performance.We have the following empirical observation and analysis of the results, shown in Table 2:

• •

  In general, for the three submodels of HG4SM (including the representation-focused component, the interaction-focused component, and both), the both-component setting often outperforms each single-component setting but worse than the triple-component setting by introducing HIN (i.e. HG4SM). It demonstrates that introducing HIN helps the model learn comprehensive knowledge and thus gives an explicit estimation.

• •

  The introduction of second-order relevance modeling can provide stable improvement (e.g. 1.0%$\sim$1.6% of AUC gains) to every first-order solution. This demonstrates that applying HIN to the semantic match can effectively exploit the neighboring nodes’ information contained in user-behavior networks, benefiting the final relevance estimation between central nodes.

To further evaluate HG4SM’s performance in the real search scene, we deploy it to an online e-commerce search platform and report its A/B test results in Table 3. Four widely-used online business metrics are adopted: 1) Conversion Rate (CVR): Total order number / total click number; 2) User Conversion Rate (UCVR): Total order number / search user number; 3) UV-value: Total Gross Merchandise Volume / total user number; and 4) Revenue Per Mile (RPM): 1000 * search GMV / search number.

The results of A/B tests show that our HG4SM outperforms the existing online DNN model in this platform on all of the business metrics. For example, HG4SM improves 5.7% and 0.5% of UV-values under both price sort and default sort. It indicates that 1) our HG4SM model has a low time complexity, which makes it easy to cooperate with other online serial models, and 2) HG4SM provides accurate relevance estimation between queries and items, so that it provides users efficient and intelligent search experiences.