GLOW : Global Weighted Self-Attention Network for Web Search

The first-stage of document retrieval task can be described as, given one query q, the system produces a fixed amount of documents D from a mass of candidates. d represents one instance from D. Since in web search d is always equipped with multi fields contents. Let $\mathcal{F}_{d}=\{F_{1},F_{2},\cdots,F_{k}\}$ denote a set of fields associated with the document d.

For a single q, d pair, a deep matching model usually describes a matching score based on the representation of q and d.

where $\Phi$ is a model function to map each query and document to a representation vector, and $Func$ is the scoring function based on the similarity between them.

As show in Figure 1, to fit large scale document matching scenario, GLOW is built on a Siamese manner, we employ two GLOW Encoders to represent queries and documents respectively. From a horizontal view, GLOW is comprised of four parts, Word level global weight generation, Token level representation, Semantic level representation and Matching score. One prerequisite on data preparation before training is that we simply prepend a [CLS] (classification) token to both query and document. For document side, a [SEP] (separating) token is inserted across different fields to constitute the combined fields representation.

We introduce semantic level representation in Section 3.3 and 3.4, which takes GLOW Encoder by stacking 3 times. Then we describe how to generate the word level global weight in Section 3.5, what we adopt finally as global weight is BM25. The whole word weight sharing is described in Section 3.6 to explain how we map weight from whole words to WordPiece tokens. For the token level representation, we use the sum of token embeddings and position embeddings to form the token representation for query side. For document, we introduce the combined fields represention in Section 3.7, it adds field embeddings to differentiate multi-fields in documents. We adopt cosine similarity to describe the matching score. Section 3.8 explains how we optimize the matching score with training labels.

Let’s define $\mathbf{X}\in\mathbb{R}^{d\times T}$ is a $d$-dimensional sequence embedding input of one query or document with length of $T$, $x_{i}$ is the $i$th token in the sequence. $Q$,$K$ and $V$ are matrices initiated by $\mathbf{X}$ multiplying different weight matrices. The attention score matrix in such a sequence is denoted by $\mathbb{A}\in\mathbb{R}^{T\times T}$. For a token pair $x_{i}$, $x_{j}$, $q_{i}$ and $k_{j}$ are the column selection from $Q$ and $K$ according to $i$ or $j$, its attention score $A_{ij}$ is calculated in Scaled Dot-Product Attention as $A_{ij}=\frac{{q_{i}}\cdot{k_{j}^{T}}}{\sqrt{d}}$. In (AttentionisAllyouNeed, ), they claim the attention unit is already a weighted sum of values, where the weight assigned to each value is learned from $q_{i}$ and $k_{j}$. Whereas, in information retrieval area, it is well equipped with prior knowledge to represent the weight of one word. We enrich the attention calculation with these techniques. Assuming $w_{i}\in\mathbb{R}^{T}$ represents the global weight of the $i$th token in the sequence, $w_{i}$ is a non-trainable scalar. In this paper we use BM25 to represent this global importance. Thus a new weighted attention score $A_{ij}^{w}$ is computed as

where $q$ and $k$ share the same shape and only differ in random initialization. Symmetrically, the weighted attention score of $A_{ji}$ can be represented in the right part of Eq. 2. The right part of Figure 1 presents how Weighted Self-Attention works. With importing the weight information and packing all $w_{i}$ into $W$, we formally define Global Weighted Self-Attention (GWSA) as

where $W$ is one dimension vector and its multiplicand is a matrix . $\odot$ represents a Hadamard product by repeating $W$ to perform element-wise multiplication. Eq 4 explains this special operation.

GLOW Encoder picks this Weighted Self-Attention as its block unit. It is also built upon multi-head structure by concatenating several Weighted Self-Attention instances. With re-scaling by $W^{o}$, we can get a Complex Weighted Self-Attention (CWSA). A fully connected Feed-Forward network is then followed as the other sub-layer. In both sub-layers, layer normalization(layerNorm, ) and residual connection(resnet, ) are employed to facilitate the robustness of GLOW Encoder.

We formulate the whole encoding logic in Algorithm 1.

A key point of GLOW is to find an appropriate way to represent global weight. BM25 and its variants show the superiority in global weight representation for document ranking tasks against other alternatives. We leverage BM25 to generate the global weight scores for a query and BM25F(Robertson1994OkapiAT, ) to compute the weight scores for a multi-fields document. BM25F is a modification of BM25 in which the document is considered to be composed from several fields with different degrees of importance in term of relevance saturation and length normalization. Both BM25 and BM25F depend on $tf$ and $idf$, $tf$ means TermFrequency, it describes the number of occurrences of the word in the field. While $idf$ (InverseDocFrequency) is a measure of how much information the word provides, i.e., if it’s common or rare across all documents. It is the logarithmically scaled inverse fraction of the documents that contain the word. For word $i$, $idf_{i}={\rm log}\frac{N-df_{i}+0.5}{df_{i}+0.5}$, where $N$ is a scalar ¹¹1Here we set it by 100,000,000 indicting how many documents we are serving in system and $df$ is the number of documents where the word $i$ appears.

Sub-word based approach has been proved efficient to alleviate out of vocabulary issue and limit vocabulary size. BERT uses WordPiece to produce tokens from original raw text. One shortcoming of this methodology is that we cannot directly apply the word-level prior knowledge. Moreover, in some NLP tasks, token-level weight is not enough to distinguish the importance of different words. The latest BERT model has proved that upgrading the Mask Language Model task to whole word level²²2https://github.com/google-research/bert improves performance. In our task, the $W$ used for attention score is also based on whole word weights. That is, we first collect and calculate weights in whole word level, then give the same word weight to tokens corresponding to one word. By this way, one WordPiece token may has different weight representation if it occupies in different words. We also conduct experiment in Analysis section to compare the effect of token-level weight generation and word-level. The results suggest the word-level manner is superior than token-level.

In ad-hoc retrieval tasks, there are always multiple sources of textual description (fields) corresponding to one document. Lots of studies (neuralrankingmodelmultiplefields, ; bm25ForWeightedFields, ) reveal that different fields contain complementary information. Thus, to obtain a more comprehensive understanding of document, when encoding the document into semantic vector space, we need to take multiple fields into consideration.

The well-known fields for a document in web search are title, header, keyword, body, and the URL itself etc. These fields are primitive from the website and can be fetched from HTML tags. Another kind of fields, like anchor, leverage the description from the brother website. Via this way, we can infer with useful information from other documents. In addition to this, click signal is also with high quality and can be easily parsed from the search log. When a user clicked on the document d with a query q, we will add q to the clicked query field of d.

The special properties of these document fields make it difficult to unify them into one semantic space. One common approach (neuralrankingmodelmultiplefields, ) is to separately encode the multiple fields respectively and learn a joint loss across these fields. Other alternatives unify all fields by simply concatenating all these fields with spaces.

As shown in Figure 3 we translate the segment definition of pre-next sentence in BERT to different fields in document by mapping multiple fields into different segments. Field embeddings are introduced to differentiate fields. Every field has a max length constrain, we set it to 20 tokens for anchor, URL and title fields. For clicked query fields, since a popular document may exist a large magnitude of click instances, we only pick the top 5 clicked queries for one document with a max length of 68 tokens. For all these fields, we pad them according to the need. To obtain an unified document embedding, a [CLS] token is added at the beginning of the combined fields, and a [SEP] token is also inserted between each field to distinguish different fields.

We can achieve a sequence of semantic embeddings after GLOW Encoder. Inspired by (devlin-etal-2019-bert, ), using the embedding of [CLS] in the last layer as the matching features is already good enough. Nogueira et al.(nogueira2019passage, ) also proved that in passage ranking task, adding more components upon Transformer does not help too much(qiao2019understanding, ). Therefore, GLOW uses the embedding of [CLS] as the semantic representation for the query and document, the matching score $s$ is measured by cosine similarity on the query and document vectors.

We adopt a binary cross entropy loss to optimize the model, which determines whether a query-document pair is relevant or not. We also tried pair-wise loss and found it had no extra improvement. Prior works (qiao2019understanding, ; nogueira2019passage, ) also confirm on this.

where $y$ is the label denoting if query-document is relevant, $\delta$ represents Sigmoid function. $w$ and $b$ are used to generate weighted cosine similarity to fit the Sigmoid function.

We evaluate the performance of GLOW on MS MARCO³³3The source code of our measurement on MS MARCO is available at https://github.com/GLOW-deep/GLOW and Bing’s large scale data points. Table 1 illustrates the properties of the training data with these two datasets, it can be observed that the number of queries and documents in MS MARCO are within a pretty lower scale, only three hundred thousand queries and three million documents. It is hard to identify the deep matching model’s quality for industrial web search engines with solely measurement on MS MARCO. Thus we expand the data scale by collecting more training data from Bing’s search logs. In the Bing’s internal data points, we scale up the query counts to 30 million and documents to 140 million. What’s more, we also add clicked queries as the extra field in the internal set.

To answer RQ1, we compare the retrival performance GLOW with aforementioned baselines on MS MARCO and Bing’s large scale data points. The evaluation results are listed in Table 2.

The evaluations on MS MARCO and Bing’s large scale dataset reveal that GLOW can not only address the real encoding problem for industrial search engine, but also extends to be a general solver for ordinary document matching task in IR community.

To study RQ2, as aforementioned, three key components empower the embedding quality of GLOW. To further investigate the individual contribution of each part, we examine three GLOW variants and compare the NCG and NDCG results with BERT and GLOW full implementation on the Bing’s internal dataset. Each variation disables a component while keeps others unchanged. The results reported in Table 3 indicate that all these three components are critical to GLOW, missing anyone of them would degrade the performance of GLOW. Detailed analyses are described as follows.

The design purpose of GLOW is to enhance the attention for those words having higher weights. We conduct some case studies to answer RQ3. Comparing BERT and GLOW, Table 4 illustrates top 5 words which have the highest attention scores for each document. Here the attention scores are generated from the last layer’s hidden state outputs, for each token, we firstly multiple all others by an inner product and average the value, then sort all tokens’ attention score. For those words belong to multiple tokens, we sum up the attention score to fetch those words’ attention.

From Table 4, we can observe that GLOW is more effective to capture topical and semantic words from documents. Doc1 tries to share the answer to question ”can low sodium levels cause seizures”, it is pretty clear here ”low”, ”sodium” and ”seizures” are the topical words to these document. GLOW successfully rank these 3 words into Top5 while BERT only rank ”low” to #2. In Doc2, ”sheriff” cannot rank into top 5 highest attention words in BERT, while it ranks second position in GLOW. This document has a strong preference on describing the officer salary information at $sheriff$ area, weakening this information makes the retrieval system harder to get this document back when people searches a query related with ”sheriff patrol”. Similar phenomenon takes place at Doc3, BERT failed to emphasize ”Hang Son Doong”, which is vital to describe the topic of this document. But GLOW successfully enhance ”Hang Son Doong“’s attention. It is also explainable since ”Hang Son Doong” are all rare words, their global weights are pretty higher than other words, in addition, they repeat twice in this document so their BM25 value are higher than other words.

Then to answer RQ4, we evaluate the efficiency of GLOW from the points of view of model complexity and training cost. The training trend reveals GLOW is easier to converge while retaining the same training parameters with BERT.