Traditional IR rivals neural models on the MS MARCO Document Ranking Leaderboard

We use our FlexNeuART  retrieval toolkit[4]¹¹1https://github.com/oaqa/FlexNeuART, which ingests data (queries and documents) in the form of multi-field JSON entries. A field can include any textual information associated with a document. In particular, each MS MARCO document has three text attributes: URL, title, and body (main document text with HTML formatting removed). A field can be provided as is, i.e., without any modification, or it can be preprocessed and split into tokens. Given an original textual attribute, e.g., the title, it is possible to generate multiple fields, which are tokenized, lemmatized, stopped, or otherwise processed in different ways.

In the current system, a textual attribute of an MS MARCO document can be tokenized using Spacy[12] or split into BERT word pieces [10, 19]. The tokenized field can be further lemmatized and/or stopped (stopping is not applied to BERT word pieces). In the case of URL, we heuristically preprocess input before applying Spacy: We remove prefixes such as http://, slashes, and other punctuation signs.

Our submission retrieves 1000 candidate documents using Apache Lucene²²2https://lucene.apache.org/ with a tuned BM25 [17] and re-ranks documents using an LambdaMart [6] model, which aggregates 13 features described below.

Overall we compute 13 features, which include simple textual similarity features as well as the lexical translation features, namely, IBM Model 1 log-scores [1]. The corresponding JSON descriptor can be found online. BM25 and IBM Model 1[1] log-scores are core ingredients. The full set of features includes:

• •

  BM25 [17] scores, which are normalized using the sum of query term IDFs;

• •

  cosine similarity scores;

• •

  relative query-document overlap scores: the number of query terms that appear in documents, normalized by the total number of query terms.

• •

  normalized proximity scores (similar to ones used in our prior TREC experiments [3]): Each scoring component generates two feature values;

• •

  IBM Model 1[1] log-scores, which are normalized by the number of query terms.

IBM Model 1[1] is a lexical translation model, which is trained using a parallel corpus (a bitext). The parallel corpus consists of queries paired with respective relevant documents. Training relies on the expectation-maximization algorithm [5] implemented in MGIZA [16].³³3https://github.com/moses-smt/mgiza/

Overall we compute four IBM Model 1 log-scores. Three scores are computed for non-lemmatized attributes: URL, title, and body, which are tokenized with Spacy[12]. One score is computed for body, which is split into BERT word pieces [10, 19].

We take several measures to maximize the effectiveness of IBM Model 1 (see § 3.1.1.2 [2] for details). Most importantly, it is necessary to have queries and documents of comparable sizes. It is true for URL and title, but not for body, which typically has hundreds of tokens. To resolve this issue, for each pair of query $q$ and respective relevant document $d$, we first split $d$ into multiple short chunks $d_{1}$, $d_{2}$, …$d_{n}$. Then, we replace the pair $(q,d)$ with a set of pairs $\{(q,d_{i})\}$.

Although our traditional system is reasonably efficient, it is research software, which lags somewhat in speed compared to an optimized traditional system. In particular, it contains an inefficient module computing proximity score in time $O(|D|\times w)$, where $w$ is the size of the window (to count word pairs) and $|D|$ is the document length. It can be implemented to run $w$ times faster.

Furthermore, we can improve computation of IBM Model 1. First, translation tables produced by MGIZA can be pruned more aggressively with a small loss in accuracy. Second, the algorithm runs in time $O(|D|\times|q|)$, where $|q|$ and $|D|$ are query and document lengths. This can be reduced to $O(|D|)$ by precomputing a query-specific reverse-translation table (§ 3.1.2.1 [2]). Precomputation requires extra time, but it is totally worth the effort when we need to re-rank a large number of documents.

Last but not least, re-ranking requires random-access to data stored in a forward index. Reading forward-index entries from memory can be quite fast. However, storing several fields per documents can be space inefficient. At the same time, if data is read from a disk, reading can become a primary computational bottleneck. FlexNeuART  generates and reads forward indices for each field separately, because it is a simpler and more flexible approach. In a production system, all field entries belonging to the same document can be stored jointly. Because recent consumer SSD drives can sustain about 25K random reads per second⁴⁴4https://ssd.userbenchmark.com/SpeedTest/693540/Samsung-SSD-970-EVO-Plus-1TB, it is quite practical to re-rank hundreds of documents per query by reading forward-index entries from a fast SSD disk rather than from main memory.