Transformers for Headline Selection for Russian News Clusters

For each headline from the training corpus, embeddings are obtained from various Russian and multilingual models. We use pretrained BERT-based models trained both on Russian monolingual corpora (RuBERT [5], SBERT [7]) and in multiple languages (mT5 [14], XLM-RoBERTa [12]) including Russian. In addition, a multilingual version of USE [11] embeddings is used. The mentioned models show state-of-the-art results on a number of NLP benchmarks [13], including those in Russian language [8], so it was natural to test them on the task of selecting the best headline for the cluster.

To obtain a headline embedding, we take the average word embeddings from the layer 19 (of 25) for SBERT, XLM-R and mT5 and layer 8 (of 13) for RuBERT, considering the length of the headline. In the case of mT5 model, which is trained mainly for solving seq2seq tasks, the decoder is removed and the embedding is taken from layer 19 of the encoder. We use recommended pretrained tokenizers from the transformers library [10]. These tokenizers are based on WordPiece and SentencePiece models [4].

Then we train a classifier on top of the embeddings. We use CatBoost ranking model [2], which is a gradient boosting over decision trees algorithm. Pairs of headline embeddings are fed to the classifier as input, while the best headline in the pair is considered the ”positive” element of the pair, and the other one is considered ”negative”. We chose PairLogitLoss (1) as the target loss function.

An ensemble of ranking models is trained based on different features. The number of decision trees in CatBoost is set to $10^{3}$, the best epoch is chosen based on the validation score. We obtain the final headline rank by averaging the ranks predicted by each of the models, and then each pair is assigned one of the left, right, or draw labels depending on the resulting rank difference $r_{r}-r_{l}$. More specifically, the rank difference $r_{r}-r_{l}<0$ means the winner is left, $r_{r}-r_{l}>0$ means the winner is right, and $|r_{r}-r_{l}|\leq 0.1$ correspond to draw.

To explore models and compare the results, we train classifiers on top of each type of embeddings separately. Every model was trained on an Nvidia Tesla P100 GPU provided by Google Colab. Table 2 shows the results of using various embeddings and training classifiers on top of them. The table shows that the best result is obtained by the ensemble of the five models mentioned in the paper (referred as Blend-5), but the single multilingual model mT5 shows a comparable accuracy.

During the Dialogue Evaluation 2021 competition we achieved 86.00% and 85.40% accuracy for the public and private test sets by taking the average word embeddings in the top layer. But further experiments show that middle layers perform better than top layers with the result of 87.28% and 86.60% accuracy respectively.

In this section, we analyze the model and explore the impact of several aspects of out approach. We list some examples of wrongly predicted labels and report the evaluation results for different variants of the model on the private LB test set.

Error analysis. We selected 300 examples where the model confused between left and right labels and skipped examples with draw label. Errors can be divided into several types as listed at (2.3)–(2.3) together with several corresponding examples and their translation to English. The model is more likely to prefer titles with lack of facts and sometimes tends to choose verbose headlines. Other wrongly selected headlines include opinions and biased titles. Finally, having a pair of almost equivalent titles, the model could choose the wrong label.

Headlines containing insufficient facts

gold: Ту-22МЗМ Казанского авиазавода испытали на сверхзвуке
gold: Kazan aircraft factory’s Tu-22MZM tested at supersonic mode

pred: В ОПК рассказали об испытаниях модернизированного ракетоносца Ту-22М3М
pred: The defense industry told about the tests of the modernized Tu-22M3M missile carrier

gold: День проведения парада Победы будет нерабочим — Песков
gold: Victory Parade day will be non-working — Peskov

pred: Песков заявил о большой вероятности объявления еще одного выходного
pred: Peskov said about the high probability of announcing another weekend

Verbose headlines

gold: С 27 мая москвичи могут бесплатно сдать тест на антитела
gold: From May 27, Moscow residents can take an antibody test for free

pred: <<Как и где сдать тест на антитела к коронавирусу в Москве. С 27 мая это может сделать любой желающий
pred: How and where to take an antibody test against coronavirus in Moscow. Anyone can do it from May 27

gold: Украинский суд признал нацистской символику дивизии СС <<Галичина>>
gold: The Ukrainian court found that the symbols of the SS division ”Galicia” were Nazi

pred: <<Победа справедливости, здравого смысла и закона>>: Вятрович проиграл суд по делу о символике СС <<Галичина>>
pred: ”Victory of justice, common sense and law”: Vyatrovich lost the court case concerning the SS ”Galicia” symbols

Biased headlines

gold: Роскомнадзор начнет блокировать в России пиратские приложения
gold: Roskomnadzor will begin to block pirated applications in Russia

pred: Госдума приняла спорный законопроект о блокировке приложений с пиратским контентом
pred: State Duma passed controversial bill on blocking applications with pirated content

gold: Доллар в обменниках ускорил рост
gold: Dollar in exchange offices accelerated growth

pred: Заманивают иностранцев под покупку облигаций. Почему гривня снова падает
pred: Luring foreigners to buy bonds. Why is the hryvnia falling again

Equivalent pairs of headlines

gold: Россияне массово забирают валюту из банков
gold: Russians massively withdraw currency from banks

pred: Жители страны массово снимают валюту с банковских счетов
pred: Residents of the country are massively withdrawing currency from bank accounts

gold: Россиянам разъяснили, когда можно будет поехать в отпуск за рубеж
gold: Russians were clarified when it will be possible to go on vacation abroad

pred: Россиянам озвучили возможные сроки возобновления поездок за границу
pred: Russians were announced the possible date of the resumption of trips abroad

Thus, a good headline can be defined as precise, short, unbiased, and containing as many significant facts as possible. However, classifiers based on language models can rank headlines which do not meet these criteria higher than manually selected ones. We assume that adding more training data or introducing multitask learning, combining another training objectives, such as information extraction, could help achieve better results.

Sentence representations. We explore a different way to obtain sentence embeddings from language model’s top layer output: using the embeddings of the first token, known as [CLS] token, which is sometimes more common than averaging the word embeddings. We compare the results for all models, except mT5, which doesn’t have the [CLS] token in its dictionary. However, as shown in Table 3, averaging show slightly better results for most of the models. We assume this is because embeddings of the [CLS] token contain high-level semantic meaning [13], while for headline selection task it is more important not to lose the token-level information to meet the previously formulated criteria of a good headline. Moreover, we analyze whether sentence embeddings in the middle layers can be more suitable than in the last ones. Experiments show that layers 17 to 19 (of 25) for SBERT, XLM-R and mT5 and layers 8 to 9 (of 13) for RuBERT perform better than the top layers. The reason for this may be the same: top layers embed more high-level semantic meaning than the middle ones.