HeadlineCause: A Dataset of News Headlines for Detecting Causalities

Same headlines: two headlines are considered as the same if they are about the same things or differ in minor details. In other words, if they describe the same event, they should be considered the same.

An example of headlines we consider same (sample en_tg_572):
A: Exclusive: NextVR acquired by Apple (Updated)
B: Apple Buys Virtual Reality Company NextVR

Causality: the first headline causes the second headline if the second headline is impossible without the first one. If the first event did not happen, then the second event must not be happening too.

An example of a news headline pair with a causal relation (sample en_tg_1153):
A: Oklahoma spent $2 million on malaria drug touted by Trump
B: Gov. Kevin Stitt defends $2 million purchase of malaria drug touted by Trump

This type of causality is known as necessity causality. There are other possible definitions of causality, including sufficient causality or a cost-based concept of causality. They are described in Roemmele et al. [2011]. We have several reasons to use this particular definition. First, it is aligned with our goals stated in the introduction. Second, it is easy enough to be used in a crowdsourcing project.

Refutation: the second headline refutes the first one if the second headline makes the first one irrelevant. Refutations are a subset of causal pairs, and every refuting headline is an effect of some cause, but not every effect of a cause is a refutation.

An example of a news headline pair with a refutation (sample en_tg_496):
A: Report: Microsoft acquiring Microvision, a leader in ultra-miniature projection display
B: Microsoft denies MicroVision acquisition

We used two sources of news documents: the Lenta corpus¹¹1https://github.com/yutkin/Lenta.Ru-News-Dataset and documents from the Telegram Data Clustering Contest²²2https://contest.com/docs/data_clustering2. We additionally parsed the Lenta website³³3https://lenta.ru to obtain fresher news documents and hyperlinks between documents. Lenta is one of the oldest Russian news websites, and the dataset contains over 800 thousand news from 1999 to 2020. The Telegram news dataset was published in 2020 and contains data from hundreds of sources in over ten languages from October 2019 to May 2020.

The original data for the Telegram dataset contains various types of documents. We trained a FastText classifier on separate crowdsourcing annotations and open datasets to differentiate news from other documents. These annotations and the classifier itself are available in the separate GitHub repository ⁴⁴4https://github.com/IlyaGusev/tgcontest. The classifier is not ideal, but the majority of the resulting documents are news.

The licenses for both datasets are not provided. Our dataset uses only news headlines, provides links for all used documents, and provides authors where possible, which can be considered fair use. We also emailed Lenta asking for permission to publish their data.

For both datasets, we used four filters to extract candidates for annotation:

• •

  A presence of a hyperlink between two documents

• •

  An affiliation of documents to the same website

• •

  A cosine distance between LaBSE embeddings [Feng et al., 2020] with a threshold

• •

  A presence of different locations in headlines

We combined these filters in various combinations in different annotation pools to collect diverse data. We did not have a specific algorithm or scheme for applying these filters and used some of them as different problems emerged. For example, at some moment we observed that our trained model was making errors in linking news headlines about similar events, but in different locations, so we annotated more data with the last filter. The first two filters were used as the main ones.

Another problem we detected was a subset of headlines that our models considered causes without even looking at effects. This type of error is common for sentence-pair inference tasks [Gururangan et al., 2018]. To handle this, we trained a single headline classifier to estimate a priori probability of a headline being cause or effect, sampled negative pairs for the original task from the strongest examples, and annotated them.

To estimate a final role of a sampling method based on a cosine distance between LaBSE embeddings, we plotted a dependency between the distance and the number of causal pairs. It can be seen in Figure 2. We used this distance as an upper threshold in our candidate sampling. It is clear from the figure that for the Russian annotations we did not lose any of the information as with the increasing threshold the number of causal pairs is decreasing very fast. It is not so obvious for the English annotations, so we additionally plotted Figure 1. One can see that the ratio of causal pairs is also decreasing.

To estimate a final role of a sampling method based on a cosine distance between LaBSE embeddings, we plotted a dependency between the distance and the number of causal pairs. It can be seen in Figure 2. We used this distance as an upper threshold in several candidate samplings. It is clear from the figure that we did not lose many causal pairs above the threshold for the Russian annotations, as with the increasing threshold, the number of causal pairs is decreasing very fast. For English annotations, this regularity is not apparent from Figure 2, so we additionally plotted Figure 1. One can see that the ratio of causal pairs in English is also decreasing with the distance.

We annotated every candidate pair with Yandex Toloka⁵⁵5https://toloka.ai/, a crowdsourcing platform. We chose this platform as we are very familiar with it, and it has many workers that are native Russian speakers, which is useful for Russian annotation. The task was to determine a relationship between two headlines, A and B. There were seven possible options: titles are almost the same, A causes B, B causes A, A refutes B, B refutes A, A linked with B in another way, A is not linked to B. An annotation guideline was in Russian⁶⁶6https://ilyagusev.github.io/HeadlineCause/toloka/ru/instruction.html for Russian news and in English⁷⁷7https://ilyagusev.github.io/HeadlineCause/toloka/en/instruction.html for English news. Ten workers annotated every pair. The annotation interface is in Figure 3. The total annotation budget was 2173$, with the estimated hourly wage paid to participants of 1.13$. Initially, it was 45 cents, but we reconsidered this wage because of ethical reasons, as it is lower than the minimum wage in Russia. Annotation management was semi-automatic. Scripts are available in the project repository⁸⁸8https://github.com/IlyaGusev/HeadlineCause.

As for the quality control, we required annotators to pass training, exam, and their work was continuously evaluated through the control pairs ("honeypots"). The threshold was 70% correct examples for training and 80% correct examples for exams and honeypots. No additional language abilities check was done, as these thresholds should remove workers without them. All examples from training and exam are also available in the project repository.

Annotation statistics are in Table 1. As for the number of pairs with links and the same sources, the filtering system causes these high numbers. As for the number of annotators, historically, the Russian annotation was done earlier than English and was split into two considerable periods, so more workers were involved.

We aggregate annotations in two settings. The first setting, Full, includes all seven possible classes. The second setting, Simple, unites some of the classes to simplify the task. The alignment between labels is presented in Table 2. We include only samples with an agreement of 70% or higher in the final dataset. The agreement is calculated relative to the setting, with three labels for the Simple setting and seven labels for the Full setting.

Agreement distribution for both tasks is in Table 3. Agreement between workers is very different for Russian and English. There are several possible reasons for this. Firstly, English workers are less homogeneous than Russian ones, as they are from a more extensive list of countries, and English is not native for some of them. Secondly, there is a difference in the complexity of the task itself. The headlines are more challenging in English documents, as they include a bigger list of entities, including local ones. The diversity of news agencies is also more considerable in the English dataset than in Russian.

The distribution by the most popular countries of English-speaking workers is in Table 1. Most of the workers are probably not native speakers. It affects the annotation quality, but the aggregation with an overlap of 10 and tight quality control should help to maintain it.

We use the Majority vote (MV) aggregation method. There are several possible alternatives: the Dawid and Skene [1979] method, aggregation by skill, and some others. Some of them are supported by crowdsourcing platform itself. We chose the MV as it is easily interpretable and yielded consistent results in our experiments. We also tried to use the Dawid-Skene method, but the training on resulting annotations was hard and yielded poor metrics, so we abandoned it early. However, we still do not know whether the poor results came from the method itself or the complexity of the examples it brings, and it is the subject of future experiments.

The final statistics for aggregated annotations are presented in Table 4 and Table 5. All datasets are imbalanced. For the Simple task, only 25% of samples contain causal relations.

We remove pairs that are not consistent with timestamps. In other words, if a temporally following headline is a cause of a temporally preceding headline, we consider an annotation of this pair inaccurate. There were 981 (10%) such pairs for the English dataset and 540 (5%) pairs for the Russian one.

These numbers drop to 202 (5%) and 132 (2%) if we consider only pairs with more than 70% agreement (Simple task) and from the same sources. Different sources can have different timestamps policies or different promptness of reaction to the events, so it can be incorrect to compare them.

We split the dataset into train, validation, and test sets by time. The main reason is a possible entity and event bias in the news domain. Trained models should work well with the unseen entities, locations, and events, and the best way to emulate these factors is to split by time. We consider a maximum of left and right timestamps as a timestamp of a pair. The training dataset contains the first 80% of pairs, the validation dataset contains the next 10% of pairs, and the test dataset utilizes the remaining 10%.

Lenta and Telegram corpora have very different densities of news in time and different time spans affected. So we split these two datasets separately and unite the resulting splits from both sources, so we have samples from both datasets in the train, validation, and test sets equally presented.

We apply two augmentations⁹⁹9https://github.com/IlyaGusev/HeadlineCause/blob/main/headline_cause/augment.py to the train and validation datasets. The first one adds symmetrical pairs to enforce a model to be logical in case of swapping the headlines. The second one adds typos to the left, right, or both headlines to make a model more robust.

The symmetrical augmentation doubles the size of the dataset. Typos are applied for 5% of the dataset, with original pairs preserved. Typos are just swaps of adjacent letters.

For this task, we consider causality ROC AUC on two classes as a main metric. To calculate it, we unite Left-right and Right-left classes to be able to vary a classifier threshold.

We used a CatBoost [Prokhorenkova et al., 2018] classifier over TF-IDF features as a baseline model. The primary reason to use such a model is to check lexical biases or data leaks. For this task, causality ROC AUC was 71% for Russian and 62% for English, so we concluded no huge leaks.

The results for the Simple task are in Table 6. Models are working better with Russian as the training dataset for Russian is more extensive, and the average agreement on the remaining samples is higher than for English. The final score for both languages is over 95%, which means the models can predict test set labels far from random.

We use a checklist [Ribeiro et al., 2020] methodology to evaluate different aspects of the model. We do not include in Table 7 tests that models pass without fail, only those that fail in a considerable number of cases. The typos test ensures the robustness of models, swapping order checks their logic, and the different locations test inspects whether they can detach causes and effects that are certainly not connected. The swap-order test is also similar to Commutative Test from Laban et al. [2021]. The first two groups of tests match the augmentation methods introduced to reduce the failure rate.

We also hypothesized that the accuracy of models is higher for samples with a higher agreement. Indeed, as shown in Figure 4, it is an almost linear dependence between these two parameters, so we conclude that the agreement can be used as a proxy value of the complexity of a specific pair.

The results for this task are in Table 8. We provide F-score for every class and a total multiclass accuracy. The number of samples with refutation is too small to define whether we can reliably detect refutations with our model.

Additionally, we trained a GPT-2 Radford et al. [2019] model to predict effect headlines for the causes. The examples of such predictions are in Table 9. One can see that some continuations are reasonable, and it is possible to select the probable ones. Still, many continuations are not correct, and probably the larger dataset and the larger model will fix that.

In future, the generator can be used to create an augmented dataset, as it can generate grammatically correct but meaningless continuations. It will be much harder task for a detection model to correctly identify such examples.