Robin: A Novel Online Suicidal Text Corpus of Substantial Breadth and Scale

The Robin dataset is a suicidal text corpus designed to offer state-of-the-art breadth and scale, with the intention of enabling next-generation suicide prevention machine learning models. With this in mind, the dataset consists of 1,104,711 online posts, collected from a variety of subreddits from the social media website Reddit.com. PushShift is an open source API which allows users of the API to powerfully search and aggregate submissions and comments from Reddit. PushShift was used to greatly reduce the collection time of the Robin dataset Baumgartner et al. (2020).

Constructing a dataset using data from Reddit brings numerous benefits. First, outside of a few online acronyms, posts on Reddit have relatively few structures in place that differ from those commonly employed in the English language at large. As a result, the Robin dataset can be used for the machine learning detection of suicidal content, rather than suicidal content limited to a single platform. Datasets constructed using other platforms, such as Twitter often contain artificially shortened self-writing and website-specific acronyms and abbreviations; consequently, they may struggle to generalize to the typical English employed on websites like Facebook and Reddit. All Reddit posts are publicly available, which makes scraping them relatively easy compared to websites where profiles may be private, such as Facebook or Instagram.

Reddit is divided into countless subreddits, which are individual online communities where users post about specific topics. We curated data from several subreddits, with an underlying rationale for each. Suicidal posts make up 20% of the dataset and were sourced from the subreddit SuicideWatch, which allows users to post a cry for help when they are feeling suicidal thoughts. We operate under the assumption of self-labeling, assuming that all posts on SuicideWatch are expressing suicidal ideation or sentiment; human annotation later on demonstrates that this is a quite reasonable assumption. We included 13 additional subreddits, each specifically selected to either offer an additional category of suicidal text or to provide general, non-suicidal text. Suicidal text is nuanced and spans a range of categories, only one of which–suicidal ideation–is cause for concern. Incorporating these additional categories, like suicide bereavement, allows models using Robin to train on common false positives. The categories of suicidal text included in Robin and their subreddit sources are described in detail in Table 1.

Robin was constructed using the web API service PushShift, which allows users to query specific subreddits and download available posts from as early as the beginning of 2019. For each post, we collected the title, body, subreddit, length, and identifying id. The specific breakout of the dataset by subreddit is shown in Table 2.

Relatively little pre-processing was needed. First, the title and body were concatenated to create a single unit of text; this was done to make the corpus platform-agnostic, as many platforms do not have post titles. Then, each post was assigned a suicidal/non-suicidal label, and the reddit post IDs were replaced with dataset-specific identifiers. Finally, we removed subreddit-specific features not reflecting the syntax of plain English. This was necessary for the subreddit tifu, in which the body of posts always begins with the token TIFU.

Whenever handling mental health data, it is of the utmost importance that adequate steps have been taken to ensure anonymity and privacy. Although Reddit usernames are rarely self-identifying, we made the decision not to include them as a feature in the dataset given the sensitive nature of this work. However, while the posts were being scraped, we tracked the number of unique authors on each subreddit. Additionally, each Reddit-specific post identifier was replaced with a dataset-specific post identifier, creating an additional barrier separating the scraped posts in Robin from their online Reddit versions. It’s worth noting that these posts are publicly available online. Still, we took the aforementioned steps to ensure that they are anonymized in the context of the Robin dataset. All scraping and post collection was done in compliance with the Reddit Terms of Service and Conditions.

Data labeling was conducted under the assumption that all posts on SuicideWatch are suicidal and all posts from the other subreddits are non-suicidal. In order to assess our quality of labeling, we randomly selected 1000 posts from the dataset for human annotation with an even split of 500 suicidal and 500 non-suicidal. Each post was annotated by five annotators using Amazon’s MTurk service, with a fee of $0.03 per post (corresponding to a wage of around $18 per hour). Annotators were given the following prompt: ‘Is the author of this social media post suicidal?’. Since each post is labeled by five annotators, we select the label with the majority of votes, with no chance of a tie. Each post was labeled by five annotators, but these may not be the same five annotators for each post.

Of the 500 posts labeled as suicidal in the sample, a majority of annotators classified 398 as suicidal and 102 as non-suicidal. Of the 500 posts labeled as non-suicidal in the sample, a majority of annotators classified 487 as non-suicidal and 13 as suicidal. These results yield an annotator F1-score of 0.87 and accuracy of 0.89, indicating fairly close agreement between our labeling and the annotator labeling. Additionally, there was considerable inter-annotator agreement, with an average Fleiss’ Kappa of 0.78. If anything, these labeling results indicate that Robin is overly biased in labeling content as suicidal. That said, false positives on this problem are far less impactful than false negatives.

While, on the one hand, this can be used as validation for our existing labeling, on the other hand, it can also be used to assess the performance of untrained humans on this task, assuming that our original labeling was accurate. This is probably not a perfectly true assumption, but it can offer insight nonetheless. Some of the suicidal posts that were labeled as non-suicidal by the annotators can well be seen as suicidal given the correct context.

We conducted an extensive linguistic analysis of the Robin dataset, examining the most common bigrams and trigrams in each subreddit, as well as the frequency of suicidal terms.

We also conducted a t-SNE dimensionality reduction to visualize a random sample of 5,000 posts from our dataset to better understand the structure of our data. We used TF-IDF vectorization for this subset of the data. Figure 1 shows that the lexical content of the suicidal data tends to have some distance from other types of posts, but is scattered throughout the visualization. This suggests that a purely lexical model may not be adequate to distinguish suicidal from non-suicidal text.

We present a brief overview of notable previously published datasets in Table 3.

Shing et al. (2018) presented a corpus which contained a time-series of posts for 1868 Reddit users. They constructed their corpus by finding 11,129 users which had posted at least once in SuicideWatch and had at least 10 Reddit posts between January 1, 2008 and August 31, 2015. Then, they randomly sampled 1,097 users from this. Of the 1,097, they were able to successfully annotate the posts of 934 users. To form the overall dataset, they included an equal number of control users who were not suicidal. Although this dataset uses a similar data collection methodology to ours, the size of the corpus they present is over an order of magnitude smaller than ours. Further, they excluded all users with less than 10 posts. Users on SuicideWatch often create a new account which they intend to use once to post on SuicideWatch. Omitting this type of post leaves the corpus with an incomplete picture of suicide ideation.

Roy et al. (2020), Sinha et al. (2019) both generated datasets via the Twitter API. The Twitter API only allows users to collect tweets that match specified search queries. As a result, these datasets only captured tweets which contain semantically obvious suicidal intent, i.e. “I am thinking about killing myself.” This approach is flawed for two reasons. First, there is an inclusion bias against tweets which express suicidal ideation but do not contain obvious keywords like ‘suicide’ and ‘die’. Second, selecting Tweets by keywords alone over-simplifies the problem of suicidal sentiment classification (allowing bag-of-words models to excel by virtue of how the dataset is constructed) and injects a major bias for what constitutes suicidal text.

We want to emphasize that there is a dearth of comparable corpora that are publicly available. We emailed authors of related work for dataset access, and no emails were returned. We believe this further highlights the need for the Robin corpus: note that the Robin dataset is freely and publicly available under the Creative Commons BY 4.0 license. It is available for download $<$here$>$.

The authors are aware of and acknowledge several limitations in our dataset compared to other work. Although our validation achieved high inter-annotator agreement, our annotators were online workers based in the United States, not experts. The size of the dataset meant that the 1,000 posts which were randomly sampled to be validated comprise a small fraction of the total dataset. Finally, our data originates from anonymous Reddit postings. This introduces bias in the demographics for which our model is finely tuned towards the male, English-speaking demographic group with which Reddit is associated (Tigunova et al. 2020).

Although researchers are increasingly emphasizing deep learning techniques, traditional machine learning techniques such as logistic regression still offer impressive performance for many tasks. To this end, we examined a breadth of traditional learning methods to establish a comparative benchmark for subsequent deep learning approaches. We trained the following models: logistic regression, linear support vector machines, complement and multinomial Naive Bayes, random forest, and XGBoost. Each model was hyperparameter tuned using F1-score and cross-validated on five folds of data. Hyperparameter tuning grids were relatively coarse, especially for tree-based approaches such as random forest and XGBoost, due to environmental and computational concerns (see Appendix for exact tuning details).

Each traditional learning model was paired with six different vectorization approaches, with hyperparameter tuning happening for all six pairs. The six vectorization approaches used were unigrams, bigrams, and unigrams and bigrams; each was used with either bag-of-words vectorization or a term frequency-inverse document frequency (TF-IDF) transformation. For traditional machine learning approaches, we performed vectorization within each fold. Additionally, all standard English stop words were removed, and the vocabulary was capped at 20,000 unique words.

Once all traditional models were adequately tuned, the best model was selected for further analysis based on F1-score. In this additional analysis, we benchmarked the model using several additional metrics, measured subreddit-specific performance, generated receiver operating characteristic (ROC) and precision-recall curves, and examined linguistic markers by measuring the feature importance embedded within the model.

We also applied the BERT transformer model (Devlin et al. 2018). BERT pre-trained models can be fine-tuned to specific tasks, allowing for noteworthy performance on particular objectives while remaining context-aware due to BERT’s extensive pre-training on BooksCorpus and English Wikipedia. Our results use a fine-tuned pre-trained BERT model (’bert-base-uncased’), implemented in the library Transformers by HuggingFaces, with a 768-node fully connected classification and softmax layer appended onto the end (Wolf et al. 2020). We truncated posts to 512 tokens, and padded any posts which were shorter than this length to meet the formatting requirements of BERT. We used a tokenizer from Transformers to pre-process the text to a BERT-specific format, and used the popular Adam Optimizer for our training (Kingma and Ba 2014). We trained BERT on several subsets of our dataset and continued to see an increase in performance as the size of the dataset we used continued to grow. When initially fine-tuning BERT, we tried a learning rate of 0.01, which quickly led to over-fitting. We finalized with a learning rate of $5\cdot 10^{-6}$ and batch size of 10 over a single training epoch. Due to computational constraints when benchmarking BERT’s performance, we did not use cross validation and used no tuning besides learning rate, whereas we used five-fold nested cross validation for traditional machine learning techniques. For traditional machine learning approaches, we performed vectorization within each fold. We split our dataset into an 80-20 train-test split for traditional machine learning and for BERT. The random seed we used for model training was 42.

After hyperparameter tuning, the best-performing model was logistic regression paired with TF-IDF vectorization of unigrams and bigrams; this architecture yielded an F1-score of 0.85 ($\pm 0.002$) when tested on approximately 200,000 unseen posts. Other techniques also offered noteworthy performance, with even the worst-performing model achieving an F1-score of 0.73 ($\pm 0.001$).

Beyond F1-score, the logistic regression model achieved a precision of 0.89, recall of 0.82, accuracy of 0.94, and Matthews’ Correlation Coefficient (MCC) of 0.82. The model cumulatively classified 861,092 true negatives, 180,191 true positives, 22,670 false positives, and 40,747 false negatives across all of the cross-validation folds. The ROC and Precision-Recall curves demonstrate the model’s excellent capability in distinguishing between classes, achieving AUCs of 0.98 ($\pm 0.000$) and 0.92 ($\pm 0.000$) respectively (Figure 2).

We also examined the linguistic markers of the logistic regression model by charting the fifteen most strongly weighted features as determined by coefficient magnitudes in Appendix Table A1. While some terms, such as ‘suicidal’ and ‘kill’ are obvious linguistic markers, others, such as ‘anymore’ and ‘goodbye’ are far more subtle. Many of these terms may not indicate suicidal ideation depending upon the context, emphasizing potential shortcomings of such naive language models.

As expected, BERT was able to learn the nuances of suicidal text and outperformed the traditional methods in every measured metric. When evaluated on roughly 200,000 test posts, the model classified 36,939 suicidal posts as suicidal, 3,288 suicidal posts as non-suicidal, 156,440 non-suicidal posts as non-suicidal, and 3,332 non-suicidal posts as suicidal, achieving an F1-score of 0.92, precision of 0.92, recall of 0.92, accuracy of 0.97, and Matthews’ Correlation Coefficient (MCC) of 0.90.

BERT also outperformed traditional models with respect to receiver operating characteristic (ROC) and precision-recall curves, achieving AUCs of 0.99 and 0.97 respectively (Figure 2). BERT also fared better with differentiating between suicidal ideation and bereavement; BERT classified 30% of bereavement posts as suicidal, whereas logistic regression classified 39% as suicidal (See Appendix for subreddit-specific results).