MTLHealth: A Deep Learning System for Detecting Disturbing Content in Student Essays

Particularly in the period of 2018-2019, a wave of papers introduced systems that leveraged new approaches to transfer learning, a paradigm in which models trained on one type of data are re-purposed for a new task. These systems used unsupervised pre-training methods to initialize models for a variety of downstream tasks.

The power of transfer learning is apparent in its successful application to a number of challenging benchmark tests. SQuAD is one such challenge, in which the answer to a question must be highlighted within a context paragraph (Wang et al., 2018). GLUE is another benchmark, consisting of nine diverse language understanding tasks, including natural language inference, sentiment analysis, and paraphrase generation (Rajpurkar et al., 2018). On both datasets, deep learning systems have matched human baseline performance.

Two overarching trends in the literature appear to be responsible for much of the progress in language understanding. One is the use of pre-training on massive datasets using objectives similar to language modeling. The other is an increased reliance on Transformer neural network layers.

An emerging sub-field of research within computational psychology involves applying general NLP techniques to analyze the connection between language use and mental health. Many studies make use of textual data from web sources, particularly from Twitter and Reddit.

The Association for Computational Linguistics holds a yearly workshop in Computational Linguistics and Clinical Psychology (CLPsych). CLPysch has sponsored the creation of multiple mental health datasets and held competitions on machine learning for mental health. Past competitions have centered around efforts to triage posts on the suicide support website ReachOut.org and to predict depression and anxiety levels throughout life based on youth essays (Milne et al., 2016) (Lynn et al., 2018). The most recent involved predicting the degree of suicide risk on the r/SuicideWatch subreddit (Zirikly et al., 2019).

A paper by Gkotsis demonstrated the ability of simple fully-connected and convolutional neural network architectures to classify Reddit posts as originating from mental health subreddits vs non mental health subreddits (Gkotsis et al., 2017). A study by Yates used similar methods on user posts in non mental-health subreddits to predict whether those same users would later announce a depression diagnosis on r/depression (Yates et al., 2017).

Some prior work has directly explored the problem of detecting disturbing content in student responses. Earlier research at ACT produced a disturbing content pipeline based on an ensemble of multiple non-neural machine learning methods trained on selected Reddit posts (Burkhardt et al., 2017). A study by the American Institutes for Research built a classifier for a large internal dataset of constructed response and compared the performance of several varieties of recurrent neural network architectures. (Ormerod and Harris, 2018).

MTLHealth is trained on a variety of labeled datasets as described below.

• •

  Emotion Classification: $\sim$6k Tweets annotated with 11 emotions including anger, disgust, joy, sadness, and trust. (Mohammad et al., 2018)

• •

  Toxic Comment Classification: $\sim$150k Wikipedia comments annotated with labels including toxic, hate speech, obscene, and threat.¹¹1https://www.kaggle.com/c/jigsaw-toxic-comment-classification-
  challenge

• •

  UK Birth Cohort Essays: $\sim$10k essay responses from 11 year olds in 1969 who were asked to imagine their lives at age 25. Each essay is annotated with a numerical score of its author’s depression level as measured by a psychological examination following the British Social Adjustment Guide. (University College London, 2018) (Zirikly et al., 2019)

• •

  Subreddit Data: $\sim$357k user posts in Reddit forums focusing on issues related to mental health along with control posts from subreddits unrelated to mental health. Annotated with subreddit origin information. Some examples are: r/StopSelfHarm, r/SuicideWatch, r/addiction, and control.²²2 Mental health data obtained via personal correspondence with author of (Gkotsis et al., 2017) and control taken from Reddit dump at https://files.pushshift.io/reddit/comments/.

• •

  ACT Constructed Responses: Consists of 102 constructed responses annotated with disturbing status. 42 flagged as disturbing (label of 1) , 80 flagged as normal (label of 0).

The presence of disturbing content in student responses is a relatively rare occurrence. To bolster the MTLHealth system, non ACT datasets were used to encourage the system to flag a wider range of disturbing responses.

The selected datasets were chosen to ensure the system is informed by data that is diverse in terms of domain (short tweets vs long Reddit posts), age of speakers (children vs adults), time period (1960’s vs present day), and topic. Emotion and toxicity are somewhat distinct from mental health, but may be strongly correlated with mental health outcomes (Hu et al., 2014).

The non-profit AI organization HuggingFace maintains pytorch-transformers, a library for the deep learning framework PyTorch that allows users to build models initialized with pre-trained weights from Google ³³3https://github.com/huggingface/pytorch-transformers. MTLHealth uses BERT-Base uncased. BERT-Base uncased is the smaller version of BERT, with 12 Transformer layers, a hidden dimension of 768, and nearly 110M total parameters (Devlin et al., 2018).

Following the usage guide for BERT, MTLHealth pre-pends all input sentences with the special token ‘[CLS]’. After fine-tuning, the vector output corresponding to this token serves as the overall representation of the sentence. The pooled output is passed through multiple fully-connected layers with ReLU functions Eq. (9) before being passed to a task-specific output layer.

Training neural networks involves selecting hyper-parameters.

The ‘learning rate’ is a multiplier for the parameter updates which are calculated during backpropagation. This value affects convergence of the model, if the learning rate is set too low, training could take excessively long. If the learning rate is too high, training may be unstable and the system may never converge. MTLHealth uses $lr=2e-5$.

The ‘maximum sequence length’ defines a maximum number of tokens beyond which input sentences are truncated. Unlike many training schemes, MTLHealth does not use a fixed batch size. Instead, it accepts a hyper-parameter of ‘maximum tokens per batch’ and attempts to batch sequences of similar sequence length to be close to this value. This scheme follows the implementation found in (Rush, 2018).

MTLHealth makes use of a technique called dropout, where a percentage of the output is randomly set to zero, which helps to prevent over-fitting by removing noise in the representations (Srivastava et al., ). Dropout probability is set at $p=0.5$.

During fine-tuning for each task, all layers, including those belonging to the pre-trained model, are trained using the ‘Adam’ optimizer, a variation of the standard stochastic gradient descent algorithm that incorporates the concept of ‘momentum’ to automatically scale the learning rate (Kingma and Ba, 2014).

Training typically involves multiple passes through the data, which is known as an epoch. Training is allowed to continue for a maximum number of training epochs $max\_epochs=20$. If a chosen task-specific evaluation metric does not improve for several consecutive epochs $patience=5$, training stops automatically.

Hyper-parameter values not found here are listed in the Appendix.

There are three basic problem formulations that MTLHealth is trained on: classification, multi-label classification, and regression. A BERT model was trained for each dataset with an appropriate problem formulation: classification for the Reddit dataset, multi-label classification for the Twitter Emotion and Wikipedia Toxicity datasets, and regression for the 1969 UK Essay dataset.

• •

  Classification involves identifying a sentence as belonging to one of $C$ classes. Classes are mutually exclusive. The output activation function is a logistic softmax. Eq. (8). Classification is optimized with the cross-entropy loss function. Eq. (11).

• •

  Multi-label classification involves identifying a sentence as belonging to 0-C of $C$ classes. Classes are not mutually exclusive, and a sentence can potentially carry some, all or none of the class labels. The output activation function is a sigmoid. Eq. (7). Multi-label classification is optimized with the cross-entropy loss function . Eq. (11).

• •

  Regression involves predicting a numerical score for a sentence. The output activation function is an identity. Regression is optimized with the mean squared-error loss function Eq. (10).

After training, each BERT model can be applied to any input text. For regression, the output is a predicted score. For classification (including multi-label), the output is the probability of belonging to each class. In turn, these predictions serve as input features for a simple logistic regression classifier. This classifier yields a single yes or no label indicating whether the text includes disturbing content. This system is trained to classify the ACT constructed response data.

All training and experiments were conducted on single p2.xlarge AWS instance with access to half of an NVIDIA Tesla K80 GPU with 12 GB RAM.

Table 5 reports the results of an ablation study, which demonstrated the usefulness of the features extracted by the various BERT pipelines. The Reddit model produced by far the most predictive features of any single model, and the toxicity pipeline produced the least useful features. The best results were obtained using all feature sets besides the depression scores from the UK Data Essays. Fig (3) represents the magnitude of the feature weights learned by the final logistic regression in MTLHealth. Features like ‘suicideWatch’ ,‘severe_toxic’, and ‘sadness’ received high positive weights, meaning they are positive contributions to classifying a response as disturbing. Similarly, features like ‘optimism’,’trust’, and ‘control’ were highly negative indicators of disturbing content.

The use of transfer learning and Transformer architectures allows MTLHealth to benefit from the state-of-the art methods in 2019. The incorporation of diverse training data also makes MTLHealth unique among disturbing content detection systems in its capabilities across a variety of tasks.

MTLHealth was trained on over 200 times more text examples than the prior work at ACT. The older version only uses Reddit sources, while MTLHealth is able to make predictions on varying input domains and problem formations. (Burkhardt et al., 2017). Whereas the old system used an ensemble of non-neural machine learning models all trained on one dataset, MTLHealth builds an ensemble that incorporates information from several different datasets into its predictions. This should predispose MTLHealth towards more generally applicable predictions, although this must be confirmed with validation on more data.

The paper from the American Institutes for Research (AIR) does not report conventional classification metrics, instead focusing on the total number of disturbing essays that are caught by choosing a fixed percentage of the essays with the highest probability of being disturbing (Ormerod and Harris, 2018). Crucially, unlike the AIR system, most of the training data for MTLHealth comes from publicly available datasets.

The name MTLHealth refers both to its focus on mental health, and to the eventual goal of incorporating a training paradigm known as multi-task learning into the package. A good deal of literature indicates that simultaneously training a system on related tasks using shared parameters can yield better results than training separate models for each individual dataset (Hashimoto et al., 2016) (McCann et al., 2018). Future work will extend MTLHealth to an architecture based on MT-DNN (Liu et al., 2019a). In essence, each task will have a small number of output layers specific to the task, while the lower level representations will be in common for all tasks. This scheme will require the model to learn representations that are widely applicable across tasks.

Additionally, the MTLHealth software must undergo more rigorous review and testing before it is put into operation as part of the disturbing content detection protocol for ACT. Specifically, it should conduct more extensive empirical validation of MTLHealth predictions as additional constructed responses are annotated and transcribed. Given enough of this internal data, MTLHealth could potentially be trained to classify constructed responses in an end-to-end manner, i.e. without the intermediate feature extractors, or with the student data as part of an ensemble with the existing trained models. It must also consider the availability of manual graders in order to decide a threshold above which a probability estimate from MTLHealth should trigger an alert.