Lightme: Analysing Language in Internet Support Groups for Mental Health

This study used state-of-the-art methods to develop hand-crafted features derived from the Reachout online support forum CLPsych:2017 . The model aims to achieve the best classification performance for crisis posts and competitive results for other classification labels, described in Section 2. We conducted a qualitative analysis of the post, which requires the immediate attention of moderators. The study has two aims:

• •

  Shed some light about the linguistic characteristics of the urgent posts.

• •

  Examine the feasibility of lexical resources in an ML classification system for triage post using the Reachout dataset.

This study used a collection of posts from the Australian Reachout mental health online forum released by the Computational Linguistics and Clinical Psychology Shared Task (CLPsych) CLPsych:2017 . Participants range from 18 to 25 years old. All of the posts are written in English. Each post in the dataset is labelled with a semaphore pattern to indicate the urgency of the post, and the required attention of the moderator, as shown in Table 1. Label distribution across the training and testing dataset of the Reachout online forum is given in Table 2.

Precision, recall and F-measure were used to examine the performance of the classifier. Precision is defined as the proportion of correctly classified posts into a particular label by the ML model. Recall is defined as the proportion of the labels that are successfully classified. The F-measure is the mean of precision and recall. Macro f-score metric is preferred since it gives more weight to infrequent yet more critical labels, such as red and crisis. Similar to Altszyler:2018 , the f-score for crisis versus non-crisis was reported. This metric measured the classifier’s capability to detect the most severe cases. Details of the official evaluation matrices are described below;

• •

  Macro-averaged F-score: The macro-averaged f-score is calculated among crisis, red and amber, and after excluding the green class.

• •

  F-score for flagged vs. non-flagged: This metric separates the posts that moderators need to action (i.e. crisis, red, amber) compared to posts that can be safely ignored (i.e. green). This is the most important metric in CLPsych since it measures the classifier’s capability to identify the post that requires moderator attention.

• •

  F-score for urgent vs. non-urgent: This metric is the average F1-score among urgent (crisis + red) and non-urgent (amber + green) labels.

A search of key computing and health databases (IEEE, ACM, PubMed and PsycINFO) were conducted to identify the key components of previous text classifiers for ISGs. Table 3 shows the features and methods used by the best performing classifiers using the Reachout dataset, more details in milne-EtAl:2016:CLPsych and CLPsych:2017 .

The Reachout dataset $D=\{x_{i},y_{i}\}^{n}_{i=1}$ consist of $n$ training instances, where the $i$th instance is a feature vector $x_{i}$ and label $y_{i}$. The classification task is to predict the label $y_{i}$ given the feature vector $x_{i}$ for each forum post such that:

We trained a Support Vector Machine (SVM) multi-class classifier with linear kernels Vapnik:1963 . SVM is a supervised ML method used widely in text classification. This method used a state-of-the-art triage classification using the Reachout dataset. Hyper-parameters¹¹1In machine learning, an hyper-parameter is a parameter whose value is set before the learning process, while the value of other parameters are derived via learning. were selected with a grid search²²2Grid search is a way of choosing the best hyper-parameters, and consist of exhaustively searching through a subset of the hyper-parameter space of a learning algorithm. scheme with a 5-fold Cross-Validation over the training set. The C hyper-parameter³³3The C hyper-parameter referrers to the regularization value, which serves as a degree of importance that is given to miss-classification. The larger the value, the less the wrongly classified examples are allowed. is 1 with $l_{1}$ regularization type, and the loss function⁴⁴4A loss function or cost function measures how good a prediction model does in terms of being able to predict the expected outcome. is hinge, the maximum number of iterations is 2000.

In order to compare Lightme (which used SVM) against other ML classifiers, we also trained K-Nearest Neighbour (KNN), and Naïve Bayes. Since deep learning models are the state-of-the-art in many natural language processing applications, we trained two neural network classifiers: Multi Layer Perceptron (MLP) and Recurrent Neural Networks (RNN) with Long Short Term Memory (LSTM).

The feature set is shown in Table 4. All the features were derived from the post themselves. No features derived from the forum structure or interactions between posts were used. We included additional language features such as MPQA, offensive language, and mental health lexicons.

During feature extraction, negation was model as in cimino2014linguistically . Thus, when a $term_{i}$ from a post is found in a lexicon, its negation is checked by inspecting the term $term_{i-1}$. As in cimino2014linguistically , we used a list of negation terms:

If a negation term was found, the polarity of the term was shifted when the lexicon differentiate between positive and negative terms (e.g., the PERMA lexicon); otherwise, it was skipped and not included as a feature.

The results of the triage classification experiment using the different features, including lexical resources and treating negation, are presented in Table 5. Best results are highlighted in boldface. Exclusive use of lexicon features resulted in lower performance for all classes (flagged, urgent and crisis) and the overall performance (macro F1-score). Treating negation when using only lexicons did not boost the classification performance. However, adding Term Frequency-Inverse Document Frequency (TF-IDF) contributed to improving the classification performance for all classes⁷⁷7TF-IDF is the amount of times a word appears in a document weighted by the number of meaningful words across multiple documents. Best results were achieved with features that included “TF-IDF + lexicons with negation”, F1-score of 0.44. The most complex set of features (included all features in Table 4) showed competitive results with the most state-of-the-art triage classification system by Altszyler:2018 , and the baseline classification system by milne-EtAl:2016:CLPsych .

We also experimented on a linear and nonlinear classification method. Naïve Bayes was trained with the Lightme feature set since it is an easy and fast linear classification method suitable for classifying large chunks of data. Similarly, KNN was trained with the same feature set due to its practicality and ease. Hyper-parameters such as the number of neighbours were selected with a grid search scheme using a range from 1 to 25. Table 6 shows the results of Naïve Bayes and KNN, which underperformed SVM and other state-of-the-art systems. Compare to Naïve Bayes and KNN, SVM is known to perform better on rich feature sets such as the one presented in this study.

MLP was trained using the same set of features as Lightme with hidden layer sizes that varied between 100 and 300 nodes, depending on the development set. The RNN+LSTM model was trained with pre-trained word embeddings and without features since one of the advantages of this type of model is its ability to learn feature representations automatically. Important hyper-parameters such as the number of epochs, size of the hidden layer and batch size were tuned using a portion of the training set as the development set. As shown in Table 6, the deep learning models underperformed the other models. This is not surprising as deep learning models are data-hungry and the size of Reachout is small, especially some of the classes (e.g., red and crisis) only have a few instances.

We randomly selected 40 crisis posts to analyse from the training dataset. We then used open coding to understand linguistic characteristics. Through the qualitative analysis of the selected crisis posts, we identified six linguistic characteristics. We extracted selected phases of crisis posts (including the post id) that matched the given linguistic profile, and suggested recommendation of features to the model.

This study demonstrates a solution that utilises a variety of lexicon-based resources and supervised ML techniques to assist trained moderators to efficiently moderate ISGs. Contrast to other similar research; this study extracted lexicon-based features from the textual content of posts which may avoid possible biases during classification. The classification experiment found one of our classifier (Liteme) achieved the best results for the crisis post (0.52 F1-score) and competitive results in the other classes (i.e., non-green, flagged, and urgent posts). These results may indicate that it is possible to build a strong classifier that can process only textual features extracted from individual messages. However, the experimental results also demonstrated that using only lexicons was not enough to classify posts into all relevant classes. Exclusive use of vocabulary in the Reachout dataset was built into the solution which may have introduced some noise that may have impacted on the classification performance of flagged and urgent posts. Furthermore, this study demonstrated the limitations of utilizing lexicons, especially their ability to only capture information at the ’word’ level. This may prevent their ability to understand the contextual meaning at the ’sentence’ level.

Furthermore, the findings suggest that using mental health lexicons can have an impact on the classification of posts requiring immediate response by trained moderators. This is unsurprising given the distinct domain-specific properties of lexicons, especially their association with certain mental and behavioural health theoretical constructs Kornfield:2018 . Lastly, six linguistic characteristics were identified in the qualitative analysis of crisis posts. Interestingly, we found a person in crisis will use words or language associated to hopelessness, publish short posts containing concise negative emotional responses, publish long posts containing variations of emotions, express dissatisfaction with locally available health services, use storytelling to express crisis, and seek the advice of peers during a crisis.

Our best classifier showed comparative results with the state-of-the-art systems for triage classification with the Reachout dataset and the baseline classifiers. The baseline system by milne-EtAl:2016:CLPsych used uni-grams and bi-grams as features, and a default scikit-learn logistic regression classifier scikit-learn . We also found the classification performance of the best system by Altszyler:2018 from the CLPsych 2017 Shared Task also utilized an SVM classifier. This classifier used a richer set of features, including features from the forum structure and interactions between posts, which outperformed our system for flagged and urgent posts. Interestingly, our approaches showed better results in identifying crisis posts. This is an important category for this problem, especially the moderator’s need to immediately respond to these posts. Adding features derived from the forum structure may help to improve the classification performance. However, the trade-off is the expense of not properly classifying posts from new users.

Similar to other systems, our triage text classifier found using TF-IDF with lexicons improved classification performance. kim-EtAl:2016:CLPsych received the best results in CLPsych 2016 Shared Task when using TF-IDF weighted n-grams and post embeddings using Sent2vec in an SGD classifier, and a set of twelve fine-coarse grained labels, instead of the coarse-grained four labels. The system by Brew:2016 also weights n-grams with TF-IDF producing similar results. Similarly, the use of TF- IDF showed comparative results to triage text classifiers for Twitter Odea:2015 , and an online support forum for substance abuse Kornfield:2018 .

The qualitative findings appeared to support prior research that found similar patterns of online interactions in people with mental ill-health using social media. As highlighted in previous research, online peer-to-peer interactions can improve health and psychosocial outcomes by facilitating a range of positive behaviours that can empower people, such as seeking information and emotional support Naslund2014 . However, these online networks can also become harmful when social media content begin to promote self-harm, suicide, or pro-eating disorder behaviours Gerrard2018 ; Dyson2016 . Particularly, social media posts that promote “problematic” content that may be difficult to identify by specifically moderating hashtags in online communities Gerrard2018 .

Most of the qualitative findings may be translated into features that could improve classification performance. As noted, the qualitative findings of the crisis posts could be used to distinguish salient linguistics characteristics of the language used in urgent messages for moderators. For example, specific features for detecting hopelessness may improve detection of crisis messages. Suggestions for future work may also include differentiating posts that provide support or seek advice to other peers and identifying participants roles, such as leaders, influence, and opinion users Carron-Arthur:2015 . Furthermore, various types of help-seeking behaviours can be identified, such as users wanting to share their personal stories of struggle Smithson:2011 . The analysis of satisfaction with available services can play a role in developing enhanced mixed reality care approaches combining eHealth and on-site services vanGenderen2018 .

There was a limitation to this study. First, our classifier is restricted to one dataset. More data is needed to generalize the model to avoid overfitting. Second, the training set was relatively small. This may have had implications to our approach and subsequent results. Third, an error analysis was not conducted. The error analysis could have examined why certain posts were misclassified or classified correctly.

The current study examines a triage classifier using features derived only from the textual content of the post. Various lexicons were used to analyse the value of lexical resources on the text classifier for triaging posts. Lexical resources alone were not enough to build a good performing classifier; however, a solution that includes lexicons with other features derived from the content of the posts performed well in identifying crisis posts. Qualitative investigation on the crisis posts found six salient linguistic characteristics. While qualitative findings are still formative, more work is needed to translate these findings into features that can improve the overall performance.