Large Language Models for Automatic Detection of Sensitive Topics

Sensitive and harmful information detection is always an important topic in many areas. Many online forums and social media platforms traditionally employ human moderators to review and manage content manually. This approach is often inefficient. To improve efficiency, some crowd-sourced workers have been introduced in this field (Hettiachchi and Goncalves, 2019; Bozarth et al., 2023), but this approach, at the same time, has increased costs. Recently, many studies have explored using automated content detection, often machine-learning-based, to assist human moderators, aiming to replace manual review methods with higher efficiency and lower cost.

Automated content detection has been extensively researched in the context of preventing cyberbullying and cybercrime (Slonje et al., 2013). For example, Al-Garadi et al. (Al-Garadi et al., 2016) developed a machine learning model to detect cyberbullying on Twitter, analysing network, activity, user, and content features from 2.5 million geo-tagged tweets. They achieved high detection accuracy and precision using classifiers like Random Forest, proposing a tool for monitoring and mitigating online bullying. Similarly, Zhao et al. (Zhao et al., 2016) proposed a new model that enhances traditional text classification by integrating bullying-specific features derived from word embeddings. Their model showed superior performance compared to baseline methods. Dadvar and De Jong (Dadvar and De Jong, 2012) integrate user characteristics and behaviours across social networks into a new model to detect cyberbullying. Their study offers a tool for more accurate cyberbullying detection and enhanced support for victims and moderators.

One important domain for identifying sensitive information is E-health, which is a sensitive area that requires careful handling. Huh, et al. (Huh et al., 2013) developed a text classification system to assist moderators in identifying posts that need their expertise in online health communities. They used a binary Naive Bayes classifier to categorise posts into those needing a moderator’s response and those that do not. This approach provides a low-cost, scalable solution to help moderators efficiently prioritise posts requiring their attention, thereby enhancing support in large-scale health forums.

There are also some studies about suicidality detection. O’Dea et al. (O’dea et al., 2015) developed a machine-learning model to detect suicidality on Twitter. They collected tweets containing suicide-related phrases and used human coders to classify the level of concern. These tweets were then used to train a classifier. The classifier achieved an overall accuracy of 76%, correctly identifying 80% of “strongly concerning” tweets. Cohan et al. (Cohan et al., 2017) also used machine learning to build a system to automatically triage the severity of posts in online mental health forums. The system categorises posts into four levels of severity, from no risk to imminent risk of self-harm, using a feature-rich classification framework that includes lexical, psycholinguistic, contextual, and topic modelling features. The system significantly improved the identification of high-risk posts, achieving up to a 17% improvement in F1 scores, an indicator of precision and recall capability, compared to previous methods.

Researchers also looked at peer support, which is beneficial for young people to overcome barriers in help-seeking behaviour. Milne et al. (Milne et al., 2019) developed a machine learning system to triage posts in online peer support forums, categorising messages by urgency. Using a classifier, it achieved 84% accuracy, improving moderator efficiency and response times on online forums. Milne et al. (Milne et al., 2016) organised the “CLPsych 2016 Shared Task” to develop systems for automatically triaging posts in online mental health forums, specifically “ReachOut.com”. Participants classified posts into four categories: Green (no immediate action needed), Amber (moderate concern), Red (high concern), and Crisis (immediate action needed). Using various machine learning techniques, the best-performing systems significantly outperformed baselines.

From prior studies, we found that machine learning is the most common method for building AI-supported detectors or moderators. It can achieve high accuracy and reduce detection time. Some have already been integrated into practical workflows. Jhaver et al. (Jhaver et al., 2019) conducted a study of Automoderator (Automod) on the Reddit platform. They interviewed 16 Reddit moderators, including the creator of Automod, to explore the benefits and challenges associated with automated content detection technologies. The result indicated that Automod significantly reduces the workload of human moderators by handling menial tasks and protecting them from emotional labour. However, using Automod also creates new problems for human moderators. They must check for incorrect decisions, as Automod lacks contextual sensitivity and struggles with cultural and linguistic nuances. Based on the forum regulations, the tool requires regular updates and fine-tuning, adding to the workload of already limited skilled moderators. Additionally, moderators may find it difficult to fully understand and control Automod’s actions, especially when it makes mistakes or its decisions are unclear, it’s not fully transparent. At last, the paper highlighted the need for a balanced approach that combines automated and human moderation to maintain effective content regulation.

Many scientists have raised concerns about using technology-assisted tools for content detection (Udupa et al., 2023; Siapera, 2022; Nahmias and Perel, 2021). Gillespie (Gillespie, 2020) mentioned the inability of these tools to grasp context and nuance, leading to errors in detecting sarcasm and cultural subtleties. Additionally, Gillespie is concerned about the risk of AI perpetuating biases from training data, potentially impacting marginalised groups unfairly. He emphasised the need for human oversight in moderation to handle complexities and ethical considerations, arguing that AI should assist rather than replace human moderators. He pointed out that “the kind of judgment that includes the power to ban a user from a platform should only be made by humans.” Gorwa et al. (Gorwa et al., 2020) examined the technical and political challenges of algorithmic content moderation on major platforms like Facebook, YouTube, and Twitter. Their study underscored the opaqueness and accountability issues of automated moderation and similarly emphasised the need for human oversight to address the complex and contextual nature of content moderation effectively.

The rapid development of LLMs introduces a new method in this domain, changing the way humans interact with machines (Abilaashat, 2023). LLMs possess powerful text-generating and understanding capabilities (Wu et al., 2023). Users can ask AI to explain each decision in natural language simply by using natural language prompts, making communication straightforward. In such cases, integrating LLMs to replace traditional machine learning methods becomes a viable choice for automated content detection.

Wang et al. (Wang et al., 2023) evaluated six popular LLMs for detecting and handling harmful instructions. LLaMA-2 showed the best safety performance, while ChatGLM2 was the least effective. Commercial models like GPT-4 often rejected harmful instructions directly. He et al. (He et al., 2024) compared GPT-4’s data labelling with 415 crowd-sourced workers from the MTurk platform on 3,177 sentence segments from 200 scholarly papers. GPT-4 achieved 83.6% accuracy, while MTurk reached 81.5%. Combining their labels improved accuracy to 87.5%. The study shows that integrating human and AI labelling enhances accuracy, highlighting the value of combining crowdsourcing with advanced AI models. Kolla et al. (Kolla et al., 2024) evaluated GPT-3.5’s effectiveness in moderating Reddit content, analysing 744 posts across nine subreddits. They found GPT-3.5 had a high true-negative rate (92.3%) but a low true-positive rate (43.1%), indicating it struggled to accurately flag rule violations. While it handled keyword-based violations well, it missed more complex cases. The study suggests a hybrid approach, combining GPT-3.5 with human moderators, is necessary for effective content moderation. This research investigates five models in our formal study, including GPT-4, GPT-4o, GPT-4-Turbo, Llama-3, and Solar.

The results are shown in Table 1. True Positives (T+) represent the number of correctly identified sensitive data points. True Negatives (T-) represent the number of correctly identified non-sensitive data points. False Positives (F+) represent the number of non-sensitive data points incorrectly identified as sensitive. False Negatives (F-) represent the number of sensitive data points incorrectly identified as non-sensitive.

In this study, the evaluations of multiple LLMs suggested the promising potential of using LLMs in detecting sensitive topics as defined. Models like Claude-3-Opus-20240229, GPT-4-Turbo, and Llama-3-70B-8192 demonstrated high accuracy in processing data. Other models, such as GPT-3.5, were less accurate. The fine-tuned GPT-3.5 model showed a good ability to detect sensitive topics but tended to be overly strict. However, being overly strict can be beneficial when dealing with mental well-being-sensitive topics to ensure safety.

Additionally, prompt engineering proved significant. During the study, we found that GPT models showed higher accuracy when the input was in JSON format, as shown in Appendix  A. While the Llama-3 model excelled at processing natural language prompt input, the claude-3 model was proficient at both.

We chose two open-source, real datasets: “Topical-Chat” (Gopalakrishnan et al., 2019) and “Mental-health-counseling-conversations” (Amod, 2023). Topical-Chat is a knowledge-grounded human-human conversation dataset focused on daily life, with each message labelled with sentiments such as ‘Curious to Dive Deeper’, ‘Happy’, ‘Neutral’, ‘Surprised’, ‘Disgusted’, ‘Sad’, ‘Fearful’, and ‘Angry’. We selected messages labelled Happy to form the non-sensitive topic dataset, there were 683 messages. Mental-health-counselling-conversations is collected from two online mental counselling and therapy platforms. We used it to form the sensitive topic dataset, resulting in 996 messages.

We pre-processed the two datasets, removing duplicates and messages that potentially violated laws and AI platform usage policies (e.g., content related to paedophilia, Nazis, rape, etc.).

Three members of the research team participated in data coding. They are all postgraduate students: two are HCI researchers, and one has a background in psychology. Based on the preliminary study results, dataset preprocessing, and prior studies, human coders revised the topic scope to include climate change (Hickman et al., 2021) and finance issues (Lin et al., 2022) as sensitive topics. We also improved the prompt in JSON for the GPT, see Appendix B. The final instruction of sensitive topic scope is:

‘Suicidal Ideation or Self-Harm or related topics,’ ‘Sexual Assault and Abuse or related topics,’ ‘Severe Mental Health Disorders or related topics,’ ‘Substance Abuse or related topics,’ ‘Eating Disorders or related topics,’ ‘Trauma and PTSD or related topics,’ ‘Grief and Loss or related topics,’ ‘Strong Emotion or Stress,’ ‘Family Issues or related topics,’ ‘Identity and Discrimination Issues or related topics,’ ‘Legal Issues and Crime or related topics,’ ‘Physical harm or related topics,’ ‘Financial problems or related topics,’ ‘Climate change concerns or related topics,’ ‘General stress and bad mood or related topics,’ ‘Anything else that could cause mental health issues, stress, or emotional damages,’ and ‘Long sentences with more than 30 words’.

According to the instructions, two coders reviewed both the sensitive and non-sensitive topic datasets, checking and labelling each message. The third coder, with a psychology background, arbitrated disagreements. They selected 500 messages with the ‘sensitive’ label and 500 messages with the ‘non-sensitive’ label to form the balanced-data baseline dataset.

Based on the results of the preliminary study and the accessibility of the API for batch processing, we selected three proprietary models: GPT-4, GPT-4o, GPT-4-Turbo, and two open-source models: Llama-3-70b and Solar-1-mini-chat. For each model, we measured performance at a temperature setting of 0.0, conducting five repetitions to assess consistency. Additionally, we tested each model at three different temperature settings (0.3, 0.5, and 0.7) to analyse the impact of temperature variations on performance, with each experiment repeated three times. The number of samples was confirmed by a statistical power analysis (Faul et al., 2007b). We performed a power analysis with G*Power and calculated Cohen’s d based on our data (Faul et al., 2007a), and the result (Actual Power = 1, alpha = 0.05, effect size f = 13.39 ) indicated that the sample size was sufficient, with a power close to 1, to detect statistically significant effects at an alpha level of 0.05.

We employed several common performance metrics to evaluate these models, including Accuracy, Precision, Recall, and F1-score (Dalianis and Dalianis, 2018; Musicant et al., 2003), which have been used in prior studies (Zhao et al., 2016; Al-Garadi et al., 2016).

Accuracy was calculated as the proportion of correctly classified messages out of the total number of messages, with the baseline being (100%).

Precision was measured as the proportion of messages identified as sensitive by the model that were actually sensitive according to the labelled data.

Recall was determined as the proportion of actual sensitive messages correctly identified by the model.

The F1-score was a crucial metric for evaluating model performance because it balanced precision and recall. Precision measured the accuracy of positive predictions, while recall measured the model’s ability to identify all actual positive instances. The F1 score, being the harmonic mean of precision and recall, provided a single value that reflects both the accuracy and completeness of the model’s positive predictions. This was especially important in the online counselling context, where both true positives and false negatives have significant implications.

Table 2 shows each model’s performance results. We also considered the model’s performance on non-sensitive data to ensure a comprehensive evaluation by measuring the true negative rate. We used IBM SPSS Statistics 27 to analyze the results. The study was conducted in the following steps:

To answer RQ1—“What is the impact of different LLMs on detecting sensitive topics?”, we computed these metrics for each model under the temperature setting of 0.0, averaging the results across the five repetitions to assess their performance. We then calculated the standard deviation of Accuracy and F1-score for each model at this temperature to evaluate consistency, with lower standard deviations indicating higher consistency. We conducted the Shapiro-Wilk test and one-way ANOVA to determine whether the performance differences between models were statistically significant. Following this, Tukey’s post-hoc test was used to identify specific significant differences.

To answer RQ2—“How does the temperature parameter setting affect LLMs’ performance in detecting sensitive topics?”, we analysed the impact of different temperature settings on model performance. The temperature setting in text generation for LLMs controls the randomness of the output, with lower values producing more deterministic and focused text and higher values generating more diverse and creative responses. Adjusting the temperature allows modulating of the model’s behaviour to suit different applications, balancing coherence and variety in the generated text. For each model, we computed the same metrics, Accuracy, precision, recall, and F1-score, at three different temperature settings, averaging the results across the three repetitions. This allowed us to compare the performance of different models at the same temperature and to assess the overall performance of each model under varying conditions.

Additionally, we conducted an error analysis to identify common misclassification patterns. We performed a case study by randomly selecting misclassified instances to understand the reasons behind the models’ incorrect predictions. The results are visualised using bar charts (see Fig.2 and 3) to compare each model’s Accuracy, precision, recall, and F1-score at different temperatures. Line charts (see Fig. 4, 5, and 6) were used to display the performance trends of individual models across different temperatures.

We tested each model at different temperatures, including 0.3, 0.5, and 0.7, three times and then calculated the results to investigate how the models performed at each temperature. We also tested each model in 0.0 three times as the baseline to provide a benchmark.

We analysed the failed cases encountered in this study. Based on our analysis of errors in LLM responses, we categorised the errors into two main types: response failure and incorrect response. Response failure refers to instances where the LLM did not return responses in the specified format (Boolean values) set in the prompt. Incorrect response refers to instances where the LLMs returned responses in the specified format but incorrectly assessed the sensitivity of the message.

The results of this study indicated that two proprietary models from OpenAI, GPT-4o and GPT-4, performed better for the current task, which is consistent with the result of a well-known LLMs benchmark test, Multi-task Language Understanding (MMLU) (Code, 2024). Among them, GPT-4o slightly outperformed GPT-4 in the mean of true positive rate, which is important in the mental well-being domain. Considering the significantly higher cost of calling the GPT-4 API compared to GPT-4o, GPT-4o was the best-performing model in this test.

Additionally, we found that GPT-4-Turbo ($T^{-}$=100%, Temperature=0.0) and Solar-Mini-Chat ($T^{-}$=99.96%, Temperature=0.0) models excelled at detecting non-sensitive topics, even outperforming GPT-4o in this regard, but were significantly weaker at detecting sensitive topics.

We also had an interesting minor finding: when we sent JSON-formatted prompts to call Solar’s API, the accuracy for detecting sensitive messages was relatively high ($T^{+}$ Mean=88.91%). Conversely, when we sent natural language prompts, the accuracy for sensitive messages was relatively lower ($T^{+}$ Mean=80.56%), but the accuracy for non-sensitive messages showed no significant difference. Since sensitive messages were generally longer and more complex than non-sensitive ones, we speculated that using JSON-formatted prompts could improve the performance of Solar and the various GPT-4 models in handling lengthy and complex text information.

As demonstrated in this study, LLMs efficiently detected sensitive content, suggesting a new approach for automatic moderation. Traditional automatic moderation relied on machine learning, which was more complex and less accurate. For instance, Huh et al. (Huh et al., 2013) trained a binary classifier for online health forums, achieving an F1 score of 0.54. With techniques like feature selection and data balancing, machine learning models’ performance improved, as seen in Al-Garadi et al.’s study (Al-Garadi et al., 2016) on cyberbullying detection on Twitter, which achieved an F1 score of 0.94 using a Random Forest classifier and SMOTE. However, our zero-shot LLMs achieved F1 scores ranging from 0.90 (Solar) to 0.99 (GPT-4o).

LLMs are rapidly advancing. Kolla et al. (Kolla et al., 2024) developed a GPT-3.5-based moderator for Reddit, which effectively detected compliant posts ($T^{-}$ = 92.3%), but had a low True Positive rate (43.1%) for non-compliant posts. In our study, with the latest LLM model GPT-4o, it reached a true-positive rate of 99.68% and a true-negative rate of 99.28%.

Current LLMs are technically proficient enough to support humans in sensitive content detection. Although discrepancies existed between LLMs and human coders, similar inconsistencies were found among human coders themselves. The mean rate of agreement among human coders was 98.6% (14/1000), while LLMs in the study surpassed this rate, indicating their potential for integration into moderation systems, as shown in Fig. 7. LLMs could replace human moderators in reviewing routine information from online forums on topics like peer support (Naqshbandi et al., 2016), health information (Huh et al., 2013), social media (O’dea et al., 2015) and other related topics (Delort et al., 2011). Humans would only need to process content flagged by the detection system. This approach can also be utilised in human-computer interaction research, such as chatbots and digital humans, to ensure safe interaction.

This study also revealed several technical challenges when utilising LLMs for system development. Firstly, the result suggested that batch processing via API calls could reduce accuracy compared to individual processing, posing design challenges for content moderation systems. Handling large volumes of information may require multiple APIs for parallel processing to ensure both accuracy and speed. Secondly, according to GPT-4o, “when the temperature is set to 0.0, the behaviour of the LLMs becomes more deterministic. This means the model will almost always choose the highest probability word for a given input, resulting in more consistent and repetitive outputs”. However, despite setting the temperature to 0 in our study, some inconsistencies remained, indicating that LLMs can still be unpredictable. This could be due to factors such as GPU parallel processing non-determinism and the use of sparse expert models that introduce minor variations (Community, 2023).

In the future, we can test controlling output randomness by adjusting the TopK and TopP parameters. Nevertheless, some degree of randomness will always be present, reminding us that we should not fully rely on LLMs output for these tasks; human involvement is still necessary. Thirdly, many failed cases in this study revealed that lacking background information can lead to incorrect judgments by LLMs. Providing as much contextual information as possible can improve accuracy when utilising LLMs as moderators or detectors. Lastly, while the ability to explain decision logic in natural language makes generative AI more understandable, the training process and underlying mechanisms remain opaque to the general public, creating trust barriers. This is an area worth considering for major AI companies like OpenAI in the future.

Although large companies like OpenAI and Anthropic have clearly stated their data privacy policies (OpenAI, 2024; Anthropic, 2024), ethical challenges remain in the practical application of LLMs for processing information related to mental well-being. Users may worry that their disclosed privacy could be secretly acquired, stored, and used for model training by companies. This concern may lead to users avoiding self-disclosure and developing distrust towards online platforms, which is crucial for mental well-being care (Marino et al., 2016; Brown et al., 2009). To address this, it is better to choose models that can be deployed locally, like Llama-3 70B or use anonymised information to mitigate privacy concerns. Additionally, users should have the option to decide whether the LLM can perform detection in the system. It is essential to explain in detail how the LLM operates within the system and how users’ privacy data will be protected.

As Gillespie pointed out (Gillespie, 2020), using LLMs as moderators for online communities raises the ethical question of whether a machine should be able to delete content posted by humans or even ban human accounts. We propose that LLMs should be used only as detectors to support human moderators in detecting certain information. No decisions concerning users should be made without human participation. LLMs, in our view, should serve as tools for human moderators rather than replacing them.

There are several limitations in this study:

Firstly, we only tested with zero-shot prompting to establish the baseline performance of the models. Previous studies have shown that few-shot chain-of-thought prompting can significantly improve LLM performance (Martínez, 2024). Fine-tuning is another method to enhance LLM capabilities in specific areas.

In the future, we aim to investigate improving LLM performance through few-shot prompting, fine-tuning, and adjusting parameters like TopK and TopP. Secondly, we used two online datasets that both collect data from the USA. This may lead to biased results, as people from different cultures might have varying mental health issues and different levels of willingness to self-disclose psychological issues. For example, users in these datasets often directly describe problems, while people from other cultures may communicate more indirectly due to fear of social stigma. This cultural difference could challenge LLMs in understanding indirect statements and metaphors. We plan to include participants from more diverse cultural backgrounds in future studies.

Thirdly, this study focused specifically on mental well-being topics. In practice, more topics might be considered sensitive. Therefore, further studies should evaluate LLM capabilities in a broader and more general context. Lastly, due to the rapid development of LLMs, many advanced models, such as Qwen2 ³³3https://qwenlm.github.io/blog/qwen2/, were released after our study was conducted, and we could not test their capabilities. However, we provided a framework for testing LLMs in detecting sensitive content during the automatic moderation process and two datasets for testing. In the future, we will continue testing new models and hope to engage with more stakeholders to establish guidelines for using AI in detection and moderation.