SLM-Mod: Small Language Models Surpass LLMs at Content Moderation

We curate our data from the publicly available dataset of Reddit comment removals between May $10^{\text{th}}$, $2016$ and February $4^{\text{th}}$, $2017$ by Chandrasekharan et al. (2018) by sampling 10K comments (5K moderated and 5K unmoderated) for 15 popular subreddits from Reddit’s landing page. For each subreddit, we split its data into 80/20 train/test sets. Appendix A includes the complete list, description, and subscriber statistics of the subreddits.

For our study, we evaluate the $4-$bit quantized versions of three small language models (SLMs): Llama-3.1-8b Dubey et al. (2024), Gemma-2-9b Team et al. (2024), and Mistral-nemo-instruct Mistral AI (2024), and three large language models (LLMs): Cohere’s Command R+ Cohere For AI (2024), OpenAI’s GPT-4o; and GPT-4o-mini OpenAI (2024). For the LLMs, a temperature of 0 was used to ensure consistency across runs, and a top_p of $0.75$ was used.

We fine-tune 15 subreddit-specific SLMs using rank $16$ Low-Rank Adaptation (LoRA) Hu et al. (2021) for $1$ epoch on a balanced sample of total 8,000 moderated and unmoderated comments.

The task for the language models is to determine a moderation outcome for a given comment when provided with the comment, its preceding context, and subreddit rules. For a language model $\mathcal{M}$ and a comment $\mathcal{T}$ along with its context $\mathcal{C}$, and subreddit rules $\mathcal{R},$ we prompt the model with a prompt $p.$ Consequently, the moderation task is:

where $\mathcal{D}_{\mathcal{M}}$ represents the moderation decision generated by the language model $\mathcal{M}$.

We define two sets of tasks: (a) In-domain Moderation, which involves moderating comments from the same subreddit on which the model was fine-tuned; (b) Cross-domain Moderation, which involves moderating comments from a subreddit different from the one on which the model was fine-tuned. Note that this distinction is primarily useful for SLMs. For off-the-shelf LLMs, the prompt is subreddit-specific but the models themselves are not, so we always prompt with the correct rules.

We use rule-based prompting, drawing on prior work Kumar et al. (2024), by querying the models with text, context (preceding comment), and rules from each subreddit using the following prompt $p$:

We obtained rules for each subreddit by querying Reddit using the PRAW API.²²2https://praw.readthedocs.io/en/stable/ For in-domain tasks, we use the rules of the original subreddit, whereas, for cross-domain tasks, we use the rules of the source subreddit since we assume that we do not have rules for the target subreddit.

For our evaluation tasks, we focus on the metrics of accuracy, precision, and recall. These give us a holistic picture of the efficacy of different models on content moderation tasks, along with an insight into how different models handle violating comments in terms of the precision/recall trade-off.

In this section, we compare the performance of SLMs versus LLMs on in-domain content moderation tasks. Figure 2 provides a visual depiction of the accuracy, recall, and precision across models.

In order to get a deeper understanding of how SLMs tackle content moderation and where they falter, we complement our quantitative results with a qualitative analysis. However, due to the scale and complexity of the data, qualitative analysis of the entire dataset is infeasible. Therefore, we focus on one specific community, r/changemyview that has been a community of interest in various prior content moderation works Srinivasan et al. (2019); Koshy et al. (2023); Jhaver et al. (2017). r/changemyview is a subreddit with 3.7M subscribers for debating opinions where users invite others to challenge their perspectives with thoughtful counterarguments, and if their view is changed, then they can award a delta ($\Delta$) to a commenter. Moreover, our fine-tuned SLMs showed strong overall performance on this subreddit, achieving an average of 91% accuracy, 90% precision, and 93% recall, so it would be valuable to further examine the few niche error cases to conduct additional exploration. These factors make it an interesting subreddit to qualitatively compare the differences between content moderation with SLMs versus LLMs, where looking at the false positives and negatives offers valuable insights into the types of errors made by the models.

We retrieved all comments from the test set where the SLMs made an error and inspected them both manually and computationally. Overall, there were 15.2% (152/1000) false positives and 11.5% false negatives (115/1000) in the moderation outcomes where at least one SLMs made a mistake.

In this section, we investigate the cross-domain performance of SLMs on moderation tasks. Figure 4 shows a visual depiction of the findings. We choose three communities examined in prior work Kumar et al. (2024), namely r/IAmA, r/askscience, and r/movies as our three subreddits for the discussion of cross-domain performance. Appendix D provides complete cross-domain results.

We observe that r/askscience is the subreddit with the lowest average cross-domain performance at 58% with the highest performance of 65% given by the Mistral fine-tuned for r/science and r/history. On the other hand, r/movies was the easiest of the three subreddits, with the highest average cross-domain performance at 68.6% and a highest performance of 80% given by the Mistral fine-tuned for r/aww and r/nba, and Gemma model fine-tuned for r/aww. Finally, for r/IAmA, the average accuracy was 63.7% and the highest performance came from the Mistral model fine-tuned for r/politics at 75%.

It is noteworthy that while the moderation accuracy of cross-domain models is not at par with in-domain SLM models, the best-performing cross-domain SLMs outperform best-performing LLMs for these subreddits, with relative advantages of 6.5% for r/IAmA, 11.6% for r/askscience, and 0.1% for r/movies, indicating the promise of cross-community based approach for content moderation using SLMs over using LLMs.

Furthermore, we see that there are some models that show a promising overall cross-domain performance. Specifically, models trained on r/changemyview, r/nba, and r/movies show an average cross-domain accuracy between 64.6 and 67.8%, and can therefore be used as ‘meta-experts’ to moderate communities that would benefit more from a cross-domain moderation as compared to in-domain moderation. One example is r/AskReddit, which would benefit from the cross-domain models from 10 other communities compared to its in-domain model based on Llama-3.1-8b and Gemma-2-9b, and 7 others for Mistral-NeMo-Instruct.

Due to the promising performance of cross-domain models, we further investigate why some subreddits may benefit from cross-domain moderation, and whether there is a pattern to the cross-domain accuracy of an SLM on a target subreddit by testing the impact of subreddit size, description and rules. Our hypothesis is that relative sizes of subreddits can play a role in determining content moderation performance across communities as similarly-sized communities may have similar underlying norms. Accordingly, the similarity of topics as well as community rules between subreddits may have a direct impact on cross-community content moderation outcomes, due to the similarity in content or in the way in which the content is moderated.

For size, we construct a matrix of relative number of subscribers of the source subreddit w.r.t to the target subreddit, and for description and rules, we get the subreddit description and rules using PRAW and use Cohere’s embed-english-v3.0 Cohere For AI (2023) to generate embeddings and compute two cosine-similarity matrices between pairs of source and target subreddit descriptions and rules.

We then conduct a column-wise (in this case, subreddit-wise) t–test for non-correlation between the relative-size matrix, and the cosine similarity matrices for description and rules, with the cross-domain accuracy matrices for each model, using the Pearson correlation coefficient Schober et al. (2018) as our test-statistic ($r$). The alternative hypothesis was that there exists a positive correlation.

We find that there was no statistically significant ($\alpha=0.05$) positive correlation between relative size and the performance of cross-domain models on a subreddit. For subreddit description, we find two statistically significant positive correlations with Llama-3.1-8b on r/AskHistorians ($t(12)=0.459,p=.049$) and Gemma-2-9b on r/askscience ($t(12)=0.535,p=.024$). Finally, for correlation between subreddit rules and cross-domain performance we again find only two significant positive correlations with Llama-3.1-8b on r/nba ($t(12)=0.462,p=.048$) and Gemma-2-9b on r/anime ($t(12)=0.459,p=.049$). Full results of the t-test can be found in Appendix E.

This result signifies that while the topic of the subreddit and the community rules may play some role in determining the cross-domain performance, the overall notions of what determines the cross-community performance go beyond just subreddit sizes, description, and rules.

Our findings indicate a significant advantage in employing SLMs for content moderation tasks over LLMs, consistently showing superior performance in terms of both accuracy and recall. Our findings suggest that SLMs adopt a more aggressive approach compared to LLMs, resulting in significantly higher recall for SLMs, though at the cost of slightly lower precision relative to LLMs. However, prior work has shown that moderators, who are overburdened already, may not be able to attend to all instances of norm-violating and undesirable behavior Park et al. (2022); Chandrasekharan et al. (2019). Hence, the higher recall of SLMs compared to LLMs could actually provide a benefit for more reliable detection of harmful content that could otherwise be overlooked and stay on the platform for longer—potentially leading to further undesirable outcomes Lambert et al. (2022).

Apart from performance, a key strength of SLMs lies in their ability to be fine-tuned for specific communities, allowing for moderation that aligns closely with the unique needs of individual subreddits. Additionally, specialized communities and those serving sensitive populations Saha et al. (2020) may require extra safeguards and considerations when using automated content moderation tools, which can be more easily fine-tuned with SLMs. This approach improves moderation accuracy by better reflecting the decisions of community-specific moderators. In addition, community norms can evolve over time, and performing continual pretraining Ke et al. (2023) on SLMs with incrementally collected data can help the models stay updated in accordance.

Finally, SLMs for content moderation are a cheaper and more scalable option for platforms like Reddit, which manage large volumes of user-generated content and provide them agency over their models and data without the reliance on third-party APIs. LLMs, on the other hand, are expensive to query, rate-limited, and mostly closed-source.

The precision/recall trade-off between LLMs and SLMs discussed in the previous section also has implications for their potential usage from a moderation design perspective. Specifically, content moderation can either be done in an automated manner with no moderator involvement or in a moderator-in-the-loop manner where triaged and reported comments are manually moderated.

Since LLMs are better at accurately identifying violating comments while minimizing false positives, they are more suited to be used as automated moderation tools as they are less likely to wrongfully penalize community members by moderating potentially benign comments due to their cautious approach. SLMs, on the other hand, are more aggressive at flagging comments with higher recall, which makes them more suited to scenarios where the priority is to ensure that potentially harmful content is quickly triaged and sent for further review by human moderators. This will ensure the flagging of seemingly undesirable behavior in a time-sensitive manner and not leave such content visible on the platform for long periods, and upon manual inspection, if the comment is benign, it can be allowed to exist by the moderator.

We observe that along with great in-domain performance, SLMs also perform well in cross-domain settings which suggests that norm violation representations learned by fine-tuned SLMs can generalize effectively across different online communities, making them a viable option for moderating content in new or growing communities.

Moreover, we saw that certain ‘meta-expert’ models have high average cross-domain performance, often providing certain communities with more performance than their in-domain models. This indicates that SLMs have the capability to learn shared norm violation representations, and by leveraging cross-community similarities, cross-domain SLMs can be used for moderation without needing specific training data for every new community, thus enabling faster and more cost-efficient deployment of automated moderation tools across platforms. Specifically, a new community developing its rules can make use of cross-domain experts to identify which expert provides the community with the highest content moderation performance, and use that to inform their own community rules.

However, we show that determining which cross-domain expert would provide a community with the highest benefit is a nuanced task that goes beyond measures like similarity in subreddit sizes, descriptions, and rules. This provides an avenue for future research to explore the development of strategies to identify the right set of cross-community experts as well as meta-moderation models. Using a mixture of experts (MoE) framework, for example, could allow a community to draw from specialized models trained on different clusters of subreddits with shared themes, rules, or behaviors. This would enable automated moderation systems to dynamically route content to the most appropriate expert and ensure that moderation decisions are well-aligned with specific community norms.

Since closed-source models undergo frequent updates, it is crucial to assess the stability of their performance in content moderation tasks. To analyze how different releases of the same model can affect performance, we compare the initial release of OpenAI’s GPT-4o model (May 13, 2024) with the latest version (August 06, 2024).

From Table 1, we observe that the accuracy of GPT-4o declined from 67.7% to 65.8%, accompanied by a huge reduction in recall from 55.7% to 44.7%. Conversely, the average precision improved from 73.6% to 76.3%. Crucially, 10.7% or 2,166 comments across all subreddits that were correctly moderated by the May release of the model, were no longer moderated by the August version, highlighting a lack of trust that could be placed in these closed-source LLMs for content moderation.

Given the black-box nature of closed-source models and the lack of knowledge of upcoming updates, ensuring consistent moderation performance would pose a challenge when using closed-source models. Moderators may instead consider using open-source language models, which tend to be more stable since moderators can choose not to update the models and can be fine-tuned more easily to meet specific moderation requirements.

We chose 15 highly popular subreddits for our study with a good mix of science, AMA, entertainment, history, sports, and political subreddits in order to capture a wide spectrum of underlying norms and subreddit characteristics. We therefore believe that our findings are representative and scalable to a wider range of communities. However, future studies could expand the scope to incorporate a larger number of subreddits for increased robustness of our findings and more open-sourced models for content moderation.

Due to availability of publicly available text-based datasets for Reddit comment removals, we resorted to purely text-based content moderation. However, undesirable behavior may occur in multiple other modes like images and future studies can expand on our insights to explore the performance of vision-language models in content moderation settings.

Community norms and notions of undesirable behavior may change over time and models fine-tuned on a specific date range of posting activity may not necessarily generalize to newer comment removals or an evolved representation of content that moderators would remove. Therefore, future studies can evaluate the adaptability of models on temporal distribution shifts in online content and determine the efficacy of techniques like domain-adaptive pretraining (DAPT) Gururangan et al. (2020) and continual pretraining Ke et al. (2023) in keeping SLMs update with the latest norms.