GLARE: Detecting Harmful Memes
by Combining Global and Local Perspectives
(Appendix)

We train all the models using Pytorch (paszke2019pytorch) on a NVIDIA Tesla T4 GPU, with 16 GB dedicated memory, with CUDA-10 and cuDNN-11 installed. For the unimodal models, we import all the pre-trained weights from TORCHVISION.MODELS¹¹1http://pytorch.org/docs/stable/torchvision/models.html subpackage of the PyTorch framework. The non pre-trained weights are randomly initialized with a zero-mean Gaussian distribution with a standard deviation of 0.02. From the dataset statistics table (Table 1 in manuscript), we can observe a label imbalance problem for both harmfulness intensity ([Very Harmful, Partially Harmful] vs. Harmless) and target ([Individual, Organization, Community] vs. Entire Society) classification tasks. To deal with the imbalance problem, we use focal loss (FL) lin2017focal, which down-weighs easy examples and focuses training on hard ones. We train GLARE in a multi-task learning setup, where the loss due to target identification is considered only if the meme is partially harmful or very harmful. We train our models using Adam kingma2014adam optimizer and negative log-likelihood (NLL) loss as the objective function. In Table 1, we furnish the details of hyper-parameters used for the training.

To assure a satisfactory quality of the Harm-C and Harm-P datasets, we impose four well-defined and fine-grained filtering criteria during the data collection and annotation process. The criteria are as follows:

1. 1.

   The meme text must not be in code-mixed or non-English language.

2. 2.

   The meme text must be readable. (e.g., blurry text, incomplete text, etc. are not allowed)

3. 3.

   The meme must not be unimodal, i.e. it should not contain only textual or visual content.

4. 4.

   The meme must not contain several cartoons. (we add these filtering criteria as cartoons are often very hard to be interpreted by AI systems)

Figure 1 shows some example memes which were rejected during filtering process due to the four criteria mentioned above.

We use the crowd-sourcing platform pybossa²²2https://pybossa.com/ (c.f. Figure 2) to build an annotation interface that shows each meme and request annotations for harmfulness levels and target. Before beginning the annotation process, we requested every annotator to thoroughly go through the annotation guidelines and conducted several discussion sessions to understand if all of them can understand exactly what harmful content is and how to differentiate it from humorous, satirical, hateful, and non-harmful content. The average inter-annotator agreement scores (Cohen’s $\kappa$) bobicev-sokolova-2017-inter for Harm-C and Harm-P $0.683$ and $0.790$, respectively.

This section analyzes the lexical (word-level) statistics of the Harm-C and Harm-P datasets. Figure 4 shows the length distribution of the meme text for both the tasks across two datasets. Furthermore, Table 2 shows the top-5 most frequent words by every separate class in the combined validation and test sets of two datasets. We observe that for very harmful and partially harmful classes, names of US politicians and COVID-19 oriented words are frequent. In the target classes, we notice the presence of various class-specific words such as ‘trump’, ’joe’, ‘obama’, ’republican’, ’wuhan’, ‘china’, ’islam’. To alleviate potential bias caused by these class-specific systems, we intentionally included harmless memes of related to these individuals, groups and entities, which is described more in the Section $4$ of main manuscript.