Towards Patronizing and Condescending Language in Chinese Videos: A Multimodal Dataset and Detector

In this section, we outline the construction of the PCLMM dataset. We developed a comprehensive semantic definition of PCL in Chinese to create annotation guidelines. Using six key vulnerable community categories from the Chinese internet, we compiled a keyword list and collected videos via targeted searches. The dataset was manually annotated by three annotators, followed by sentiment and toxicity analysis.

PCL typically targets vulnerable groups, but this definition often doesn’t align with the Chinese context, where the concept of vulnerable groups differs from that in English-speaking communities. For example, PCL toward immigrants is rare in China due to policy reasons. Building on [4, 6], we proposed a comprehensive definition of PCL tailored to the Chinese context, serving as our annotation guide.

Chinese PCL refers to discriminatory, falsely sympathetic, and hypocritical remarks directed at six vulnerable groups within the Chinese community: disabled individuals, women, the elderly, children, single-parent families, and low-income groups. A key feature of PCL is the speaker’s condescending attitude, making statements that do not improve the group’s situation. PCL expressions are often accompanied by contemptuous and discriminatory facial expressions. To minimize subjective discrepancies, we specified the following cases to be annotated as non-PCL:

• •

  Vulnerable individuals describing their own experiences of unfair treatment.

• •

  Objective news reports on discriminatory incidents.

• •

  Public service announcements containing discriminatory content but lacking discriminatory intent.

Based on II-B, we identified six major vulnerable communities on the Chinese internet. We expanded each community into a list of 10 commonly used keywords and designed a lexicon of offensive and discriminatory terms as query keys. These were matched with the keyword list to generate the final search set (e.g., adding the query discrimination to the keyword elderly care). Our search list included 1800 keyword-value pairs, retrieving 2654 preliminary videos. We retained videos ranging from 30 seconds to 5 minutes and filtered out damaged and irrelevant videos, and we also used masking techniques to obscure all possible watermarks that might disclose user privacy. Finally, we got 715 high-quality annotable samples.

Two trained Ph.D. students annotated the videos, with a third as the reviewer (two males, one female, aged 25-30, all in computer science and focused on toxicity detection). Annotators were compensated based on the number of annotations completed. Videos were labeled as PCL or non-PCL following II-B. An initial set of 30 videos (20 non-PCL, 10 PCL) was used to reach consensus on discrepancies. To minimize harm, annotators were limited to 20 videos per day and reported their psychological state. The CVAT tool [13] further recorded facial expression spans for PCL videos, while non-PCL videos had no patronizing spans. Fleiss’ Kappa [14] measured inter-annotator agreement (IAA = 0.72), with the third annotator resolving discrepancies. Finally, we obtained 196 PCL and 519 non-PCL videos.

The PCLMM dataset contains 715 videos, totaling 21 hours of content, with an average video length of 1.80 mins and a frame rate of 30 FPS, comprising 2.3M frames. Approximately 27.4% of the videos were labeled as patronizing, aligning with the distribution of PCL data on internet platforms. Detailed dataset statistics are shown in Table I.

Given a dataset of video samples $V$, the task is to classify videos targeting vulnerable groups as either PCL ($y=1$) or non-PCL ($y=0$). Each video $V$ is represented by a sequence of frames $F=\{f_{1},f_{2},\dots,f_{n}\}$ and a subset of facial expression frames $F_{v}=\{f_{1v},f_{2v},\dots,f_{nv}\}$. If a frame $f_{n}$ lacks facial expressions, $f_{nv}$ is filled with a zero vector. The audio sequence is $A=\{a_{1},a_{2},\dots,a_{l}\}$, and the transcribed text sequence is $T=\{w_{1},w_{2},\dots,w_{m}\}$. The goal is to develop an attention-based multimodal classifier $X:X(F;F_{v};A;T)\rightarrow y$, where $y\in\{0,1\}$.

We used the Vision Transformer (ViT) [17] to extract features from videos. Given a sequence of frames $F=\{f_{1},f_{2},\dots,f_{n}\}$, ViT extracted feature vectors for each frame $f_{i}$. The feature vector $\mathbf{z}_{i}$ is computed as:

Here, $\mathbf{z}_{i}\in\mathbb{R}^{d_{v}}$ is the $d_{v}$-dimensional feature vector encoded by ViT for each frame $f_{i}$.

To capture patronizing facial expressions in the video, we first used MTCNN (Multi-task Cascaded Convolutional Networks) [18] for face detection. Next, FER-VT (Facial Expression Recognition using Vision Transformers) [19] encoded facial features using grid-wise attention and visual transformers to capture long-range dependencies. For each video frame $f_{i}$, if MTCNN detected a face, FER-VT extracted the facial feature vector $\mathbf{z}_{i}^{v}$; otherwise, a zero vector was assigned.

We used FFmpeg [20], a widely-used multimedia package, to extract high-quality audio from the videos, and then applied the Mel Frequency Cepstral Coefficient (MFCC) [21] to extract audio features. The extracted audio sequence $A$ was encoded as $\mathbf{z}^{a}$.

We used Whisper [22], a speech recognition model by OpenAI, to transcribe audio into text. For text encoding, we utilized RoBERTa-Chinese [23] and a fine-tuned RoBERTa trained on the CCPC dataset [6] for patronizing language detection (We call it BERT-PCL). These models extract the CLS token from each transcript, producing a feature vector $\mathbf{z}^{t}$.

In our model, we used a unified Cross-Modality Multi-Head Attention (MHCA) mechanism to fuse information across different modalities. The general form of MHCA is:

Here, $\mathbf{Q}_{i}$ is the query from modality $i$, and $\mathbf{K}_{j}$ and $\mathbf{V}_{j}$ are the key and value from modality $j$. By varying $i$ and $j$, the interaction between different modality pairs is expressed as:

The resulting attention features are then aggregated into a unified multimodal representation:

We employed the BCEWithLogitsLoss as our loss function, which is suitable for binary classification tasks. The loss is computed as:

Our experiments were conducted using two NVIDIA A800-80G GPUs with 5-fold cross-validation to ensure robust training. We trained for 20 epochs, averaged the top five performances, and used a batch size of 10 with a learning rate of 1e-4. All code was implemented in PyTorch. Evaluation metrics included Precision, Recall, F1-score, and Accuracy, standard in toxicity detection. Notably, the ViT architecture provides an efficient solution, making it ideal for multimodal models. Its performance in short video analysis matches that of VideoMAE [24], which is why we chose ViT as the baseline instead of VideoMAE for modality fusion.

We employed a strategy of progressively integrating multiple modalities, beginning with single modalities. The experimental results are presented in Table II. (1) In single-modality scenarios, the text modality yielded the highest detection performance, while using audio alone resulted in poor outcomes, underscoring the ongoing importance of text in toxicity detection. (2) In multi-modality scenarios, incorporating the video modality often leads to superior results. For dual-modality setups, combinations that include video achieved an average F1 score of 75.46, whereas those without video only reached 66.56. This trend is also evident in tri-modal configurations, highlighting the critical supportive role of video in feature understanding. Moreover, the facial expression modality only demonstrates optimal performance when combined with the video modality. (3) Our proposed MultiPCL, which integrates four modalities, significantly outperforms all baselines, with performance improvements of 6.51%, 4.27%, and 2.22% over the best single, dual, and tri-modal setups, respectively. This confirms the effectiveness of our detector.

We further conducted ablation experiments (Table III) on the MultiPCL detector to demonstrate the role of MHCA. Our experiments showed that replacing MHCA with a standard fully connected layer resulted in nearly a 4% decrease in F1 score, highlighting the critical role of MHCA in capturing the relationships between different modalities.