EnCBP: A New Benchmark Dataset for
Finer-Grained Cultural Background Prediction in English

Our work relies on the hypothesis that news articles from mainstream news outlets of a country or district reflect the local language use patterns. Thus, we construct 5 text corpora with news articles posted by New York Times, Fox News, and the Wall Street Journal in the US, BBC in UK, Big News Network - Canada in Canada (CAN), Sydney Morning Herald in Australia (AUS), and Times of India in India (IND) for EnCBP-country. For EnCBP-district, we construct 4 corpora from Coosa Valley News, WJCL, and Macon Daily in Georgia (GA), Times Union, Gotham Gazette, and Newsday in New York (NY), NBC Los Angeles, LA Times, and San Diego Union Tribune in California (CA), and Hardin County News, Jasper Newsboy, and El Paso Times in Texas (TX). We stream news articles from Media Cloud ²²2https://mediacloud.org/, a platform that collects articles from a large number of media outlets, using its official API.

To maintain consistent mentions of events and named entities (NEs) in the corpora, we limit the articles to those under five frequently discussed topics, namely “global warming", “abortion", “immigration", “social safety net", and “mandatory vaccination". 1,000 news articles published between Jan. 1, 2020 and Jun. 30, 2021 are sampled from each news outlet to form our corpora.

After data collection, we remove duplicates and overly short documents (less than 100 words) to ensure data quality. We also replace the mentions of countries and districts with the “[country]" and “[district]" special tokens. Then, we chunk the remaining news articles into paragraphs and label the documents with the country or district codes of the news outlets by which they are posted. We adopt paragraph-level annotations since asking the validators to read an overly-long document may cause them to lose track of culture-specific information when they are making judgments. Most state-of-the-art DL models also have input length limits that are not capable of encoding full-length news article. To avoid overly simplifying the task, we remove paragraphs containing NE mentions that are mainly used by news media in specific countries or districts. We quantify the specificity of NEs using inverse document frequency (IDF) scores.

From the filtered news paragraphs, we sample 2,000 paragraphs from the corpus of each culture group to form the annotated dataset. Table 1 provides the statistics of the label and topic distribution of the instances in EnCBP.

To ensure that the cultural background labels in EnCBP correlate with writing styles, we randomly sample 50 instances from each class and manually validate them on MTurk. In each questionnaire, we pair the sampled instance with another random news paragraph from EnCBP and ask three annotators whether the first, second, or both paragraphs are posted by media outlets in a specific country or district. We manually check the instances to ensure there are no country- or district-specific mentions remaining to avoid potential information leakage. For quality control purposes, we only hire crowdsourcing workers from the country or US state matching the label of the instances sampled for validation.

To ensure the quality of annotations in EnCBP, we hire crowdsourcing workers from MTurk to validate randomly sampled data points. Since all the news articles are written by native English speakers and the culture groups are not strictly separated from each other, it is difficult for a validator to identify whether a news paragraph is written by a journalist from the same cultural background as them. Instead, we provide each validator with a news paragraph posted by an international or domestic news outlet in the country or district they live in (MTurk allows for filtering based on location) and a randomly selected news paragraph from our dataset. The validators are asked to compare the two news paragraphs and decide which of the two paragraphs (or both) were written by their local news outlets through analyzing the use of words, phrases, and sentence structures. To avoid information leak and bias in the validation process, the mentions of countries and districts are replaced with “[country]" and “[district]" special tokens at the pre-processing stage of the dataset. An example questionnaire is shown in Figure 1.

We display the validation accuracy (ACC), i.e., the proportion of the validators’ answers that match the labels of those instances in EnCBP, and inter-annotator agreement rate (IAA) in Table 2. Since we have three options in each of the questionnaires, the ACC of random guess is around 33% for each culture group. We quantify IAA with Fleiss’ $\kappa$ Fleiss (1971), a widely used metric for evaluating IAA. The Fleiss’ $\kappa$ in Table 2 range from moderate ($>0.40$) to substantial agreement ($>0.60$). We infer from the relatively high ACC and IAA that: 1) news writing styles are affected by the cultural backgrounds of journalists and 2) writing styles in each culture group are identifiable by local residents. Since we removed country- or state-specific NEs and mentions of countries or states from the paragraphs, and as the distributions of topics and sentiments are balanced across corpora, the chance that the validators make their judgments based on these external information is low.

After data validation, we divide both EnCBP-country and EnCBP-district into training, development, and test sets with a 80%/10%/10% split and a random state of 42. To show the predictability of cultural background labels with NLP models, we benchmark the EnCBP-country and EnCBP-district separately with BiLSTM, bert-base-cased, and roberta-base models. We train the BiLSTM model for 20 epochs with a learning rate of 0.25 and fine-tune the other models for five epochs with a learning rate of 1e-4 on both subsets.

Table 3 displays the average F1-macro scores across five runs with different random seeds for model initialization. For all the models, the standard deviations of the five runs are at most 0.98 on EnCBP-country and 1.84 on EnCBP-district, indicating that randomness does not severely affect the predictions of models, and that the culture-specific writing styles can be modeled by DL models. Both the BERT and RoBERTa models outperform the BiLSTM model with large margins, which suggests the importance of deep neural network architectures and large-scale pre-training for the task. We also note that all the three models perform worse on EnCBP-district, which may be caused by both the more difficult task setting and the higher level of noise in EnCBP-district, since local news outlets target audiences from all over the country. In the rest of this paper, we use the bert-base-cased model for the analyses and discussions since it is less resource-consuming than the roberta-base model, while the findings potentially apply to other model architectures as well.

Since all the documents in EnCBP come from news articles, we assume they are well-written and grammatically correct. In addition, LMs trained on a grammatical corpus should produce similar perplexities on the corpus with each label if the writing styles are consistent across corpora. Thus, to examine culture-specific differences in writing styles, we fine-tune a bert-base-cased model on the training corpus of each class in EnCBP with the MLM objective and evaluate perplexity of the fine-tuned models on all the test corpora.

As Table 4 shows, BERT models usually produce the lowest perplexities on the test portions of their training corpora, and the cross-corpus perplexities are usually considerably higher. This supports our hypothesis that English writing styles are culture-dependent, and that the writing styles across cultures are different enough to be detected by LMs. Meanwhile, we find that the cultural domain compatibility differs for different pairs of corpora, e.g., the IND corpus is more compatible with the UK corpus than other countries or districts. The relations are not symmetric either, e.g., while the LM trained on the CAN corpus well adapts to the US corpus, the US LM performs the worst on the CAN corpus among the five countries. These potentially result from the effects of geographical, geo-political, and historical backgrounds on the formation of cultural backgrounds. For instance, the US could be said to have greater influence on Canadian culture than vice versa. Potentially for similar reasons, compared to TX and GA, NY has a more consistent writing style with CAN. We also note clear cultural domain compatibility gaps between liberal (NY and CA) and conservative states (GA and TX), which, agreeing with Imran et al. (2020), shows that ideologies and policies of a district potentially has an effect on its culture-specific writing styles. We provide additional topic-level LM analysis in Appendix A.

To verify if the different expressions across classes in the EnCBP datasets are triggered by cultural differences, we analyze the distributions of topics and sentiment scores for each class. Specifically, we model the topics of each corpus using BERTopic Grootendorst (2020) and analyze sentiments of text using Stanza Qi et al. (2020).

We apply two-sided Kolmogorov-Smirnov (KS) tests on the topic distributions of each pair of classes to see whether the topic distributions for each country or state are similar. For all pairwise comparisons, the null hypothesis (which is that the distributions are identical) cannot be rejected using the KS test, with all p-values being above 0.1, and most in fact being above 0.7. This potentially results from both topic control at the data collection phase and data filtering eliminating paragraphs containing NEs with high IDF scores. Additionally, the sentiment score distribution is relatively consistent across classes (28.02% to 34.97% instances with negative sentiments). Since the classes in EnCBP contain documents that are similar in topics and sentiments, it is likely that the differences in linguistic expressions across classes are caused by cultural differences.

In addition to automatic evaluations, we manually examine distinguishable English expressions for each culture group in EnCBP. Specifically, we extract phrases with high TF-IDF values for each corpus in EnCBP, retrieve news paragraphs that contain these phrases, and examine sentence structures and phrase usages in these representative instances.

From our analyses, we find that the different writing styles of countries and districts in EnCBP are affected by the choice of words or phrases, the ordering of phrases, and degrees of formality. For example, the phrases “in the wake of", “in the lead up to", and “the rest of the world" are much more frequently used by AUS news outlets than the others. Also, the use of auxiliaries, especially the word “may", is more frequent in the UK corpus, in the context of politeness. The US corpus is in general more colloquial than the other corpora, as the journalists often write subjective comments in the news articles. Additionally, the ways of referencing speeches differ across corpora, e.g., the quoted text usually appears prior to the “[name] said" in the UK corpus but reversely in the US corpus. In the EnCBP-district subset, the sentence structures are more consistent across corpora, while the mentions of NEs and wordings differ more. For example, the word “border" appears frequently in the TX corpus but less in the other corpora when discussing the “immigration" topic. Though the observations summarized from EnCBP may not be universally applicable to other datasets or text domains, they are validated by native speakers of English to be accounting for the high ACC in manual validations.

The datasets used in our evaluations are:
PAWS-Wiki Zhang et al. (2019) is a PI dataset containing English Wikipedia articles. Each instance in PAWS-Wiki consists of a pair of sentences and a label indicating whether the two sentences are paraphrase (1) or not (0). There are 49,401 training instances, 8,000 development instances, and 8,000 test instances in this dataset.
CoNLL-2003 English NER dataset Tjong Kim Sang and De Meulder (2003) contains news articles from Reuters news only, so the dataset has a more consistent UK writing style, compared to the other datasets we utilize. Each word in the documents is annotated with persons (PER), organizations (ORG), locations (LOC), or miscellaneous names (MISC) NE label in the IOB-2 format. We adopt the official data split of the CoNLL-2003 dataset in the experiments, where there are 7,140, 1,837, and 1,668 NEs in the training, development, and test sets, respectively.
Go-Emotions Demszky et al. (2020) is an ER dataset containing 58,009 English Reddit comments. Instances in this dataset are labeled with 28 emotion types including neutral, in the multi-label classification form. We split the dataset into training, development, and test sets with a 80%/10%/10% split using 42 as the random seed. To be consistent with other evaluations, we switch the annotations to the multi-class classification form by duplicating the data points associated with multiple labels and assigning one emotion label to each copy. This results in an ER dataset containing 199,461 training instances, 35,057 development instances, and 34,939 test instances after removing instances with no labels.
Stanford Sentiment Treebank (SST-5) Socher et al. (2013) is a document-level SA dataset containing sentences from movie reviews. The documents are annotated with sentiment scores, which are turned to fine-grained (5-class) sentiment labels after pre-processing. Using the official data split, we divide the dataset into training, development, and test splits containing 156,817, 1,102, and 2,211 instances, respectively. Note that the training set of SST-5 contains a mixture of phrases and sentences, while the development and test sets contain only complete sentences.
SST-2 is the coarse-grained SST-5 dataset, in which each document is labeled with positive (1) or negative (0) sentiments. There are 67,349 training instances, 872 development instances, and 1,821 test instances in this dataset.
QNLI Wang et al. (2019) is a natural language inference (NLI) dataset with a question answering background. Each instance in QNLI contains a question, a statement, and a label indicating whether the statement contains the answer to the question (1) or not (0). There are 104,743 training instances, 5,463 development instances, and 5,463 test instances in this dataset.
STS-B Cer et al. (2017) is a benchmarked semantic textual similarity (STS) dataset. Each instance in STS-B is a pair of sentences manually annotated with a semantic similarity score from 0 to 5. The dataset contains 5,749 training instances, 1,500 development instances, and 1,379 test instances.
RTE is a textual entailment (TE) dataset. Each instance in RTE contains a pair of sentences and a label indicating whether the second sentence is an entailment (1) or not (0) of the first sentence. The RTE dataset we use is a combination of RTE1 Dagan et al. (2005), RTE2 Bar-Haim et al. (2006), RTE3 Giampiccolo et al. (2007), and RTE5 Bentivogli et al. (2009) datasets, which contains 2,490 training instances, 277 development instances, and 3,000 test instances.
Emotion Saravia et al. (2018) is a Twitter-based ER dataset labeled with six emotion types, i.e., sadness (0), joy (1), love (2), anger (3), fear (4), and surprise (5). There are 16,000 training instances, 2,000 development instances, and 2,000 test instances in this dataset.