What are Your Pronouns? Examining Gender Pronoun Usage on Twitter

We use the full span of the collected data and consider only users who had at least 10 posts to ensure we have sufficient data points per user. For each user, we build a series of feature sets as follows:

• •

  Text ($\text{dim}=768\times 20$): the BERTweet embeddings (Nguyen, Vu, and Tuan Nguyen 2020) of their most recent 20 tweets, zero-padded if they had less than 20 tweets. We use this value since most (60%) of the users had less than 20 tweets. BERTweet is the state-of-the-art language model developed specifically for Twitter, producing 768-dimensional embeddings per tweet.

• •

  Text Pronouns ($\text{dim}=7$): the number of times I, you, we, he, she, it, they, and other gender-nonconforming pronouns appeared in their tweets (see Appendix).

• •

  User ($\text{dim}=6$): The last entry of user-level metadata including whether they are verified and their number of friends, followers, favorites, statuses, and lists.

• •

  Activities ($\text{dim}=9$): The number of original tweets, retweets, quoted tweets, and replies the user posted as well as the number of hashtags, URLs, mentions, characters, and tokens used.

• •

  Mentions ($\text{dim}=\text{1K}$): We gather the top 1K most frequently mentioned users in our dataset and build a vector representing how frequently they were mentioned. Note that we treat retweets as a subset of mentions.

We purposely omit user biography as a feature because it is our source of ground truth labeling of gender pronouns.

We run two experiments. Experiment (1) is the binary classification task of detecting whether a user discloses gender pronouns or not; experiment (2) is the multiclass classification task of detecting the category of gender pronoun disclosed. To address data imbalance, we sample 10K users from each of the gender pronoun groups (he series, she series, and non-binary) followed by 30K NP users, totaling 60K users. We gather two such random samples of 60K users, one for hyperparameter-tuning and one for final testing via 5-fold CV. For experiment (2), the 30K NP users are not included. As we will show below, this decision is in part driven by the relative ease in which the models are able to distinguish users with from users without pronouns in experiment (1). In this way, we can investigate the factors that separate users of each gender pronoun group. We train and test a deep neural network (DNN) for each prediction task, using each feature set in isolation as ablation studies, as well as all feature sets in combination. For every input scenario, we fine-tune the model using randomized grid search to test over 40 combinations of hyperparameters, fixing 20% of the data for validation. The exact model architecture and hyperparameters searched can be found in the Appendix. We use the macro-F1²²2Since the data is balanced, macro-F1 is equivalent to micro-F1. score as the classification metric and use two baseline methods for comparison: Random, which predicts random labels based on the distribution of labels in the training set, and Majority, which predicts the majority label.

For experiment (1), the binary classification distinguishing GP users from NP users, we find that using all of the feature sets improves predictive performance over the baseline Random and Majority models (Fig. 3). Embeddings of the textual data are the best single predictors of gender pronouns, achieving a macro-F1 of 84% ($SE=0.05\%$). User activities are the second best predictors at 77% macro-F1 ($SE=4.53\%$), albeit with a much larger variance. The feature set on top mentions achieves an impressive score of 69% macro-F1 ($SE=2.12\%$). Including all features yields a slightly better score of 85% ($SE=0.24\%$). These results indicate that users’ sharing activities encode substantial signals to predict whether they disclose gender pronouns.

For experiment (2), the multiclass classification of gender pronoun category, we note a similar improvement in classification performance over the baselines (Fig. 3). However, the improvement is more modest than what was achieved in experiment (1), The model using all feature sets attained only a macro-F1 of 47% ($SE=0.19\%$), indicating that discerning a user’s gender pronoun category is more challenging. Textual features remain the best predictors of gender pronoun category at 46% macro-F1 ($SE=0.19\%$), whereas activities features are the worst predictors at 35% macro-F1 ($SE=2.50\%$). The feature set of top mentions is the best single set of features after the textual features, yielding a score of 42% ($SE=1.12\%$).

Since using the feature set of highly mentioned users alone achieves a non-negligible improvement over the baselines in both experiments, we further explore whether there is a link between gender pronoun disclosure and the gender pronoun categories of the 1K highly mentioned users. To this end, we first check if the 1K highly mentioned users belong to any gender pronoun group. 12% of the highly mentioned users have gender pronouns (she series: 6%, he series: 4%, and non-binary: 2%). We then compute the relative frequency of them getting mentioned by GP users as opposed to NP users. The result is depicted in Fig 4. Across the board, all highly mentioned GP users are mentioned significantly more by GP users (Mann-Whitney U test, $p<0.001)$. These differences are also significant across all four groups of users using a Kruskal-Wallis H test ($p<0.001$). Highly mentioned users who use non-binary pronouns, in particular, are 10 times more likely to be mentioned by GP users than by NP users ($p<0.001$).

In any social network, either online or offline, ideas, emotions, and behaviors can diffuse from one to another, a process known as social contagion (Christakis and Fowler 2013; Ferrara and Yang 2015a; Kramer, Guillory, and Hancock 2014; Rosenquist, Fowler, and Christakis 2011; Tsvetkova and Macy 2014; Pachucki, Jacques, and Christakis 2011; Hodas and Lerman 2014; Mønsted et al. 2017; Ferrara and Yang 2015b). Prior works have also shown that people who congregate in social communities share similar traits, interests, and tendencies, resulting in homophilic communities (McPherson, Smith-Lovin, and Cook 2001; Kossinets and Watts 2009; Bisgin, Agarwal, and Xu 2012). Putting network homophily and contagion together results in the social network effect (VanderWeele and An 2013). In this work, we elicit social network effects from observable sharing activities, such as retweeting and mentioning other users who disclose gender pronouns. While it is preferable to use the follower/following relationships among users (Hodas and Lerman 2014; Ferrara and Yang 2015a), this is not computationally feasible in practice given Twitter API’s rate limits. In lieu of the follower/following network, we use the retweet and mention networks under the assumption that the act of retweeting or mentioning can be treated as a form of a stronger, more explicit signal of social network exposure to peer influence as well as social endorsement (Boyd, Golder, and Lotan 2010; Metaxas et al. 2015).

To measure the network effects on gender pronoun adoption, we restructure our data into temporally disjointed subsets. Consider two time windows of data $T_{1}$ and $T_{2}$, where $T_{1}$ occurred prior to $T_{2}$. We build a directed, weighted, and attributed network $G=(V,E)$ from tweets posted in $T_{1}$ as follows. We first identify the set of GP users $U_{p}$ who appeared in our dataset with gender pronouns during $T_{1}$. Then we extract the network interactions $E=\{e\}$ that occurred in $T_{1}$, where each edge $e=(u,v,w)$ is a directed edge where at least one of the users involved in the interaction is a GP user, i.e., $u\in U_{P}$ or $v\in U_{P}$. The weight of the edge $w$ represents the frequency of the mention or retweet interactions from u to v. In the context of this paper, retweeting a user is one way to mention them, therefore the retweet network edges are a strict subset of the mention network edges. We then retain the largest weakly connected component of the network. The NP users in the network compose the set $U_{NP}$, where $V=U_{P}\sqcup U_{NP}$. In the ensuing period $T_{2}$, we check if any of the users in $U_{NP}$ adopted gender pronouns. Note that a substantial amount of users would have unknowable pronoun status if we do not see their tweets in $T_{2}$.

We repeatedly build $T_{1}$ and $T_{2}$ data using sliding windows of discretized time periods starting from Feb 1, 2020. We experiment with spans of four weeks for graphs built on both retweet edges and mention edges. This produces twenty-one $T_{1}$ and $T_{2}$ pairs, ending the last $T_{2}$ period on Oct 9, 2021. For graphs with retweet edges, we further explore a larger span of eight weeks to control for data biases due to time span selection. This produces ten $T_{1}$ and $T_{2}$ pairs, ending the last $T_{2}$ period on Oct 23, 2021.

The adoption experiment data sets used in this section are described in Table 1. We show the number of target users included in the experiments and how many of those eventually adopted pronouns. These users are all part of $U_{NP}$, that is, users who were connected to a GP user ($U_{P}$) in $T_{1}$ but who themselves did not have gender pronouns in $T_{1}$. Overall, the proportion of gender pronoun adoption is very low at around 2%, rendering the data extremely imbalanced. This comes as no surprise given the relatively low gender pronoun usage we have seen so far in this paper, even though, by design, every user is connected to at least one GP user through retweeting or mentioning them. Of all the gender pronouns adopted, she series is the most popular pick. He series and non-binary pronoun adoptions follow closely behind, with he series pronouns slightly more frequently adopted.

To explore the relationship between data collected from $T_{1}$ and the adoption of gender pronoun in $T_{2}$ of all users in $U_{NP}$, we use the following sets of features:

• •

  Text ($\text{dim}=768$): the BERTweet embeddings of the tweets each user posted in $T_{1}$, concatenated as a single string.

• •

  User ($\text{dim}=768+6$): the BERTweet embedding of the user’s biography and user-level metadata including friends count, followers count, favorites count, statuses count, listed count, and whether they are verified. We use the last entry in $T_{1}$ if a user appeared more than once.

• •

  Network ($\text{dim}=5+5+128$): the number of neighbors they have (i.e., they retweeted or mentioned), the number of GP neighbors, and the number of neighbors with she series, he series, or non-binary pronouns. We also compute all weighted versions of the previous numbers of neighbors. Additionally, we include the node2vec (Grover and Leskovec 2016) graph embeddings learned from the retweet or mention network (see Appendix).

• •

  Activities ($\text{dim}=11$): the number of original tweets, retweets, quoted tweets, or replies the user posted during $T_{1}$. We also count the number of hashtags, URLs, mentions, characters, and tokens used. Finally, we include the number of first-person singular pronouns used in the tweets, which convey individualism, and the number of first-person plural pronouns, which convey collectivism (Twenge, Campbell, and Gentile 2013).

We conduct two experiments. Experiment (1) is the binary classification of whether the user will adopt gender pronouns in $T_{2}$ and experiment (2) is the multiclass classification of which of the three gender pronoun categories the user adopts in $T_{2}$. Users who did not adopt any gender pronouns are excluded from experiment (2), as was done in §5. Our experiments are repeated for all data scenarios: retweet networks over four- and eight-week time spans and mention networks over four-week time spans (i.e., $T_{i}$ is either four or eight weeks). For each data scenario, we use the last $T_{1}$–$T_{2}$ pair as the test set, the second to last $T_{1}$–$T_{2}$ pair as the validation set, and the remaining data as the training set. Similar to §5, we built a deep neural network (DNN) for these tasks. We first tune the model hyperparameters via randomized grid search using the training set. Please see the Appendix for the exact model architecture and hyperparameter search space. Macro-F1 is used as the evaluation metric. Following §5, we explore the results using each feature set in isolation as well as in combination, and we use the Random and Majority models as baselines.

We present our results to address RQ3 in Fig. 6. The experimental results of the four-week and eight-week time spans of the retweet network are averaged due to the similarity in their results. Differently from Section 5.2.1, we recognize lower classification performances in experiment (1), the binary classification of predicting whether a user will adopt any type of gender pronouns in $T_{2}$ given the features in $T_{1}$, probably due to the limited observation period leveraged in the training phase. Using all of the features sets in $T_{1}$ features, our approach performs slightly better than the Random and Majority baselines on the retweet network, achieving 56% macro-F1 compared to the baseline score of 50%. The mention network prediction tasks achieve better performance at scores of 59% macro-F1. Combining all features yields very little improvement over using features in isolation. All singular categories of features perform relatively equivalently. User-level features are the most predictive, while network features are some of the lowest. We note that this could be an artifact of our study design limited by computational constraints, forsaking the larger social network of users who are not directly connected to a GP user. Therefore, we do not know how many other NP users they were connected to. The ineffectiveness of the network features becomes even more apparent using the mention network. One explanation is the underlying difference between retweets and mentions: retweets are often seen as endorsements (Boyd, Golder, and Lotan 2010; Metaxas et al. 2015) whereas mentions can represent social conversations (Leavitt et al. 2009).

The predictive performance of experiment (2), which is the multiclass classification of which of the three gender pronoun category a GP adoptee chooses, yields further insights. While user-level features remain the singularly most predictive set of features, we now see a large improvement gained from using network features. This shows that network features contain cues as to which gender pronoun a user will adopt. We also see that the performance is better in general on the retweet network than on the mention network, suggesting that the retweet network signals encode more cues as to which gender pronoun category a user will adopt. This process could unfold via either social homophily or social contagion or both (Shalizi and Thomas 2011). Social networks are mediums for online communities of people with shared interests and similar backgrounds, forming homophilic clusters of users. It can also be a pathway for social influence, wherein one exerts behavioral changes on another by exposure. While we cannot disentangle the intertwining effects of the two forces, we provide additional analyses on the social network effects below.

To deepen our understanding of the link between the social network and gender pronoun adoption, we analyze how gender pronoun adoption in $T_{2}$ is linked to the relative number of GP neighbors they had in $T_{1}$. A user’s neighbor is someone they retweeted in a retweet network and someone they mentioned in a mention network. We begin by establishing the expected number of type $x$-gendered neighbors any NP user would be connected to in $T_{1}$, given by:

$$E[x]=\frac{1}{|U_{NP}|}\sum_{u\in{U_{NP}}}\mathcal{N}_{u}(x),$$

(1)

where $\mathcal{N}_{u}(x)$ denotes the number of $x$-gendered neighbors a user $u\in U_{NP}$ had in $T_{1}$, $x\in\{\textit{she},\textit{he},\textit{non-binary}\}$. The relative number of $x$-gendered neighbors for users who adopted $y$ gender pronouns in $T_{2}$ is thus given by:

$$R(x,y)=\left.\frac{1}{|U^{y}|}\sum_{u\in U^{y}}\mathcal{N}_{u}(x)\middle/E[x],\right.$$

(2)

where $U^{y}\subset U_{NP}$ represents the set of users with pronoun status $y$ in $T_{2}$, $y\in\{\textit{she},\textit{he},\textit{non-binary},\allowbreak\textit{no pronouns}\}$. If $R(x,y)<1$, then users who adopted pronoun status $y$ in $T_{2}$ had less $x$-gendered pronoun neighbors in $T_{1}$. Alternatively, if $R(x,y)>1$, then users who adopted pronoun status $y$ in $T_{2}$ had more $x$-gendered pronoun neighbors in $T_{1}$.

Fig. 7 displays the relative number of neighbors $R(x,y)$ separately for the retweet and mention networks. We aggregate the statistics for each individual in every timeframe. The retweet networks of four-week and eight-week time spans are combined in this figure due to the similarity of their results. There are several key insights. First, we observe that users who did not adopt pronouns (red bars) retweeted or mentioned an expected number of GP users. That is, $R(x,\textit{no pronouns})\approx 1$. This is both a sanity check and a result that shows that users who remained without gender pronouns had network exposures that did not deviate from the norm. Interestingly, users who adopted she and he series pronouns retweeted and mentioned slightly more neighbors of the same gender pronoun category but slightly fewer neighbors of the opposite gender pronoun category.

Importantly, we see that users who adopted any gender pronouns at all had substantially more neighbors who use non-binary pronouns. Those who adopted she and he series pronouns had 1.5 times non-binary neighbors in the retweet network. This number is particularly sizeable for non-binary pronoun adoptees, who retweeted over 3 times as many and mentioned over 2.5 times as many non-binary neighbors as the expected baseline.