Sensemaking About Contraceptive Methods Across Online Platforms

LeAnn McDowall,¹ Maria Antoniak,² David Mimno¹

Abstract

Selecting a birth control method is a complex healthcare decision. While birth control methods provide important benefits, they can also cause unpredictable side effects and be stigmatized, leading many people to seek additional information online, where they can find reviews, advice, hypotheses, and experiences of other birth control users. However, the relationships between their healthcare concerns, sensemaking activities, and online settings are not well understood. We gather texts about birth control shared on Twitter, Reddit, and WebMD—platforms with different affordances, moderation, and audiences—to study where and how birth control is discussed online. Using a combination of topic modeling and hand annotation, we identify and characterize the dominant sensemaking practices across these platforms, and we create lexicons to draw comparisons across birth control methods and side effects. We use these to measure variations from survey reports of side effect experiences and method usage. Our findings characterize how online platforms are used to make sense of difficult healthcare choices and highlight unmet needs of birth control users.

Introduction

Birth control users include nearly two thirds of women in the U.S. between the ages of 15-49—almost 50 million women (Daniels and Abma 2020). In addition to providing a means of family planning, birth control methods can also be used to treat and manage many medical conditions, including acne, endometriosis, cancer risks, gender dysphoria, and irregular and painful menstruation (Schindler 2013; Nahata, Chelvakumar, and Leibowitz 2017). However, birth control methods are not one-size fits all, and the choices of whether to use birth control and which method to select are complicated by personal beliefs, cost and accessibility, and a wide array of side effects that are difficult to predict and identify (Manzer and Bell 2022; Yoost 2014; Polis et al. 2016).

When navigating this decision, birth control users face a sensemaking challenge that involves dealing with social stigma, understanding contraceptive methods and potential side effects, and seeking normalcy after upsetting experiences seeking and using contraception. This challenge leads many birth control users to the internet (Yee and Simon 2010; Russo et al. 2013), where they can engage in community sensemaking activities, e.g., seeking and sharing information, reviews, advice, hypotheses, and stories (Carron-Arthur et al. 2015; Yang et al. 2019).

Prior work has shown that these various sensemaking activities can vary in frequency across different online platforms (De Choudhury, Morris, and White 2014; Rivas et al. 2020; Zhang, N. Bazarova, and Reddy 2021). Different healthcare conditions (Sannon et al. 2019) and side effects (De Choudhury, Morris, and White 2014) can result in different behavior, and recent work has demonstrated differences in the online sensemaking practices surrounding reproductive healthcare topics like pregnancy and vulvodynia (Young and Miller 2019; Andalibi and Garcia 2021; Chopra et al. 2021). However, while birth control researchers have examined the frequencies of different method discussions over time (Nobles, Dredze, and Ayers 2019; Merz et al. 2020), a comparison across platforms of the difficult decision-making of birth control users and their sensemaking practices has so far been unexplored.

We study birth control users’ unique combination of sensemaking activities—deciding between methods, interpreting side effects, calculating risks, etc.—and we use computational text analysis methods to discover the intersections between these activities and online platforms. Because different methods and side effects could lead to different activities, we first explore which birth control methods and side effects are more likely to be discussed on different online platforms (RQ1). Using lexicons crafted for our online settings, we compare these discussion frequencies to the distribution of experienced side effects as reported via a nationally representative survey by Nelson et al. (2017), indicating increased or decreased interest in discussing these concerns online. Next, using topic modeling and hand-annotations, we discover and characterize the kinds of sensemaking activities that birth control users engage in online and how these differ by platform (RQ2). We contrast these activities with prior work on related online healthcare communities, and we highlight the particular strategies of birth control users.

Contributions.

We identify and characterize how birth control users make sense of their experiences and options via three large English-language online platforms (Reddit, Twitter, and WebMD). Their unique combination of sensemaking strategies included storytelling, risk analysis, timing and calculations, causal reasoning, method and hormone comparison, and information and explanations; in particular, storytelling is used to prepare for and overcome painful insertion experiences. We examine these patterns across birth control methods, side effects, and platforms, unlike prior work, and we compare our observations of side effect discussions to survey data, finding large variations and highlighting the importance of integrating different methods. Throughout this study, we employ a mixture of lexicons (which we provide as a community resource¹¹1https://github.com/maria-antoniak/birth-control-across-platforms), topic modeling, and hand annotation, demonstrating how a detailed and diverse analysis across online platforms can expand our understandings of social sensemaking, web and social media activity, and vital healthcare issues.

Related Work

Sensemaking in online communities. As explained by Weick, Sutcliffe, and Obstfeld (2005), sensemaking necessarily involves communication and is an “activity that talks events and organizations into existence,” in part through retrospective storytelling. It is a process that relies on collaborative problem solving (Pirolli and Card 2005), and more recently, it has been defined in closely related work by Andalibi and Garcia (2021) as “how individuals make sense of complex phenomena by constructing mental models that draw on new or existing experiences, information, emotions, ideas, and memories.” In online healthcare communities, the individual works to make sense of their healthcare experience, and the community collectively gathers information, compares stories, and makes sense of a shared experience (Mamykina, Nakikj, and Elhadad 2015). Sharing experiences can help narrators make sense of their stories (Tangherlini 2000) and can transfer important information to others without firsthand experience (Bietti, Tilston, and Bangerter 2019), leading users through a transformative process via the gathering and organizing of information (Genuis and Bronstein 2017; Patel et al. 2019). This process is contextual; each community can rely on different strategies (Young and Miller 2019).

Healthcare activity across platforms. Specific platform affordances can facilitate different types of healthcare-related disclosure and interactions. Zhang, N. Bazarova, and Reddy (2021) formulate a social media disclosure ecology in which platform affordances like anonymity, persistence, and visibility control can predict, e.g., pandemic-related disclosures. Health information seeking behavior can differ across platforms, with search engines more frequently used for serious and stigmatized conditions and Twitter more frequently used for symptoms (De Choudhury, Morris, and White 2014), and community affordances such as threaded conversations and hashtags can influence participation decisions of those with invisible chronic health conditions (Sannon et al. 2019). The distribution of content type can also differ by platform; e.g., prior work found that sharing experiences is less frequent on social networks than forums, although these rates can depend on healthcare topic (Rivas et al. 2020).

Reproductive healthcare and online sensemaking. Studies of reproductive healthcare communities have emphasized some unique sensemaking practices. For example, in a study of sensemaking practices after pregnancy loss, Andalibi and Garcia (2021) highlighted the importance of seeking emotional validation, rather than just information, when trying to re-discover normalcy. Young and Miller (2019) similarly supported these dual information management and emotional needs in a study of a vulvodynia Facebook group, and Chopra et al. (2021) discussed polycystic ovary syndrome and why personal tracking is an important communal activity that involves comparison to others’ experiences. Whether these themes hold for online birth control communities is not currently known.

Two recent studies have focused specifically on online discussions of birth control. First, Nobles, Dredze, and Ayers (2019) used search queries to track interest in different birth control methods, finding that U.S. political events correlate strongly with increased information seeking behavior online. Second, Merz et al. (2020) examined the prevalence and sentiment of tweets mentioning different birth control methods. The authors found that long-acting methods like the IUD were mentioned more often on Twitter, and the proportion of these tweets increased over time. While most tweets expressing sentiment about contraception were negative, tweets about long-acting methods were more likely to express positive sentiment.

Data

We collected data related to birth control from three prominent online platforms: posts and comments on r/BirthControl, birth control reviews on WebMD, and Twitter posts and replies related to birth control. These are all English-language subcommunities of larger websites, and while we have limited demographic information, we observe that the majority of location-specific posts (e.g., politics, insurance) are about the U.S. Table 1 summarizes these datasets.

Community	# of Posts	Vocab Size	Mean Tokens	Year Range	Posts Dist. (2007-2020)	Moderation	Structure
Reddit Posts	68,958	49,088	79	2010-2020	{sparkline}14 \sparkspike0.03333333333333333 0.0 \sparkspike0.1 0.0 \sparkspike0.16666666666666666 0.0 \sparkspike0.23333333333333334 0.0 \sparkspike0.3 0.0 \sparkspike0.36666666666666664 0.005445744251714401 \sparkspike0.43333333333333335 0.042839854780153286 \sparkspike0.5 0.07678499394917306 \sparkspike0.5666666666666667 0.11308995562726905 \sparkspike0.6333333333333333 0.15615671641791046 \sparkspike0.7 0.19763513513513514 \sparkspike0.7666666666666666 0.3000605082694635 \sparkspike0.8333333333333334 0.4996470350947963 \sparkspike0.9 0.8377369907220653 \sparkspike0.9666666666666667 0.9	user moderators	forum posts
Reddit Comments	264,912	67,837	32	2010-2020	{sparkline}14 \sparkspike0.03333333333333333 0.0 \sparkspike0.1 0.0 \sparkspike0.16666666666666666 0.0 \sparkspike0.23333333333333334 0.0 \sparkspike0.3 0.0 \sparkspike0.36666666666666664 0.00546811055845873 \sparkspike0.43333333333333335 0.061820814707548945 \sparkspike0.5 0.11436284077315369 \sparkspike0.5666666666666667 0.184824970093811 \sparkspike0.6333333333333333 0.261208524837877 \sparkspike0.7 0.33968866083233645 \sparkspike0.7666666666666666 0.48256783982874774 \sparkspike0.8333333333333334 0.5731033180129699 \sparkspike0.9 0.9 \sparkspike0.9666666666666667 0.8297220298432285	user moderators	replies to forum posts
Twitter Posts	499,796	398,910	12	2006-2020	{sparkline}14 \sparkspike0.03333333333333333 0.0 \sparkspike0.1 0.001111416026344676 \sparkspike0.16666666666666666 0.006683932491767288 \sparkspike0.23333333333333334 0.09772742864983534 \sparkspike0.3 0.38393249176728866 \sparkspike0.36666666666666664 0.6351279500548848 \sparkspike0.43333333333333335 0.9 \sparkspike0.5 0.6975061745334796 \sparkspike0.5666666666666667 0.5365669593852909 \sparkspike0.6333333333333333 0.7475662047200877 \sparkspike0.7 0.7191787870472008 \sparkspike0.7666666666666666 0.7062277030735454 \sparkspike0.8333333333333334 0.6884759193194292 \sparkspike0.9 0.8438271816684961 \sparkspike0.9666666666666667 0.7510856888035127	company	tweets (no retweets)
Twitter Replies	211,896	73,896	12	2007-2020	{sparkline}14 \sparkspike0.03333333333333333 0.0 \sparkspike0.1 0.00010469986040018614 \sparkspike0.16666666666666666 0.002006747324336901 \sparkspike0.23333333333333334 0.02511051651931131 \sparkspike0.3 0.0624709167054444 \sparkspike0.36666666666666664 0.1275767798976268 \sparkspike0.43333333333333335 0.197987436016752 \sparkspike0.5 0.19395649139134483 \sparkspike0.5666666666666667 0.20385062819916241 \sparkspike0.6333333333333333 0.15432759422987435 \sparkspike0.7 0.18976849697533738 \sparkspike0.7666666666666666 0.33249185667752446 \sparkspike0.8333333333333334 0.4761575151233132 \sparkspike0.9 0.8317705909725454 \sparkspike0.9666666666666667 0.9	company	replies to tweets
WebMD Reviews	18,110	17,487	45	2006-2020	{sparkline}14 \sparkspike0.03333333333333333 0.0 \sparkspike0.1 0.1261818181818182 \sparkspike0.16666666666666666 0.5632727272727273 \sparkspike0.23333333333333334 0.822909090909091 \sparkspike0.3 0.762909090909091 \sparkspike0.36666666666666664 0.8763636363636365 \sparkspike0.43333333333333335 0.9 \sparkspike0.5 0.6854545454545455 \sparkspike0.5666666666666667 0.592 \sparkspike0.6333333333333333 0.39163636363636367 \sparkspike0.7 0.34581818181818186 \sparkspike0.7666666666666666 0.18 \sparkspike0.8333333333333334 0.14363636363636365 \sparkspike0.9 0.12654545454545457 \sparkspike0.9666666666666667 0.06872727272727273	company	reviews (no reply structure)

Table 1: Overview of the three datasets used for comparison, including only texts mentioning our target birth control methods.

Reddit.

r/BirthControl²²2https://www.reddit.com/r/birthcontrol/ is a user-created and user-moderated online forum dedicated to birth control. Users are pseudonymous and range from one-time questioners to experienced question answerers. As of November 20, 2022, r/BirthControl had 107,537 members. According to a Pew Research Center survey of U.S. residents, more men (15%) than women (8%) use Reddit, and more White (12%) and Hispanic (14%) than Black (4%) survey respondents use Reddit (Perrin and Anderson 2019), but these platform-wide distributions are unlikely to be representative of r/BirthControl, for which more detailed demographics are unavailable. Prior work on a related subreddit found that 81% of the users identified themselves as white (Nobles et al. 2020).

Posts tend to be longer than comments, there are more comments than posts in our dataset, and both posts and comments have increased in frequency over time. For each document, we collected the title (for posts), text, date, and user-assigned tag (for posts). We removed comments written by the parent post’s author, and we also removed stickied comments (auto-generated or mod-written comments). We did not include documents deleted by the time of collection.

WebMD.

WebMD³³3https://www.webmd.com/sex/birth-control/birth-control-pills is a healthcare website that has hosted news, information, prescription discounts, community forums, and user-written medication reviews. We focused on the medication reviews as a contrast to the Reddit and Twitter datasets. The majority of reviews are written by women (71%), and unlike Twitter and Reddit, the website has declined in visitors and reviewers since 2009 (Yun 2019). Unlike Twitter and Reddit, WebMD reviews do not allow users to interact directly, though reviews sometimes mention other reviews or the general trends observed in other reviews.

We gathered reviews by querying birth control as a condition on the drugs and supplements portion of the site and selecting medications that had more than 5 reviews.⁴⁴4At the time of data collection, this search term yielded both preventative and emergency contraception, but the terms must be queried separately as of January 2021. For each WebMD review, we collect the birth control name, the date of the review, and the review text.

Twitter.

Twitter is a large social network where discussions range widely from personal to global topics. Compared to the general public, Twitter users are more likely to be Democrats, and they also skew younger than users of YouTube, Facebook, or Instagram (Perrin and Anderson 2019). The gender and racial distribution on Twitter are close to uniform; out of a set of U.S. survey respondents, 24% of men and 21% women report using Twitter, while 24% of Black, 25% of Hispanic, and 21% of White respondents report using Twitter (Perrin and Anderson 2019). On Twitter, birth control discussions take place in the context of many other discussion topics, and while users can group their conversations using hashtags, there are not separated communities like subreddits. Moderation is organized by Twitter, rather by users as on Reddit.

Using the Twitter Academic API (v2), we collected all the English Twitter posts containing a set of keywords corresponding to our three target birth control methods. Separately, we collected all the Twitter replies containing the same set of keywords. Recent work has shown that this API returns reliable representations of the full tweet space (Pfeffer et al. 2022). See §Methods below for the design of the Twitter-specific keywords. We removed Twitter handles from tweet texts. The high vocabulary size for Twitter posts (Table 1) partly reflects many unique URLs shared in these documents, which can be indicative of information-providing behavior. There are many possible design decisions when gathering Twitter data, and we chose not to collect the full conversations (replies and parent posts of the keyword-containing posts and replies) to (a) limit the size of our collection, (b) target texts explicitly discussing birth control, and (c) more closely replicate prior work on birth control tweets (Merz et al. 2020).

Methods

Birth Control Method Lexicon

To explore the prevalence across platforms of different birth control methods, we develop a lexicon to match each document to the primary method being discussed. Due to data scarcity and space limitations, we do not consider methods that are not available on all three platforms (e.g., WebMD does not include the male condom), and of the remaining methods, we limit our analysis to the three most commonly discussed reversible, non-emergency methods, mirroring prior work such as Merz et al. (2020).⁵⁵5Using the lexicons described above for Reddit, we find that the implant (8,578 posts) is discussed more than twice as often as the shot (3,529 posts) or barrier methods like the male condom (3,029 posts). This contrasts with reported rates of contraception use in the U.S., where the pill is used by 14.0%, the IUD by 8.4%, the male condom by 8.4%, the implant by 2.0%, and the shot by 2.0% of women aged 15-49 (Daniels and Abma 2020). Our goal is to measure prevalence of discussions rather than prevalence of usage because (1) we are interested in the level of interest and concern of birth control users regardless of actual usage and (2) other study designs (e.g., surveys) are better suited to studying usage rates.

We rely on sets of keywords to assign each text a primary birth control method. Because of their different structures, each platform requires its own set of keywords and matching techniques. All of these lexicons are made available for future research. To estimate the lexicon performance, we checked 450 documents balanced across Reddit posts, Twitter posts, and Twitter replies.⁶⁶6We omitted Reddit comments because of our method detection strategy, which defaulted to the method in the parent post if no method was found in the comment; and we omitted WebMD reviews because our method detection strategy already used hand-labeling of medication tags. Our precision and recall scores were perfect (1.0).

WebMD.

The URLs for WebMD reviews include a drugname parameter. This parameter includes the medication name followed by its class. We were able to classify 99.2% of the WebMD reviews by mapping these names to their corresponding method, and the remaining reviews were for methods outside our target three.

Reddit.

To assign Reddit posts and comments to the birth control methods, we use a custom set of keywords. In all cases, the text was assigned to the method that had the highest keyword count (of all the methods).⁷⁷7We could instead have assigned each Reddit post and comment to as many methods as were mentioned in its text, rather than assigning each post and comment only to the method mentioned most frequently. We chose the latter assignment because it better aligns with the WebMD assignments; WebMD reviews are explicitly assigned by the user to a single method (even if other methods are also mentioned). We do not find that these different assignment techniques affect our distributional results. If the highest keyword count was not for one of our target three methods, or if multiple methods were mentioned with equal frequency, we discarded the text. We follow a similar procedure for Reddit comments, but in cases where the comments do not contain a birth control keyword, we assign the comment to the birth control method of its parent post.

We developed the Reddit methods keyword set by (a) drawing terms from the WebMD keyword set, (b) drawing terms from the Twitter keyword set, (c) supplementing the WebMD and Twitter sets with general terms that are overloaded on Twitter but usable on Reddit (e.g., “pill”), and (d) iteratively running the assignments and examining the unassigned posts and comments. We were able to assign all but 4.6% of the Reddit posts and 32.6% of the Reddit comments to a birth control method. Manual examination of the unassigned posts reveals discussions of pregnancy scares, access (e.g., online appointments and prescriptions), and treatment of side effects with non-contraceptive medications. The remainder of these unassigned posts discussed topics that are adjacent to birth control, e.g., insurance, menstruation, but are not explicitly connected to a birth control method.

Twitter.

We use a more limited keyword set, derived from a similar survey of birth control tweets by Merz et al. (2020), to query for tweets about each of our target birth control methods. Our focus was on precision rather than recall, as the Twitter API requires a keyword match to retrieve tweets on a particular topic; if we included keywords like “pill”, our results would contain many false positives, unlike Reddit and WebMD where the topic is already constrained to birth control. After gathering our initial Twitter dataset, we then apply the full Reddit keyword set, using the same methods described for Reddit above. As with the Reddit data, we find that more texts can be assigned to only the pill, IUD, or implant than to a combination of methods; 12.7% of posts and 17.6% of the gathered tweets either mentioned multiple methods with equal frequency or most frequently mentioned a method not in our set of three target methods.

Side Effects Lexicon

To measure the frequency of discussions about side effect birth control, we develop a lexicon of terms and patterns. Because we do not use the side effects lexicon for data collection (unlike the methods lexicon), we can rely on one lexicon across our three datasets. As with the methods lexicon, we focus specifically on measuring the prevalence of discussions rather than the prevalence of experiences, as these are better measured via surveys (Nelson et al. 2017). The frequency of discussions could be influenced by the prevalence of experiences and the level of concern birth control users on this platforms have about particular side effects, as well as platform setting and a user’s individual goals. We select patterns that match any discussion of the side effect, whether or not it is mentioned in the affirmative.

We grounded our development of the side effects lexicon in prior work. We matched the side effect categories from Nelson et al. (2017) as closely as possible; this study conducted a nationally representative survey of U.S. birth control experiences and reported prevalences of side effect experiences across different birth control methods. In addition to these, we added lexicon categories for pain, skin changes, PMS, appetite changes, sexual partner feeling IUD strings, and heart attack. We identified these additional categories and patterns from topic models trained on our datasets and from other work; for example, we also drew on work by Barr (2010) that discusses a variety of side effects and their known frequencies and associations with different birth control methods. For each side effect, we then iteratively queried and made updates to the lexicon when we encountered false positives or false negatives.

Source	# Texts with Side Effect	% Texts with Side Effect
Reddit Posts	53,027 / 72,731	73%
Reddit Comments	119,780 / 238,568	50%
Twitter Posts	61,698 / 513,017	12%
Twitter Replies	47,049 / 244,140	19%
WebMD Reviews	16,176 / 18,110	89%

Table 2: The coverage of the side effects lexicon.

Lexicon evaluation.

To evaluate our lexicon, for each side effect we randomly selected a set of matching and non-matching texts, balanced across platforms and match status, resulting in a set of 1,040 texts. We then manually checked whether each text does or does not contain a discussion of the target side effect. Across the side effects, we found precision of 0.98 and recall of 0.96. We show the lexicon coverage in Table 2.

Comparing observations with survey responses.

We compare the distributions of side effect mentions to prior work surveying people in the U.S. about their experiences with birth control side effects. This allows us to compare the frequency of side effect experiences with the frequency of online discussions. Differences between these distributions can indicate healthcare needs that users are addressing via the internet. We compare to surveys (rather than controlled studies) because surveys better approximate the self-reported anecdotes and personal experiences shared online. We first converted each distribution to a ranking, where lower ranks represent greater percents of discussions $r_{platforms}$ or experiences $r_{survey}$ . We then find the difference between the ranks for each side effect, $d_{ranks}$ .

d_{ranks}=r_{survey}-r_{platforms}

(1)

These ranks avoid issues in directly comparing percents, which might on average be higher or lower. When $d_{ranks}$ is positive, it indicates more online discussion than expected given the frequency of reported side effect experiences.

Refer to caption — Figure 1: Distribution of the side effect mentions across the platforms and methods. Bars represent the percents of documents for the specified platform or method mentioning any side effect that also mention the specified side effect. See Table 2 for the denominator sizes. Error bars indicate the standard deviation over 20 bootstrapped samples of the datasets.

Sensemaking Topic Model

We compare sensemaking activity distributions through an unsupervised, bottom-up analysis using topic modeling. Topic modeling is an automatic method to identify prominent themes and discourses in a dataset (Blei, Ng, and Jordan 2003) and is a popular unsupervised technique for the analysis of online health communities (Nobles et al. 2018; Abebe et al. 2020). While some sensemaking themes might be shared across settings, other themes might be more prevalent in certain platforms and side effects. This process helps us to avoid biases in our data coding (which might overlook certain themes while over-representing others) by quickly revealing frequent themes in each dataset (Nelson 2020).

Model training.

We trained a latent Dirichlet allocation (LDA) topic model (Blei, Ng, and Jordan 2003) on each of our datasets, by combining the Reddit posts and comments and the Twitter posts and replies for training, resulting in three models, using MALLET⁸⁸8http://mallet.cs.umass.edu/ for training. LDA remains a highly consistent and reliable model (Harrando, Lisena, and Troncy 2021; Hoyle et al. 2022), especially when trained via Gibbs sampling for smaller datasets. We balanced each training set, sampling 8,000 documents for each method for each of the Reddit and Twitter datasets and 800 documents for each method for the WebMD dataset (see Figure 2 for method distributions). By balancing the training set, we avoid weighting the topics toward a certain method.

We removed a set of frequent stopwords, numbers, punctuation, and duplicate documents from the training sets. Removing stopwords and duplicate documents has been shown to improve the legibility of the final topics, whereas stemming can be harmful (Schofield and Mimno 2016; Schofield, Magnusson, and Mimno 2017; Schofield, Thompson, and Mimno 2017). To avoid capitalization discrepancies, we lowercase all text. We experimented with different numbers of topics and found 35 to be interpretable across the datasets; at this number of topics, we observed topics that were neither too fine-grained (e.g., splitting a single theme across multiple topics) nor too high level (e.g., combining themes that should be separated), and that produced reasonable evaluation scores (see below). However, we emphasize that there is no “correct” number of topics and that this method is used for exploration and interpretation.

Model evaluation.

While we do not have space here to list the full sets of topics for each dataset, we provide the topics, most probable words, and example paraphrased documents in our code repository for manual examination. We report human evaluation of our topics following the recommendations in Hoyle et al. (2021). Using the word intrusion task (Chang et al. 2009), we show two expert annotators (the first two authors) and one non-expert annotator a set of the four most probable words plus an “intruder” word that has low probability for the current topic but high probability for another topic. We report the proportion of topics for which the annotators identified the intruder.

We found that our Reddit and WebMD topics have performance much higher than a random baseline of 0.2 (annotator accuracies for Reddit: 0.71, 0.74, 0.77, WebMD: 0.54, 0.66, 0.77) while the Twitter topics have lower performance but are still substantially above the random baseline (Twitter: 0.46, 0.46, 0.51). The first (non-expert) annotator consistently had lower scores. The lower performance on Twitter is expected, as text processing methods are notoriously challenged by the short text lengths and non-standard language (Gimpel et al. 2011), and the short tweets require contextual knowledge to interpret. This vulnerability of the word intrusion task to esoteric topics is noted by Hoyle et al. (2021).

We also calculate the “UMass” Coherence: the log probability that a document containing at least one instance of a higher-ranked word also contains at least one instance of a lower-ranked word (Röder, Both, and Hinneburg 2015). We find the highest mean coherence scores across topics for Reddit ( $-416$ ), lower scores for WebMD ( $-577$ ), and lowest scores for Twitter ( $-712$ ). We note the criticisms of this and other automatic metrics in Hoyle et al. (2021); in comparison to human evaluation, automatic metrics can exaggerate differences between models.

Identifying sensemaking themes.

Across the datasets, we find that many of the topics discuss birth control methods (including one method in their 10 most probable words), side effects, pregnancy, and access (costs, appointments). We then identify a series of cross-cutting sensemaking topics. These topics include discussions of information seeking/sharing, educational links and resources, how-to explanations, experience seeking/sharing, emotional support, and other sensemaking-related discussions. Two researchers independently coded the topics as more or less related to sensemaking. The researchers then conferred and agreed upon a final set of topics and assigned them to thematic clusters. During this coding, we relied on the sensemaking definition from Andalibi and Garcia (2021): “sensemaking describes how individuals make sense of complex phenomena by constructing mental models that draw on new or existing experiences, information, emotions, ideas, and memories.” We also drew inspiration from prior work studying sensemaking in online healthcare communities, particularly those works also focused on reproductive healthcare (see §Related Work). We further explore and validate these sensemaking topics by hand-labeling a small subsection of the data with social support goals (Yang et al. 2019). After coding 150 documents for each dataset, we measured the agreement between the annotators using Krippendorff $\alpha$ , as each document could receive zero, one, or more labels. Our agreement was acceptable, with a score across the labels of 0.74.⁹⁹9Lower scores are not surprising for subjective language labeling tasks (Artstein and Poesio 2008; Godwin and Piwek 2016), and our scores are substantially higher than the agreement scores for a very similar classification task in Rivas et al. (2020).

To compare the topics across the platforms, we aligned the topics from the different models using Jensen-Shannon divergence (JSD) for the word distributions associated with each pair of topics. JSD is frequently used in prior work to compare topic distributions (Hall, Jurafsky, and Manning 2008; Fang et al. 2012). After manual examination of the ranked topic pairs, we categorized topics with JSD scores below 0.6 as aligned across the datasets, and those with scores above 0.8 we considered diverging.

Results

Results: Birth Control Methods Across Platforms

We compare the rates of discussion of the different birth control methods across platforms and over time. We find that the types of questions that users ask and the type of information they share differs based on birth control method. We measure the rise and fall of discussions of different birth control methods over time, and we compare these distributions across platforms. Importantly, these patterns do not necessarily indicate real-world increases or decreases in use of different methods, but they do show the concerns and interests of users on these platforms.

Figure 2 shows the frequency of documents for each medication method by platform over time. The pill is the most popular method in both Reddit Posts and WebMD Reviews, while the pill and IUD are tied in Reddit Comments. Discussions of the pill on WebMD are much more frequent that discussions of other birth control methods and do not show a rise in interest for the IUD, counter to both the other datasets and our original hypothesis. On Twitter, the pill begins as the most popular but is replaced by the IUD in posts, while replies always center on the IUD. Twitter replies also increase sharply after 2016, differing from the Twitter posts (so is unlikely to be a symptom of our keywords). This suggests that the IUD generates more discussion on Twitter, especially post-2016, compared to the other birth control methods. On Reddit, the number of posts discussing the pill and IUD are similar, with slightly more posts discussing the pill, though this difference is erased in the comments. The implant is always the least discussed of the three methods.

Results: Side Effects Across Platforms

Figure 2 shows the distributions of the side effects across the different platforms, where frequency proportions are calculated by dividing the number of texts mentioning the specified side effect by the total number of texts mentioning any side effect (i.e., $p(s_{i}|a)$ where $s_{i}$ is the specific side effect and $a$ is any side effect). Discussions of different methods have changed dramatically over time on Twitter, with discussions of the IUD rising and the pill falling. The pill remains most most mentioned in Reddit posts.

Figure 1 shows the distribution of side effects across platforms. We find that pain, cramps, menstrual bleeding irregularities, mood changes, and skin conditions are the most commonly discussed side effects. In particular, pain is consistently and frequently discussed across the platforms and is an outlier among the side effects. Discussions of stroke are more frequent in Twitter posts than in the other datasets, but for all other side effects, Twitter has the lowest discussion frequencies (perhaps because of stigma around publicly sharing such information compared to the more frequently pseudonymous settings on Reddit and WebMD). Except for pain, menstrual bleeding, and severe effects, WebMD has the highest frequency of side effects discussions compared to the other platforms. In particular, WebMD has higher frequencies for discussions of mood changes, skin conditions, and headache discussions. Reddit has has the highest frequency of discussions for menstrual bleeding irregularities.

Figure 1 also compares the side effects by birth control method. Across the birth control methods, we again find that pain is a frequently mentioned side effect, but it is most frequently mentioned in discussions that also mention the IUD rather than the pill or implant. The implant is the only method shown to cause weight gain (Barr 2010), so this association is expected—but the discussions of weight gain for the other methods are less expected and could indicate that this potential side effect is a concern across methods. Menstrual bleeding, mood changes, headache, and libido are least often discussed in with the IUD, as are general mentions of side effects, while nausea, stroke, and skin conditions are most often discussed in discussions of the pill.

Results: Comparison to Observed Distributions

Figure 3 shows the differences between our observed distributions of side effect mentions and the reported distributions of side effect experiences in Nelson et al. (2017) (described above). We find large differences between how frequently side effects are discussed online compared with how frequently they are reported in the survey from Nelson et al. (2017). For example, strokes are rarely experienced according to Nelson et al. (2017), but when mentioned with the pill, they are discussed more frequently on Twitter in comparison to other side effects; this is unsurprising, since sensational topics are frequently discussed on Twitter. Mood changes are more likely to be discussed across the platforms for the implant in comparison to the survey data. Dizziness is more likely to be discussed for the IUD than expected, while weight gain is universally discussed less frequently than expected. These patterns indicate cases in which users turn to the internet more or less frequently than expected based on their self-reported experiences in the survey.

Results: Sensemaking Across Platforms

Our final set of sensemaking themes included: storytelling (e.g., implant insertion and removal on Twitter), risk analysis (e.g., risks of serious side effects on WebMD), timing and calculations (e.g., daily timing of pill on WebMD), method and dose comparison (e.g., hormone dosages and comparison on Reddit), causal reasoning (e.g., weight changes and causes on Reddit), and information and explanations (e.g., research on the male pill on Twitter). We show four of these themes (others omitted for space) in Figure 4, with the topics from each platform included in that theme and their probabilities for each birth control method. We observe a greater number of storytelling topics for the IUD and implant than for the pill, and examination of these topics shows that insertion and removal experiences fuel this pattern. Each method is associated with risks and is included in method comparisons, but the pill in particular is included in discussions of timing and calculations.

Using Jensen-Shannon divergence to compare topics across the platforms, we find that the most aligned ( $JSD<0.6$ ) topic categories were: weight changes, general side effects, IUD insertion experiences, implant insertion experiences, menstrual timing and cycles, and bleeding changes. Each of our three datasets included at least one representative topic from these categories.

The most diverging ( $JSD>0.8$ ) topics were: causes and side effects of vaginal infections and explanations of how birth control works on Reddit; IUD jokes and random, viral folk stories, and implant news about unexpected experiences on Twitter; and treating or causing other medical issues and secondhand advice and experiences on WebMD. These topics were the most unique to their training dataset, without directly comparable topics in the other datasets.

Discussion

Methods and side effects across platforms.

In response to our first research question, we find that birth control discussions on Twitter, WebMD, and Reddit substantially differ in their distributions of methods and side effect mentions. We cannot claim that using a method or experiencing a side effect cause people to choose a specific platform, but our observations add detail and sometimes contradict prior findings. For example, in a study of general healthcare information seeking, De Choudhury, Morris, and White (2014) found that people more often use search engines for serious and stigmatized conditions and more often use Twitter to discuss symptoms. We do not include search engines in our study, but in our comparison across platforms, we find that Twitter has a higher frequency of severe side effect discussions for birth control, while WebMD and Reddit have higher frequencies of general side effect discussions. Our focus on birth control might explain these variations, as many of the birth control side effects are themselves highly stigmatized (e.g., menstrual bleeding, vaginal discharge). The discussion of severe side effects on Twitter is likely related to Twitter’s tendency toward sensational content, and could also explain the relative frequency of IUD discussion on Twitter; the IUD has been associated with both severe side effects and potential legal bans (Nobles, Dredze, and Ayers 2019). These findings emphasize the importance of considering platform setting when studying online patterns related to specific health conditions, medications, and side effects.

Online sensemaking practices of birth control users.

In response to our second research question, our identification and comparison of topics that align with sensemaking themes reveals important activities specific to birth control and to the different platforms and methods. Prior work has found that online communities focused on reproductive healthcare (e.g., pregnancy, vulvodynia) employ strategies related to validation (Andalibi and Garcia 2021), information management (Patel et al. 2019; Young and Miller 2019; Andalibi and Garcia 2021), personal tracking (Chopra et al. 2021), and identifying causation (Patel et al. 2019). We found these themes again in our birth control communities, with variations; for example, personal tracking is also a prominent theme in birth control discussions, but it is focused on pill timings and calculations and on self-observations of side effects. Identifying causation is a common concern for those experiencing infertility (Patel et al. 2019), and we find this theme again in our datasets but focused on side effects like weight gain. This combination of themes reflects the unique problems facing birth control users, as they choose between “least bad” options, struggle to identify and treat side effects, determine risk of pregnancy given their circumstances, and avoid rare but alarming outcomes like stroke and heart attack.

Storytelling, pain, and the IUD.

We would like to particularly highlight storytelling as an important sensemaking strategy for birth control users. Storytelling is known to help communities work through trauma (Tangherlini 2000), but birth control users employ storytelling specifically to address physical pain. Pain is a major cross-cutting theme across the platforms; it is the most frequently and consistently discussed side effect, and it is most often mentioned alongside the IUD. While Barr (2010) identifies pain as the second most common side effect (after bleeding changes) and Dickerson et al. (2013) identifies pain as the most common side effect for the IUD and second most common side effect for the implant, we find a much larger gap between pain and the next most frequent side effects in our analysis. Pain is mentioned frequently in posts about the IUD and in posts whose probable topics are about seeking and sharing IUD insertion stories. Pain is inherently a subjective experience that cannot be precisely communicated (Scarry 1987), but sharing of personal experiences provides one way for a community to build a sense of what is normal or to be expected (Patel et al. 2019; Andalibi and Garcia 2021). We suggest that there is an urgent unmet need for (a) honest education and preparation before IUD insertions and (b) pain treatment options during this procedure.

Online discussions differ from survey reports.

In comparison to the survey results in Nelson et al. (2017), we find not only large differences in the frequency at which different side effects are discussed but also differences across the platforms. It could be that the differences from the survey are due to the demographic distribution opting into the survey versus those opting to post online; future work surveying the demographic distribution of these users would address this question, but our results take a first step at measuring differences between these settings. We find, e,g., while mood changes are discussed at similar frequencies in comparison to the survey data across the platforms, other side effects like strokes, bloating, fatigue, bleeding, and dizziness might be discussed more, less, or at equal frequency ranks to the reports in Nelson et al. (2017), depending on the platform. Bloating is less frequently mentioned on all the platforms and for all the methods in comparison to the survey data, perhaps indicating a general lack of concern about this side effect in contrast to its reported frequency. But bloating is much less frequent on Twitter for the pill and implant, which could also indicate that this particular setting is less suited to discussing this side effect, perhaps because of the embarrassing or intimate nature of this side effect.

Stigma, privacy, and contextual disclosures.

Birth control can be a controversial, stigmatized, and intimate topic. This can lead birth control users to seek out additional information privately. For example, in a set of interviews of young Black and Hispanic women, Yee and Simon (2010) found that a greater number reported seeking decision-marking support on the internet, citing its privacy, in comparison to other sources of information (e.g., talking to physicians, reading provided information). Interpreting side effects, analyzing the risk of pregnancy, or normalizing a painful experience require disclosing personal details and stories. Birth control users might analyze the risk and benefit of making these disclosures in certain settings and to certain audiences. While social penetration theory (Altman and Taylor 1973) posits that more disclosures are possible as social bonds deepen, prior work has also found that intimate language can be frequent among both close connections and strangers (but not in between) (Pei and Jurgens 2020). The variations we observe across methods, side effects, and sensemaking practices could be indicative of platform affordances for privacy, audience size, and anonymity, each of which can affect decisions to self-disclose. For example, we found that side effect discussions are less frequent on Twitter, where users are facing a much larger and non-specialized audience, unlike the other platforms. Giving users more platform-specific tools to control their audience size and membership (Mondal et al. 2014) could allow for more productive discussions of this sensitive topic.

Other factors influencing platform decisions.

It is important to note that decisions to seek information online can correlate with demographic characteristics. For example, in a survey of U.S. young adults, those with a sexual risk history (early sexual activity, involvement in an unintended pregnancy) less frequently reported using the internet as a source and more frequently reported seeking information from a doctor/nurse, and men more frequently reported using the internet than women (Khurana and Bleakley 2015). It is also possible that users follow a birth control journey, where different needs at different points in one’s journey can lead one to different platforms, as has been reported for other healthcare topics (Sannon et al. 2019; Andalibi and Forte 2018). These journeys can intersect with methods; for example, while the pill is a popular first method, many people report switching to the IUD as they gain more experience with birth control (Nelson et al. 2017). This would mirror the journeys of those with invisible chronic illnesses who move from one platform to another as their needs evolve and as they grow more comfortable with self-disclosure (Sannon et al. 2019). This is mirrored in intra-community research that models user trajectories in online cancer support groups, finding that users often transition from information-seeking to information-sharing roles over time (Yang et al. 2019).

Recommendations.

Our results show that patterns found in one platform are not necessarily replicable on other online platforms, even when grounded in the same health topic. We recommend that social media researchers compare results across multiple platforms to increase confidence in shared patterns. If relying on a single platform, its choice should be guided by an understanding of which platforms might be more less or conducive to discussions of the desired methods, side effects, or sensemaking activities. Our work also highlights benefits of mixing computational tools like topic modeling with lexicon-based methods and hand-annotation, and how these methods can be used to characterize difficult-to-identify themes like sensemaking practices; we recommend taking this careful approach, putting unsupervised results in context with fine-grained measurements.

Broader Impacts, Ethics, and Limitations

Our study was considered exempt under our institution’s IRB. However, while Reddit, Twitter, and WebMD posts and replies are “public,” they can contain highly personal information, requiring a balance between potential harms and potential benefits to the community, as described in the guiding principles of the Belmont Report: respect for persons, beneficence, and justice.¹⁰¹⁰10https://www.hhs.gov/ohrp/regulations-and-policy Considering possible harms, e.g., re-identification of those using stigmatized medications, we do not collect any user-specific information, and we do not infer medical conditions for individual users; instead, we rely on patterns averaged across many users. We also anonymize and paraphrase any direct quotations. To protect users’ agency to edit or delete their data at its original source, we release our data collection lexicons but not copies of the collected data. We balance these concerns and protective actions against the benefits of this research; among other benefits, our work highlights the unmet pain treatment needs of a population known to be discriminated against by physicians (Samulowitz et al. 2018) and examines the kinds of support needed by those facing difficult healthcare choices.

Limitations

Social media does not necessarily represent offline events, and unlike the surveys we use for comparison, we cannot control for demographics. We study texts written in English and platforms that attract a U.S. audience, as the authors are all most familiar with this setting. We focus on one specific community out of many, and we do not expect that the findings in this paper will necessarily generalize outside of the U.S. or to other online spaces; indeed, one of our results indicates that different platforms display different patterns. Finally, averaging over posts allows us to track patterns and make comparisons but can also reduce nuance. Our work is best read in conjunction with ethnographic studies like Homewood and Heyer (2017) and Daley (2014), which highlight individual voices of birth control users.

References

Abebe et al. (2020) Abebe, R.; Giorgi, S.; Tedijanto, A.; Buffone, A.; and Schwartz, H. A. A. 2020. Quantifying Community Characteristics of Maternal Mortality Using Social Media. In Proceedings of WWW.
Altman and Taylor (1973) Altman, I.; and Taylor, D. A. 1973. Social penetration: The development of interpersonal relationships.
Andalibi and Forte (2018) Andalibi, N.; and Forte, A. 2018. Announcing Pregnancy Loss on Facebook: A Decision-Making Framework for Stigmatized Disclosures on Identified Social Network Sites. In Proceedings of CHI.
Andalibi and Garcia (2021) Andalibi, N.; and Garcia, P. 2021. Sensemaking and Coping After Pregnancy Loss: The Seeking and Disruption of Emotional Validation Online. In Proceedings of CSCW.
Artstein and Poesio (2008) Artstein, R.; and Poesio, M. 2008. Inter-coder agreement for computational linguistics. Computational Linguistics.
Barr (2010) Barr, N. G. 2010. Managing adverse effects of hormonal contraceptives. American Family Physician.
Bietti, Tilston, and Bangerter (2019) Bietti, L. M.; Tilston, O.; and Bangerter, A. 2019. Storytelling as Adaptive Collective Sensemaking. Topics in Cognitive Science.
Blei, Ng, and Jordan (2003) Blei, D. M.; Ng, A. Y.; and Jordan, M. I. 2003. Latent Dirichlet allocation. JMLR.
Carron-Arthur et al. (2015) Carron-Arthur, B.; Ali, K.; Cunningham, J. A.; and Griffiths, K. M. 2015. From help-seekers to influential users: a systematic review of participation styles in online health communities. JMIR.
Chang et al. (2009) Chang, J.; Gerrish, S.; Wang, C.; Boyd-Graber, J. L.; and Blei, D. M. 2009. Reading tea leaves: How humans interpret topic models. In Advances in NeurIPS.
Chopra et al. (2021) Chopra, S.; Zehrung, R.; Shanmugam, T. A.; and Choe, E. K. 2021. Living with Uncertainty and Stigma: Self-Experimentation and Support-Seeking around Polycystic Ovary Syndrome. In Proceedings of CHI.
Daley (2014) Daley, A. M. 2014. What influences adolescents’ contraceptive decision-making? A meta-ethnography. Journal of Pediatric Nursing.
Daniels and Abma (2020) Daniels, K.; and Abma, J. C. 2020. Current Contraceptive Status Among Women Aged 15–49: United States, 2017–2019. NCHS Data Brief No. 388.
De Choudhury, Morris, and White (2014) De Choudhury, M.; Morris, M. R.; and White, R. W. 2014. Seeking and Sharing Health Information Online: Comparing Search Engines and Social Media. In Proceedings of CHI.
Dickerson et al. (2013) Dickerson, L. M.; Diaz, V. A.; Jordan, J.; Chirina, S.; Goddard, J. A.; Carr, K. B.; and Carek, P. J. 2013. Satisfaction, early removal, and side effects associated with long-acting reversible contraception. Family Medicine.
Fang et al. (2012) Fang, Y.; Si, L.; Somasundaram, N.; and Yu, Z. 2012. Mining Contrastive Opinions on Political Texts Using Cross-Perspective Topic Model. In Proceedings of ICWSM.
Genuis and Bronstein (2017) Genuis, S. K.; and Bronstein, J. 2017. Looking for “normal”: Sense making in the context of health disruption. JASIST.
Gimpel et al. (2011) Gimpel, K.; Schneider, N.; O’Connor, B.; Das, D.; Mills, D.; Eisenstein, J.; Heilman, M.; Yogatama, D.; Flanigan, J.; and Smith, N. A. 2011. Part-of-Speech Tagging for Twitter: Annotation, Features, and Experiments. In Proceedings of ACL.
Godwin and Piwek (2016) Godwin, K.; and Piwek, P. 2016. Collecting reliable human judgements on machine-generated language: The case of the QG-STEC data. In Proceedings of INGL.
Hall, Jurafsky, and Manning (2008) Hall, D.; Jurafsky, D.; and Manning, C. D. 2008. Studying the History of Ideas Using Topic Models. In Proceedings of EMNLP.
Harrando, Lisena, and Troncy (2021) Harrando, I.; Lisena, P.; and Troncy, R. 2021. Apples to Apples: A Systematic Evaluation of Topic Models. In Proceedings of RANLP.
Homewood and Heyer (2017) Homewood, S.; and Heyer, C. 2017. Turned On / Turned Off: Speculating on the Microchip-Based Contraceptive Implant. In Proceedings of DIS.
Hoyle et al. (2021) Hoyle, A.; Goel, P.; Hian-Cheong, A.; Peskov, D.; Boyd-Graber, J.; and Resnik, P. 2021. Is Automated Topic Model Evaluation Broken? The Incoherence of Coherence. Advances in NeurIPS.
Hoyle et al. (2022) Hoyle, A. M.; Goel, P.; Sarkar, R.; and Resnik, P. 2022. Are Neural Topic Models Broken? ArXiv, abs/2210.16162.
Khurana and Bleakley (2015) Khurana, A.; and Bleakley, A. 2015. Young adults’ sources of contraceptive information: variations based on demographic characteristics and sexual risk behaviors. Contraception.
Mamykina, Nakikj, and Elhadad (2015) Mamykina, L.; Nakikj, D.; and Elhadad, N. 2015. Collective Sensemaking in Online Health Forums. In Proceedings of CHI.
Manzer and Bell (2022) Manzer, J. L.; and Bell, A. V. 2022. “Did I Choose a Birth Control Method Yet?”: Health Care and Women’s Contraceptive Decision-Making. Qualitative Health Research.
Merz et al. (2020) Merz, A. A.; Gutiérrez-Sacristán, A.; Bartz, D.; Williams, N. E.; Ojo, A.; Schaefer, K. M.; Huang, M.; Li, C. Y.; Sandoval, R. S.; Ye, S.; et al. 2020. Population attitudes toward contraceptive methods over time on a social media platform. American Journal of Obstetrics and Gynecology.
Mondal et al. (2014) Mondal, M.; Liu, Y.; Viswanath, B.; Gummadi, K. P.; and Mislove, A. 2014. Understanding and specifying social access control lists. In Symposium On Usable Privacy and Security.
Nahata, Chelvakumar, and Leibowitz (2017) Nahata, L.; Chelvakumar, G.; and Leibowitz, S. 2017. Gender-affirming pharmacological interventions for youth with gender dysphoria: when treatment guidelines are not enough. Annals of Pharmacotherapy.
Nelson et al. (2017) Nelson, A. L.; Cohen, S.; Galitsky, A.; Hathaway, M.; Kappus, D.; Kerolous, M.; Patel, K.; and Dominguez, L. 2017. Women’s perceptions and treatment patterns related to contraception: results of a survey of US women. Contraception.
Nelson (2020) Nelson, L. K. 2020. Computational grounded theory: A methodological framework. Sociological Methods & Research.
Nobles et al. (2018) Nobles, A.; Dreisbach, C.; Keim-Malpass, J.; and Barnes, L. 2018. “Is this an STD? Please help!”: Online information seeking for sexually transmitted diseases on Reddit. In Proceedings of ICWSM.
Nobles, Dredze, and Ayers (2019) Nobles, A. L.; Dredze, M.; and Ayers, J. W. 2019. “Repeal and replace”: increased demand for intrauterine devices following the 2016 presidential election. Contraception.
Nobles et al. (2020) Nobles, A. L.; Leas, E. C.; Dredze, M.; and Ayers, J. W. 2020. Examining Peer-to-Peer and Patient-Provider Interactions on a Social Media Community Facilitating Ask the Doctor Services. In Proceedings of ICWSM.
Patel et al. (2019) Patel, D.; Blandford, A.; Warner, M.; Shawe, J.; and Stephenson, J. 2019. “I feel like only half a man” Online Forums as a Resource for Finding a “New Normal” for Men Experiencing Fertility Issues. In Proceedings of CSCW.
Pei and Jurgens (2020) Pei, J.; and Jurgens, D. 2020. Quantifying Intimacy in Language. In Proceedings of EMNLP.
Perrin and Anderson (2019) Perrin, A.; and Anderson, M. 2019. Share of US adults using social media, including Facebook, is mostly unchanged since 2018. Pew Research Center.
Pfeffer et al. (2022) Pfeffer, J.; Mooseder, A.; Hammer, L.; Stritzel, O.; and Garcia, D. 2022. This Sample seems to be good enough! Assessing Coverage and Temporal Reliability of Twitter’s Academic API. ArXiv, abs/2204.02290.
Pirolli and Card (2005) Pirolli, P.; and Card, S. 2005. The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis. In Proceedings of International Conference on Intelligence Analysis.
Polis et al. (2016) Polis, C. B.; Bradley, S. E.; Bankole, A.; Onda, T.; Croft, T.; and Singh, S. 2016. Typical-use contraceptive failure rates in 43 countries with Demographic and Health Survey data: summary of a detailed report. Contraception.
Rivas et al. (2020) Rivas, R.; Sadah, S. A.; Guo, Y.; and Hristidis, V. 2020. Classification of Health-Related Social Media Posts: Evaluation of Post Content–Classifier Models and Analysis of User Demographics. JMIR Public Health and Surveillance.
Röder, Both, and Hinneburg (2015) Röder, M.; Both, A.; and Hinneburg, A. 2015. Exploring the Space of Topic Coherence Measures. In Proceedings of WSDM.
Russo et al. (2013) Russo, J. A.; Parisi, S. M.; Kukla, K.; and Schwarz, E. B. 2013. Women’s information-seeking behavior after receiving contraceptive versus noncontraceptive prescriptions. Contraception.
Samulowitz et al. (2018) Samulowitz, A.; Gremyr, I.; Eriksson, E. M.; and Hensing, G. 2018. “Brave Men” and “Emotional Women”: A Theory-Guided Literature Review on Gender Bias in Health Care and Gendered Norms towards Patients with Chronic Pain. Pain Research & Management.
Sannon et al. (2019) Sannon, S.; Murnane, E. L.; Bazarova, N. N.; and Gay, G. 2019. ”I Was Really, Really Nervous Posting It”: Communicating about Invisible Chronic Illnesses across Social Media Platforms. In Proceedings of CHI.
Scarry (1987) Scarry, E. 1987. The body in pain: The making and unmaking of the world. Oxford University Press, USA.
Schindler (2013) Schindler, A. E. 2013. Non-contraceptive benefits of oral hormonal contraceptives. International Journal of Endocrinology and Metabolism.
Schofield, Magnusson, and Mimno (2017) Schofield, A.; Magnusson, M.; and Mimno, D. 2017. Pulling Out the Stops: Rethinking Stopword Removal for Topic Models. In Proceedings of EACL.
Schofield and Mimno (2016) Schofield, A.; and Mimno, D. 2016. Comparing Apples to Apple: The Effects of Stemmers on Topic Models. TACL.
Schofield, Thompson, and Mimno (2017) Schofield, A.; Thompson, L.; and Mimno, D. 2017. Quantifying the Effects of Text Duplication on Semantic Models. In Proceedings of EMNLP.
Tangherlini (2000) Tangherlini, T. R. 2000. Heroes and lies: Storytelling tactics among paramedics. Folklore.
Weick, Sutcliffe, and Obstfeld (2005) Weick, K. E.; Sutcliffe, K. M.; and Obstfeld, D. 2005. Organizing and the process of sensemaking. Organization Science.
Yang et al. (2019) Yang, D.; Kraut, R. E.; Smith, T.; Mayfield, E.; and Jurafsky, D. 2019. Seekers, providers, welcomers, and storytellers: Modeling social roles in online health communities. In Proceedings of CHI.
Yee and Simon (2010) Yee, L.; and Simon, M. 2010. The role of the social network in contraceptive decision-making among young, African American and Latina women. Journal of Adolescent Health.
Yoost (2014) Yoost, J. 2014. Understanding benefits and addressing misperceptions and barriers to intrauterine device access among populations in the United States. Patient Preference and Adherence.
Young and Miller (2019) Young, A. L.; and Miller, A. D. 2019. ”This Girl is on Fire”: Sensemaking in an Online Health Community for Vulvodynia. In Proceedings of CHI.
Yun (2019) Yun, H. 2019. The Decline of WebMD? NYC Data Science Academy: Data Science Blog.
Zhang, N. Bazarova, and Reddy (2021) Zhang, R.; N. Bazarova, N.; and Reddy, M. 2021. Distress Disclosure across Social Media Platforms during the COVID-19 Pandemic: Untangling the Effects of Platforms, Affordances, and Audiences. In Proceedings of CHI.